[The following is from my monthly reflections journal for the UBC Scholarship of Teaching and Learning Leadership program I’m attending this year (a year-long workshop focused in part on general improvements in pedagogy, but also in large part on learning about SoTL and developing a SoTL project). Warning–a long post!]
I recently read an article by Ursula Wingate of the Department of Education and Professional Studies at King’s College London, entitled The impact of formative feedback on the development of academic writing,” Assessment & Evaluation in Higher Education Vol. 35, No. 5 (August 2010): 519-533. I am very interested in this article b/c it deals with a question I myself have wondered, in relation to my teaching in Arts One: why do some students improve so much in their writing over the course of the year, and why do some fail to do so? I was thinking this might be a future SoTL project for me, and I’m glad to see that there is literature on this…so I’ve got some work in the future, looking through that literature!
In this article Wingate reports on a study focused on two research questions:
(1) Can improvements in student writing be linked to the use of the formative feedback?
(2) What are the reasons for engaging or not engaging with the assessment feedback? (p. 523)
The sample was a set of essays by 62 students in a first year course focused in part on writing (the course was part of a program in applied linguistics). Comments on essays were coded according to which category they belong to in terms of assessment criteria, and the researchers compared the comments in each category from an early essay to those on a later essay from each student. They separated students into three main categories: those whose essays showed equally high achievement over the two assignments, those whose marks improved by at least 10% between the two essays, and those whose marks didn’t show much of a difference between the two essay (+ or – 5%). After this separation they ended up with 39 essays. They list a few reasons why they didn’t include other students in the study, but those aren’t crucial to what I want to comment on here (I think). Mostly they were looking to find why some students improve a lot and some don’t, so the second two groups make sense, but the first group (those who were consistently high achievers) were included to see if they could find out in interviews some useful information about how/why they are academically engaged.
They found, unsurprisingly, that the comments on the second essays for those who improved significantly had fewer negative things to say, or suggestions for improvement (this was true for those who were consistently scoring well over the two essays too), but that those who didn’t improve much had similar numbers and types of comments on the second essay as on the first, showing that those marking the essays still saw the same kinds of problems on the second as on the first.
They then invited students from those three groups for interviews, and they ended up with 12 interviews. They asked two sets of questions:
The first set of interview questions aimed at eliciting information on students’ engagement with the feedback. It was assumed that engaged students would remem- ber well the details of the feedback comments and be able to describe how they used them. The following questions were asked: (1) Which comments did you receive on your assignments? (2) Did you find the comments useful? (3) How did you use them?
The second set of questions tried to elicit information on motivation and self- perception as possible reasons for the engagement with feedback comments: (4) Are you enjoying the programme? (5) What do you find difficult in writing at university? (p. 525)
They found, again unsurprisingly, that those whose essays had improved significantly seemed to have paid more attention to the comments than those whose essays had not. The first group were able to remember some of the specific comments on the first essay and they were able to say fairly precisely how they tried to improve for the second (including specific actions they took, like asking others to read over their essays, looking at writing reference books, etc.). Those students whose essays didn’t seems to change much were less able to recall the specific comments and didn’t say a lot about how they used them to improve their essays.
But what I find most interesting about this study (since some of it is, well, what you might expect) is the discussion of why some students may be paying less attention to comments than others. The answers they found in their interviews (though of course there are likely others) had to do in part with engagement with their programs of study and their confidence in their abilities as writers. It is the latter that struck me the most in reading this study, as it’s something I hadn’t really thought about before, but when I did it seemed so obvious…why hadn’t I considered it? Several of the interviewees whose writing hadn’t improved much said that they didn’t feel they were up to the task of university writing, that they were very worried about their perceived lack of ability and that anxiety hindered their writing process. It makes sense that if one gets a low mark and is already feeling a lack of confidence, that one may not be motivated to pay attention to the comments or feel that one could possibly improve even if one did.
Of course, to note that this is a possible issue is one thing; to determine what might be done about it is quite another, since it’s very complex. But the article makes a couple of suggestions that those who mark and give comments on essays might consider. I find them persuasive. The researchers note that in their analysis of particular comments, they found a discrepancy in the tone and number of critical comments on essays:
…the comments for the high-achieving students contained far more hedging devices such as ‘tend to’, ‘occasional’, ‘might/would have enhanced’, whilst the comments for lower achieving students contained more imperatives, such as ‘use headings’, ‘see hand- book’ and modals such as ‘you must’, ‘you need to’. Also, the high-achieving students received considerably more positive comments than the students in the other two cate- gories. (p. 526)
The “other two categories” are those students whose essays improved significantly from the first to the second (and who didn’t start out with high scores), and those whose essays did not improve significantly (and who also didn’t start out with high scores).
I had never considered thinking about my own comments in this way, but I think I might be doing the same thing. In my desire to provide students with as many hints as possible for how to improve, I am probably giving the essays that are weaker more negative comments (b/c there is more to improve) than those that are stronger–which is of course reasonable. But I am most likely not also giving the same amount of positive comments to the papers that are weaker as I am to those that are stronger. In fact, the more I think about it, I’m probably mostly focusing on the negative. This is through my desire to help, of course, because I want students to have specific feedback on what they can improve. It’s also because writing is so complex that many things can go wrong and I’m hoping students can fix them all! I also haven’t paid attention to the style of my comments between stronger and weaker essays–am I more hedging in my comments on the former and more imperative in my comments on the latter? I’ll have to check.
I also find another suggestion in the article particularly relevant to me, and something I will consider further in my own marking:
The approach taken by the tutors in this research was to comment explicitly on every weakness discovered in an assignment. The feedback was designed to avoid some pitfalls such as academic discourse that might be difficult to understand for students. Nevertheless, it was obviously not balanced enough to prevent weaker students being discouraged. It is therefore important to explore a more effective delivery of feedback that avoids weaker students receiving an overwhelming amount of criticism on various topics at one time.
I am pretty sure I probably overwhelm some students with my voluminous comments. I always thought it was a good thing to give more, and more specific comments rather than fewer and vague ones because: (a) I thought it would help students see better what they need to improve; (b) it shows I am really thinking about their essays carefully and trying to understand their arguments, and I think I actually am understanding most of them by reading this carefully; (c) students can then get a better sense of why they got the mark they did. But it is possible I am giving “an overwhelming amount of criticism on various topics at one time,” and that this can be discouraging to some students.
Wow…what a thought-provoking idea for me, one that may change my marking practices, though I’ll have to give it more thought.
