This is part of a series of posts in which I summarize and comment on research literature about different methods of doing peer assessment. Earlier posts in this series can be found here and here, and part 1 corresponding to this particular post is here.
In this post I summarize, as briefly as I can, a complex study on differences between how students speak to each other when doing peer assessment when it’s in person versus on a discussion board (mostly asynchronous, but students also did some posting to the discussion boards in a nearly synchronous environment, during class time).
Hewett, B. (2000) Characteristics of Interactive Oral and Computer-Mediated Peer Group Talk and Its Influence on Revision, Computers and Composition 17, 265-288. DOI: 10.1016/S8755-4615(00)00035-9
This study looked at differences between ways peers talk in face to face environments and computer-mediated environments (abbreviated in the article as CMC, for computer-mediated communication). It also looked at whether there are differences in the ways students revise writing assignments after these different modes of peer assessment and feedback.
There were several research questions for the study, but here I’ll focus just on this one:
How is peer talk that occurs in the traditional oral and in the CMC classroom alike and different? Where differences exist, are they revealed in the writing that is developed subsequent to the peer-response group sessions? If so, how? (267)
Participants and data
Students in two sections of an upper-level course (Argumentative writing) at a four-year university participated; one section engaged in face to face peer assessment, and the other used computer-mediated peer assessment, but otherwise the two course were the same, taught by the same instructor. The CMC course used a discussion board system with comments organized chronologically (and separated according to the peer groups), and it was used both during class, synchronously (so students were contributing to it while they were sitting in a class with computers) and outside of class, asynchronously.
Peer group conversations were recorded in the face to face class, and the record of conversations from the CMC class could just be downloaded. The author also collected drafts of essays on which the peer discussion took place. Data was collected from all students, but only the recordings of conversations in one peer group in each class (oral and CMC) were used for the study. I’m not sure how many students this ended up being–perhaps 3-4 per peer group? [update (Feb. 28, 2013)] Looking at the article again, a footnote shows that there were four students in each group.
One of those groups, the CMC group, engaged in both computer-mediated peer discussion as well as oral discussion at a later point–so this group provides a nice set of data about the same people, discussing together, in two different environments. Below, when talking about the “oral” groups, the data include the group that was in the oral only class, plus the CMC group when they discussed orally.
Results
Nature of the talk in both environments
Not surprisingly, the student discussion in the face to face groups was highly interactive; the students’ statements often referred to what someone else had said, asked questions of others, clarified their own and others’ statements, and used words and phrases that cued to others that they were listening and following along, encouraging dialogue (e.g., saying “yes,” “right,” “okay,” “exactly”) (269-270).
In the CMC discussions, the talk was less interactive. Multiple threads of discussion occurred on the board, and each students’ comments could pick up on several at a time. This created a “multivocal tapestry of talk” that individuals would have to untangle in order to participate (270). At times, students in a peer group would respond to the paper being discussed, but not to each other (271), so that the comments were more like separate, standalone entities than part of an interactive conversation.
In addition, the possibility for asynchronous communication, though it could be convenient, also left some students’ comments and questions unanswered, since others may or may not return to the board after the synchronous group “chat” time had ended.
Subjects of the talk in each environment
Hewett found that face to face discussion had more talk about ideas, wider issues raised in the papers, and information about the contexts surrounding the claims and issues discussed in the papers, than in the CMC discussion (276). The CMC groups tended to focus more on the content of what was written, and showed less evidence of working together to develop new ideas about the topics in the essays. Hewett suggests: “Speculative thinking often involves spinning fluid and imperfectly formed ideas; it requires an atmosphere of give-and-take and circumlocution,” which is more characteristic of oral speech (276).
Effects of different kinds of peer assessment on revisions
For this part of the study, Hewett looked at changes in drafts of the same essay by each student and compared them to the peer evaluation discussion transcripts. She also looked at the peer texts each student read and commented on to see if there was trace evidence of influence by those other texts.
That is a particularly interesting part of the analysis–I hadn’t thought to do such a thing, but of course reading someone else’s work can have an influence on your own writing as well as the discussion that goes on in a peer group about your work. In discussing this process she cites Mortensen (1992), which might be a good place to look to think about how to do this; I don’t know for sure because I haven’t looked at it yet.
Through analyzing these documents, Hewett identified three types of idea exchange between students:
- Direct idea exchange: when one student directly suggests a revision to another student, who then incorporates that into a later draft
- Intertextual idea exchange: two types
- Imitation: a student imitates in her own work something from another students’ work that she read (such as an idea, an interpretation of a text, a way of writing an introduction or conclusion, or any number of things)
- Indirect sharing: a student uses in his own work something that was said in peer conversation about another students’ work
- Self-generated idea exchange: a student uses in her own work something that she said in the peer conversation about another students’ work. Alternatively, it could be something that that student says in the group discussion about her own work, that she then incorporates into her own work later.
In both discussion types, direct idea exchanges were far more numerous than the other two kinds. In the oral discussion, there were more (but not a great deal more) intertextual idea exchanges than in the CMC group, where there were more direct idea exchanges than in the oral group (281-282). This may be due, Hewett speculates, to the fact that in oral discussion the comments can appear to be more addressed to the group, even when one is directing one’s comments partly to a specific individual, than in the CMC discussion. In the latter, students used names to direct their comments to other students, and even though they were posted to a board open to the group, students could feel that the comments are more directed to the individual specifically and less to the group as well. Thus, students may not have paid much attention to comments that were addressed to other students–they could ignore them entirely, not reading them, whereas in the oral discussion students would (if they were listening!) hear most or all of the conversation (283).
In both environments, self-generated idea exchange was second out of the three types (after “direct”), though there were more of those in the oral discussion than in the CMC discussion. Hewett doesn’t comment much on why this might be the case, saying only that “If students are not interactively generating ideas with and for each other, they may be less likely to find ideas for themselves in the direct suggestions they give to their peers” (284). As noted above, collaborative generation of ideas happened more frequently in the oral discussion than in the CMC discussion.
Intertextual types were the least common (though there were more in the oral discussion than the CMC), and of those, “imitation” was very rare in both groups. Hewett suggests that concerns about copying each others’ ideas, or plagiarism, might contribute to there being very few imitation types of idea exchange (282).
Conclusions and Implications
Given the different nature of the peer conversations in each environment, Hewett concludes that
speculating and conversing hypothetically and abstractly about writing-in-progress may be more challenging in an online than an oral environment. (284)
Still, using computer-mediated writing can be useful in that comments are saved for later review, whereas in oral speech what is said may be easily forgotten unless written down during or shortly after the conversation.
Thus, Hewett suggests:
A combination of the two media might work even better by offering the ease of talking abstractly in the earlier stages of thinking and writing and the benefit of CMC-generated notes for revising at home. (284)
My thoughts
I am glad Hewett included one peer group that did both oral and CMC discussion, because otherwise the differences could be partly explained by the choices students made between the two types of classes. Those who were not comfortable with an oral environment might have chosen the CMC one, or vice versa. That may still be playing in here. I think a study like this would be best if students in the same class did both types of peer discussion, to help mitigate this factor. Of course, then, you’re still going to run into the fact that doing the different formats at different times will affect the study (e.g., for the second and subsequent peer discussions, the peers know each other better than the first). Perhaps one could have some groups do oral first and then CMC, and other groups could do CMC first and then oral.
I really like that Hewett emphasized the various ways students can exchange ideas in peer evaluation environments. We can’t just look at whether students incorporate direct suggestions from their peers in their essays, but should also look at what else was said in the discussion to see if they are incorporating things that were said about someone else’s paper, or things that those students themselves said–the discussion may have sparked them to come to realize things they would like to change about their writing that they wouldn’t have realized without the discussion.
The conclusion I’m reaching from the studies I’ve read so far is one I would have guessed without them, but it’s good to see some evidence: oral discussion and written comments are both useful, for different purposes, in peer assessment. In Arts One tutorials, in which students meet in groups of four once a week to do peer review of each others’ papers (with the professor present), I have in the past only focused on the oral discussion. I haven’t required students to also give each other written comments (or, when I did, I was pretty lax about it). That will now change!
Next I will consider studies that look at oral peer evaluation vs. written, online, synchronous (text-based chats).
Works cited
Gere, Anne R., & Abbott, Robert D. (1985). Talking about writing: The language of writing groups. Research in the Teaching of Writing, 19(4), 362–381.
Mortensen, Peter L. (1992). Analyzing talk about writing. In G. Kirsch, P.A. Sullivan (Eds.), Methods and methodology in composition research (pp. 105–129). Urbana, IL: Southern Illinois University Press.
