Tag Archives: peer review

Conference Follow-Up Questions on Peer Assessment

Someone who attended my conference presentation last month about Peer Assessment (see the blog entry here for slides), sent me some follow-up questions that I thought might be useful to capture here. Thanks for your questions! My responses are signaled by >>.

* How do you train students in the use of a rubric and in effective peer review, particularly in such a large class?

>> we developed the Peer Assessment Training workshop, which can be adapted for anyone to use https://peerassessment.arts.ubc.ca/ . If you have access to Canvas there’s a template and a demo in the Commons. If you don’t, stay tuned to our peerassessment website… we’re working on a fully open WordPress version for launch soon.

* How do your students respond to being graded by novice peers like themselves, rather than a more expert instructor or TA?  Does it take some convincing, or do you just present the evidence of the effectiveness of peer assessment and move on?

>> there is a range of opinions… but there’s a range of opinions about every pedagogical decision! I show them the evidence, make sure the assessment isn’t valued too highly, and give people a form to submit to have their grade re-evaluated by me if they want. That takes care of most concerns.

* You mentioned Peter Graf also assesses the quality of the peer assessments as well; do you know how he handles this?  (For instance, does the student being evaluated also reciprocally evaluate their peer reviewer?  Or is it something the TA does?)

>> He and his TAs grade the comments. It moves pretty quickly when they’re exported in a spreadsheet.

* In your slides, you mention two additional challenges: Students don’t trust each other, and comments were poor quality.  How did you address those challenges?  Any recommendations/ideas for how you would do it in the future?

>> The strategies above generally address these concerns.

 

>> If you’d like to try it out but are nervous about scale, you could always treat it as an opt-in pilot, so a sub-group of students try it out and give feedback. In a class of 440 I’m sure you can find at least a dozen students willing to participate… that’s one nice thing about very large classes!

peerScholar V Canvas Peer Review

If these terms mean anything to you, you might be interested in checking out my short report, where I pit peerScholar and Canvas Peer Review tool head-to-head. Yes, it’s absolutely like Batman V Superman: really we’re all actually friends just trying to  reach a common goal (i.e., facilitate student peer review).* Extra extra thanks to the students who participated and gave feedback, making this report possible.

Finding a Tool to Facilitate Peer Review in Large Classes

*Obviously I’m Wonder Woman in this analogy.

2013/2014 Student Evaluations Response Part 1/4: Intro Psych

Thank you to each of my students who took the time to complete a student evaluation of teaching this year. I value hearing from each of you, and every year your feedback helps me to become a better teacher. As I explained here, I’m writing reflections on the qualitative and quantitative feedback I received from each of my courses.

 

After teaching students intro psych as a 6-credit full-year course for the past three years, in 2013/2014 I was required to transform it into 101 and 102. Broadly speaking, the Term1/Term2 division from the 6-credit course stays the same, but turning Term 2 of Psyc 100 into a semi-standalone Psyc 102 proved more challenging than converting Term 1 into 101. Because these two courses really still form one unit in my mind, and I structure the courses extremely similarly, I will discuss them in tandem.

Across both courses, quantitative feedback was similar (albeit a bit higher in 101 than 102). Students rated the textbook equally high (4.2 & 4.3/5), which makes sense because I use the same text for both, and many students have told me informally that they enjoy reading the book (some qualify this endorsement with “for a textbook”). Check out my overall UMI scores from this year and all previous years here (click to enlarge, and click here for further discussion of UMIs):

IntroPsycHistoricUMIs.2009.2014

 

In the qualitative feedback, many of the same positive and helpful features were highlighted by students in both courses. Overall, students report enjoying and finding valuable the clickers, classroom discussions (often tied to clicker questions), films, opportunities to apply what is being learned, the 3-midterm format that helps stay on top of things even if it’s slightly annoying to be so frequently tested, music before class, the organization of class periods, my enthusiasm and energy, my effort to learn many students’ names, and the Invitational Office Hours. Capturing many of these commonly-mentioned features, one student from Psyc 102 wrote,

It was incredibly impressive how she tried to learn the name of every single student that she interacted with, despite the size of the class. The IOH were also a surprisingly fun experience. The class was very interactive, which definitely helped me learn, and even though I was unhappy about having three midterms at first I think I have to conclude it made studying for each one much easier and less stressful.”

One new element I added to both courses this year were five mini-papers which I called “Writing to Learn” (W2L) assignments (replacing a single 500 word paper I used to assign to be marked by TAs; see last year’s reflection for rationale). Students picked a topic from each of the two chapters about to be tested, wrote a paragraph explaining and applying the concept to their lives, then read 5 peers’ papers and gave feedback to them using peerScholar software. Students received feedback from their peers, and were able to choose any two topics to improve and reproduce on the final exam (no notes!). Overall, students reported finding the Writing to Learn assignments helpful for learning, and some mentioned that reading others’ work was helpful as well (both of these results are consistent with past research on similar writing assignments and peer review). My TAs have reported being able to grasp whether students knew what they were talking about from the writing section on the exams – and my test scores were higher than in previous years, so the goal of increasing learning seems to have been met! However, of the students who mentioned the W2L assignments, many noted that quality of peer feedback received was low. Dr. Peter Graf and I are just starting a project to deal with this very issue of enhancing peer feedback. It may take a couple of years to figure it out in a way that’s scalable to 300-400 students at a time, but we’re working on it.

Interestingly, a couple of students in each course noted my responses to student incivility. In one case it was failing to follow instructions to complete the bubbles during the exam time given, and another case (mentioned a few times in 102, actually), was my response to students talking in class. Side chatter is really only a problem in my first year courses – and it’s a consistent one that varies in severity year-to-year with different cohorts. Interestingly, Gillian Sandstrom and I have a paper about to come out in Teaching of Psychology showing that some chatter is a good thing: it can build a sense of community in a large class. But it can feel disrespectful and distracting to me. Perhaps I should consider building in even more opportunities for structured conversation, because clearly it’s going to happen anyway.

In 102 this year, rather than devoting a whole week to Chapter 2 Research Methods – which I do in 101 in the same place I did back when I taught the 6-credit version – I decided to split it up and cover topics as they came up throughout the term. For example, I used intelligence testing as a chance to discuss measurement and survey designs, and social psychology (specifically Milgram’s studies) as a chance to address the ethics of deception in research designs. Although I think this was a solid solution in theory, in practice there were definitely times when I felt like I was awkwardly wedging topics in to 102. Indeed, a few students mentioned this flow problem too – and it seems to be students who took both 101 and 102 with me who noticed the difference. Hopefully next year I’ll be able to smooth topics out a bit more effectively, perhaps cutting even more material to make more room for these new topics and ensuing discussions.

Although I still would prefer to teach intro psych as a unified whole with the same students over the whole year, apparently that’s not an option any longer. I have begun the process of converting this course into two halves effectively, and given the feedback above, I think I’m heading in that direction.

“Writing-to-Learn” in Intro Psych

While flying to the APS convention in Washington DC today, I was perusing a back-issue of the journal Teaching of Psychology. I came across an interesting article that made me consider a new paper-writing option for my intro psych class. In two studies, the researchers asked intro psych students to upload 16 brief (1-2 paragraph) statements that expanded on one concept from a 10-concept shortlist from each chapter. The paragraphs were graded for completion only (just a check to make sure they were topical), and were then coded for topic choice. Students responded better to midterm exam questions on topics they had previously written about than to questions on topics they had not previously written about–even when students had been told specifically which concepts to write about (Study 2).

What I’m finding particularly intriguing about this study is that (a) students seemed to learn the concepts better after writing about them, (b) the task seems pretty simple and straightforward for students (“What I learned about the concept is…” or “An example of the concept in my life is…”), (c) it may promote strong study habits that can be transferred to other topics and courses, and (d) because it’s ungraded and tracked online, it seems logistically manageable in a very large class.

Indeed, my classes are pretty large (depending on the year/term, they range from 250 to 400 students). Couple that with very limited TA resources, and fitting in writing is always a challenge. Currently I assign one 600-word application-style paper in each term. Recently I added a peer feedback step using peerScholar software, and that worked out pretty well. Informally, students reported learning from reading others’ papers, yet few seemed to revise their work at all, let alone implementing changes based on others’ feedback.

Maybe the assignment in this study could be a worthwhile pursuit. In this research paper, the authors noted that completion of all 16 was worth 10% of the students’ grade. I’m not sure I could offer that just for completion… we have pretty strict grading standards here in UBC Psych. But once I cut it in half (separating term 1 101 from  term 2 102) that’s only 5%… that seems more manageable.

I continue to ponder… as I lie awake… on Pacific time… in Eastern timezone.