Tag Archives: learning assessments

2014/2015 Student Evaluations Response Part 3: Psyc 102

Thank you to each of my students who took the time to complete a student evaluation of teaching this year. I value hearing from each of you, and every year your feedback helps me to become a better teacher. Each year, I write reflections on the qualitative and quantitative feedback I received from each of my courses, and post them here.

After teaching students intro psych as a 6-credit full-year course for three years, in 2013/2014 I was required to transform it into 101 and 102. Broadly speaking, the Term1/Term2 division from the 6-credit course stays the same, but there are some key changes because students can take these courses in either order from any professor (who uses any textbook). These two courses really still form one unit in my mind so I structure the courses extremely similarly. I have summarized the quantitative student evaluations in tandem. As can be seen in the graph, quantitative ratings of this course haven’t changed too much over the past few years, and students rate my teaching in these courses very similarly. However, I will discuss them separately this year because of some process-based changes I made in 102 relative to 101.

IntroPsycHistoricUMIs.LastFiveYears

My response to Psyc 101 included a formal coding of comments into various categories. Oh to have the open time of summer! I’m a bit more pressed for time now as I work on my Psyc 102 preparations, so as I was reading the comments I picked out themes a bit less formally. Two major themes emerged (which map on roughly to those identified using more a formal strategy for Psyc 101): class management, and tests. Interestingly, I changed the weighting of the Writing to Learn assignments from Psyc 101 (Term 1 in 2014/15) to Psyc 102 (Term 2 in 2014/15), to avoid relying on peer reviews and just do them for completion points only. The number of comments about that aspect of the course dropped close to zero, despite the actual tasks of the assignment staying the same (see my response to Psyc 101, linked above, for discussion of why I was compelled to make changes in 102 last year).

Again, a major theme in the comments was that tests are challenging. I don’t think they’re any more challenging than in my 101 course, but maybe there’s a perception that they will be easier because the content seems more relatable, and so people are more surprised by the difficulty in 102. Not sure. Just like in my 101, they draw from class content, overlapping content, and some textbook-only content, and they prioritize material that follows from the learning objectives (which I post in advance to help you know what material will be explored in class the next day). MyPsychLab is a source of practice questions, as are your peers and the learning objectives.

 
In addition to content, time is tight on the tests. Before implementing the Stage 2 group part, my students didn’t have 25 questions in 50 minutes… they had 50 questions in 50 minutes. Now, we have 25 questions in about 28-30 minutes, which is actually more relaxed than before. Although many people report finding value in the group part of the test, it’s not universally loved. A few people mentioned that it’s not worth it because it doesn’t improve grades by very much. My goal here is to promote learning. I’m stuck with the grading requirements: we have to have a class average between 63 and 67%. That’s out of my control. The group tests add an average of about 2% to your test grade, which you may or may not value. But importantly for me, they improve learning (Gilley & Clarkston).

The second most frequent comment topic related to various aspects of classroom dynamics. I thought I’d take this opportunity to elaborate on some choices I make in class.

I do my best to bring high energy to every class. Many people report being fueled by that enthusiasm—that’s been my most frequent comment for many years across many courses. However, a few people don’t love it and feel it’s a bit juvenile or just too much. I bring this up here as a heads up: Although I’d love to have you join us, if you’re not keen on the way I use my voice to help engage people, you might enjoy a different section of 102 better.

In class, occasionally I comment when a student is doing non-course related things on a device, and invite them to join us. A couple of people mentioned this in evaluations from last year. My intention here is to promote learning (i.e., to do my job). Research shows that when people switch among screens on their laptops, they’re not just decreasing their own comprehension, but the comprehension of all the people within view of the screen (Sana, Weston, & Cepeda). I occasionally monitor and comment on this activity (e.g., during films) so that I can create a class climate where anyone who wants to succeed can do so.

 
Sometimes I wait for the class to settle, and sometimes I start talking to the people in the front (which might be perceived as incomprehensible to the people at the back of the room). I get impatient sometimes too, particularly toward the end of the year (I’m only human after all!). I don’t like to start class until the noise level is settled, out of fairness of people who are sitting at the back but still want to be involved, and, to be honest, talking while others are talking and not listening makes me feel disrespected. One change I made in my 101 class this past term might help us with starting class in 102. I decided to move the announcements from verbal ones at the start of class to a weekly email I send out on Friday afternoons. This change seems to have improved people’s recognition that when I’m ready to start class, we’re starting with content right away, so settling happens more quickly. Hopefully this will help us out in 102.

 
As always, many thanks for your feedback. It challenges me to think about ways I can improve in my teaching, and to reconsider decisions I have made in the past. Sometimes I make changes, and sometimes I reaffirm the decisions I made before. This space gives me a chance to explain changes or re-articulate why I continue to endorse my past decisions. Student feedback is an essential ingredient to my decision-making process. Thank you!

2014/2015 Student Evaluations Response Part 2: Psyc 101

Thank you to each of my students who took the time to complete a student evaluation of teaching this year. I value hearing from each of you, and every year your feedback helps me to become a better teacher. As I explained here, I’m writing reflections on the qualitative and quantitative feedback I received from each of my courses.

 

After teaching students intro psych as a 6-credit full-year course for three years, in 2013/2014 I was required to transform it into 101 and 102. Broadly speaking, the Term1/Term2 division from the 6-credit course stays the same, but there are some key changes because students can take these courses in either order from any professor (who uses any textbook). These two courses really still form one unit in my mind so I structure the courses extremely similarly. I have summarized the quantitative student evaluations in tandem. Students rate my teaching in these courses very similarly. However, I will discuss them separately this year because of some process-based changes I made in 102 relative to 101.

IntroPsycHistoricUMIs.LastFiveYears

To understand the qualitative comments in Psyc 101, I sorted all 123 of them into broad categories of positive, negative, and suggestions, and three major themes emerged: enthusiasm/energy/engagement, tests/exams, and writing assignments. Many comments covered more than one theme, but I tried to pick out the major topic of the comment while sorting. Twenty-one comments (18%) focused on the positive energy and enthusiasm I brought to the classroom – enthusiasm for students, teaching, and the discipline (one other comment didn’t find this aspect of my teaching helpful). All nine comments about the examples I gave indicated they were helpful for learning.

Thirty-four comments focused largely on tests (28%). Last year I implemented the two-stage tests in most of my classes, and I expected to see this format discussed in these evaluations. All but one of the five comments about the two-stage tests were positive. Other positive comments mentioned that tests were fair and that having three midterms kept people on track. Yet, the major theme was difficulty. Of the 17 negative comments that focused largely on tests (representing 14% of total comments), the two biggest issues were difficultly and short length (which meant missing a few questions had a larger impact). The biggest theme of the suggestions (5 comments) requested practice questions for the exams. There is a study guide that students can use that accompanies the text. I wonder how many students make use of that resource? Do they know? Would that help students get a better sense of the exam difficulty? This is a major question I want to ask my upcoming Student Management Team (SMT) this year. Still, it’s important to keep in mind that these comments represented a minority in the context of all 123 that were given.

Twenty-five comments (20%) focused largely on the low-stakes writing-to-learn assessments. Five times throughout the term students write a couple of paragraphs explaining a concept and then applying it to understand something in their lives. They are each worth 2%. When I implemented this method two years ago I also added a peer assessment component, such that 1% has been completion, and 1% comes from the average of their peers’ ratings of their work. In year one I used Pearson’s peerScholar application, and in year two (2014/2015) I switched to the “Self and Peer Evaluation” feature in Connect (UBC’s name for Blackboard LMS)… which was disastrous from the data side of things (e.g., giving a zero instead of missing data… and then counting the zero when auto-calculating the student’s average peer rating!). As I expected, most of the comments about the assignments were about peer review failures: missing comments, missing data, distrust of peers as being able to rate each other, distrust at only receiving 3 ratings that can be wildly different sometimes, difficult to get a good mark, suspicion that peers weren’t taking it seriously. For 2015/2016, I have made three major changes to help improve this aspect of the course:

  1. Return to the peerScholar platform instead of Connect, which should fix part of the missing data problem (and it can now integrate with my class list better than it did before),
  2. Review 4 or maybe 5 peers rather than 3,
  3. Implement a training workshop! With the help of a TLEF, Peter Graf and I have been developing an online training module for our courses to help students learn to use our respective peer assessment rubrics and test out the rubric on some sample essays. Our main hypothesis is that students will be more effective peer reviewers—and will feel like peers are more effective reviewers—as a result of going through this process. More to come!

As can be seen in the graph above, quantitative ratings of this course haven’t changed too much over the past few years. The written comments are useful for highlighting some areas for further development. I look forward to recruiting a Student Management Team this year to make this course an even more effective learning experience for students!

2014/2015 Student Evaluations Response Part 1: Psyc 217

Thank you to each of my students who took the time to complete a student evaluation of teaching this year. I value hearing from each of you, and every year your feedback helps me to become a better teacher. As I explained here, I’m writing reflections on the qualitative and quantitative feedback I received from each of my courses.

 

Research Methods is the course I have taught more than any other: since 2008 I have taught 1022 students across 14 sections. I am also the course coordinator, which means I help new research methods instructors prepare for their sections including ordering textbooks, I organize the labs including TFs and room bookings, and I facilitate the poster session. And I write the Canadian edition of the text we use (2nd edition forthcoming February 2016). I spend a lot of time thinking about this course!

Over the years I have incorporated many suggestions initiated by students in student evaluations, and have made changes to the course design based on student evaluation feedback. In 2014/2015, I implemented one major change to the course: I converted my three tests and the final exam to a two-stage format. Students write the test first on their own, then break into groups to write it again (see my blog post about this method, with resources). The largest challenge I knew I faced going in was timing, particularly for the tests: Our class periods are just 50 minutes long, and my tests were already notoriously lengthy. It was with this lens that I approached reading my student evaluations for this past year. Do I keep the two-stage tests?

I examined the quantitative data first. As is clear from the graph, there were no major differences relative to previous years. Notably, The fairness of the instructors’ assessments of learning item was rated higher than usual, though that difference was small. No indication of a disaster. Yay!

Psyc217historicUMIs.LastFiveYears

Next, I evaluated the qualitative data. As I sorted into positive and negative columns, two topic themes seemed to be emerging: tests and enthusiasm. As in past years, students appreciated the energy and enthusiasm I bring to the classroom (especially at 9am and especially with this topic). Out of 128 comments, 29 of them (23%) specifically mentioned energy or enthusiasm (with just a couple of those recommending I tone it down a bit).

Coincidentally, the same proportion of comments (29, 23%) mentioned the tests in some way. Six comments endorsed the three (rather than two) test format, indicating it helped them keep on top of studying, although three comments mentioned that tests 2 and 3 were too close together, and another three indicated they would have preferred two tests. Seven comments mentioned enjoying the two-stage format were positive, indicating that it provides opportunities to work together, make friends, and receive immediate feedback. The two negative comments that specifically mentioned the two-stage format did not disagree with it per se, but felt that this format exacerbated a different problem: feeling rushed. Seven comments specifically mentioned feeling rushed during exams. Two others indicated that the fix implemented for test #2 worked well to address the timing issue. Still, it seems that timing during tests was the clearest challenge in my course. Despite my best efforts to shrink the tests, there is a small group of students reporting they remain too long for the required tasks. I’ll consider strategies for preparing students for this pace.

Overall, the two-stage tests seemed to work well for most students, and grades were still within the acceptable range for our department. I enjoyed giving exams much more than I used to, and I was able to relax and hear the conversations students were having as they debated correct answers. Anecdotally, I was able to witness deeper learning and (mostly) positive group dynamics during the second stage (luckily I have other peoples’ data to offer as evidence that it works to promote learning!). Two-stage exams: you’re staying!

Two-Stage Exam: Introduction and Resources

Time to dive in! After thinking about them for a long time, this term I’m converting my exams into two-stage exams.

  • Step 1. I shorten the exam so it’s doable in about 2/3 of the testing time slot.
  • Step 2. Students write the exam individually.
  • Step 3. Students immediately — during the same class period — write the same exam again in groups of 4.
  • Step 4. Grade the exams as usual, but 90% of the score comes from individual, and 10% from team, with a guarantee that if you do better than the team score you get 100% weight for the individual (which very rarely happens, so I’m told).

Why am I making this change?

Four key reasons:

  1. Data. A growing pool of evidence is showing that team tests help students learn. See references below.
  2. Feedback. My classes are very large, so I struggle to give any personalized feedback at all, especially timely feedback. By re-doing the test immediately with peers, they get to immediately discuss the questions and come to the right answer (according to data).
  3. Exam improvement. Based on my evaluations, a small but consistent group of students find my exams very difficult and/or too long. Because I still only have 50 minute classes to work with, this change will force me to shorten my exams, culling and distilling to just the most effective questions that measure deep learning.
  4. Community. I value collaboration and building a supportive community. Research papers and instructors who have used this method report extra benefits beyond learning: students have more rapport with each other and are more willing to participate with their peers in class throughout the term. Also, Gillian Sandstrom and I have a research paper in press showing the more students talk in class, the more they feel like part of a community and interested in the class. So… back to data.

Interested? Here are some quick and effective resources for implementation:

  1. Videos by the CWSEI team depicting Two-Stage Exams in action.
  2. Jones, F., Gilley, B., Harris, S. (2013). Tips for successful two stage exams. The EOS-SEI Times, 6(9). Retrieved http://www.cwsei.ubc.ca/Files/EOS/EOS-SEITimes_4.1_GroupExams.pdf
  3. Jones, F., Gilley, B., Lane, E., Caulkins, J., & Harris, S. (2011). Using group exams in your classes. The EOS-SEI Times, 4(1). Retrieved http://www.cwsei.ubc.ca/Files/EOS/EOS-SEITimes_4.1_GroupExams.pdf
  4. PHAS-CWSEI Team. (2012). Two-stage (group) exams. CWSEI–PHYS & ASTRO Newsletter. Retrieved http://www.cwsei.ubc.ca/Files/PHAS/PHAS-CWSEI_Newsletter_Summer-2012.pdf
  5. Brett Gilley, aka @ModernHydra

Data

Dahlstrom, O. (2012). Learning during a collaborative final exam. Educational Research and Evaluation: An International Journal on Theory and Practice, 18, 321-332.

Eaton, T. T. (2009). Engaging students and evaluating learning progress using collaborative exams in introductory classes. Journal of Geoscience Education, 57, 113-120.

Gilley, B. H., & Clarkston, B. (2014). Collaborative testing: Evidence of learning in a controlled in-class study of undergraduate students. Journal of College Science Teaching, 43, 83-91.

  • A particularly well-designed example.

Leight, H., Saunders, C., Calkins, R., & Withers, M. (2012). Collaborative testing improves performance but not content retention in a large-enrollment introductory biology class. CBE—Life Sciences Education, 11, 392-401.

  • The title might be alarming here… they showed no effect of the 2-stage exam on final exam performance (compared with material that had been previously tested only with individual tests). I’m ok with this. Not every study is going to find the same effect (particularly ones with some execution oddities like this one), yet this is still a “no-change” effect with no evidence that student learning decreases. Moreover, students still enjoyed the process and found it less stressful than the individual-only tests. No harm done, potential benefits.

Rieger, G. W., & Heiner, C. E. (2014). Examinations that support collaborative learning: The students’ perspective. Journal of College Science Teaching, 43, 41-47.

Roediger, III, H. L., & Marsh, E. J. (2005). The positive and negative consequences of multiple-choice testing. Journal of Experimental Psychology: Learning, Memory, & Cognition, 31, 1155-1159.

  • Two-stage tests might help to fight the negative consequences of MC tests: you remember what you answered (and thought was right), not what actually was right.

Sandstrom, G. M., & Rawn, C. D. (in press/2014). Embrace chattering students: They may be building community and interest in your class. Teaching of Psychology.

Zipp, J. F. (2007). Learning by exams: The impact of two-stage cooperative tests. Teaching Sociology, 35, 62-76. doi: 10.1177/0092055X0703500105

 

 

 

2013/2014 Student Evaluations Response Part 4: Psyc 218

Thank you to each of my students who took the time to complete a student evaluation of teaching this year. I value hearing from each of you, and every year your feedback helps me to become a better teacher. As I explained here, I’m writing reflections on the qualitative and quantitative feedback I received from each of my courses.

This is my second time teaching Psyc 218: Analysis of Behavioural Data, and I must say, I truly love teaching students this course. I didn’t make many changes from the last time, with two exceptions: (1) I attempted to even out the midterm difficulty in response to feedback and self-analysis from last time, (2) I integrated a new reading and treatment of the “New Statistics” debate/movement going on in our discipline right now. I didn’t change the textbook, overall teaching methods (largely lecture punctuated frequently by clicker questions where students practice calculations and interpretation), or the assignments (which we all have to do across sections).

Psyc218historicUMIs.2012.2014

Like last time, quantitative feedback was quite positive, and even a bit higher than last time (e.g., 4.7/5 on average for the “overall efficacy” item, up from 4.4; see the graph above). Qualitative feedback helps me to figure out specifics of what’s going well and what isn’t. Overwhelmingly, the most common comments noted the energy and enthusiasm I bring to class – which was awesome. I really had fun every day, so I’m glad I can help students build positive vibes toward statistics. Quite a few students commented on the high expectations I have for them: indeed, this course is challenging. Some students felt they were appropriately supported to rise to the challenge, whereas others felt pushed a bit too far. Given the differences there, I think the course is probably pitched at an appropriate level, at least when coupled with the way I handled it. Quite a few students asked for more problems to practice with. There are about 30 in the book at the end of each chapter, and every day in class there are clicker questions. However, I know that very few of these problems are at the high level of difficulty I ask for on exams. I wonder if I could offer students a model of how to take a research paper (of which there are a gazillion they could find on their own) and turn it into a problem set. Is there a common set of questions students could ask to help them link the course material to a research article? Hmm. Think more about how to help students learn how to make their own problems/examples. That will be more useful in the long run for the students than me trying to come up with a whole pile of artificial ones.

I was delighted with the number of students who mentioned that incorporating the New Statistics (vs. NHST) framework was motivating and engaging. In the context of discussing these current issues, one student wrote,

“I think that’s really important in terms of training us to be strong, statistically sound researchers. As well, it made me feel like I’m truly part of this field, instead of just being a student taking a course. Overall I would give Dr. Rawn an A+ because she has truly changed the way I think about statistics and about the field in general.”

The identity shift that this student in particular expresses is profoundly important to me. This and similar comments like it bolster my belief that incorporating current issues in statistics and research methods is crucial when training the next generation of psychological scientists. Overall, I’m quite pleased with how this course went, and overall the students seem to be as well.