Tag Archives: learning assessments

2013/2014 Student Evaluations Response Part 4: Psyc 218

Thank you to each of my students who took the time to complete a student evaluation of teaching this year. I value hearing from each of you, and every year your feedback helps me to become a better teacher. As I explained here, I’m writing reflections on the qualitative and quantitative feedback I received from each of my courses.

This is my second time teaching Psyc 218: Analysis of Behavioural Data, and I must say, I truly love teaching students this course. I didn’t make many changes from the last time, with two exceptions: (1) I attempted to even out the midterm difficulty in response to feedback and self-analysis from last time, (2) I integrated a new reading and treatment of the “New Statistics” debate/movement going on in our discipline right now. I didn’t change the textbook, overall teaching methods (largely lecture punctuated frequently by clicker questions where students practice calculations and interpretation), or the assignments (which we all have to do across sections).

Psyc218historicUMIs.2012.2014

Like last time, quantitative feedback was quite positive, and even a bit higher than last time (e.g., 4.7/5 on average for the “overall efficacy” item, up from 4.4; see the graph above). Qualitative feedback helps me to figure out specifics of what’s going well and what isn’t. Overwhelmingly, the most common comments noted the energy and enthusiasm I bring to class – which was awesome. I really had fun every day, so I’m glad I can help students build positive vibes toward statistics. Quite a few students commented on the high expectations I have for them: indeed, this course is challenging. Some students felt they were appropriately supported to rise to the challenge, whereas others felt pushed a bit too far. Given the differences there, I think the course is probably pitched at an appropriate level, at least when coupled with the way I handled it. Quite a few students asked for more problems to practice with. There are about 30 in the book at the end of each chapter, and every day in class there are clicker questions. However, I know that very few of these problems are at the high level of difficulty I ask for on exams. I wonder if I could offer students a model of how to take a research paper (of which there are a gazillion they could find on their own) and turn it into a problem set. Is there a common set of questions students could ask to help them link the course material to a research article? Hmm. Think more about how to help students learn how to make their own problems/examples. That will be more useful in the long run for the students than me trying to come up with a whole pile of artificial ones.

I was delighted with the number of students who mentioned that incorporating the New Statistics (vs. NHST) framework was motivating and engaging. In the context of discussing these current issues, one student wrote,

“I think that’s really important in terms of training us to be strong, statistically sound researchers. As well, it made me feel like I’m truly part of this field, instead of just being a student taking a course. Overall I would give Dr. Rawn an A+ because she has truly changed the way I think about statistics and about the field in general.”

The identity shift that this student in particular expresses is profoundly important to me. This and similar comments like it bolster my belief that incorporating current issues in statistics and research methods is crucial when training the next generation of psychological scientists. Overall, I’m quite pleased with how this course went, and overall the students seem to be as well.

2013/2014 Student Evaluations Response Part 1/4: Intro Psych

Thank you to each of my students who took the time to complete a student evaluation of teaching this year. I value hearing from each of you, and every year your feedback helps me to become a better teacher. As I explained here, I’m writing reflections on the qualitative and quantitative feedback I received from each of my courses.

 

After teaching students intro psych as a 6-credit full-year course for the past three years, in 2013/2014 I was required to transform it into 101 and 102. Broadly speaking, the Term1/Term2 division from the 6-credit course stays the same, but turning Term 2 of Psyc 100 into a semi-standalone Psyc 102 proved more challenging than converting Term 1 into 101. Because these two courses really still form one unit in my mind, and I structure the courses extremely similarly, I will discuss them in tandem.

Across both courses, quantitative feedback was similar (albeit a bit higher in 101 than 102). Students rated the textbook equally high (4.2 & 4.3/5), which makes sense because I use the same text for both, and many students have told me informally that they enjoy reading the book (some qualify this endorsement with “for a textbook”). Check out my overall UMI scores from this year and all previous years here (click to enlarge, and click here for further discussion of UMIs):

IntroPsycHistoricUMIs.2009.2014

 

In the qualitative feedback, many of the same positive and helpful features were highlighted by students in both courses. Overall, students report enjoying and finding valuable the clickers, classroom discussions (often tied to clicker questions), films, opportunities to apply what is being learned, the 3-midterm format that helps stay on top of things even if it’s slightly annoying to be so frequently tested, music before class, the organization of class periods, my enthusiasm and energy, my effort to learn many students’ names, and the Invitational Office Hours. Capturing many of these commonly-mentioned features, one student from Psyc 102 wrote,

It was incredibly impressive how she tried to learn the name of every single student that she interacted with, despite the size of the class. The IOH were also a surprisingly fun experience. The class was very interactive, which definitely helped me learn, and even though I was unhappy about having three midterms at first I think I have to conclude it made studying for each one much easier and less stressful.”

One new element I added to both courses this year were five mini-papers which I called “Writing to Learn” (W2L) assignments (replacing a single 500 word paper I used to assign to be marked by TAs; see last year’s reflection for rationale). Students picked a topic from each of the two chapters about to be tested, wrote a paragraph explaining and applying the concept to their lives, then read 5 peers’ papers and gave feedback to them using peerScholar software. Students received feedback from their peers, and were able to choose any two topics to improve and reproduce on the final exam (no notes!). Overall, students reported finding the Writing to Learn assignments helpful for learning, and some mentioned that reading others’ work was helpful as well (both of these results are consistent with past research on similar writing assignments and peer review). My TAs have reported being able to grasp whether students knew what they were talking about from the writing section on the exams – and my test scores were higher than in previous years, so the goal of increasing learning seems to have been met! However, of the students who mentioned the W2L assignments, many noted that quality of peer feedback received was low. Dr. Peter Graf and I are just starting a project to deal with this very issue of enhancing peer feedback. It may take a couple of years to figure it out in a way that’s scalable to 300-400 students at a time, but we’re working on it.

Interestingly, a couple of students in each course noted my responses to student incivility. In one case it was failing to follow instructions to complete the bubbles during the exam time given, and another case (mentioned a few times in 102, actually), was my response to students talking in class. Side chatter is really only a problem in my first year courses – and it’s a consistent one that varies in severity year-to-year with different cohorts. Interestingly, Gillian Sandstrom and I have a paper about to come out in Teaching of Psychology showing that some chatter is a good thing: it can build a sense of community in a large class. But it can feel disrespectful and distracting to me. Perhaps I should consider building in even more opportunities for structured conversation, because clearly it’s going to happen anyway.

In 102 this year, rather than devoting a whole week to Chapter 2 Research Methods – which I do in 101 in the same place I did back when I taught the 6-credit version – I decided to split it up and cover topics as they came up throughout the term. For example, I used intelligence testing as a chance to discuss measurement and survey designs, and social psychology (specifically Milgram’s studies) as a chance to address the ethics of deception in research designs. Although I think this was a solid solution in theory, in practice there were definitely times when I felt like I was awkwardly wedging topics in to 102. Indeed, a few students mentioned this flow problem too – and it seems to be students who took both 101 and 102 with me who noticed the difference. Hopefully next year I’ll be able to smooth topics out a bit more effectively, perhaps cutting even more material to make more room for these new topics and ensuing discussions.

Although I still would prefer to teach intro psych as a unified whole with the same students over the whole year, apparently that’s not an option any longer. I have begun the process of converting this course into two halves effectively, and given the feedback above, I think I’m heading in that direction.

Creative Advertisement Showcase 2014

A couple of weeks ago in my section of Psyc 208 we held the Creative Advertisement Showcase, which was a fantastic celebration of what my students had discovered throughout the term! Previously, each team of students identified a learning challenge they face, investigated primary sources for insight into understanding and addressing the challenge, and summarized those sources in annotated bibliographies and team abstracts. (See the Team Project Guide for a full description of this multi-part assignment.) The purpose of the Creative Advertisement was to get the word out to fellow students about research-based techniques and strategies for addressing the learning challenges they face. They truly were creative! Projects ranged from video and live games to posters to live skits and demonstrations to videos and pamphlets… an impressive variety! Check out some of their videos and websites (ordered by team #)…

Team 4’s Sleep Fairy:  https://www.youtube.com/watch?v=PGOWxnPizOk 

Team 5’s Culture Shock: https://www.youtube.com/watch?v=GyZcynf2jk4&feature=youtu.be

Team 22’s Motivation Makeover: https://www.youtube.com/watch?v=l6syynl18XA

Team 11’s Loneliness: https://drive.google.com/folderview?id=0B_qDfS_x4VtvLWhIZXRNS2pxVHc&usp=sharing

Team 18’s Sleep: https://docs.google.com/file/d/0B0xsEQGqsy3IVGczVVpZTGZkcGM/edit

Here’s a shot of Team 19’s Effects of Internet Use on Learning station…

… and one of Team 3’s Social Loafing game based on Apples to Apples

Thanks to everyone for a solid effort on these projects and throughout the term! Study smartly for your finals… and remember that no matter how you do on them, that’s not a reflection of your worth as a person.

Student Evaluations of Teaching 2012/2013: Part 3 Special Topics Psyc 208

Thank you to each of my students who took the time to complete a student evaluation of teaching this year. I value hearing from each of you, and every year your feedback helps me to become a better teacher. As I explained here, I’m writing reflections on the qualitative and quantitative feedback I received from each of my courses.

Psych 208: Special Topics

Overall, I thought this course was pretty smooth. Over the past four years I’ve developed a clearer vision for what this course is (see the syllabus), and I think that’s reflected in greater coherence, greater integration across in-class teaching methods, content, and learning objectives. Learning appraisals are in decent shape (although could probably use some refreshment in the coming year). As I have matured, I have matured this course.

After such self-assessment, I was pleased to note that students rated this course more positively than any previous iteration, right on par with my other more traditional courses (check out the quantitative data here). Reading the qualitative evaluations was almost overwhelming. Student after student noted how useful this course was, how much they applied these concepts to their everyday life, how they built skills they’d take with them into future courses and their careers. I am absolutely blown away by what people said about this course. When I first envisioned this course, I wanted it to be useful. So much of psychology (particularly social psychology) can improve our lives, and that’s exactly what I dreamed this course would help people do: apply our amazing research to improve their lives. To that end, numerous students wrote things like,

I found myself always referencing the course subject matter to my friends and applying it within my own life”, “this has been one of the most beneficial classes of my university career,” and “this will be one of those courses that I look back on knowing that it was a good use of my time” – even if they noted they were about to graduate.

All of this positivity was despite (or because of?) the extent to which this course challenged students in various ways (e.g., “not an easy A”) yet was perceived as valuable (e.g., “the course project on group work is such a valuable skill that students need”).  The usefulness of our course material got through to these students, and I’m absolutely thrilled!

Given these positive comments, I think it’s worthwhile to note that as I’ve matured (with) this course, there’s one key tweak I made in 2012 that I emphasized even more in 2013. During the first week, I am very explicit about the collaborative, applied, interactive nature of this course. I invite people to explore with me some ways of learning that are, for some people, uncomfortable and new. I also invite people who aren’t up for such exploration at this time to choose a different section or course, with no hard feelings. Along with better development of the curriculum and assessments, I think this tweak goes a long way toward student-course-instructor fit.

One area for growth that was noted a few times in the qualitative data was lack of clarity about learning assessment requirements. In my view, the handouts and LOs I give to students pretty clearly map on to my rubrics and exams. But there’s obviously a disconnect: a small yet larger proportion of students than in any of my other courses report lack of clarity for what to expect from grading. Moreover, my “fairness of evaluations” rating was the only UMI across all my 2012/2013 courses to fall below 4 out of 5. To remedy this disconnect, I have a couple of ideas: First, I will consider giving—up front—my rubrics for all components to the assignment. A couple of students suggested this, and I think it will help. Second, I will consider ways to give more advice for the exams, especially the midterm. One option would be to give a list of the short answer questions, a subset of which will appear on the exam (will this increase learning? If so, great! Will this increase the mean and/or reduce variance? If so, stress!).

[One of the challenges that I face (that’s not exactly popular with students, in my experience) is that attempts to change anything to do with grading run this risk of inflating the mean and/or changing the distribution of grades. Like all faculty in my department, I am bound by departmental requirements to have a mean around 65% and a standard deviation around 14% in all 100 and 200 level courses. Therefore, interventions that increase the mean or shrink the SD present real concerns that force me to confront this reality: any improvement in clarity might simultaneously require an increase in difficulty.]

Quite a few students mentioned enjoying the readings from one book, but finding the book that drew from a sport psychology perspective a bit less helpful and/or easily applied to academic life. Many of the readings will change in 2013/2014, largely because many of the chapters I currently rely on are out-of-date and must be changed anyway. Therefore, it’s a great opportunity to re-think the whole set of readings. I will be making every effort to get the custom course-pack down to one publisher, ideally with less of an emphasis on sport. Given the current offerings I’ve seen, I think that’s possible. I’ll also be thoughtfully considering the length of readings, as a (small) group of students mentioned feeling like there was too much to read.

Overall, in my view and the students’, this offering was the most successful iteration of Psyc 208 to date!

“Writing-to-Learn” in Intro Psych

While flying to the APS convention in Washington DC today, I was perusing a back-issue of the journal Teaching of Psychology. I came across an interesting article that made me consider a new paper-writing option for my intro psych class. In two studies, the researchers asked intro psych students to upload 16 brief (1-2 paragraph) statements that expanded on one concept from a 10-concept shortlist from each chapter. The paragraphs were graded for completion only (just a check to make sure they were topical), and were then coded for topic choice. Students responded better to midterm exam questions on topics they had previously written about than to questions on topics they had not previously written about–even when students had been told specifically which concepts to write about (Study 2).

What I’m finding particularly intriguing about this study is that (a) students seemed to learn the concepts better after writing about them, (b) the task seems pretty simple and straightforward for students (“What I learned about the concept is…” or “An example of the concept in my life is…”), (c) it may promote strong study habits that can be transferred to other topics and courses, and (d) because it’s ungraded and tracked online, it seems logistically manageable in a very large class.

Indeed, my classes are pretty large (depending on the year/term, they range from 250 to 400 students). Couple that with very limited TA resources, and fitting in writing is always a challenge. Currently I assign one 600-word application-style paper in each term. Recently I added a peer feedback step using peerScholar software, and that worked out pretty well. Informally, students reported learning from reading others’ papers, yet few seemed to revise their work at all, let alone implementing changes based on others’ feedback.

Maybe the assignment in this study could be a worthwhile pursuit. In this research paper, the authors noted that completion of all 16 was worth 10% of the students’ grade. I’m not sure I could offer that just for completion… we have pretty strict grading standards here in UBC Psych. But once I cut it in half (separating term 1 101 from  term 2 102) that’s only 5%… that seems more manageable.

I continue to ponder… as I lie awake… on Pacific time… in Eastern timezone.