Tag Archives: student evaluations of teaching

Reflecting on a Seminar: My First Fully Online Course

As I mentioned in this previous post, I am working through feedback from my students.  All quantitative data, as well as links to all previous blog posts (since 2011), are available here.

Last round I focused on PSYC 218 Statistics. For this installment, I focus on PSYC 417 Advanced Seminar in Applied Psychology of Teaching and Learning, which will be transitioning to the new course code PSYC 427 for my W2021 Term 2. The first time I taught this course, 10 students journeyed with me in May-June 2019. That pilot syllabus, an overview of the course, and reflections from that year, and my August 2019 (!) draft of Summer 2020’s syllabus are available here. By the time Summer 2020 actually occurred, we were well into the heart of the pandemic. Sixteen brave students joined me from around the world in July-August 2020 as we figured out how to teach and learn online. Here’s how I adapted the syllabus. Notably, both these first two offerings were during the summer and therefore were condensed into 6 weeks. Next time I offer the course (likely my last offering for at least a while), it will be stretched to a regular term.

I had never taught students entirely online before, and don’t have a ton of experience with upper level seminars. I tried my best to apply the Guiding c Principles when redesigning the course, particularly when it came to compassion and options and flexible deadlines. If you compare the Policies sections from my three pandemic courses in three successive terms, you can see the progression I made in articulating and embracing options and flexibility: PSYC 417 (Summer 2020), PSYC 217 (W2020 T1), and PSYC 218 (W2020 T2).

To be honest I don’t remember too many details about how I went about adapting the course. It was such a frightening and overwhelming flurry. I do recall that it’s when I came up with a weekly organizer and intensive module structure in Canvas, which I went on to elaborate in subsequent courses because it kept my organized. I left some notes to myself that I want to capture here:

  • The students collaboratively made an excellent participation rubric. I still found it challenging to apply, but it was better.
  • Students scoffed at my 5 hour estimate for the final paper, indicating it took them much much longer.
  • The skill of “summarize an article” was coming up a lot and I should have students practice that specific skill deliberately. It’s a building block for both the reading reflections and the final paper (and any paper, really).
  • Lesson 6 (end of Week 3) plan was too much. We were all overwhelmed by then.
  • Cut reading reflections to best 4 of 5 rather than requiring all 5.
  • A second marker (TA) was essential for actually being able to give timely feedback throughout the course, and reliably grade papers on time by the deadline. (Special thanks to Carolyn Baer!)

Notably, none of these notes were on the tech. Maybe those notes are embedded in my weekly lesson plan notes. I recall getting the hang of annotations at some point during that term. But I can’t help but think about how tech ended up fading into the background in PSYC 217 and PSYC 218 evaluations too. I’m curious to know what students in PSYC 417 put in their comments…

Quantitative Data Summary. The aggregate quantitative data from both offerings appears below. Please click on the graph to enlarge it. Importantly, the response rate was low. Just 7 out of 16 students responded, which is below the threshold for a class this small. Nonetheless, based on the data I have, the second offering went quite well. The most room to grow is in perceived fairness of assessments. I’m not altogether surprised by this, and I’m glad to see a boost from the previous iteration given the changes to the project. The biggest jump was in Clear Expectations — again not surprising given I’d had a chance to actually clarify my expectations.

Qualitative Data Summary. Two key themes emerged from the comments. First, students felt supported in a well-structured course. The asynchronous/ synchronous balance worked well, as did the Canvas structure, and general openness to flexibility. Second, it was a lot of work. Expectations where high (a couple of folks perceived too high), and there were a lot of deadlines. I suspect that some of this intensity will be mitigated by stretching out over 13 weeks rather than condensing into 6, but I still hear the call to think carefully about whether each and every assignment is essential, and where I can build in further flexibility.

Prospective students might be interested in this one comment in particular: “This course is truly intensive. The estimated amount of hours it takes to do this course is way lower than reality. If you are doing anything other than this course in the summer, I wouldn’t recommend it. You will have to produce work above and beyond other high achievers. If you have other commitment in the summer, and produce an average work quality, expect to see lower grades than your average.” I would invite someone who is spending exorbitant hours on this course to meet with me to see if and where to incorporate efficiencies. Yet the point is a valuable one for some folks: this isn’t an easy A. Contrast with someone else’s perspective: “This course reminded me that grades are by far, not the most important part of learning. Nor are they always able to provide sufficient feedback for students. The comments I received on my work far outweighed the value of the grade.

Two of my favourite comments:

“Taking PSYC417A with Dr. Rawn has genuinely changed my life. Between the quality and frequency of the feedback I received and the fact that the course format was premised upon Self–Determination Theory, I feel I’ve improved more in this course than any other. I believe I’ve improved as a thinker, writer and learner. The information communicated by Dr. Rawn, together with the course activities, provided a scaffolding for students to gradually become more autonomous in their learning. Her compassion, flexibility, and acumen worked synergistically to construct an environment that felt safe enough for me to push my own limits, and improve in ways I wasn’t aware I could.” — WOW. I am honoured to have had such an impact on this student. Thank you so much, whoever you are, for sharing this feedback.

“Dr. Rawn manages to engage her students in the content, and helps give students the freedom to connect course content to passions outside of the course all while creating an environment that is both accepting, and demands the very best work product. I purposefully look to take courses with her because I am never disappointed. She works so incredibly hard, but makes it look easy.” — Whoever you are, thanks for your ongoing trust in me!

My sincere thanks to everyone who took the time to offer ratings and feedback, and to everyone who journeyed along with me while I learned to teach online from the dining table in my open-concept condo. Although some of the details are a bit blurry by now, I do have a deep sense of how much we all worked to motivate each other and learn together in those unprecedented times. You kept me going.

Reflecting on Statistics (Emergency Pivot and Mid-Pandemic)

As I mentioned in this previous post, I am working through feedback from my students.  All quantitative data, as well as links to all previous blog posts (since 2011), are available hereALERT! This post has become extra ridiculously long. Writing it is helping me think and process this overwhelming year, so I’m just going with it.

Last round I focused on PSYC 217 Research MethodsFor this installment, I focus on PSYC 218 Advanced Statistics for Behavioural Sciences, in which I have taught over 1000 students across 11 sections since 2012. Like 217, it’s a required course for BA Psychology Majors. I regularly teach this course in Term 2 (January start), two sections back-to-back of about 100 students each. This post will focus on my W2019 Term 2 results (which started in January 2020 business as usual, and pivoted in a state of emergency to remote learning). It will then compare with results from my mid-pandemic January 2021 fully online offering. I’ll start with an overview of what changed across these offerings, as well as within the 2020 offering during the pandemic onset.

Both years’ iterations featured the same Learning Objectives and broad approach (compare syllabi here and here). As is true in all sections for the past 10 or so years, both iterations featured a series of assignments with real data collected during the term from student activities. These assignments were handed in and graded online, as usual. Both iterations featured three tests and a cumulative final exam. In January 2020, these tests started out as Two-Staged, but I abandoned that practice once the pandemic hit as I couldn’t figure out the logistics. In January 2020, I started out using iClickers during each class (as I’d been doing for years) and after the pivot I kept the questions but didn’t attempt to record results — those participation points were considered completed. By January 2021, I learned to quickly create a “quiz” in Canvas that students could use to respond to clicker-style questions (even if they were joining asynchronously). As always, many students approach this course with trepidation, often after having identified as being “bad at math” — sometimes holding tight to this identity for decades. Anxiety management is a regular part of my teaching practice in this course. Let’s just say anxiety was, ahem, heightened in both of these years.

******My memory of the emergency pivot at the onset of the pandemic, because apparently I need to write this******

UBC didn’t officially close campus until end-of-day on Friday March 13, 2020, but rumours started circulating a couple of days before. On Wednesday or Thursday of that week, I hosted an impromptu (f2f!) session with colleagues to share what little I knew about pre-recording and uploading videos in Canvas — I’d given a talk in Florida at the end of January (including being on the last flight from Vancouver to Toronto that had started in Shanghai, the epicentre), and had learned some basics of using Camtasia to record a lesson with pauses for clicker questions. By Friday morning, I had already announced to my students that they were welcome to stay safe at home and watch the recording instead. A handful of students arrived, maybe half dozen each section, and I stood at the door and handed out disinfectant wipes for students to wash their desks and chairs. We didn’t know then that COVID was primarily passed through the air, so no one had masks. I remember leaving my office that afternoon after receiving that official email of the closure, not knowing when I’d return, grabbing everything I could carry home with me on the bus. I remember when I was getting off the bus, the driver was remarking to someone how unusually empty it was, and I chimed in that UBC had just closed because of the pandemic. It was news to them. Turns out, I have visited my office just twice since that day (late March and August), both times for 30 minute visits to grab what I could carry. This is the longest I’ve been away from UBC campus since I started as a graduate student in Fall 2003.

I had exactly one weekend to figure out how to teach my students statistics online, and I knew pre-recording everything wasn’t going to work for me. I chose live synchronous classes using Collaborate Ultra (CU), because it was FIPPA-compliant and already integrated into my Canvas course so I didn’t have to direct students elsewhere or figure out how to save and upload the videos for those who needed to watch later. I don’t have a home office, so I perched at my dining table in our open-concept space, overlooking the galley kitchen, living room area, and modest balcony. I’ve been working there ever since. Because of bandwidth concerns, and how CU is set up, I never saw most of those students’ faces again. That was heartbreaking.

When I reflect back on that time, I think of it as a time of terror and chaos and confusion, which translated into about ten million emails. My heart broke daily to hear my students’ stories and anxieties poured out in my inbox, and especially those who were waiting for my permission to book a flight to home before the borders closed. My answer was, of course, a resounding yes — we can figure out how to solve their absence later. They needed to stay safe. I remember at least once a student who joined class from a lineup at an airport, which drove home how desperate students were, but also how much I just needed to keep teaching. That hour to focus on statistics was an island of distraction for some students, and for me too [yup, I’m crying now… like I did most days of 2020… shout out to my husband for going through all this alongside me]. We were all in panic for our safety — grocery store shelves were emptied, we were afraid to leave our homes at all, insomnia was rampant. I also remember the community of colleagues that strengthened on Twitter during those early days. We posted threads of support and ideas, like what it was like to give an exam online using different features and settings, so other colleagues could learn from what happened. We shared resources and advice in a flurry — I probably spent as much time on Twitter as anything else, and that was such important, vital time. Thank you to my husband, my students and my Twitter community of colleagues for being a life raft in those early days.

************************

Although the plan for January 2020’s offering had started out business as usual, by the end of the term, everything post-March 16 became optional. Some students thrived (relatively speaking) during this time, but many were crippled by anxiety, fear, emergency travel, and general chaos of living through the terrifying onset of a pandemic. Policy changes at the UBC and Faculty of Arts levels opened the gates widely for late withdrawals and other accommodations. I and my colleagues worked hard to translate those policy changes at the Department and course level to support our students as best we could. Ultimately, our PSYC 218 teaching team agreed to ample accommodations, including essentially treating the last third of the course as optional. Implementation details were a bit different across sections; see Syllabus 2020 Amendment 2 Options for End of Term PSYC 218 001 and 002 for where mine landed.

By the time I was preparing my January 2021 offering, I had already taught two entire courses during a pandemic. I also had the experience of teaching the last third of this very course online during the emergency pivot. Both of these sets of experiences made this course an extra emotional one, due to exhaustion and memories of panic. But also, I and the students were much more fluent in all the technologies and how to run/manage class time, which made the logistics easier. Rather than making many changes, like I had for PSYC 217 in Fall term, I kept my PSYC 218 course as similar to “business as usual” as I could. But I layered in lots of flexibility in deadlines, as well as the option to customize the weighting of some course assessments; see the syllabus for more details on assessments and policies. I also switched over to Zoom, as it was now integrated in Canvas and FIPPA-compliant. This change made it possible to see some of my students’ faces every class period — which made an immeasurable positive difference to my well-being and enthusiasm each day. I remain so grateful to those students who chose to turn on their cameras.

So, with this lengthy backdrop in mind, let’s begin exploring student feedback! I’ll start with quantitative results. As shown below, response rates have been down since the pandemic began. The year before all this (W2018 Term 2, starting January 2019), my response rate was 44%. Since then, the response rate has been 31% (just after the emergency pivot 2020), and 30% (this past term). I’m actually surprised it’s stayed this high, as I can’t offer class time in the same way I used to do (folks just exit, whereas in The Before Times everyone in attendance would remain seated with a kind of social norm present to complete them).

Because the response rate has dropped, it is tough to compare data from The Before Times to after. Aggregate ratings from 2020 and 2021 appear a bit higher than 2019, but given the response rate, it’s possible that the folks with the less positive attitudes simply didn’t respond. But the big picture is that the numbers are (remarkably) not all that different. The one exception is Fair Evaluations. Since the pandemic, I cut Two-Stage tests (which were identified as a source of challenge in PSYC 217), and offered more time for tests, as well as more flexibility and choice. Fair Evaluations item was exceptionally high in 2020, likely owing to the tremendous flexibility from the pandemic pivot onward (basically, if you submitted something, it only counted if it improved your grade, but you didn’t have to submit anything else–a decent proportion of LOs were already measured by then). 2021 ratings were also higher than The Before Times, which could have stemmed from lack of two-stage tests (while keeping the full period of time for relatively few test questions), and/or offering the ability to customize how much the tests, final exam, and other components were worth. When we go back, I’ll likely reinstate two-stage tests, but continue the customization options and flexibility. If dropping two-stage tests was driving this perceived (relative) boost in fairness, that item should drop back to its former level.

Qualitative data time! This is always an emotional and difficult undertaking for me, which is why I start with quant to orient me to what I’m looking for. I have 7 pages to work though for 2020, and 6 pages for 2021. I try to roughly code student comments into four quadrants along two continua (thanks to Jan Johnson for teaching me this strategy about a decade ago). The first is valence (positive-negative) and the second is ‘under my control, changeable’ on the one end, and ‘not under my control or not willing to change’ on the other. ‘Not willing to change’ is usually because I have data behind that decision (but I might rethink implementation) or it’s just not possible given the time-effort that I have to give in the context of my other commitments. It’s also pretty common to read evaluations that directly conflict with each other (I loved this! alongside I hated this!), so I need to look for common themes.

Key points from January 2020 offering qualitative data. At the risk of sounding boastful, I must admit that, by far, the most frequent comments were about positive qualities I brought: #1 care and compassion (especially during COVID but also before), #2 enthusiasm and passion, #3 fun and engaging and participative (often with caveats like surprisingly, for statistics; even when moved online), #4 encouragement and support. I appreciated the specific things various people identified as helpful: weekly announcements including wellness moments, lots of resources and tips, well-considered syllabus and pacing, I was available and responsive, recorded lessons, two-stage exams, lots of in-class participation, thorough preparation and organization, fair assessments that broke the course up in manageable ways (even if it looked like a lot at the start). One person noted the helpfulness of studying from explicit Learning Objectives, and another remembered I brought granola bars in case they hadn’t had breakfast. I think one bigger lesson here is that lots of little touches I’ve added here and there over the many years of teaching can add up to many opportunities to reach one more studentI’ve been teaching my own courses since 2008. This didn’t happen overnight. Each of these additions came from reflective practice, listening to my students, reading evaluations carefully (trust me, they weren’t always this positive), going to teaching-focused conferences/workshops and learning from colleagues, staying open to change, and persisting through incremental change given the capacity I had that term (especially when my dreams were bigger than what I could reasonably offer at the time).

To be honest, there weren’t too many strong themes emerging from the negative side, but each of these were identified by 2-3 people: expensive (especially SPSS — but now as of 2021 that’s free thanks to UBC!), wanted more time for the individual portion of the two-stage tests, desire for practice tests *with* time pressure. One person noted that my energy and reassurance was “just too much” sometimes, but given those qualities were major themes that lots of people found helpful, I can’t dwell on this. Can’t be all things to everyone.

Some example comments:

“She does her best to create the best learning environment for students, with lots of opportunities for connecting and applying ideas. She teaches a tough subject but makes it very easy to understand and having her as a teacher helped me achieve a lot this semester.”

“Dr. Rawn stepped up and illustrated an incredible amount of compassion, consideration and honestly, love for us during the pandemic. She illustrated how much she cared about us and treated us as humans rather than just students. Words aren’t enough.” — I admit, this one made me cry

“Encouraged an environment of learning and encouraged everyone to push their boundaries. Showcased care, affection and concern for every student and was very involved and helpful to anyone who wanted the help.”

I was surprised that there were no comments whatsoever about the technological choices I made in that frantic flurry, choices that seemed so important in the moment. Like many faculty thrust into emergency online teaching, I agonized about and spent countless hours learning and choosing and implementing and troubleshooting various technologies. Was the tech simply not salient in hindsight… because I put in that time and energy so they went reasonably well? Or did all the technology issues fall away, taking a backseat to the feelings of compassion, care, enthusiasm (despite our circumstances), and sense of common humanity it seems I was able to convey? As I concluded in my last post, we teach students, not topics or courses. I’ll add here that we teach students, sometimes through technology and sometimes despite the limitations of technology. What matters is not the specific tool, it’s how we’ve used it to reach and support the human beings, learners, students, who are looking to us to light their path.

[On returning to writing: I’m returning to finish this after almost a month away from this post. Along with taking some time off, and responding to a surprising number of emails, I’ve been doing a fair amount of reading (highly recommend Oluo’s So you want to talk about race, Morton’s Moving up without losing your way, and Bear & Gareau’s Indigenous Canada). Sometimes I need to write, and sometimes I need to read.]

Key points from 2021 qualitative data: These data indicate that the course went very well in the eyes of most respondents. Most comments could be grouped like this: (1) feeling my concern for students and their learning, for example through accommodations and flexibility, encouragement, and using student feedback; (2) clarity, strong organization, and preparation that helped scaffold learning and keep people on track; (3) fun and engaging, despite the difficulty and high standards; (4) I brought passion, enthusiasm, and encouragement. Negative comments were hard to code, as they tended to be fairly idiosyncratic. Four people wanted more straight lecture with less participation and/or breakout groups. Four people mentioned tests in a negative way: two people requested more — and more challenging — practice tests; two people requested more questions on tests (so each is worth less). For comparison, four people mentioned tests in a positive way: they were applied (rather than drills or memory), fair learning assessments, and appreciated the extra time I gave. Again, comments about the technologies were largely absent. Two of my favourite comments:

“I was very afraid of this course for so many reasons but Dr. Rawn has been absolutely amazing and I genuinely wouldn’t have been able to achieve what I have and learned this much if it wasn’t for her. She is so incredibly caring towards us and does (way too much) work just to help us! The lectures were always engaging and participation was encouraged. The switch to online learning seemed flawless (although I know it wasn’t) and she was always so well prepared and laid  everything out for us in a way we could understand. Also I loved the autonomy given to us by being allowed to choose our weighting for certain marks. I cannot thank Dr. Rawn enough for this semester and how much she helped in terms of this class but also be a positive part of my life during these sad times.”

“PSYC 218 is not an easy course but having Dr. Rawn as the instructor made things a lot more bearable. She is passionate about the topic and that is transmitted through the classes to students. She goes the extra mile with her teaching, implementing strategies such as the Self–Determination Theory so that her students feel more engaged with the
course and it is definitely working. Furthermore, her analogies, i–clicker questions and examples are incredibly useful. A lot of the things she implements are things that I would love to see other teachers doing as well. But what I like the most about her is the transparent and effective communication she has with her students. She actively seeks feedback
and that is why she knows how her students are feeling, she makes us feel understood and supported.”

Thank you so much to everyone for your time and engagement with this feedback process!

Next time on the blog… I reflect on student feedback and my experiences in PSYC 417 Advanced Seminar in Psychology of Teaching and Learning, which was the very first course I taught fully online, back in July-August 2020. (Note that course code is changing to PSYC 427 for 2021, and I’ll teach it in Winter Session Term 2.)

Reflecting on Research Methods (Pre- and Mid-Pandemic)

As I mentioned in this previous post, I am working through feedback from my students. All quantitative data, as well as links to all previous blog posts (since 2011), are available here. ALERT! This post has become ridiculously long. Writing it is helping me think and process this overwhelming year, so I’m just going with it.

For this installment, I focus on PSYC 217 — the course I have taught the most: 24 sections and almost 2000 students since 2008. I regularly teach this course in the Fall term, two sections back-to-back of almost 100 students each. This post will focus on my pre-pandemic Fall 2019 results (business as usual), versus results from my mid-pandemic Fall 2020 online offering.

Both iterations featured a group research project scaffolded by labs led by Teaching Fellows (TFs), that culminated in an individual APA-style manuscript as well as a group poster. This course component is the same across all sections for the last 10 years. Over summer 2020, I worked with two graduate students as well as the other instructors to create Canvas modules to support all PSYC 217 students through each lab, in preparation for the fully online experience in the Fall. During the term, TFs continued to act as support, trying to guide groups through their projects.

In The Before Times, I assigned almost every chapter in the text (and a few short supplementary articles), held classes three times a week for 50 minutes, and measured learning primarily through the project deliverables (as above) plus 3 tests and a final exam. All tests and the exam were two-staged. Here’s the syllabus, which honestly hadn’t really changed much in about 5 years. Neither had my lesson plans, which were dotted with clicker questions, and included many demonstrations and discussions and illustrative examples that had been honed over many years. With the help of Arts ISIT, I set up the system so all lessons were recorded and posted automatically on Canvas (yes, even back in Fall 2019).

Fall 2020 was not business as usual. Early in Summer 2020 I hastily decided that my old lesson plans and course strategies would largely not work in a fully online environment, that also needed to support a fully asynchronous experience for learners joining from all around the world. (In hindsight, I probably could have adapted more than I thought I could, but that needed 9 months of online teaching for me to realize.) See the syllabus for details. Major changes: slashed content by about 3 chapters, shifted to two tests and a final exam (none of which were two-staged), added weekly low-stakes quizzes (from the textbook publisher’s materials), added option to customize some aspects of assessment weighting, and drastically changed how I thought about class time. Mondays became Q&A, where I answered questions from the previous week’s discussion posts, and answered questions live. (Unfortunately I labelled this as “optional” so it wasn’t well-attended/watched despite folks who came finding it really helpful). Wednesdays I held class in a form similar to The Before Times, featuring selected topics from that week’s chapter and bringing people together for discussion and demos as best I could. Fridays were for independent work (e.g., discussion posts, quizzes) and/or Labs. To help students stay on track, I curated everything in Weekly Modules, with an opening page that integrated everything to do and think about that week. As I had been doing for years, I closed each week with an announcement with reminders, though these were more elaborate than usual.

Now that I have oriented to the major changes in the course, on to student feedback! Quantitative data show that students in 2019 and 2020 rated the course remarkably similarly. Response rates were down a bit in 2020, but not to a worrisome degree. Click to enlarge the graph below:

Interestingly, clear expectations did not change, which might be attributable to the doubling or tripling of efforts to keep students on track with reminders and organizers — that extra effort might be necessary to keep commensurate with face-to-face. Slight drops in communicates content effectively and inspires interest might be related to the relative drop in synchronous class time. For perhaps the first time ever, fair evaluations nudged a little higher than inspires interest, which might be related to the customize grade weights. Concern for students has long been an area of strength for me, and I’m not surprised to see this rated highly this year due to the lengths I took to approach all decisions with compassion (though cf. 2018 Section 2?). Overall efficacy is a bit lower (in Section 1 only?) but doesn’t seem meaningfully so.

Qualitative data time! This is always an emotional and difficult undertaking for me, which is why I start with quant to orient me to what I’m looking for. I have 7 pages to work though for 2020, and 6 pages for 2019. I try to roughly code student comments into four quadrants along two continua (thanks to Jan Johnson for teaching me this strategy about a decade ago). The first is valence (positive-negative) and the second is ‘under my control, changeable’ on the one end, and ‘not under my control or not willing to change’ on the other. ‘Not willing to change’ is usually because I have data behind that decision (but I might rethink implementation; see example below) or it’s just not possible given the time-effort that I have to give in the context of my other commitments. It’s also pretty common to read evaluations that directly conflict with each other (I loved this! alongside I hated this!), so I need to look for common themes.

Key points from 2019 qualitative: On the negative side, the modal issue was the quizzes. Quite a few students (~9) mentioned they felt quite pressed for time on the individual portion, and another couple mentioned there was too little time remaining for the group portion to be effective. I need to think more about this. The in-term quizzes are already quite brief, and I’ve tinkered with length before — too few points and students get stressed that each question becomes valued at close to a percent of their final grade. This is the one downside of a 50 minute class and two-stage exams. But this timing thing is becoming too common a complaint for my liking.

Many students mentioned the use of examples in a positive way, particularly appreciating how applied they were. But also a few students mentioned a desire to be given more examples, or more different examples, or using the examples from the textbook in class. I’m not sure how to do this without increasing lecture time (sacrificing demos, activities, engagement).

On the positive side, the clear themes are two of my signature strengths in the classroom (as I have learned because of reading evaluations over the years): bringing enthusiasm every day, and caring about students — both their learning and broadly as fellow humans. The vast majority of comments of any kind included mention of one or both of these qualities. There were various additional notes of what worked well: asking questions and activities to keep people engaged, using real applied examples, recording and uploading lessons (2 people), office hours, explaining reasons why I do things a certain way, structure/sequencing, wellness moments in weekly announcements. My two favourite comments:

She’s very passionate, which makes learning more interesting and easier. I really liked how she included the class a lot and used questions and examples to actually help students learn in class, instead of expecting them to just take notes and learn later, like most teachers. I also really liked to set up of the course with the groups and labs and group tests. Group tests and time to discuss in class about questions with my group really helped me to learn.” [emphasis added]

Dr. Rawn is one of the best professors I have ever had. She made the classes so engaging and interesting, and time and time again showed her genuine concern for her students’ learning and wellbeing. I visited her office hours once and overheard talking to another student about different ways he could improve his wellbeing and performance in a course. I just wish she had longer office hours because I could tell she wants to connect more with her students but has a lack of time to do so.” [emphasis added… to highlight a sentence that fills me with All The Feels. Check out the opening bullet of my previous post.]

Dr. Rawn has perfected the formula for this class.” –> this student gets the decade+ process behind the course as it was. Which is why I’m filled with terror to begin reading 2020 comments… but here we go…

Key points from 2020 qualitative: Wow. That was a lot to process. On the negative side I have a long list of things people mentioned as not working for them (which was starting to alarm me), but not much in the way of clear themes. Upon reflection, I take this as a good sign, in light of the fact that I simply cannot please 200 people all the time with all the decisions I make across a 13+ week timespan (while in a pandemic teaching a large class online for the first time). A few folks mentioned they wanted more “lecture” (i.e., me talking) and less reliance on the text and less participation. However, at least as many people (if not more) appreciated the engagement in active learning and value of student-directed Q&A (plus, you know, All The Research on active learning). A few people noted there were too many small assignments, but again a few people mentioned appreciating the range of activities available to show learning. If I group a few comments together about labs and the paper, there are some folks who didn’t feel sufficiently supported in the lab portion (Fair enough. We all tried our best and knew things weren’t as smooth as in person.). One student mentioned wishing a better guide to help them navigate course content, but many students mentioned the navigation, organization, and structure of the course as a real strength. Interestingly, enthusiasm barely made it on the list at all — apparently that’s something that comes through in my face-to-face teaching but not so much online.

And yet my heart sinks to read there was one time I didn’t respond in a caring way to a student and it clearly upset them deeply and soured their whole experience of the course. Reading a comment like that just breaks my heart. I am human and I make mistakes in the moment and wish I could take back how words came out of my mouth, and what exactly those words were. But I can’t. I tried to fix it then and that clearly did not work. So although I deeply regret that I couldn’t reach that student, I have to force myself to learn and move on, to always live what I know: every single interaction with a student matters. Even when that interaction is happening anonymously online. And I have to recognize that, by far, the biggest theme across 2020 qualitative comments was that I cared.

Many students mentioned that I cared and that made a difference for them. I cared that they learned, and they noted I worked hard for them which made them motivated to work hard too. I responded to emails consistently and in timely ways, and I asked for feedback each week and used it to make real changes students experienced. I also cared about them as human begins who were learning in a pandemic. Many students mentioned my concern for their well-being, compassion and flexibility, Wellness Moments in announcements, and how I chose to highlight self-care and compassion in examples I used to teach the content. I found it interesting that each of these specific choices was mentioned more than once, and this theme of care was a much bigger deal than anything about the course content or technology used or assessments or anything else. We teach people, not topics or courses. My two favourite comments:

Dr. Rawn was highly adaptive, and showed great care and concern for her students. She produced a safe, and engaging learning environment. It was clear that she had her students well–being in mind when she designed this class. Her lectures were effective in producing clarity, and her Ask Dr. Rawn Sessions allowed us to further learn, and develop a sense of community in discussion our ideas with peers.

Dr. Rawn went above and beyond to teach this course. Her lectures and labs were very engaging and fun. Also, she provided useful resources. Even outside the class, she made sure that the students were on track with quizzes, discussion boards, and take home surveys. When I first came to this class, I had little hope with how it was going to be taught, considering we couldn’t conduct experiments in person. But Dr. Rawn gave me so much hope and motivation towards my project. I really appreciate a professor like this who overcame the problem of COVID–19 and social isolation, and to be able to bring us all together and work hard.

Next time on the blog… I reflect on student feedback and my experiences in PSYC 218 Statistics, where I changed relatively little about the course, and had 2 terms of experience teaching online under my belt already.

Reflecting on the first Seminar in Applied Psychology of Teaching and Learning

In May-June 2019 (Summer Term 1) I taught a pilot course: Seminar in Applied Psychology of Teaching and Learning. Please see the first syllabus for details on this pilot offering: Syllabus.PSYC417.S2019.Rawn.SeminarApplPsychTeachLearn.V2

Course Overview

This course is designed as an intensive, active seminar to help you apply your understanding of psychological science to help other people learn, while developing professional skills relevant to teaching. You may begin to shift your identity from a student to a member of a teaching team.

If you enjoy this course, you might consider applying to become an Undergraduate Teaching Assistant in the Psychology Department or elsewhere. This course will help you strengthen that application. Yet this course is designed as a springboard for many future work or study endeavours (e.g., course/curriculum design, instructional design, management, teaching at any level, human resources/training, graduate school, group facilitation, academic

What did Students say?

All 10 students from the Pilot course in Summer 2019 provided rich feedback throughout the course as well as in the Student Evaluations of Teaching at the end of the term. Thank you!

Quantitative results are reported here. The qualitative comments, as usual, help to contextualize the numbers. Students reported feeling challenged, in a positive way. The highlights:

Dr. Rawn’s high expectations of us and bid to push us out of our comfort zones made certain parts challenging but it was welcome, given the standing of the course and the objectives it sets out towards. Really well designed for students who might be considering become TAs or instructors themselves in the future.”

I really loved the sand–box elements of the course in which we were given the opportunity to help build elements of the class and muddle through behind–the–scenes challenges.”

The discussions, peer reviews, hands-on activities and presentations (even though I dislike those) are the most effective parts of the course at promoting learning.”

Great course! One of my takeaways that was not an explicit part of the curriculum was actually the structure and planning of a graduate–type seminar (which i will need for my later teaching).”

Overall, this course was interesting and there isn’t anything like it at UBC right now so I think many students would like it and benefit from taking it.

In planning the next offering (coming Summer 2020 Term 2), I made two key changes in response to problems fairly identified by students (plus one more key change). First, I will not be counting marks for the first Reading Reflection (#0). A couple of students reasonably pointed out that it was difficult to know how to write that first Reading Reflection, especially without a rubric (which I hadn’t created yet). So although I’ll still expect a best effort and will “grade it” accordingly, I won’t count those points. This year, I’ll also be able to give more concrete tips in advance because the rubric exists already. These concrete tips will help address a request by a couple of other students for more clarity on assignments.

Second, I have moved the course material on peer review and using rubrics earlier in the term. A couple of students noted that their peer reviews were not as reliable or helpful as they’d hoped, especially early on in the term. Hopefully this earlier discussion will help improve the usefulness and reliability peer reviews. (Note that peer review scores ultimately contribute very little % to each grade, and they are all checked and adjusted if needed by our TA or by me.)

A third change came from my own reflections on the assignments and grading them, along with feedback from my TA Kyle Gooderham (thanks Kyle!). In hindsight, the major project was over-complicated. Asking students to invent a study strategy or learning resource, pilot it, and anchor it in the literature was just too much (especially in a 6 week course). Thus, I have revised the major project to clarify its purpose. In a nutshell, the task is to take an existing strategy or resource, ground it in research evidence, and use that evidence to convince others to use it (or not to use it, if the evidence is weak/contradictory).

In an unprecedented move for me, I actually have next summer’s syllabus prepared. Of course, it’s subject to change at this point. But I wanted to do it now while the course was reasonably fresh, and so I can bring it to the Psychology Department to propose its own course code. If you’re interested, here is next year’s draft: Syllabus.SeminarApplPsychTeachLearn.2020.V1.TOPOST.August.2019. Feedback is welcome!

Student Evaluations of Teaching Feedback: PSYC 217 from Fall 2017

Thank you to all the students who provided their ratings and comments on all my courses last year (and always). I always take care to analyze the data and consider written comments (see this page for mutli-year graphs and averages), and I often write a reflection on my blog while I prepare for the next set of students. Today is that day!

As you can tell from this graph, numerical ratings last year were consistent with  those I have received in recent years, and indicate a generally high level of satisfaction with the course. Notably, students in Section 1 (at 9am) rated the fairness of evaluations as higher than typical–which must be a fluke because I changed them very little from the previous year, and not at all from Section 2.

Reading through the comments was interesting because it also showed a difference between Section 1 and Section 2. For the most part, students in Section 1 wrote about how engaging the classes were and how I showed concern for student well-being throughout the course. A couple of people noted that tests were challenging in both content and timing–but that’s not out of the ordinary. When I moved on to read Section 2, similar positive feedback appeared too, but a much greater proportion of students reported frustrations with the assessments on various dimensions (frequency, wording, difficulty, timing, length, two-stages). The assessments were the same for both Sections. I don’t recall seeing such discrepancy before. A part of me is inclined to chalk this up to randomness, but I also have a nagging sense that it’s time for a thorough review of my exam questions. Although I review the data each year and make small tweaks, I haven’t done a close examination and renovation in this course for a while. I’ve slotted that in my calendar for this fall term!