For as long as I’ve been teaching I have been analyzing qualitative comments from students, often sharing insights (see blog tag student evaluations of teaching and Teaching Evaluations (Archive W2008-W2020) for links by course). The last few years, although I’ve read the comments, I haven’t done as thorough analysis as I used to do. So, as I prepare for 2026/27 academic year newly refreshed from sabbatical, I wanted to dive deeply into my last three years’ of data. (Please see my Student Evaluations of Teaching page for quantitative analyses and an overview of UBC’s SEI measurement instrument.)
Here are my research questions:
- What do students think I should keep doing in each course, and what changes might I consider making this year? (What do I have capacity for? What is within my control?)
- For PSYC 217 Research Methods: Comparing 2021 to my fully renewed 2023 and adjusted 2024 offerings, how do students receive the course? Are there different strengths and suggestions before and after the renewal? Overall, is the renewal doing what I hope it is doing? That is, strengthening students’ critical understanding of our discipline’s dominant ways of knowing, while developing their (researcher) identities?
- For PSYC 218 Intro Stats for Psych Majors: I went from three midterm tests in 2023 to two in 2024, in order to decrease students’ experience of stress. I know it lost that checkpoint but did create some space for more review. How did students feel about this change? What approach do I take for next year?
- Transformation is the core concept of my new (draft) teaching statement. Is it supported by students’ reported experiences in my courses, particularly my recent offerings? How does it show up? I can make a case based on how I have approached my course (re)designs, and how I am positioned in my heart and mind as a teacher, but to make such claims I need a validity check from students.
To assist me with this analysis, I chose to use a large language model (LLM) in the form of Microsoft Copilot (Basic) at UBC–which means that the data stay secure within UBC and are not sent elsewhere. I receive Student Experience of Instruction data aggregated into one pdf file per section of each course. For each course each year (2021, 2023, 2024 for each of PSYC 217 and 218) I copied qualitative responses to all five questions from both sections, and pasted them into a word document. I skimmed the entire dataset so I had a sense of what students had written. There is no way to identify any students in the dataset, which means that the data meet low risk criteria for sharing with this instance of Copilot. I uploaded the file, with a few key prompting questions I’ll share in future posts with my take on the results.
Why did I use an LLM to help with this analysis? I am not a big user of AI tools. But I turned to it in this case for a few key reasons:
- I had available a secure, closed, option. Microsoft Copilot Basic ensures data stay secure and within UBC. I would not have used any other option which would have sent these data out into cyberspace.
- Given that all I had was written comments with no way to put them into context or ask follow-up questions, I couldn’t really use social constructivist or other approaches that rest on shared meaning-making. I’m looking for a basic thematic analysis of students’ perspectives to inform my prep for next year, evaluate some recent changes, and to begin to test out my new teaching statement focus (not in a comprehensive way, but in a starting way).
- Reading (even just skimming) student comments is an emotional roller-coaster. Elation and joy at successfully reaching students meaningfully gets shattered by someone else’s negative interpretation, sometimes of the exact same course element. It’s really hard to keep perspective (e.g., proportion and nature of positive to negative comments) and to maintain focus on what is in my control to change, when I’m feeling everything on the roller-coaster. I wondered if an AI tool could help me learn from my students’ feedback while caring for my own heart as a human being.
- Speaking of being human, I was also aware of confirmation bias (seeking evidence of what I already think I know), and other biases that might creep in to my conclusions. I figured AI doesn’t care about me and my ego, so has the potential to tell me a version of truth that might be hard for me to see (or admit) myself. “Bracketing” (trying to separate one’s personal connection to the data for qualitative analyses) is harder the closer it is to ones’ Self… and I can’t think of a closer dataset to my Self than students judging my teaching practice and course design.
- The time saving elements were appealing, but I would have painstakingly analyzed all 80 or so pages of small-font text (as I have for many years) if I thought it would reap substantially more meaningful results than what AI could do, given my purposes. And, I could always do that anyway if this didn’t work out.
What did I find? Stay tuned!