All posts by Isabeau Iqbal

Calibrated Peer Review: An introduction

Student

I am enthusiastically involved in a project in which I am helping instructors implement writing assignments that use student peer feedback into their courses (see note 1). I am loving this initiative and the learning; plus, it is a neat extension of the work I have been doing on peer review of teaching.

Today’s post is a brief introduction to Calibrated Peer Review (CPR), a web-based writing and peer review tool that is being used in one of the re-designed courses.

Calibrated Peer Review (CPR) automates the process of distributing writing assignments to the students and then manages a peer review process that involves four steps, in which students:

  1. Submit a writing assignment
  2. Undergo a process whereby their review skills are calibrated
  3. Review peers’ writing, and
  4. Assess their own writing assignment

The instructor need not grade the assignments and the CPR system automatically compiles grades (Likkel, 2012; Schneider, 2015).

According to the CPR website, compelling reasons to use CPR include that it:

  • Allows students to hone their writing skills
  • Helps student learn to use higher-order thinking skills (in the writing and review process)
  • Promotes students’ critical thinking abilities
  • Encourages students to gain a deeper understanding of the topic and discipline
  • Reduces time needed by instructors to grade

With the exception of the last point, research on the CPR has drawn varied conclusions about the effectiveness of CPR for the above.

Some of the questions that remain inconclusive in the literature are:

  • Do the students’ writing skills improve?
  • Does engaging in the process promote students’ conceptual understanding of X?
  • Do students feel more confident as writers?
  • Do students’ believe the CPR process helped them augment their conceptual understanding of X and/or become better writers?

I cannot yet comment on the above from personal experience because the CPR assignment I have been working on launches next week.  I can attest to the fact that, though instructor load may be lightened overall (i.e., when CPR is used in multiple assignments and/or in next iterations of the same course), the time involved in getting to know CPR and setting up the assignment has been significant.

Note 1: The project I am working on is a Teaching and Learning Enhancement Fund granted initiative. See here and search for “Bradley” (the principal investigator) for brief information about that TLEF.

References:

Likkel, L. (2012). Calibrated Peer Review™ essays increase student confidence in assessing their own writing. Journal of College Science Teaching, 41(3), 42-47.

Schneider, S. C. (2015). Work in progress: Use of Calibrated Peer Review to improve report quality in an electrical engineering laboratory.  Paper presented at the 2015 American Society for Engineering Education Zone III Conference, Springfield, MO.

Overview of Calibrated Peer Review (2016). Retrieved from http://cpr.molsci.ucla.edu/Overview.aspx

Photo by CollegeDegrees360 https: //flic.kr/p/cEJnWs, CC BY-SA 2.0

 

Research Projects – Past

Research Projects – Past

Social Network Analysis in Teaching and Learning

Manuscript: Poole, G., Iqbal, I., & Verwoord, R. (2018). Small significant networks as birds of a feather. International Journal for Academic Development, 1-12. https://doi.org/10.1080/1360144X.2018.1492924

Project: With Dr. Gary Poole and Roselynn Verwoord, we examined how post-secondary instructors use significant networks to support their professional growth as teachers and SoTL scholars.

Based on our analysis of existing research on small significant networks pertaining to teaching and learning, our research posed the following questions:

(1) How are educators using networks in their own contexts to expand, refute or build their stories of teaching and learning and of SoTL?
(2) Do instructors perceive greater similarity among network members than among randomly chosen colleagues?
(3) Are there relationships among perceived similarity, value of interactions, and impact of the network on one’s teaching and research on teaching?
(4) What strategies can be employed to enhance the value of one’s networks?

Student Peer Feedback: Between 2016-2017, I worked with two UBC instructors who implemented student peer feedback approaches in their courses. Together, we carefully considered the course design, researched and selected student peer approaches, implemented and evaluated these (in terms of student learning and other) and then made further modifications to the course based on the evaluation data.

Evaluation of the Course Design Intensive: Beginning in April 2015, I led a program evaluation of the Course Design Intensive, a workshop offered through the Centre for Teaching, Learning and Technology.  (Link to more information).

The Educational Developer’s Portfolio (February 2014-February 2016). I was part of a collaborative research project inquiring into the Educational Developer’s Portfolio. Collectively, we gathered data from educational developers through World Cafes and inquired into the possibilities for the Educational Developer’s portfolio and what is needed to support a culture of portfolios.  We produced two publications from this project: (1) The Educational Developer’s Portfolio Guide (2015) and (2) a journal article titled “Exploring the potential of educational developer portfolios” in To Improve the Academy.

Team members on this project were: Jeanette McDonald (lead), Natasha Kenny, Erika Kustra, Judy Chan, Debra Dawson, and Paola Borin (and, of course, me).

Planning the Integrated Respirology Module (November-December 2014). With colleagues from Pharmaceutical Science, I conducted a program evaluation study related to an integrated Respirology Module in the Faculty of Pharmaceutical Sciences. I interviewed members of the planning team to determine what they perceived were the successes and challenges of planning an integrated module. The intention of this research was to help with future module development.

Student and Faculty Member Perceptions of the Student Evaluations of Teaching: A Qualitative Study. (July 2013-June 2014). This was a collaborative project conducted with John Lee (who was an undergraduate student in the Pharmacy program), Marion Pearson, and Simon Albon. We have had our paper accepted to Currents in Pharmacy Teaching and Learning (scheduled to be published in Spring 2016).

How do Physicians Learn to be Good Patient Educators? (January-June, 2013). This was a study being conducted by Dr. Terese Stenfors-Hayes, when she was a Post-Doctoral Fellow at the Centre for Health Education Scholarship. In my role as research assistant, I conducted literature reviews and carried out data analysis.

Faculty Members’ Professional Growth in Teaching Through the Summative Peer Review of Teaching and Other Departmental Practices.  In 2012, I completed a doctoral research study.  Please see here for information on my doctoral research.

 

Significant conversations

Come Together

As I prepare for a pre-conference workshop and conference workshop at the International Society for the Scholarship of Teaching and Learning 2016, I am reading and blogging on how university instructors learn about teaching through personal networks (my four previous posts on the topic can be found here). The ISSoTL workshops are part of a research project I am collaborating on with Gary Poole and Roselynn Verwoord.

Today’s post looks at “significant conversations” and consists of my notes from a paper by Roxå and Mårtensson, two lovely people and terrific scholars I had the pleasure of meeting in 2008.

Reference: Roxå, T. & Mårtensson, K. (2009). Significant conversations and significant networks: Exploring the backstage of the teaching arena. Studies in Higher Education, 34(5), 547-559.

**************

This paper presents results from a study in which Roxå & Mårtensson surveyed 109 university instructors to learn more about their teaching and learning conversation partners. Participants were from the following disciplines: engineering studies, social sciences, humanities.

Participants responded to these questions:

  1. With how many people do you have engaging conversations about teaching and learning?
  2. Where are these conversational partners found?
  3. What characterizes your conversations? (Please describe them.)
  4. Do you consider your local professional culture to be supportive or nonsupportive of such conversations about teaching and learning? (this question was only included in the later questionnaires)

The researchers drew on Handal’s (1999) concept of critical friends to “focus the respondents on individuals with whom they had sincere and serious discussions about teaching and learning” (p.550).

Summary of results

With how many people do you have engaging conversations about teaching and learning?

  • 83% of respondents had up to 10 conversational partners (there were differences among the disciplines)
  • Roxå & Mårtensson found that “university teachers rely on a limited number of individuals to test ideas or solve problems related to teaching and learning” (p.556)

Where are these conversational partners found?

The majority of participants discussed teaching with colleagues within their own discipline.  Conversational partners, however, were located within or outside the individual’s institution and discipline, and therefore the authors concluded that “significant networks” have no boundaries surrounding them.

What characterizes your conversations?

Private conversations:  

  • conversations rarely took place in formal meetings (they took place “backstage”1)
  • though many were backstage, the conversations were not isolated from the surrounding culture

Trustful conversations:  

  • conversations were about a range of topics (intellectual and emotional)
  • there was mutual trust among partners and partners often shared similar interests and values
  • at times, the conversations did not align with the official discourse within the participant’s culture/context

Intellectually intriguing conversations:  

  • conversations dealt with important disciplinary content and challenges about how to support student learning
  • participants used these conversations to make sense of experiences, deal with problems, and plan/evaluate actions.
  • Roxå & Mårtensson found that most participants were not drawing on pedagogical literature and theory as they were having these conversations; nor were they making public the results of their inquiry. Rather, they were using “personal theories” (p.556)

“Do you consider your local professional culture to be supportive or non-supportive of such conversations about teaching and learning” (this question was posed to only 50 of the participants)

  • There is a clear link between how encouraging a culture is and number of conversational partners (i.e., in a supportive culture, individuals have more conversational partners)
Implications

Significant conversations have the potential to help university teachers see things through someone else’s perspective. They may shape and/or expand an individual’s identity as teachers.

In the words of the authors: It is likely that these conversations open up the possibility of constructing and maintaining–and perhaps partly changing–an understanding about the realities of teaching.” (p.555)

**************

Footnote 1: Erving Goffman’s wrote about the concept of “front stage” and “back stage” behaviors in his book The Presentation of Self in Everyday Life (1959). Goffman proposes that we have two different modes of presenting our selves: one when we are ‘on’ for others (front stage) and another when we let down our guard (back stage). For a succinct introduction to these concepts, see here at “Everyday Sociology“. 

University instructors’ key learning connections

PATOLA CONNECTION

“Data revealed that participants’ learning networks were based around both physically and emotionally close ties, which appeared the most homphilious with respect to occupation”(p.67)

As I prepare for a pre-conference workshop and conference workshop at the International Society for the Scholarship of Teaching and Learning 2016, I am reading about how university instructors learn about teaching through personal networks (sometimes referred to as “significant networks” or “social networks”). This work draws on Social Network Theory and is part of a research project I am collaborating on with Gary Poole and Roselynn Verwoord.

Today’s post is a summary of findings from the following paper:
Pataraia, N., Margaryan, A., Falconer, I., Littlejohn, A., & Falconer, J. (2013). Discovering academics’ key learning connections: An ego-centric network approach to analysing learning about teaching. Journal of Workplace Learning, 26(1), 56-72.

Below is a summary of the findings, as presented in the paper. Participants were 37 academics from UK-based universities.

The form of participants’ personal learning networks relating to teaching  
  • The highest percentage of connections that academics considered key to their learning and teaching were within their department and institution (56%).
  • Participants used their network’s expertise, information, and guidance, to carry out work-related tasks and  solve teaching-related problems. They reached out to people they perceived as being most helpful for addressing a given challenge/situation.
  • The majority of participants emphasized that a good personal relationship was critical for establishing and maintaining learning connections.
  • Most academics’ learning networks are comprised largely of other academics.
Homophily evident in participants’ networks relating to teaching
  • In this study, homophily was not seen in terms of gender or experience level but it was seen in the type of profession. That is, academics’ networks were comprised mainly of other academics.
  • Academics with more than 11 years of experience had colleagues of all ranks in their networks; however early- and mid- career academics mainly had colleagues who were at a higher rank than them (the authors refer to this as “hierarchical levels”).
Physical proximity
  • Participants’ networks are largely made up of people who are physically proximate and within the organization.
Length of time known
  • Academics’ networks were comprised of colleagues that individuals had known for periods of time ranging from short to long. As can be expected, the most experienced academics had the most diversity in their network with respect to the length of time they had known their colleagues.
  • Academics interacted most often with strong-tie connections than with weak-tie connections. (For a brief introduction to the concept of strong and weak ties, see here).
Discussion section of paper – More take-aways

“Data revealed that participants’ learning networks were based around both physically and emotionally close ties, which appeared the most homphilious with respect to occupation” (p.67)
This suggests that creation of learning networks and maintenance is aided by:
– Physical proximity
– The strength of tie in terms of friendship
– Homophily of occupation

Participants (often) shared more than one type of relationship with their contacts (i.e., individuals in the network were both a professional acquaintance and friend).

“Despite the widespread popularisation of technologies, participants tended to favour face-to-face encounters for their learning, which occurred largely with their institutional colleagues.” (p.67)

Academics’ networks are diverse in terms of hierarchical position (compared to self) and length of time they had known their contacts. This type of diversity should be favourable for learning new teaching practices.

“Given that heterogeneity in the network structure was more visible among experienced participants, we may hypothesise that their networks stand a better chance of promoting serendipitous learning and innovation.” p.68

Recommendations for educational developers
  • Help academics recognize value/potential of personal networks for learning about teaching
  • Teach networking skills and raise awareness about importance of networks.
  • Provide more opportunities for networking.