educational development

Evaluating the Course Design Intensive

Rainy Days and Mondays

As do several other teaching and learning centres, ours offers a Course Design Intensive (CDI). During this 3-day course, participants make progress on the design or redesign a course for post-secondary students.

Since 2015, I have been leading a program evaluation of our CDI. The process and methodology have been messy and inconsistent…and have taught me a lot about program evaluation. In this blog post, I share on the retrospective pre-test (RPT), one of the approaches I have used as part of our multi-faceted evaluation [for a 2-page description of our program evaluation, see here].

Retrospective pre-test

The retrospective pre-test is a survey that is administered at the same time as the post-test. Learners are asked to answer questions about their level of understanding, confidence or skill after an intervention. They are then asked to think back to their understanding prior to the intervention and to answer the same questions, but from the perspective of the present moment. See here for more information, including a brief description of strengths and weaknesses of this approach.

What we used to do before

Prior to December 2016, we did the following:

Before the CDI

The survey asked participants to consider the learning outcomes for the CDI and, using a Likert Scale, rate: (1) how important is this skill in course design?; (2) how confident are you in your current skills in this area? [see here for pre-CDI program evaluation survey].

On day 3 of the CDI

On the last day of the CDI, participants would complete a survey that had the same questions as above (#1 and #2) and this question: (3) how helpful has the CDI been in learning this skill? [see here for post-CDI program evaluation survey]

What we do now: Retrospective pre-test and post-test

Instead of administering two surveys at two different times, we now administer the retrospective pre-test and post-test at the same time. After consulting different articles about the benefits and disadvantages of one method over another, I surmised that the main advantage, in the case of the CDI, was mostly practical: one survey vs two.  To access our survey, see here.

Additional resources on the retrospective pre-test

The Retrospective Pretest: An Imperfect but Useful Tool (Harvard Family Project, 2005)

The Retrospective Pretest Method for Evaluating Training (Evaluate Webinar, 2015)



  1. Thanks go to Dr. Chris Lovato for introducing me to retrospective pre-test.
  2. Photo credit: Bill Dickinson “Rainy Days and Mondays” https ://




Educational developer skills, knowledge and competencies

Lavender Field (Beauty of Simplicity)
Over the past few days, I have re-read a few texts that address the skills, knowledge and competencies of educational developers. (The texts are listed at the bottom of this page).

Rather than re-hash the details here (and because I can’t reproduce the useful [copyrighted] visuals), I want to point you to specific sections of these resources as they are useful for helping educational developers articulate, assess and reflect on the skills, knowledge and competencies that we bring to our work.

To see figures that visually depict competencies of entry-level, senior and director-level educational developers, go to pages 19, 20 and 21 of the McDonald, Kenny, Kustra, Dawson, Iqbal, Borin, & Chan (2016) publication (similar visuals are in the Dawson et al. paper).

For example, a look at Figure 3.2 on p.20 of the Educational Developer’s Portfolio Guide (McDonald et al., 2016) shows:

Senior Educational Developer: Competencies

Educator, Course Design, Instructional Strategies, Program Development Strategies, Educational Strategies

Senior Educational Developer: Skills, Abilities and Knowledge

Interpersonal skills, Conflict Resolution, Mediation, Diplomacy, Trust, Listening, Empathy, Educational Leadership, Self Reflection, Peer Mentor/Coach, Model, Consultation, Formal Education in Pedagogy, Organizational Behaviour, Literacy

(I am highlighting the senior level because this is where I position myself)

Want to rate/assess your own skills, knowledge and attributes? –> Go to Appendix C (pages 67 and 68) of the Educational Developer’s Portfolio.

To read more about skills, knowledge, competencies and threshold concepts in educational development, see:

And though I haven’t written directly about Dr. Julie Timmerman’s work in this post, her paper is excellent and well worth the read:

  • Timmermans, J. (2014).  Identifying threshold concepts in the careers of educational developers.  International Journal for Academic Development, 19, 305-371. doi:10.1080/1360144X.2014.895731


A FAILFaire in educational development

FAILFares are not about celebrating failures, but rather about providing ‘a space in which people can celebrate taking risks and the open and honest sharing of information …so that we could learn from these things.’  – (Trucano, 2011)

I recently read an interesting blog post, by Michael Trucano (@trucano), that described his experience of organizing and hosting a FAILFaire for the World Bank. It got me thinking about the application of this concept to educational development. 

A FAILFaire, I learned from the post, is an event that recognizes projects, within an organization, that have not worked: “the pilots that never got anywhere, the applications that are not delivering, the projects that are not having any measurable impact on the lives of people, and the cultural or technical problems that arise.”(MobileAction NGO, quoted in blog post). The philosophy driving FAILFaire initiatives is that sharing lessons about what doesn’t work can encourage people to be innovative and entrepreneurial because lack of results if a likely outcome of any innovation.

Trucano cautions that these events are not about celebrating failures, but rather about providing “a space in which people can celebrate taking risks and the open and honest sharing of information (even and especially about what doesn’t work or isn’t working) so that we could learn from these things.” 

He proposes that FAILFaires have two main objectives:

  1. to generate lessons learned from experience and determine how these may be useful to other colleagues working on similar projects;
  2. to encourage open dialogue among colleagues about how to respond to professional challenges, in the hopes of addressing these more productively.

In his blog post, Trucano shares seven ground rules for presenters and also offers other practical suggestions and lessons learned from his own experience.

Possible Applications to Educational Development

Within our educational development community, I see many applications. Keeping the two overall objectives in mind (above), FAILFaire events could include:

  • A FAILFaire within your Centre for centre staff only
  • A FAILFaire at a conference (i.e. lessons learned from educational development lessons or learned in SoTL research design or implementation)
  • Help a receptive department or Faculty organize a FAILFaire in which faculty members and other instructors share lessons learned from their teaching and learning failures (if you manage this, please let me know!)
  • Encourage those you work with to reflect on lessons learned from risks they took in teaching or educational development
  • If you blog or do podcasts, consider sharing an educational development or teaching failure and what you have learnt from it (I’m going to hold myself accountable to doing this in the next few months).
  • And/or, as University of Waterloo’s Centre for Teaching Excellence has done, make it the theme of your annual teaching and learning conference (for 2016, this CTE has made the theme “Learning from Challenge and Failure” — well done Julie Timmermans and colleagues!)

Why bother? Because:

Only if we understand what doesn’t work in this field, can we collectively learn and get better.* 


*FastCo article “How FAILFaire Turns Epic Fails Into Successes”

Thank you to the lovely Dr. Julie Timmermans for the conversations that inspired this post.


Educational developer’s portfolio

Educational developer’s portfolio: Resources for creating your own

For the past two years, I have had the pleasure of collaborating with a dynamo group of educational developers on the Educational Developer’s Portfolio*. This initiative allowed me to take my interests and experience in the area of teaching portfolios, and apply it to educational development. Good stuff!

The authors of that Guide are delighted to be able to share this free downloadable resource with our community!

Also, Judy Chan and I recently offered a webinar on the Educational Developer’s portfolio for the Educational Developers Caucus. Below are some resources that might be of interest:

Additional resources can be found on WikiPODia, from the conference session that Jeanette McDonald, Debra Dawson, Erika Kustra, Judy  Chan and I co-facilitated (Natasha Kenny and Paola Borin collaborated to plan the session but were not able to make it to POD 2015 ).

Resources from POD include, but are not limited to:

  • framework for aligning a portfolio.
  • worksheet for beginning to develop a section of your educational developer’s portfolio (‘workshops facilitated’)

And, for those who like to see samples, here are portfolios Judy and I showed and talked about during the webinar:

Celebrating the ED GuidePhoto: taken by Jeanette McDonald. Cake made in honour of first EDC Guide (ours!) in the Series.


*”An educational developer’s portfolio is a tool used to articulate, reflect upon, and provide evidence of an educational developer’s beliefs, values, ethical principles, practices, approaches, development, and impact.” (McDonald et al., 2016, p.12)


Guide reference:

McDonald, J., Kenny, N., Kustra, E., Dawson, D., Iqbal, I., Borin, P., & Chan, J. (2016). Educational Development Guide Series: No. 1. The Educational Developer’s Portfolio. Ottawa, Canada: Educational Developers Caucus. Download here.

For more information about the EDC Guide Series, see here.

Helping: What it means in educational development


This table is taken from Schein, 2009 (p.7).

Educational development: “The profession dedicated to helping colleges and universities function effectively as teaching and learning communities” (from Felten, Kalish, Pingree, & Plank, 2007, p.93)

As an educational developer, helping is important.  Whether I am program planning, consulting, or facilitating, my ultimate aim is to help (to enhance teaching and learning in some way). Schein (2009) notes that there is helpful help and unhelpful help. I know I have done both.

In order to better understand what it means to help, I am reading Schein’s book  “Helping: How to offer, give and receive help”.  Below are some sense-making notes I have taken and quotes I find particularly useful from the first three chapters (future blog posts will explore the other chapters).

Schein begins by describing two cultural principles that are fundamental to understanding the helping relationship:

  • “….all communication between two parties is a reciprocal process that must be, or at least must seem to be, fair and equitable” (p.11)
  • “… all relationships in human cultures are to a large degree based on scripted roles that we learn to play early in life and which become so automatic that we are often not even conscious of them” (p.11-12)

He applies the notions of “social theatrics” and “social economics to describe communication within a cultural context.

According to Schein, “every helping relationship is in a state of imbalance” in the beginning (p.35). That imbalance exists largely because of the unequal power dynamics; the client (the term he uses for anyone seeking or being offered help) is “down” and the helper is “up”. “Being thrust into the role of help is immediately a gain in status and power…” (p.33),  Schein notes. The helping process is often impeded because the people involved fail to recognize the initial imbalance. Consequently, neither the helper or client initially knows what to expect and what to give the relationship.  So that our help may be helpful, we must address and deal with the imbalance.

Doing so, however, can be difficult because the helper and client may fall into traps.

Traps the helper may fall into Traps the client may fall into
  • dispensing wisdom prematurely
  • meeting defensiveness with more pressure
  • accepting the problem and over-reacting to the dependence
  • initial mistrust
  • relief
  • looking for attention, reassurance and/or validation instead of help

A successful helping relationship requires that the helper intervene in ways that “build up the client’s status” (p.47). By addressing the imbalance of power, the helping relationship may further develop and become productive.

Reference: Schein, E. (2009). Helping: How to offer, give and receive help.  San Francisco, CA: Berrett-Koehler Publishers.