Qualtrics is UBC’s survey platform. I’m mostly happy to just use anonymous links in it, and when I’m not, I can often distribute targeted links via e-mail addresses.

However, as instructors, we do not have access to a list of our students’ e-mail addresses in our classes. So, how do we send a survey to students in a course where we want to know who responded, with some confidence?

It turns out we can have students log in to Qualtrics via CWL.

They key is to create an authenticator. I figured this out with help from the documentation and from published tips from Firas Moosvi. (Thanks, Firas!)

The key is to open the “survey flow”:

Then add an “Authenticator” block. I’m not sure whether it matters where you put the block, but since it seems to behave like a conditional, I placed my authentication-guarded survey questions in the block nested beneath the authenticator. Finally, configure the authenticator to use “SSO” and “Shibboleth” and likely to “Capture respondent identifying info”:

Be sure to click “Apply” on this screen and then go to the main survey screen (the clipboard icon above) and click “Publish” or you will not see the updates.

Above, I am capturing given name and surname (but not preferred name, which is unavailable) as well as email. I am also capturing a unique identifier exported by UBC, but which cannot be mapped back to CWL.

Note that what I wrote above does not capture CWL login or student number. According to this document listing attributes you can use, it is possible to capture these via shibboleth. Unfortunately, according to Warren Code:

From Joss Ives, Qualtrics expert in PHAS: The only fields I have been able to capture are “lastName”, “sn” (which is the same as lastName) “givenName”, “email” (email) and “ubcEduPersistentID” (which does not seem to correspond to any identifier on any other system) So it’s enough to uniquely identify somebody in a smaller class

LT Hub confirmed roughly the same set of fields.

So, that’s what we have to work with! I suspect the email is whichever one the student registers with the student registration system as their preferred email, as opposed to (for example) the CWL@student.ubc.ca e-mail address.

The UBC Faculty of Science Equity and Inclusion Scholars Reading Group is reading the paper Impact of equity models and statistical measures on interpretations of educational reform by Idaykis Rodriguez, Eric Brewe, Vashti Sawtelle, and Laird H. Kramer. The paper argues that educational research aimed at increasing equity should approach two factors more carefully: what “equity” means and how to report results so they are most usable. For the former, the paper presents three different models of equity. For the latter, the paper encourages reporting effect sizes with confidence intervals and not just statistical significance, both to make results more meaningful and to facilitate meta-analyses. They then reinterpret the results of Reducing the gender gap in the physics classroom by Mercedes Lorenzo, Catherine H. Crouch, and Eric Mazur under different models of equity with careful evaluation of effect sizes.

The paper’s discussion of statistical reporting and its re-analysis of the Lorenzo et al. results are both interesting, but they’re not what caught my attention. I read another paper recently—Student Perceptions of Fairness and Security in Versioned Programming Exams by Chinedu Emeka and Craig Zilles—where I found the paper results interesting but really got caught up in the models presented in related work. In that paper, it was different models of justice in grading practices by which students might judge a course.

In this paper, I got nerd-sniped by the models of equity. What is equity, and what do we mean by it?

I had a vague idea in my head from previous papers that matches the equity vs. equality graphics out there, although I hadn’t actually seen them before I discussed the paper with my partner and daughter and they both said it was “like the fence diagram”.

Two of this paper’s models generally match my understanding of the most common contrast drawn in those pictures. The third is an interesting alternative (or maybe two), and not really the same as the “liberation” or “justice” models presented in some of those fence diagrams.

Below, I’ll discuss the notions of equity the paper laid out, using completely fake data on pre-test and post-test results in some educational intervention for Indigenous and White students. In my examples, the Indigenous students enter with lower average scores because of the previous impacts of racism. If that already feels racist (assuming Indigenous students are worse than White students, insisting on comparing Indigenous students to White students), then keep reading to the third definition of equity.

Equity of Parity

First is equity of parity:

… parity is achieved when the distributions of achievement scores in two or more groups have the same postinstruction average despite differences in preinstruction averages. … one must acknowledge that the instruction benefits the ‘‘less prepared’’ students more than the ‘‘well prepared’’ students.

Loosely speaking, this matches my pre-existing definition of equity. We allocate resources to different “types” of students so that everyone achieves similar learning outcomes. We “close the achievement gap” as both the Rodriguez et al. paper puts it and many other resources online.

The paper does point out that this may not quite be what’s intended in those graphics, however. Equity of parity in outcomes in a single course is an episodic sort of equity. If successful, it closes the achievement gap in this one course in its learning outcomes, leaving untouched the inequities that led to the gap in the first place and the outcomes of other metrics outside the reform’s specific focus.

Equity of Fairness

Next is equity of fairness:

… defining equity as that which promotes justice and freedom from bias or favoritism [2]. … The equity of fairness perspective implies an impartial treatment that would be reflected in an equal gain (posttest score – pretest score) in conceptual understanding for all students groups, regardless of initial understanding or group.

The paper’s definition of equity of fairness loosely matches my pre-existing definition of equality. This approach is aiming for a “colour-blind” outcome. Everyone is treated equally in some measurable sense. There’s more to say to even truly nail down this definition. Should we say the “same gains” are achieved if the mean difference in score is the same? The mean percentage increase in score? The “normalized learning gain”?

It feels clear looking at this that it is a disappointing definition of equity, maybe even a “straw” argument. However, on a second read, I realized this actually is another notion of equity and not quite the notion I previously had for equality. That’s because despite differences in their circumstances coming in, both groups had the same gains. That may well mean that more resources are allocated to the group that had been disadvantaged coming in.

A more strict notion of “equality” might mean that resources are allocated to students entirely independent of which group they belong to, or perhaps independent of any other factors. Measuring the success of this equality approach doesn’t involve pre-test or post-test results at all; it involves measuring the distribution of resources without reference to outcomes.

Despite that, this model is problematic at best in that it perpetuates the initial inequity between groups.

Equity of Individuality

Last is equity of individuality:

…relying on a comparison group perpetuates the idea that marginalized students are not worth studying in their own right and therefore sustains upper-middle class whites as the normative group [5,6].

The equity of individuality model promotes individual excellence of students within groups.

It took me three readings and a quick read of A “Gap-Gazing” Fetish in Mathematics Education? Problematizing Research on the Achievement Gap by Rochelle Gutiérrez to feel like I understood what was going on here. I see this as a lens on equity that differs in two important ways from the ones above.

First, the focus is solely on the specific group for whom we seek justice. In my imaginary example, we have identified that Indigenous students have been and are subject to injustice. In both previous definitions of equity, the response is to place them in opposition to a “norm” of White students and to try to make the Indigenous students do as well as the White ones. Gutiérrez points out that that perspective bakes in marginalization and injustice. Why shouldn’t we instead focus on the circumstances, needs, and desires of the Indigenous students exclusively? Why are Indigenous students themselves not a group worthy of study?

Second, the focus is on individual achievement. Returning to Gutiérrez again, they point out that the distribution of achievement within each of these groups is going to dwarf the difference between groups. So, why not focus on trying to help individuals achieve excellent outcomes?

Entangled in this as well is a greater focus on context and explanation rather than simply success or failure. My knee-jerk reaction is to shout “qualitative methods” to the rooftops when I see that, but Gutiérrez again points out that while qualitative methods may well be important to this lens, the distinction is not really about qualitative vs. quantitative but about the two factors above: a focus on the specific group and a focus within that group on individual excellence.

As a lens, this model focuses attention on my choice in my fake data to compare (imaginary) Indigenous students to (imaginary) White students and my real choice in manufacturing the data to place the Indigenous students’ pre-test scores below the White students’ scores. Gutiérrez’s paper emphasizes how the achievement gap model of equity plays into this problematic narrative of deficiency. Why not invent this world:

In a sense, this model dodges the question of allocation of resources within the course across groups. The focus is on one group and how we allocate resources within that group, not on a comparison across groups.

On the other hand, instructors still will be allocating resources to groups, perhaps unequally or inequitably, and this model doesn’t really let us judge that. Furthermore, this model brings up a meta-question about how society should allocate resources to educational reform across groups if the educational reform’s focus is entirely within-group.

Which is Right? Which is Missing?

Yeah. I’m gonna cop out and tell you there is no Right Model here. I certainly prefer the parity and individuality models over the fairness one. However, the fairness model is actually a great tool for understanding the distinction between equity and equality. It emphasizes that even a “do no (more) harm” approach cannot necessarily blithely allocate the same resources to every student.

I’m also not going to brilliantly identify the gaps in these models. I will say that, by the nature of the Rodriguez et al. paper, they are all pretty narrowly focused on one course and one (set of) metrics. More context and a thicker understanding might be important to your model of equity. (That is where the “liberation” and “justice” pictures in the fence diagrams come in.) That seems especially important since the factors that lead to injustice often involve weaving discriminatory factors into unstated assumptions. (All my students come in knowing X. Everyone’s family includes Y. The non-technical word Z means the same thing to all my students.)

Why Not Equity?

I’ve approached equity as something obviously-to-be-desired. There are certainly critiques of equity as a goal, both recent and longstanding, aimed at avoiding equity becoming a drag on the success or excellence of individuals or groups.

I’m not convinced by the urgency of these critiques in my own current cultural and practical context. From my perspective, it seems clear that work is needed to overcome the impact of both historical and ongoing discrimination across many dimensions. Furthermore, we should be striving to create educational interventions that benefit students as a whole. So the nightmare scenario in the imaginary results graph above really is a failure of an educational intervention, regardless of whether our definition of equity might admit it as a “success”.

That said, these critiques highlight the danger of blindly applying a definition of equity and also the added risks to consider in a situation that is “zero-sum” such as selecting a fixed number of “winners” from a pool of candidates.

Skimpy Little Conclusion

I will come out of this paper trying to think more carefully about how my instructional design choices impact individual students with different personal circumstances and groups of students who face structural barriers to success.

NOTE: This is a draft; critical feedback, comments, and questions are very much welcome! (Leave a reply or e-mail wolf@cs.ubc.ca.)

If you’re in academia and teaching is an important part of your job, then you’ll probably need a teaching portfolio for promotion or tenure at some point. Unfortunately, despite the fact that you have dozens/hundreds/thousands of students, teaching is often a solitary endeavor, especially compared to research. How, then, do you compellingly document your teaching experiences and innovations to others?

Here’s my advice, much of it inspired by my long-ago participation in the Engineering Teaching Portfolio Program offered by U. Washington’s Center for the Advancement of Engineering Education.

I wrote this advice as I was drafting my tenure packet—CV and portfolio—and promotion packet. The tenure packet from Sep 2008 was for promotion from “Instructor I” to “Senior Instructor” at UBC and, of course, for tenure. The promotion packet from Sep 2013 was for promotion from “Senior Instructor” to “Professor of Teaching”, a somewhat unexpected process since the rank did not exist when I began at UBC. For context, check out UBC’s tenure and promotion policies. The guidelines for promotion to Professor of Teaching have the most detail, although they were published well after my tenure case.

Prelude

Both my tenure and promotion packets are bulky but designed to be skimmed. I aimed for a “multi-resolution” packet that offers busy readers a short, sweet summary they can read, skipping the rest, but with hooks available to dive deeper, all the way down to evidence (artifacts of teaching, evaluation records, research citations, etc.) supporting the claims I make in my portfolio.

One minor caveat for non-Computer Scientists: Conference publications in CS are at least as prestigious as journal publications (and often more so). More broadly and importantly, you’ll want to adjust disciplinary or cultural issues like this for your own context in both my packets and my advice!

Finally, be aware that the packets are written to make me sound like I walk on water. It’s all true mind you, but spun to the max. Don’t get intimidated; you can sound like you walk on water, too!

Beforehand

Here are a few steps you can take from the start of your teaching career to prepare yourself for your tenure or promotion case.

1. Be a pack rat. Save everything and file it away. Meeting agendas, notes from students and colleagues regarding your teaching, links to interesting student artifacts, etc.

2. Write, write, write. Write a “post mortem” after each class describing what worked well and poorly, write analyses of your exams (how and why you planned them and how they turned out), write any commentary on your own teaching that you can motivate yourself to put down.

Doing it all the time will help you tremendously as a teacher. Doing it at least sometimes and filing it away will be invaluable for your portfolio.

3. At least once a year—merit review, anyone?—write out a one paragraph story of each major teaching-related effort you make, like course design, new pedagogical approaches, student research supervision, and so on.

My mentor Paul Carter suggested this to me and suggested my CV as the place to keep it. Using this approach, your CV recaps points in your promotion portfolio… but it also makes the CV stronger and forces you to write bits and pieces of your portfolio every time you update the CV. At my promotion, I received strong advice to brutally trim this in the submitted version, but the same text migrated over to the rest of my packet!

Writing the Portfolio

The writing process itself can take a while on the calendar and a fair number of hours. Get started early, and try out these steps. I started a year or more in advance with the first step below.

1. Review some other people’s portfolios. When I say “review”, I mean review like you would a conference or journal submission. Question how and whether they’ve accomplished their main points and mark the document up with recommendations. If the author is still working on their portfolio, this is peer feedback, but it’s worthwhile regardless so that you can discover what makes an effective portfolio.

2. Write three crappy teaching philosophy statements, with three totally different philosophies. Don’t worry; they won’t go into your portfolio. Their purpose is to encourage you to think creatively about what you’ve accomplished as a teacher and why you did it.

3. Make a list of artifacts you can use as evidence of your successful teaching:

• Your favorite stuff that you remember.
• Anything that catches your eye as you peruse your pack-rattish hoardings.
• Things that click with the crappy philosophies you’ve written.
• At least one example of every type of thing (lecture, assignment, interactive exercise, demonstration, exam, etc.) you’ve ever put together for a course.
• Examples of cases where you did not do that well, especially where you have the evidence you used to discover the problem and steps you took to resolve it.

4. Throw your crappy teaching philosophies away.

5. Make an “affinity diagram” of your artifacts. Put a brief description of each artifact on a post-it note and then stick the post-its on a whiteboard arranged into related groups. Look for ideas about your teaching in those groups. Is there a cluster of artifacts that are all about bringing your research ideas into the classroom? A cluster about encouraging students to take responsibility for their own learning? A cluster about current cultural references? Whatever.

Give each cluster a title—grab a whiteboard marker, circle the post-its, and write a name next to the circle—and take a picture of the diagram you created.

Now, repeat this process a couple more times but intentionally make very different clusters. Hopefully those crappy teaching philosophies you wrote have keyed you to be able to view your teaching from very distinct perspectives.

Here’s a summary I drew while creating my tenure packet of some of the groups in my affinity diagrams:

Circled nodes are cluster titles; uncircled nodes represent artifacts.

6. The cluster titles you created represent new themes for a new teaching portfolio developed bottom-up from—and supported by—the evidence you’ve collected to build your portfolio. Plop down the cluster titles as section headers in a text editor with brief notes of its artifacts below each one.

My teaching portfolio for tenure grew out of the clusters in the image above. Compare those circled nodes to these section headers from the portfolio: “Motivated Learners”, “Active, Reflective, Social Learners”, “Reflective Teaching”, and “Practical, Respectful, Efficient Teaching”.

7. Flesh out, paring back to the artifacts that really make your point. The best artifacts are the ones that look good in themselves and also support multiple clusters effectively, so you don’t need quite as many to tell your story.

8. Now, write a good teaching philosophy to pull it all together into a coherent story.

When you’re done, you’ll have a coherent story that describes the most important elements of your teaching with strong evidence (your artifacts) backing up each point in the story. The artifacts themselves can be appendices; your audience doesn’t need to read these because your philosophy statement will reference them in context whenever they’re needed.

Letter Writers

You may also need to recommend letter-writers for your case. For my tenure case, I proposed six letter-writers, a minimum number of whom (perhaps 2?) were guaranteed to be asked as letter-writers in my case. My department independently chose the remaining letter-writers. (If you submit a list, know whether their choice is independent of or exclusive with your list! If their list must not overlap yours, then you may want to avoid putting all your best prospects on your list!)

This process was challenging, perhaps more so for assessing teaching than research. After all the main products of research are intentionally very public. In the end, I made a long list of candidates who I could be reasonably confident were familiar with some aspect of my work, using the sections of my teaching portfolio to guide my brainstorming of names. Then, on advice from my mentors, I added Canadians in prominent undergraduate management or advising positions and filtered out almost everyone who was: (1) not tenured/promoted themselves yet, (2) too well-known to me (e.g., had co-taught with me, co-organized a major event, or co-published something more significant than a panel), or (3) did not have a PhD.

Finishing Up

Now, get and give lots of feedback until you’re ready to send it off. Look for feedback from mentors, peers, colleagues, and anyone involved in the tenure and promotion process to whom you have access!

There’s one more point that’s easy to miss; don’t miss it. Creating your portfolio this way takes a great deal of time and work. One benefit of the work (hopefully!) is tenure and promotion.

But you should also benefit by gaining insight into what matters to you in your teaching and what you want to do next. Read over this document with fresh eyes thinking about where you want to go in your teaching now that you’ve sent your case off. After all, you’re getting tenure or, to be pessimistic, you’re about to be denied it. In either case, now’s the time to try something risky and innovative that takes your career to the next level!

Links

Here are links to the materials I submitted for tenure and promotion:

• Tenure packet (submitted as separate documents):
  • CV
  • Teaching Portfolio
  • Anonymized proposed letter-writers
• Professor of Teaching promotion packet (submitted as a web-based portfolio)

Here are links to a few other materials that helped me as I made my packets or that I’ve found interesting since then.

• Jodi Tims’ SIGCSE slides on promotion and tenure from the 2016 New Educators Workshop
• Related articles from the fantastic Tomorrow’s Professor series
  • My favorite: Answers to Common Questions About the Teaching Portfolio; includes a list of questions to ask yourself:
    • Is every claim made in the narrative supported by hard evidence in the appendices
    • Is the portfolio sufficiently reflective? Does is include a balance of items from oneself, from others, and from student learning?
    • Does the portfolio clearly identify what you teach, how you teach it, and why you teach it as you do?
    • Is a complete table of contents and appendices included?
    • Does the portfolio contain reflective observations?
    • Have efforts at growth and improvement been cited?
    • Are numerical student rating results included for several courses over several years?
    • Have any department or institutional factors influenced your teaching effectiveness
    • Would including some charts, tables, or graphs enhance the portfolio?
  • Others, vaguely ordered by relevance:
    • Writing A Teaching Philosophy Statement
    • Building Your Teaching Portfolio
    • Items for Inclusion in a Teaching Portfolio
    • Making Teaching and Learning Visible
    • Making Visible the Intellectual Work in Teaching
    • Promotion, Tenure, and the Engaged Scholar
    • Personal Philosophies of Teaching: A False Promise?