Hello wonderful people,
This was a fun one for me!
I focused on an aspect of my work (exam planning / building / supporting), and drilled down into why I find it so meaningful — being able to create positive experiences for students.
To capture some of my ideas, I wrote a post on Medium.
Additionally, I also created some dummy exams on Canvas for you to get a better sense of what remote proctoring is like.
Simply sign in with your CWL to join the course. Just a heads up that this part of my assignment won’t be the most mobile-friendly, as you won’t be able to take these exams on your smartphone…
As always, please feel free to leave me any feedback or questions here!
Lyon
Academic Integrity enforcement is a tricky thing. There are the technical issues, recording audio and video for example. Neither can be done from within a browser without a plugin or external program. Browser security is such that webpages cannot request access to those resources. There are privacy and security issues, what is being stored, who has access to what is being stored, where is that data being stored, how do authorized personnel access what was stored? There are moral issues, should students be required to acquiesce to surveillance during testing, is remote proctoring truly different from in person exam invigilation, does cheating take away from the value of degrees of those who do not cheat? Finally, is remote proctoring the best way to discourage dishonesty, would better assessment design or methodologies serve better? There are many questions that make the topic have no easy answers. I can say that in the 90’s, I got an engineering degree where there was never anyone besides the instructor and maybe one or two TAs in the exam room with us and cheating was not a problem. I think the real question we need to be asking is why are students feeling the motivation to cheat and how do we tackle that problem?
Lyon;
A great topic! There was a lot of discussion about exams at my school in May, and a lot of the concerns that you brought up were raised. In the end, teachers were asked to design open book tests that asked higher order questions and had a fair but firm time limit. The hope was that students would need to have at least some basic stored subject knowledge in order to be able to complete the test in the given time as looking everything up would take too long for them. I also think they considered using zoom with mics muted in order to invigilate. In the end, we returned for the last few weeks of the term before summer and the students were asked to take the newly designed exams at school while socially distanced.
The biggest concern I would have about third party software would be the security concern. Where are the videos stored? Who has access to them? As well, your comments about increased student anxiety are very valid. I remember a story a high school friend told me about writing an exam in our gym. She was hot, and wanted to remove her sweater, but was so terrified that any large movement could be construed as her cheating and therefore she choreographed the extrication of herself from her sweater into a long series of small movements designed to allow her not to move her gaze from her exam paper, and not to draw anyone’s attention. She came to mind when you spoke of how the idea of being ‘watched’ might affect student performance.
I wonder if we are approaching assessment wrong, or missing an opportunity here. Perhaps the answer lies in finding a more holistic way to assess. If we moved more towards a portfolio style of assessment, where a student creates an artefact that demonstrates their knowledge and understanding, or as a classmate has already suggested, a group or open task that must be completed under timed conditions, we may be able to get around the academic honesty issue AND end up with assessment that is potentially digging more deeply.
Thanks for your project!
Suzzie
Hi Lyon,
Excellent choice for your A3 project. It’s very current and could have significant repercussions on tertiary education. It was smart to situate yourself in term of this choice of topic and explain the rationale behind it. Your take on proctoring exams was insightful and most welcome. Thinking about the students instead of just coming up with numerous and ingenious ways to detect or prevent cheating is a valid point. Students don’t perform well if you put them in a stressful environment. By considering how this might impact them and what could be done to alleviate test anxiety and unfamiliarity with this new technology is a good start.
The call for integrating proctoring solution within the LMS is justified. The students know how to navigate this system already and they will know what to expect. Why add another layer of difficulty? At the beginning of their studies, many universities require students to take an orientation course to understand their LMS. In that course, they could familiarize themselves with the online exam guidelines and policies. Asking a third party to get involved would lead to unnecessary complications and headaches for all parties. The cost is another issue. Why should students pay for their online exams? Do we ask students to pay for printing the exam papers? Why is this cost passed on to the students?
Although there are issues linked to this new technology, I believe we will have the means to add more credibility to online courses and degrees. In the past, online courses had a certain stigma attached to them because it was impossible to verify the student’s identity. Hopefully with better online proctoring products and guidelines, more institutions and establishments might look at online courses in a more favourable way.
Hi Lyon,
Do you know of any other alternatives to lockdown browser or proctorio?
I’m curious of ways to invigilate an open-book exam. Allow access to everything except for student-to-student communication.
Hi Adrian,
I’m sure there are alternatives out there, but I have only worked (extensively) with the two that you mentioned.
Usually, it’s a matter of “making it work” with one of these solutions when it comes to open-book exams. Doing open-book with LDB is a little tougher because you must specify which domains students will be allowed to visit (virtually impossible to define all of these, unless you’re doing a semi-open-book and only allowing certain websites). With Proctorio, web activity and screens can be recorded — video / audio can be disabled if desired…
Hi Lyon,
I really enjoyed your work. In terms of design, I really appreciated the simplicity of it. Not bogged down by too much information, easy to follow, and clear.
I’ve been designing online courses for many years and exam proctoring is always being discussed. I’ve used ProctorU, Proctorio, Examity and now Proctortrack coming up soon. There are shortcomings with every single one and when I saw your A3, I thought YES! This is so needed for our future!
I have experienced first-hand the chaos that ensues when a proctoring service fails to function correctly. It’s disastrous or everyone involved; the institution, teacher, help desk, designers, and most of all, students. Very often these exams are high stakes and students are riddled with anxiety. A smooth experience is what they need, not a clunky piece of software that ends up being frustrating and difficult to use.
I do agree with your four ideas on how to improve remote proctoring. The only things I would add is the ability of the proctoring service to be easy to use for the teacher. Very often it’s a lot of work for a teacher to sift through all the potential issues. Some proctoring software flag incredibly easily, leaving the teacher to go through multiple videos and images trying to figure out if cheating took place. The system isn’t easy to use and causes an incredible amount of work.
Great post!
Hi Nataie, thanks!
Great point on the chaos — especially when there is no way of reaching students all at once to provide them with information. Exams should test knowledge, not how good their computer is or how effectively they can troubleshoot issues that might come up.
Agreed on the ease of use for instructors as well — Proctorio, for example, can capture lots of student “behaviours” and flag them depending on how they compare to the rest of the exam-takers (student A used more keystrokes than other students). Sometimes, simplicity might be key…
Hi Lyon,
What a great topic! You bring up a very important point on user experience and I am sure with that awareness and your knowledge about the technology involved, you will be able to make changes for students.
Maintaining academic integrity is already a challenging task pre-COVID-19 and now with most, if not all, courses going remote, institutions are left to navigate resources in a short period of time. It is ironic that the use of technology can be a double edged sword. In this case when we talk about the two examples you gave, LockDown Browser and Proctorio, both seem to induce fear from students. That fear could be a result from unfamiliarity of the systems, like you mentioned; or the worry that their actions are recorded and can be interpreted against them. Also it is difficult to ensure technical support will be available to students and instructors in a timely manner.
Recently there was a huge discussion (at least on Reddit) on the use of Proctorio and the biggest concern students had was over security. Who can access the recorded sessions, how long are each recording kept, and most importantly, whether those video recording will be used for other purposes (data mining to sell to other companies). There is so much for to consider but I like your suggestion that major LMS should have similar systems embedded. Thanks for sharing this project with us!
Thanks Pei! I definitely think that learning management systems should offer more functionality when it comes to academic integrity and honesty.
The Proctorio incident has been on my mind for a while — it clearly illustrated student dissatisfaction with the solution, as well as big mistakes made by the CEO and the company. Their extension on Google Chrome has a 1 star rating, and I think a lot of the issues stem from students not knowing what to expect. Being able / not able to re-enter an exam (after disconnecting from internet, for example) is actually a setting! Guidelines need to be provided so students aren’t left wondering…
Hi Lyon,
I like how enthusiastic you are about exams! especially how you focus so highly on the student experience.
The integrity of exams are a very interesting challenge at home – I liked the examples you set up. Overall you roundly addressed traditional assessments, the issues and some solutions.
This diverges a bit … but, my stance on this time period is that it is the best opportunity to revolutionize traditional assessments. So much of closed book exams assess the lower rungs of Bloom’s Taxonomy, like ‘remembering’. Open book exams, or group exams would be easier to integrity and more closely resemble the real world and can more deeply assess the students.
You and Tyler bring up a great point — there is no better time than now perhaps to rethink how we assess students. Presentations are hot right now, and so is case work. I guess it’s up to us to figure these things out!
What an interesting topic. And one I have no experience with. Myself I’m working on moving away from exams in the traditional sense while teaching in person and for my purposes, an online exam just wouldn’t serve a purpose – but that’s only my narrow perspective. So it’s great to see a whole other world here in your presentation!
I really liked this work around you offered: “Perhaps students should be recorded in brief snippets and only at randomized intervals, rather than constantly throughout the entire exam. Still images captured from webcams and screens can also be used to supplement and verify the information gathered.”
And I liked your closing line a lot too: “After all, the end user is not the institution they signed a contract and agreement with — it is the students themselves…” Anything putting the student first is greta by me!