Looking for a way to assess high-level learning (such as application, analysis, and evaluation) with a question that is quick and easy to grade, even by graders unfamiliar with the topic area? If so, read on!
The issue with traditional open-ended problems
While question formats like multiple choice have a role, sometimes you need to have students demonstrate that they can perform detailed calculations, analyze a scenario, or evaluate different options. Giving a more open-ended question on an assignment or exam often works well in terms of having students demonstrate these competencies, but they can be extremely difficult to mark, requiring a high-degree of judgement any time the student solution deviates from the marking key solution.
Advantages of this novel “spot the errors” format
Instead of providing students a blank page to solve a problem, consider providing a sample solution (with errors) as part of the question. The student’s task is then to identify the errors and describe what should have be done instead. This can be elevated further by requiring students to identify the most significant errors they identify and justify those choices. This tends to focus student tasks on higher-level thinking (i.e., judgement, evaluation, etc.), and it simplifies marking as there isn’t a need to try to follow a student’s thought process through an entire solution.
Other advantages of this format include the following:
- Developing students’ ability to quickly review and assess others’ work is a very important skill
- Grading time is significantly reduced (more TA time can be used for higher-value interactions with students)
- Judgement required by graders is significantly reduced (there is more consistency and fairness in grading)
- Writing time for students is reduced (more time is spent thinking instead of writing)
- Mistakes do not propagate (in a traditional problem, an early mistake could mean a student has to redo much of the solution; think of times you have seen a student erase a whole page of work during an exam – that adds stress!)
- Questions normally requiring drawing, equations, or calculations can (usually) be answered using simple text, which works well online
A sample traditional problem is provided below along with the same basic question in this “spot the errors” version suitable for use online assuming the use of essay-style textboxes.
Traditional 'Pen and Paper' Problem
A traditional "pen and paper" problem might look like what is shown below. The remainder of the page would be left blank for students to write their solutions.
It wouldn't be too much of a stretch to imagine a solution from at least some students looking something like this:
It is not well organized, a previous answer was erased (but not fully) making it more difficult to follow, and there are missing steps and elements in the solution. The final answer is off by many orders of magnitude, but there are elements that are correct. Apart from just identifying the errors that were made, the grader really should rework the solution with the errors propagated through in order to be able to assign part marks for the rest of the work. This requires time and mental energy from the grader. They need to try to figure out the student's thought process and degree of understanding, and they need to exercise judgement in assigning part marks.
What's more, consider how it would be possible to ask this type of question on an online exam and ask students to write out equations and demonstrate their ability to work through a solution like this step-by-step. Most likely, students would need to work with pencil and paper and then scan and upload their answer. While certainly doable, it does raise concerns about equity (do all students have access to the technology for scanning, and for doing so at a high quality?) as well as integrity (if students are scanning and uploading their solutions, who else might they share those with? how do you know they are not using the technology inappropriately while answering the question?).
Novel 'Spot the Errors' Problem
This novel "Spot the Errors" format for the same question might look like as shown below. At first this looks like a much longer question, but it is really just three parts: the task for the students at the top, the problem that forms the basis for the scenario, and the mock student solution they are reviewing.
Under normal circumstances, in in-class exams, space could be left on the right of the page and students could write their answers in pencil there. For an online exam, as suggested in the question instructions here, students could write their answer in a textbox on their computer. The sample solution could look like this:
| Line 2: RE calculated using diameter in cm rather than radius in m Line 3: sphere is polished; 0.0125 roughness ratio is unrealistic Line 5: surface area used instead of frontal area Line 6: the fluid pushes the sphere to the right |
To answer this problem, the student needs to have a high-level understanding of the topic; they need to know how to answer the question themselves to be able to identify when someone has done it incorrectly. To a novice viewing the sample solution at a superficial level, everything would likely look familiar and appear plausible.
In contrast to the traditional problem, there is little judgement required in grading this as graders can be given a list of all errors and the associated mark weight. For example, perhaps the grades for each of the four errors could be 1, 2, 1, and 2, respectively, recognizing that the error on Line 2 is a simple oversight but the errors on Lines 3 and 6 represent a more conceptual misunderstanding. You could look at part marks for partially complete/correct variations of the full answers, and it is comparatively easy to remark papers if a new error you had not anticipated is identified.
Tips for using this format
If using this format, some ideas to consider:
- give students multiple opportunities to engage in developing these same types of skills in the course (such as by converting some tutorials to this format, providing past exams in this format for practice, or having students regularly exchange work to informally peer grade)
- this works on many question types, not just calculation; I've used it on design problems, problems where students apply a tool or process, engineering drawings, coding, technical communication, and more
- consider basing the errors in the mock solution on common errors you see students make,
- for calculation-style questions, make sure it is clear students should not propagate corrections from earlier steps (i.e., students should focus on each line of the solution independently, and assume all prior work is correct when considering that line),
- be explicit whether or not you want students to recheck any calculations, look up values, etc. (I trust students can do this and usually do not want students using exam time to show this),
- consider indicating how many errors are in the mock solution or how many errors students should identify,
- consider awarding marks based on how significant the error is (i.e., trivial errors carry less mark weight than major conceptual errors), and
- consider that very few (if any) students will spot all errors; I usually give 100% for students who correctly identify a significant subset of all errors.
An alternative to this format to further emphasize high-level judgement and evaluation—and to further streamline marking—is to ask student to identify the one, two, or three most significant errors. You would need to clarify what you mean by “most significant” but this could include the errors that most significantly impact the final conclusions, or that most significantly demonstrate a lack of understanding of the concepts, etc. In this way, students first need to identify all the errors they can, and then second evaluate and rank those errors by significance to identify the most significant one(s). In this case, part of the mark is based on how significant the error they identify is.
Lastly, there can be some student grumbling when you use this approach. Typically, this has to do with two things:
- the familiar marking system where most students who at least wrote something could expect some part marks as been removed; for a student struggling with the concepts, everything in the mock scenario in this format could look plausible.
- it can be difficult for some students to judge whether they are on the right track with this style question. With a typical calculation-type question, not having all of the parameter values to solve a particular equation or getting a result that was off by many orders of magnitude were cues that there was something wrong in what they were doing, directing them to look back at their work. There isn't the same kind of real-time feedback here.
These issues are worth acknowledging as part of a describing why you are using this type of question, including the various benefits for them mentioned above. Importantly, remind students you are intentionally trying to assess their high-level thinking, and that the skills you are assessing are valuable for any professional.