Question pools

Many LMS quiz tools and other online question platforms will allow you to create pools of question to facilitate randomization between students.  Also called “question sets,” by organizing questions this way, you can give each student a random selection of questions, and usually present questions in a random order.

An Example Question Pool Structure

Imagine these three sets of question pools, in this case phrased as multiple choice but with only the question stems shown.

Notes and possible explanations for why these pools might be set up using the above indicated number of assigned questions include the following:

• Pool A (2 of the 3 questions assigned): Each student will be missing one of the three questions at random.  This creates a modest benefit in terms of integrity in that there is a 33% chance that any two students will see the same two questions, and a 17% chance that any two students will see the same two questions in the same order.  Notice that the three questions are independent from each other (they focus on different aspects of Bloom’s Taxonomy) and therefore they are suitable to be answered in any order, such as is the case with most platforms that will assign questions within a pool in random order.
• Pool B (2 of the 2 questions assigned): In this case, all students answer both questions and only the order might vary; this is sometimes done to make it more difficult for students to inappropriately collaborate during an exam, especially if students are not allow to revisit previous questions.  Here there is a 100% chance that any two students will see the same two questions, and a 50% chance any two students will see the same two questions in the same order.
• Pool C (1 of the 2 questions assigned): Here there is a 50% chance any two students will see the same question.  Since these two questions test basically the same concept using different wording, in most cases you would likely want to avoid asking both of these questions of a single student (answering one question essentially answers the other, and one question may even give away the answer to the other).

How Many Questions to Include?  How Many to Ask?

There are no definitive rules on how to use question pools, but some factors to consider:

• how much you are relying on the question pools to act as a barrier to academic misconduct (recall, reducing opportunities to cheat is only one approach to enhancing academic integrity – see the Academic Integrity pages for details.)
• the number of students writing the examination or quiz (the larger the number of students, the larger the number of questions in each pool in general)
• the number of different times each exam or quiz will be offered (if you have all students writing at the same time, there is less of a concern of information flow and question pools can be smaller)
• the time pressure on students during each exam or quiz (the less spare time students have during the exam, the less chance of duplicating materials; however, use this approach with caution since some students read/answer more slowly, and here you are increasing pressure on students)
• the grade weight associated with the exam or quiz (the higher the grades associated with the questions, the more significant the pressure towards misconduct will be)
• how many times you plan to reused the same questions for different assessments (do you plan to use the same questions again in a follow-up quiz or in a subsequent year?)

To give one point of reference, for low-stakes quizzes as part of the introduction to engineering courses at UBC, we typically assign each student 6-8 questions out of 20-24 questions in total organized in 4-5 pools.  This gives a ratio of roughly 3:1 in terms of total questions available to the number of questions asked of each student.  The fact that these are low-stakes quizzes (<0.5% of the course grade per quiz) suggests a lower ratio is sufficient, but the class size of about 1000 students per year and the fact that questions are largely re-used from year to year suggests a higher ratio.

This table below gives some indication of the probability (rounded) that any two students at random will get the same combination of questions.  The values in parentheses indicate probability of the same questions appearing in the same order (for when order is also randomly determined by the online platform).  The top row of data in the table can equally be interpreted as the probability that any two students at random will get the same question independent of pool size and order.

Other considerations

When asking questions in pools, it is important to try to ensure all questions in a given pool are similar in level (see Bloom’s Taxonomy), difficulty, and length.  Still, despite our efforts, it may be that some questions are in fact easier or more difficult than others.  Normally this is not an issue in terms of fairness as all students are given the same questions in a typical exam, but with question pools this might no longer be the case.  Some students might be randomly assigned a more difficult combination of questions and others an easier combination.  Consider reviewing individual question averages and applying a correction to overall exam grades as appropriate remove effects of question difficulty.