As a result of our pilot test, we noted following issues that needed to be changed in our experiment protocol:
- We should counterbalance for which type of annotation the participant is asked to find first, having some find the low visibility annotations first and others find the other high visibility annotations first. We found that there was a significant learning effect depending on the order that the participant was asked to do these tasks. This doubles the number of conditions from four to eight.
- We decided to have a printout of the annotations that the participants need to find to show the participant. This clarifies the task for the participant.
- We wanted to have a copy of the interfaces with a dummy video tutorial that the participant can use to familiarize themselves with both interfaces, before starting the experiment.
- We decided to have a time limit of 15 minutes to prevent the study from running too long if the participant gets stuck on a tutorial.
- Some of the wording on questionnaire needed to be changed. Specifically, we were asking users for their “comments/opinions” which was confusing, since we often used the term “comments” in our experiment with a different meaning.