Category Archives: Update 6

We added a script so that our studies could be conducted in a consistent manner.

Script

Minor changes were made to the consent form to fit the context of our study, such as adjusting the time the experiment should take and what the participant should expect to do.

MedFiconsentform

We made separate questionnaires for each one of our interfaces and also a post study one. Questions were also reworded for clarity.

GulfOEQuestionnaire

We compiled a list of all the comments on both videos for the participants to use during the search for the comments.

ListofComments

For our study, we compared two interfaces, System Red (Youtube) and System Blue (our interface), to observe how users completed tutorial videos and searched for comments or annotations in each setting. We recruited 8 participants for our study and had each person complete a tutorial video on both interfaces, as well as follow-up questionnaire questions. We wanted to determine which interface helped users finish an entire task, and find comments or annotations faster.

After running our study we found that over half of the participants failed to complete at least one of two tutorials. Our results can’t support that finding annotations is faster on System Blue compared to System Red, but there is a trend in the data to System Blue being faster. Despite this, our qualitative data results showed that the majority of participants prefer to System Blue to watch educational videos (6/8 participants).

As a result of our pilot test, we noted following issues that needed to be changed in our experiment protocol:

• We should counterbalance for which type of annotation the participant is asked to find first, having some find the low visibility annotations first and others find the other high visibility annotations first. We found that there was a significant learning effect depending on the order that the participant was asked to do these tasks. This doubles the number of conditions from four to eight.
• We decided to have a printout of the annotations that the participants need to find to show the participant. This clarifies the task for the participant.
• We wanted to have a copy of the interfaces with a dummy video tutorial that the participant can use to familiarize themselves with both interfaces, before starting the experiment.
• We decided to have a time limit of 15 minutes to prevent the study from running too long if the participant gets stuck on a tutorial.
• Some of the wording on questionnaire needed to be changed. Specifically, we were asking users for their “comments/opinions” which was confusing, since we often used the term “comments” in our experiment with a different meaning.