a. Pilot Test
In our pilot study we found that the participant did not really understand the top suggestions section of the event details screen for our interface. They did not use the top suggestions section when trying to find consensus. To fix this in our experiment we will explain more clearly how our prototype works and specifically clarify what the top suggestion section is intended for.
Also it was not clear when the participant started the task and the experimenter timing the task did not know when to start or stop the stopwatch. To fix this, in the actual experiment we will say “start” to the participant to indicate when to start the stopwatch. The participant will say done when they have an answer to notify the timer when to stop the stopwatch.
Another problem we found was trying to film the participant’s interaction with our mobile device. It was difficult to find an angle to film where the participant was not blocking the view for our camera. So learning from that, we decided to place the phone in a specific location marked on the table where the phone is visible for the camera.
b. Experiment Abstract
We conducted our experiment on 10 participants in the room X339. Each participant was read a script informing them they will be videotaped and observed during the experiment.
Participants were given a chat long, which varied in length, and were required to interact with the interface and determine what consensus was reached. Their interactions with each interface were timed and they were asked to note what they determined was the correct consensus.
After completing the interactions with the interfaces, users were then asked to fill out a questionnaire which gathered information.
There was no significant difference found between our prototype and Facebook messenger. 2 participants scored 75% on the accuracy test and 8 scored 100%.
c. Revised Supplementary Experiment Materials
We removed the pre-experiment questionnaire after our feedback session as it led us to decide against using participants with event planning experience. We also changed the post-experiment interview into a post-experiment questionnaire.