a. Revised Goals of Experiment

We are making changes to our first two experiment goals which proposed that we compare two potential design alternatives. Instead, we have built a prototype based on the more promising of the two alternatives and intent to compare our prototype to an existing interface, which in this case will be Facebook Messenger. The goal of the experiment is to determine if our prototype affords faster consensus building in comparison to Facebook Messenger.

b. Experiment Method

Participants:
We anticipate recruiting 10 volunteers who are university students, with varying degrees of event planning experience and some experience with Facebook Messenger. Participants will be recruited from our network of friends as most of them are university students that meet our requirements.

The nature of consensus building involves reading group messages and determining the outcome of the discussion. Therefore it makes sense that participants should be relatively similar in their English proficiency. We don’t have a test to determine how proficient our participants are, but instead we have set requirements in place that will allow us to screen out ESL students. This will ensure all participants are proficient in the English language even if it is not their first language.

Requirement for participants (verbal confirmation while recruiting):

1. Must be a native speaker of the English language, or have completed a 1st year university level English course.
2. Must have completed at least a grade 12 or equivalent English course.

 

Conditions:

We are looking to compare the time it takes participants to identify the general consensus made by a group within our prototype and Facebook messenger.

Our prototype (please refer to Figure 1 from Task Details) offers a chat interface for users to provide their input and to endorse other people’s inputs. The top suggestions from the chat interface show up as highlighted details on the event page, allowing users to get a sense of the top trending ideas from the chat screen.

The other interface – Facebook Messenger (refer to Figure 2 from Task Details) – consists of a standard chat interface which allows multiple users to communicate with each other in a single window. There is a minimum of 2 participants required for a chat to occur, and there is a chronological ordering of the messages. Participants cannot filter the chat in any way but are able to scroll up to see the history of messages even if they join the group late.

Tasks:

The experimental task would consist of identifying the group consensus reached in two different interfaces (refer to the Figures in Task Details). The participant will be required to use our prototype or Facebook Messenger (alternating for each participant) with a mock discussion. They will be asked to identify the group consensus for an event based on their interactions with each interface. We will record both the time it takes for them to figure out the group consensus, and the accuracy of their answers.

Design:

The experiment would be a 2×2 design with the following factors and levels:

Interfaces: Our prototype, Facebook chat
User experience with event planning: Rare, Frequent

Interfaces would be within-subjects design and user experience would be between-subjects design. The dependent variable would be the time taken by participants to identify general consensus made out of the group settings and also, the accuracy of their answers.

Procedure:

Upon Arrival:
After a brief introduction, we will inform the participant that they will be given the scenario for the experiment and 2 pre-filled chat interfaces. The first interface will be our prototype and the second will be a Facebook Messenger chat log. The scenario will describe the situation surrounding the chat logs. The participant’s goal will be to identify the consensus reached for the TBD (To Be Determined) time and place details. In addition, they can stop the experiment at any time if they feel uncomfortable. This introduction will be followed by a quick questionnaire which will allow us to gather information about them (Refer to questionnaire in Supplementary experiment materials).

Experiment Phase 1 (Our Prototype):
We will initially present the scenario, followed by our prototype to the participant. They will be informed of the task which is to identify the consensus reached for a time and place for the event. Participants will be given an opportunity to request clarification if needed before proceeding.

For the task, the time and place of the event would be left as TBD and participants would be asked to identify the group consensus reached for both time and place. Once they start interacting with the interface the timer will start and will be stopped once participants answer the question set forth. We will then record both, the time it takes for the participant to identify the consensus, and the accuracy of the answers provided by our participants.

Experiment Phase 2 (Facebook Messenger):
We will then have the participant use Facebook Messenger. They will be informed of the task which is to identify the consensus reached for both time and place details. Participants will be given an opportunity to request clarification if needed before proceeding.

For the task, participants would be asked to identify the group consensus reached for both time and place. The timer will start once they begin interacting with the interface and stop when they answer the question set forth. We will then record both, the time it takes for the participant to identify the consensus, and the accuracy of the answers provided by our participants.

The consensus in this task would be different from the previous task to normalize any biases that are present (i.e: participant would not be asked to identify the same consensus again since they have seen it before in the previous task).

Post Experiment:
After completing both tasks, participants will be interviewed to gain information regarding user experience and feedback about the prototype and interactions.

Apparatus:

The experiment would be conducted in a well-lit classroom in the Computer Science building. The pre-experiment questionnaire and scenario will be loaded onto an iPad. The experiment will be conducted using one mobile device that has both interfaces. We will be filming participants performing the tasks with a camera. A cell phone timer will also be utilized to monitor the participants interactions.

Independent and Dependent Variables:

Independent variables would be the interfaces and user experience with event planning, as mentioned in Design section. The dependent variable would be the time taken by participants to identify general consensus reached out of the group settings, and accuracy of the answers.

We intend to measure the time taken by timing our participants from the point they start interacting with interface to the time our participant identifies the group consensus on time and place. We also intend to measure the accuracy of the answers provided by our participants by comparing their response to the correct answers.

Hypotheses:

H1: There is a difference in the time taken by participants to identify group consensus reached between our prototype and Facebook chat.

H2: There is a difference in the time taken by participants with frequent event planning experience than participants with little event planning experience in identifying group consensus in both our prototype and Facebook chat.

Planned Statistical Analysis:

Since each of our independent variables have two levels and we are only interested in the differences between the groups and do not need to know which is better than the other, we plan to use ANOVA to perform our statistical analyses.

Expected Limitations of Planned Experiment:

We anticipate that there might be issues arising from the experiment in terms of measuring the time taken for participants to give the correct answer to our question. Participants would vary in the way they present their answers, as some might talk through their thought process which may prolong the time. Another limitation might be the participant’s reading levels will be different regardless of the screening procedures. Some people naturally read faster or skim efficiently than others. This can cause an increase or decrease in time-to-find the consensus. The last limitation might be that since we plan to perform the experiment on a mobile device, there may be a discrepancy in terms of comfort and familiarity between participants in terms of using such a device. This can cause some very slight but notable discrepancies in speed, because participant will be faster or slower depending on their experience.

c. Supplementary Experimental Materials:

Pre-Experiment Questionnaire
Post-Experiment Interview Questions
Scenario Presented to Participants