Putting the script and interview questions on the top of the page for easier access:
Interview Script and Questions:
https://docs.google.com/document/d/1_foU-aBvDJ906hsayNaUU5svqDjUmoN7hVkyJBFb7wU/edit?usp=sharing
Blog Update #4a Revised goal(s) of experiment:
As mentioned in our proposed goals, we will be focusing on the first two goals, which are:
- Goal 1: Does the interface help the user establish or feel a sense of trust with other users from the information provided in the profile page, private message, and mutual friends.
- Goal 2: Are the users able to use the application to connect with suitable jam buddies for each other?
In order to narrow down our focus, and attain specific insight from our data, we revised our experiment goal to be more specific as follow:
- Based on the information provided in the profile page interface, the user will establish a sense of trust in looking for a new jam member, and this would develop further into connecting with the new jam member through sending a message or adding as a friend.
To test if our application succeeded in reaching this goal, we will be comparing our application to an existing application, BandFriend. Both applications support the task that allows the users to express their trust and connection with new members, but we believe that our application, Band&Jam, would have faster user’s selection time on new members, and have higher user preference from our selected workflow.
Blog Update #4b Experiment method
Participants:
We will have five participants who are over the age of 18 years old. The participants will all be musicians or play an instrument, albeit at different skill levels to provide some variety amongst the participant pool. We will recruit our participants through personal connections such as friends and friends of friends.
Conditions:
The participants will complete the given task on the existing application, BandFriend, that is available on the market now, and our application, Band&Jam.
Tasks:
We have provided tasks for users to interact with our system. First, users will walk around with the app after going through the tutorial of the app, the system will mock the data and populate the app with other users which they have passed by. Second, users will be given a pre-populated list of user matches based on proximity encounters, attributes, and mutual friends. Third, users will then be asked to explore the interface and choose 5 users in which they want to jam with the most. Fourth, users will be presented with another existing app(BandFriend) and do steps one to three again. Lastly, a semi-structure interview will follow by the end of the tasks where users will be asked some questions about the two apps.
Design:
The formal experimental design used will be a 2 sample (2 conditions) experiment based on the comparison of one sample mean with a constant. 5 participants will partake in the 2 sample experiment and will be conducted using within-participant design. The sample mean we will measure using our new design relates to proximity and its influence on how comfortable/quickly a participant took to add friends. The constant measure will not include proximity but will be an application currently used in the real world. The application we will be using as a constant is known as BandFriend. The comparison of performance data and performance requirements between the new design and BandFriend will determine whether our new design, focused on proximity, meets key design requirements. Performance data pertaining to speed and user experience will be obtained using Likert scales and video recordings as measurement tools.
Procedure:
Procedures of the experiment can be sectioned into three major components as following:
We will begin by briefing the participants on the process of the experiment, including the purpose of walking around, the tasks they will be expected to follow, and the prototype they will be working with. We will not disclose the entire experiment to the participant at this point, but enough information that they will be able to follow the tasks. Then, we will present the consent form to the participant. After signing the consent form, we will show our prototype to the participant and proceed to make our way to the HCI lab following our predetermined route.
When we have made our way to the HCI lab, we will first let the user navigate the two applications on their own. The order of whether BandFriend or Band&Jam will be used first will be randomized to cancel out any learning effect. At the same time, we will be noting any usability problems on the application. Subsequently, we will ask the participant to choose five musicians on the each application based on the information that is present on the application.
By performing these tasks, we wish to measure the preference level between BandFriend and Band&Jam and the execution time of the choosing task in order to find out whether our application has a better approach to the problem.
Lastly, we will be conducting a brief semi-structured interview. The interview will focus on the reasoning behind the selection of jam partner, what information will help in the decision-making process, and whether there are any previous unvocalized usability issues present on the prototype.
Apparatus:
The Apparatus that will be required for this experiment is our medium fidelity prototype and BandFriend on a mobile device, a Camera for video recording and photo taking, and laptops or notebook for note taking purposes.
Independent and dependent variables:
We will be using two separate interfaces as our independent variables. One interface will be the prototype of our application, Band&Jam, and the other will be an app in the market currently in the market named BandFriend. Our dependent variables will relate to the user interactions with each app. Firstly, the users will be timed on how long it takes them to find five members to form a band in each app and will represent our first dependent variable. The second dependent variable will be a qualitative measurement of the user’s preference between apps using a Likert scale. To summarize:
- Independent variables:
- Application interfaces – BandFriend VS Band&Jam
- Dependent variables:
- User performance time to pick their jam members (quantitative measurements)
- User preferences between apps (qualitative measurements)
Hypotheses:
Task performance time
- H0: User takes similar amount of time to select 5 jam members on both app – Band&Jam and BandFriend
- H1: User takes lesser time to select 5 jam members for Band&Jam than BandFriend
User preference
- H0: Users prefer both designs equally
- H1: Users prefer Band&Jam more than BandFriend
Planned statistical analyses:
For our statistical analysis, we will be using t-test with (p = 0.05) to determine the significance of our values. We have chosen this analysis as there is only a single factor comparison for our experiment with two conditions. We will also analyze on qualitative data, such as Likert scale answers.
Expected limitations of the planned experiment:
There are several limitations that we encountered when designing our experiment. First, the number of participants are limited to five due to time and resources constraint. We do not have the time to recruit too many participants to create a larger scale study environment. Second, places or locations are limited to UBC main hall, we are not able to conduct the experiment at many different locations, again, because of time and resource constraint. We believe that by running the experiment at various locations is highly recommended in the future if given enough time. Lastly, the list of matches is also limited to only certain pre-populated data. We have tried to eliminate all possible factors that could potentially bias our experiment. However, given the nature of our platform, the list of matches has to be faked since it is impossible to have carried out the intended idea (matching based on proximity) with only Medium fidelity prototype along with other limitations.