Blog Update #7a – Final Conclusions and Recommendations:

Building consensus quickly is an important part of spontaneous/short-term event planning. We studied consensus building in group settings using qualitative and quantitative methods. The results of our statistical analysis ultimately proved inconclusive and did not support our hypotheses. It is possible that this was a result of unforeseen and uncontrolled errors in the experimental design. One limitation that comes to mind, is the amount of “Wizard of Oz” design involved in our experiment. We focused more on how clearly the information was visible to users as they made a conclusion on the group consensus, therefore a lot of information was fixed. This isn’t completely irrelevant to the larger problem of consensus building as from the perspective of the user organizing the event, being able to quickly determine what works best for everybody is a useful feature. But we weren’t observing consensus building in its natural form which usually happens in real time due to the logistical issues involved in planning such an experiment within the amount of time delegated to us. This could produce some effects, such as how suggestions and votes are updated/handled, which could also change the visibility of the feature as well.

From the qualitative data collected, we saw that participants liked the top suggestions feature and ability to vote on others’ suggestions. In addition, our participants found it easier to identify the consensus reached in our interface as compared to Facebook Messenger, which lacked this function. This was no surprise as consensus building was the priority of our interface thus the implementation of this feature. Yet, many participants would still look through the entire chat in our interface despite acknowledging the consensus in Top Suggestions.

Due to the positive reception of some of the features in our interface, the experiment could be run again, but with more emphasis placed on the top suggestions feature and modifying it to make it clearer and visible to the user. To tackle the issue of a busy interface and reduce the number of screens the user has to go through, it might be beneficial to utilize a hybrid system combining the full chat and Top Suggestions; users see the full chat on the events page until the length has exceeded a certain amount of lines, at which point the interface operates as it was implemented for this experiment. This could allow for quicker read throughs and reduce the number of steps involved to complete the task.

Ultimately, using the findings that we learn from our experiment and the above recommendations, we can move forward with a new iteration of our application.

Update #7b: Report Your Reflections

What were the most significant ways in which the design concept and the actual interface design changed under the influence of user involvement? What were the biggest surprises for you – the things you learned from or about users that you would not have predicted based on your own experience and intuition?

Our field study worked out well for us because it helped narrow down the scope of our project after talking to our users. In our field study we found that spontaneous event planning was a good direction to go since it was a less explored direction of event organization and there was more opportunity to improve the experience of this task.

Another surprise was that even though users confirmed they fully understood the tasks required of them they disregarded them when actually performing the tasks. The most prominent example was when we told them to say “Done” when they are ready to write their answers, almost all the participants starting their answers down without saying “Done”. This affected the way we stopped the timer and could have impacted our results.

Additionally, many users failed to comprehend the “Top Suggestions” and voting features when performing tasks. When conducting a walk through of the prototype, users appeared to understand the features and interacted with it. However, there seemed to be a disconnect as many users suggested we implement a voting system and make popular group choices more prominent. However, since we frequently changed who was running the experiment in our group between sessions, there may have some deviations from the script between individuals that may have affected how participants understood the instructions.

Did the methods you chose for your evaluation and prototyping get at what you were looking for? In hindsight, would a different approach (process, not specifics of your interface) have been better?

For prototyping, we chose to use Axure as our medium fidelity prototyping tool. We found for the purposes of testing a user’s ability to identify consensus, our prototype had enough functionality to test our hypotheses. On a horizontal level, Axure was more than sufficient to accomplish the basic functionality required for the prototype. The implementation of the chat interface, which acted as the vertical element of the prototype, was a bit more difficult to pull of due to some of the limitations of Axure (such as persisting messages) meaning we couldn’t test it as thoroughly, but as participants were not required to interact much with it the experiment wasn’t affected. The amount of effort and time to create the prototype was ideal in that Axure provided tools to easily implement many of the basic features of our interface. Axure also allowed us to test our prototype on mobile devices making interaction more realistic to the user.

For quantitative evaluation methods, we timed our participants’ speed and measured the accuracy of the answers. These worked as they fit our research topic and got us the appropriate data to analyze. As for qualitative analysis, we provided a post-experiment questionnaire. In hindsight, we could have asked more specific questions in the questionnaire (e.g. specific questions targeting our top suggestions and voting features).

What were the most, and least, valuable among the methods you used, either generally or specifically for your project?

One of the most valuable methods was using Axure for creating our prototype. It allowed us to create a medium fidelity prototype within the time constraints for the project while also supporting the correct scope for experimentation.

However, our interface walkthrough for the experiment was not effective. We designed thinking that it would help our user familiarize themselves with our interface, but most participants did not seem to learn our interface as well as we thought they would. For example, one participant suggested we use a voting system to rate the top suggestions despite it being shown in the walkthrough.