#7a: Conclusion and Recommendations

Conclusion:
Overall, the trends observed our data showed that the idea of a chat-bot for event planning was greatly appreciated by users, especially introverts. Initially, the idea was to design a system for introverts, although results showed that it benefited both personality types. However, due to time and experimental limitations, we were unable to do a large enough experiment to fully validate our hypotheses. We evaluated a small subset of functionality in our design, focusing on event-planning assistance according to the user’s interest and availability. However, user profiles were faked, as were common interests and schedules for the purpose of this experiment, so this should be explored more.

In terms of performance time for the inclusion of Social Butterfly, we concluded that for task 1 (group), the results were almost significant, and for task 2 (solo), significant. According to summary statistics, the main effect was lower time on average when using Social Butterfly. We can conclude that users understood the rationale behind the onboarding experience. However, there was some confusion around inputting the data, specifically how to indicate the levels of an interest, or how to fill out the ‘how much me-time’ page, as it was not indicated on the user profile provided. Users felt that Social Butterfly was abrupt, intrusive and lacked context to suggest relevant events. Providing event options was also a common suggestion. Due to experimental limitations and time constraints could not explore this further, so cannot draw conclusions about the appropriateness of our onboarding options at this time.

Recommendations:
Overall, minor adjustments are needed, but our approach was validated for this aspect of the design. However, there are many other areas that need exploring. One of the main concepts that we couldn’t explore in detail through our experiment was how to measure and accommodate social burnout. Further research in this area would be beneficial in developing a more robust design for the target audience of introverts. With what our study was able to cover, however, some other things to consider are: how larger group dynamics will affect the group task, bot personality (i.e. when and how should chat-bot jump in the conversations), the value of the chat-bot in a solo chat setting, and testing the connection between the onboarding preferences input and recommendations provided by the Butterfly. Additionally, more research can be done in terms of using natural language processing and artificial intelligence to help chat-bot.  

#7b: Reflection on overall design process and experience designing an interactive system 

Beginning our project with a field study was a great way to start. We were able to mine information about what interfaces people liked using for socializing with friends, which then directed our work. We also learned that people, especially introverts, prefer face-to-face interaction, which surprised us, and would not have been reflected in our experiment design without the field study. Interviewing people shifted our understanding of what people want from a socializing interface, validated our concept of social burnout, and added an understanding of event attendance motivations.

Prototyping iteratively was another great method of learning. Beginning with the low-fi prototype allowed us to identify places where we were recreating existing interface functionality. This led us to explore a complementary interface, rather than a replacement, which led to our final design.

The affinity diagram was very effective for lifting qualitative themes from both our field study and our experiment. Piloting our study helped refine our experiment for clarity. Ideally, we would have done a couple more pilot sessions because we still ran into errors in the experimental process that could possibly have been clarified. Furthermore, this would have also helped to refine our hypotheses metrics of success (e.g. more clarification about the definition of error rate. We defined error rate as the number of times a plan was declined in the conversation, however, there was ambiguity: did we count it as multiple errors when multiple participants turned down the same event? Or was it considered one error because the event was the same?)

In terms of other things that didn’t work as well, the experiment was very rushed. A long term field study would likely be more effective at getting at some of the things we were looking for. It felt too soon for such a quantitative study, when many of the ideas we were exploring were still a bit vague and undefined. In addition, we should have spent more time refining our data collection strategy for the experiment. We ended up not using the video for analysis, and it would have been better to know that up front. Screencasting would have been more useful. We also could have collected more qualitative data, which would have cemented our ideas and outcomes for future research. From a strictly logistical perspective, we collected non-continuous data for user satisfaction, which rendered us unable to run an ANOVA test on our results, and compromised our statistical integrity. We also had self-reported introverts and extroverts. For more valid results, we should have our participants do an intensive social classification demographic survey (of which there are many approximately hour long options).

Additionally, with more time, we would have liked to take more time to design the chatbot before conducting the experiment, to figure out specific cases to explore and account for. Interviews about the medium-fi prototype would also have been helpful, in order to collect more rich data and get a greater understanding of the needs for the onboarding experience.