A. FINAL CONCLUSIONS AND RECOMMENDATIONS

Overall, we think that we have successfully validated our proposal, although we have also identified major gaps that need further work. The main idea behind our design was that anyone can feel comfortable reading and sharing political news, although today that activity is commonly perceived as a special interest of a particular kind. We tend to interpret this attitude as something that needs to be changed, rather than a constraint that could discourage further work.

As our experiment showed, our prototype didn’t surpass Reddit overall, however it didn’t fall behind either: no significant differences were observed in performance speed and perceived user satisfaction. As we were putting our prototype next to a fully polished app, used daily by millions of people, the comparison was somewhat not in our favor from the beginning. In that light, we interpret the fact that the prototype finished close to Reddit in terms of performance speed and user satisfaction as a success. We conclude that our proposal is viable, although further development is needed to make it really comparable to existing popular systems.

One aspect of our app that calls for further in-depth study is the moderation process. Based on our experimental results, we concluded that our moderation proposal shows promise, however our subject pool was quite small to understand how moderation will work in real-world scenarios. Therefore, we recommend a further study that would involve a larger pool of participants, as well as a bigger sample of the articles that have to be moderated during the task.

Other recommendations include:

• Supporting Android as well as iOS: we didn’t have Android support in our experiment, which slowed down several participants with no iOS experience;
• Make sharing much quicker, available directly from the browser: copying the URL and article headline from one app to another was very time-consuming;
• Detecting the names of political figures in the article automatically: our participants needed to find those names in each article, which took a lot of time;
• Including more features in the prototype: it will make the comparison with an existing system more fair.

Crucially, a key insight from our project is that credibility and trustworthiness need to be considered as central design problems when designing digital media. Filtering the stream of news is a crucial feature expected today by many users, and the lack of it could lead to the users being less engaged with a particular medium overall.

 

B. REFLECTION ON THE DESIGN AND EVALUATION PROCESS

We gained a large amount of insight from our users that would not have been predicted based on our own experience and intuition:

• During the field study interviews, we initially assumed that sharing news would be a more prominent feature.  However, we learned that a large majority of our users were consumers of content, not sharers.  This apprehension towards sharing political news with strangers and acquaintances was not predicted based on our own insight.

• We considered addressing this issue by allowing users to post anonymously, but the problem of slanderous posting arose.
• This idea then eventually evolved in into the user moderation of news articles.

In addition to insight about our users, we gained experience with design methods.

• We conducted multiple brainstorming sessions throughout the entire design process.  These brainstorming sessions provided two key benefits:
  • First, we were able to better conceive novel ideas as a group while sharing our thought processes.
  • Second, we realized which ideas were more popular because of the immediate reactions received.
• In order to organize the scattered thoughts created during a brainstorming session, we began to use affinity diagrams.
• Affinity diagrams helped organize the entire group’s ideas into useful themes that could be delved into deeper.  In conjunction with brainstorming, this method was the most useful for generating ideas and coming to a consensus.
• The pilots provide good feedback that helps avoid collecting useless data during the actual experiment.  They also helped polish the experimental procedure, this aided in making the actual experiment more professional.

Prototyping

• It was exciting to see our proposal come to life when the prototype was made. It would have been hard to present our ideas to the users in any other way: the prototype is almost universally accessible, and our participants were comfortable sharing opinions about it.

Experiment

• We used both interviews and questionnaires to gather data during the studies.  We found that interviews were better than the questionnaires in collecting useful data.
• The participants would interpret the long form open-ended questions on the questionnaire as optional.  This was likely due to sheer laziness.
• Once asked to provide the same answers verbally, all the participants complied and provided far more rich and useful data.
• We didn’t find video and screen recordings particularly valuable for gaining new insight about our proposal. It worked ok for documenting the experiment, but we didn’t really gain any new information from the footage.
• Although they provided rich data, the analysis of video data was far too time consuming and unrealistic.
• Having an additional interviewer present during the experiment filled in the gap of missing data that one interviewer may not have been able to collect, thus negating the need for video recordings.

What we would’ve changed

• Perhaps we would avoid fully transcribing user interviews, as that was very time-consuming. An approach that would have worked better is having two interviewers, with one guiding the interview and the other taking very detailed notes.