Category Archives: Update 7

After completing Milestones I to IV, we reflected on our design process and had several points to consider:

• Our design and evaluation focused on an aspect of a collaborative video platform but not the entire system. The original vision of our design was to create a collaborative video platform, to help users watch educational videos and seek useful guidance when a concept was unclear. At times, it was difficult to realize the relationship between low level details of our design and the high level design goals. For example, timing users completing the video task could show us if interface type affects how users complete tasks in a video. Similarly, comment search time could be considered as a metric of how easy it is for users to seek help. In hindsight, we could have focused on testing different aspects of the system, that would help reinforce the relationship between the low level details and the high level design goals.
• When developing the medium-fidelity prototype, we felt restricted by our ability to implement it in Javascript and HTML. Specifically, we wanted the segments to be labelled on the video which we could not implement due to our limited experience with the Youtube player API and time constraints. The visual appearance and graphic design of the website also suffered due to our lack of experience designing websites.
• One of the things that was surprising when conducting the study was that users’ abilities varied greatly. Although piloting indicated that the difficulty of the tasks was reasonable, during the actual experiment, some participants struggled greatly with the task while others found it easy. As a result, user performance variation should be considered more carefully when designing a study.
• Lastly, we could have performed another field study to see how people would interact with our segments and annotation system. We found that when designing our experiment, we had to make many assumptions about how annotations would be used and the type of content that they would hold. Having a better idea of this would have allowed us to develop a stronger and more valid experiment.

We were unable to find a statistically significant difference in overall task completion time or comment/annotation search time between System Blue (our developed system) and System Red (YouTube). However, 6 out of 8 participants agreed that it was easier to complete tasks on System Blue than on System Red, and that they would be more likely to use System Blue over System Red for watching educational video. 6 out of 8 participants also indicated that they liked the feature of having the video divided into smaller segments for navigation. Although no statistically significant timing differences were found, a majority of participants preferred System Blue’s annotation and video segmentation system.

From the results of our experiment, we can recommend that the overall approach of our system was valid. However, there are various recommendations that can be made to further improve both the design of our system and experiment. Firstly, we could modify segments by integrating them into the video playback bar by adding markers, instead of only  having hyperlinks under the video. This would make our segment design more visually salient and possibly affect the way that users interact with it. Secondly, we could focus more on learning about the types of interactions users would have with annotations to enlighten our interface design. It could also be useful to determine which types of annotations people post, as well as what they find useful for their video completion goals. Thirdly, since our experiment only examined one type of video (tutorials) it could be useful to include a wider range of video types, such as informational videos. Lastly, we would recommend that the experiment be conducted again, with some minor changes. The videos being used in the experiment should be changed. The current videos proved to be too challenging for some users and resulted in many users not completing the prescribed tasks in time.