Test 1

Fifth project milestone: Test – Part 1

This is the first part of the Test stage of Design Thinking. The goal of this milestone is to craft an experiment design and to complete your medium-fidelity prototype.

Note that there is a relatively high degree of interaction between parts 1 and 2 during this stage of the project. While they are presented sequentially in this document, some of the steps will need to be done in parallel. You will need to brainstorm possible experiment goals and designs in order to help you scope your med-fi prototype(s); on the flip side you will not be able to commit to an experiment design until you know how far you will get with your prototype. Note that you need to submit your proposed goals for the experiment as an interim stage, and then incorporate the instructor feedback before proceeding.

1. Experiment Design

By now, you have one design approach (or possibly two), which you’ve subjected to a qualitative walkthrough analysis. It’s time to determine using a quantitative controlled manner whether this approach is effective.

Proposed Goal(s) of Experiment:

Assess the key usability challenges or uncertainties of your design approach. What needs to be answered to help establish whether this is a good design approach? Use the insights gained during the cognitive walkthrough to inform your reflection, as well as any other input you have obtained (e.g. by discussing with other teams).  

Some typical types of evaluation goals:

  • Design A v.s. Design B
    If you have developed two prototypes, or there is a competitor system for certain tasks that your system supports, the main goal is likely to determine which of the two systems is most effective overall (or for given tasks). This represents a classic evaluation of Design A vs. Design B.
  • Achieved performance threshold
    Another possible goal is whether users will be able to do a certain task within a threshold amount of time, or within a threshold number of errors. The goal in this case would be to test the task against a reasonable usability requirement target. This approach is well covered in Section 10.8 of the Experiments in Support of Design chapter — which the course staff can provide to you as a supplemental reading. Note that if you take this approach, you need to clearly justify the usability requirement target.
  • User group A v.s. User group B
    Another possibility may be that you believe demographics (i.e. subject individual difference) will impact the usability of your system and so you may want subjects in two different demographic groups. In this case, the user groups will be an independent variable in your experiment, and the main goal is likely to determine the performance differences between user groups. One example of this is comparing usability issues between novice and expert users.

Interim reporting of  the proposed goal(s) of experiment: Make a comprehensive list, then rank items by (a) importance and (b) your ability to test them within the scope of 544. Then clearly identify which of the goals you will address in your evaluation. As a guideline, you should have 1 substantial goal or up to 3 smaller goals.

Length: approximately half a page

Where: Please submit your PDF document to Canvas with a title “Test_1-interim report-<team name>.pdf

Final Goal(s) of Experiment:

1.1 Report the (revised) final goal(s) of experiment: Incorporating the feedback you received from your interim post, indicate the final  goals that you will address in your experiment.

Length: approximately a third of a page

Design the Experiment:

Now it is time to actually design the experiment that will evaluate your system. This will require a lot of effort, much more than the informal evaluations you have conducted to date.

Determine your hypotheses. These are very specific instantiations of some of your evaluation goals. They need to be testable. You will likely have 2 to 4 hypotheses.

Consider your intended participants (i.e. the recruitment criteria) and how you will recruit them. If user group is an independent variable in your experiment, then consider the different recruitment criteria for each user group.

Determine the method you will use to test your hypotheses. This forms the bulk of the experiment design and includes:

  1. The independent and dependent variables
  2. The formal experimental design (e.g., a 2 x 3 mixed factorial (split-plot) design, more specifically a 2 levels of expertise (between subjects) x 3 interfaces (within subjects) design)
  3. The experimental tasks that your subjects will carry out
  4. Specific details on how you will measure the dependent variables
  5. The statistical analysis you expect to carry out.

Create all the study instruments you will need. These will likely include questionnaires, interview questions, observation sheets, and the script you will follow so that all subjects get identical instructions.

Every experiment has limitations. You need to carefully think through the limitations of your chosen design, in particular with respect to different kinds of validity that were discussed in class.

1.2 Report the experiment method: This includes a detailed description of the following components, that should appear in the order given below unless you have good reason to adjust:

  • Participants – describe participants, including total number expected and recruiting approach 
  • Conditions – briefly describe what will be compared in the experiment (drop this section if only comparing performance against threshold) 
  • Tasks – briefly describe tasks to give the gist of what participants are expected to do, point to appendices for detailed task descriptions 
  • Design – formal experimental design  
  • Procedure – describe the sequence of activities each subject is expected to follow 
  • Apparatus – describe the physical setup for the experiment (use photos to illustrate if appropriate) 
  • Independent and dependent variables – includes exactly how you intend to measure each dependent variable 
  • Hypotheses – remember to state these in terms of the independent and dependent variables 
  • Planned statistical analyses – every dependent variable should be present in at least one hypothesis
  • Expected limitations of the planned experiment – stated in terms of types of validity that is threatened

As an example, please see the upcoming reading from Findlater et al. which closely follows this structure.

In an appendix, include: each of your study instruments, your script, and a revised consent form. If you need more space to clarify the tasks than available in the main body of the report, such as showing screen captures of your system, or a competitor’s system against which you will be evaluating your system, include those in the appendix as well.

IMPORTANT: In terms of the level of detail required for the experiment deliverables, you should be able to give a classmate from another team your experiment description and materials and they should be able to carry out your experiment.

Length: up to approximately 3 pages (+ appendix for all supplemental experiment materials)

2. Medium Fidelity Prototype

Complete your implementation of your medium fidelity prototype.

2.1 Prototype demonstration: Document your final interface prototype implementation, using screenshots/screen capture, photos, or the equivalent for your project. Include explanatory/descriptive captions for figures, or narration for video, as appropriate. Where appropriate, justify design elements with respect to what you have learned about human abilities and limitations.

Length: There is no specific word limit for this part. However, please avoid excessive length and concentrate on conveying key aspects. If in doubt, consult course staff.

Presentation and report:

As with the previous milestones, your team will briefly present your work on this milestone in class: 5 minutes per team. Your presentation should focus mainly on the experiment design (~4 min), but in order to ground that, you should first very briefly present the status medium-fi prototype (~1 min). You will have the opportunity in the next design review class to present the prototype more fully.

For your report, include an appendix that identifies each team member’s contributions. You many optionally include additional appendices for content that does not fit into the main sections of the report.

Teams need to submit two pdfs in Canvas:
1) a pdf of their report, filename: “Test_1-report-<team name>.pdf
2) a pdf of the presentation, filename:
 Test_1-presentation-<team name>.pdf
There is one submission slot for each type of pdfs you should upload in Canvas. Please submit accordingly with the specified name.