Experimental Protocol

Fifth project milestone: “Test” – Design Experimental Protocol

Testing is an important stage of Design Thinking. The goal of this milestone is to craft an experiment design and to complete your medium-fidelity prototype.  Note that there is a relatively high degree of interaction between parts 1 and 2 during this stage of the project. While they are presented sequentially in this document, some of the steps will need to be done in parallel. You will need to brainstorm possible experiment goals and designs in order to help you scope your med-fi prototype(s); on the flip side you will not be able to commit to an experiment design until you know how far you will get with your prototype.

  1. Experiment Design

By now, you have one design approach (or possibly two), which you’ve subjected to a qualitative walkthrough analysis. It’s time to evaluate the effectiveness of this approach in quantitative controlled manner.

Establish Proposed Goal(s) of Experiment:

Assess the key usability challenges or uncertainties of your design approach. What needs to be answered to help establish whether this is a good design approach? Use the insights gained during the cognitive walkthrough to inform your reflection, as well as any other input you have obtained (e.g. by discussing with other teams).

Some typical types of evaluation goals:

  • Design A vs. Design B
    If you have developed two prototypes, or there is a competitor system for certain tasks that your system supports, you may want to determine which of the two systems is most effective overall (or for given tasks).
  • Achieved performance threshold
    Another possible goal is whether users will be able to do a certain task within a threshold amount of time, or within a threshold number of errors. The goal in this case would be to test the task against a reasonable usability requirement target. Note that if you take this approach, you need to clearly justify the usability requirement target.
  • User group A vs. User group B
    Another possibility may be that you believe demographic variables or other individual differences (e.g., subject expertise) will impact the usability of your system. In this case, the user groups will be an independent variable in your experiment, and the main goal is to determine if there are performance or perceptual differences between user groups.

Interim reporting of the proposed goal(s) of experiment:

Make a comprehensive list, then rank items by (a) importance and (b) your ability to test them within the scope of 544. Then clearly identify which of the goals you will address in your evaluation. As a guideline, you should have 1 substantial goal or up to 3 smaller goals.

Length: approximately half a page

Where: Please submit your PDF document to Canvas with a title “Experiment-interim report-<team name>.pdf 

Draft of Experiment Protocol

Design the Experiment:

  • Determine your hypotheses. These are very specific instantiations of some of your evaluation goals. They need to be testable. You will likely have 2 to 4 hypotheses.
  • Consider your intended participants (i.e. the recruitment criteria) and how you will recruit them. If user group is an independent variable in your experiment, then consider the different recruitment criteria for each user group.
  • Determine the method you will use to test your hypotheses. This forms the bulk of the experiment design and includes:
  1. The independent and dependent variables
  2. The formal experimental design (e.g., a 2 x 3 mixed factorial (split-plot) design, more specifically 2 levels of expertise (between subjects) x 3 interfaces (within subjects) design)
  3. The experimental tasks that your subjects will carry out
  4. Specific details on how you will measure the dependent variables, e.g, task completion time, think-aloud verbalizations
  5. The statistical analysis you expect to carry out.
  • Create all the study instruments you will need. These will likely include questionnaires, interview questions, observation sheets, and the script you will follow so that all subjects get identical instructions.

Every experiment has limitations. You need to carefully think through the limitations of your chosen design, in particular with respect to different kinds of validity that were discussed in class.

Write up of the experiment protocol: 

The following components should appear in the order given below (unless you have good reason to adjust). As an example, please see Findlater et al. (on the reading list in Week 12, Nov. 24) which closely follows this structure.

  • Participants – describe participants, including total number expected and recruiting approach
  • Conditions – briefly describe what will be compared in the experiment (drop this section if only comparing performance against threshold)
  • Tasks – briefly describe tasks to give the gist of what participants are expected to do; point to appendices for detailed task descriptions
  • Design – formal experimental design
  • Procedure – describe the sequence of activities each subject is expected to follow; flow diagrams can be helpful for illustrating the sequence of events
  • Apparatus – describe the physical setup for the experiment; use photos to illustrate if appropriate
  • Independent variables – specify each, and include exactly how you intend to define, e.g., how will novice and expert users be determined?
  • Dependent variables – specify each, and include exactly how you intend to measure each one and justify this measurement approach
  • Hypotheses –state in terms of the independent and dependent variables
  • Planned statistical analyses – every dependent variable should be present in at least one hypothesis
  • Expected limitations of the planned experiment – stated in terms of types of validity that is threatened
  • Appendices: each study instrument, script, and consent form. If you need more space to clarify the tasks than available in the main body of the report, such as showing screen captures of your system, or a competitor’s system against which you will be evaluating your system, include those in the appendix as well.

IMPORTANT: In terms of the level of detail required for the experiment deliverables, you should be able to give a classmate from another team your experiment description and materials and they should be able to carry out your experiment.

Length: up to approximately 3 pages (+ appendix for all supplemental experiment materials)

Deliverable: Experimental protocols will be shared in class on Nov. 26.  We will do some workshopping on your protocols during class.  After the class, revise your protocols based on the feedback you received and submit them to Canvas, “Experiment-protocol-<team name>.pdf