Mid-Fi Prototypes and User Testing

Due: Thursday, Mar 31 11:59PM
Point Total: ~15% of your TA grade (6.8% overall)

Assignment Overview
Your assignment is to convert a subset of your more promising sketches from the last assignment into interactive, mid-fidelity prototypes and to test them with four users. You will create and test two different versions of your prototypes. As I've noted before in class, comparing two designs is far easier/better than evaluating one single design in isolation. For example, it's easier for a user to express preference between one design over another rather than articulating, exactly, why they don't like a single design.

Following from the IRB protocols we discussed in class, your participants must sign an informed consent document before taking part in your test.

Your report should include figures of your prototypes, a description of your study method, and a breakdown of your primary findings from the user testing.

What to Do

  1. To start, iterate and refine the four primary tasks that users should be able to accomplish with your application (based on learnings/reflections from the last assignment).

  2. Then, transition to the first core part of the assignment: riff, iterate, and create two different mid-fidelity prototypes for the four primary tasks. That is, you must create "Prototype #1" that allows your users to accomplish the four tasks one way and "Prototype #2" that allows your users to accomplish the four primary tasks another way. The prototypes should be functionally different so that your users can compare and contrast their experiences with both of them. So, for example, the two prototypes should represent the tasks in fundamentally different ways. Remember, the focus here is not on aesthetics/beauty but rather on understandability, usability, approachability, and, to some degree, layout, widget type, etc.

    For creating the mid-fi prototypes, I suggest Balsamiq; however, your team can explore and choose other options. I want the focus to be on layout, navigation, understandability vs. aesthetic so tools that allow you to maintain a sketch-like look are preferred. For Balsamiq, I invited the entire class to our class account (link), which Balsamiq was kind enough to donate to us for educational use.

  3. Once, you've created the two prototype designs, beta test them with members of your team and make requisite changes. This "eating your own dog food" is a good way to catch errors before investing time in testing with actual users.

  4. Now you're ready for real user testing (the second core part of the assignment). Recruit four independent users to test out your paper prototypes. Ideally, these users should be representative of your target population. If this is not possible, then find reasonable alternatives. You cannot use members of this class. Each user testing session must be done in isolation (that is, you cannot have more than one user testing at a time) with at least two experimenters present. For the testing session, you should follow this protocol:
    1. First, download and modify this IRB "informed consent" template to fit your project [link]. At the beginning of the user testing session, read the "Purpose of this Study" section of the consent form out loud to your participants. This should be done consistently for each participant. Then, give your participants a chance to read the entire consent form, ask questions, and, if they agree to participate, have them sign the form. If they do not agree to participate, simply wish them a nice day and recruit another participant (it can be slightly awkward but this happens!). If they do agree to participate, provide a copy of the form and take the signed copy for yourself (please scan in the signed consent form and include it in your report appendix).

    2. After the informed consent process, you can begin user testing the prototypes. Follow the method described in the Nielsen Norman "Paper Prototyping: A How-To Video" but adapt it for your interactive prototypes. Because you have two different prototype designs for your tasks, you should fully test one prototype for each of the four tasks before testing the other prototype. Because all subjects are exposed to all conditions, this is a "within subjects" study design. Consequently, you need to counterbalance your prototype conditions. In this case, two users should test Prototype #1 then Prototype #2 and the other two users should test Prototype #2 then Prototype #1. The condition order for all four participants should be set before you test your first participant.

    3. As the user is interacting with your prototypes, ask them to "think-aloud." One experimenter should record notes about how the participant was using the prototype, the problems/successes encountered, and comments made during the testing sessions. Scan in and include these raw notes in your appendix for each user testing session. I would like each individual student to serve at least once as a "observer/recorder" and at least once as the experimenter running the session.

    4. After both prototypes have been evaluated, provide your participant with a short paper survey with a few questions asking about which prototype the user liked better and why as well as questions seeking participant ideas for improvement. Scan in and include these responses in your appendix for each user testing session.

  5. After user testing, analyze your collected data (e.g., observational notes, post-study survey data). As you have only four users, it does not make sense to perform statistical analyses but you can report the raw data (e.g., in tables or graphs), which should indicate trends (or the lack thereof).

  6. Finally, you need to write up your report. See below.

The Report

Submit a report of no more than 5 pages of text. Images, tables, figures, etc. are strongly encouraged, do not count against the page limit, and are thus free. Please use the following full section headers (style Headings 1) in your report; it makes it easier for us to grade. So, for example, you should literally have headings with Section 1: Abstract and Section 2: Task Descriptions in your report.

Title Page

At the top of the report, please include a title (centered and bold) followed by the names of each team member. For each person listed, include 1-2 sentences on their primary role/accomplishments on this assignment.

Section 1: Abstract

The abstract should provide a 5-7 sentence overview of your report including: (i) a description of your application, (ii) a description of your four primary tasks, (iii) a high-level description of the two mid-fi prototypes and how they differ, (iv) a description of your evaluation method, and (v) a summary of your primary findings.

Section 2: Task Descriptions

Describe your four primary tasks (iterate from previous incarnations). An intro paragraph should summarize the four tasks. Each subsequent paragraph should start with a bolded name of the task and a more lengthy description. If the tasks changed from the previous assignment, please describe these changes and why they were made (e.g., was it based on your own design reflections, was it based on the experiences of storyboarding, was it based on feedback from in-class critiques, etc.).

Section 3: The Mid-Fi Prototypes

The introductory paragraph to this section should introduce the two mid-fidelity prototype designs and, at a high-level, the primary similarities and differences between each. Include screenshots! Why are they different? Then, there should be two sub-sections (one sub-section for each mid-fidelity design). Name the sub-sections Mid-Fidelity Prototype Design #1: <name> and Mid-Fidelity Prototype Design #2: <name>. In each of these sub-sections, describe the mid-fidelity prototype for the four primary tasks. Again, make it easy for your readers to understand the most important points and, in particular, how you made different design choices with each design--include as many screenshots as possible (with annotations, when relevant) to support your points. You should include as many inline figures as is appropriate to help describe your prototype but not so many that it makes your report difficult to read. If the mid-fidelity prototypes can be accessed online, please include a link. Finally, tools like FluidUI allow you to zoom out and take a screenshot of the "flow diagram" (see below). Please include this in the appendix (if applicable).

external image 7705073436_94bc341e5b_o.jpg



Section 4: Testing the Mid-Fi Prototypes
This section should have four sub-sections focused on:
  • Pilot testing the paper-prototypes among your group and the resulting changes that were made either to the study protocol or to the paper prototypes themselves
  • Participant recruitment and demographics, which includes a description of how you recruited participants and demographics relevant to your project. Justify why these participants were appropriate for your project.
  • Study method, which details how you performed your study including the study protocol, the location, the length of the study, and a description of the data collection instruments (e.g.,the post-study survey).
  • Analysis, which describes how you analyzed the data you collected.


Section 5: Results from User Testing

This section should detail the primary results from user testing.
  • What did users generally think of your application idea?
  • How did they react to the two different prototypes?
  • Were there some features of each prototype that were more promising than others? Could they be consolidated into one design?
  • How might you change your design based on the user testing?

Section 6: Learnings from this Project Assignment

This section should offer reflections on what you learned from this project assignment and what you would do differently if you had to do the assignment again.

Section 7: Appendix

The appendix should include:
  • Screenshots of the two designs that cover the four primary tasks
  • A scan of the four signed informed consent forms
  • Raw notes from four user testing sessions (notes should be clearly marked with a timestamp and session number)
  • A scan of the post-study paper survey responses (these should also have session numbers)

Submission

Please submit a PDF of your report to Canvas.

Rubric

You can download the rubric as an Excel file below:



We've also taken a screenshot of the rubric for your convenience. You can click on the image below to enlarge.
TA04-MidFiPrototypesandUserTesting-GradingRubric_v1b_MLM_cropped_650w.png