Moderated Testing

Traditionally, usability tests have been moderated. 0:00 A moderated test just means that there's a moderator present asking 0:03 questions in order to deeply understand each choice the participant makes. 0:07 They may be sitting in your office and using your prototype, or you may go out 0:12 and observe someone test your product whatever you intend for it to be useful. 0:17 Alternatively, moderated testing can be done remotely. 0:22 There are many remote testing services that allow you to stay in the office 0:26 while observing and holding discussions 0:30 with people using your service from the comfort of their own home. 0:33 So what are the advantages of a moderated study? 0:38 Probe deeper. 0:41 When you're with a participant at each step, 0:42 you can ask what they're thinking and why they've made a particular choice. 0:44 For example, why did you click here, or did you see this instructional text? 0:50 Capture participant reactions. 0:54 Did you notice that sigh or look of confusion? 0:57 You can add that to your notes as additional data. 1:00 Without someone there, nonverbal reactions can easily be missed. 1:03 [SOUND] Protect sensitive information. 1:08 Say you're working on a product that really shouldn't 1:11 be revealed to the public until much, much later. 1:15 By moderating, particularly in person. 1:18 You've limited any exposure to the highly vetted group of study participants, 1:21 who've all ready signed an non-disclosure agreement, or NDA. 1:26 [SOUND] Evaluate low fidelity concepts. 1:30 If testing early and possibly even with just a paper prototype, you will need 1:33 to be present to guide the participant on what features are functional. 1:38 [SOUND] Keep on track. 1:43 If you notice the participant is focusing somewhere irrelevant to 1:44 the study's objectives, you could politely guide them back to the task. 1:49 Test complex products. 1:54 When testing products such as business software, you will need to make sure 1:55 that your participant has the background to complete the required test. 2:00 And in addition, 2:05 you will likely need to provide clarifying information throughout the study. 2:07 Let's say that your study is the perfect candidate to be moderated. 2:12 For example, you have a click through prototype for 2:17 your complex business software. 2:21 One next question is, will you be running this study at your office, 2:23 in the wild, or remotely? 2:28 A moderated study needs to be in real time but 2:30 you do not need to be in the same physical space as your participants. 2:33 If you do choose to bring the participants into your office, you will need to make 2:38 sure your office is set up in a way that will make people comfortable and 2:42 limit distractions. 2:46 Ideally, this would include a dedicated space in the form of usability lab. 2:47 Let's take a look at the key attributes of a usability lab. 2:53 First, you need a room with a closed door to minimize distractions during the study. 2:57 Inside the room, you'll need a computer with a monitor. 3:04 An external monitor rather than just a laptop is preferred, so 3:08 that it's easier for you and the participant to see the screen. 3:12 Attach a camera to the monitor if it doesn't have one already, 3:16 to record the study. 3:20 And confirm that the microphone in your computer is working. 3:21 Make sure there's a keyboard and standard mouse available as well. 3:24 There should be at least two chairs in the room. 3:29 Place the chair for the participant directly in front of the monitor. 3:31 And place your own chair slightly behind, and next to the participant's chair. 3:35 You can also include an additional chair for a note taker towards the back. 3:40 The key component of usability study is also being able to invite outside 3:45 observers. 3:50 You can do this in a couple of ways. 3:51 One is to have a couch or row of seats in the back. 3:53 However, I believe this can be intimidating for the participant and 3:57 prefer to put the observers in a totally separate conference room. 4:01 And stream the study into that room. 4:06 The observers would then be able to hear the audio, see the product being used, 4:09 and if possible also see the participants facial expressions. 4:14 Make sure to set up a messaging system between the usability lab and 4:18 the observation room, so that the observers can text, email, or 4:22 chat their questions throughout. 4:26 You may need specialized software to record, observe and 4:28 analyze the final videos. 4:32 You can find the list of options in the teacher's notes. 4:34 Remember to compensate your participants generously. 4:37 They're taking time out of their day to commute to you and 4:40 to provide you with the valuable feedback that you need. 4:43 The exact costs will vary by location, but a rough guideline, 4:46 if you're in a metro area and asking people to come in for about an hour, 4:51 anticipate providing $100 per person. 4:55 Testing a product in person is great for keeping things confidential, and for 4:58 testing physical hardware when you space design. 5:03 However, you will be limited to the people living close to your office. 5:06 Now, imagine designing a product meant to be used by everyone in the world, 5:12 as I did there in my How representative do you think our research 5:16 would have been if we tested with just people in the US, or California? 5:21 Or those in Silicon Valley willing to commute to our office in Mountain View? 5:26 We'd be left with a group of early adopters, tech enthusiasts, and 5:31 frequent usability testers. 5:36 Not exactly diverse representative group. 5:38 Now, let's go back to the usability lab setup we just described, and 5:42 adjust it for a remote testing study. 5:46 Remove an extra chair and take a seat in front of the monitor yourself. 5:49 You can use remote testing software to allow the participant to take control of 5:54 your screen, and use the prototype available in much the same way as before. 5:59 Alternatively, you can take an available conference room and 6:03 run the session straight from there. 6:06 If you include outside observers, they can then join you in the same room or 6:08 from their own desks if the session's being streamed. 6:12 Remote testing is rarely seeded for 6:16 hardware products, paper prototypes or highly confidential material. 6:18 Mobile devices can also be tricky, 6:22 depending on the kind of hand gestures are important, or if an install is required. 6:24 However, remote testing is a fantastic solution for 6:29 the online products you're most likely to be working on. 6:32 Remote testing is convenient. 6:36 You don't need to leave the office, and it provides access to a much, 6:38 much wider range of participants. 6:43 While working on advertising software at Google, 6:45 we relied heavily on moderated remote testing. 6:48 Because of this, 6:51 we were able to test frequently as we developed new features and products. 6:52 However, it was also important to occasionally visit our users in their 6:57 workspaces to understand the greater context in which our tools are being used. 7:01 In one site visit, we're able to discover that clients would use 7:06 an additional monitor that displayed other software, 7:10 such as spreadsheets to make up the features that our product was missing. 7:13 A very insightful observation that would have been impossible to discover remotely, 7:18 or in a lab. 7:23