1 00:00:00,000 --> 00:00:04,918 [MUSIC] 2 00:00:04,918 --> 00:00:07,508 Usability testing is a great form of quality 3 00:00:07,508 --> 00:00:10,658 research designed to get to the root of behaviors, so 4 00:00:10,658 --> 00:00:14,990 you can anticipate how people would use your product and why. 5 00:00:14,990 --> 00:00:19,270 However, if you want to get into the data representing how people are using your 6 00:00:19,270 --> 00:00:23,750 product today, that's where quantitative testing comes in. 7 00:00:23,750 --> 00:00:26,334 To kick off our quantitative method deep dive, 8 00:00:26,334 --> 00:00:29,379 let's discuss A/B testing starting with an example. 9 00:00:30,441 --> 00:00:34,880 Ever notice that your Facebook feed has a different design than before? 10 00:00:34,880 --> 00:00:37,290 But when you mention it to your other friends on Facebook, 11 00:00:37,290 --> 00:00:40,150 they don't know what you're talking about. 12 00:00:40,150 --> 00:00:44,230 That's likely because you're part of an A/B test that Facebook is running. 13 00:00:44,230 --> 00:00:47,590 They are evaluating if the new design you're seeing 14 00:00:47,590 --> 00:00:50,310 performs better than the original. 15 00:00:50,310 --> 00:00:53,669 They may want to know if you'd spend more time on Facebook or 16 00:00:53,669 --> 00:00:58,412 if you're more likely to click on an ad when using this new feed design. 17 00:00:58,412 --> 00:01:02,863 During an A/B test, one group of visitors to your website sees the specific version 18 00:01:02,863 --> 00:01:06,560 of it, while another group sees another version. 19 00:01:06,560 --> 00:01:11,393 At the end of the study, you will need to determine which of the website versions 20 00:01:11,393 --> 00:01:14,352 performed better based on a hypothesis such as, 21 00:01:14,352 --> 00:01:18,552 version B will receive more ad clicks than version A. 22 00:01:18,552 --> 00:01:24,273 So how do you know when to run an A/B test, and how to get started running one? 23 00:01:24,273 --> 00:01:29,440 First, you may notice something is off in your analytics data. 24 00:01:29,440 --> 00:01:34,540 Perhaps visitors are abandoning your site or app before reaching their goals. 25 00:01:34,540 --> 00:01:35,843 In our Amazon example, 26 00:01:35,843 --> 00:01:40,080 this was when people were adding air conditioners to their shopping carts but 27 00:01:40,080 --> 00:01:44,410 a high number of them were leaving before making a purchase. 28 00:01:44,410 --> 00:01:47,670 Second, find out why. 29 00:01:47,670 --> 00:01:51,320 One of the ways to do this is something we've already discussed, 30 00:01:51,320 --> 00:01:53,100 a usability study. 31 00:01:53,100 --> 00:01:57,050 This may lead you to discover what is throwing the users off track. 32 00:01:57,050 --> 00:01:59,690 Perhaps it's the need for more product information or 33 00:01:59,690 --> 00:02:01,670 maybe the checkout button is hard to find. 34 00:02:02,740 --> 00:02:05,870 Third, propose a potential solution. 35 00:02:05,870 --> 00:02:08,440 What can address the problem you've identified? 36 00:02:08,440 --> 00:02:10,970 Perhaps it's adding more product information or 37 00:02:10,970 --> 00:02:13,440 maybe it's moving the checkout button. 38 00:02:13,440 --> 00:02:17,370 Propose one design solution that you can test against the current site and 39 00:02:17,370 --> 00:02:18,730 an A/B experiment. 40 00:02:19,810 --> 00:02:22,840 If you follow the steps we just discussed, 41 00:02:22,840 --> 00:02:26,230 you may be ready to start running an A/B test. 42 00:02:26,230 --> 00:02:29,160 Let's take a look at the technical steps involved and 43 00:02:29,160 --> 00:02:32,960 how to create a test that can produce statistically significant results. 44 00:02:34,080 --> 00:02:39,176 When we talked about testing the current site versus a potential design solution, 45 00:02:39,176 --> 00:02:43,543 this was in reference to the control versus experiment. 46 00:02:43,543 --> 00:02:48,890 The control group is a group of people who will see the original website design. 47 00:02:48,890 --> 00:02:52,410 It is used as the baseline for your experiment. 48 00:02:52,410 --> 00:02:55,640 The new design should be shown to the experimental group 49 00:02:55,640 --> 00:02:57,590 to see if it outperforms the control. 50 00:02:58,790 --> 00:03:02,132 If you want to have additional experimental groups, that would be 51 00:03:02,132 --> 00:03:06,438 referred to as multivariate testing, but we won't be going into that in this class. 52 00:03:07,649 --> 00:03:12,153 For a website or app that receives few visitors, you should plan on assigning 53 00:03:12,153 --> 00:03:16,460 roughly half the visitors into the control and half into the experiment. 54 00:03:17,460 --> 00:03:21,200 If you have many visitors like in the Facebook example, 55 00:03:21,200 --> 00:03:25,500 you may need to sign as little as 1% or even less. 56 00:03:25,500 --> 00:03:27,302 Simply because even at 1%, 57 00:03:27,302 --> 00:03:32,297 you will quickly have enough volume to draw meaningful conclusions. 58 00:03:32,297 --> 00:03:36,778 The design alternatives that you're testing must be limited to contain 59 00:03:36,778 --> 00:03:39,360 specific isolated changes. 60 00:03:39,360 --> 00:03:43,804 Examples include, the color of the critical button, 61 00:03:43,804 --> 00:03:50,380 placement of a specific element, text variations, or perhaps a header image. 62 00:03:51,595 --> 00:03:56,102 This is important so at the end of the study you can say, with confidence, 63 00:03:56,102 --> 00:04:00,180 what exactly caused the difference in performance. 64 00:04:00,180 --> 00:04:06,051 For example, was it the color, the placement, the text, or the image. 65 00:04:06,051 --> 00:04:10,769 Even small, iterative changes matter, and can add up to large 66 00:04:10,769 --> 00:04:15,854 changes in the total revenue or lead generation from your product. 67 00:04:15,854 --> 00:04:20,520 A/B testing will compare two versions of a product in order to find which 68 00:04:20,520 --> 00:04:23,972 one is better at achieving a higher conversion rate. 69 00:04:25,077 --> 00:04:27,238 What is a conversion rate? 70 00:04:27,238 --> 00:04:31,753 It is the rate at which visitors to your site get to a particular goal that you 71 00:04:31,753 --> 00:04:33,380 have chosen. 72 00:04:33,380 --> 00:04:38,172 Examples of this include, making a purchase, signing up for 73 00:04:38,172 --> 00:04:41,684 a service, or subscribing to a newsletter. 74 00:04:43,148 --> 00:04:47,355 How long should one of these studies run for? 75 00:04:47,355 --> 00:04:52,236 Unfortunately, there's no single answer since much of it will depend on how 76 00:04:52,236 --> 00:04:54,990 much traffic your site gets. 77 00:04:54,990 --> 00:04:57,960 Start with how big you want your sample size to be, and 78 00:04:57,960 --> 00:05:00,870 stop your study when you've reached that size. 79 00:05:00,870 --> 00:05:05,100 The sample size can be determined by the size of the conversion improvement 80 00:05:05,100 --> 00:05:09,790 you're looking for and the confidence you'd like to have in that answer. 81 00:05:09,790 --> 00:05:11,644 I've provided a link to a guideline for 82 00:05:11,644 --> 00:05:15,547 helping you determine the sample size in the teacher's notes. 83 00:05:15,547 --> 00:05:20,200 If your site traffic or user behavior varies according to the day of the week, 84 00:05:20,200 --> 00:05:23,120 extend your test to about two weeks. 85 00:05:23,120 --> 00:05:26,950 That way, you test each day of the week an equal amount of time. 86 00:05:26,950 --> 00:05:28,730 It should also help account for 87 00:05:28,730 --> 00:05:32,650 unique spikes do to things like in newsletter coming out. 88 00:05:32,650 --> 00:05:37,442 A holiday, the consumers payday, or people just looking around your site, but 89 00:05:37,442 --> 00:05:40,836 coming back much later to complete the actual purchase. 90 00:05:41,856 --> 00:05:45,214 Once you've understood the basics behind A/B testing, 91 00:05:45,214 --> 00:05:48,240 implementation should be the easy part. 92 00:05:48,240 --> 00:05:50,755 There are many tools out there to help you. 93 00:05:50,755 --> 00:05:53,194 Just make sure you know what you want to test and 94 00:05:53,194 --> 00:05:56,330 what success metrics you're going to compare. 95 00:05:56,330 --> 00:05:59,780 Purchases, sign ups, subscriptions, or something else.