1 00:00:00,890 --> 00:00:05,540 In the previous video, we talked about why carrying out experiments are necessary. 2 00:00:05,540 --> 00:00:07,748 But what exactly do we mean by an experiment? 3 00:00:07,748 --> 00:00:11,486 The Lean Startup movement, developed by Eric Ries, 4 00:00:11,486 --> 00:00:16,490 introduced the concept of the build, measure, learn feedback loop. 5 00:00:16,490 --> 00:00:21,180 Each experiment we've been talking about will consist of a single turn [SOUND] of 6 00:00:21,180 --> 00:00:21,813 the loop. 7 00:00:21,813 --> 00:00:24,085 The first part of the loop is the build stage. 8 00:00:24,085 --> 00:00:25,830 [SOUND] In the build stage, 9 00:00:25,830 --> 00:00:29,963 we start by asking what metric would confirm our hypothesis. 10 00:00:29,963 --> 00:00:32,604 The metric is important because it determines how we 11 00:00:32,604 --> 00:00:35,030 move forward with our model. 12 00:00:35,030 --> 00:00:38,600 Be careful about picking the right metric and not one that inflates your 13 00:00:38,600 --> 00:00:42,720 data to confirm your desired assumptions rather than the truth. 14 00:00:42,720 --> 00:00:44,225 [SOUND] Once we have a metric, 15 00:00:44,225 --> 00:00:47,056 we design the experiment to gain data on that metric. 16 00:00:47,056 --> 00:00:52,312 We do so by [SOUND] developing our MVP, or minimum viable product. 17 00:00:52,312 --> 00:00:55,350 The MVP is a version of our product that enables us to 18 00:00:55,350 --> 00:00:59,307 complete a single iteration of the build, measure, learn loop, 19 00:00:59,307 --> 00:01:04,450 with a minimum amount of effort and the least amount of development time. 20 00:01:04,450 --> 00:01:06,794 This is a vague guideline for sure, but 21 00:01:06,794 --> 00:01:11,706 you get a better sense of what that means when taking the experiment into account. 22 00:01:11,706 --> 00:01:14,724 One of the first few experiments we need to conduct is to 23 00:01:14,724 --> 00:01:18,790 measure the hypothesis that our product fits the market. 24 00:01:18,790 --> 00:01:21,910 This is a assuming we've tested the assumption that the market exists 25 00:01:21,910 --> 00:01:23,360 to begin with. 26 00:01:23,360 --> 00:01:26,360 In the build stage of this experiment, all we need to 27 00:01:26,360 --> 00:01:31,250 show a customer is some prototype that can convey the bare bones of our product. 28 00:01:31,250 --> 00:01:33,940 You might think, as most people do, that at this point, 29 00:01:33,940 --> 00:01:35,520 you need to build a product. 30 00:01:35,520 --> 00:01:37,878 But no, you don't have to. 31 00:01:37,878 --> 00:01:41,439 Remember, [SOUND] an MVP is a version of our product that enables us 32 00:01:41,439 --> 00:01:44,937 to complete a single turn around the loop with minimum effort and 33 00:01:44,937 --> 00:01:47,092 the least amount of development time. 34 00:01:47,092 --> 00:01:51,444 We can get those same insights by using wireframes, slide decks, or 35 00:01:51,444 --> 00:01:53,550 even basic sketches. 36 00:01:53,550 --> 00:01:56,380 If you put down a working app in front of someone, 37 00:01:56,380 --> 00:01:59,500 they can tend to get tied up in features and design. 38 00:01:59,500 --> 00:02:04,390 By using a low fidelity MVP, something quick like wireframes, we can get 39 00:02:04,390 --> 00:02:08,690 those insights relatively quickly and then move on to the measure and learn stages. 40 00:02:09,750 --> 00:02:13,379 Regardless of the type of experiment, once you have your MVP, 41 00:02:13,379 --> 00:02:16,740 you invite your customers to use it and give you feedback. 42 00:02:16,740 --> 00:02:18,651 [SOUND] This is the measure phase. 43 00:02:18,651 --> 00:02:23,830 So the first step is to enter the build phase as quickly as possible with an MVP. 44 00:02:23,830 --> 00:02:27,650 Once the MVP is done, you move on to the measure phase. 45 00:02:27,650 --> 00:02:32,011 In the measure phase, you [SOUND] analyze whether your product development efforts 46 00:02:32,011 --> 00:02:35,453 in the build phase actually translated to meaningful progress. 47 00:02:35,453 --> 00:02:38,540 This is why the metric is so important. 48 00:02:38,540 --> 00:02:41,590 If you pass the test, our assumption is validated. 49 00:02:41,590 --> 00:02:45,460 The metrics you choose depends on the stage you are in the company and 50 00:02:45,460 --> 00:02:47,000 the type of experiment. 51 00:02:47,000 --> 00:02:51,000 They can range from things like number of meetings set up with potential customers 52 00:02:51,000 --> 00:02:55,535 for sales calls to cost per acquisition, monthly occurring revenue and so on. 53 00:02:55,535 --> 00:02:59,170 [SOUND] After the measure phase, it's the learn [SOUND] phase. 54 00:02:59,170 --> 00:03:02,381 Take the insights that you have gained from this experiment and 55 00:03:02,381 --> 00:03:05,160 apply that to your product or service. 56 00:03:05,160 --> 00:03:09,657 If the test fails, you discard those assumptions and keep experimenting. 57 00:03:09,657 --> 00:03:13,102 It is important to note that we're not just building lots of MVPs, 58 00:03:13,102 --> 00:03:15,900 running an experiment and discarding it. 59 00:03:15,900 --> 00:03:20,780 No, our product is basically the evolution of these MVPs. 60 00:03:20,780 --> 00:03:22,793 Let's say we have a website [SOUND] up and 61 00:03:22,793 --> 00:03:25,121 we need to increase our activation efforts. 62 00:03:25,121 --> 00:03:29,377 So we conduct an experiment to test the effectiveness of our call [SOUND] to 63 00:03:29,377 --> 00:03:30,369 action button. 64 00:03:30,369 --> 00:03:34,180 Our metric we're measuring here is an account creation. 65 00:03:34,180 --> 00:03:36,338 We built several [SOUND] MVPs. 66 00:03:36,338 --> 00:03:37,020 Oh, that's right. 67 00:03:37,020 --> 00:03:38,950 That's another important point. 68 00:03:38,950 --> 00:03:42,720 A test does not have to be restricted to one MVP. 69 00:03:42,720 --> 00:03:44,000 We can create several and 70 00:03:44,000 --> 00:03:48,070 test each version among a different segment of our customer base. 71 00:03:48,070 --> 00:03:52,570 In our case, we create several MVPs with different [SOUND] button styles, copy, 72 00:03:52,570 --> 00:03:53,641 and positioning. 73 00:03:53,641 --> 00:03:56,930 [SOUND] The MVP that passes [SOUND] the test then becomes the next 74 00:03:56,930 --> 00:03:58,746 iteration [SOUND] of the product. 75 00:03:58,746 --> 00:04:02,300 In this way, we only add to the product those features that pass our 76 00:04:02,300 --> 00:04:05,470 experiments and confirm our assumptions. 77 00:04:05,470 --> 00:04:09,100 Everything else that adds no value is discarded. 78 00:04:09,100 --> 00:04:10,130 If you work on a product or 79 00:04:10,130 --> 00:04:14,210 service like this when you start out, by the time you launch, you will have 80 00:04:14,210 --> 00:04:18,000 a product that has customer-tested features that you know will succeed. 81 00:04:19,020 --> 00:04:22,550 When we carried out the exercise in the previous stage and built our business 82 00:04:22,550 --> 00:04:27,560 model, we laid out a lot of assumptions to make our business plan work. 83 00:04:27,560 --> 00:04:31,500 Our goal with these experiments is to test these assumptions as quickly as 84 00:04:31,500 --> 00:04:34,550 possible so we get rid of all the wrong ones. 85 00:04:34,550 --> 00:04:38,680 So for example, my first assumption was that a market existed for 86 00:04:38,680 --> 00:04:42,740 project management software focused around really small groups. 87 00:04:42,740 --> 00:04:45,690 To experiment this, I first need to develop my metric. 88 00:04:45,690 --> 00:04:48,170 Now, that's fairly simple in this case. 89 00:04:48,170 --> 00:04:53,700 I'm going to measure yes or no responses to a survey asking the very same question. 90 00:04:53,700 --> 00:04:56,520 The pass fail bar that I'm setting here is that I want at 91 00:04:56,520 --> 00:05:00,100 least a 70% positive response rate. 92 00:05:00,100 --> 00:05:00,795 Why so high? 93 00:05:00,795 --> 00:05:04,790 Remember, we said that with our initial assumptions like these, 94 00:05:04,790 --> 00:05:09,210 we want a really high pass rate because each successive experiment is going to 95 00:05:09,210 --> 00:05:10,654 further whittle down this number. 96 00:05:10,654 --> 00:05:14,795 That if only 20% of my survey respondents indicated yes, 97 00:05:14,795 --> 00:05:18,260 then the number that will actually pay me is much smaller. 98 00:05:18,260 --> 00:05:20,109 This is not a scalable business model. 99 00:05:21,290 --> 00:05:24,280 My MVP in this case is a simple survey and 100 00:05:24,280 --> 00:05:27,942 I will carry it out both in person and using an online survey. 101 00:05:27,942 --> 00:05:32,650 The in-person survey will help me get some feedback right from the beginning, 102 00:05:32,650 --> 00:05:37,900 while the online survey will help me reach a much larger number of respondents. 103 00:05:37,900 --> 00:05:42,110 Similarly, I can conduct experiments for all my assumptions, from things like 104 00:05:42,110 --> 00:05:47,230 landing pages using AB tests, product testing using focus groups, and so on.