Experiments

In the previous video, we talked about why carrying out experiments are necessary. 0:00 But what exactly do we mean by an experiment? 0:05 The Lean Startup movement, developed by Eric Ries, 0:07 introduced the concept of the build, measure, learn feedback loop. 0:11 Each experiment we've been talking about will consist of a single turn [SOUND] of 0:16 the loop. 0:21 The first part of the loop is the build stage. 0:21 [SOUND] In the build stage, 0:24 we start by asking what metric would confirm our hypothesis. 0:25 The metric is important because it determines how we 0:29 move forward with our model. 0:32 Be careful about picking the right metric and not one that inflates your 0:35 data to confirm your desired assumptions rather than the truth. 0:38 [SOUND] Once we have a metric, 0:42 we design the experiment to gain data on that metric. 0:44 We do so by [SOUND] developing our MVP, or minimum viable product. 0:47 The MVP is a version of our product that enables us to 0:52 complete a single iteration of the build, measure, learn loop, 0:55 with a minimum amount of effort and the least amount of development time. 0:59 This is a vague guideline for sure, but 1:04 you get a better sense of what that means when taking the experiment into account. 1:06 One of the first few experiments we need to conduct is to 1:11 measure the hypothesis that our product fits the market. 1:14 This is a assuming we've tested the assumption that the market exists 1:18 to begin with. 1:21 In the build stage of this experiment, all we need to 1:23 show a customer is some prototype that can convey the bare bones of our product. 1:26 You might think, as most people do, that at this point, 1:31 you need to build a product. 1:33 But no, you don't have to. 1:35 Remember, [SOUND] an MVP is a version of our product that enables us 1:37 to complete a single turn around the loop with minimum effort and 1:41 the least amount of development time. 1:44 We can get those same insights by using wireframes, slide decks, or 1:47 even basic sketches. 1:51 If you put down a working app in front of someone, 1:53 they can tend to get tied up in features and design. 1:56 By using a low fidelity MVP, something quick like wireframes, we can get 1:59 those insights relatively quickly and then move on to the measure and learn stages. 2:04 Regardless of the type of experiment, once you have your MVP, 2:09 you invite your customers to use it and give you feedback. 2:13 [SOUND] This is the measure phase. 2:16 So the first step is to enter the build phase as quickly as possible with an MVP. 2:18 Once the MVP is done, you move on to the measure phase. 2:23 In the measure phase, you [SOUND] analyze whether your product development efforts 2:27 in the build phase actually translated to meaningful progress. 2:32 This is why the metric is so important. 2:35 If you pass the test, our assumption is validated. 2:38 The metrics you choose depends on the stage you are in the company and 2:41 the type of experiment. 2:45 They can range from things like number of meetings set up with potential customers 2:47 for sales calls to cost per acquisition, monthly occurring revenue and so on. 2:51 [SOUND] After the measure phase, it's the learn [SOUND] phase. 2:55 Take the insights that you have gained from this experiment and 2:59 apply that to your product or service. 3:02 If the test fails, you discard those assumptions and keep experimenting. 3:05 It is important to note that we're not just building lots of MVPs, 3:09 running an experiment and discarding it. 3:13 No, our product is basically the evolution of these MVPs. 3:15 Let's say we have a website [SOUND] up and 3:20 we need to increase our activation efforts. 3:22 So we conduct an experiment to test the effectiveness of our call [SOUND] to 3:25 action button. 3:29 Our metric we're measuring here is an account creation. 3:30 We built several [SOUND] MVPs. 3:34 Oh, that's right. 3:36 That's another important point. 3:37 A test does not have to be restricted to one MVP. 3:38 We can create several and 3:42 test each version among a different segment of our customer base. 3:44 In our case, we create several MVPs with different [SOUND] button styles, copy, 3:48 and positioning. 3:52 [SOUND] The MVP that passes [SOUND] the test then becomes the next 3:53 iteration [SOUND] of the product. 3:56 In this way, we only add to the product those features that pass our 3:58 experiments and confirm our assumptions. 4:02 Everything else that adds no value is discarded. 4:05 If you work on a product or 4:09 service like this when you start out, by the time you launch, you will have 4:10 a product that has customer-tested features that you know will succeed. 4:14 When we carried out the exercise in the previous stage and built our business 4:19 model, we laid out a lot of assumptions to make our business plan work. 4:22 Our goal with these experiments is to test these assumptions as quickly as 4:27 possible so we get rid of all the wrong ones. 4:31 So for example, my first assumption was that a market existed for 4:34 project management software focused around really small groups. 4:38 To experiment this, I first need to develop my metric. 4:42 Now, that's fairly simple in this case. 4:45 I'm going to measure yes or no responses to a survey asking the very same question. 4:48 The pass fail bar that I'm setting here is that I want at 4:53 least a 70% positive response rate. 4:56 Why so high? 5:00 Remember, we said that with our initial assumptions like these, 5:00 we want a really high pass rate because each successive experiment is going to 5:04 further whittle down this number. 5:09 That if only 20% of my survey respondents indicated yes, 5:10 then the number that will actually pay me is much smaller. 5:14 This is not a scalable business model. 5:18 My MVP in this case is a simple survey and 5:21 I will carry it out both in person and using an online survey. 5:24 The in-person survey will help me get some feedback right from the beginning, 5:27 while the online survey will help me reach a much larger number of respondents. 5:32 Similarly, I can conduct experiments for all my assumptions, from things like 5:37 landing pages using AB tests, product testing using focus groups, and so on. 5:42