Bummer! This is just a preview. You need to be signed in with a Pro account to view the entire video.
Adding Simplicity40:50 with Cap Watkins
It’s always exciting to work at a company that’s growing and flourishing. However, along with success, you’ll inevitably be thrown some new problems as well. For one, as your organization grows in complexity, it’s tempting to allow your product development process to do the same. Unfortunately, a more complex process can easily result in slower launches, less transparency and unclear overlaps in teammates’ roles and responsibilities.
[BLANK_AUDIO] 0:07 [SOUND] Hi. 0:12 So it's funny, I, it was hush-hush, hush-hush stuff at Amazon until this week. 0:13 I am so sorry about that phone, everyone. 0:19 That is partially on me. 0:23 So I always, I like to start with this slide every 0:26 time I give a talk because I find it pretty inspirational. 0:28 Like, if this cat can ride bacon through space, then so can everyone, right? 0:33 I usually go, that usually goes better, so, yeah. 0:39 [LAUGH] Hi. 0:42 So I'm Cap. 0:44 I'm a product design manager at Etsy. 0:45 Does everybody here, does anyone here not know what Etsy is? 0:47 Google it. 0:52 No, it's okay. 0:53 So Etsy is actually it was a startup founded in Brooklyn in 2005. 0:57 We just had our ninth birthday yesterday. 1:01 And ever since those humble origins, the community has been 1:04 selling a variety of things like handmade and craft manufactured products, 1:08 vintage goods 1:14 and supplies for making more stuff. 1:18 That's kind of a nice circle. 1:21 Etsy has grown to over a million active sellers in 200 countries. 1:25 And today, there are over 20 million items available for purchase on Etsy. 1:31 And that's growing pretty rapidly. 1:35 So actually in 2013, our community sold approximately $1.4 billion worth of goods. 1:36 It's pretty big. 1:42 Did not expect that. 1:45 actually, I started Etsy, and I saw, it's 2012, and I saw that graph, 1:47 and I was like, I should not 1:51 be responsible for anything remotely resembling that graph. 1:52 We also serve up about 1.4 billion page views a month, 1:58 which is a lot. 2:02 And we're also growing as a company, so today 2:04 nearly 500, actually 600 people work in our offices 2:07 in Brooklyn, Hudson, New York, San Francisco, Berlin, Dublin, 2:10 and a lot of other offices around the world. 2:14 And I'm gonna throw this video on. 2:17 This is, there's not a lot of sound. 2:19 There is sound, but it's not very good, so I'm going to narrate this. 2:20 But this is just to show you what it's kind of like at Etsy. 2:23 Everybody always asks about the office. 2:25 So this was the opening for our brand studio. 2:27 We had to like create a new part of the office, and so we 2:30 were just taking video of some of the collateral we've created over the years. 2:34 This is, these patches at Etsy, everybody does what's called boot camping where 2:36 you go work with a team that's not yours for about six weeks. 2:41 And at the end, you get one of these cool badges. 2:44 And the idea is to collect them all and then you know all of Etsy. 2:46 So like really nice stuff. 2:52 We had a holiday shop in Soho a couple of 2:53 years ago for two weeks that was a pop-up shop. 2:55 That was pretty neat. 2:58 When you come to Etsy, you have to sign one of those, Howdy, My Name is stickers. 3:00 Make all sorts of stuff, it's mostly internal [INAUDIBLE]. 3:06 I have one of those cool tote bags; we just makes, it's fun to have a 3:14 brand as it is to make stuff and then we get to have it, which is great. 3:20 Some Etsy folks, everyone's pretty stoked to work 3:24 there um; yeah, hold on, it gets real good. 3:28 There, that's nice. 3:35 So that's Matt [LAUGH]. 3:37 He's actually the sixth employee at Etsy. 3:37 So he's been there nine years. 3:39 He heads up our values and impact team. 3:41 And 3:43 this is where it gets amazing. 3:47 So my coworker who was taking this video didn't realize that when you turn it, it 3:48 doesn't it doesn't change the [LAUGH] yeah, so anyway. 3:52 Like I said, we're growing. 4:01 So to give you some perspective into 2010, we had two product designers with Etsy. 4:03 In 2011, it went up to about six people, and at 4:08 the end of 2012 when I started, we had 12 product designers. 4:11 We ended last year with 19, and now we're at 25 so far in 2014, which 4:17 might seem like a lot, but like I said, Etsy's nearly 600 people. 4:22 So it kind of makes a little bit of sense. 4:27 Engineering has about 180 or so folks, so this helps keep us balanced out. 4:28 And as I've been growing, I've been trying to 4:34 figure out, like how does a, like, almost 30 4:36 person design team think about building for a constantly 4:39 changing product working with that many engineers and product managers. 4:41 And we realized we had to find a common way of working together, 4:45 of working across like, all of our teams and across the entire organization. 4:48 And we never really formalized that process but 4:52 I've been thinking about it a lot lately, 4:57 and I realize by accident we kind of 4:59 came up with our own principles of product design. 5:01 The first of which is most importantly, be a product designer. 5:05 Maybe not that kind of product. 5:10 Actually feel like this guy a lot of the time, managing designers at Etsy. 5:13 Like, I'm like, what the hell is that? 5:17 But the key word being product. 5:21 So at Etsy, are designers are generalists, which 5:23 means they do user experience design, visual design, and 5:25 their own front end implementation, so they're responsible for 5:28 the entire stack and that's awesome and pretty rare. 5:30 But we expect more than that even. 5:34 We expect our designers to be a part of the entire product development process. 5:36 From like inception all the way through to like, data analysis and iteration. 5:39 And the way we do that is so most companies 5:45 that I've ever worked at, like, the product development cycle looks 5:48 a lot like this, where you do some up front stuff 5:51 in that beginning gray area, then you design a little bit. 5:55 Then you develop some stuff, and then you measure. 5:57 And this is a really long process, maybe like four or five months, and at 6:00 the end, you all get fired because it's been a long time, and you're wrong. 6:04 And the crappy part about this that designs 6:11 like a very finite part of this process. 6:12 I would much prefer something that looks like this, and this is what we 6:15 try to do Etsy, where like, it's just kind of there the entire time. 6:19 Because the truth is that design isn't just 6:22 about like executing it's about what we build. 6:24 When we build it strategically thinking about like what's 6:28 coming in what order and when we should do things. 6:31 And most importantly, like why we're doing something. 6:34 We want to inject design and like user 6:37 empathy into every part of the product process. 6:38 So, yeah, so how do we encourage that? 6:42 First of all, we try to get designers 6:44 involved super early, before doing any designer engineering work. 6:46 So when I started at Etsy, I brought with 6:51 me some stuff from Amazon for better or worse. 6:53 And one of the things was to, the beginning 6:57 of every design process should really be writing stuff. 7:00 And so we get people to write down problems and tenets 7:05 for each project they start before they've done any work at all. 7:08 And this is like, something that I wrote down from my very first project at Etsy. 7:12 I was work, working on redesigning our item review system. 7:15 And just like kind of writing down the problems 7:19 and the tenets like, is a pretty complicated product. 7:21 And I shared this with my product 7:23 manager and engineering partners and like, we actually 7:24 argued about these for like two, discussed, 7:27 sorry [LAUGH] these for a couple of days. 7:29 And kind of honed them down and like, it was like, oh, I don't 7:32 think this is a real problem, or like I think we should add this tenet. 7:35 And at the end, what we wound up with was like a 7:39 shared, perspective that we all agreed on and could move forward with. 7:41 And like, what's really nice about writing stuff down is that it take, this took 7:45 like, I don't know, 10 minutes for me to jot down and like send around. 7:48 And it helped raise a lot of disagreement 7:52 on alignment before I dedicate any time to work. 7:55 It also has the added benefit of getting everybody together in the design process. 7:59 All of a sudden, everyone is designing together cuz that's what this is. 8:03 Another way we do this is through small teams. 8:07 So even for big projects, that item reviews 8:09 project took like in total about eight months 8:11 with small shipments in between, but there were 8:15 only three or four of us on the project. 8:16 It was me, one engineer and a PM and a product marketing person. 8:18 And what's nice about the small teams is it forces collaboration. 8:22 So towards the end of the project, things were getting really tight and really 8:26 fast, and we realized that we needed to add a feature like, basically overnight. 8:29 So this is not, I did not draw this. 8:34 Actually, the engineer I was working with drew this. 8:37 We sat down at a table, and we knew we had to do it. 8:39 And like we sketched this out in like five or ten minutes. 8:41 And we literally, it's hard to tell what this is looking at it. 8:44 But we shipped exactly this. 8:48 This was our spec, and we literally built off of that. 8:50 And about 24 hours later, we shipped the product. 8:52 yeah, that's crazy. 8:56 The other thing we do is ask questions a lot. 8:59 So are we solving the right problems? 9:00 Does this product still even make sense to do? 9:03 so, an example of that is we were 9:06 working on, this is our seller experience currently. 9:07 Does anyone run a shop on Etsy? 9:10 Yes. 9:13 Awesome. 9:14 That's awesome; there's always one. 9:15 Actually, did you know like if you live in the United States, you 9:16 have 99% chance of living in the same zip code as an Etsy seller. 9:18 Boom. 9:22 [LAUGH] 9:22 It's really awkward when no one raises their hand during that, 9:26 I'm like come on So this is, this is our seller experience. 9:28 And this left nav is, like, it's, so, Etsy's nine years old. 9:32 We haven't really redesigned much of the site like, in a big holistic way. 9:35 We're starting to do that now, but like, this is what people deal with now. 9:40 This list of links on the left is the nav, 9:42 and it's like 28 or 30 links long, which totally sucks. 9:44 Like, everybody at Etsy knows it and someone was trying to 9:48 figure out how to fix this, and so they were just 9:53 trying to roll it up and like what if we just 9:55 collapsed the whole thing, and like, it was just real messy. 9:57 And finally, we stopped, and we're like, hey, should we be doing this? 10:01 Like, this isn't, this should, is like, not going to work. 10:04 Just to collapse it down. 10:07 This requires like a complete rethinking of like, how 10:07 all this stuff fits together and how we're presenting it. 10:10 And so we stopped that project and like ditched it until we could 10:13 like dictate the time and resources to like really do the project right. 10:15 Which I think is really hard to do but it only 10:20 happened because we kept asking ourselves like, should we keep doing this. 10:22 So yeah, so like, with the holistic approach in mind like 10:28 the second product design principle that Etsy uses, you know, no silos. 10:30 So as a small design team like, we're at a distinct advantage in a company this big. 10:35 It's way easier to see across 25 people than it is to see across, like 600 or 180. 10:40 So individually, while we may be in our own product teams we 10:46 actually meet as a design team, like a few times a week. 10:50 To like, essentially, trade information, and like show work to each other, 10:53 and start to like connect those 10:57 dots across this like, constantly growing organization. 10:58 again, the benefit of small team, and that won't last forever. 11:02 But it's working pretty well for us right now. 11:04 So not only are we able to easily like look across 11:07 all of the Etsy product, but we kind of have to. 11:10 As designers, it's our job to think about the whole product. 11:14 An example of that, this is a project I worked on, so we as part 11:19 of the item reviews project, we wanted to put them onto, into the Purchases page. 11:21 So like if you purchase something, and you came back to look 11:26 at your receipt, like you should be able to review if from here. 11:29 That just was how it should be. 11:31 But the problem is that like this is pretty information 11:33 dense and just adding something to it feels pretty bad. 11:36 Not to mention, it doesn't do a good job of like telling buyers what they need to 11:39 know, which is like, where the hell's my stuff and who do I talk to about it. 11:43 But the problem of redesigning this is, like literally, this page 11:49 is the culmination of every single, like, feature ever invented on Etsy. 11:51 So like gift cards, shipping labels, payments, wedding registry purchases, 11:57 digital downloads, custom orders, all have logic in this page. 12:01 And all, like, have different, like, ways they show up. 12:04 So what I wound up doing, cuz I didn't build those features is I 12:09 went and was able to talk to every designer who'd ever touched that page. 12:11 Found out exactly, like, what they needed there, why they needed it. 12:15 Even found out when like, they 12:19 implemented something incorrectly which was pretty fun. 12:20 And yeah, so we wound up going from this to something like 12:26 this, which is a lot clearer, like, way fewer features showing up. 12:29 Using design patterns from the more modern parts of our site, and 12:35 this also allows people to focus again on those two things they 12:40 care most about, like tracking your package is like a gigantic button, 12:42 and then like contacting the shop is also like a gigantic button. 12:45 Those are the two things people really come here to do. 12:48 And 12:52 the thing is like if I hadn't spoken to 12:54 everybody, and if I'd just like stayed in my, like, 12:55 little siloed team, I would have assumed a lot 12:59 about how this stuff works and why it was there. 13:01 And I would have been wrong over and over and over. 13:03 And I would have wasted a lot of my 13:06 time and my team's time and so communication across our 13:07 teams is like, super imperative, and we encourage all 13:11 of our designers to be super transparent with each other. 13:14 Which leads me to the next principle, which is share. 13:18 so, yeah, that's dangerous. 13:23 So just like we don't wanna silo our 13:27 projects we also don't wanna silo the design process. 13:28 And designers at Etsy work in a lot of different ways. 13:32 Whether just sketching I actually sketched this 13:35 just for this presentation, that's not real. 13:38 People do this I like; I just decided I needed a nice sketch, not 13:41 one of my chicken scratch sketches; I probably spent like 20 minutes on this. 13:45 Just for you. 13:49 Whether it's sketching or like Photoshop or Sketch, we have designers who have 13:52 work primarily internal that goes straight to code and work in them, which is nuts. 13:58 Assholes. 14:04 I'm just jealous. 14:07 So we don't actually care what your process looks like 14:10 at Etsy as a designer, like as long as it's transparent. 14:12 Because transparency helps us socialize and cross-pollinate ideas it 14:15 creates more serendipity across the projects, and it prevents surprises. 14:19 Like the worst thing that ever happened, all the pain I've ever felt at any 14:22 company, is because someone got surprised, or 14:26 like I surprised someone or was surprised myself. 14:28 It also, like when you're transparent, it shows you value each other's feedback. 14:31 And so we do this in a few ways. 14:37 So we use base camp pretty religiously. 14:39 This is like problems and tenets. 14:43 We also have if this feature, and we're just improving it 14:45 through the features strengths we want to hang on to those. 14:47 So apparently, this project had zero strengths. 14:50 The existing features sucked that bad. 14:52 But we use base camp pretty religiously. 14:55 What's really cool is we put the entire product design team on 14:57 every single project so we all get all the updates all the time. 14:59 And every morning there's like, the best part of my day 15:03 is like, getting this one single digest email that base camp sends. 15:06 It has like, everything from the day before. 15:09 You sit there with your coffee for 30 15:12 minutes, roll through that stuff like, dive into things 15:14 that are interesting and then, like start to 15:16 connect people together based on what you're looking at. 15:18 And we do critique in there and like, make suggestions which is pretty nice. 15:21 A little asynchronous. 15:26 We also print stuff out. 15:28 Those four people looking at printed designs are all engineers. 15:31 I was sitting at my desk after we printed this stuff, 15:35 just working, and one of my designers across the room is like 15:36 doing this widely at me and I look, and she goes like 15:38 this and points at them, and I was like, yes, that's awesome 15:42 Yeah, I mean, they could of been saying it sucks, but I don't know. 15:48 whatever, they're looking at it; it's fine So 15:51 all this transparency allows designers to move really quickly. 15:54 So this is actually, we decided, one of our designers 15:59 decided to do an audit of all the emails we send. 16:01 And we decide, we printed them out and decided 16:03 to see how many designers worth of emails we send. 16:05 Turns out two designers. 16:08 Two designers worth of email to buyers. 16:10 That's way too much email. 16:13 But all this transparency allows us to move pretty quickly. 16:15 Which is great, because at Etsy, we have a 16:18 culture of shipping fast and of breaking things pretty quickly. 16:19 Which leads me to the fourth principle, which is celebrating failure. 16:24 I asked the Internet if I should use this; they said no. 16:29 And I did it anyway, so Mad Men, the best. 16:31 So one of my co-workers, we're just gonna hold that for a second. 16:36 One of my co-workers like is always saying that she, 16:40 she says this all the time that perfection is a pursuit. 16:42 Yeah, sorry, that really hurt that guy. 16:46 It was not great. 16:48 That was a great episode though. 16:50 No? 16:52 No Mad Men fans? 16:52 anyway, perfection's a pursuit. 16:54 [LAUGH] So and that's we have a culture of 16:56 like, fast failures and service of longer term wins. 16:59 It's our job as designers to embrace that. 17:02 So like I said, there are basically two approaches to like product development. 17:04 One is a single timeline where like, boom, design, develop, measure, get fired. 17:07 And 17:12 the problem with that is, like, it assumes we're correct. 17:15 And the problem is that we're not. 17:17 We're like, wrong a lot. 17:19 The problem is that like, you have all this space where 17:21 like, you didn't validate your ideas, so like, you're like, doing all 17:23 this work and like, there, you, go off, it just goes 17:26 off into development, and like, you were probably wrong the entire time. 17:30 You don't know whether you're doing a good or bad thing. 17:34 So at Etsy, what we try to do is take that single 17:36 timeline and break it up into like short, like iterative measurable cycles. 17:38 Evenly weighted, like blocks of time. 17:44 It gives us the opportunity to sanity check. 17:46 And the nice thing about that is it allows you to screw up. 17:49 Or learn about your ideas and make them better. 17:53 so, one example of that is The Listing Page. 17:56 So last year, one of the teams at Etsy decided to 17:58 redesign the Listing page, which is like, our item level page. 18:02 It's the highest traffic page on Etsy. 18:06 People get here from like Etsy.com from Google search from like, ads that we 18:07 run, ads that sellers run, Facebook, like Pinterest, like this is the page at Etsy. 18:13 It had never been redesigned; clearly, it had only been added to. 18:18 So like, even, we've even added another Add 18:22 to Cart button at the bottom of the page. 18:23 Double down, that works. 18:26 That's a good design principle, double it. 18:29 [LAUGH] 18:31 The problem also is that, like this doesn't really represent what 18:34 Etsy is, so at Etsy, every quarter, everybody does two, like 18:36 a support rotation, and we go sit for like three or 18:41 four hours with our support team and actually answer customer emails. 18:44 So everybody at Etsy does this. 18:48 And every time I'm on the support rotation I would get these emails. 18:50 And it would just, all it would say would be like, can I get that in blue? 18:54 I have no idea what they're talking about. 18:58 They like, what they had done was come to this page and 19:00 like, oh, that's really cool I wonder if it comes in blue. 19:02 And like instead of realizing that they're buying from a person, and they need 19:05 to hit that like, tiny contact link down there; they wanted to Help, to like 19:07 all through our help flow, to finally get to that form to finally fill 19:11 it out, to send me the email that says, can I get that in blue. 19:15 And I don't even know where they came from. 19:18 Like, it's a huge bummer, so yeah, we decided to maybe redo that. 19:21 And this is Kim. 19:28 She is the designer who like led this project and shortly 19:29 after I married her, thanks, Etsy not because of this project 19:32 even though it did went really well. 19:39 And she told me it wasn't just about better conversion on the 19:41 page, and it was also about like just telling you what Etsy is. 19:44 So this is the first pass she did. 19:47 It took about six weeks to get from like starting all the 19:49 way to like, this first, we were shipping this about six weeks later. 19:52 It's a bit better, there's a subtle callout at the top, the shop name. 19:55 A more dynamic presentation, you can kind of see 19:59 it at the bottom here, it's like the shop owner. 20:01 More structured information in like that, what we call the buy box. 20:04 It's in the top righthand corner. 20:07 It's like materials, and like shipping information. 20:08 I'd never really been present, like when you were making the 20:11 purchasing decision so, we were trying to get that up there. 20:14 And so it feels a lot better than like, 20:18 this guy it starts to feel a little bit better. 20:20 So like I said, it took four to six 20:23 weeks, we're kind of refashioning some stuff, setting ourselves up. 20:25 And so we shipped it, and the result was like, total disaster. 20:29 It actually like, bounce rates went way up, which if you're not 20:36 familiar, like it means, someone hit this page, and then left Etsy. 20:39 Like didn't like stay in Etsy, just totally left. 20:42 And pro, and didn't come back most likely. 20:44 It actually performed worse than the page that 20:47 hadn't been touched in probably like seven years. 20:49 It also turns out to like, so we had taken the item 20:53 description, and kinda rolled it up, cuz they can get really long. 20:54 Turns out that they're really important. 20:57 They actually have, the other problem is there is no max length 20:59 to our descriptions, it's however big it can get in the database. 21:02 Which is something like 12,000 characters. 21:06 And sellers will fill that space, with description information. 21:09 And it turns out buyers need to see that. 21:12 We also we found out later that putting shipping information 21:15 upfront turns people off, they want to see that later. 21:17 I have no idea why, you guys are weird. 21:19 [LAUGH] 21:22 And so I think like, normally if you were in that long development cycle 21:24 and you shipped this and it failed like, it would be pretty hard to take. 21:29 Like you would feel pretty bad, It would be a huge bummer, 21:31 but the secret is like we knew that thing was gonna fail. 21:35 We had actually planned that way so we knew that like, 21:38 we were just kind of setting ourselves like a design, a new 21:41 design baseline to work away from, and still keeping that old control 21:43 alive so we could like make sure we're doing a good thing. 21:47 So from there we started shipping really fast, like, iterations so, 21:50 two weeks later we shipped this, and this even better right. 21:54 There is like shop banner at the top, with more items from the shop. 21:57 Even more items from the shop in the right hand side, 22:00 like really trying to make this a super shoppable like, if 22:02 you don't see what you want here there is more stuff 22:05 for you even more contact info inside at the description right like. 22:07 Making the photo the first thing that you see, 22:12 so this is, like, feeling a little bit better and 22:17 it performed a little bit better than the old one. 22:18 Then we shift this and flipped the page cuz, yeah, fuck it, 22:22 who knows, had no effect, turns out And so after we, like, we did a bunch of 22:28 these and we finally round up with this, this is the final thing that we shipped. 22:36 And yeah, I mean it's pretty, it's pretty great. 22:41 It works out pretty well. 22:44 This big shop banner at the top with the big identity mark, 22:46 feels really nice and tells you this is not Etsy selling you something. 22:48 And like we really worked out what 22:53 information really needed to be in that buy 22:54 box, hid shipping policies because people hate 22:56 that, told you the descriptions get really long. 23:01 And I'm so sorry about this footer, it does not look like this anymore. 23:05 We felt real bad about that. 23:09 So this took about six months, this was actually 23:12 the biggest one we've ever had, conversion one we've ever 23:14 had, in Etsy history, which means sellers are making a 23:16 lot more money than they were before, which is awesome. 23:20 It took about six months, and if we'd 23:24 spent just six months working on that single 23:26 litera, like design that single thing and building 23:27 a single thing, we would have been totally screwed. 23:29 So we benefit a lot by doing these like short, iterative, measurable steps. 23:33 And just to show you like here are only, just a few of the iterations we launched. 23:37 We wound up shipping 21 versions of this in six months. 23:41 And some of these are like so subtly different it's hard to tell, 23:45 we did all of these. 23:50 And we're just shipping, like so, so fast. 23:53 So this secret, right is just accepting that, our designs are not perfect. 23:57 And that really helps us cuz it helps us try 24:01 riskier things, and makes failure not just tolerable but desirable. 24:02 You begin to want to fail because it's not, it's about learning 24:07 and like making your stuff better, and not just like winning right now. 24:10 So yeah in the spirit of total absolute failure, the, the final product design 24:15 principle we have at Etsy is that all of these principles are subject to change. 24:20 Because the truth is that like, we grow up, like as organizations and as teams. 24:25 We change, like, the teams change, our companies change 24:30 like what they need to, like stay stable changes 24:33 All these things that are working for us right 24:37 now, at 25 people are going to totally break. 24:38 We're starting to feel some of it now, like not from 12 24:41 to 25 people in like a year and a half is pretty nuts. 24:44 So, some of the things we used to do just 24:47 aren't working anymore, and we have to be able to like 24:48 essentially iterate just like we would on the product and 24:50 like, be okay with like, whatever we're doing not working anymore. 24:52 The most important thing for us to embrace 24:58 like, that impermanence and to be super flexible. 25:00 And to shift fast and break things into learning. 25:03 So, yeah. 25:07 Figured there'd be time for questions. 25:08 [SOUND]. 25:15 >> two, two questions, kind of quick ones. 25:19 The first is when you started talking about the, redesigning the product 25:23 page you mentioned like, one of our designers decided to do this. 25:27 What's your process for figuring out, like who owns that process for figuring 25:30 our what you're gonna design next and, and when you can do it? 25:35 And then just to give you both of them at once, when 25:38 you are testing something as major as like the, product detail page. 25:41 Where's your threshold when you guys are pumping out, you 25:46 know, doing your multi-varied testing or doing different iterations of it? 25:49 Like when do you know like alright fuck that 25:52 we gotta try something else like this is no good? 25:54 >> Yeah. 25:56 so, the first question it's actually the designer hadn't 25:58 so like her team had decided to like pursue this. 26:00 [COUGH] the way we do this at Etsy actually changed it in the 26:03 last year and we're going to do this for the second time this year. 26:06 So I am sure this will be better, than the first time. 26:09 But what we do now it is very bottoms up so like 26:12 we in October we have this thing called slush from October to December. 26:14 So, our peak times obviously are the holidays, right? 26:19 So try like not to ship experiments between October and December because, 26:24 don't touch it, it's like this is the time, do not break it. 26:27 Our office guys don't sleep for three months, but we so we use that 26:32 time actually to kind of like step back and like, phew, and like look 26:37 back over the last year and the team start to plan together what they 26:40 would like to do in the next year and build business cases around that. 26:43 And essentially that just likes, get presented 26:47 to our like CEO, CFO, CTO and like. 26:50 I, I don't think we've ever been 26:54 told no, because we're all pretty reasonable people, 26:54 and we're not trying to do in totally 26:57 insane things, or things that's aren't very impactful. 26:59 So that's kind of how it works. 27:02 Like designers isn't like this and like the PMs will all 27:05 get together and like, just like brainstorm what they'd like to do. 27:07 Narrow focus and then like, make the cases. 27:09 So then the way we know things aren't working, so we time box stuff. 27:14 So the listing page thing, six months was like, we could have just kept going. 27:19 I mean, like, we probably will at some point just keep going on that, right? 27:22 Cuz there's way more wins than what we got out of, like, even that one great release. 27:27 So so it's a time box, so like, we're like hey, this is gonna get done in like Q2. 27:31 and if it doesn't get done in Q2 we kind of 27:35 have to reevaluate at that point, like, is this worth, like, 27:37 spending a few more months on, or is it work just 27:40 like [SOUND] like, chopping it off, and like not doing it. 27:43 In service of whatever was next, so that's really just calls within the 27:46 team at that point, but yeah that's kinda how we decide to do that. 27:50 Tanking numbers isn't really a problem for 27:54 us, we just shut that down, like start over. 28:01 >> You talked a little bit about working with product managers, and because 28:10 product designers as Etsy, you know, kind of own the whole process. 28:15 How do you, I guess define the difference between the roles of product designers 28:21 and product managers at Etsy, and how do you facilitate the collaboration process? 28:27 >> Sure. 28:32 so, I think just like product design is like, involved in the whole process. 28:33 Like, so are product management and engineering again 28:37 these teams are really small, so it's really 28:41 hard for one person to go off into like, a place and just like, create a thing. 28:43 so, I mean we try just to get everybody involved in the 28:48 entire process, all the way through, and that's, it's just, it's very organic. 28:51 I mean we try to pair designers that like, some designers are 28:56 like more productive than others like they think more productive than other designers 29:00 and so like, if you've a PM who is maybe isn't that 29:03 productive like, maybe you'd pair those two people, same thing with an engineer. 29:05 Like productive engineers verses like, I just want 29:09 to build an engineer, like that's totally fine. 29:12 We just try to find those like, balance is in the teams. 29:14 So that like, great work gets done. 29:17 And sometimes it is like a super product designer, 29:19 super product the product manager and a super product engineer. 29:22 As long as those three people align you're fine. 29:25 [LAUGH] But if any of them have very 29:27 strong opinions like, then it's a little more 29:29 like, lets work that out, But yeah, I 29:31 mean, I think it's like, it feels pretty organic. 29:34 I don't know how we're doing it, it's a total accident. 29:36 [LAUGH] 29:38 I don't, that's not helpful at all, I'm so sorry. 29:41 [LAUGH]. 29:43 >> Thank you. 29:46 My question is about, you talk a lot about measuring or failing starting again. 29:50 I just didn't know about your data and analytics team, or if 29:56 it's, if it is the designers it sounds like, again all very organic. 30:00 But how, how is that relationship, how many people are you know. 30:04 >> Sure. 30:08 It's our data, we have a data analyst team that's pretty big, I would guess. 30:10 I wanna say 12 people are on that team, and they build 30:15 out, like all of these, it's like data stack, it's like, custom, 30:19 so like our analytics tool is totally custom, like they way we 30:23 measure experiments is like, we built our own tools for this stuff. 30:25 And are constantly improving them, which is pretty nice, because then we can go 30:29 straight to the source, if something's going 30:32 wrong or we don't understand why somethings happening. 30:34 So it's, I mean, it's pretty healthy like, they 30:38 have like a data analysis assigned to each team. 30:41 >> Okay. 30:43 >> And actually like everybody has access to all of 30:43 the data so you can, I mean, if a designer 30:46 knew how to run a sequel query, they could like, 30:48 pull up whatever they needed to know, on their own. 30:50 >> Yeah. 30:52 >> [INAUDIBLE] access to that, but like, for 30:52 the like, the deeper like trend like, questions. 30:54 It's nice to have that person, who is the expert on doing that. 30:57 Just going like, hey like, in the last year how many 31:00 shops have with fewer than 10 items sold more than a $1,000? 31:04 You know, it's just like, you can ask really complicated like, trend 31:07 questions and like, they'll find the answers for you or help you understand. 31:11 >> And, my last question is about, you know 31:15 you said you, you pretty much have full stack designers. 31:17 I'm interested to know like, where you're hiring from like, where, 31:20 you know, where people are coming from, what were their previous jobs? 31:22 >> Wherever I can find them. 31:25 It's funny I, so [COUGH], I'm, I'm gonna offend somebody in here. 31:28 I really don't like drivel, I'm gonna make some people really happy. 31:32 That's always what they're still like this like [SOUND]. 31:35 You can see like people get up and walk out. 31:37 [LAUGH]. 31:39 >> But what's really nice about it is it's 31:41 like a place like, you can go to like just. 31:42 They have a list of like every designer, 31:45 in whatever country or state you'd like to look. 31:47 In order of like, so the default sort is 31:51 popularity, which whatever, but you can reverse that sort, which 31:53 is really nice, and so I'm like, I should not be telling people this is how I do it. 31:58 But, like, I've recruited most of the designers that are working 32:02 at Etsy now, and the way I've done this is basically going 32:05 through there, through that exact sort, and like not looking at 32:08 their actual profile but trying to get to through to their website. 32:13 Cuz I want to see if they wrote their own website super cool and 32:15 important, or have a get hub or something like that where they've been writing code. 32:19 And then like examples of process work, like I'm looking 32:23 for that which you also don't find on, on the dribbles. 32:26 So that's one way, another way is 32:30 just like trolling the internet, just looking, not 32:32 trolling like trolling, but like, like the 32:35 old way of trolling, putzing around the internet. 32:38 You know, seeing like a hashtag for a designer, 32:42 or something, like for a design conference, like this one, 32:46 and like looking at who's talking about it, and 32:48 then just trying to like kill people out of there. 32:50 yeah, it's, it's hard, I mean I think I did the math and like, I've, let's see 32:54 we, I did phone screens for like 100, 140 designers in the last 32:59 like year and then, from there I think I wanna say 16 or 18 came 33:04 in for interview loops, and out of that number, we hired like 14 of those. 33:10 So like, there's a huge drop off from the phone 33:14 screen, but like it actually like works out pretty out well. 33:16 It's just a big time investment for me, many than anything else, yeah. 33:19 [BLANK_AUDIO] 33:24 >> Hey what level of fidelity of deliverables do 33:28 you tend to do when you're working super literally. 33:32 Cuz I know that if you're trying to get 33:35 something out really quickly, you don't want to spend 33:36 all of your time making super high fidelity, unless 33:38 you are I guess, coding them yourself for doing CSS. 33:40 >> Yeah, so we are coding them ourselves. 33:44 Which makes the deliverables different. 33:46 I guess depending on which side you're talking to. 33:48 We've actually gotten a little more high fidelity lately as we've gotten into 33:50 like, we're actually getting more and 33:53 more into like, prototyping for qualitative research. 33:54 Which is a new thing at Etsy since last year. 33:58 So some of the designers are like, starting to try to 34:01 like, get like, get into clickable prototypes faster, things like that. 34:04 Again, like, the process really varies. 34:10 Like, some people, like, I know for a fact that 34:12 some people go straight to code and that's where they design. 34:14 Like, there's not like, there's no wire frame. 34:16 They may sketch something just to get their head around it, but that's it. 34:19 And other folks will do wire frames, or 34:22 use a flow, it just, like, kind of depends. 34:24 But yeah I it, like that, I'm not answering these 34:29 very well, I'm really sorry, it just like, it's so varied. 34:33 yeah, like whatever it takes kind of. 34:38 >> Hi there, so it sounds like you're talking a 34:44 lot about a collaborative design, and I really embrace that. 34:47 What have you found has worked best in terms of the 34:50 balance between collaborative design and 34:55 preventing that, becoming designed by committee? 34:57 >> So the collaborative design, again like the small teams really help. 35:01 [COUGH] it's not I mean we're not like showing work, so at Amazon I, 35:05 what I had to do was show work to like, my team, my director, 35:08 my creative director, the director of product, the VP of product, the VP of 35:13 some other stuff, and then Jeff, like to get it signed off on at all. 35:17 And that is, that becomes very design-by-committee 35:22 after a while, cuz you're just trying to, 35:24 like, navigate this like all these different 35:25 personalities to get someone to go like this. 35:27 But because the teams really small and fairly autonomous, 35:31 like, we don't, there is no like formal review process. 35:33 For someone to like get something shipped, is 35:37 the transparent thing is super important for us. 35:40 That just everybody knows what's going on and 35:41 can like, act on something if they need to. 35:43 But I think, yeah, when you only have three or four people working 35:46 on something, like the committee thing is really hard to like, make happen. 35:48 Because it is just so small. 35:52 And the team is usually, again like, if you can get alignment on 35:53 the team like in the early stages, then it's just like, it's gravy. 35:55 Zima committee is totally crappy [LAUGH]. 36:01 Try not, try to get away from that as much as possible. 36:03 >> [SOUND] so, on the second design, when you were mentioning that you guys did 36:06 different versions of it, and then you, you had it out for maybe like two weeks. 36:12 And you were like this isn't not working, we're gonna do another version. 36:18 Did you guys gather data based on that one design, 36:21 or are you guys doing like AB testing around it? 36:24 >> yeah, so after, when we shipped the first 36:27 design we were actually already working on the second one. 36:29 >> Mm-hm. 36:31 >> Because we knew we just wanted to like, keep going. 36:31 We had some ideas about what wasn't going 36:33 to work, and we just wanted to keep moving. 36:35 [COUGH] but yeah, but what we would ten, tend to do would be like. 36:37 So we had these really, we named our releases, it was, 36:40 it was the Henry Project, I know this team was very weird. 36:45 I don't know why it was named Henry. 36:47 But like the first release was named Henry, the second one 36:49 was Enrique, and then Hank, and then Henrietta, and then Henny. 36:51 And each of those releases were mutivarient so, like 36:55 you'd have like this like, base design, new design. 36:57 And then they would three or four variants of that design, 37:01 like smaller variants to like figure out what wasn't really gonna work. 37:03 And then they would like basically take all of that data, do a fresh redesign, do 37:07 this and then do variants of that design, 37:12 and then again, like, like, four or five times. 37:13 So we were running, at some points, three 37:17 or four different variants, this various, like, simultaneously. 37:18 It's nice to be at that scale, like, 1.4 billion page views 37:22 lets you get to, like, what we call statistical significance really quickly. 37:24 Which means that, like, you know that it's not just, like noise happening. 37:27 You, like, you have like, you have a system that will 37:32 tell us, like, with confidence this is what's happening, like for sure. 37:34 And before that it tells you, it just says, 37:38 waiting on data and you just have to wait. 37:40 And it won't tell you anything, which is nice. 37:42 It keeps you out of there. 37:45 >> So, I've always really liked the idea of iterating quickly and, you know, 37:48 throwing out tests, and figuring out and 37:52 validating the designs that you actually go through. 37:54 But as a company, as you grow larger and larger, how 37:57 do you mitigate the risk of you know, putting out a 38:00 design and affecting a large enough of a user base where 38:03 you know, these people are selling things on your site, right? 38:07 >> Uh-huh. 38:10 >> So part of their livelihood depends on that. 38:10 >> Yes. 38:13 >> How do you mitigate that with the test that you are putting out, to make 38:13 sure that you are not affecting such a 38:17 large group of users and affecting their lives. 38:18 >> Sure. 38:21 There's a couple things we tend to only run experiments on buyers. 38:21 Like, so that includes some sellers obviously cuz sellers 38:25 are some of our most, like, avid buyers as well. 38:27 But we run percentage ba, it's percentage, we bucket percents of people. 38:31 so, I don't remember what the percentage is 38:35 where for the listing page, but because we 38:37 have so much traffic it's easy to do 38:38 lower, percentage tests and get somewhere fairly quickly. 38:40 And we try to learn as quickly as possible so we can know if we're 38:44 doing a bad thing and, like, kind of, shut it down if we need to. 38:47 The thing with the sellers we were doing is we don't run experiments on sellers. 38:50 Like, just on sell, like, on seller tools, like the things they're using 38:54 to do their jobs and run their business, we don't experiment on that. 38:58 typically, because we don't like that's someone's business 39:02 and like changing someone's workflow really sucks, and then 39:04 like taking it away because it was just an 39:07 experiment, like really blows, so we don't do that. 39:08 We had this thing, we have a feature 39:12 called teams on Etsy, which was built originally 39:13 to allow sellers that were doing similar things 39:16 to like, group themselves, so you'd be like. 39:18 A seller would start like photographers in Brooklyn, and would 39:21 invite and try to find like those people and like 39:25 kind of form a little group, and it's like a 39:28 support group where you can like help each other out. 39:29 Hey this worked for me, like they were using 39:31 this as like small mini forums, private like team forums. 39:32 We finally figured out that we can use that feature 39:37 to to run to like, to flag someone into a prototype. 39:39 Its like a ship, we get ship a feature and like, say 39:45 like, if you're in this team that we created you get the feature. 39:48 And so that's how we ship things to 39:51 sellers, so we'll be like we'll start a team, 39:53 be like, hey, there's this new team that 39:54 improves like, how you purchase shipping labels on Etsy. 39:56 Join, and we'll tell all the sellers, join this team if you want 39:59 all the new stuff and can te, and can give us feedback on it. 40:01 And they all, and a lot of people will join, and 40:04 then like they know when they're getting into it what they're getting. 40:06 And then we can like ask them questions, little more qualitative, right? 40:08 Like, it's like really hard to quantify, 40:11 like what makes someone faster at their job. 40:13 Especially when like it's, there's only a million sellers. 40:17 Like 1.4 billion page use, let me tell you, it's hard to get significance. 40:19 But that's really helpful. 40:23 Like it's, like, that's all we do now is just ship to 40:24 these prototype teams, and allow them to like, discuss it and tell 40:26 is this is, this sucks, I would never use this, and then 40:29 they leave the prototype because then they don't want that feature anymore. 40:31 And then we're like oh now, we changed it, come back. 40:34 Try to get him back. 40:37 But yeah I mean that's how we try to 40:39 mitigate that, I think we're out of time, so, thanks. 40:41 >> [SOUND] 40:43
You need to sign up for Treehouse in order to download course files.Sign up