1 00:00:00,050 --> 00:00:05,139 [SOUND] Hi, welcome to the Dev Team Show. 2 00:00:05,139 --> 00:00:06,610 My name is James. 3 00:00:06,610 --> 00:00:11,120 In this episode, we're gonna be talking about QA testing automation. 4 00:00:11,120 --> 00:00:15,320 Joining me for the panel discussion is Craig Dennis, who's a teacher here at 5 00:00:15,320 --> 00:00:20,180 Treehouse, Nathan Menge, who's a guest teacher here at Treehouse and 6 00:00:20,180 --> 00:00:23,910 Doug Darst, who's the director of engineering at Treehouse. 7 00:00:23,910 --> 00:00:25,100 Welcome to the panel. 8 00:00:25,100 --> 00:00:25,720 >> Thank you. >> Thank you. 9 00:00:25,720 --> 00:00:26,430 >> Thanks Jed. 10 00:00:26,430 --> 00:00:29,670 >> So I thought we would start by doing a little bit of a level set. 11 00:00:29,670 --> 00:00:33,620 And let's make sure that we all understand what we mean by QA testing within 12 00:00:33,620 --> 00:00:35,850 the context of this conversation. 13 00:00:35,850 --> 00:00:38,400 So Doug, what do we mean by QA testing? 14 00:00:38,400 --> 00:00:40,653 >> Well, QA is really a measuring body. 15 00:00:40,653 --> 00:00:43,240 They measure the software against requirements. 16 00:00:43,240 --> 00:00:46,360 And those requirements might be functional requirements, 17 00:00:46,360 --> 00:00:49,340 meaning what is the software supposed to do? 18 00:00:49,340 --> 00:00:51,730 And they might be non-functional requirements, 19 00:00:51,730 --> 00:00:54,160 meaning how is the software supposed to do that? 20 00:00:54,160 --> 00:00:57,890 So in terms of performance, or scalability, or security, 21 00:00:57,890 --> 00:01:01,810 quality assurance is really concerned with the processes that you put in place and 22 00:01:01,810 --> 00:01:05,050 methodologies that you put into place to ensure that you have quality 23 00:01:05,050 --> 00:01:10,050 built into your product as early as possible in the process. 24 00:01:10,050 --> 00:01:13,940 As compared to quality control, which is really at the back end of the process, 25 00:01:13,940 --> 00:01:16,110 where your trying to prevent bugs from getting out the door. 26 00:01:16,110 --> 00:01:20,260 So quality assurance is really that up front proactive piece of the puzzle. 27 00:01:20,260 --> 00:01:25,280 >> Okay, so, we talk a lot about unit testing as developers here at Treehouse. 28 00:01:25,280 --> 00:01:27,970 So is unit testing part of QA testing? 29 00:01:27,970 --> 00:01:29,300 Or is that something different? 30 00:01:29,300 --> 00:01:32,070 >> Well, it all falls into the whole testing realm. 31 00:01:32,070 --> 00:01:35,813 So every piece of testing is good testing, right? 32 00:01:35,813 --> 00:01:38,492 But typically, when we talk about quality assurance, 33 00:01:38,492 --> 00:01:42,312 we're talking about looking at the requirements, and running a specific test 34 00:01:42,312 --> 00:01:46,520 to exercise the software to ensure that we're meeting the customer's needs. 35 00:01:46,520 --> 00:01:47,690 So unit testing and 36 00:01:47,690 --> 00:01:52,340 integration testing is typically performed by the developers against their code. 37 00:01:52,340 --> 00:01:56,880 And it's done in smaller pieces, so atomic pieces, testing individual functions or 38 00:01:56,880 --> 00:01:57,930 methods. 39 00:01:57,930 --> 00:02:03,240 Whereas QA testing starts to focus more on the system as a whole in testing, making 40 00:02:03,240 --> 00:02:08,210 sure the product in it's final form really meets the needs that it's intended to. 41 00:02:08,210 --> 00:02:14,740 >> So does that always black box testing, or is it black box and white box testing? 42 00:02:14,740 --> 00:02:19,010 >> Well it can be both and companies should consider doing both. 43 00:02:19,010 --> 00:02:22,849 Black box testing means that you're sitting down at the computer like the user 44 00:02:22,849 --> 00:02:26,686 would, and you really don't have any information about how it was designed or 45 00:02:26,686 --> 00:02:28,330 implemented behind the scenes. 46 00:02:28,330 --> 00:02:33,420 So all you see in front of you is the user interface, and you're testing it 47 00:02:33,420 --> 00:02:37,030 based on your understanding of what the software is supposed to do. 48 00:02:37,030 --> 00:02:38,060 White box testing, 49 00:02:38,060 --> 00:02:40,940 on the other hand means you have that information behind the scenes. 50 00:02:40,940 --> 00:02:43,280 You have access to the code, typically. 51 00:02:43,280 --> 00:02:47,270 You can see how it was implemented, all the air checking that might be built 52 00:02:47,270 --> 00:02:50,000 into the code and you can test it from a different angle. 53 00:02:50,000 --> 00:02:51,718 So both of them have pros and cons. 54 00:02:51,718 --> 00:02:58,080 >> So Nate, are developers involved in QA testing or is this sometimes 55 00:02:58,080 --> 00:03:03,080 like a different type of person on the team in terms of experience and skills. 56 00:03:03,080 --> 00:03:08,330 >> It depends on what size and budget your business has. 57 00:03:08,330 --> 00:03:11,470 If you're a small shop with just a few people, 58 00:03:11,470 --> 00:03:16,531 you'll have people doing multiple roles, coming at it with developers, 59 00:03:16,531 --> 00:03:21,405 going through their code with just a mind to what the user experience is. 60 00:03:21,405 --> 00:03:26,210 Whereas if you're a large corporation, you might have a large staff that is entirely 61 00:03:26,210 --> 00:03:31,320 there to run through it with no knowledge of how the underlying system is built. 62 00:03:31,320 --> 00:03:35,660 >> Okay, so there might be different kinds of tests. 63 00:03:35,660 --> 00:03:37,800 So you have testers with development experience, 64 00:03:37,800 --> 00:03:41,310 and testers who may not do any development work, who may be like, 65 00:03:41,310 --> 00:03:46,900 would be fair to say they'd have their business experts or domain experts. 66 00:03:46,900 --> 00:03:47,930 >> Yeah, exactly. 67 00:03:47,930 --> 00:03:51,010 If you consider like an accounting system, for 68 00:03:51,010 --> 00:03:54,120 example, you probably wanna a little bit of both. 69 00:03:54,120 --> 00:03:58,125 So domain expertise would be really handy, because you'd want to have people who were 70 00:03:58,125 --> 00:04:00,635 actually maybe accountants in the real world, who can come in and 71 00:04:00,635 --> 00:04:03,815 understand how an accounting system is really supposed to be behave. 72 00:04:03,815 --> 00:04:06,435 And they bring a certain level of expertise 73 00:04:06,435 --> 00:04:09,560 from a business standpoint that you really can't get otherwise. 74 00:04:09,560 --> 00:04:13,310 So they might be sitting down testing the accounts payable module, and 75 00:04:13,310 --> 00:04:16,250 they know how invoices are entered into the system. 76 00:04:16,250 --> 00:04:20,235 And they can test that mission critical functionality very precisely. 77 00:04:20,235 --> 00:04:24,957 But on the flip side, they might not have the technical skills to do test automation 78 00:04:24,957 --> 00:04:27,190 and some of those types of activities. 79 00:04:28,300 --> 00:04:33,177 Conversely, people with a technical background can do the automated testing, 80 00:04:33,177 --> 00:04:37,706 but they probably don't have that deep, deep subject matter expertise. 81 00:04:37,706 --> 00:04:41,020 >> Yeah, I think that one of the things that comes into play with trying to 82 00:04:41,020 --> 00:04:43,950 decide who's doing what, is the development cycle itself. 83 00:04:43,950 --> 00:04:46,030 Like how long, does it take. 84 00:04:46,030 --> 00:04:51,881 You don't have a team waiting or testing a release ahead of time. 85 00:04:51,881 --> 00:04:55,781 So yeah, I have seen the short release cycle work really well with support teams, 86 00:04:55,781 --> 00:04:59,072 because they're critical for this is how the product needs to work. 87 00:04:59,072 --> 00:05:01,511 We need to make sure that this works no matter what, 88 00:05:01,511 --> 00:05:03,510 test this right here sort of thing there. 89 00:05:04,710 --> 00:05:08,370 >> So you mentioned that that QA testing, quality assurance testing, 90 00:05:08,370 --> 00:05:11,220 is something that should be done early and often. 91 00:05:11,220 --> 00:05:13,503 So how early? 92 00:05:13,503 --> 00:05:16,388 Does that happen pretty much at the beginning of the project throughout 93 00:05:16,388 --> 00:05:17,630 the lifetime? 94 00:05:17,630 --> 00:05:20,083 >> Yeah, I would suggest that it really should be. 95 00:05:20,083 --> 00:05:23,268 Even before the first piece of code that's been written, 96 00:05:23,268 --> 00:05:25,810 you can be doing testing on the requirements. 97 00:05:25,810 --> 00:05:28,390 When we talk about quality assurance being proactive, 98 00:05:28,390 --> 00:05:32,370 you can look at the requirements and find potential bugs that you can prevent. 99 00:05:32,370 --> 00:05:35,950 And obviously the sooner that you find the bug or potential bug, 100 00:05:35,950 --> 00:05:37,130 the cheaper it is to address. 101 00:05:37,130 --> 00:05:41,540 So when you're reviewing requirements, it might state something like 102 00:05:41,540 --> 00:05:46,300 provide the ability to report on a list of employees in the system. 103 00:05:46,300 --> 00:05:49,250 But you could start asking questions around well how 104 00:05:49,250 --> 00:05:50,340 quickly should that be done? 105 00:05:50,340 --> 00:05:52,150 What if there's 1,000 employees? 106 00:05:52,150 --> 00:05:54,420 What if there's a million employees in the system? 107 00:05:54,420 --> 00:05:55,780 What should the performance be? 108 00:05:55,780 --> 00:05:58,710 And if there's a wait, what should the UI experience be? 109 00:05:58,710 --> 00:06:01,140 So you can start fleshing those sorts of ideas and 110 00:06:01,140 --> 00:06:04,840 concerns up front, rather than after that code has already been implemented, and 111 00:06:04,840 --> 00:06:06,320 it's more costly to address. 112 00:06:06,320 --> 00:06:08,710 >> Yeah, it's interesting, cuz in my experience, 113 00:06:08,710 --> 00:06:13,640 I can definitely vouch for having more of a waterfall approach. 114 00:06:13,640 --> 00:06:18,710 Where the development team is kind of churning away at making the application or 115 00:06:18,710 --> 00:06:23,930 the website, and maybe what I would describe as informal QA testing going on. 116 00:06:23,930 --> 00:06:25,818 Where the developers are kind of poking and 117 00:06:25,818 --> 00:06:28,360 prodding at it and making sure things are working. 118 00:06:28,360 --> 00:06:33,210 But we work until we're like, all right, now we feel like we have a release. 119 00:06:33,210 --> 00:06:35,710 Whether that's something that could actually go into production or 120 00:06:35,710 --> 00:06:39,405 not may be debatable, but it's some form of milestone, right. 121 00:06:39,405 --> 00:06:43,640 And we push that out to a staging server and then say, all right, 122 00:06:43,640 --> 00:06:45,372 now that person's in QA. 123 00:06:45,372 --> 00:06:48,521 So what you're talking about sounds a little different from that in terms 124 00:06:48,521 --> 00:06:49,790 of process. 125 00:06:49,790 --> 00:06:52,460 >> Yeah, I think it's really important to get QA, 126 00:06:52,460 --> 00:06:56,650 whatever that might look like in your company, involved early on. 127 00:06:56,650 --> 00:07:00,960 Just again, to find those bugs or potential bugs early in the process and 128 00:07:00,960 --> 00:07:02,010 starts with the requirements. 129 00:07:02,010 --> 00:07:05,160 And then, you don't wanna wait until the whole feature or 130 00:07:05,160 --> 00:07:07,010 application is developed before you get it into QA. 131 00:07:07,010 --> 00:07:09,010 You wanna be testing in an iterative mode, 132 00:07:09,010 --> 00:07:11,300 giving that feedback to development quickly. 133 00:07:11,300 --> 00:07:14,830 One, because the developers' minds are kinda wrapped around that piece of code. 134 00:07:14,830 --> 00:07:17,650 And so there's not a lot of context switching as 135 00:07:17,650 --> 00:07:21,140 compared to if you wait a month before QA starts getting into it. 136 00:07:21,140 --> 00:07:21,780 >> Right. 137 00:07:21,780 --> 00:07:26,850 >> And it's just kinda good practice to kind of be working in an iterative flow. 138 00:07:26,850 --> 00:07:31,510 And that way you're building quality and testing for quality the whole way. 139 00:07:31,510 --> 00:07:32,517 So what is? Okay, 140 00:07:32,517 --> 00:07:36,150 so QA testing still feels a little nebulous to me. 141 00:07:36,150 --> 00:07:38,280 What does that actually look like, Craig? 142 00:07:38,280 --> 00:07:42,550 In your experience, what does QA look like from a manual prospective. 143 00:07:42,550 --> 00:07:45,600 Is that a room of people or people sitting down in a room and 144 00:07:45,600 --> 00:07:47,000 banging away at the app, or? 145 00:07:47,000 --> 00:07:48,750 >> Right, I mean, that definitely can be. 146 00:07:48,750 --> 00:07:51,760 Like you just talked about, I think, you push something up to release. 147 00:07:51,760 --> 00:07:54,442 Here's a new feature that you have all talked about and thought about. 148 00:07:54,442 --> 00:07:58,655 And, so I think that that manual QA can be done by stakeholders, 149 00:07:58,655 --> 00:08:00,612 like Doug was talking about. 150 00:08:00,612 --> 00:08:03,293 People who are interested in this is the feature that I requested, and 151 00:08:03,293 --> 00:08:05,510 it's coming alive and I'm going to test it and I know it. 152 00:08:05,510 --> 00:08:07,300 I think that that's a good manual QA place. 153 00:08:07,300 --> 00:08:11,450 I think there are definitely people, there are definitely companies that still 154 00:08:11,450 --> 00:08:15,880 have an entire QA team dedicated where the thing will come to them and 155 00:08:15,880 --> 00:08:17,220 they will do manual tests. 156 00:08:17,220 --> 00:08:21,658 And they'll have paper check boxes even, going through that sort of thing. 157 00:08:21,658 --> 00:08:26,640 >> To describe like these are our test cases and when we do a testing cycle, 158 00:08:26,640 --> 00:08:29,630 we split these up and we are gonna run through these actions. 159 00:08:29,630 --> 00:08:34,022 >> Yeah, and I think, I mean I've seen a very like you know, 160 00:08:34,022 --> 00:08:39,493 there's definitely automated QA, but I have seen very successfully, 161 00:08:39,493 --> 00:08:45,341 sites that are going live multiple times a day, cuz continuous deployment. 162 00:08:45,341 --> 00:08:49,540 There's a, I need to make sure that the thing that makes us money is working. 163 00:08:49,540 --> 00:08:50,130 Yes, it does. 164 00:08:50,130 --> 00:08:50,935 Okay, good, go ahead. 165 00:08:50,935 --> 00:08:52,200 >> [LAUGH] >> The deploy's good. 166 00:08:52,200 --> 00:08:55,450 Yeah, that sort of manual QA I think is always, probably, 167 00:08:55,450 --> 00:08:58,968 I don't know of anybody really who's ever fully given that up. 168 00:08:58,968 --> 00:08:59,870 Hey, the site went live. 169 00:08:59,870 --> 00:09:02,090 We need to make sure that that thing that's- 170 00:09:02,090 --> 00:09:03,860 >> Putting eyeballs on the thing, and 171 00:09:03,860 --> 00:09:06,040 actually you get that sense of okay, yeah, 172 00:09:06,040 --> 00:09:08,120 I can see that it's working and we're good to go. 173 00:09:09,270 --> 00:09:13,634 So Nate, what are some of the pains of doing this manual process? 174 00:09:13,634 --> 00:09:17,360 That sounds like kind of arduous work. 175 00:09:17,360 --> 00:09:20,490 >> Yeah, so the major pains are just the repetitiveness of it. 176 00:09:20,490 --> 00:09:24,340 If you're working with a product that's in multiple browsers on multiple operating 177 00:09:24,340 --> 00:09:26,920 systems, you have to literally do the exact same thing over and 178 00:09:26,920 --> 00:09:32,750 over again across all, possibly across all permutations of these platforms. 179 00:09:32,750 --> 00:09:37,780 Some of the pains are just the amount of output you might end up with. 180 00:09:37,780 --> 00:09:41,580 I've worked for a major company that's FDA regulated, and 181 00:09:41,580 --> 00:09:43,390 I can tell you exactly what the QA testing looked like, 182 00:09:43,390 --> 00:09:44,940 because there's this half-ream of paper- >> [LAUGH] 183 00:09:44,940 --> 00:09:46,190 >> Because it had to be printed out, 184 00:09:46,190 --> 00:09:47,338 signed >> Wow. 185 00:09:47,338 --> 00:09:49,950 >> With actual signatures, verifying that we did the work. 186 00:09:49,950 --> 00:09:53,600 And that these individuals are responsible to say that this is quality. 187 00:09:53,600 --> 00:09:54,720 And it's filed with the FDA. 188 00:09:54,720 --> 00:09:57,580 So every single test- >> As you were doing test cases, 189 00:09:57,580 --> 00:10:01,490 you were signing that saying that saying I tested this and it passed. 190 00:10:01,490 --> 00:10:02,170 >> Yeah. 191 00:10:02,170 --> 00:10:03,300 >> Wow, okay. 192 00:10:03,300 --> 00:10:04,580 It was all done at the end but. 193 00:10:04,580 --> 00:10:05,410 >> Right. 194 00:10:05,410 --> 00:10:06,400 >> Yeah. 195 00:10:06,400 --> 00:10:10,780 >> So how bad can bad get when you are doing manual testing? 196 00:10:10,780 --> 00:10:15,870 Does anyone have a horror story, in terms of what that cycle looked like? 197 00:10:15,870 --> 00:10:20,688 >> So same company I work for, I worked on a product that was in 14 languages, 198 00:10:20,688 --> 00:10:23,250 deployed not only in those languages but 199 00:10:23,250 --> 00:10:26,360 in regionalizations that differ per language. 200 00:10:26,360 --> 00:10:30,157 So you're dealing with, already just localization and 201 00:10:30,157 --> 00:10:32,698 regionalization of 30 plus cases. 202 00:10:32,698 --> 00:10:37,586 And then on top of that, operating systems and browsers, 203 00:10:37,586 --> 00:10:42,378 so there were permutations across 100 plus variables 204 00:10:42,378 --> 00:10:46,592 that literally took months of testing by a group. 205 00:10:46,592 --> 00:10:51,747 That was not small, probably 20 at its smallest to 50 at our biggest, 206 00:10:51,747 --> 00:10:56,418 of people just working months at a time to verify this for release. 207 00:10:56,418 --> 00:10:57,822 And we're FD regulated, so 208 00:10:57,822 --> 00:11:00,811 we're always on the heavy side of what we need to sign off on. 209 00:11:00,811 --> 00:11:04,798 But by the time we're done testing, the next product is ready to ship, and 210 00:11:04,798 --> 00:11:06,520 we have to start all over again. 211 00:11:06,520 --> 00:11:07,144 >> And repeat? 212 00:11:07,144 --> 00:11:08,880 >> This will repeat the cycle. 213 00:11:08,880 --> 00:11:10,650 >> Wow, I can imagine that with, 214 00:11:10,650 --> 00:11:13,450 for instance, it's like mobile devices these days. 215 00:11:13,450 --> 00:11:17,194 The permeations that you have to test just keep expanding and expanding. 216 00:11:17,194 --> 00:11:19,326 It's not probably even a linear increase, right, 217 00:11:19,326 --> 00:11:21,176 it's probably exponentially increasing. 218 00:11:21,176 --> 00:11:23,210 >> [LAUGH] >> Every time you add a device and 219 00:11:23,210 --> 00:11:25,900 it has any sort of flexibility, different multiple 220 00:11:25,900 --> 00:11:30,980 browsers on the same Android device, that's already dozens of permutations. 221 00:11:30,980 --> 00:11:32,842 >> So let's talk about automation then, 222 00:11:32,842 --> 00:11:35,910 because we've mentioned it a little bit here and there. 223 00:11:35,910 --> 00:11:39,790 Automation can help a lot, but what does that look like? 224 00:11:39,790 --> 00:11:43,020 What are we talking about when we say we are gonna automate the QA testing process? 225 00:11:44,720 --> 00:11:46,070 >> Well, it can really vary. 226 00:11:46,070 --> 00:11:50,650 It depends on what you have as a product and where you wanna go with it. 227 00:11:50,650 --> 00:11:55,350 If you have the people, you can dedicate a team to just start going through your mail 228 00:11:55,350 --> 00:12:00,750 processes and writing code to just repetitively knock out these cases. 229 00:12:00,750 --> 00:12:04,830 But a lot of companies don't have the staff to have a dedicated team, so 230 00:12:04,830 --> 00:12:10,240 you've got a small, manual group of people who are in their spare time, 231 00:12:10,240 --> 00:12:11,990 which doesn't really exist in the first place, 232 00:12:11,990 --> 00:12:17,120 trying to automate some stuff from the side to get through their workflow. 233 00:12:17,120 --> 00:12:21,560 >> Right, that's if you have an existing application with an established QA 234 00:12:21,560 --> 00:12:26,340 process, what you're suggesting is you kind of almost moonlight this switch 235 00:12:26,340 --> 00:12:31,060 over to automated testing, and you just kind of slowly chip away at it and 236 00:12:31,060 --> 00:12:33,026 maybe you get some momentum, eventually. 237 00:12:33,026 --> 00:12:37,000 >> It's certainly not an ideal, but it's a situation I definitely have seen multiple 238 00:12:37,000 --> 00:12:39,660 times where you just don't have the staff to dedicate to it. 239 00:12:39,660 --> 00:12:42,830 So you have to just slowly eke your way out. 240 00:12:42,830 --> 00:12:48,070 And you focus on your worst case, your biggest time consuming tests and 241 00:12:48,070 --> 00:12:49,860 automate those out. 242 00:12:49,860 --> 00:12:51,650 Then with the time that saves you, 243 00:12:51,650 --> 00:12:55,000 hopefully you can write some more tests that saves more time. 244 00:12:55,000 --> 00:12:57,510 And slowly and slowly, you chip away at 245 00:12:57,510 --> 00:13:00,840 this massive block of testing until you have something that's manageable. 246 00:13:00,840 --> 00:13:04,839 >> So it sounds like, regardless of where you're at in sort of the lifetime of, 247 00:13:04,839 --> 00:13:08,221 even if you're just starting a new application or a new project or 248 00:13:08,221 --> 00:13:11,790 even if it's a legacy application that's been being maintained for 249 00:13:11,790 --> 00:13:16,210 a long period of time, that automation can help in all those cases. 250 00:13:16,210 --> 00:13:18,940 >> Yeah, for sure, I think there's a lot of companies 251 00:13:18,940 --> 00:13:22,680 start off with a manual test group and they do a lot of manual testing. 252 00:13:22,680 --> 00:13:26,550 And it's quick to get going with that, right, you can get some quick results. 253 00:13:26,550 --> 00:13:29,500 But it's also kind of hard to shift that momentum, once 254 00:13:29,500 --> 00:13:34,510 you have that established there might be a bias against leaving a manual testing. 255 00:13:34,510 --> 00:13:38,870 Like Nathan was saying, it's hard to carve out the time to dedicate to automated 256 00:13:38,870 --> 00:13:42,605 testing, and having the skill set on your team to write that code. 257 00:13:42,605 --> 00:13:46,140 But I think he is absolutely right that you just have to start chipping away at 258 00:13:46,140 --> 00:13:49,610 some of your mission critical pieces of functionality that have to work. 259 00:13:49,610 --> 00:13:54,010 And really focusing on those things that are super time consuming, or prone to 260 00:13:54,010 --> 00:13:58,205 error to free up your manual testers to work on other areas of the application. 261 00:13:58,205 --> 00:14:02,660 And use the automation as an accelerator for your testing efforts. 262 00:14:02,660 --> 00:14:06,730 >> For sure, and on that I've never met a manual tester who doesn't want to do 263 00:14:06,730 --> 00:14:07,520 automation testing. 264 00:14:07,520 --> 00:14:11,570 So it's like always a benefit of you're empowering people learn how to code and 265 00:14:11,570 --> 00:14:15,800 they're very excited usually about learning how to do it as long as it's, 266 00:14:15,800 --> 00:14:18,060 I think you can chip away at that. 267 00:14:18,060 --> 00:14:21,220 >> So to be clear, when we're talking about this automation piece, 268 00:14:21,220 --> 00:14:22,910 we're gonna leverage some sort of tool. 269 00:14:22,910 --> 00:14:26,660 And there's probably a variety of tools that you can use to do this. 270 00:14:26,660 --> 00:14:29,250 But you're actually talking about, for instance, 271 00:14:29,250 --> 00:14:34,310 if it's a web-based application, you're automating browsing from page to page or 272 00:14:34,310 --> 00:14:37,510 from screen to screen, clicking buttons, filling out fields. 273 00:14:37,510 --> 00:14:38,850 Is that what we're talking about? 274 00:14:38,850 --> 00:14:40,260 >> Yeah, definitely. 275 00:14:40,260 --> 00:14:42,670 >> So to set up those test cases, 276 00:14:42,670 --> 00:14:45,560 typically you're writing code to do that, is that true? 277 00:14:45,560 --> 00:14:47,682 >> That's true. >> Are there some tools that, 278 00:14:47,682 --> 00:14:50,360 I feel like I've have heard of tools. 279 00:14:50,360 --> 00:14:51,690 I don't know if that's Cucumber or 280 00:14:51,690 --> 00:14:54,860 what I'm thinking of, that are more natural language? 281 00:14:54,860 --> 00:14:58,260 >> So yeah, Cucumber is what they call a behavioral driven design tool, 282 00:14:58,260 --> 00:15:02,330 where you literally write out just natural human sentences. 283 00:15:02,330 --> 00:15:07,720 And basically under the hood those sentences are interpreted into 284 00:15:07,720 --> 00:15:09,390 automated steps with the tool. 285 00:15:09,390 --> 00:15:13,950 But Cucumber particularly I really enjoyed because you could get your 286 00:15:13,950 --> 00:15:16,731 people who knew business of your product, 287 00:15:16,731 --> 00:15:20,607 like pure business analysts who didn't understand code. 288 00:15:20,607 --> 00:15:25,622 You could get them to write things out in a sort of verbose way that could 289 00:15:25,622 --> 00:15:31,095 easily transfer into Cucumber steps which a test engineer could then take. 290 00:15:31,095 --> 00:15:33,050 >> Yeah. >> And automate the whole system. 291 00:15:33,050 --> 00:15:37,640 But the big benefit is that it leaves the tests in a state that 292 00:15:37,640 --> 00:15:41,690 any person can just read them and exactly understand what everything does. 293 00:15:41,690 --> 00:15:43,510 You don't have to parse code. 294 00:15:43,510 --> 00:15:45,800 It's literally just reading sentences. 295 00:15:45,800 --> 00:15:52,050 When this, then that, Given X, when Y, then Z. 296 00:15:52,050 --> 00:15:53,880 >> Right. >> So it's a structured language, but 297 00:15:53,880 --> 00:15:57,630 it's natural and broad enough that people don't feel too confined, and 298 00:15:57,630 --> 00:16:00,530 they're not looking at curly braces and semicolons for instance, 299 00:16:00,530 --> 00:16:03,320 and getting kind of hung up on what that looks like. 300 00:16:03,320 --> 00:16:04,800 >> Yeah. >> I've seen that used for 301 00:16:04,800 --> 00:16:06,700 a great place for business logic, where you're like, 302 00:16:06,700 --> 00:16:08,800 what happens when you're logged in and you can't. 303 00:16:08,800 --> 00:16:09,490 Are you supposed to be able to click that? 304 00:16:09,490 --> 00:16:14,000 And you're like, well, let's read the test together as a product team. 305 00:16:14,000 --> 00:16:16,510 It's pretty neat to look- >> The testing documentation is a really 306 00:16:16,510 --> 00:16:17,154 powerful tool. 307 00:16:17,154 --> 00:16:18,555 >> Yeah. >> Interesting, so 308 00:16:18,555 --> 00:16:24,406 is that a common way of doing this, or is that still a little bit more fringey, 309 00:16:24,406 --> 00:16:27,979 in terms of how to go about- >> In five different places I've worked, 310 00:16:27,979 --> 00:16:28,732 we only did it once. 311 00:16:28,732 --> 00:16:29,244 >> Okay. 312 00:16:29,244 --> 00:16:31,094 [LAUGH] >> And I loved it, but 313 00:16:31,094 --> 00:16:34,668 that was also the only place where I was able to come in at the start and 314 00:16:34,668 --> 00:16:37,020 was heavily involved in decision process. 315 00:16:37,020 --> 00:16:40,310 Everywhere else I've worked, I come in late, and 316 00:16:40,310 --> 00:16:42,850 I have to work with what already exists. 317 00:16:42,850 --> 00:16:45,670 >> Right, so you might end up with more of a hybrid approach then. 318 00:16:45,670 --> 00:16:48,560 So if you have a blended team of some developers and 319 00:16:48,560 --> 00:16:52,250 some business experts, stakeholders, 320 00:16:52,250 --> 00:16:56,260 demand experts, whatever the term that we're using there, collaborating. 321 00:16:56,260 --> 00:16:59,180 So you still might have some manual process where the people who 322 00:16:59,180 --> 00:17:01,932 aren't able to write code are responsible for that. 323 00:17:01,932 --> 00:17:06,300 Maybe they're also working shoulder to shoulder with the developers to write or 324 00:17:06,300 --> 00:17:09,520 do the programming to write the automated portions. 325 00:17:09,520 --> 00:17:13,090 Is that kind of a fair way to describe how that can work? 326 00:17:13,090 --> 00:17:14,310 It certainly can work that way. 327 00:17:14,310 --> 00:17:16,380 I mean, there's no really one way to do this, 328 00:17:16,380 --> 00:17:18,870 with given your staff, what you have. 329 00:17:18,870 --> 00:17:21,540 If you don't have technical people except for 330 00:17:21,540 --> 00:17:24,740 your developers, then basically developers have to write the main task. 331 00:17:24,740 --> 00:17:27,330 Otherwise you're not gonna have automated tasks. 332 00:17:27,330 --> 00:17:31,443 >> Yeah, I would say that in general from a QA perspective, you wanna be one, 333 00:17:31,443 --> 00:17:34,465 transparent with all your testing and testing tools. 334 00:17:34,465 --> 00:17:34,993 But two, 335 00:17:34,993 --> 00:17:39,236 have a system in place that everyone can participate in the testing effort. 336 00:17:39,236 --> 00:17:42,438 So it's not just responsibility if you have a dedicated QA staff to 337 00:17:42,438 --> 00:17:45,525 do all the testing, you wanna bring in your development staff, 338 00:17:45,525 --> 00:17:47,600 your product managers, if you have them. 339 00:17:47,600 --> 00:17:51,240 Customer support is a great resource for 340 00:17:51,240 --> 00:17:54,190 helping to test because they know where all the problem areas are. 341 00:17:54,190 --> 00:17:59,890 And so, but I agree with Nathan that you use the right tool for the job. 342 00:17:59,890 --> 00:18:03,222 So you don't have to be locked into one tool and try to apply that everywhere. 343 00:18:03,222 --> 00:18:04,620 It's gonna be a combination of tools, 344 00:18:04,620 --> 00:18:06,600 depending on what it is you are trying to test. 345 00:18:06,600 --> 00:18:08,714 >> Yeah, no golden hammer, so to speak. 346 00:18:08,714 --> 00:18:09,339 >> Yeah, right. 347 00:18:09,339 --> 00:18:12,577 >> One of the gateway tools that I've seen work is there's, 348 00:18:12,577 --> 00:18:14,925 I think it's Selenium that produces it. 349 00:18:14,925 --> 00:18:16,970 It's a little browser plug-in and you click it. 350 00:18:16,970 --> 00:18:20,310 It just kind of records your clicks as you go, and then it generates a little, so 351 00:18:20,310 --> 00:18:24,040 like this is what you did and code below and it's a thing that you 352 00:18:24,040 --> 00:18:26,680 can take a look and go, I can see how we could put this across the side. 353 00:18:26,680 --> 00:18:27,874 So that's a great sales tool. 354 00:18:27,874 --> 00:18:32,493 I've seen that convert teams from maybe we shouldn't do manual testing if the manual 355 00:18:32,493 --> 00:18:34,644 tests can produce these sets of scripts. 356 00:18:34,644 --> 00:18:36,467 I mean, it just kind of works across. 357 00:18:36,467 --> 00:18:38,960 Right, having a recorder I can imagine it would be huge, 358 00:18:38,960 --> 00:18:41,890 cuz I can totally see how you're gonna say, hey, I'm just going to 359 00:18:41,890 --> 00:18:45,930 install this thing and have you go through what you would normally do manually. 360 00:18:45,930 --> 00:18:48,050 But we're also gonna capture some of that, and 361 00:18:48,050 --> 00:18:51,340 I'm imagine there's some finessing maybe you need to do here and there. 362 00:18:52,480 --> 00:18:55,562 >> Sure, yeah, if you would record the script that's gonna use the same value you 363 00:18:55,562 --> 00:18:57,820 typed in every time, so you'll wanna randomize that. 364 00:18:57,820 --> 00:19:00,406 It definitely is a great jumpstart to get the script going. 365 00:19:00,406 --> 00:19:03,540 >> In general, I found it's fine for a starting point, 366 00:19:03,540 --> 00:19:05,450 but that is not gonna be your goal. 367 00:19:05,450 --> 00:19:09,600 These are core and playback scripts that have the weakness that if there's any sort 368 00:19:09,600 --> 00:19:14,460 of logical changes in your code, they quickly become more work to maintain than 369 00:19:14,460 --> 00:19:19,115 it would be if you had a more abstract implementation 370 00:19:19,115 --> 00:19:24,150 through something like Selenium or Cucumber. 371 00:19:24,150 --> 00:19:27,847 There's all kinds of testing frameworks for just about every language and 372 00:19:27,847 --> 00:19:29,264 every framework out there. 373 00:19:29,264 --> 00:19:33,198 >> Okay, we've kind of been, or at least in my mind anyway during this discussion, 374 00:19:33,198 --> 00:19:36,505 I keep thinking web applications because that's what I have the most 375 00:19:36,505 --> 00:19:37,470 experience with. 376 00:19:37,470 --> 00:19:41,467 But I would assume that even if you are doing like mobile development whether that 377 00:19:41,467 --> 00:19:45,229 be native or react native or something else, or even desktop development, 378 00:19:45,229 --> 00:19:49,463 for instance, that there's probably tools that will assist with automating that QA 379 00:19:49,463 --> 00:19:50,730 testing as well? 380 00:19:50,730 --> 00:19:54,367 >> Yeah, absolutely, I mean, there's emulators on the mobile side that helps 381 00:19:54,367 --> 00:19:57,040 you, and you can apply automation tools to those. 382 00:19:57,040 --> 00:19:59,840 On the desktop side, it's been a long time since I worked in that area, 383 00:19:59,840 --> 00:20:03,033 but there are tools like Silk and whatnot that Basically, 384 00:20:03,033 --> 00:20:07,160 much like Selenium allows you to interact with the UI and drive the application, 385 00:20:07,160 --> 00:20:09,610 as if a user was sitting right there at the keyboard interacting with you. 386 00:20:09,610 --> 00:20:11,800 >> That makes total sense, okay. 387 00:20:11,800 --> 00:20:14,140 So help me out here. 388 00:20:14,140 --> 00:20:18,520 You're investing in writing code, so it's kinda another code base that's 389 00:20:18,520 --> 00:20:22,710 to the side of the code base in your application that you're working on. 390 00:20:22,710 --> 00:20:25,260 The application's not gonna be static. 391 00:20:25,260 --> 00:20:27,540 It's gonna continue to grow and change and evolve over time. 392 00:20:27,540 --> 00:20:29,260 What are we talking about here? 393 00:20:29,260 --> 00:20:34,765 Are we taking on, is that overhead gonna end up killing us in the long run? 394 00:20:34,765 --> 00:20:36,250 And is there ways to help manage that? 395 00:20:37,310 --> 00:20:40,120 >> So first, the overhead testing, 396 00:20:40,120 --> 00:20:45,630 you're talking something that we're using to minimize the cost of testing. 397 00:20:45,630 --> 00:20:49,110 So as long as you're focused on quality, automation 398 00:20:50,150 --> 00:20:54,460 done correctly is only going to minimize the amount of work you have to do. 399 00:20:54,460 --> 00:20:57,430 You will have to do some work to maintain your testing suite. 400 00:20:57,430 --> 00:21:02,160 But now, moving forward, you're running these tests every time you add a feature. 401 00:21:02,160 --> 00:21:05,770 So these tests, if they're gonna break, tou're gonna see what they're gonna break, 402 00:21:05,770 --> 00:21:08,050 you're gonna fix those tests, and you're gonna move forward. 403 00:21:08,050 --> 00:21:09,770 So it shouldn't ever really get stale and 404 00:21:09,770 --> 00:21:13,280 out of touch with your code base, it should evolve with it. 405 00:21:13,280 --> 00:21:15,390 >> Right, so there's going to be maintenance. 406 00:21:15,390 --> 00:21:19,280 But I guess it's fair to say that even if you're doing manual testing, those manual 407 00:21:19,280 --> 00:21:24,496 testers would have to update written use test cases and I relearn the application. 408 00:21:24,496 --> 00:21:29,310 So it's kinda just repurposing that effort into maintaining code and to your 409 00:21:29,310 --> 00:21:34,180 point there is probably a large enough net gain savings from automating this, 410 00:21:34,180 --> 00:21:40,500 that It's hard to exceed that in terms of maintenance, and from a code perspective. 411 00:21:40,500 --> 00:21:43,170 >> Yeah, totally, it's on the proactive, you're gonna pay for 412 00:21:43,170 --> 00:21:44,471 it on the reactive side of it, right? 413 00:21:44,471 --> 00:21:45,220 [LAUGH] >> Right? 414 00:21:45,220 --> 00:21:47,170 >> Like if these bugs are showing up, you're gonna be paying for 415 00:21:47,170 --> 00:21:47,700 it later anyway. 416 00:21:47,700 --> 00:21:49,810 Development trying to go and catch that and fix that. 417 00:21:49,810 --> 00:21:52,200 And then fixing that, like, did I break something else over here? 418 00:21:52,200 --> 00:21:54,100 >> Right. >> But I've see these test runs go 419 00:21:54,100 --> 00:21:58,195 where you like, I just change that one field and then you break 30 tests, 420 00:21:58,195 --> 00:22:01,710 this form was used all over the places, and this one little bit. 421 00:22:01,710 --> 00:22:04,784 I mean that just, when you see that happen, you're like wow, 422 00:22:04,784 --> 00:22:08,554 this is great that did that, nobody tested, nobody either sit there and test 423 00:22:08,554 --> 00:22:12,562 those or think about all these difference scenarios that it's on the code there. 424 00:22:12,562 --> 00:22:13,230 >> Yeah. 425 00:22:13,230 --> 00:22:16,960 >> You got to the thing there is you got to focus on 426 00:22:17,990 --> 00:22:21,430 you got to put effort the some amount of effort into your testing 427 00:22:21,430 --> 00:22:23,830 that you do in your development because it's important. 428 00:22:23,830 --> 00:22:25,400 It's important that your code is working. 429 00:22:25,400 --> 00:22:26,400 >> Right, that makes sense. 430 00:22:26,400 --> 00:22:29,160 >> I think you have to go into it knowing that a little bit of planning goes 431 00:22:29,160 --> 00:22:29,680 a long way. 432 00:22:29,680 --> 00:22:34,170 So a lot of times in automation systems, people will create frameworks. 433 00:22:34,170 --> 00:22:36,540 And so, you can have common functionality. 434 00:22:36,540 --> 00:22:39,844 That everyone can use in their scripts so, you don't want everyone at 435 00:22:39,844 --> 00:22:43,540 the beginning of their script to write the code to log in to the system, right. 436 00:22:43,540 --> 00:22:46,145 So you create some common code that everybody can utilize and 437 00:22:46,145 --> 00:22:47,330 those things go a long way. 438 00:22:47,330 --> 00:22:50,700 And then again, when that changes in the actual application, 439 00:22:50,700 --> 00:22:54,340 hopefully there's just one place that you can change it, and so it minimizes that. 440 00:22:54,340 --> 00:22:55,950 So, a little bit of planning goes a long way. 441 00:22:55,950 --> 00:22:57,510 >> That makes a lot of sense. 442 00:22:57,510 --> 00:22:59,900 So imagine I'm managing a deaf team. 443 00:22:59,900 --> 00:23:03,540 Maybe I'm managing the product manager or I'm some decisions or 444 00:23:03,540 --> 00:23:06,176 some control over how we move ahead here. 445 00:23:06,176 --> 00:23:09,070 We've been doing what we've been doing with manual testing for 446 00:23:09,070 --> 00:23:11,230 a long time and you've come to me and said, 447 00:23:11,230 --> 00:23:14,780 hey we've got to automate some of this, but maybe I'm not buying it. 448 00:23:14,780 --> 00:23:18,380 Maybe I'm interested, but I'm like I'm unsure if this 449 00:23:18,380 --> 00:23:21,880 really gonna have the return on the investment that we need to make. 450 00:23:21,880 --> 00:23:23,780 Do you have any other stories that you can share, 451 00:23:23,780 --> 00:23:27,120 where you've seen that sort of, you're at a decision point here. 452 00:23:27,120 --> 00:23:29,890 And I like to hear a success story. 453 00:23:29,890 --> 00:23:31,860 >> Sure, yeah, I've got a good story. 454 00:23:31,860 --> 00:23:34,992 I think a lot of times, if you're facing that type of question, 455 00:23:34,992 --> 00:23:37,950 all you have to do is look towards real life and what happens. 456 00:23:37,950 --> 00:23:43,340 And one of my previous companies, we had a large ERP system, 457 00:23:43,340 --> 00:23:47,730 and it was a system where multiple users can be in there at one time, so like 40, 458 00:23:47,730 --> 00:23:49,240 50, 60 people at one time. 459 00:23:49,240 --> 00:23:53,610 And a bug was reported that occasionally a race condition would occur, so 460 00:23:53,610 --> 00:23:56,770 there'd be some type of contention over a particular record in the database or 461 00:23:56,770 --> 00:24:00,910 what have you It was very hard to reproduce, but it was very impactful, 462 00:24:00,910 --> 00:24:04,640 that when it occurred it crashed the system and everyone got knocked out. 463 00:24:04,640 --> 00:24:07,460 So a very important bug. 464 00:24:07,460 --> 00:24:11,140 We tried to reproduce this manually, meaning getting our dev and QA staff and 465 00:24:11,140 --> 00:24:12,030 customer support staff, 466 00:24:12,030 --> 00:24:15,790 so very expensive, it took a long time to get set up to begin with. 467 00:24:15,790 --> 00:24:16,580 Trying to reproduce it. 468 00:24:16,580 --> 00:24:19,130 Once we reproduced it, it was wait a minute. 469 00:24:19,130 --> 00:24:22,180 We didn't have enough debugging going on in the system. 470 00:24:22,180 --> 00:24:24,890 We didn't have enough logging to really understand what the problem was. 471 00:24:24,890 --> 00:24:26,180 So we gotta do this all again, right? 472 00:24:26,180 --> 00:24:28,790 So hours and hours and hours of investment. 473 00:24:28,790 --> 00:24:31,570 Fortunately, we had the foresight of on the side having 474 00:24:31,570 --> 00:24:35,210 engineers writing some automated tests to do this exact same thing and 475 00:24:35,210 --> 00:24:39,690 eventually we were able to reproduce it more frequently and with less effort. 476 00:24:39,690 --> 00:24:42,570 And so it started paying for itself right there. 477 00:24:42,570 --> 00:24:46,402 But then, beyond that once the fix was in place, 478 00:24:46,402 --> 00:24:52,198 we could those same scripts and let them run 24/7 for an entire week and 479 00:24:52,198 --> 00:24:56,610 make sure that that risk condition did not occur again. 480 00:24:56,610 --> 00:25:00,640 And then, a third point is that that could then be added to our test suite, and 481 00:25:00,640 --> 00:25:02,454 run every time we do that release. 482 00:25:02,454 --> 00:25:02,973 >> Right. 483 00:25:02,973 --> 00:25:07,364 >> So something that we didn't have in place that took some time to do, 484 00:25:07,364 --> 00:25:12,063 really got rid of the need to have this massive manual testing effort that 485 00:25:12,063 --> 00:25:13,970 just wasn't sustainable. 486 00:25:13,970 --> 00:25:17,090 There's no way you could do that on a regular basis and you certainly went more 487 00:25:17,090 --> 00:25:20,770 running for 24/7 for a week to make sure the bug was really gone. 488 00:25:20,770 --> 00:25:25,594 So, automation and just solve these type of volume and those are issues that you 489 00:25:25,594 --> 00:25:30,274 are trying to deal with when you are talking about people and only been working 490 00:25:30,274 --> 00:25:34,636 for eighteen hours a day, [LAUGH] >> Yeah, having done a lot of manual QA 491 00:25:34,636 --> 00:25:40,178 test, specially first involve the domain experts it would really nice to know for 492 00:25:40,178 --> 00:25:45,395 a fact that when you run on your automated QA test that they actually run, what 493 00:25:45,395 --> 00:25:51,320 would be the results instead of having fingers cross that everyone is doing. 494 00:25:51,320 --> 00:25:55,509 In lieu of like having signed pieces of paper that said I read, 495 00:25:55,509 --> 00:25:57,290 I did this thing, right? 496 00:25:57,290 --> 00:25:57,928 >> Right. >> And 497 00:25:57,928 --> 00:26:02,073 that would be really great as from a developer's perspective just having that 498 00:26:02,073 --> 00:26:03,970 sort of, like you know that it ran. 499 00:26:03,970 --> 00:26:05,305 You know whether it passed or failed. 500 00:26:05,305 --> 00:26:07,120 >> That's some really interesting tools that exist now. 501 00:26:07,120 --> 00:26:10,020 Like I know we use coveralls where I work right now. 502 00:26:10,020 --> 00:26:12,820 And it actually is just a website that our 503 00:26:12,820 --> 00:26:17,130 test coverage goes to like our test run and it generates a report. 504 00:26:17,130 --> 00:26:19,317 That report gets hosted on coveralls so 505 00:26:19,317 --> 00:26:22,510 we can go back at any moment of any day of any build really. 506 00:26:22,510 --> 00:26:27,560 We can see what was test coverage, what test ran, what was the results. 507 00:26:27,560 --> 00:26:31,939 It's continuing to evolve, but it's amazing what convenience 508 00:26:31,939 --> 00:26:35,610 is being built into the testing frame work these days. 509 00:26:35,610 --> 00:26:40,180 >> Yeah, has anyone had the experience of implementing automated QA testing and 510 00:26:40,180 --> 00:26:44,390 actually can tie that or associate that to happier users, 511 00:26:44,390 --> 00:26:48,510 like users feeling like things are more stable or that the product is better? 512 00:26:48,510 --> 00:26:53,240 Is that something, you been able to see that association so strongly? 513 00:26:53,240 --> 00:26:56,900 >> I think it may be a qualitative association. 514 00:26:56,900 --> 00:27:00,466 I think when your QA staff is sleeping well because they know that the test 515 00:27:00,466 --> 00:27:01,930 are all passing, [LAUGH] >> [LAUGH] 516 00:27:01,930 --> 00:27:05,180 >> I think that alone is worth the money. 517 00:27:05,180 --> 00:27:09,060 But certainly, like Nathan was saying, when you have the transparency into 518 00:27:09,060 --> 00:27:12,590 the test results, and you can rerun these tests whenever you need to. 519 00:27:12,590 --> 00:27:16,246 And even things where you're at the end of a release cycle, and all of a sudden 520 00:27:16,246 --> 00:27:19,690 you've got to fix a bug, you have time to rerun all your tests manually. 521 00:27:19,690 --> 00:27:22,310 Or maybe a new operating system version comes out or 522 00:27:22,310 --> 00:27:25,240 new browser version comes out, and you've got to rerun these tests. 523 00:27:25,240 --> 00:27:28,060 Well, nobody really plans for those sorts of things, and so 524 00:27:28,060 --> 00:27:28,970 you don't have time for it. 525 00:27:28,970 --> 00:27:33,273 And so, there's a lot of peace of mind knowing that a new browser just came out, 526 00:27:33,273 --> 00:27:35,908 we can do a full regression of our application and 527 00:27:35,908 --> 00:27:37,660 have confidence that it works. 528 00:27:37,660 --> 00:27:41,590 Or if there is a bug, we're gonna be the first ones to find it another end users. 529 00:27:41,590 --> 00:27:42,195 >> That is great. 530 00:27:42,195 --> 00:27:43,150 >> Yeah. 531 00:27:43,150 --> 00:27:46,732 >> I saw right there, you hit on something there about I think that it also allows 532 00:27:46,732 --> 00:27:48,450 for better release cycles, right? 533 00:27:48,450 --> 00:27:51,783 So if you're asking specifically about what your users are expecting, yeah, 534 00:27:51,783 --> 00:27:54,880 you can release more quickly because you know that everything's there. 535 00:27:54,880 --> 00:27:59,402 You don't have the QA period they're of traditional waterfall, 536 00:27:59,402 --> 00:28:04,165 like you had said, where you can release a new feature on Wednesday and 537 00:28:04,165 --> 00:28:07,900 Thursday and Friday, because the tests are working. 538 00:28:07,900 --> 00:28:10,534 You get in the habit of writing those tests alongside of it and 539 00:28:10,534 --> 00:28:13,997 you know that that's going and you can show that the business logic is working to 540 00:28:13,997 --> 00:28:16,652 everybody who's concerned about that feature [CROSSTALK]. 541 00:28:16,652 --> 00:28:18,046 >> So that's interesting. 542 00:28:18,046 --> 00:28:21,890 So you have an issue that says, at this feature and part of completing that 543 00:28:21,890 --> 00:28:25,424 feature is, you have the automated QA test that goes along with it, 544 00:28:25,424 --> 00:28:29,250 along with your unit test from a development perspective, and so on. 545 00:28:29,250 --> 00:28:30,460 >> Right. 546 00:28:30,460 --> 00:28:31,210 >> Yeah, that's great. 547 00:28:32,630 --> 00:28:35,530 So for anyone who's now convinced, 548 00:28:35,530 --> 00:28:40,910 I have to automate my QA testing, any advice on how to get started? 549 00:28:40,910 --> 00:28:45,510 >> Well, I guess one of the things I would just reiterate or emphasize is again, 550 00:28:45,510 --> 00:28:49,660 if you have a QA team it's not just the QA teams responsibility. 551 00:28:49,660 --> 00:28:54,360 Bring everybody into the effort, development certainly, product management, 552 00:28:54,360 --> 00:28:55,470 customer support. 553 00:28:55,470 --> 00:28:56,570 Develop a system or 554 00:28:56,570 --> 00:29:00,510 a methodology where everybody can help add tests to the system. 555 00:29:00,510 --> 00:29:03,770 Really try to make your tests repeatable, get the benefit out of them. 556 00:29:03,770 --> 00:29:09,130 Make them so that they're tied into your continuous integration system and 557 00:29:09,130 --> 00:29:10,210 really have them work for you. 558 00:29:10,210 --> 00:29:12,320 They're not things that you just run at the end of the release. 559 00:29:12,320 --> 00:29:13,680 They're being run every day. 560 00:29:13,680 --> 00:29:16,350 Really take the advantage of the machine power that you have. 561 00:29:16,350 --> 00:29:17,350 >> Nate? 562 00:29:17,350 --> 00:29:22,350 >> Yeah, I think the main thing is to just look at your ROI on it, is just 563 00:29:22,350 --> 00:29:27,150 find what you biggest pinpoints are and that's where you start and you just move 564 00:29:27,150 --> 00:29:31,900 down the list of worst case offenders for what takes your time to validate. 565 00:29:31,900 --> 00:29:33,250 >> Okay, Craig? 566 00:29:33,250 --> 00:29:37,660 >> I think it's important to respect the tests as code. 567 00:29:37,660 --> 00:29:39,920 The same way that you treat your other code in the system. 568 00:29:39,920 --> 00:29:41,500 Show that same level of respect. 569 00:29:41,500 --> 00:29:42,704 And that- >> Reviews. 570 00:29:42,704 --> 00:29:45,860 >> Yeah, yeah, totally, and best practices, right. 571 00:29:45,860 --> 00:29:48,100 Hey, this is duplicated, why are we duplicating? 572 00:29:48,100 --> 00:29:50,535 Let's figure out a way to make this, let's make good fixtures, 573 00:29:50,535 --> 00:29:52,070 let's make it fun to write tests. 574 00:29:52,070 --> 00:29:57,036 That's something that happens too, I see that, people, sometimes you'll get into 575 00:29:57,036 --> 00:30:00,860 it's kind of a mess to write this test and I don't make it more fun. 576 00:30:00,860 --> 00:30:04,322 Take in the software the same way that you would in and 577 00:30:04,322 --> 00:30:08,390 develop a component, same thing with the test there. 578 00:30:08,390 --> 00:30:11,600 >> Excellent, well thanks for joining the discussion today. 579 00:30:11,600 --> 00:30:13,160 Great discussion and thanks for 580 00:30:13,160 --> 00:30:16,870 sharing your experiences about QA testing and QA testing automation. 581 00:30:16,870 --> 00:30:17,600 >> Great. >> Thank you. 582 00:30:18,660 --> 00:30:22,360 >> And thanks for watching And be sure to check the notes for this video, 583 00:30:22,360 --> 00:30:25,490 we'll have resources listed there for, QA testing, 584 00:30:25,490 --> 00:30:30,670 to learn more about QA testing or to learn more about automation of those QA tests. 585 00:30:30,670 --> 00:30:33,710 Also, be sure to rate the video and let us know how we're doing. 586 00:30:33,710 --> 00:30:34,430 See you next time. 587 00:30:34,430 --> 00:30:38,290 [SOUND]