1
00:00:00,050 --> 00:00:05,139
[SOUND] Hi, welcome to the Dev Team Show.

2
00:00:05,139 --> 00:00:06,610
My name is James.

3
00:00:06,610 --> 00:00:11,120
In this episode, we're gonna be
talking about QA testing automation.

4
00:00:11,120 --> 00:00:15,320
Joining me for the panel discussion is
Craig Dennis, who's a teacher here at

5
00:00:15,320 --> 00:00:20,180
Treehouse, Nathan Menge, who's
a guest teacher here at Treehouse and

6
00:00:20,180 --> 00:00:23,910
Doug Darst, who's the director
of engineering at Treehouse.

7
00:00:23,910 --> 00:00:25,100
Welcome to the panel.

8
00:00:25,100 --> 00:00:25,720
>> Thank you.
>> Thank you.

9
00:00:25,720 --> 00:00:26,430
>> Thanks Jed.

10
00:00:26,430 --> 00:00:29,670
>> So I thought we would start by
doing a little bit of a level set.

11
00:00:29,670 --> 00:00:33,620
And let's make sure that we all understand
what we mean by QA testing within

12
00:00:33,620 --> 00:00:35,850
the context of this conversation.

13
00:00:35,850 --> 00:00:38,400
So Doug, what do we mean by QA testing?

14
00:00:38,400 --> 00:00:40,653
>> Well, QA is really a measuring body.

15
00:00:40,653 --> 00:00:43,240
They measure the software
against requirements.

16
00:00:43,240 --> 00:00:46,360
And those requirements might
be functional requirements,

17
00:00:46,360 --> 00:00:49,340
meaning what is the software
supposed to do?

18
00:00:49,340 --> 00:00:51,730
And they might be
non-functional requirements,

19
00:00:51,730 --> 00:00:54,160
meaning how is the software
supposed to do that?

20
00:00:54,160 --> 00:00:57,890
So in terms of performance,
or scalability, or security,

21
00:00:57,890 --> 00:01:01,810
quality assurance is really concerned with
the processes that you put in place and

22
00:01:01,810 --> 00:01:05,050
methodologies that you put into place
to ensure that you have quality

23
00:01:05,050 --> 00:01:10,050
built into your product as early
as possible in the process.

24
00:01:10,050 --> 00:01:13,940
As compared to quality control, which is
really at the back end of the process,

25
00:01:13,940 --> 00:01:16,110
where your trying to prevent
bugs from getting out the door.

26
00:01:16,110 --> 00:01:20,260
So quality assurance is really that up
front proactive piece of the puzzle.

27
00:01:20,260 --> 00:01:25,280
>> Okay, so, we talk a lot about unit
testing as developers here at Treehouse.

28
00:01:25,280 --> 00:01:27,970
So is unit testing part of QA testing?

29
00:01:27,970 --> 00:01:29,300
Or is that something different?

30
00:01:29,300 --> 00:01:32,070
>> Well,
it all falls into the whole testing realm.

31
00:01:32,070 --> 00:01:35,813
So every piece of testing is good testing,
right?

32
00:01:35,813 --> 00:01:38,492
But typically,
when we talk about quality assurance,

33
00:01:38,492 --> 00:01:42,312
we're talking about looking at the
requirements, and running a specific test

34
00:01:42,312 --> 00:01:46,520
to exercise the software to ensure that
we're meeting the customer's needs.

35
00:01:46,520 --> 00:01:47,690
So unit testing and

36
00:01:47,690 --> 00:01:52,340
integration testing is typically performed
by the developers against their code.

37
00:01:52,340 --> 00:01:56,880
And it's done in smaller pieces, so atomic
pieces, testing individual functions or

38
00:01:56,880 --> 00:01:57,930
methods.

39
00:01:57,930 --> 00:02:03,240
Whereas QA testing starts to focus more on
the system as a whole in testing, making

40
00:02:03,240 --> 00:02:08,210
sure the product in it's final form really
meets the needs that it's intended to.

41
00:02:08,210 --> 00:02:14,740
>> So does that always black box testing,
or is it black box and white box testing?

42
00:02:14,740 --> 00:02:19,010
>> Well it can be both and
companies should consider doing both.

43
00:02:19,010 --> 00:02:22,849
Black box testing means that you're
sitting down at the computer like the user

44
00:02:22,849 --> 00:02:26,686
would, and you really don't have any
information about how it was designed or

45
00:02:26,686 --> 00:02:28,330
implemented behind the scenes.

46
00:02:28,330 --> 00:02:33,420
So all you see in front of you is
the user interface, and you're testing it

47
00:02:33,420 --> 00:02:37,030
based on your understanding of what
the software is supposed to do.

48
00:02:37,030 --> 00:02:38,060
White box testing,

49
00:02:38,060 --> 00:02:40,940
on the other hand means you have
that information behind the scenes.

50
00:02:40,940 --> 00:02:43,280
You have access to the code, typically.

51
00:02:43,280 --> 00:02:47,270
You can see how it was implemented,
all the air checking that might be built

52
00:02:47,270 --> 00:02:50,000
into the code and
you can test it from a different angle.

53
00:02:50,000 --> 00:02:51,718
So both of them have pros and cons.

54
00:02:51,718 --> 00:02:58,080
>> So Nate, are developers involved
in QA testing or is this sometimes

55
00:02:58,080 --> 00:03:03,080
like a different type of person on the
team in terms of experience and skills.

56
00:03:03,080 --> 00:03:08,330
>> It depends on what size and
budget your business has.

57
00:03:08,330 --> 00:03:11,470
If you're a small shop
with just a few people,

58
00:03:11,470 --> 00:03:16,531
you'll have people doing multiple roles,
coming at it with developers,

59
00:03:16,531 --> 00:03:21,405
going through their code with just
a mind to what the user experience is.

60
00:03:21,405 --> 00:03:26,210
Whereas if you're a large corporation, you
might have a large staff that is entirely

61
00:03:26,210 --> 00:03:31,320
there to run through it with no knowledge
of how the underlying system is built.

62
00:03:31,320 --> 00:03:35,660
>> Okay, so
there might be different kinds of tests.

63
00:03:35,660 --> 00:03:37,800
So you have testers with
development experience,

64
00:03:37,800 --> 00:03:41,310
and testers who may not do any
development work, who may be like,

65
00:03:41,310 --> 00:03:46,900
would be fair to say they'd have their
business experts or domain experts.

66
00:03:46,900 --> 00:03:47,930
>> Yeah, exactly.

67
00:03:47,930 --> 00:03:51,010
If you consider like an accounting system,
for

68
00:03:51,010 --> 00:03:54,120
example, you probably wanna
a little bit of both.

69
00:03:54,120 --> 00:03:58,125
So domain expertise would be really handy,
because you'd want to have people who were

70
00:03:58,125 --> 00:04:00,635
actually maybe accountants in
the real world, who can come in and

71
00:04:00,635 --> 00:04:03,815
understand how an accounting system
is really supposed to be behave.

72
00:04:03,815 --> 00:04:06,435
And they bring a certain
level of expertise

73
00:04:06,435 --> 00:04:09,560
from a business standpoint that
you really can't get otherwise.

74
00:04:09,560 --> 00:04:13,310
So they might be sitting down testing
the accounts payable module, and

75
00:04:13,310 --> 00:04:16,250
they know how invoices
are entered into the system.

76
00:04:16,250 --> 00:04:20,235
And they can test that mission
critical functionality very precisely.

77
00:04:20,235 --> 00:04:24,957
But on the flip side, they might not have
the technical skills to do test automation

78
00:04:24,957 --> 00:04:27,190
and some of those types of activities.

79
00:04:28,300 --> 00:04:33,177
Conversely, people with a technical
background can do the automated testing,

80
00:04:33,177 --> 00:04:37,706
but they probably don't have that deep,
deep subject matter expertise.

81
00:04:37,706 --> 00:04:41,020
>> Yeah, I think that one of the things
that comes into play with trying to

82
00:04:41,020 --> 00:04:43,950
decide who's doing what,
is the development cycle itself.

83
00:04:43,950 --> 00:04:46,030
Like how long, does it take.

84
00:04:46,030 --> 00:04:51,881
You don't have a team waiting or
testing a release ahead of time.

85
00:04:51,881 --> 00:04:55,781
So yeah, I have seen the short release
cycle work really well with support teams,

86
00:04:55,781 --> 00:04:59,072
because they're critical for
this is how the product needs to work.

87
00:04:59,072 --> 00:05:01,511
We need to make sure that
this works no matter what,

88
00:05:01,511 --> 00:05:03,510
test this right here sort of thing there.

89
00:05:04,710 --> 00:05:08,370
>> So you mentioned that that QA testing,
quality assurance testing,

90
00:05:08,370 --> 00:05:11,220
is something that should be done early and
often.

91
00:05:11,220 --> 00:05:13,503
So how early?

92
00:05:13,503 --> 00:05:16,388
Does that happen pretty much at
the beginning of the project throughout

93
00:05:16,388 --> 00:05:17,630
the lifetime?

94
00:05:17,630 --> 00:05:20,083
>> Yeah,
I would suggest that it really should be.

95
00:05:20,083 --> 00:05:23,268
Even before the first piece
of code that's been written,

96
00:05:23,268 --> 00:05:25,810
you can be doing testing
on the requirements.

97
00:05:25,810 --> 00:05:28,390
When we talk about quality
assurance being proactive,

98
00:05:28,390 --> 00:05:32,370
you can look at the requirements and
find potential bugs that you can prevent.

99
00:05:32,370 --> 00:05:35,950
And obviously the sooner that you
find the bug or potential bug,

100
00:05:35,950 --> 00:05:37,130
the cheaper it is to address.

101
00:05:37,130 --> 00:05:41,540
So when you're reviewing requirements,
it might state something like

102
00:05:41,540 --> 00:05:46,300
provide the ability to report on
a list of employees in the system.

103
00:05:46,300 --> 00:05:49,250
But you could start asking
questions around well how

104
00:05:49,250 --> 00:05:50,340
quickly should that be done?

105
00:05:50,340 --> 00:05:52,150
What if there's 1,000 employees?

106
00:05:52,150 --> 00:05:54,420
What if there's a million
employees in the system?

107
00:05:54,420 --> 00:05:55,780
What should the performance be?

108
00:05:55,780 --> 00:05:58,710
And if there's a wait,
what should the UI experience be?

109
00:05:58,710 --> 00:06:01,140
So you can start fleshing
those sorts of ideas and

110
00:06:01,140 --> 00:06:04,840
concerns up front, rather than after that
code has already been implemented, and

111
00:06:04,840 --> 00:06:06,320
it's more costly to address.

112
00:06:06,320 --> 00:06:08,710
>> Yeah, it's interesting,
cuz in my experience,

113
00:06:08,710 --> 00:06:13,640
I can definitely vouch for
having more of a waterfall approach.

114
00:06:13,640 --> 00:06:18,710
Where the development team is kind of
churning away at making the application or

115
00:06:18,710 --> 00:06:23,930
the website, and maybe what I would
describe as informal QA testing going on.

116
00:06:23,930 --> 00:06:25,818
Where the developers
are kind of poking and

117
00:06:25,818 --> 00:06:28,360
prodding at it and
making sure things are working.

118
00:06:28,360 --> 00:06:33,210
But we work until we're like, all right,
now we feel like we have a release.

119
00:06:33,210 --> 00:06:35,710
Whether that's something that could
actually go into production or

120
00:06:35,710 --> 00:06:39,405
not may be debatable, but
it's some form of milestone, right.

121
00:06:39,405 --> 00:06:43,640
And we push that out to a staging
server and then say, all right,

122
00:06:43,640 --> 00:06:45,372
now that person's in QA.

123
00:06:45,372 --> 00:06:48,521
So what you're talking about sounds
a little different from that in terms

124
00:06:48,521 --> 00:06:49,790
of process.

125
00:06:49,790 --> 00:06:52,460
>> Yeah,
I think it's really important to get QA,

126
00:06:52,460 --> 00:06:56,650
whatever that might look like in
your company, involved early on.

127
00:06:56,650 --> 00:07:00,960
Just again, to find those bugs or
potential bugs early in the process and

128
00:07:00,960 --> 00:07:02,010
starts with the requirements.

129
00:07:02,010 --> 00:07:05,160
And then, you don't wanna wait
until the whole feature or

130
00:07:05,160 --> 00:07:07,010
application is developed
before you get it into QA.

131
00:07:07,010 --> 00:07:09,010
You wanna be testing in an iterative mode,

132
00:07:09,010 --> 00:07:11,300
giving that feedback to
development quickly.

133
00:07:11,300 --> 00:07:14,830
One, because the developers' minds are
kinda wrapped around that piece of code.

134
00:07:14,830 --> 00:07:17,650
And so
there's not a lot of context switching as

135
00:07:17,650 --> 00:07:21,140
compared to if you wait a month
before QA starts getting into it.

136
00:07:21,140 --> 00:07:21,780
>> Right.

137
00:07:21,780 --> 00:07:26,850
>> And it's just kinda good practice to
kind of be working in an iterative flow.

138
00:07:26,850 --> 00:07:31,510
And that way you're building quality and
testing for quality the whole way.

139
00:07:31,510 --> 00:07:32,517
So what is?
Okay,

140
00:07:32,517 --> 00:07:36,150
so QA testing still feels
a little nebulous to me.

141
00:07:36,150 --> 00:07:38,280
What does that actually look like, Craig?

142
00:07:38,280 --> 00:07:42,550
In your experience, what does QA
look like from a manual prospective.

143
00:07:42,550 --> 00:07:45,600
Is that a room of people or
people sitting down in a room and

144
00:07:45,600 --> 00:07:47,000
banging away at the app, or?

145
00:07:47,000 --> 00:07:48,750
>> Right, I mean, that definitely can be.

146
00:07:48,750 --> 00:07:51,760
Like you just talked about, I think,
you push something up to release.

147
00:07:51,760 --> 00:07:54,442
Here's a new feature that you have
all talked about and thought about.

148
00:07:54,442 --> 00:07:58,655
And, so I think that that manual
QA can be done by stakeholders,

149
00:07:58,655 --> 00:08:00,612
like Doug was talking about.

150
00:08:00,612 --> 00:08:03,293
People who are interested in this is
the feature that I requested, and

151
00:08:03,293 --> 00:08:05,510
it's coming alive and
I'm going to test it and I know it.

152
00:08:05,510 --> 00:08:07,300
I think that that's
a good manual QA place.

153
00:08:07,300 --> 00:08:11,450
I think there are definitely people,
there are definitely companies that still

154
00:08:11,450 --> 00:08:15,880
have an entire QA team dedicated
where the thing will come to them and

155
00:08:15,880 --> 00:08:17,220
they will do manual tests.

156
00:08:17,220 --> 00:08:21,658
And they'll have paper check boxes even,
going through that sort of thing.

157
00:08:21,658 --> 00:08:26,640
>> To describe like these are our test
cases and when we do a testing cycle,

158
00:08:26,640 --> 00:08:29,630
we split these up and
we are gonna run through these actions.

159
00:08:29,630 --> 00:08:34,022
>> Yeah, and I think,
I mean I've seen a very like you know,

160
00:08:34,022 --> 00:08:39,493
there's definitely automated QA,
but I have seen very successfully,

161
00:08:39,493 --> 00:08:45,341
sites that are going live multiple
times a day, cuz continuous deployment.

162
00:08:45,341 --> 00:08:49,540
There's a, I need to make sure that
the thing that makes us money is working.

163
00:08:49,540 --> 00:08:50,130
Yes, it does.

164
00:08:50,130 --> 00:08:50,935
Okay, good, go ahead.

165
00:08:50,935 --> 00:08:52,200
>> [LAUGH]
>> The deploy's good.

166
00:08:52,200 --> 00:08:55,450
Yeah, that sort of manual QA I
think is always, probably,

167
00:08:55,450 --> 00:08:58,968
I don't know of anybody really
who's ever fully given that up.

168
00:08:58,968 --> 00:08:59,870
Hey, the site went live.

169
00:08:59,870 --> 00:09:02,090
We need to make sure
that that thing that's-

170
00:09:02,090 --> 00:09:03,860
>> Putting eyeballs on the thing, and

171
00:09:03,860 --> 00:09:06,040
actually you get that sense of okay, yeah,

172
00:09:06,040 --> 00:09:08,120
I can see that it's working and
we're good to go.

173
00:09:09,270 --> 00:09:13,634
So Nate, what are some of the pains
of doing this manual process?

174
00:09:13,634 --> 00:09:17,360
That sounds like kind of arduous work.

175
00:09:17,360 --> 00:09:20,490
>> Yeah, so the major pains
are just the repetitiveness of it.

176
00:09:20,490 --> 00:09:24,340
If you're working with a product that's in
multiple browsers on multiple operating

177
00:09:24,340 --> 00:09:26,920
systems, you have to literally
do the exact same thing over and

178
00:09:26,920 --> 00:09:32,750
over again across all, possibly across
all permutations of these platforms.

179
00:09:32,750 --> 00:09:37,780
Some of the pains are just the amount
of output you might end up with.

180
00:09:37,780 --> 00:09:41,580
I've worked for
a major company that's FDA regulated, and

181
00:09:41,580 --> 00:09:43,390
I can tell you exactly what
the QA testing looked like,

182
00:09:43,390 --> 00:09:44,940
because there's this half-ream of paper-
>> [LAUGH]

183
00:09:44,940 --> 00:09:46,190
>> Because it had to be printed out,

184
00:09:46,190 --> 00:09:47,338
signed
>> Wow.

185
00:09:47,338 --> 00:09:49,950
>> With actual signatures,
verifying that we did the work.

186
00:09:49,950 --> 00:09:53,600
And that these individuals are responsible
to say that this is quality.

187
00:09:53,600 --> 00:09:54,720
And it's filed with the FDA.

188
00:09:54,720 --> 00:09:57,580
So every single test-
>> As you were doing test cases,

189
00:09:57,580 --> 00:10:01,490
you were signing that saying that
saying I tested this and it passed.

190
00:10:01,490 --> 00:10:02,170
>> Yeah.

191
00:10:02,170 --> 00:10:03,300
>> Wow, okay.

192
00:10:03,300 --> 00:10:04,580
It was all done at the end but.

193
00:10:04,580 --> 00:10:05,410
>> Right.

194
00:10:05,410 --> 00:10:06,400
>> Yeah.

195
00:10:06,400 --> 00:10:10,780
>> So how bad can bad get when
you are doing manual testing?

196
00:10:10,780 --> 00:10:15,870
Does anyone have a horror story,
in terms of what that cycle looked like?

197
00:10:15,870 --> 00:10:20,688
>> So same company I work for, I worked
on a product that was in 14 languages,

198
00:10:20,688 --> 00:10:23,250
deployed not only in those languages but

199
00:10:23,250 --> 00:10:26,360
in regionalizations that
differ per language.

200
00:10:26,360 --> 00:10:30,157
So you're dealing with,
already just localization and

201
00:10:30,157 --> 00:10:32,698
regionalization of 30 plus cases.

202
00:10:32,698 --> 00:10:37,586
And then on top of that,
operating systems and browsers,

203
00:10:37,586 --> 00:10:42,378
so there were permutations
across 100 plus variables

204
00:10:42,378 --> 00:10:46,592
that literally took months
of testing by a group.

205
00:10:46,592 --> 00:10:51,747
That was not small, probably 20 at
its smallest to 50 at our biggest,

206
00:10:51,747 --> 00:10:56,418
of people just working months at
a time to verify this for release.

207
00:10:56,418 --> 00:10:57,822
And we're FD regulated, so

208
00:10:57,822 --> 00:11:00,811
we're always on the heavy side
of what we need to sign off on.

209
00:11:00,811 --> 00:11:04,798
But by the time we're done testing,
the next product is ready to ship, and

210
00:11:04,798 --> 00:11:06,520
we have to start all over again.

211
00:11:06,520 --> 00:11:07,144
>> And repeat?

212
00:11:07,144 --> 00:11:08,880
>> This will repeat the cycle.

213
00:11:08,880 --> 00:11:10,650
>> Wow, I can imagine that with,

214
00:11:10,650 --> 00:11:13,450
for instance,
it's like mobile devices these days.

215
00:11:13,450 --> 00:11:17,194
The permeations that you have to test
just keep expanding and expanding.

216
00:11:17,194 --> 00:11:19,326
It's not probably even a linear increase,
right,

217
00:11:19,326 --> 00:11:21,176
it's probably exponentially increasing.

218
00:11:21,176 --> 00:11:23,210
>> [LAUGH]
>> Every time you add a device and

219
00:11:23,210 --> 00:11:25,900
it has any sort of flexibility,
different multiple

220
00:11:25,900 --> 00:11:30,980
browsers on the same Android device,
that's already dozens of permutations.

221
00:11:30,980 --> 00:11:32,842
>> So let's talk about automation then,

222
00:11:32,842 --> 00:11:35,910
because we've mentioned it
a little bit here and there.

223
00:11:35,910 --> 00:11:39,790
Automation can help a lot,
but what does that look like?

224
00:11:39,790 --> 00:11:43,020
What are we talking about when we say we
are gonna automate the QA testing process?

225
00:11:44,720 --> 00:11:46,070
>> Well, it can really vary.

226
00:11:46,070 --> 00:11:50,650
It depends on what you have as a product
and where you wanna go with it.

227
00:11:50,650 --> 00:11:55,350
If you have the people, you can dedicate a
team to just start going through your mail

228
00:11:55,350 --> 00:12:00,750
processes and writing code to just
repetitively knock out these cases.

229
00:12:00,750 --> 00:12:04,830
But a lot of companies don't have
the staff to have a dedicated team, so

230
00:12:04,830 --> 00:12:10,240
you've got a small, manual group of
people who are in their spare time,

231
00:12:10,240 --> 00:12:11,990
which doesn't really
exist in the first place,

232
00:12:11,990 --> 00:12:17,120
trying to automate some stuff from
the side to get through their workflow.

233
00:12:17,120 --> 00:12:21,560
>> Right, that's if you have an existing
application with an established QA

234
00:12:21,560 --> 00:12:26,340
process, what you're suggesting is you
kind of almost moonlight this switch

235
00:12:26,340 --> 00:12:31,060
over to automated testing, and you just
kind of slowly chip away at it and

236
00:12:31,060 --> 00:12:33,026
maybe you get some momentum, eventually.

237
00:12:33,026 --> 00:12:37,000
>> It's certainly not an ideal, but it's a
situation I definitely have seen multiple

238
00:12:37,000 --> 00:12:39,660
times where you just don't have
the staff to dedicate to it.

239
00:12:39,660 --> 00:12:42,830
So you have to just
slowly eke your way out.

240
00:12:42,830 --> 00:12:48,070
And you focus on your worst case,
your biggest time consuming tests and

241
00:12:48,070 --> 00:12:49,860
automate those out.

242
00:12:49,860 --> 00:12:51,650
Then with the time that saves you,

243
00:12:51,650 --> 00:12:55,000
hopefully you can write some
more tests that saves more time.

244
00:12:55,000 --> 00:12:57,510
And slowly and slowly, you chip away at

245
00:12:57,510 --> 00:13:00,840
this massive block of testing until
you have something that's manageable.

246
00:13:00,840 --> 00:13:04,839
>> So it sounds like, regardless of where
you're at in sort of the lifetime of,

247
00:13:04,839 --> 00:13:08,221
even if you're just starting a new
application or a new project or

248
00:13:08,221 --> 00:13:11,790
even if it's a legacy application
that's been being maintained for

249
00:13:11,790 --> 00:13:16,210
a long period of time, that automation
can help in all those cases.

250
00:13:16,210 --> 00:13:18,940
>> Yeah, for sure,
I think there's a lot of companies

251
00:13:18,940 --> 00:13:22,680
start off with a manual test group and
they do a lot of manual testing.

252
00:13:22,680 --> 00:13:26,550
And it's quick to get going with that,
right, you can get some quick results.

253
00:13:26,550 --> 00:13:29,500
But it's also kind of hard
to shift that momentum, once

254
00:13:29,500 --> 00:13:34,510
you have that established there might be
a bias against leaving a manual testing.

255
00:13:34,510 --> 00:13:38,870
Like Nathan was saying, it's hard to carve
out the time to dedicate to automated

256
00:13:38,870 --> 00:13:42,605
testing, and having the skill set
on your team to write that code.

257
00:13:42,605 --> 00:13:46,140
But I think he is absolutely right that
you just have to start chipping away at

258
00:13:46,140 --> 00:13:49,610
some of your mission critical pieces
of functionality that have to work.

259
00:13:49,610 --> 00:13:54,010
And really focusing on those things that
are super time consuming, or prone to

260
00:13:54,010 --> 00:13:58,205
error to free up your manual testers to
work on other areas of the application.

261
00:13:58,205 --> 00:14:02,660
And use the automation as an accelerator
for your testing efforts.

262
00:14:02,660 --> 00:14:06,730
>> For sure, and on that I've never met
a manual tester who doesn't want to do

263
00:14:06,730 --> 00:14:07,520
automation testing.

264
00:14:07,520 --> 00:14:11,570
So it's like always a benefit of you're
empowering people learn how to code and

265
00:14:11,570 --> 00:14:15,800
they're very excited usually about
learning how to do it as long as it's,

266
00:14:15,800 --> 00:14:18,060
I think you can chip away at that.

267
00:14:18,060 --> 00:14:21,220
>> So to be clear, when we're
talking about this automation piece,

268
00:14:21,220 --> 00:14:22,910
we're gonna leverage some sort of tool.

269
00:14:22,910 --> 00:14:26,660
And there's probably a variety of
tools that you can use to do this.

270
00:14:26,660 --> 00:14:29,250
But you're actually talking about,
for instance,

271
00:14:29,250 --> 00:14:34,310
if it's a web-based application, you're
automating browsing from page to page or

272
00:14:34,310 --> 00:14:37,510
from screen to screen,
clicking buttons, filling out fields.

273
00:14:37,510 --> 00:14:38,850
Is that what we're talking about?

274
00:14:38,850 --> 00:14:40,260
>> Yeah, definitely.

275
00:14:40,260 --> 00:14:42,670
>> So to set up those test cases,

276
00:14:42,670 --> 00:14:45,560
typically you're writing code to do that,
is that true?

277
00:14:45,560 --> 00:14:47,682
>> That's true.
>> Are there some tools that,

278
00:14:47,682 --> 00:14:50,360
I feel like I've have heard of tools.

279
00:14:50,360 --> 00:14:51,690
I don't know if that's Cucumber or

280
00:14:51,690 --> 00:14:54,860
what I'm thinking of,
that are more natural language?

281
00:14:54,860 --> 00:14:58,260
>> So yeah, Cucumber is what they
call a behavioral driven design tool,

282
00:14:58,260 --> 00:15:02,330
where you literally write out
just natural human sentences.

283
00:15:02,330 --> 00:15:07,720
And basically under the hood those
sentences are interpreted into

284
00:15:07,720 --> 00:15:09,390
automated steps with the tool.

285
00:15:09,390 --> 00:15:13,950
But Cucumber particularly I really
enjoyed because you could get your

286
00:15:13,950 --> 00:15:16,731
people who knew business of your product,

287
00:15:16,731 --> 00:15:20,607
like pure business analysts
who didn't understand code.

288
00:15:20,607 --> 00:15:25,622
You could get them to write things out
in a sort of verbose way that could

289
00:15:25,622 --> 00:15:31,095
easily transfer into Cucumber steps
which a test engineer could then take.

290
00:15:31,095 --> 00:15:33,050
>> Yeah.
>> And automate the whole system.

291
00:15:33,050 --> 00:15:37,640
But the big benefit is that it
leaves the tests in a state that

292
00:15:37,640 --> 00:15:41,690
any person can just read them and
exactly understand what everything does.

293
00:15:41,690 --> 00:15:43,510
You don't have to parse code.

294
00:15:43,510 --> 00:15:45,800
It's literally just reading sentences.

295
00:15:45,800 --> 00:15:52,050
When this, then that,
Given X, when Y, then Z.

296
00:15:52,050 --> 00:15:53,880
>> Right.
>> So it's a structured language, but

297
00:15:53,880 --> 00:15:57,630
it's natural and broad enough that
people don't feel too confined, and

298
00:15:57,630 --> 00:16:00,530
they're not looking at curly braces and
semicolons for instance,

299
00:16:00,530 --> 00:16:03,320
and getting kind of hung
up on what that looks like.

300
00:16:03,320 --> 00:16:04,800
>> Yeah.
>> I've seen that used for

301
00:16:04,800 --> 00:16:06,700
a great place for business logic,
where you're like,

302
00:16:06,700 --> 00:16:08,800
what happens when you're logged in and
you can't.

303
00:16:08,800 --> 00:16:09,490
Are you supposed to be able to click that?

304
00:16:09,490 --> 00:16:14,000
And you're like, well, let's read
the test together as a product team.

305
00:16:14,000 --> 00:16:16,510
It's pretty neat to look-
>> The testing documentation is a really

306
00:16:16,510 --> 00:16:17,154
powerful tool.

307
00:16:17,154 --> 00:16:18,555
>> Yeah.
>> Interesting, so

308
00:16:18,555 --> 00:16:24,406
is that a common way of doing this, or
is that still a little bit more fringey,

309
00:16:24,406 --> 00:16:27,979
in terms of how to go about-
>> In five different places I've worked,

310
00:16:27,979 --> 00:16:28,732
we only did it once.

311
00:16:28,732 --> 00:16:29,244
>> Okay.

312
00:16:29,244 --> 00:16:31,094
[LAUGH]
>> And I loved it, but

313
00:16:31,094 --> 00:16:34,668
that was also the only place where I
was able to come in at the start and

314
00:16:34,668 --> 00:16:37,020
was heavily involved in decision process.

315
00:16:37,020 --> 00:16:40,310
Everywhere else I've worked,
I come in late, and

316
00:16:40,310 --> 00:16:42,850
I have to work with what already exists.

317
00:16:42,850 --> 00:16:45,670
>> Right, so you might end up with
more of a hybrid approach then.

318
00:16:45,670 --> 00:16:48,560
So if you have a blended
team of some developers and

319
00:16:48,560 --> 00:16:52,250
some business experts, stakeholders,

320
00:16:52,250 --> 00:16:56,260
demand experts, whatever the term that
we're using there, collaborating.

321
00:16:56,260 --> 00:16:59,180
So you still might have some manual
process where the people who

322
00:16:59,180 --> 00:17:01,932
aren't able to write code
are responsible for that.

323
00:17:01,932 --> 00:17:06,300
Maybe they're also working shoulder to
shoulder with the developers to write or

324
00:17:06,300 --> 00:17:09,520
do the programming to write
the automated portions.

325
00:17:09,520 --> 00:17:13,090
Is that kind of a fair way to
describe how that can work?

326
00:17:13,090 --> 00:17:14,310
It certainly can work that way.

327
00:17:14,310 --> 00:17:16,380
I mean,
there's no really one way to do this,

328
00:17:16,380 --> 00:17:18,870
with given your staff, what you have.

329
00:17:18,870 --> 00:17:21,540
If you don't have
technical people except for

330
00:17:21,540 --> 00:17:24,740
your developers, then basically
developers have to write the main task.

331
00:17:24,740 --> 00:17:27,330
Otherwise you're not gonna
have automated tasks.

332
00:17:27,330 --> 00:17:31,443
>> Yeah, I would say that in general
from a QA perspective, you wanna be one,

333
00:17:31,443 --> 00:17:34,465
transparent with all your testing and
testing tools.

334
00:17:34,465 --> 00:17:34,993
But two,

335
00:17:34,993 --> 00:17:39,236
have a system in place that everyone
can participate in the testing effort.

336
00:17:39,236 --> 00:17:42,438
So it's not just responsibility if
you have a dedicated QA staff to

337
00:17:42,438 --> 00:17:45,525
do all the testing,
you wanna bring in your development staff,

338
00:17:45,525 --> 00:17:47,600
your product managers, if you have them.

339
00:17:47,600 --> 00:17:51,240
Customer support is a great resource for

340
00:17:51,240 --> 00:17:54,190
helping to test because they know
where all the problem areas are.

341
00:17:54,190 --> 00:17:59,890
And so, but I agree with Nathan that
you use the right tool for the job.

342
00:17:59,890 --> 00:18:03,222
So you don't have to be locked into one
tool and try to apply that everywhere.

343
00:18:03,222 --> 00:18:04,620
It's gonna be a combination of tools,

344
00:18:04,620 --> 00:18:06,600
depending on what it is
you are trying to test.

345
00:18:06,600 --> 00:18:08,714
>> Yeah, no golden hammer, so to speak.

346
00:18:08,714 --> 00:18:09,339
>> Yeah, right.

347
00:18:09,339 --> 00:18:12,577
>> One of the gateway tools
that I've seen work is there's,

348
00:18:12,577 --> 00:18:14,925
I think it's Selenium that produces it.

349
00:18:14,925 --> 00:18:16,970
It's a little browser plug-in and
you click it.

350
00:18:16,970 --> 00:18:20,310
It just kind of records your clicks as you
go, and then it generates a little, so

351
00:18:20,310 --> 00:18:24,040
like this is what you did and
code below and it's a thing that you

352
00:18:24,040 --> 00:18:26,680
can take a look and go, I can see how
we could put this across the side.

353
00:18:26,680 --> 00:18:27,874
So that's a great sales tool.

354
00:18:27,874 --> 00:18:32,493
I've seen that convert teams from maybe we
shouldn't do manual testing if the manual

355
00:18:32,493 --> 00:18:34,644
tests can produce these sets of scripts.

356
00:18:34,644 --> 00:18:36,467
I mean, it just kind of works across.

357
00:18:36,467 --> 00:18:38,960
Right, having a recorder I
can imagine it would be huge,

358
00:18:38,960 --> 00:18:41,890
cuz I can totally see how you're
gonna say, hey, I'm just going to

359
00:18:41,890 --> 00:18:45,930
install this thing and have you go through
what you would normally do manually.

360
00:18:45,930 --> 00:18:48,050
But we're also gonna capture some of that,
and

361
00:18:48,050 --> 00:18:51,340
I'm imagine there's some finessing
maybe you need to do here and there.

362
00:18:52,480 --> 00:18:55,562
>> Sure, yeah, if you would record the
script that's gonna use the same value you

363
00:18:55,562 --> 00:18:57,820
typed in every time, so
you'll wanna randomize that.

364
00:18:57,820 --> 00:19:00,406
It definitely is a great
jumpstart to get the script going.

365
00:19:00,406 --> 00:19:03,540
>> In general, I found it's fine for
a starting point,

366
00:19:03,540 --> 00:19:05,450
but that is not gonna be your goal.

367
00:19:05,450 --> 00:19:09,600
These are core and playback scripts that
have the weakness that if there's any sort

368
00:19:09,600 --> 00:19:14,460
of logical changes in your code, they
quickly become more work to maintain than

369
00:19:14,460 --> 00:19:19,115
it would be if you had a more
abstract implementation

370
00:19:19,115 --> 00:19:24,150
through something like Selenium or
Cucumber.

371
00:19:24,150 --> 00:19:27,847
There's all kinds of testing frameworks
for just about every language and

372
00:19:27,847 --> 00:19:29,264
every framework out there.

373
00:19:29,264 --> 00:19:33,198
>> Okay, we've kind of been, or at least
in my mind anyway during this discussion,

374
00:19:33,198 --> 00:19:36,505
I keep thinking web applications
because that's what I have the most

375
00:19:36,505 --> 00:19:37,470
experience with.

376
00:19:37,470 --> 00:19:41,467
But I would assume that even if you are
doing like mobile development whether that

377
00:19:41,467 --> 00:19:45,229
be native or react native or something
else, or even desktop development,

378
00:19:45,229 --> 00:19:49,463
for instance, that there's probably tools
that will assist with automating that QA

379
00:19:49,463 --> 00:19:50,730
testing as well?

380
00:19:50,730 --> 00:19:54,367
>> Yeah, absolutely, I mean, there's
emulators on the mobile side that helps

381
00:19:54,367 --> 00:19:57,040
you, and
you can apply automation tools to those.

382
00:19:57,040 --> 00:19:59,840
On the desktop side, it's been a long
time since I worked in that area,

383
00:19:59,840 --> 00:20:03,033
but there are tools like Silk and
whatnot that Basically,

384
00:20:03,033 --> 00:20:07,160
much like Selenium allows you to interact
with the UI and drive the application,

385
00:20:07,160 --> 00:20:09,610
as if a user was sitting right there
at the keyboard interacting with you.

386
00:20:09,610 --> 00:20:11,800
>> That makes total sense, okay.

387
00:20:11,800 --> 00:20:14,140
So help me out here.

388
00:20:14,140 --> 00:20:18,520
You're investing in writing code, so
it's kinda another code base that's

389
00:20:18,520 --> 00:20:22,710
to the side of the code base in your
application that you're working on.

390
00:20:22,710 --> 00:20:25,260
The application's not gonna be static.

391
00:20:25,260 --> 00:20:27,540
It's gonna continue to grow and
change and evolve over time.

392
00:20:27,540 --> 00:20:29,260
What are we talking about here?

393
00:20:29,260 --> 00:20:34,765
Are we taking on, is that overhead gonna
end up killing us in the long run?

394
00:20:34,765 --> 00:20:36,250
And is there ways to help manage that?

395
00:20:37,310 --> 00:20:40,120
>> So first, the overhead testing,

396
00:20:40,120 --> 00:20:45,630
you're talking something that we're
using to minimize the cost of testing.

397
00:20:45,630 --> 00:20:49,110
So as long as you're focused on quality,
automation

398
00:20:50,150 --> 00:20:54,460
done correctly is only going to minimize
the amount of work you have to do.

399
00:20:54,460 --> 00:20:57,430
You will have to do some work
to maintain your testing suite.

400
00:20:57,430 --> 00:21:02,160
But now, moving forward, you're running
these tests every time you add a feature.

401
00:21:02,160 --> 00:21:05,770
So these tests, if they're gonna break,
tou're gonna see what they're gonna break,

402
00:21:05,770 --> 00:21:08,050
you're gonna fix those tests,
and you're gonna move forward.

403
00:21:08,050 --> 00:21:09,770
So it shouldn't ever really get stale and

404
00:21:09,770 --> 00:21:13,280
out of touch with your code base,
it should evolve with it.

405
00:21:13,280 --> 00:21:15,390
>> Right, so
there's going to be maintenance.

406
00:21:15,390 --> 00:21:19,280
But I guess it's fair to say that even if
you're doing manual testing, those manual

407
00:21:19,280 --> 00:21:24,496
testers would have to update written use
test cases and I relearn the application.

408
00:21:24,496 --> 00:21:29,310
So it's kinda just repurposing that
effort into maintaining code and to your

409
00:21:29,310 --> 00:21:34,180
point there is probably a large enough
net gain savings from automating this,

410
00:21:34,180 --> 00:21:40,500
that It's hard to exceed that in terms of
maintenance, and from a code perspective.

411
00:21:40,500 --> 00:21:43,170
>> Yeah, totally, it's on the proactive,
you're gonna pay for

412
00:21:43,170 --> 00:21:44,471
it on the reactive side of it, right?

413
00:21:44,471 --> 00:21:45,220
[LAUGH]
>> Right?

414
00:21:45,220 --> 00:21:47,170
>> Like if these bugs are showing up,
you're gonna be paying for

415
00:21:47,170 --> 00:21:47,700
it later anyway.

416
00:21:47,700 --> 00:21:49,810
Development trying to go and
catch that and fix that.

417
00:21:49,810 --> 00:21:52,200
And then fixing that, like,
did I break something else over here?

418
00:21:52,200 --> 00:21:54,100
>> Right.
>> But I've see these test runs go

419
00:21:54,100 --> 00:21:58,195
where you like, I just change that one
field and then you break 30 tests,

420
00:21:58,195 --> 00:22:01,710
this form was used all over the places,
and this one little bit.

421
00:22:01,710 --> 00:22:04,784
I mean that just,
when you see that happen, you're like wow,

422
00:22:04,784 --> 00:22:08,554
this is great that did that, nobody
tested, nobody either sit there and test

423
00:22:08,554 --> 00:22:12,562
those or think about all these difference
scenarios that it's on the code there.

424
00:22:12,562 --> 00:22:13,230
>> Yeah.

425
00:22:13,230 --> 00:22:16,960
>> You got to the thing
there is you got to focus on

426
00:22:17,990 --> 00:22:21,430
you got to put effort the some
amount of effort into your testing

427
00:22:21,430 --> 00:22:23,830
that you do in your development
because it's important.

428
00:22:23,830 --> 00:22:25,400
It's important that your code is working.

429
00:22:25,400 --> 00:22:26,400
>> Right, that makes sense.

430
00:22:26,400 --> 00:22:29,160
>> I think you have to go into it knowing
that a little bit of planning goes

431
00:22:29,160 --> 00:22:29,680
a long way.

432
00:22:29,680 --> 00:22:34,170
So a lot of times in automation systems,
people will create frameworks.

433
00:22:34,170 --> 00:22:36,540
And so, you can have common functionality.

434
00:22:36,540 --> 00:22:39,844
That everyone can use in their scripts so,
you don't want everyone at

435
00:22:39,844 --> 00:22:43,540
the beginning of their script to write
the code to log in to the system, right.

436
00:22:43,540 --> 00:22:46,145
So you create some common code
that everybody can utilize and

437
00:22:46,145 --> 00:22:47,330
those things go a long way.

438
00:22:47,330 --> 00:22:50,700
And then again, when that changes
in the actual application,

439
00:22:50,700 --> 00:22:54,340
hopefully there's just one place that you
can change it, and so it minimizes that.

440
00:22:54,340 --> 00:22:55,950
So, a little bit of
planning goes a long way.

441
00:22:55,950 --> 00:22:57,510
>> That makes a lot of sense.

442
00:22:57,510 --> 00:22:59,900
So imagine I'm managing a deaf team.

443
00:22:59,900 --> 00:23:03,540
Maybe I'm managing the product manager or
I'm some decisions or

444
00:23:03,540 --> 00:23:06,176
some control over how we move ahead here.

445
00:23:06,176 --> 00:23:09,070
We've been doing what we've been
doing with manual testing for

446
00:23:09,070 --> 00:23:11,230
a long time and
you've come to me and said,

447
00:23:11,230 --> 00:23:14,780
hey we've got to automate some of this,
but maybe I'm not buying it.

448
00:23:14,780 --> 00:23:18,380
Maybe I'm interested, but
I'm like I'm unsure if this

449
00:23:18,380 --> 00:23:21,880
really gonna have the return on
the investment that we need to make.

450
00:23:21,880 --> 00:23:23,780
Do you have any other
stories that you can share,

451
00:23:23,780 --> 00:23:27,120
where you've seen that sort of,
you're at a decision point here.

452
00:23:27,120 --> 00:23:29,890
And I like to hear a success story.

453
00:23:29,890 --> 00:23:31,860
>> Sure, yeah, I've got a good story.

454
00:23:31,860 --> 00:23:34,992
I think a lot of times,
if you're facing that type of question,

455
00:23:34,992 --> 00:23:37,950
all you have to do is look towards
real life and what happens.

456
00:23:37,950 --> 00:23:43,340
And one of my previous companies,
we had a large ERP system,

457
00:23:43,340 --> 00:23:47,730
and it was a system where multiple users
can be in there at one time, so like 40,

458
00:23:47,730 --> 00:23:49,240
50, 60 people at one time.

459
00:23:49,240 --> 00:23:53,610
And a bug was reported that occasionally
a race condition would occur, so

460
00:23:53,610 --> 00:23:56,770
there'd be some type of contention over
a particular record in the database or

461
00:23:56,770 --> 00:24:00,910
what have you It was very hard to
reproduce, but it was very impactful,

462
00:24:00,910 --> 00:24:04,640
that when it occurred it crashed
the system and everyone got knocked out.

463
00:24:04,640 --> 00:24:07,460
So a very important bug.

464
00:24:07,460 --> 00:24:11,140
We tried to reproduce this manually,
meaning getting our dev and QA staff and

465
00:24:11,140 --> 00:24:12,030
customer support staff,

466
00:24:12,030 --> 00:24:15,790
so very expensive, it took a long
time to get set up to begin with.

467
00:24:15,790 --> 00:24:16,580
Trying to reproduce it.

468
00:24:16,580 --> 00:24:19,130
Once we reproduced it,
it was wait a minute.

469
00:24:19,130 --> 00:24:22,180
We didn't have enough debugging
going on in the system.

470
00:24:22,180 --> 00:24:24,890
We didn't have enough logging to really
understand what the problem was.

471
00:24:24,890 --> 00:24:26,180
So we gotta do this all again, right?

472
00:24:26,180 --> 00:24:28,790
So hours and hours and
hours of investment.

473
00:24:28,790 --> 00:24:31,570
Fortunately, we had the foresight
of on the side having

474
00:24:31,570 --> 00:24:35,210
engineers writing some automated
tests to do this exact same thing and

475
00:24:35,210 --> 00:24:39,690
eventually we were able to reproduce it
more frequently and with less effort.

476
00:24:39,690 --> 00:24:42,570
And so it started paying for
itself right there.

477
00:24:42,570 --> 00:24:46,402
But then,
beyond that once the fix was in place,

478
00:24:46,402 --> 00:24:52,198
we could those same scripts and
let them run 24/7 for an entire week and

479
00:24:52,198 --> 00:24:56,610
make sure that that risk
condition did not occur again.

480
00:24:56,610 --> 00:25:00,640
And then, a third point is that that could
then be added to our test suite, and

481
00:25:00,640 --> 00:25:02,454
run every time we do that release.

482
00:25:02,454 --> 00:25:02,973
>> Right.

483
00:25:02,973 --> 00:25:07,364
>> So something that we didn't have
in place that took some time to do,

484
00:25:07,364 --> 00:25:12,063
really got rid of the need to have this
massive manual testing effort that

485
00:25:12,063 --> 00:25:13,970
just wasn't sustainable.

486
00:25:13,970 --> 00:25:17,090
There's no way you could do that on a
regular basis and you certainly went more

487
00:25:17,090 --> 00:25:20,770
running for 24/7 for a week to
make sure the bug was really gone.

488
00:25:20,770 --> 00:25:25,594
So, automation and just solve these type
of volume and those are issues that you

489
00:25:25,594 --> 00:25:30,274
are trying to deal with when you are
talking about people and only been working

490
00:25:30,274 --> 00:25:34,636
for eighteen hours a day, [LAUGH]
>> Yeah, having done a lot of manual QA

491
00:25:34,636 --> 00:25:40,178
test, specially first involve the domain
experts it would really nice to know for

492
00:25:40,178 --> 00:25:45,395
a fact that when you run on your automated
QA test that they actually run, what

493
00:25:45,395 --> 00:25:51,320
would be the results instead of having
fingers cross that everyone is doing.

494
00:25:51,320 --> 00:25:55,509
In lieu of like having signed
pieces of paper that said I read,

495
00:25:55,509 --> 00:25:57,290
I did this thing, right?

496
00:25:57,290 --> 00:25:57,928
>> Right.
>> And

497
00:25:57,928 --> 00:26:02,073
that would be really great as from
a developer's perspective just having that

498
00:26:02,073 --> 00:26:03,970
sort of, like you know that it ran.

499
00:26:03,970 --> 00:26:05,305
You know whether it passed or failed.

500
00:26:05,305 --> 00:26:07,120
>> That's some really interesting
tools that exist now.

501
00:26:07,120 --> 00:26:10,020
Like I know we use coveralls
where I work right now.

502
00:26:10,020 --> 00:26:12,820
And it actually is just a website that our

503
00:26:12,820 --> 00:26:17,130
test coverage goes to like our test
run and it generates a report.

504
00:26:17,130 --> 00:26:19,317
That report gets hosted on coveralls so

505
00:26:19,317 --> 00:26:22,510
we can go back at any moment of
any day of any build really.

506
00:26:22,510 --> 00:26:27,560
We can see what was test coverage,
what test ran, what was the results.

507
00:26:27,560 --> 00:26:31,939
It's continuing to evolve, but
it's amazing what convenience

508
00:26:31,939 --> 00:26:35,610
is being built into the testing
frame work these days.

509
00:26:35,610 --> 00:26:40,180
>> Yeah, has anyone had the experience
of implementing automated QA testing and

510
00:26:40,180 --> 00:26:44,390
actually can tie that or
associate that to happier users,

511
00:26:44,390 --> 00:26:48,510
like users feeling like things are more
stable or that the product is better?

512
00:26:48,510 --> 00:26:53,240
Is that something, you been able to
see that association so strongly?

513
00:26:53,240 --> 00:26:56,900
>> I think it may be
a qualitative association.

514
00:26:56,900 --> 00:27:00,466
I think when your QA staff is sleeping
well because they know that the test

515
00:27:00,466 --> 00:27:01,930
are all passing, [LAUGH]
>> [LAUGH]

516
00:27:01,930 --> 00:27:05,180
>> I think that alone is worth the money.

517
00:27:05,180 --> 00:27:09,060
But certainly, like Nathan was saying,
when you have the transparency into

518
00:27:09,060 --> 00:27:12,590
the test results, and you can rerun
these tests whenever you need to.

519
00:27:12,590 --> 00:27:16,246
And even things where you're at the end
of a release cycle, and all of a sudden

520
00:27:16,246 --> 00:27:19,690
you've got to fix a bug, you have time
to rerun all your tests manually.

521
00:27:19,690 --> 00:27:22,310
Or maybe a new operating
system version comes out or

522
00:27:22,310 --> 00:27:25,240
new browser version comes out, and
you've got to rerun these tests.

523
00:27:25,240 --> 00:27:28,060
Well, nobody really plans for
those sorts of things, and so

524
00:27:28,060 --> 00:27:28,970
you don't have time for it.

525
00:27:28,970 --> 00:27:33,273
And so, there's a lot of peace of mind
knowing that a new browser just came out,

526
00:27:33,273 --> 00:27:35,908
we can do a full regression
of our application and

527
00:27:35,908 --> 00:27:37,660
have confidence that it works.

528
00:27:37,660 --> 00:27:41,590
Or if there is a bug, we're gonna be the
first ones to find it another end users.

529
00:27:41,590 --> 00:27:42,195
>> That is great.

530
00:27:42,195 --> 00:27:43,150
>> Yeah.

531
00:27:43,150 --> 00:27:46,732
>> I saw right there, you hit on something
there about I think that it also allows

532
00:27:46,732 --> 00:27:48,450
for better release cycles, right?

533
00:27:48,450 --> 00:27:51,783
So if you're asking specifically about
what your users are expecting, yeah,

534
00:27:51,783 --> 00:27:54,880
you can release more quickly because
you know that everything's there.

535
00:27:54,880 --> 00:27:59,402
You don't have the QA period
they're of traditional waterfall,

536
00:27:59,402 --> 00:28:04,165
like you had said, where you can
release a new feature on Wednesday and

537
00:28:04,165 --> 00:28:07,900
Thursday and Friday,
because the tests are working.

538
00:28:07,900 --> 00:28:10,534
You get in the habit of writing
those tests alongside of it and

539
00:28:10,534 --> 00:28:13,997
you know that that's going and you can
show that the business logic is working to

540
00:28:13,997 --> 00:28:16,652
everybody who's concerned about
that feature [CROSSTALK].

541
00:28:16,652 --> 00:28:18,046
>> So that's interesting.

542
00:28:18,046 --> 00:28:21,890
So you have an issue that says, at this
feature and part of completing that

543
00:28:21,890 --> 00:28:25,424
feature is, you have the automated
QA test that goes along with it,

544
00:28:25,424 --> 00:28:29,250
along with your unit test from
a development perspective, and so on.

545
00:28:29,250 --> 00:28:30,460
>> Right.

546
00:28:30,460 --> 00:28:31,210
>> Yeah, that's great.

547
00:28:32,630 --> 00:28:35,530
So for anyone who's now convinced,

548
00:28:35,530 --> 00:28:40,910
I have to automate my QA testing,
any advice on how to get started?

549
00:28:40,910 --> 00:28:45,510
>> Well, I guess one of the things I would
just reiterate or emphasize is again,

550
00:28:45,510 --> 00:28:49,660
if you have a QA team it's not
just the QA teams responsibility.

551
00:28:49,660 --> 00:28:54,360
Bring everybody into the effort,
development certainly, product management,

552
00:28:54,360 --> 00:28:55,470
customer support.

553
00:28:55,470 --> 00:28:56,570
Develop a system or

554
00:28:56,570 --> 00:29:00,510
a methodology where everybody can
help add tests to the system.

555
00:29:00,510 --> 00:29:03,770
Really try to make your tests repeatable,
get the benefit out of them.

556
00:29:03,770 --> 00:29:09,130
Make them so that they're tied into
your continuous integration system and

557
00:29:09,130 --> 00:29:10,210
really have them work for you.

558
00:29:10,210 --> 00:29:12,320
They're not things that you just
run at the end of the release.

559
00:29:12,320 --> 00:29:13,680
They're being run every day.

560
00:29:13,680 --> 00:29:16,350
Really take the advantage of
the machine power that you have.

561
00:29:16,350 --> 00:29:17,350
>> Nate?

562
00:29:17,350 --> 00:29:22,350
>> Yeah, I think the main thing is to
just look at your ROI on it, is just

563
00:29:22,350 --> 00:29:27,150
find what you biggest pinpoints are and
that's where you start and you just move

564
00:29:27,150 --> 00:29:31,900
down the list of worst case offenders for
what takes your time to validate.

565
00:29:31,900 --> 00:29:33,250
>> Okay, Craig?

566
00:29:33,250 --> 00:29:37,660
>> I think it's important to
respect the tests as code.

567
00:29:37,660 --> 00:29:39,920
The same way that you treat
your other code in the system.

568
00:29:39,920 --> 00:29:41,500
Show that same level of respect.

569
00:29:41,500 --> 00:29:42,704
And that-
>> Reviews.

570
00:29:42,704 --> 00:29:45,860
>> Yeah, yeah, totally,
and best practices, right.

571
00:29:45,860 --> 00:29:48,100
Hey, this is duplicated,
why are we duplicating?

572
00:29:48,100 --> 00:29:50,535
Let's figure out a way to make this,
let's make good fixtures,

573
00:29:50,535 --> 00:29:52,070
let's make it fun to write tests.

574
00:29:52,070 --> 00:29:57,036
That's something that happens too, I see
that, people, sometimes you'll get into

575
00:29:57,036 --> 00:30:00,860
it's kind of a mess to write this test and
I don't make it more fun.

576
00:30:00,860 --> 00:30:04,322
Take in the software the same
way that you would in and

577
00:30:04,322 --> 00:30:08,390
develop a component,
same thing with the test there.

578
00:30:08,390 --> 00:30:11,600
>> Excellent, well thanks for
joining the discussion today.

579
00:30:11,600 --> 00:30:13,160
Great discussion and thanks for

580
00:30:13,160 --> 00:30:16,870
sharing your experiences about QA
testing and QA testing automation.

581
00:30:16,870 --> 00:30:17,600
>> Great.
>> Thank you.

582
00:30:18,660 --> 00:30:22,360
>> And thanks for watching And
be sure to check the notes for this video,

583
00:30:22,360 --> 00:30:25,490
we'll have resources listed there for,
QA testing,

584
00:30:25,490 --> 00:30:30,670
to learn more about QA testing or to learn
more about automation of those QA tests.

585
00:30:30,670 --> 00:30:33,710
Also, be sure to rate the video and
let us know how we're doing.

586
00:30:33,710 --> 00:30:34,430
See you next time.

587
00:30:34,430 --> 00:30:38,290
[SOUND]