Reporting Quality5:20 with Ryan Saul
One of the most important parts of quality assurance is collecting lots of data and reporting that back to your company for decision making.
- Do we need more tests?
- Are we catching most of the bugs that we could be?
- What do our customers think of our product? Are they happy with it?
- Coverage report: How many tests belong to different parts of the software?
- Escaped bug report: A report of all bugs that have been reported by customers and not internally.
- User feedback report: What are customers saying about the software? Look for trends in feedback.
One of the most important parts of quality assurance is collecting lots of data, and 0:00 reporting that back to your company for decision making. 0:05 QA is a very data-driven role. 0:09 Ask yourself the following questions about where your product is right now. 0:12 [SOUND] Do we need more tests? 0:16 [SOUND] Are we catching most of the bugs that we could be? 0:19 [SOUND] What do our customers think of our product? 0:22 Are they happy with it? 0:25 These are important things to answer. 0:27 You need to let your company know how the software is performing 0:29 out in the real world, or if there's too many bugs. 0:32 Or if something isn't totally ready for release, so 0:36 that the company can make decisions based around that. 0:38 In this video we'll go over some easy reports that can help you do that and 0:42 answer those questions. 0:46 So the first report we'll cover is the coverage report. 0:48 This is the simple overview of your testing efforts. 0:51 The idea here is to show what tests are covering what parts of your application, 0:55 and how much more testing is needed to cover that gap. 1:00 So let's say we have an application that has three main parts. 1:03 Login, or authentication, creating cards on a board, and printing out those cards. 1:06 Currently, we have two tests for authentication, another four tests for 1:12 creating cards on the board, and we don't have any tests for printing out the cards. 1:16 This tells us that there's a lack of testing happening in one part of the app 1:21 and that we need to write some new tests before we can feel confident in it. 1:25 Another thing that the coverage report can show is where you're automation efforts 1:29 should be going. 1:32 If you track how long tests take to run, 1:34 then this will be a very easy decision to make. 1:36 The longest tests should probably be automated first 1:38 because they save you a lot of time. 1:42 But if you don't have that kind of data, 1:44 look at automating areas that require a lot of testing to complete. 1:46 This is a simple way to illustrate that data can be very powerful in helping 1:50 to make decisions. 1:55 Both for your business and for your own QA efforts. 1:56 You always have limited time. 2:00 So it's good to know where to spend it. 2:01 So now that the software has been released some customers are reporting bugs. 2:04 This is bad. 2:09 And we need to know about these bugs as part of the QA team. 2:10 The difference between you finding a bug before the software is released and 2:14 a customer finding a bug is huge. 2:18 Keep a tally of how many bugs are being found by the customers each month. 2:21 Log them just like bugs that are found in development. 2:26 These are escaped bugs that have found their way into the released product. 2:30 Fixing these is much more costly than fixing them in the key way process. 2:34 We want to put these bugs into an escaped bugs report 2:39 to show to the rest of the team. 2:42 This report basically tells you if you have some areas in the development or 2:45 testing process that need to be worked on. 2:49 This report will also help justify putting more investment into that. 2:52 We want to make customers feel more confident in their product 2:57 each time that there is a new release. 3:01 Further more, every time a bug gets released or 3:03 out into the wild, it costs much more than it would to fix it before it was released. 3:06 Finally, another simple report is gathering user feedback. 3:12 If you're unfamiliar with the Agile manifesto, 3:16 I recommend checking out the Treehouse course on Scrum basics which goes over it. 3:19 But a very important aspect of Agile is user feedback. 3:25 How does customer feedback fit into QA, though? 3:29 Basically this will inform if you are looking for 3:33 the right things in the software when it is being tested. 3:35 How simple are the workflows? 3:39 Are the buttons confusing? 3:42 Is the documentation helpful? 3:43 These are the types of things you can get from user feedback. 3:46 This is just a simple report that I put together with all 3:50 the feedback that we've gotten back from our customers on the RSVP app so far. 3:53 We aggregate the positive and negative feedback as well as other feedback for 3:58 those that didn't specify. 4:03 We also take in all of the feedback and 4:05 calculate the satisfied percentage by the amount of users giving 4:07 positive feedback divided by everyone else who has given feedback. 4:11 Your way of calculating this number may be a little different, and 4:15 you may not even really need this number for your purposes. 4:19 But I feel like it's a really nice way to show 4:22 how many users are satisfied by your product. 4:24 You can solicit customer feedback from a support or sales team. 4:28 What are your customers saying about the software? 4:33 This can be tricky. 4:36 When generating a user feedback report, 4:38 you wanna gather all the feedback you can get. 4:40 And look for patterns in what the customers are saying. 4:43 This isn't an easy task though. 4:47 It's often done by a product team, so 4:50 you might be able to get that sorta data from them. 4:52 In QA, we wanna know if our tests are looking for the right things and 4:56 if we need to revisit old tests. 4:59 This is why user feedback is great. 5:02 I really just want to go over some simple reports that you can keep 5:05 in your back pocket. 5:08 In reality, just about any piece of data you're collecting, 5:10 could probably find it's way into a useful report. 5:13 So, be on look out for things your company may find useful. 5:17
You need to sign up for Treehouse in order to download course files.Sign up