Coming up with a testing strategy doesn’t have to be stressful. Prioritizing features to test, and generating test cases for each feature can be fairly quick and painless. This episode covers a strategy for both that can be applied to many types of software.
Transcript for episode 150 of the Test & Code Podcast
This transcript starts as an auto generated transcript.
PRs welcome if you want to help fix any errors.
00:00:00 Today I want to talk about a general software testing strategy that works for like 80% of the cases I’ve encountered. I’m hoping this quick episode will save you time and get you developing test cases quickly.
00:00:15 This episode is brought to you by Python Term. There’s a cool new version out version 2021 One. The big new thing is called Code with Me, and it is a collaborative development feature. You control what people can do and see, but really you share a link with them and you can collaborate with them on some code. The remote person doesn’t even need to have an idea of their own installed to collaborate with you. Code With Me also includes embedded audio and video calls along with chat messaging. Bring pair programming back, or even just spend a few minutes with somebody else talking and working through some code issues. Very cool new feature. Pycharm is always looking out for ways to save us time. Visit Testingcode.com PyCharm to try this out, and you get four months of free PyCharm pro to try out all the cool features that will save your time.
00:01:21 Welcome to Testing Code.
00:01:32 People often stress out when they are either asked to write tests or even just on their own. They figure that they ought to. And unit tests are a different thing. And I don’t even want to touch that snake nest right now. I’m talking about feature tests or functionality tests.
00:01:48 Not only does this function do the thing, I think it’s supposed to do unit tests, but I’m talking about does the system work in the way it’s supposed to type of tests. These are the tests that everyone wants to have written and solid and working for all the software they use, but they don’t want to write them themselves for their own software. And I think it’s just because they don’t have an idea for the strategy for how to do it. So that’s the strategy we’re going to talk about today. Also, really, the strategies we’re going to talk to will should work for fine grain unit tests as well, or they may. Anyway, let’s jump in. There are two stages to this that I’m going to discuss. First, deciding which features or bits of functionality to test. Second, for each of those which test cases to implement, that’s it. So which features should you test? And don’t say test all the things, because that really doesn’t help with prioritization. If you’ve got time and the payoff is valuable, go for it. Test all the things, but you still have to decide on an order. So which features? I generally prioritize features to test based on the following factors. First, recent new features new areas of code new functionality that has been recently repaired or refactored or otherwise modified core your product’s unique selling propositions the essential functions that must continue to work in order for your product to be useful. Three risk areas of the application that pose more risk such as areas important to customers but not regularly used by development teams or parts that use third party code that you don’t quite trust. Fourth, problematic areas, functionality that frequently breaks or often gets defect reports against it. And fifth, expertise features are algorithms understood by a limited subset of people. So again, that was recent core risk problematic. And expertise, they’re not really in order themselves.
00:03:50 If something checks all of the boxes, well then test the hell out of it. And if something checks none of these, well, maybe put that feature off until later. Everything else is a judgment call on your part, of course. And let’s go through some examples to make this more solid. You’ve just added some code for a new feature. You should test it. It’s recent. A markdown converter should be tested on how it converts markdown to HTML. If it also converts to PDF, well, that’s cool, but it’s not the primary function. Make sure the HTML conversion is working solid. Before you test the PDF conversion, there’s a chunk of code that only one engineer understands. Well, you got to test that. And then after you’ve tested it, assign all the defects and new features in that area to a new person so that you can double up on expertise. Single points of failure are terrible things to leave, as is for too long. But really, that’s a management thing, not a testing thing. So let’s go back. Well, what if the PDF conversion was an afterthought, but you get lots of customer defects in that area? Well, guess what? That’s now one of your USPS or your unique selling propositions, and your customers are using it for that. So test it more. Well, hopefully that helps prioritize which features to test. So for all of those, we need to get test cases.
00:05:09 And for the test cases, I think of a few things. First, start with a nontrivial happy path test case. Then look at test cases that represent interesting sets of input. Interesting starting States, interesting in States, interesting output, all possible error States. Well, that makes sense. I’m just looking at the input and the output, trying to find interesting things that I can put in or interesting things that come out interesting starting States and end States. And the error States. That’s kind of obvious, isn’t it? Well, more specifically, and if you want to look this up more, the names for this is equivalent to partitioning and boundary value analysis. I kind of combine them. However, when I start talking to people about equivalent partitioning and boundary values, their eyes glaze over. And actually the odds are that just with those last two senses, I’ve lost about half of you already. Let’s go back to real world. Let’s look at some examples. Let’s say we’ve got a check register application, and one of the features that we’ve decided is really important is entering a check. That makes sense. Here’s a happy path test case. Enter a check with a check number of one, two, three made out to Bob with the value of $10. It’s nontrivial. The happy path should work. We check the output, check the instate, everything’s good. Test is good. Now let’s look at interesting inputs. We had $10 before. What if we set $0 or $0.01? Or how about negative $0.01? Just the boundary value is around zero. How about lots of money like a billion dollars or something? And if you know the Max length or the Max value, then use that.
00:06:55 We made the check out to Bob in the happy path. What if we made it out to an empty string, or a single character string, or a really long string, or maybe something with Unicode characters in it? These are sorts of interesting strings that we could do. And again, if we know that there’s a Max value of Max length of the string, then we need to test that and then maybe like right past it to see what happens if we get past it. So those are interesting inputs. How about interesting output? What’s the output of that? Well, it’d probably be the check register value, the ending value, maybe. So interesting output would be when it goes below zero or when it goes past the Max maybe.
00:07:38 How interesting starting and ending States. Well, when the register balance reaches zero or when it hits the Max value, that’s interesting.
00:07:49 Hopefully that works.
00:07:51 How about if it crosses it, when it goes from negative to positive? If we’re allowing negative balances and then from positive to negative, and when it hits its Max value, if there is one, what does it do then?
00:08:05 Those are all kind of the interesting output in the starting and ending States kind of all blend together. But we just want a set of test cases, so it doesn’t really matter. How would air conditions. This also is kind of meshed in with the other ones, but are there expected exceptions? Do we throw an exception or is there some error case? If we go negative, what about Invalid input? What if we pass something not a string into there or something or not a number? What happens then? How does it deal with that? Those are air conditions we can test for. We can also just search the code for asserts and exceptions and see if we can hit those with some input. That’d be interesting. So again, I’m going to go through these at the end. But really, these are just a few questions that can generate a huge number of test cases, but it’s not really a huge number. I mean, that’s manageable. That’s what we were looking for. We’re looking for a reasonable set of manageable test cases that really pretty thoroughly test the feature and then we move on to the next feature and don’t really worry about whether or not we did it thoroughly enough. You really want to get a good decent I think this is pretty solid for all of the core features, all of the recent features. So let’s actually look through those again. So which features we’re looking at? We’re looking at the recent features, the core features, the risky ones, problematic ones and the ones that require scarce expertise. And then for each of those are the ones that we prioritize, we go through them one at a time. We look at happy path test cases and then interesting input, interesting output, interesting starting States, interesting ending States, and then all of the possible error States that we can think of. Now, if you’ve done that, you actually are better than like 80% of the other software out there. So there you go. I hope this helps if you’ve got other strategies or a different set of strategies that work for you, please let me know. I’d love to hear about it.
00:10:05 Thank you, PyCharm for sponsoring the show. Save time, use PyCharm, especially if you write tests. Check them out at testandcode.com, PyCharm that link is also in the show notes at testandcode.com 150. Thank you also to Patreon supporters join them at testandcode.com support. That’s all for now, go out and test something.