Some functionality is important enough to make sure the test behavior coverage is thorough. In this episode, we discuss 3 techniques that can be combined to quickly generate test cases. We then talk about how to implement them efficiently in pytest.
Transcript for episode 39 of the Test & Code Podcast
This transcript starts as an auto generated transcript.
PRs welcome if you want to help fix any errors.
Test & Code | Episode 39: Thorough software testing for critical features
Welcome to Test & Code, a podcast about software testing and software development. I am you host Brian Okken, and today we’re going to talk about how to go about exhaustively testing a particular functionality of your code.
Before we get started I’ve got a few people I want to thank. I want to thank Felix Haberleâ I’m probably pronouncing your last name wrong, I’m sorry about that Felix. Felix is the first contributor to the Cards project other than myself, so now there are two contributors, so that’s exciting. We’re actually going to talk about what Felix proposed and helped out with a little later.
Next, Patreon supporters, I couldn’t do this without you. And thirdly the Slack channel folks, right now there are it looks like 354 members, if you’ve got a question about software testing in Python, these men and women in here are amazing, they help out people all the time. There are people around paying attention, if you ask a question it won’t go unanswered for very long. So thank everybody on Slack and thanks Felix and thanks Patreon supporters. And so next, let’s get into it.
On the last episode, episode 38, we talked about prioritizing software tests. The plan for today, for this episode, was to take like a particular feature, like the delete functionality of Cards. As a reminder, Cards is a task application little thing, a command line thing to demonstrate some of the testing techniques. I was going to take something like delete and use the equivalents, partitioning and boundary value analysis to develop a more thorough tests, but I had this wonderful opportunity and the opportunity was that Felix had submitted a pull request, he thought that it would be neat to have a feature to filter, so once we have a whole bunch of tasks listed, it would be great if we do not list them all all the time. So it would be a way to filter them. And this is a common thing, so you’ve got a team of people and let’s say you want to list out the ones that don’t have any owners yet, so that you can assign owners so you can pick something that you might want to work on, or you want to look at all the tasks that you have or all the done ones.
So the request initially was for two flags, one of them for an owner flag on list to say hey just list those of a particular owner and that makes sense. And the other thing is a no owner flag, so to list the items that don’t have an owner yet. But this is completely reasonable, easy to understand. And Felix and went ahead and did initial implementation and I appreciate that, and initial tests. And the initial tests were in the command line interface tests and these were great, it was a great start, so thanks Felix. I took it and ran with it, I am like, “Yeah, this is a good thing to have, I’m excited about having my first contributor to the project, awesome.” As I said, I am dog fooding this application trying to use it at work and yes, I want to be able to filter things really well.
And this is sort of a feature that’s going to be used a lot, so I really want the testing to be thorough. This is a great opportunity to go ahead and touch on a few things. If somebody says cards list you can add a filter, you can add a flag that says no owner, you can add a flag that says owner, and then list the owner. And then you can also say whether you want to list the done items, a true or false flag for that. So if you don’t pass in the done flag, all of the items are going to get returned and listed whether or not they are done, and if you pass in true, it’s all the items that are done, the completed ones. And if you pass in false, it filters those and just lists the ones that are not done. This makes sense.
The no owner is kind of self-explanatory, if you pass in no owner, you get the items that don’t have any owner on. The owner is, let’s talk about what you can pass in. If you don’t pass it in, it lists all the tasks regardless of owner. If you pass in something, that something could be an owner, it could be a string that it’s the same as one of the owners that’s in the list, and that’s assigned to the Cards. It could be somebody that isn’t in the list yet, this person doesn’t have any cards that are there.
I realized when I was playing with this that once there is an owner if you want to remove the owner you can’t really do that, but you can update something with an empty string. So an owner of empty should make sense, so for instance if we didn’t have the no owner flag passing in an empty owner, that should be the same as saying I want to list the ones with no owner, so I want to make sure that works.
And then they can work in combination, you can pass in let’s say that I pass in no owner, and an owner of Okken say and done true. That means I want to list all of the items that are listed that I own, plus I want to list all the owners that don’t have an owner, plus of all of those I only want to list the ones that are done. So all that makes sense, it kind of makes sense if you think about it, and it’s a little complicated, so this is a risky thing, there’s complexity, there’s risk. So according to our prioritization from episode 38, maybe we should thoroughly test this and I totally think so. So how do we do that?
I’m just going to touch on a lot of these. Equivalents partitioning is a cool way to say I’ve got this infinite set of, I can’t completely test anything, I can’t test all of the possibilities because the set is infinite for a lot of cases. In our case, done can either be none, true or false, no big deal. No owner can be there or not there, but the owner field can be not there, it can be a string but the string can be anything, so that’s infinite, you can’t test every string. So how do you split those up? And we’ve already sort of, like enum values or booleans are easy because you don’t really have to split them up, they are already kind of in small sets, but the owner one, I’ve split those up into saying whether the owner exists in the set, the owner doesn’t exist in the set or an empty string. So I have narrowed that down to say I’m partitioning all the input of owner to say those different things, it’s either none, it’s an owner that’s in the set, an owner that’s not in the set or an empty string. And those I can test more, but they’re all going to fall, all of the possible inputs are going to fall into those categories. So as long as I test those like one element from those categories, the theory for equivalence partitioning is that all other tests are going to be equivalent to these tests, so that’s good. I think that actually learning all these is going to be better if we just touch on them when we need them.
Boundary value analysis. This is, it’s good to read up on boundary value analysis, but really, we make mistakes on the edges and also we make mistakes on how to describe and define requirements on the edges. Once you learn about it and start using it, it’s just something you kind of use. In our case, the values of owner, the boundary that I’m considering is the empty string. I always like add this when I come up with the equivalence, I do equivalence partitioning first and then I think about boundary value analysis. And usually I think about it as the edges of my equivalence partitions. And before I learned about boundary value analysis, I probably wouldn’t have said well what about the empty string, but that’s one of those things, that’s one of those boundaries. And so that’s why it’s in the test. But I also wanted to define it. before I defined it, before I thought about it, I didn’t know what it should do.
How do I test all of this? How many test cases could there possibly be? And it’s actually pretty big, so if I take the done, they multiply out. So if I take the done element, it’s none, true or false, that’s three. The no owner flag could be there or not that’s two choices, so that’s six combinations, and the owner if I set those partitions as none, so an empty string none, an owner that’s in the set and an owner that’s not in the set, it’s 4 times 6, it’s 24 different possible test cases. That’s kind of a lot. But some of these you can collapse them down, so we’re not going to do 24 test cases, but we’re going to do quite a bit.
What if I really do want to thoroughly do these? So another way to thoroughly test something is to use decision tables and as far as I can tell, what I do for a table based testing is similar to decision table testing. If you look it up, decision table testing, it’s actually all like the Wikipedia pages and stuff. It seems to be a little confusing, but in this case, what I am going to do is I’m just going to write a table and I’m going to write a column for the done flag, the owner flag, and then no owner flag and then go through, leaving all of the other elements the same. So I am going to start out with none, none, none and I am going to say okay, so there’s no owner, the no owner flag is not passed in, if I don’t pass in an owner but I do pass in a done, or what the value of the done flag it’s none, true or false so that’s three test cases.
And then counting in binary, I am going to toggle the next one, so if I have the owners the next one I am going to cover, the next value of owner and then go through all the cases of done. And then keep doing that until I get all through my equivalence partitions for owner and then I’m going to repeat everything with whether or not I pass in the no owner flag. If you do this in a spreadsheet or something, actually it doesn’t take that long to do all of this and then I can collapse some, so one of the things I noticed was the empty string and no owner, in a lot of cases it returns the same set of values, actually in all the cases. So coming up with what the result should be. It’s easy in that case.
The other thing is the case where the owner is listed but not there, I added an owner but that owner doesn’t exist in the set that’s going to return an empty string or an empty list, so I don’t have to test all combinations, it’s like one test case. So I boiled it down to like 18 test cases or 16 which is still a lot, but I only need one test and that’s thanks to pytest, and I’m not going to list all this out but I used test parameterization. And if you go to the cards, github.com/okken/cardsâ I’m going to keep working on this, but for right now it’s version 0.1.11 and if you go into the tests and then look at the cards db under there, there’s a new file called test list filter, and I actually listed out all the test cases, but they are all one-liners.
So what I did was I really pretty much set up a table with test parameterization. And the elements can be the no owner, the owner, the done and the indices. The indices, I’ve got like a test set of cards with just five cards in it and then which combination of those five get pulled out based on which flag. I’m also using the ID field parameterization, to make use of listing these out well, so that once they’re a little terse in how I’ve written it down but with the id I explain all of the combinations. So one test, 16 test cases, 16 parameters that go in the parameter sets that go in to one test. And yeah, and also a fixture to set it all up. So there we have it. I’ve got using a little bit of bounder value analysis, a little bit of equivalence partitioning and a little bit of parameterization. Actually, a lot of parameterization, but a little bit of decision tables. I’ve come up with it, but I think it was a thorough test case.
Is this silly? Is it silly that I’ve tested this many test cases? And maybe you could argue some of these don’t really need to be there. I’m going to run it in PyCharm and rerun all of my tests, I can’t remember how long it takes. So all of the tests including all of the command line interface tests and everything, I’m still up to 0.75 seconds. So as far as I’m concerned, I don’t need to start calling test cases yet, I mean, sometimes we’ll get to the point where we want to call test cases, but right now, I think I’m good with this.
Great, so we touched on a lot of stuff. If I want to really thoroughly test something I can pull out my equivalence partitioning tool, my boundary value analysis and decision tables and say, hey, yes I think I’ve exhaustively tested this.
Now how about the split up? So whether you tested the command line interface or test through the user interface at the top or test through a subcutaneous API, and we are going to talk about this a little bit more in the next episode, but I slimmed down the command line interface test that Felix had put in to just do with that the CLI test I just want those to essentially just test the flags to make sure that the flags get through correctly. So I could have mocked, but I didn’t. Let’s take a look. It’s just in the test CLI, there is a test list filter that is one test case to make sure that works in one test, in passing in all of the flags, no owner, owner and done. I just want to make sure that. Plus, I am dog fooding it, but I like to have one of these in here. And then, all of the combinations though use the API because it’s a lot easier to work with. And yeah, those are the levels.
There are a couple other things I did recently, I actually moved things around a lot, I moved the Cards into a source directory, the implementation code, because I was using my own advice and because you should use a source directory and there’s an episode of Python Bytes that we talked about that, I can’t remember which one though. Because they are good, and we have a lot of tests to make sure it works.
The other thing is there’s more documentation now. If you remember at the beginning of this I had some manual tests and one of the first things you’re going to do when you automate often is take your manual tests and try to automate those. And I did that, and we’ll talk about that in the next episode.
I hope you’re having fun with this because I am. Please join Felix and I and then start hacking on this cards app. It’s fun, I think, and there are already some suggestions for new features. One of the things that Felix suggested was a plugin system, I don’t know how to do plugins, but that might be fun, if anybody else knows how to use plugins, add plugins to a project, that might be neat. But I don’t know if we really need a plug in system yet, but, what the heck.
Anyway, that’s all, I’m going to stop blathering on right now and let you get back to testing your own code, and so I will see you next time on Test & Code. If you want to ask me questions go ahead, there’s a contact forum on testandcode.com, and there’s also me on Twitter @brianokken. Thanks a lot, and I’ll talk to you next time.