Parametrization is the ability to take one test, and send lots of different input datasets into the code under test, and maybe even have different output checks, all within the same test that you developed in the simple test case.
Transcript for episode 87 of the Test & Code Podcast
This transcript starts as an auto generated transcript.
PRs welcome if you want to help fix any errors.
There’s a cool feature of Pytech called Parameterization, and it’s totally one of the super powers of pytest, and you really need to understand it to get the most power out of Pi test. It’s actually a handful of features that all work together, and there’s a few ways to approach it. So if you don’t know what Parameterization is, you take one test, maybe a happy path through the system, maybe a focus test, just testing one function or piece of the system. Let’s say you’re sending a simple input data into some code under test and checking the output and behavior. That’s kind of a normal test. Now, Parameterization is the ability to take that test and send lots of different input data sets into the code under test, maybe even have different output checks, all with the same test that you’ve developed with a simple test case. Super powerful, but something that trips people up all the time. That’s the focus of today’s show. Thank you to Pie Charm for sponsoring this episode. Check them out at testandcode.com PyCharm and boost your productivity today.
Welcome to Testing Code, the podcast about software development, software testing, and Python.
Yeah, I thought it would be fun with this episode to shake things up a bit when I have an interview that’s probably going to take the whole episode. But what is just me like this episode? I thought it might be fun to try out a different format or maybe try several different formats. Let me know what you think. First Updates I’ll start us off with an update on what’s new with Pythest. Well, more specifically, a hiccup I had when updating and how I got around it. Next, fun with Flags many of us use a handful of tools every day, but might not know the power they hold. I’m going to share a Get option that I just learned then the Parameterization bit that I talked about at the beginning of the show. So pytest is up to version 5.1. .2 now, as of the 30 August at work, I was using the for something release and I tried to bump up to the latest one and it didn’t work right away. So let me tell you what happened. So in the five 1.0 version they removed some deprecated features and I applaud this even if it causes me a bit of extra work. I mean, it’s a small team focused on an awesome tool and if they’re taking away some of the features that make the code structure a little bit better and it’s not really taking away features, it’s taking away things that we really shouldn’t have been using in the first place. So good job by test team, but it did cause me a hiccup. So what happened?
Well, actually I’ll just go through a couple of steps first. So if you’re going to update any version, especially Pi test, my suggestion is that you upgrade first. Before you upgrade, run all your tests to make sure that your tests are working as you expected, then upgrade to the newest version, then run your test again. If you don’t have any issues, then great. If you do have some issues, then kind of work through them. So that’s what I did. So what I did, there’s a page on the pipeline site that’s the change log, and there’s a page on there for deprecated features. I used both of those and all links to both of those in the show notes. So what happened in my case? In my case there is the configuration of your test run and the things like which flags you passed in and stuff like that, and it’s called a config object and that you used to be able to grab that with the global pytest dot config object and that’s not there anymore. That was deprecated, but the functionality is still there. So in my case I had a hook function. It’s pytest report header hook, and we were using that to grab basically just grab whatever options people are using and print that at the top of the report.
It was nice for tracking how a test was run, and we were also grabbing it and putting it like in a special output. So we’re saving our output to a database, so we’re saving that as well. Within that report header, we were accessing pipes config and that went away. So what do I do? Well, the pytest report header hook has a config object already. We should have just been using that. So I just replaced Pinterest config with just config and the whole thing worked just fine. Now your mileage may vary. Maybe you’re doing something extra fancy and this is probably what’s going to happen. Normal test code isn’t going to break with versions, but if you’re doing something like you’ve got some hooks or something where you’re grabbing some internal bits of pytest and hoping that it will work well.
So the change log will tell you which features were deprecated and then the deprecation list. It’s a really great list that talks about if this fails for you, what you should do differently. So it isn’t completely thorough. Again, volunteer team, but they’ve done a really good job. So thank you Pythons team for helping me get through this. So Python is up to 5.1 .2 and I’m really liking it. Fun with flags. I thought it would be kind of fun to try just to introduce one or two flags from common tools that we use every day because maybe there’s something new. And what’s new with us, with me was I’ve been using Git for a long time for open source project, but at work we’ve been migrating to get from an older version control tool. And one of the things that people have asked me is if I clone a repository and I want to select a branch of somebody else’s. If I just say get branch, it only shows me the branches that are new or local already on my machine. I don’t see them all. Well, there’s a dash, a flag for all. It’s just get branch A and that’s it. It shows you all of the branches that are on the remote repo. So cool, so simple, but two characters and it gets exactly what people want.
Thank you PyCharm for sponsoring this episode. Pycharm today is a different powerhouse than it was even a few months ago. The team has done some amazing work, not only in their pipe test support, but in other areas as well. One area that I’m really impressed with is the recent changes to the Jupiter notebook support. More and more work is being done in notebooks, in all sorts of fields, and people like me want all of the power of notebooks with all of the power of a great editor. I love my editor Tweaks keyboard shortcuts and custom theme, and I don’t want to give it up just when I’m working on a notebook. And now I don’t have to create, edit and run Jupyter notebooks right in by PyCharm. They recently added more run controls. You can run all the cells or the currently selected sell. You’d expect that, but this next bit is super cool. With a selected cell, you can run everything above it or the selected sell or everything below it. This is way neat and helpful for working on and debugging changes and exploring data. You can try out the new notebook support yourself by going testandcode.com/pycharm before September 25 and they’ll give you four months to try out the pro version.
Now the meat of the show path.
I should have named this something easier to say paths to parameterization, going from one test to many. Now I’ve had some feedback from people that say that I’m rushing through stuff a little bit fast, and I appreciate that feedback. You do not actually have to have read my book before you listen to the podcast. Of course I would appreciate it if you’d buy a copy and actually buy ten extra copies to give to your family and friends. Test parameterization is a mouthful, but it’s also if you take a simple test that’s just doing something simple, let’s do the dumb edition. One. So if I’ve got a sum function, takes two numbers and spits out an output, I can have a test that says given one and two, I pass it. To sum, the answer should be three. Now that’s one test, but I want to check the sum function with a whole bunch of stuff. And parameterization is a way to say to have one test, but multiple test cases. So I can give it the input of like one and two with the expected output of three as one parameterization. And then I can have a whole bunch of others. So that’s what we’re talking about today. And on the face of it, it’s not too difficult. But it trips people up and I find myself, I am helping a lot of people get through how to do this. I’ll tell you kind of how I go through the thought process to get parameterization working. One of the difficulties is there’s a couple of ways to do it. It’s not just one feature. So test parameterization. You can also use fixtures and fixtures are if you’re not familiar with them, there’s just an extra function that runs before a test runs. So if a test says like, I’m using fixture X and Y and those X and Y should be functions that exist with a decorator that says pytest fixture above it.
And pytest will look for those and run those before it runs the test. Why am I bringing that up? Because that’s one of the ways you can do Parameterization. So there’s a marker. pytest Mark parameterization. Wait, that’s not right. Pytest Mark parameterize. Is that right? Now I’m messing myself up. One moment. Yes. Pytest Mark parameterize. I’ve been saying this so long that I messed myself up. Where did I was looking this up? Well, I’m going to give links in the show notes to both the pipe test documentation for both parameterizing tests and parameterizing fixtures. And I’ve got it listed, of course, in the Pytest book in both. Chapter two covers test parameterization and chapter three covers fixture parameterization. Yes, you can parameterize both. So I start with this by taking and it takes effort and I know it’s the right thing to do. And so I really try to do it. And I’m trying to teach other people to do it is to every time I approach a problem, needs a test for it. I start with one test and one happy path and I write everything into the test. I try to get it into a given when then structure. So that makes sense to me. So the top of the test is the given state. And in the simple sum case, it’s like A equals five and B equals six.
And then when the win state is the action. So in our little sum case, it would be like C equals the sum of A and B, and then what’s the answer? So in the then part, I’m going to assert that C equals. I should have picked simpler numbers, five and 611. So assert C equals eleven. Now, some people think of that as a range act assert and they’re really the same thing. It’s just how better you think about it. So I think about giving when then better syncs with me. So that’s my little test. So I write the test just as is. Now that I have that test working, I’ll actually run it and make sure it works. And then I will take the given part and see if that makes sense to put into a fixture. Because this is a long story, sort of.
I like to give the entire getting ready part into a fixture if possible. Partly because it can be shared by multiple tests and partly because if anything fails in there, the test stops with error instead of fail. And this will tell me that I didn’t get to the point where I was testing the thing I wanted to test. Okay, so that’s that. So I split it up, get down into the fixtures, make sure that my given when structure is clean, the asserts are all at the end of the action and it’s nice. So at this point I’m going to take advantage of version control and commit this all to get or whatever version control system you are using. Make sure you save this part and probably the earlier part. I commit all the time because when I’m refactoring stuff I want to make sure if I really muck things up I can go back in time a little bit and it happens all the time. Really love version control. Now that I have my test split into either a test that’s nice and concise or I’ve put my getting ready stuff into a fixture. Now I want to have that one test case and make it into more than one test case. And so I will try to take another test case like to try to make two or three different rows of parameterization and put that just on the test to see if that makes sense.
At this point what happens is every time I run the test, if the fixture is test scope, the fixture is going to run every time before each parameterization. If the fixture is modular or session scope, the fixture is going to run once and then each of the parameterizations is going to happen. So the next step really is or my scope of my fixtures correct? Are they running the right amount of time? If they’re doing something that doesn’t change every time then they might be able to be broadened in scope. It could be a modular session scope, it might make sense. It still also makes sense that the actions in the test might reset something that needs set up in the fixture, but a whole bunch of other stuff doesn’t get changed. In that case you can split up the fixture, you can have part of the fixture B module or session scope and part of the fixture B function scope. And the way you do that is just make two fixtures and have the function scope fixture depend on the module scope fixture. Now does this even make sense when I’ve got two or three parameters? If it makes sense at this level then hey, you got it. You can just add as many parameterizations as you need right here and you can turn this one test into multiple test cases and in many cases this is totally good enough. Now there’s a lot of times where the thing that I want to change for every test case, it lives in the fixture, not in the test. What do I do then? One of the options is we can just jump right into fixture parameterization. One of the options also is to say if the thing that’s changing for each test case is in the fixture, is that really getting ready for the test part or is that really part of the thing that’s getting tested? One option is the part that needs to change, push that back into the test, take it out of the fixture, put it into the test, and then I can use regular test parameterization to iterate over those different test cases. Now, in plenty of cases, it really does make sense to just leave it there and parameterize the fixture instead of the test. And of course you can mix and match these into both, but you’re going to end up with a test matrix. Just know that it can blow up into a whole bunch of test cases really fast.
Make sure that that’s what you want. Fixture parameterization.
Unfortunately, the syntax is a little different, so it is like basically two different syntaxes that you have to deal with, but they’re both not that bad. Just make sure that you don’t assume that they’re identical. Go ahead and read the pipe test documentation or the chapter in the Pytist book on fixed your parameterization and make sure you get that set up. But in a lot of cases it’s really a personal judgment call on your part whether you’re prioritizing a fixture or parameterizing a test. I want to stress again about leaning on your version control system. If you have broken this up into parameterization and you want to play with the test parameterization and you think, well maybe I should go to the fixture parameterization instead. Check your code in before you make the changes so you can put it on a branch or whatever. So you have all those versions available. All of it’s going to be a learning experience. Even if you have to go through this a few times, you’ll learn a lot. Last but not least, code reviews. Tell the people you work with say, hey, I’m parameterizing this test here was the original one. Here’s the parameterized one.
I’m not sure if I should do it in parameterized in the test or parameterized in the fixture. Even if you completely understand it, be aware that your team has to support this tests two and readability counts. Yeah, you’re all in it together. That was a pretty brief rundown of how I approach parameterizing tests. I hope that was helpful.
Thank you to PyCharm for sponsoring this episode and many previous episodes. Don’t forget to go to testandcode.com PyCharm before September 25 to get that four month pro trial and get your work done faster with a great editor. That link is also in the show. Notes at testandcode.com 87 thank you to Patreon Supporters If you get value out of this podcast, please consider pitching in a buck or two by going to test and coach.com support. Also thank you to all the great people on our slack channel. I really can’t tell you enough how awesome these people are. Join the fun at testandcode.com slack. There are now over 700 people there helping each other out with testing questions and please let me know what you think of this episode. Do you like the format change? What do you think of fun with flags? What do you think of checking in on some new versions of different tools that we use? Maybe I’ll cover the new things and like some of the other tools let me know what tools you want me to keep my eye on. I’ll definitely keep my eye on pythest and talks but what others so the easiest way to reach me is to go to testandcode.com and look for the contact form and send me an email. You can also find me on Twitter where I’m at Brian Hawkin, also at test and code or at test podcast. The last two are easier to spell of course, but I monitor at Brian Hawkins more frequently. And by the way, there’s two KS somebody else out there with one K but it’s not me. That’s all for now. Go out and test something or maybe turn one of the tests you already have into mini with some handy parameterization.