Test fixtures make your test code easier to write and results easier to read.
Transcript for episode 4 of the Test & Code Podcast
This transcript starts as an auto generated transcript.
PRs welcome if you want to help fix any errors.
I’ll be running a contest through the month of October, more details at the end of the episode and at the show notes at pythontesting.net/4. [music]
Hello everyone! My name is Brian Okken, and welcome to the Python Test Podcast. Test frameworks have a built-in mechanism to let you run code automatically before and after tests. This mechanism is called fixtures. It’s a nice word when you get used to it, and it’s easier to say than “setup and teardown and that native Pytest fixtures stuff”. It’s essential that you understand how and why you should use fixtures in almost every test you write actually, at least, that’s my plan for today, to convince you of this. It’ll make your tests easier to write, and make your test results easier to understand. Today I’m going to talk about aspects of fixtures that are common to pytest, unittest and nose. I’ll touch on some of the super cool things that pytest fixtures can do, but not too much. These things deserve their own episode, or two or three.
Let’s start with setup and teardown. A setup function is a function that runs before one or more of your tests. A teardown function is a function that runs after one or more of your tests. They’re often paired. Fixtures can be used to handle resources for the tests. A setup function will allocate or configure some resource, then a teardown function will clean up and return a resource to a safe place or a safe state. However, test frameworks really don’t care what you put in the fixture functions. They are often used for resources, but you could put anything in there. Another use for fixtures is to get test data ready for the function under test. In my field of electronic instruments, setup fixtures are often used to get test equipment into a proper state to test a particular measurement. Sometimes setting up generated signals, trigger sources, power levels et cetera. The associated teardown function can turn off a signal if appropriate, and get the instruments back into a known safe idle state.
Again, you can run really any code you want in a fixture function, but why would you? Why not just put the code in the test? The benefits of keeping setup and teardown code in the fixture rather than in the test are code reuse, cleanup of resources, errors and focussing your thinking on what you’re testing, and taking advantage of scoping to save time on setup. I’m going to cover all these today.
Code reuse. This is what most people tout as the reason. If you have multiple tests that need the same or similar environment or test data set, it makes sense from a code reuse perspective to pull all that common setup and put it into a function that can be used by multiple tests. This is obvious and good coding practise, but you can do this without fixtures, you can just put all of the common code in a function and all of your tests can call that common setup function at the beginning of the test. So why would you put it in a fixture rather than do that? Maybe we need some more reasons.
One of those reasons is cleanup of resources. Some resources that are configured during setup need to be returned to a safe state, or need to be returned to the system after running the function under test and assert code. That cleanup has to happen, regardless of what goes on during the test. Frameworks will stop execution if an assert is hit, so if you’ve got your cleanup code after your assert statements, it’ll never get run. If your resource cleanup is after that assertion, then it won’t get called. How do we get around that? One of the ways is to save the data from the function under test, or interrogate the side effects and save that information, and then do the cleanup and then do the asserts afterwards. That sort of works, messes up your code a little bit. But what if the code that we’re testing itself throws an exception that we don’t catch? Well, we could get around that by wrapping the function under test into a try-except block and then run the cleanup and then re-throw the exception or the assert. This is getting kind of complicated. We want the tests to be easy to write and simple to read, so let’s try not to get around this, let’s just put the cleanup in the teardown function, that’s what they’re there for. Let them handle this for us. The test frameworks call your teardown functions after the tests, regardless of what horrible thing might have happened during the test function proper. So you don’t have to worry about ordering, just do the function under test and the asserts in your test, put the resource acquisition and initialisation in the setup fixture, put your resource cleanup in a paired teardown fixture, and you don’t have to worry about it.
There’s another great reason - it’s my favourite reason - is the errors versus failures point. This is equally important, if not more so, than the code reuse and resource cleanup arguments. Asserts that happen in a fixture cause a test to result in an error not a failure. Asserts in the test function proper cause the test result to be a failure. So let me repeat that. If an assert happens in your test, it causes the test to fail. That’s how you fail a test. There are other ways, but that’s the normal way. An assert that happens in a fixture does not cause a failure, it causes an error. This is what we want, this is what helps to clarify your test results. Let’s give an example. Let’s say I’ve got a bunch of tests that check measurements against a known good waveform, supplied by an arbitrary waveform generator. Perhaps some tests that check the power readings, some tests that check the frequency offset jitter or whatever. Now let’s say that there’s something wrong with the generator - the arb file is missing or someone has changed the name of the file or there’s some problem with the instrument or whatever. If you have the generator setup code called from the tests themselves, then all of these tests will fail. But if I have the generator setup code called from a setup fixture, then that fixture will fail. This will get reported as one fixture failure, the tests will get reported as errors, all of those tests, not failures, and the test code itself won’t even run. This is what we want. What’s the point of running a test when the preconditions aren’t met? I utilise this a lot, along with utilising xfail and xpass results, but I’ll talk about xfail in a future episode. The errors do not indicate failures in your code, they indicate failures in the setup. This is great.
So, they make sense from the points I’ve already mentioned, to utilise fixtures, but that last point of taking advantage of the difference between test errors and test failures brings up something else. When you want to separate your tests into setup, test proper and teardown, it focusses your thinking about what you’re testing and what you are not. Some of the things that show up in test fixtures might be things that you need to test themselves, but they should be in their own tests. I do need tests that make sure that the frequency setting function works, for instance, but when the test I’m working on is focussed on some other functionality and just needs to set a frequency to get the system into a proper state, then I don’t want the frequency setting in the test, I want it in the setup fixture. If the frequency setting functionality breaks, I want one test, or very few tests to fail. I want the ones that just depend on a frequency setting for setup to just indicate errors. So this separation helps you be clear on what you are testing in each individual test, and what you’re not. It also helps you to name your tests, and to keep them short.
One more aspect of test fixtures I want to cover today is scoping, and how that can reduce your test times. You can have fixtures at different scopes, different levels: at the function or method level, class level, module or session. Separating them into different scopes allows a test environment to be set up once for many tests. Pytest, unittest and nose all allow this function, class and module scope fixture separation. Pytest allows session scopes also. In a purist view, all tests should run starting at the same system state. However, that’s ridiculously expensive in many situations. If I’ve got a thousand tests that all need to talk to a couple of remote instruments, it’s way more efficient to connect to the instruments once, at the beginning of the session. Likewise, a complicated data set retrieved from a database or a remote source doesn’t have to be retrieved before every test if the test data isn’t going to change. It can be retrieved once, before the tests that need to run it. Strategically placing resources and data configuration in different scopes saves test time. It’s up to you to decide what scope makes sense, of course.
So far I’ve talked about xunit style fixtures. These are the paired setup and teardown functions at various scopes. Pytest named fixtures can work the same way, but also they do much more, and I’m not going to go into too much detail today, but I’d like to give you some hints at some of the cool things that pytest named fixtures can do. Pytest will optimise running of tests to minimise fixture calls. Named fixtures return a value, which can be anything, a number, an object, a set of data, a dictionary, whatever, and the tests can utilise those. Named fixtures allow parameterisation, so the tests that use a fixture are called multiple times with different fixture values. Like, let’s say if you’ve got a test that tests at 800 megahertz, and you want to test it also at 900 megahertz. You don’t need to write two tests, or put that in the test, you can write one test and have the frequency settings be in the parameter fixture, it’s pretty cool. The scope of the fixture, the function, class, module, session etc is set in the fixture definition, not at the place where the test states it needs the fixture. This allows you to move things up and down in scope if necessary with little code change. Fixtures can also be marked as auto-use, to ensure that tests use the fixture even if they don’t know about it. This seems bizarre at first, but it can be very powerful. It can also be dangerous if you forget that you’ve got a bunch of auto-use fixtures. One more trick from pytest is the yield fixtures. They also deserve an episode of their own, so I’m not going to go into them here.
Show notes can be found at pythontesting.net/4. For questions and comments, I can be reached through the website, and also on Twitter @brianokken or @testpodcast. Today’s show is sponsored by all of the wonderful Patreon supporters who gave – who have gone to patreon.com/okken to donate to the show. I’ve got lots of extra resources in the planning phase for those supporters. Today’s show is also sponsored by all the great folks who’ve bought copies of Python Testing with Unittest, Nose and Pytest, at pythontesting.net/book. Quickly get up to speed on test frameworks in the comfort of your own ereader. You can also help me grow awareness of this podcast by telling a friend or colleague about the show. Please rate it on iTunes if you’ve got iTunes. Give me feedback, let me know what you’d like to hear about next or suggest some guests. You know, even if you don’t listen on iTunes, the ratings and downloads through iTunes increase the show visibility. Many other services also utilise the iTunes rating, so downloads and ratings on iTunes really do make a difference. A special thanks goes out to Michael at Talk Python to Me, for all of your support, encouragement and promotion. I talked with Michael before I got the show off the ground. His tips have helped me immensely. His show can be found at talkpython.fm, and it’s great, I listen to it every week. Another podcast I listen to every week is Podcast Init from Tobias and Chris. Thanks to them also for your support and mentions. Their show can be found at podcastinit.com.
Now, I want to talk about this contest. Last thing, contest! I’ve got a contest going. I’d like to beef up ratings on iTunes for this show. I’ve got two on there so far, woohoo. That’s not very impressive, one of those is from me. So here’s what I’m going to do. I’m going to give away copies of the Python Testing ebook that I’ve got. I’m going to give one away per week through October. How do you enter? Rate the show on iTunes, and then, if you don’t want to do that, if you think iTunes is evil, try to convince somebody else to, I’ll even enter their name, and yours. And then send me your iTunes name because I can’t get your email through iTunes, so send me your iTunes name and your email. Send it on Twitter to @testpodcast or @brianokken, or you can email it to me at brian@pythontesting.net. Increase your odds, you get one more entry if you email a friend or colleague and ask them to listen, you know, blind carbon copy me in so I can see it. Also one more entry if you, in that email, ask them to rate it on iTunes also. Bonus entries. Come up with your own idea to spread the word about the show, and tell me what you did. Now, what if you’ve already bought a copy of the book, why would you enter this contest? Well, I’ll find something else to give away. It’ll be good, I’ve got a couple emails out to some great people that have some great products, and I’ll try to get those included in the giveaways. So. I’m going to have one winner per week through the month of October. Good luck! That’s all for today, thanks for listening. [music]