This is the transcript for Test and Code Podcast episode 2
Hello everyone. My name is Brian Okken. Welcome to the Python Test Podcast (Now called “Test & Code”).
Today I want to talk about choosing a test framework. When I went to look at the different frameworks, I noticed that unittest and doctest are part of the standard library, but nose and pytest were used quite a bit also, so I figured those four were good things to look into. Really what I was looking for was not a unittest framework, but a functional test framework. They are really the same thing. I didn’t know if it was complete list so I put a post up on my website with a poll asking other people if they were interested in those, and if there is anything else. And predominantly those four came back as things that people wanted to hear about.
Well, so how do I choose?
I really don’t know how to tell you how to choose yourself, but I can walk you through the sequence that I went through. Kind of like buying a car, I listed my requirements down and then test drove them. So what are my requirements? Setup and teardown of test is an important consideration to me. So multi level fixtures are important. My philosophy towards software development means that I write tests for functionality that isn’t there yet, so being able to mark a test as expecting it to fail is important. I think it’s important to be able to continue through a test function after a failure, very unlike the assert statement. Next up, easy to write tests, minimum boilerplate code, some easily parsable reporting. We also have, since I am working with tests instruments and there is things that I want to run between the tests for every test, even if somebody forgets to put it in there- putting in required fixtures will be good. I didn’t know if that was possible. Ok, so those are my requirements. I’m not going to- I did experiment with doctest, it was a fun experiment, I did write up my findings on the website, but I am not going to talk about it here because it’s just painful, my experiment mainly is for me, it told me that it was too painful to use as a functional test framework. If you guys out there use it a lot, great, it wasn’t for me. However, I am going to discuss how unittest, nose and pytest stacked up with the requirements I needed.
Regarding multilevel fixtures, what I mean by that is being able to- well, they all do it, unittest, nose and pytest all support multi level fixtures, I can have a module level setup, followed by a class level setup, followed by a functional level setup and they can interact together. When setup takes a long time, this is a good thing to do in a module or class level setup, and have a method level setup just make sure that the setup is still correct. That’s what I use it for. So they all support it, however, pytest named fixtures are way more powerful, and take you a lot farther, the fixtures can be- fixtures can use fixtures, the fixture failure handling is smoother I believe than nose and unittest, the fixture model helps you think more of individual resource needs instead of just the entry points in setup. Just, I like it better. Ok, so multi level fixture- all of them work but I like pytest’s use of fixtures better.
About expected failures and skipping. Actually, this is something that is interestingly absent in nose, that I know of, I couldn’t find anywhere how to get expected failures to work right in nose. Unittest does do this, unit test and pytest both support marking a test as expecting to fail and skipping it. The interesting thing I think is unittest allows a skipif function, actually both unittest and pytest do this, so you can say like, you know, if you are in a certain version of the software skip this test, or really, anything. However, for the marking of expected failure, pytest does support that, pytest allows you to say if in this version it’s going to fail, I know this. unittest doesn’t have a expected failure if statement, not sure why. And, also the reporting of expected failures and skipping, I don’t know, I kind of like how pytest reports that a little bit better than unittest.
How about the requirement of continuing through a function after a failure? For a lot of functional tests and a lot of tests in general, it makes sense. So let’s say I’ve got, I don’t know, I’ve got a function or a setting that has side effects. It has five side effects. I want to make sure that all five of those side effects happen. I’ll have five assert statements, or expect statements, or something. And I actually want to know the result of all of them, even if one of them failed. By default actually all of these frameworks, unittest, nose and pytest will stop at the first assert. That’s not unexpected and assert is an exception. I kind of was desperate for this, so I made a work around. And I wanted my first attempt at a work around to be framework agnostic so it would work in all the frameworks. I described it in the post titled Delayed Assert. I also incorporated that solution into a little bit cleaner solution for as a pytest plugin called pytest expect; it’s not on PyPi but it is availabe at my site. So, I added that requirement. I made a work around that would work in all those. It works cleaner in pytest. I think that I could probably make plugins for unittest and nose as well. I just haven’t done that yet.
How about easy to write tests? After learning about all of them, I didn’t think any of them are hard, unittest, nose, pytest they are all pretty easy to write tests. The pytest assert rewriting, which I am going to talk about that in a future episode, but the assert rewriting is pretty cool, and it makes things a bit easier so you don’t have to remember or lookup a bunch of the fancy assert helper functions that you do in unittest and nose. Also the pytest fixture model makes it easier to write tests. There is more features like parameterized test functions. Yeah, there is some extra fun in pytest to make it little bit easier to write, I believe.
How about minimal boiler plate? I put this in my requirements because I did see some posts that said that unittest had too much boilerplate, but it’s really not that much boilerplate, you have to import unittest into your file and all of your tests have to be members of a class derived from unittest’s TestCase. That’s, it’s not that big of a deal, I don’t think, forcing people to write in classes when they don’t want to, even though they are not really using them as objects, that is a little weird.
How about reporting? I don’t know, this is just sort of subjective, they all report test failures and although, you know, like I said, nose doesn’t deal with the expected failures that well, but both unittest and pytest reports are acceptable. I like the pytest reporting better.
Required fixtures. I’d like to have setup and teardown that runs on every test. For my reason, I work a lot with test instruments. The instrument itself has an error log, for instance, as one example. And at the beginning of the test run it’s good to clear out the error log and then after a test runs, an individual test checks the error log and reports an error and fails the fixture if there are any errors in the error log. It’s kind of handy to put — if you are doing it in every test — it’s handy to put it in the fixtures. You can do this with fixtures with nose and unittest but the autouse feature of pytest fixtures make it trivial.
For me, pytest was a clear winner, given my requirements, but you know, everybody is going to have different requirements. One thing to consider is the extensions and plugins. There’s extensions and plugins for all of these, and you may feel like it’s required- I mean, if you’ve got the required extension that you can’t switch over, than obviously you are not really deciding on which framework, you’ve already made your decision. And keep in mind though that it’s probably not that terrible to switch. Get started testing. Write your tests. Just pick a framework and get started and if you change your mind in a few months, this probably is not going to be terrible to switch. Keep in mind also that pytest is really good at running unittest and nose. If you switch from unittest or nose to pytest it won’t be too painful. Also, interesting to note, that a lot of projects are migrating from unittest or nose to pytest. I have not heard of anybody moving away from pytest to unittest or nose.
The other thing that I wanted to bring up is speed. I have heard people say that they’ve noticed that pytest was, it just seemed a little slower to start up than the others. I actually haven’t noticed that. But for me, if it is slightly slower it’s not going to matter too much really because, well, for three reasons: one, it’s apparently not slow enough for me to notice it, and second reason- development time for me is way more valuable than computer time. So if I’ve got to let the computer run an extra second but it saves me hours of writing tests, I’ll save my time and let the computer run a little bit longer. The third reason is the tests that I am writing, the communication and data transfer time is way more, it’s way more time than the actual pytest functionality is going to be taking up anyway, so pytest isn’t going to be the bottleneck in my tests; but if speed really is an issue for you, I guess you can try them out, compare them, do some time tests and figure out if it’s an issue.
So, that’s really all my advice- it’s not really advice it’s just kind of how I went through it, how I went through those decisions. But, I want to stress something; picking a framework can seem like the most important thing that you have to figure out, because you can’t go- I was there, you can’t go forward until you pick one and it seems like a big decision that you can’t change later. But, you can change it later, and also, it shouldn’t consume too much of your time, I don’t think. Just pick something and move on, and switch if you need to, but don’t let this decision consume too much of your time. So I guess what I am saying is there is way more important work for you to be doing. You need to be making decisions about what to test on your project, and how to test it. Don’t get too hung up on which framework.
So, that’s it for now, I’d love to hear from you about what you use, and how you came to those decisions yourself, and also if you have a particular problem with one of the frameworks I’d like to hear about it. On Twitter I am @brianokken and this podcast is @testpodcast. So, thanks, I’ll see you next time.