Choosing a test framework
This transcript starts as an auto generated transcript.
PRs welcome if you want to help fix any errors.
Hello everyone. My name is Brian Okken. Welcome to the Python Test Podcast (Now called âTest & Codeâ).
Today I want to talk about choosing a test framework. When I went to look at the different frameworks, I noticed that unittest and doctest are part of the standard library, but nose and pytest were used quite a bit also, so I figured those four were good things to look into. Really what I was looking for was not a unittest framework, but a functional test framework. They are really the same thing. I didnât know if it was complete list so I put a post up on my website with a poll asking other people if they were interested in those, and if there is anything else. And predominantly those four came back as things that people wanted to hear about.
Well, so how do I choose?
I really donât know how to tell you how to choose yourself, but I can walk you through the sequence that I went through. Kind of like buying a car, I listed my requirements down and then test drove them. So what are my requirements? Setup and teardown of test is an important consideration to me. So multi level fixtures are important. My philosophy towards software development means that I write tests for functionality that isnât there yet, so being able to mark a test as expecting it to fail is important. I think itâs important to be able to continue through a test function after a failure, very unlike the assert statement. Next up, easy to write tests, minimum boilerplate code, some easily parsable reporting. We also have, since I am working with tests instruments and there is things that I want to run between the tests for every test, even if somebody forgets to put it in there- putting in required fixtures will be good. I didnât know if that was possible. Ok, so those are my requirements. Iâm not going to- I did experiment with doctest, it was a fun experiment, I did write up my findings on the website, but I am not going to talk about it here because itâs just painful, my experiment mainly is for me, it told me that it was too painful to use as a functional test framework. If you guys out there use it a lot, great, it wasnât for me. However, I am going to discuss how unittest, nose and pytest stacked up with the requirements I needed.
Regarding multilevel fixtures, what I mean by that is being able to- well, they all do it, unittest, nose and pytest all support multi level fixtures, I can have a module level setup, followed by a class level setup, followed by a functional level setup and they can interact together. When setup takes a long time, this is a good thing to do in a module or class level setup, and have a method level setup just make sure that the setup is still correct. Thatâs what I use it for. So they all support it, however, pytest named fixtures are way more powerful, and take you a lot farther, the fixtures can be- fixtures can use fixtures, the fixture failure handling is smoother I believe than nose and unittest, the fixture model helps you think more of individual resource needs instead of just the entry points in setup. Just, I like it better. Ok, so multi level fixture- all of them work but I like pytestâs use of fixtures better.
About expected failures and skipping. Actually, this is something that is interestingly absent in nose, that I know of, I couldnât find anywhere how to get expected failures to work right in nose. Unittest does do this, unit test and pytest both support marking a test as expecting to fail and skipping it. The interesting thing I think is unittest allows a skipif function, actually both unittest and pytest do this, so you can say like, you know, if you are in a certain version of the software skip this test, or really, anything. However, for the marking of expected failure, pytest does support that, pytest allows you to say if in this version itâs going to fail, I know this. unittest doesnât have a expected failure if statement, not sure why. And, also the reporting of expected failures and skipping, I donât know, I kind of like how pytest reports that a little bit better than unittest.
How about the requirement of continuing through a function after a failure? For a lot of functional tests and a lot of tests in general, it makes sense. So letâs say Iâve got, I donât know, Iâve got a function or a setting that has side effects. It has five side effects. I want to make sure that all five of those side effects happen. Iâll have five assert statements, or expect statements, or something. And I actually want to know the result of all of them, even if one of them failed. By default actually all of these frameworks, unittest, nose and pytest will stop at the first assert. Thatâs not unexpected and assert is an exception. I kind of was desperate for this, so I made a work around. And I wanted my first attempt at a work around to be framework agnostic so it would work in all the frameworks. I described it in the post titled Delayed Assert. I also incorporated that solution into a little bit cleaner solution for as a pytest plugin called pytest expect; itâs not on PyPi but it is availabe at my site. So, I added that requirement. I made a work around that would work in all those. It works cleaner in pytest. I think that I could probably make plugins for unittest and nose as well. I just havenât done that yet.
How about easy to write tests? After learning about all of them, I didnât think any of them are hard, unittest, nose, pytest they are all pretty easy to write tests. The pytest assert rewriting, which I am going to talk about that in a future episode, but the assert rewriting is pretty cool, and it makes things a bit easier so you donât have to remember or lookup a bunch of the fancy assert helper functions that you do in unittest and nose. Also the pytest fixture model makes it easier to write tests. There is more features like parameterized test functions. Yeah, there is some extra fun in pytest to make it little bit easier to write, I believe.
How about minimal boiler plate? I put this in my requirements because I did see some posts that said that unittest had too much boilerplate, but itâs really not that much boilerplate, you have to import unittest into your file and all of your tests have to be members of a class derived from unittestâs TestCase. Thatâs, itâs not that big of a deal, I donât think, forcing people to write in classes when they donât want to, even though they are not really using them as objects, that is a little weird.
How about reporting? I donât know, this is just sort of subjective, they all report test failures and although, you know, like I said, nose doesnât deal with the expected failures that well, but both unittest and pytest reports are acceptable. I like the pytest reporting better.
Required fixtures. Iâd like to have setup and teardown that runs on every test. For my reason, I work a lot with test instruments. The instrument itself has an error log, for instance, as one example. And at the beginning of the test run itâs good to clear out the error log and then after a test runs, an individual test checks the error log and reports an error and fails the fixture if there are any errors in the error log. Itâs kind of handy to put â if you are doing it in every test â itâs handy to put it in the fixtures. You can do this with fixtures with nose and unittest but the autouse feature of pytest fixtures make it trivial.
For me, pytest was a clear winner, given my requirements, but you know, everybody is going to have different requirements. One thing to consider is the extensions and plugins. Thereâs extensions and plugins for all of these, and you may feel like itâs required- I mean, if youâve got the required extension that you canât switch over, than obviously you are not really deciding on which framework, youâve already made your decision. And keep in mind though that itâs probably not that terrible to switch. Get started testing. Write your tests. Just pick a framework and get started and if you change your mind in a few months, this probably is not going to be terrible to switch. Keep in mind also that pytest is really good at running unittest and nose. If you switch from unittest or nose to pytest it wonât be too painful. Also, interesting to note, that a lot of projects are migrating from unittest or nose to pytest. I have not heard of anybody moving away from pytest to unittest or nose.
The other thing that I wanted to bring up is speed. I have heard people say that theyâve noticed that pytest was, it just seemed a little slower to start up than the others. I actually havenât noticed that. But for me, if it is slightly slower itâs not going to matter too much really because, well, for three reasons: one, itâs apparently not slow enough for me to notice it, and second reason- development time for me is way more valuable than computer time. So if Iâve got to let the computer run an extra second but it saves me hours of writing tests, Iâll save my time and let the computer run a little bit longer. The third reason is the tests that I am writing, the communication and data transfer time is way more, itâs way more time than the actual pytest functionality is going to be taking up anyway, so pytest isnât going to be the bottleneck in my tests; but if speed really is an issue for you, I guess you can try them out, compare them, do some time tests and figure out if itâs an issue.
So, thatâs really all my advice- itâs not really advice itâs just kind of how I went through it, how I went through those decisions. But, I want to stress something; picking a framework can seem like the most important thing that you have to figure out, because you canât go- I was there, you canât go forward until you pick one and it seems like a big decision that you canât change later. But, you can change it later, and also, it shouldnât consume too much of your time, I donât think. Just pick something and move on, and switch if you need to, but donât let this decision consume too much of your time. So I guess what I am saying is there is way more important work for you to be doing. You need to be making decisions about what to test on your project, and how to test it. Donât get too hung up on which framework.
So, thatâs it for now, Iâd love to hear from you about what you use, and how you came to those decisions yourself, and also if you have a particular problem with one of the frameworks Iâd like to hear about it. On Twitter I am @brianokken and this podcast is @testpodcast. So, thanks, Iâll see you next time.