Table of Contents
I think of pytest as the run-anything, no boilerplate, no required api, use-this-unless-you-have-a-reason-not-to test framework.
This is really where testing gets fun.
As with previous intro’s on this site, I’ll run through an overview, then a simple example, then throw pytest at my markdown.py project.
I’ll also cover fixtures, test discovery, and running unittests with pytest.
No boilerplate, no required api
The doctest and unittest both come with Python.
They are pretty powerful on their own, and I think you should at least know about those frameworks, and learn how to run them at least on some toy examples, as it gives you a mental framework to view other test frameworks.
With unittest, you a very basic test file might look like this:
import unittest
from unnecessary_math import multiply
class TestUM(unittest.TestCase):
def test_numbers_3_4(self):
self.assertEqual( multiply(3,4), 12)
The style of deriving from unittest.TestCase is something unittest shares with it’s xUnit counterparts like JUnit.
I don’t want to get into the history of xUnit style frameworks. However, it’s informative to know that inheritance is quite important in some languages to get the test framework to work right.
But this is Python. We have very powerful introspection and runtime capabilities, and very little information hiding. Pytest takes advantage of this.
An identical test as above could look like this if we remove the boilerplate:
from unnecessary_math import multiply
def test_numbers_3_4():
assert( multiply(3,4) == 12 )
Yep, three lines of code. (Four, if you include the blank line.)
There is no need to import unnittest.
There is no need to derive from TestCase.
There is no need to for special self.assertEqual(), since we can use Python’s built in assert statement.
Once you start writing tests like this, you won’t want to go back.
However, you may have a bunch of tests already written for doctest or unittest.
pytest can be used to run doctests and unittests.
You can extend pytest using plugins you pull from the pypi.org, or write yourself.
I’m not going to cover plugins in this article, but I’m sure I’ll get into it in a future article.
pytest example
Using the same unnecessary_math.py module that I wrote in the
doctest intro,
this is some example test code to test the ‘multiply’ function.
test_um_pytest.py
from unnecessary_math import multiply
def test_numbers_3_4():
assert multiply(3,4) == 12
def test_strings_a_3():
assert multiply('a',3) == 'aaa'
Running pytest
To run pytest, the following two calls are identical:
$ python -m pytest test_um_pytest.py
$ pytest test_um_pytest.py
And with verbose:
$ python -m pytest -v test_um_pytest.py
$ pytest -v test_um_pytest.py
I’ll use pytest, as it’s shorter to type.
Here’s an example run both with and without verbose:
$ pytest test_um_pytest.py
=================== test session starts ====================
collected 2 items
test_um_pytest.py .. [100%]
==================== 2 passed in 0.05s =====================
$ pytest -v test_um_pytest.py
=================== test session starts ====================
collected 2 items
test_um_pytest.py::test_numbers_3_4 PASSED [ 50%]
test_um_pytest.py::test_strings_a_3 PASSED [100%]
==================== 2 passed in 0.04s =====================
Testing markdown.py
The test code to test markdown.py is going to look a lot like the unittest version, but without the boilerplate.
I’m also using an API adapter introduced in a previous post.
Here’s the code to use pytest to test markdown.py:
from markdown_adapter import run_markdown
def test_non_marked_lines():
print ('in test_non_marked_lines')
assert run_markdown('this line has no special handling') == \
'this line has no special handling</p>'
def test_em():
print ('in test_em')
assert run_markdown('*this should be wrapped in em tags*') == \
'<p><em>this should be wrapped in em tags</em></p>'
def test_strong():
print ('in test_strong')
assert run_markdown('**this should be wrapped in strong tags**') == \
'<p><strong>this should be wrapped in strong tags</strong></p>'
And here’s the output:
$ pytest test_markdown_pytest.py
======================== test session starts =========================
collected 3 items
test_markdown_pytest.py FFF [100%]
============================== FAILURES ==============================
_______________________ test_non_marked_lines ________________________
def test_non_marked_lines():
print("in test_non_marked_lines")
> assert (
run_markdown("this line has no special handling")
== "<p>this line has no special handling</p>"
)
E AssertionError: assert 'this line ha...cial handling' == '<p>this line... handling</p>'
E
E - <p>this line has no special handling</p>
E ? --- ----
E + this line has no special handling
test_markdown_pytest.py:15: AssertionError
------------------------ Captured stdout call ------------------------
in test_non_marked_lines
______________________________ test_em _______________________________
def test_em():
print("in test_em")
> assert (
run_markdown("*this should be wrapped in em tags*")
== "<p><em>this should be wrapped in em tags</em></p>"
)
E AssertionError: assert '*this should...d in em tags*' == '<p><em>this ...tags</em></p>'
E
E - <p><em>this should be wrapped in em tags</em></p>
E ? ^^^^^^^ ^^^^^^^^^
E + *this should be wrapped in em tags*
E ? ^ ^
test_markdown_pytest.py:23: AssertionError
------------------------ Captured stdout call ------------------------
in test_em
____________________________ test_strong _____________________________
def test_strong():
print("in test_strong")
> assert (
run_markdown("**this should be wrapped in strong tags**")
== "<p><strong>this should be wrapped in strong tags</strong></p>"
)
E AssertionError: assert '**this shoul...strong tags**' == '<p><strong>t...</strong></p>'
E
E - <p><strong>this should be wrapped in strong tags</strong></p>
E + **this should be wrapped in strong tags**
test_markdown_pytest.py:35: AssertionError
------------------------ Captured stdout call ------------------------
in test_strong
====================== short test summary info =======================
FAILED test_markdown_pytest.py::test_non_marked_lines - AssertionError: assert 'this line ha...cial handling' == '<p>this...
FAILED test_markdown_pytest.py::test_em - AssertionError: assert '*this should...d in em tags*' == '<p><em>...
FAILED test_markdown_pytest.py::test_strong - AssertionError: assert '**this shoul...strong tags**' == '<p><str...
========================= 3 failed in 0.21s ==========================
You’ll notice that all of them are failing.
This is on purpose, since I haven’t implemented any real markdown code yet.
However, the formatting of the output is quite nice.
It’s quite easy to see why the test is failing.
pytest fixtures
Functions that do work before and/or after test functions run are collectively called “fixtures”. pytest fixtures are covered in several posts.
Here are a few:
Test discovery
The unittest module comes with a ‘discovery’ option.
Discovery is just built in to pytest.
Test discovery was used in my examples to find tests within a specified module.
However, pytest can find tests residing in multiple modules, and multiple packages, and even find unittests and doctests.
To be honest, I haven’t memorized the discovery rules.
I just try to do this, and at seems to work nicely:
- Name my test modules/files starting with ’test_’.
- Name my test functions starting with ’test_’.
- Name my test classes starting with ‘Test’.
- Name my test methods starting with ’test_'.
If I do all of that, pytest seems to find all my code nicely.
If you are doing something else, and are having trouble getting pytest to see your test code,
then take a look at the pytest discovery documentation.
Running unittest or doctest from pytest
pytest can be used to run existing unittest and doctest tests.
See:
Examples on github
All of the examples here are available in the markdown.py project on github.