Table of Contents
The doctest test framework is a Python module that comes prepackaged with Python. This post covers the basics of how to put doctests in your code, and outside of your code in a separate file.
We start with a simple working example using a silly script called unnecessary_math.py.
Then we go on to show how I’m using doctest to test markdown.py.
conceptual model of Python doctest
This is from python.org:
The doctest module searches for pieces of text that look like interactive Python sessions,
and then executes those sessions to verify that they work exactly as shown.
I like to think of using doctest as if I’m actually sitting at a Python interactive prompt, and typing stuff in. Using doctest is like saying “My session should look exactly like this. If it doesn’t something is wrong.”
Actually, I think some people do use it that way. They write some module, and then demonstrate how it works in an interactive shell, and copy/paste the session into a docstring as their doctests.
If you are trying to use doctest for TDD, and we’re just defining what we want the outcome to be BEFORE we write the code, it can be a little tricky, as doctest is super picky, even about wite space.
When using doctest and TDD, it can end up getting rather iterative:
- Write some doctests
- Run the doctests to see that they fail
- Write some code that should make it pass
- If it still fails, examine the failure.
- If it is a false failure, and the doctest is just being too picky, then modify the doctest, possibly with doctest flags, then go to 2.
- If it is a real failure, fix the code, then go to 2.
I have found that some of the nitpicky aspects of doctest can be minimized with the use of an api adapter. I’ll be using an adapter in the markdown.py example in this post.
doctest example
Here is a simple module with one function in it, along with two doctests embedded in the docstring.
This example, and other code in this post, can be found at github.com/okken/markdown.py.
This particular file is in the repo under the directory “simple_example”.
Each >>> line is run as if in a Python shell, and counts as a test.
The next line, if not >>> is the expected output of the previous line.
If anything doesn’t match exactly (including trailing spaces), the test fails.
unnecessary_math.py
def multiply(a, b):
"""
>>> multiply(4, 3)
12
>>> multiply('a', 3)
'aaa'
"""
return a * b
running doctest
You run doctest like this:
> python -m doctest <file>
or
> python -m doctest -v <file>
The ‘-v’ means verbose. Verbose is real handy when testing your doctests, since doctest doesn’t output anything if all of the tests pass.
When we run without -v, it looks like nothing happens.
> python -m doctest unnecessary_math.py
No output is good though, it means nothing failed.
Even if nothing fails, I like running with -v to make sure everything is ok.
> python -m doctest -v unnecessary_math.py
Trying:
multiply(4, 3)
Expecting:
12
ok
Trying:
multiply('a', 3)
Expecting:
'aaa'
ok
1 items had no tests:
unnecessary_math
1 items passed all tests:
2 tests in unnecessary_math.multiply
2 tests in 2 items.
2 passed and 0 failed.
Test passed.
>
doctests in a separate file from the code
One of the really cool features of doctest is the ability to put your doctests in a text file. This is especially useful for functional testing, since that allows you to use doctest to test even non-Python interfaces.
For our simple math example, I can just put the same code from the docstring into a text file.
test_unnecessary_math.txt:
>>> from unnecessary_math import multiply
>>> multiply(3, 4)
12
>>> multiply('a', 3)
'aaa'
running doctests in separate file
Running doctest on a file is the same as running it on a module.
> python -m doctest test_unnecessary_math.txt
Again, no output means it’s passing.
> python -m doctest -v test_unnecessary_math.txt
Trying:
from unnecessary_math import multiply
Expecting nothing
ok
Trying:
multiply(3, 4)
Expecting:
12
ok
Trying:
multiply('a', 3)
Expecting:
'aaa'
ok
1 items passed all tests:
3 tests in test_unnecessary_math.txt
3 tests in 1 items.
3 passed and 0 failed.
Test passed.
Cool.
example with markdown.py
For markdown.py, I don’t want to include doctests in the code. Since I’m only testing the external CLI (through an adapter), I will be using the ‘doctests in a text file’ method.
I’m not going to write tests for the entire syntax right away. My first three tests will be for paragraphs, single asterisk em tags, and double asterisk strong tags.
The test really starts at the first >>>.
Everything before that is essentially a comment.
test_markdown_doctest.txt:
To run: python -m doctest test_markdown_doctest.txt
or: python -m doctest -v test_markdown_doctest.txt
>>> from markdown_adapter import run_markdown
>>> run_markdown('paragraph wrapping')
'<p>paragraph wrapping</p>'
>>> run_markdown('*em tags*')
'<p><em>em tags</em></p>'
>>> run_markdown('**strong tags**')
'<p><strong>strong tags</strong></p>'
Well, that’s simple enough. I’ve imported ‘run_markdown’ from my api adapter. Then I throw some example strings into the script and show what I expect to come out.
testing markdown.py
Here’s the output of running doctest on my text file.
> python -m doctest test_markdown_doctest.txt
**********************************************************************
File "test_markdown_doctest.txt", line 6, in test_markdown_doctest.txt
Failed example:
run_markdown('paragraph wrapping')
Expected:
'<p>paragraph wrapping</p>'
Got:
'paragraph wrapping'
**********************************************************************
File "test_markdown_doctest.txt", line 9, in test_markdown_doctest.txt
Failed example:
run_markdown('*em tags*')
Expected:
'<p><em>em tags</em></p>'
Got:
'*em tags*'
**********************************************************************
File "test_markdown_doctest.txt", line 12, in test_markdown_doctest.txt
Failed example:
run_markdown('**strong tags**')
Expected:
'<p><strong>strong tags</strong></p>'
Got:
'**strong tags**'
**********************************************************************
1 items had failures:
3 of 4 in test_markdown_doctest.txt
***Test Failed*** 3 failures.
And with verbose.
> python -m doctest -v test_markdown_doctest.txt
Trying:
from markdown_adapter import run_markdown
Expecting nothing
ok
Trying:
run_markdown('paragraph wrapping')
Expecting:
'<p>paragraph wrapping</p>'
**********************************************************************
File "test_markdown_doctest.txt", line 6, in test_markdown_doctest.txt
Failed example:
run_markdown('paragraph wrapping')
Expected:
'<p>paragraph wrapping</p>'
Got:
'paragraph wrapping'
Trying:
run_markdown('*em tags*')
Expecting:
'<p><em>em tags</em></p>'
**********************************************************************
File "test_markdown_doctest.txt", line 9, in test_markdown_doctest.txt
Failed example:
run_markdown('*em tags*')
Expected:
'<p><em>em tags</em></p>'
Got:
'*em tags*'
Trying:
run_markdown('**strong tags**')
Expecting:
'<p><strong>strong tags</strong></p>'
**********************************************************************
File "test_markdown_doctest.txt", line 12, in test_markdown_doctest.txt
Failed example:
run_markdown('**strong tags**')
Expected:
'<p><strong>strong tags</strong></p>'
Got:
'**strong tags**'
**********************************************************************
1 items had failures:
3 of 4 in test_markdown_doctest.txt
4 tests in 1 items.
1 passed and 3 failed.
***Test Failed*** 3 failures.
As you can see. Once you’ve convinced yourself that your tests are correct, the verbose setting doesn’t add much. You will get plenty of output without verbose if there are errors.
In my case, everything FAILED!!!. But that’s good, because I haven’t implemented anything real yet, I just have a stub.
more doctest info
All of the examples in this post are available in the github markdown.py project.The math example is in a folder called ‘simple_doctest_example’.
The python.org site has pretty good information about using doctest.
On that same page is the writeup on how to use text files for your doctests.
Doug Hellmann has a great writeup on doctest that I highly recommend.It’s called Testing through documentation and it covers many of the problems that you may run into including dealing with multiple lines, whitespace, unpredictable output, etc.
I will cover some of these aspects as I get further into the implementation and testing of markdown.py.
next
Next up, I’ll take a look at implementing the same tests using unittest, the other test framework that’s part of the Python standard library.