This is a set of tips/tricks for learning and using pytest.
I’ll probably build on the list, so feel free to share with me items you think should be in the list.
Many items really deserve more explanation, and maybe full posts. Let me know if there’s something you’d like more in depth discussion of.

Note: I’m not numbering these, because I don’t want to keep track of order. Also, this is more of a brain dump, and not a prioritized list.

Special thanks to Michael Kennedy for prodding me into making this, and contributing to the initial list.

General tips

  • pytest is really easy to use simply, but don’t stop there.
    • “Write test modules and functions starting with test_ and use assert” is about all you need to know to get started.
    • It doesn’t take a lot of study to get tons more power out of pytest
    • The pytest course is a few hours.
    • The pytest book can be read in a weekend, easily.
    • Especially with teams, the book or course can provide common context about what you all know.
  • pytest has a lot of power
    • Don’t try to use it all at once.
  • Keep tests simple
    • add fixtures, parametrization, etc as necessary
  • Remember --help.
    • It expands to include plugin help and your own extensions.
    • The starter default list is online under Command-line Flags
  • pytest reference pages are darned good.

Structuring test functions

  • Arrange/Act/Assert or Given/When/Then are great templates for most of your tests.
    • The best “template” is just:
    def test_something():
    # get stuff ready for action
    ...
    # do action
    ...
    # assert the acction worked
    ...
  • Workflow tests with interleaved assertions aren’t evil
    • They’re just trickier to deal with.

Structuring a test suite

  • Utilize a directory structure to match how you like to run your code.
    • Because running a full subdirectory is easy.
    • I like to separate into functionality and features.
    • Some people like test structures based on subsystems.
    • Some based on the code structure.
  • You can also use -k sometext to filter directories or classes or test prefixes.
    • -k has logic. Examples:
      • -k "sometext and not othertext"
      • -k "(this or that) and not unwanted"

Fixtures

  • Use --fixtures.
    • It lists available fixtures, their scope, and where they’re defined.
    • You can specify a directory, file, or even class or function to find out fixtures available to just that bit of code.
  • Move complex setup into fixtures, especially when re-usable for other tests.
  • Use yield to separate setup and teardown.
  • The setup and teardown sections can be empty.
  • Beware addfinalizer().
    • It’s still supported, but will confuse people.
  • Leverage fixture scopes to improve performance.
    • Ex:
      • session scope for setting up a resource
      • function scope for setting resource to known state
  • Share fixtures between test modules/directories with conftest.py files.

Built in fixtures

  • Fixtures reference
  • There’s a few that everyone will eventually use.
    • tmp_path - Provide a pathlib.Path object to a temporary directory which is unique to each test function.
    • tmp_path_factory - Make session-scoped temporary directories and return pathlib.Path objects.
      • use tmp_path/tmp_path_factory instead of tmpdir/tmpdir_factory.
        • The path versions are pathlib.Path objects.
        • The dir versions are py.path.local objects, a legacy type.
    • capsys - to check stdout/stderr and to disable capturing.
    • monkeypatch - Temporarily modify classes, functions, dictionaries, os.environ, and other objects.
    • pytestconfig - Access configuration values and command line flags.
    • request - Provide information on the executing test function.
      • request.node.name is the test function name.

Markers

  • Use your own @pytest.mark.foo custom markers to group tests or selectively disable tests or lots of other fun tricks.
    • But don’t go too crazy with this.
    • Remember to register markers in pytest.ini file.
    • Run with --strict-markers to turn marker typos (unregistered markers) into errors.
  • You can mark all tests in a file with pytestmark 
    import pytest
pytestmark = pytest.mark.foo
    • or multiples
    import pytest
pytestmark = [pytest.mark.foo, pytest.mark.bar, pytest.mark.baz]
  • Use --markers to see available markers.
  • Play with built-in markers skip, skipif, and xfail.
    • they are useful when you need them.
  • xfail
    • xfail results in either xpass or xfail if non-strict.
      • fail or xfail if strict
    • strict can be set at decorator or globally in pytes.ini
    • You can set xfail_strict = True in pytest.ini to turn all xpass results into failures.
    • Use xfail_strict.
    • Link a defect number to the reason.
    • Use -ra, or at least -rxX to see reasons.
      • But really, just use -ra to see all non-passing reasons.
    • --runxfail basically ignores xfail marks.
      • Very useful for final stages of pre-production testing.

Parametrization

  • Learn parametrization and when to use it effectively.
  • When you find yourself using “copy/paste/modify” to create new test cases, parametrization may be called for.
  • Prefer function parametrization.
  • Use fixture parametrization if work per parameter is needed.
  • Use pytest_generate_tests based parametrization when you absolutely know it’s the only way to solve your needs.
  • Utilize ids functions to get more human readable parametrized test cases.
  • Parameters can be marked.

Plugins

  • Plugins are awesome.
  • Writing your own plugins isn’t too hard, but can be confusing at first.
    • pytest.org has a tutorial
    • The pytest book has a chapter on it.
    • I’m talking about the process during a talk at PyCascades:
    • I’m planning a mini-course around it, not recorded yet.
    • The complicated bits
      • Testing your plugin - see pytester or book or talk.
      • Packaging - same woes as the rest of Python, plus remember the pytest entry point, noted in the pytest.org tutorial, book, talk, etc.
    • But seriously, it’s not that bad. :)
  • Some of my favorites:
  • A few by me

Configuration

  • You should have a config file, either pytest.ini, or tox.ini, pyproject.toml, setup.cfg with a pytest section.
    • The config file determines the root for testing.
    • Stick it at the topmost point you’re likely to every run pytest from.
    • See Configuration for more info
  • I put this in almost every project: 
    addopts =
    --strict-markers
    --strict-config
    -ra
  • Book mark the reference to config options: Configuration Options

Debugging

  • Great debugging flags:
    • -s - turn off output capture
    • -v - verbose
    • -vv - more verbose
    • --tb - traceback style: auto/long/short/line/native/no
      • I like --tb=short for default setting in config file, then others for debugging.
    • -x - stop on first failure
    • --lf - run the last failed tests
    • --ff - failed first. Start with the failures, then run everything
    • --nf - run new test files first, then the rest sorted by file modification time
    • --sw - run last failed, then stop on first failure, then next time continue from last failing.
      • kinda like combining --lf -x but even cooler.
    • --sw-skip - just like above, but skip one failed test.
    • --pdb - start Python debugger on error.
      • Super useful for debugging with tox.