pytest slow order

I received a question the other day about combining the notion of marking slow tests and ordering them to run the slow tests first.

This post describes a bit of background and a solution to the problem.

@pytest.mark.slow

It’s possible to mark slow tests with @pytest.mark.slow and then either run or skip the slow tests.

To run slow tests: pytest -m slow
To skip slow tests: pytest -m "not slow"

With the pytest-skip-slow plugin, you can:

skip the @pytest.mark.slow tests by default
include them with pytest --slow
and run only the slow tests with pytest -m slow --slow

The pytest docs also include DIY instructions for something similar to the plugin.

pytest-order and @pytest.mark.order

The pytest-order plugin allows you to change the relative test run order.

The plugin allows for a lot of control, but for our use, it has these features:

@pytest.mark.order("first") or @pytest.mark.order(0) marked tests will be run first.
@pytest.mark.order("last") or @pytest.mark.order(-1) marked tests will be run last.

Perfect, that’s what we needed. Now, putting them together.

conftest.py mods

Adding the following to a project conftest.py does the trick:

def pytest_addoption(parser):
    parser.addoption("--slow-first", 
                     action="store_true", 
                     default=False, 
                     help="run slow tests first")

    parser.addoption("--slow-last", 
                     action="store_true", 
                     default=False, 
                     help="run slow tests last")


def pytest_collection_modifyitems(config, items):
    if config.getoption("--slow-first"):
        run_first = pytest.mark.order("first")
        for item in items:
            if item.get_closest_marker("slow"):
                item.add_marker(run_first)

    elif config.getoption("--slow-last"):
        run_last = pytest.mark.order("last")
        for item in items:
            if item.get_closest_marker("slow"):
                item.add_marker(run_last)

Explanation

The pytest_addoption hook function adds the flags --slow-first nd --slow-last.

The pytest_collection_modifyitems hook function runs after all tests are collected. We look for those marked with slow, and add an order("first") or order("last") depending on the flag passed in.

Test run

This example requires:

slow to be declared in pytest.ini. (Or not if using pytest-skip-slow plugin. If you are using the plugin, pretend --slow is added to the below examples.)
pytest-order is installed: pip install pytest-order
The above changes are made to a conftest.py file

pytest.ini:

[pytest]
markers =
  slow: slow tests

test_slow_order.py:

import pytest

def test_a_zoom():
    ...

@pytest.mark.slow
def test_b_snooze():
    ...

def test_c_zoom():
    ...

@pytest.mark.slow
def test_d_snooze():
    ...

By default, tests run in the order they appear in the file:

$ pytest test_slow_order.py -v
============== test session starts ==============
collected 4 items                               

test_slow_order.py::test_a_zoom PASSED    [ 25%]
test_slow_order.py::test_b_snooze PASSED  [ 50%]
test_slow_order.py::test_c_zoom PASSED    [ 75%]
test_slow_order.py::test_d_snooze PASSED  [100%]

=============== 4 passed in 0.01s ===============

With --slow-first:

$ pytest test_slow_order.py -v --slow-first
============== test session starts ==============
collected 4 items                               

test_slow_order.py::test_b_snooze PASSED  [ 25%]
test_slow_order.py::test_d_snooze PASSED  [ 50%]
test_slow_order.py::test_a_zoom PASSED    [ 75%]
test_slow_order.py::test_c_zoom PASSED    [100%]

=============== 4 passed in 0.01s ===============

With --slow-last:

$ pytest test_slow_order.py -v --slow-last
============== test session starts ==============
collected 4 items                               

test_slow_order.py::test_a_zoom PASSED    [ 25%]
test_slow_order.py::test_c_zoom PASSED    [ 50%]
test_slow_order.py::test_b_snooze PASSED  [ 75%]
test_slow_order.py::test_d_snooze PASSED  [100%]

=============== 4 passed in 0.01s ===============

To learn more about pytest

If you’d like to become an expert at pytest, I suggest starting with The Complete pytest Course ;)

@pytest.mark.slow#

pytest-order and @pytest.mark.order#

conftest.py mods#

Explanation#

Test run#

To learn more about pytest#