In Delayed assert / multiple failures per test, I presented a first attempt at writing an ‘expect()’ function that will allow
a test function to collect multiple failures and not stop execution until the end of the test.

There’s one big thing about that method that I don’t like.
I don’t like having to call ‘assert_expectations()’ within the test.
It would be cool to push that part into a plugin.

So, even though this isn’t the prettiest code, here’s a first attempt at making this a plugin.

Test code that uses expect()

The main goal for this first iteration is to remove the ‘assert_expectations()’ call from the tests.
I’d like to have my tests just call ‘expect()’, like this:

test_delayed_assert.py
`

from delayed_assert import expect

def test_should_pass():
    expect(1 == 1, 'one is one')

def test_should_fail():
    expect(1 == 2)
    expect(3 == 4, 'three is four')

`

Local conftest.py plugin for delayed assert

The ‘expect()’ calls are going to generate a list of failures.
We’ve got to grab that list and make the test fail somehow.

First, it seems like it would be good to make sure the ‘_failed_expectations’ list is empty when the test starts, so I’ve added a call to clear it in ‘pytest_runtest_setup()’.

Then, in ‘pytest_report_teststatus()’, I check for failures and fill in ‘report.outcome’ and ‘report.longrepr’.

conftest.py
`

import delayed_assert 

def pytest_runtest_setup(item):
    delayed_assert.clear_expectations()

def pytest_report_teststatus(report):
    if report.when == "call":
        if not report.failed:
            if delayed_assert.any_failures():
                report.outcome = "failed"
                report.longrepr = delayed_assert.get_failure_report()
                delayed_assert.clear_expectations()

`

Changes to delayed_assert.py

The code in ‘conftest.py’ calls for some new functions in ‘delayed_assert.py’.
Here are the changes to ‘delayed_assert.py’ to make this work.

delayed_assert.py
`

# ---- Called from tests

def expect(expr, msg=''):
    if not expr:
        _log_failure(msg)

# ----- Called from pytest plugin

def clear_expectations():
    global _failed_expectations
    _failed_expectations = []

def any_failures():
    return bool(_failed_expectations)

def get_failure_report():
    if any_failures():
        _failed_expectations.append('Failed Expectations:%s' % len(_failed_expectations))
        return ('\n'.join(_failed_expectations))
    else:
        return ''

# ------ Keeping _log_failure separate, mostly because it's ugly code

import inspect
import os.path

_failed_expectations = []

def _log_failure(msg=''):
    (filename, line, funcname, contextlist) =  inspect.stack()[2][1:5]
    filename = os.path.basename(filename)
    context = contextlist[0]
    msg = '%s\n' % msg if msg else ''
    _failed_expectations.append('>%s%s%s:%s\n--------' % (context, msg, filename,line))

`

Seeing it in action

Just to show you that this actually works.

test run
`

$ python -m pytest -s test_delayed_assert.py 
===================================== test session starts =====================================
platform darwin -- Python 2.7.9 -- py-1.4.26 -- pytest-2.6.4
collected 2 items 

test_delayed_assert.py .F

========================================== FAILURES ===========================================
______________________________________ test_should_fail _______________________________________
>    expect(1 == 2)
test_delayed_assert.py:7
--------
>    expect(3 == 4, 'three is four')
three is four
test_delayed_assert.py:8
--------
Failed Expectations:2
============================= 1 failed, 1 passed in 0.01 seconds ==============================

`

Possible issues and things I don’t like

This appears to work, at least in this simple test case.

I’m not thrilled that the functionality is split between two files, ‘conftest.py’ and ‘delayed_assert.py’. Not sure if it’s just me being picky though.

Here’s my initial list of possible problems.

  1. The ‘_failed_expectations’ list may not work great for parallelization, for instance with xdist.
  2. The expect() function would be cooler if it did some assert rewrite like voodoo that pytest uses to make the report very readable. Not a requirement, but would be cool.
  3. Not sure I’m thrilled with the name “delayed assert”, since the function is “expect()”, and now there is no assert. But ‘expect()’ means something different to some people, and there’s already a pytest-expect in the repository. Maybe ‘multi-fail’? Nah. That’s lame.
  4. … Really, I had more things I didn’t like about it. I think the cold medicine and porter are mixing and, … they’re gone.

I’m sure everyone reading this can find holes in my solution.
Please let me know.

Alternative solutions

A couple of folks have indicated that it would be better to hook into ‘pytest_runtest_makereport’ instead of ‘pytest_report_teststatus’.
That may be completely valid, but I haven’t played with trying to hook in there yet.

I’ve also thought about making ‘expect’ a pytest fixture, which would definitely make it possible to push everything into one file. Just not sure if I like having the tests have to declare an expect fixture.
However, that might not be that bad.

Next steps

I’d like to get some feedback on this and some ideas on how to improve it.

I’d also like to continue down the path of showing how to make this an installable pytest plugin and package and such.

Then perhaps push it to github or bitbucket, to allow contributions from others.

…

Anyway, it’s kind of fun putting this together in public. Warts and all.