A test stops execution once it hits a failing assert statement.
That’s kinda the point of an assert statement, though, so that’s not surprising.

However, sometimes it’s useful to continue with the test even with a failing assert.

I’m going to present one method for getting around this restriction, to test multiple things, allow multiple failures per test, and continue execution after a failure.

I’m not really going to describe the code in detail, but I will give the full source so that you can take it and run with it.

• Reasons for multiple assert statements and not stop execution
• Using a failure list to keep track of failures within a test
• Example test code that uses the delayedAssert module
• And an example for unittest
• The output for unittest
• The output for pytest
• The output for nose
• The delayedAssert.py module
• Feedback welcome

Reasons why you might want multiple assert statements in a test, and not stop execution

• Several related variables all need to be tested.
• Showing all of the failing conditions can help diagnose what the problem is.
• The setup for the function is significant enough that it just makes sense to check a bunch of stuff in one test.
• Functional testing is just easier if there is a way to mark a test as failing without stopping the execution.
• Mainly, assert is intended to mean that the code can’t meaningfully continue. However, testing frameworks have added the meaning of failing to the assert.

Using a failure list to keep track of failures within a test

The method I often us is to:

1. Use some function other than assert to check for failures.
2. Stuff all failures into a list.
3. At some point before the end of the test, assert that the list is empty.

I’d like to make sure the test failures are formatted meaningfully and are reasonably as easy to read as an assert failure.

I also want this functionality to work with unittest, nose, pytest, or any other framework that can use assert to fail a test.

It is possible to extend this to a pytest plugin so that it runs even easier.
I’ll write about that in a future post.

Since I’m going to reuse this for many test modules, I’ll stick these failure functions in their own module.

And an example for unittest

test_delayed_assert_unittest.py
</p> <pre> from delayed_assert import expect, assert_expectations import unittest

class DelayedAssertTest(unittest.TestCase):

def test_should_pass(self):
    expect(1 == 1, 'one is one')
    assert_expectations()

def test_should_fail(self):
    expect(1 == 2)
    x = 1
    y = 2
    expect(x == y, 'x:%s y:%s' % (x,y))
    expect(1 == 1)
    assert_expectations()

</pre> <p>

The output for unittest

</p> <pre> $ python -m unittest test_delayed_assert_unittest F. ====================================================================== FAIL: test_should_fail (test_delayed_assert_unittest.DelayedAssertTest) ---------------------------------------------------------------------- Traceback (most recent call last): File "test_delayed_assert_unittest.py", line 16, in test_should_fail assert_expectations() File "delayed_assert.py", line 36, in assert_expectations assert False, _report_failures() AssertionError:

assert_expectations() called from “test_delayed_assert_unittest.py” line 16, in test_should_fail()

FAILED (failures=1) </pre> <p>

The output for pytest

</p> <pre> $ python -m pytest test_delayed_assert.py ====================================== test session starts ======================================= platform darwin -- Python 2.7.9 -- py-1.4.26 -- pytest-2.6.4 collected 2 items

test_delayed_assert.py .F

def test_should_fail():
    expect(1 == 2)
    x = 1
    y = 2
    expect(x == y, 'x:%s y:%s' % (x,y))
    expect(1 == 1)

def assert_expectations():
    'raise an assert if there are any failed expectations'
    if _failed_expectations:

delayed_assert.py:36: AssertionError =============================== 1 failed, 1 passed in 0.03 seconds =============================== </pre> <p>

The output for nose

</p> <pre> $ python -m nose test_delayed_assert.py .F ====================================================================== FAIL: test_delayed_assert.test_should_fail ---------------------------------------------------------------------- Traceback (most recent call last): File "/Library/Python/2.7/site-packages/nose-1.3.4-py2.7.egg/nose/case.py", line 197, in runTest self.test(*self.arg) File "/Users/okken/Dropbox/delayed-assert/test_delayed_assert.py", line 13, in test_should_fail assert_expectations() File "/Users/okken/Dropbox/delayed-assert/delayed_assert.py", line 36, in assert_expectations assert False, _report_failures() AssertionError:

assert_expectations() called from “test_delayed_assert.py” line 13, in test_should_fail()

FAILED (failures=1) </pre> <p>

The delayedAssert.py module

I don’t have this on gitHub or anything yet. But feel free to copy it and use it if you like.
Not sure what more to say about it.

delayedAssert.py
</p> <pre> ''' Implements one form of delayed assertions.

Interface is 2 functions:

from expectations import expect, assert_expectations

def test_should_pass():
    expect(1 == 1, 'one is one')
    assert_expectations()

def test_should_fail():
    expect(1 == 2, 'one is two')
    expect(1 == 3, 'one is three')
    assert_expectations() 

</pre> <p>

Feedback welcome

Please let me know what you think.
I’m sure some of you have some better ways to accomplish something similar.
Another thing I’d like help with is the names.
I’m not really thrilled with “expect()” and “assert_expectations()”