Anthony Shaw joins Brian to discuss flaky tests and flaky test suites.
- What are flaky tests?
- Is it the same as fragile tests?
- Why are they bad?
- How do we deal with them?
- What causes flakiness?
- How can we fix them?
- How can we avoid them?
- Proactively rooting out flakiness
- Test design
- GUI tests
- Sharing solutions
Special Guest: Anthony Shaw.
Links:
- Dropbox article on flaky tests
- Microsoft article on flaky tests
- pytest-rerunfailures: a py.test plugin that re-runs failed tests up to -n times to eliminate flakey failures
- pytest-randomly: Pytest plugin to randomly order tests and control random.seed
- pytest-random-order: pytest plugin to randomise the order of tests with some control over the randomness
- math.isclose()
- numpy.isclose()
- pytest.approx() — approx
- Anthony's testing article on RealPython
- Ghost Inspector
- wily: A Python application for tracking, reporting on timing and complexity in tests