A discussion of how to use the xfail feature of pytest to help with communication on software projects.
Transcript for episode 165 of the Test & Code Podcast
This transcript starts as an auto generated transcript.
PRs welcome if you want to help fix any errors.
00:00:01 I recently moved from one phase to the next on a software project, and it involved adjusting the X fail settings. It’s an important transition that I don’t think I’ve talked about yet, so we’ll do that now on this episode, a reasonable Xfail policy and workflow.
00:00:32 Welcome to Test and Code.
00:00:43 If you don’t use Xfail now, you don’t have to start many projects. Probably most don’t use X Fail. First, let’s describe what Xfail is and why I use it and some of the problems with using it. X Fail is a marker you can apply to a test that you expect to fail. There’s also a function you can call at runtime to X fail a test.
00:01:07 The end result of an X Fail marked test can be X Fail if it failed or X Pass if it passed. You can also set Xfail strict to turn all of the X passes into fails. We want all of our tests to pass, though, so why in the world would we use this feature for the projects I use it on? The projects are large, with many subsystems developed by many teams, some teams supporting multiple projects. The tests are system tests involving all of the components of the system, both hardware and software.
00:01:45 During the development phase of a new feature or set of features, it’s natural and even desirable to move fast, experiment, and iterate and even if nothing goes wrong, I advocate for tracer bullet style development to get the feature working through all the levels of software for a single happy path test case. Then expand the test cases to fully cover the range of the feature, including even testing out of bounds cases. Of course, during all of this, we want to utilize our system test suite. We want to know when regressions happen, things that used to work that no longer do. We want to gradually bring online the new test cases for the new feature.
00:02:28 What happens for failures during the development phase?
00:02:31 One policy could be use the system test and CI and reject any bills that have regressions, and also don’t bring online any new tests until they pass on the system. The benefits of this policy are that the suite for all completed builds is green. The downside is that no build, you don’t have any bill to debug with if you need to debug it, and you also can’t use system tests to help develop new features. So here’s an alternate policy that takes some more management, but it does have some benefits.
00:03:09 Of course, I want to have a subset of tests that are used as a gating criteria for continuous integration.
00:03:16 Some bills are just so broken that there’s no point in trying them and wasting people’s time with it for the rest of the tests. If test fails, we want to report the failure into a defect tracking system. If the issue is likely going to have a delay, maybe it’s a remote team or someone’s on vacation or the difficult problem they don’t take a while to fix, or it’s a lower priority problem, et cetera. But the software is still testable in other parts as is.
00:03:45 Then one option is to Mark the test as X Fail and set the reason listed with the defect number and a short description. Also, I recommend always running with the X Failed strict equals True which causes X passes to show up as fails if it’s a new feature or a new expansion of the feature that just hasn’t been fully implemented yet. Well, still, I will possibly use Xfail and set the reason to pointing to the story or the planning item or whatever. The benefits of this kind of policy with X Fail is that any test failure indicates that something needs attention. If it’s a failure because of an EXPass, that means that something that used to not work now is working. We can go and check the item number to see if the defect is closed by somebody, and if it’s closed and it’s working now, great.
00:04:45 If there any other failure will be a new failure that we need to look into or something that we’re waiting on a fixed for right away. There are downsides also to this policy.
00:04:55 An X fail marked test may fail for a different reason than the defect given, but you won’t know it. This is a great reason to try to have very focused feature tests.
00:05:07 Also, for parameterized tests, I tend to only Mark the parameterization that fails and not the whole test to isolate the problem down even further. To summarize, during development, make sure you have focused test cases.
00:05:23 Set X Fail Strict to True to make sure that we know when things start working again, and set the reason in X fails on other failures to a tracking number, defect item and a summary too. Now here’s the transition. When we move from development to feature freeze and all the way through the release of a new version, we want a different behavior.
00:05:50 The action that we need to do when we do this transition is we need to go in and modify our test runs our CI system probably to pass the flag. Run Xfail, run Xfailtest for the system test suites. This essentially makes pytest ignore the Xfail Mark completely.
00:06:11 What this does is it highlights and brings attention to all failures.
00:06:17 It gives you a clear picture are some of the failures for features that never got implemented yet or fully implemented. Maybe these features aren’t ready to release. Maybe they need to go into the next release. If you decide to do that, you can move these tests out of this test suite or dynamically skip them based on the version number. Are some issues still open? Great. Now you know, fix them or talk to the teams responsible and get them fixed or some issues closed. But the test deal still fails. That’s the discussion also maybe the issue really isn’t closed, it needs to be reopened. Or maybe it’s a new issue that the ex fail was hiding and maybe you need to file a new issue. In any case, any failure makes it so that you need to look into it. So here’s another summary. Always run with X fail strict equals true I think.
00:07:17 And then all of the X fails should have a reason with a tracking number in it so you can look it up to make sure it’s tracked. And when you switch from development to feature freeze turn on the run XFL flag also have focused tests. That’s it. This is a mindset shift though to do something like this.
00:07:37 Using XFL on a project requires managing the failures but not to be afraid of them and to not freak out. All failures in a test run should just be an indication that an action is needed, be it checking on fixed items or fixing an issue or filing a new issue.
00:07:59 It usually involves communication though, so if the people on your project are really bad at communication, XFL might not be good for you. Or maybe it is a good thing for you and you can use it as an opportunity to talk more to each other and fix those communication problems. Let me know what you think.
00:08:24 Thank you Patreon supporters for sponsoring the show. Join them by going to testingco.com support. By the way, I got to say there are some people who have been supporting for a lot of time. Thank you. It’s also a great way to hear about a new episode. I’ve been actually less frequent lately as you’ve probably noticed as I’m trying to finish the pytest book second edition, I only got one chapter left to right so the pace of the podcast should pick up shortly. But anyway, after I release a new episode I go directly to the Patreon page and notify the supporters. So if you want to get on this list it’s easy to sign up. You just go to testincode.com support and there’s a link there and the cheapest way if you want to save some money, the cheapest way is to a dollar an episode and then go and there’s a setting somewhere to limit your contribution to only one dollars a month. More would be great of course, but that is all that’s needed to get an email from me whenever I release news episode. Anyway, thanks to everybody that supported and listened and shared this with friends. That’s all for now. Go out and test something.