xfail isn’t just for pytest tests. Python’s unittest has @unittest.expectedFailure.
Transcript for episode 166 of the Test & Code Podcast
This transcript starts as an auto generated transcript.
PRs welcome if you want to help fix any errors.
00:00:01 Last week we talked about X fail and Pi Test. This week we’re going to talk about Unit Test and expected Failure.
00:00:24 Welcome to Test and Code.
00:00:34 I received a question from a listener of last week’s episode on X Fail. The general question was something like how do I separate my expected failures from the rest of the test suite if I’m not using pytest? Well, let’s set aside the idea that someone is on episode 165 of Test and Code and not using by test what’s going on there? But they didn’t say what framework they were using, so I’ll assume it’s Unit Test. Unit Test has an expected failure decorator. You can apply this decorator to any test that you think might fail or you know is going to fail. If a test marked as expected failure in Unit Test fails as it’s expected, the result is an X or it’ll say expected failure. If you’re running in verbosement, if it passes, the test reports as you or unexpected success.
00:01:32 That’s cool. If everything else is working as expected, but you have some expected failure tests that are passing, the end result is a failed test suite. This is actually a cool thing, and it’s much like the X fail strict in Pi test, which I recommend.
00:01:51 So if you’re using Unit Tests, you can Mark your test with expected failure. If you expect them to fail, you’ll be notified when they pass, which is good. And the tests that are failing that are expected to fail won’t fail your test suite. But now what happens when you move to a phase of development where you really don’t want any of those X fails anymore? With unit Test, you kind of just have to manually remove them.
00:02:20 There’s not any Unit Test legs to say that I know of that say, you know, ignore them or treat them any differently. However, there’s a lot of people that use that right Unit Tests, but use Pi Test as a test runner. And if you do that, you can use the pytest flags and it almost kind of works. Let me explain. There’s the Run X Fail flag that I talked about last week, and nicely it tries to turn off the expected failure decorators in Unit Test and it kind of works. What it does is if you’ve got failing tests that are marked expected failure and you use the Run X fail, those will be reported by pipe test as failures.
00:03:11 The issue really is that it doesn’t change the passing tests at all, and that actually might be a good thing. So what happens is the tests that pass that are marked with expected failure. If you run with the runax Fail flag, they will be reported as failures. It’s because of the mechanism that Unit Test uses to raise the unexpected pass thing. It ends up being just a failure, even with bytes. So that might work fine for you. So if that’s not what you want and you really want to completely ignore the expected failure you have a third option which is interesting.
00:03:58 If you’re using pipes as a test runner a lot and you want to use these flags a lot you can replace the unit test expected failure decorator with pytest Mark xfail.
00:04:12 The xfail Mark will work fine on a unit test test if run with PY test. So if you do that, I’m going to have an example in the show notes. It’s pretty simple. You can use all these basically all the behaviors just like the normal pipe test behavior that we talked about in episode 165. But what if you sometimes still run with unit test? So I have this little example. I ran it in unit test and unit test just seems to ignore those marks which as expected and it runs it as if there was no decorator. So if you have a marked as X fail, unit test will just say the test either passed or failed. Whether or not it passed or failed, it pretty much just ignores that. So that’s actually kind of cool. I do want to have a slight Disclaimer on all this so I am doing I did a lot of research on unit test and, you know, I’ve researched by test quite a bit but I’ve never used unit tests in production on a production project and I’ve also never used this combination of pipes and unit tests together. But I would like to hear from people that have so if you are using if you’re using expected failure in unit test differently than I am, I’d love to hear from you. Also, if you’re using Pyte and unit test together, I’d like to hear from you and kind of know why. It’s an interesting corner case or maybe it’s not a corner case. Maybe that’s what a lot of people are doing. And then if you’ve got another option for dealing with this that I’ve missed, I’d love to hear from you. So thanks thank you to Patreon supporters for supporting the show. Join them at testandcode.com support or just click the link in the show notes help support the show with as little as a dollar per month. That’s all for now. I’ll go out and test something.