Dynamic scope fixtures is a new feature released in pytest 5.2. Anthony Sottile, a core pytest developers, joins Brian in discussing not only dynamic scope, but also what scope means, as well as parametrization.


Transcript for episode 90 of the Test & Code Podcast

This transcript starts as an auto generated transcript.
PRs welcome if you want to help fix any errors.


Pytis 5.2 was just released recently, and with it a cool, fun feature called Dynamic Scope Fixtures. Anthony so Tilly is one of the Pytech core developers, so I thought it would be fun to have Anthony describe this new feature for us. We also talk about parameterized testing and really, what is fixed or Scope? And then what is dynamic scope? Thank you to Regon for sponsoring this episode.

Welcome to Test and Code, a podcast about software development, software testing, and Python.

Today I’m Test And Code. I’m having Anthony on and it’s so Tilly. Yes, I got it right. Actually, I went back to episode 82 and listened again to what you said.

Nice. Perfect.

Anthony was on episode 82. We talked about a whole bunch of the cool features that have come out since Python back. Python since pytest three point out.

And now we’re at Python 5.2 and there’s some new stuff, but there’s an Oops also. So what’s with the Oops?

Yeah. So we didn’t mean to release Titus 5.2 as soon as we did.

We kind of messed up or get merge workflow because it’s a little bit complicated. We tend to merge branches back and forth. We keep features and bug fixes separately.

And Bruno accidentally merged the branch in the opposite direction. So usually we merge features into Master only when we do a feature release.

But we merge Master into features all the time. But we did it backwards and figured it’s easier to just cut a release than it is to try and undo the get oopsies.

Okay.

And so we made a release.

But it’s solid though, right? I mean, it passes tests and all that.

Yeah. Of course. We always test all of our code all the time. So it’s a solid release. It’s not like a beta release or anything.

It’s just as good as any other.

Okay, so one of the things that, as I was reading through the change list that popped out at me is the Dynamic Scope Fixtures.

Yeah, I’d like to talk about that, but we may have to back up a little bit just to make sure everybody understands what scope is.

Sure. Yeah. And actually, I think in order to explain Scope, I actually need to explain parameterization as well. So we’re just going to have the whole gamut today.

Okay. So where do we start?

Yeah, I guess we can start by just jumping into parameterization. That sounds fine. We’ll eventually get back to Scope and talk about that there.

Okay.

Yeah. So Parametrization and Pi Test is basically pytest name for table tests, which I guess if you don’t know what tablet is, we’re just going to define things indefinitely.

But the idea behind Table Tests, it allows you to write a skeleton of a test and then swap out the expected input and expected output. So basically it allows you to write one test and then test a bunch of different scenarios all at once.

Yeah.

So you might imagine, like a squaring function where you don’t want to write one test for, oh, I want to square a negative number and a positive number and a number that’s greater than six. You would just write one test and specify your expected input and expected output.

Okay. But also it doesn’t have to be the expected output.

It’s just a list of things that go into each row goes into the test and it creates a new test case.

Yeah, it could be a set of parameters, it could be like a scenario. It could be basically whatever you want to be. I just find that the most common use is like inputs and outputs.

Okay. Right.

This episode of Test and Code is sponsored by Regan. Save time with Regan. Crash Reporting detect, diagnose and destroy Python errors that are affecting your customers. With smart Python error monitoring software from Ray Gun, you can be alerted to issues affecting your users the second they happen. Regan takes you to the exact line of code where the error has occurred and tells you how many times it has happened and exactly who has been affected. Have complete visibility of your app or website so you discover and fix errors and performance issues before your customers experience them. Reagan supports all major programming language and platforms and can monitor both back end and front end code. Regan provides full stack traces and all the details you need to fix the issues quickly. Reagan also has fine tuned the filtering and notification control so you can focus on fixing important issues and problems affecting the most users and not be bombarded with redundant notifications. It takes minutes to set up. Try it yourself by going to regun.com that’s raygun.com.

And it turns out in pytest there’s kind of a bunch of ways to parameterize tests. There’s at least two built in ways that I remember. There’s a bunch of others through plugins as well.

The most common built in way is using the pytest Mark Pramatrized decorator.

It has a little bit of a creative spelling.

I remember the first time I worked with pytest, I was kind of frustrated with the spelling because the actual word is parameterize but pytest is tries without extra E at the end. It’s kind of hard over audio, but you’ll have to take my word for it.

Yeah.

You apply this. Sorry.

Well, I remember having to look that up and go, is this misspelled?

And then a quick lesson that there is two valid spellings.

You can put an E between the T and the R or you don’t have to. And one is more common in the US with the E and one is more common in like the UK which is without the E. I think I see.

So it’s my United States bias there.

Yeah, I think so.

There was a proposal a while ago to just like do away with the Wacky spelling and just change it to Python Mark Prince, but I don’t know that it’s received any traction yet, and I haven’t checked in on that proposal in a while, but I’m for it.

Explaining the spelling to people just is like one of my least favorite things to do.

Right. Okay.

But anyway, so you use this decorator on your test and you specify basically the parameters that your test takes that you’re going to substitute, as well as a sequence of tuples or Pi test Param instances that are going to be the rows in your table test, basically the parameters that you’ll substitute there.

And pytest machinery will make sure to call your test for each row in that sequence and basically get a bunch of tests for just one sequence. And it works pretty well. I use this feature a lot. It helps me simplify tests quite a bit, but not only can you parameterize tests, but you can also parameterize fixtures, which is kind of where we’ll tie into the fixture scoping later.

Okay.

With fixture scoping, it’s a little bit different.

You basically specify a Prams keyword argument on the pytest fixture decorator.

pytest just loves its decorators, and in this case, it’s just a simple sequence and you can put whatever you want in that sequence. You could have like a list of tuples or lists of strings.

The most common is usually just like a list of objects. And when pytest constructs that fixture, which it will do a bunch of times because it has that sequence there, it’ll make the parameter in that Prams list available as request pram.

And request itself is another kind of magical built in pytest fixture.

It provides, like pytest metadata. So things about the test itself, what’s been configured, the plans in this case, bunch of other stuff.

And. Yeah, again, like the Pi test, machinery knows how to set this up and configure it properly and do all that stuff.

Yeah. And it’s one of those things that once you get used to it, it’s not terrible, but you do kind of wish when you’re switching from function parameterization to fix your parameterization that you didn’t have to go through the request Ram. It’d be great if it was just variables there, just like functions, but it isn’t.

Yeah. Maybe one day I forget which plugin it is, but I think there’s a plugin that makes fixture parameterization the parameterization. Yeah. See, that where it’s hard to say, too. Yeah, I think there’s a plugin that makes it more like function parameterization, but I haven’t played with it and I don’t remember the name of it either.

Yeah, it sounds weird when we talk about it, but if you do it a couple of times, it’s not too bad.

Yeah. I don’t think I’ve ever used the fixture parameterization myself.

Oh, really?

I usually stick to function parameterization.

Okay. I use both of them extensively.

Nice.

Yeah.

And then there’s a third type of parameterization that’s built in. Well, it’s kind of a combination of the first two, and it’s called indirect parameterization.

And yeah, it kind of combines the two approaches above. So you still use the parameterized decorator and you still use it on individual tests.

But instead you’re not substituting the parameters of your test function. You’re substituting the parameters of the fixture. You also have to pass this indirect keyword argument to the parameterized decorator.

And pytest kind of hooks this up and it kind of works, but it kind of allows you to take a skeleton fixture and inject parameters on a per test basis, sort of a backwards data flow from the usual way that Pi test works from, like a fixture downwards.

Yeah.

It’s pretty weird.

I’ve literally never used this feature, but it was added for a particular use case that I think makes a lot of sense.

It was some quasi global fixture that needed per test data passed to it. And so this is kind of the only way to do that.

I think it’s fine. It works.

The one quirk that I found with it is if you use indirect parameterization, it kind of breaks the scoping feature of pytest, which we’ll get more to scopes later.

But indirect parameterization basically cancels out whatever scope you’ve set on your fixture and always sets up and tears down those fixtures for every function, which kind of makes sense because you have those parameterized inputs per test invocation. And so each time you run that test, you’re going to need to send those parameters along and you’re going to need to reconstruct the fixture.

But I don’t know, it’s one of those, like, got you. That is pretty surprising working with indirect.

Yeah. I’ll have to play with it because I’m curious about the ramifications of the dependent fixtures.

All right. I didn’t even think about that because fixtures can depend on other fixtures, just like tests can depend on fixtures. And so you might have a cascading effect where it rebuilds all the pictures. I haven’t looked into that. I don’t remember what it does there.

Okay.

One would hope it does the right thing.

And the third way or the other third way that I think about is the pipe test generate tests.

Yeah. So you can use hooks and stuff to build tests.

That’s the way I use. If I have a sparse matrix to generate a sparse matrix of parameterization.

Yeah. The other way you can do it is with the skip decorator, right?

Yeah.

If you want a sparse matrix and not have a bunch of skips, generate test is the way to go.

Yeah. And then there’s actually one other way that I just thought of. Now. There’s the new subtest functionality, which I think is only available in a plugin for now until it’s kind of finalized and the interface and API is figured out. But presumably you can use subtest as well as a parameterization.

Yeah. I hate telling people about that because it’s so weird and has so quirky.

Sorry.

Well, it’s just the whole. Okay, we’re getting off on the tangent. But a subtest, if everything passes, you have one test.

If you have five sub tests and they all fail, suddenly you have one test and five failures.

Yeah. You end up with these, like, mystery. Where are these mystery tests coming from? Yeah, it makes sense if you’re coming from like a unit test or like standard Lid unit test background, where subtests are pretty common. But in Pi test, it’s kind of anti paradigm.

The quirkiness and brokenness that I think of with the number of tests not adding up to the total or it’s more than the total or something like that.

Unit test has that broken also, so it’s not like it’s somewhere else.

We’re just matching functionality there.

Yeah.

Okay. So what does all this have to do with. Okay, so sort of we’re trying to lead into scope a little bit.

Yes.

Actually, hindsight, I don’t remember how they are.

Discussion of parameterization.

The relation is kind of where indirect parameterization affects how the scoping of fixtures work, as well as the scope of fixtures. Also affects how the fixture based parameterization works. But yeah, let’s jump into scoping now. So scoping is a way for you to decide the lifetime of a fixture. So by default, when you make a fixture for a test or for another fixture, pytest will set up and tear down that fixture every time a test runs. So if you have a database that you need to spin up or a Docker container or whatever else you’re spinning up in a fixture by default, it’s going to do that for every single test. And so that potentially adds a lot of runtime overhead to your test.

I know. Like we had this YAML fixture, my previous company, which would load this yellow file and then delete it at the end of the test. And I don’t know if you know, but YAML is kind of slow. And we found that when we switched the scope of this particular fixture, it caused our test runners time to go from six minutes to 3 seconds.

We’re basically spending all of our time serializing the animal.

Yeah. And if this stuff doesn’t change, you don’t have to do it for every test.

Yeah. That’s where you can kind of save a bunch of time. So if you divert from the default scoping, which is function scoping, you can control which level that the fixture will be run at. There’s a bunch of different scopes. And I feel like every time that I talk about this, there’s some new scope that we had, but I think the current list is function, which is the default. So it’ll set it up and tear it down for every test run. This also means it sets it up and tears it down for every parameterization as well. So like each parameter set it’ll create and destroy that fixture and then there’s class scope, which if you use class based tests in your test suite, which I don’t tend to use very often, but sometimes they’re useful.

It’ll make the fixture live for the entire lifetime of that test class.

The next bigger one after that, it’s kind of a matrix control.

The next bigger one after that is model scoping, where the fixture will live for the entire lifetime of a file.

Then there’s package scoping, which is newish. I think. I think that was in pytest five. Maybe I’m wrong there, but it’s a pretty new feature which allows the fixture to live for the entire directory of tests.

Okay.

And then you kind of have your biggest scope, which is session scoping, which lives for the entire Pi test run.

Yeah. I don’t think I’ve used package scope much.

I don’t think I have either.

And that was something before pytest added it. The only other tool that I know had package scoping was nose, had a package scope idea of it.

Interesting. Yeah. I think I’ve used it in Nose before, back when I used to use Nose.

Yeah.

Haven’t done that in a while.

So the lifetime. It also means how many tests run in it, like a module scope fixture. The setup will happen, then all the tests within that module will run, and then the teardown happens.

Sort of. It’s not guaranteed to be that way. If some of the tests don’t use a particular fixture, pytest is allowed to tear it down before the rest of the tests if they’re not using it. Right, right.

Yeah. I think the current behavior is pytest is lazy and will tear it down at the very end. But I don’t think there’s any guarantees about when it gets set up or torn down.

In fact, this actually produces kind of a difficult problem, because if you imagine tests and fixtures and dependent fixtures as kind of a graph in order to create an optimal ordering of running tests and minimizing the number of times fixtures get set up and torn down, it’s actually like a very hard constraint solving problem. And Python takes a greedy approach now, and it’s pretty good. It mostly satisfies the minimal number of setups and tear downs, but you could imagine a world where we added constraint solver to Pi test and improves run times of some test suites by a lot.

Yeah, but I don’t know that anyone is working on that yet. There is an open issue, and some people have started poking at it, but I imagine at some point Pythons will get much better about properly ordering tests and making fixtures set up and tear down a little bit more deterministic.

Okay, well, it’s already better than the brute force way to do it for sure.

Yeah. It’s way better than just like always doing everything all the time.

Yeah. Or even just always.

As we describe it, session scopes fixtures start at the beginning of the session and end of the session, it’s already better than modules around all the modules. That’s how we think about it. That’s not really how it happens, right?

Yeah. Like Pythons will wait for the first test that needs it or something.

Yeah. So that if nobody’s using it or if your test session fails and bails before the test that needs a fixture happens, that won’t ever happen. So, yeah, you and I know some of the inside things that could be better, but I don’t want to try it. That’s because we know some of the nitty gritty to the outside world. It’s already way better than anything else.

It’s pretty darn good, despite the quirks that we have. And then there’s auto use as well, which also kind of change when those get set up and torn down. Auto use, I think, is the easiest way to think about scopes because it will always be available for the entire scope.

Yeah. And I don’t know about you, but when I learned about auto use, I overused it and then found all the problems with it.

Yeah, I use it occasionally. I think sometimes it’s good.

One particular use case that I have is I’ll globally mock some environment variables or set up some thing that prevents me from talking to the public Internet during my time suite or other things like that, where I think auto uses a good fit where sets up a consistent environment for the test.

Okay, so now can we get to dynamic scope?

Are we ready yet?

Oh, yeah. Well, actually, I want to talk about one thing before we move on from there, in that expanding scope is not always a safe operation.

As you switch from function scope to module or session or larger scope, you have to start worrying about test pollution a lot more because Pythest will reuse the same instance of that fixture over and over. And so if that instance is mutable, you’ll potentially end up in a case where a test changes some global state and that causes some other tests that have different behavior.

One example that we saw taking that YAML example from before someone was reading that YAML file and then modifying a dictionary, which they thought like, oh, well, this will never actually matter, but I want to test a specific scenario where this config has a different value.

But since that dictionary was pseudo global, it affected the runtime of all the other test and code. We had to do some tricky stuff with faking, immutable dictionaries and pretending that it’s not exactly global mutable. And you can get around this with copying stuff, or if it’s like a database, you can use snapshots or transactions or other stuff like that.

So it’s something to look out for when you expand the scope. It’s kind of a trade off between complexity and speed there.

Yeah. And if anybody is listening to this and thinking we’ll just always use function scope. Well, that’s wonderful for things that are super fast, but things that are slow are annoying to have run all the time.

A common thing I do is to do two layers of fixtures have a like, for instance, a database, something that sets up a temporary database for the session and then initializes it and gets the connection already. And then that can be session scope, and then a function scope fixture called the empty database or something that it just deletes everything out of the database so that any test that needs an empty one, it’s already connected, it just cleaned it out.

And that way you can keep it both fast and correct.

Yeah, cool. Yeah.

So we can finally get to dynamic scoping, which is the new feature in Python Five Two.

We didn’t actually release that many features in this release, but this was one of them.

And the new dynamic scoping allows for fixtures to have their scope controlled by a scope call back. So before you could only control the scope of fixture based on a static string, essentially you could say like, okay, well this is session scoped and it would always be session scoped, but now you can specify the scope as a callback. So a function that takes in, I think it’s the pytech configuration and the return value of that will allow you to change the scope of your function dynamically.

I thought about a kind of cool use case for this. I want to try out at work where right now we spin up kind of a data set and then reuse that inside the test.

But if you’re working on tests repeatedly, you don’t want to share that data set. You don’t want to run that data set every time you run your test suite. It’d kind of be nice if you could set it up once and then just say, hey, Pi test, please reuse this. I already set it up for you. Please don’t set it up again.

And so you could set the scope call back to something which checks for that global state already being set up and kind of avoids that fixture set up.

Another example that’s given in the documentation is you can set up whether you’re reusing some data in a CI environment instead of setting up data and tearing it down repeatedly.

I can think of another like a bunch of really cool uses for this, but I’m interested to hear what you think about it.

I think it’s really cool. I think I read some example where it said for development it takes less time because you’re going to have a broader scope, like a session scope or something, but continuous integration. It’s okay if it runs a little slower, but you want more test isolation so you can do function scope. That’s a cool idea. I was actually thinking the reverse of if you’ve got long running tests to take advantage of the larger scope during continuous integration, during a huge test run, and when you’re trying to debug it, you can isolate things more. If there’s a test failure, you can rerun it with a tighter scope.

That makes sense. Yeah. So you could basically get all the benefits of performance from session scoping, unless you want to narrow down and see if there’s pollution going on.

Yeah. That’d be kind of a cool idea to have as part of a test phase in a test pipeline to run it as wide as possible. And if everything’s great, you’re golden. But if there’s any failures at all, you can dial some of the scopes down. The cool thing about the signature. Normally you say pipes fixture, and then you say scope equals session or scope equals function. Instead you give it the name of a function like determine scope or something.

Okay.

That function can be used in other fixtures also because it gets passed in not only the config object but the fixture name.

So you can have some logic in there that says if the name is in this list of fixtures and there’s a flag passed in called like debug or something, then I can switch it to function scope. You’ve got some control there, which is pretty cool. I’m totally going to play with this all over the place. It’ll be just like auto use, though. I’ll use it too much and then I’ll have to dial it back.

Yeah, the new shiny toy gets used too often.

I like the idea. It’s kind of fun. One of the things I wanted to share with people is it takes a while to get your head around fixtures and parameterization and scope and all that stuff.

The docs are actually really good about fixtures and parameterization, so definitely recommend reading those as well as the Babble that we have today.

And you’re doing a whole bunch. I’m going to go off on a tangent and you’re not just working on Pytechest. Didn’t you release the Python 38 beta just the other day?

Yeah. So I do some Python package among all the other Python packages that I maintain, I also do some packaging for Ubuntu.

I basically provide forward ports and back ports of Python versions.

The PPA is called Dead Snakes if you’ve heard of it before, but basically I repackage Python for the currently released releases that we’ve been doing. So right now it’s Xenial and Bionic and provide packaging scripts so that anyone could take those builds and rebuild them. One basically any flavor of Debian or a bunch of if they wanted to.

Okay.

Yeah. Just recently the 38 release candidate dropped and I quickly turned around and got the packages done for that. So if you want to try out the release candidate of Python 38 before it drops for real, you can add the Dead Snakes PPA and try it out there.

Is this all automated or did you have to notice that there’s a new release and then go and do it.

So I probably should automate this a little bit better.

But right now there’s still some human in this process. So my process right now is I manually noticed that a tag gets released either through Python.org or through Twitter.

But once I notice that the release is there, I have a script that mostly automates the version bumping. So essentially what I have to do is pull in the new release Tarball, refresh all of the patches that Debian applies upstream.

These are things like making sure that Ubuntu is detected properly and changing how some of the modules are included or not included in various packages and that sort of thing. I need to refresh those patches, then I run some tests on it to make sure that the build is still well done.

I often have to refresh symbols because Debian packages advertise what symbols you can link against in their packages, and those change with almost every release.

And then I basically packages up, package it up, sign the binaries and upload them to launchpad. And that’s kind of my process there. It used to be entirely manual.

When I first started working on Dead Snakes, it was one of those things where I used to think a lot and then the upstream maintainer didn’t really have time to maintain the package and the packages and wasn’t really working on Python as much anymore. It was one of those things where it’s like, I kind of know how to do this.

I volunteers Tribute and took over the packaging there.

That’s very cool.

I know a lot of people do depend on it.

Yeah, we use it at work as well.

It pays off.

Yeah.

When I first did it, it was entirely manual.

I basically did all of the steps in my scripts repo. I have a run books repo where I basically follow the step by step instructions so anyone could do this if they wanted to.

But I used to do all that by hand and make a lot of mistakes, and I slowly automated each portion of the packaging over time.

There’s still some manual bits, like sometimes patches won’t apply cleanly or the documentation bill breaks all the time, or what’s the other one? There’s a couple of things that break pretty often that I still have to manually interview, but it’s mostly under control.

Well, that’s very cool. So good job.

Thanks.

And actually, I think this is good.

I think it was a good discussion. I will drop a couple of links into the show notes for people to be able to look up things. And if you really want to get your head around fixtures and scoping and everything, I really recommend reading the Rocketbook on Pi test.

I actually bought a copy of that myself recently.

We wanted to start kind of a library at work and people were like, well, pytest is a thing that we use a lot, so it’d be nice if we had the pytest book and I was like, I’ll hook you up, I got this cool.

Thanks. Thanks again for coming on the show and this has been fun. I love having another Pipeline core resource that I can call up when I need to.

Happy to be on the show.

Thank you to Anthony for that great discussion and for all you do for pytest and for Python. Thank you to Patreon supporters for continuing to support the show special thank you to Susan Wright, Daniel Zimmerman and David Burns. Join them by going to testandcode.com support. Thank you to Ray Gunn for sponsoring this episode. Take charge of your app and web monitoring with Ray gun. Find out more at raegan.com show notes are at testandcode.com 90 along with lots of useful links including the link link to very good. That’s all for now. Now go out and test something.