pytest pytest is an extremely popular test framework used by many projects and companies. In this episode, I interview Raphael Pierzina (@hackebrot), a core contributor to both pytest and cookiecutter. We discuss how Raphael got involved with both proj…
Transcript for episode 24 of the Test & Code Podcast
This transcript starts as an auto generated transcript.
PRs welcome if you want to help fix any errors.
pytest is a test framework used by many projects and companies. It’s also an open source project. In today’s episode, you’ll meet Raphael Pierzina, one of the core contributors and also a contributor to the Cookie Cutter project. We discuss how he got involved with the project, his involvement in Cookie Cutter in pytest and the Adopt PY Test month, and also this past summer’s Code Sprint to get ready for Pi Test Three. And of course, we talk about some of the cool new features in Pytest Three. This is the Testing Code podcast, Episode 24. I’m your host, Brian Hawkin. Follow the show on Twitter at Testpodcast and listen to past episodes at Python testing. Net. Podcast.
Welcome to Test and Code, a podcast about software development and software testing.
Hey, before we get going, I need to tell you about an exciting project I just started with fellow podcaster Michael Kennedy. Michael is the host of the popular talk Python to Me podcast.
We’ve teamed up to bring you Python Bytes. It’s at Pythonbytes. Fm. That’s Pythonbytes.
Fm. The goal of the show is to do a weekly or bi weekly short, 15 minutes discussion of the news from the Python community, Python headlines delivered directly to your earbuds. Please check it out.
Hello. Hey.
How’s it going?
I’m doing really good. Hey, thanks for agreeing to come on the show. Yeah.
Thank you so much for having me.
Well, I probably know more about who you are, even though I don’t know very much than everybody listening. So how would you introduce yourself?
My name is Rafael Padzina, and I’ve been contributing to pytest for I think two years by now, maybe one and a half in a bit. But I started contributing to open source projects earlier than that. But I think my very first bigger contribution was to Cookie Cutter. That’s been more than two years ago.
For those of you who don’t know it, Cookie Cutter is a command line utility that helps you generate new projects based on templates.
That was started by the Django people, right?
Yeah, right. So Audrey started the project and Audrey is the wife of Daniel Or Pie Danny, and they’ve been writing the books, two scoops on Django books, and she started the project. So they’ve been running it together with Paul Moore and Michael Joseph. And I started contributing and eventually became a contributor as well.
Okay. So what drew you to Cookie Cutter?
It’s actually quite funny since it’s directly related to BYTest I was browsing.
It may have been like a hacker news post or something. The best open source projects and Python on GitHub or something. Catchy. So I was just scanning through the list and had a look at the repos, and I was really impressed to see that Cookie Cutter had a test coverage of 99%, I think.
Oh, wow.
Yeah, that got me really interested because at that time I was reading into test driven development and all the fancy things, but the projects that I was working on, they were lacking test coverage and best practices.
So that got me interested.
Okay. And then do you use Python and pytest at work?
Yeah. So I used to work at a media production company back in Germany for four years. And that was on Windows Acute application.
And that was all in Python.
Okay.
I work for Fanniel here in Scotland and also doing Python, but obviously not Acute application, but on the API team platform. Okay.
How did you get into contributing to Pipe Test?
The things that I was doing on cookie cutter was actually when I saw that they had so much test coverage and I was interested in learning Pipe Test. And that was for me the perfect playground, so to speak, because I just picked up one test after another. The architecture of cookie cutter is really simple, so it was easy to get into. And I gradually converted one test after another from Unit Test to pytest, bought myself the basic concepts from fixtures, over hooks and all of those kind of things. And then eventually all of the code base was using just by test.
Okay. And that’s what they use now still. Yeah.
Everything is Pipe Test now.
That’s great.
Yeah. And then I think when I started engaging more and more with the Python community was in April 2015. So not too long ago, actually, that was when the core contributors of Python came together at force them. And then they talked about how they could get more people interested in contributing and they organized. That was Brianna. She organized a lot of fighters month.
All right. Yeah.
Do you remember?
I remember that. So there was like a month long effort to try to help other projects convert to pytest.
Yeah, right. So I think the baseline was everyone who wants to submit Http requests know that there is a library called Requests. And for pytest, everyone, the idea was why do so few people actually know Pi test, although it’s been around for so long, and why only few people contribute to it.
And I think from that they just had a brainstorming. And so the month was four months in April where people who are experienced writing unit tests and Pi tests came together with projects or maintainers who are not running the tests under Python.
That’s a pretty good program. Do you know if they’re. Yeah, I think it was successful.
Especially very proud of that because especially our team.
That was amazing.
It was Bruno who goes by Nikolaos, I think on GitHub and Twitter contributor from Brazil.
So he does cute stuff at work. I used to do a lot of cute stuff. And the project that we were helping migrating to Python was cute browser who like Florian goes on the other compiler. I think so. He’s been the sole maintainer for a long while.
So the three of us got together, made a really good plan. Like, we had a separate repository, made an overview and issues like the different concepts and features of pytest, and then ask Florian if he knows a part of his code base that could benefit from such features. And then we were doing, like, the work on the separate repository, and whenever Florian feels comfortable of actually accepting the changes, he would just merge it upstream.
Okay.
Worked really well.
Most of the communication for that project is through the version control.
You mean for that back then or. No, for that particular project in there that was solely on email and GitHub. Okay, maybe a bit RSC, but we didn’t have voice communication or anything.
That’s pretty cool. I guess I’m doing my part to try to increase the awareness of my test.
Yeah, that’s great.
Hopefully we’re still migrating to it at work. Unfortunately, I’m still using a combination of two homegrown frameworks.
Actually, I think a lot of companies still grow their own instead of using off the shelf frameworks. I’m not sure why.
Yeah, I think so, too.
But actually, I know why I didn’t choose Pipe test right off the bat is because I didn’t know how powerful it was. The features, the fixture model, especially session, some auto use and session mode stuff would have helped right off the bat if I had known I could use them.
Did you go to the meet up in more recently where they did the code Sprint? Yeah, I forget what that was called. They had a name for it.
Yeah, we called it, I think just developer Sprint.
Okay.
I think it was official.
That was this summer.
Yeah, that was last June in Fry Book in Germany.
Just recently, actually, I met with the Corps contributors at European, I think, last year for the very first time. And that’s where I think I made, like, the first step from being just the user to being more interested in authoring plugins and so on. And it was like quite fortunate there were so many core contributors at Europe. So I just grabbed them and asked them random things and learned and learned. And then from that the cookie cutter Python plugin template was born. Do you happen to know that?
Yeah, I haven’t tried it yet, but I’m definitely going. Actually. I’m planning on trying it very soon because I’ve got a plug in I want to write.
Yeah, that was like, that was a huge step for me to just looking at templates and trying to make sense of what’s actually happening behind the scenes. And then with all the insights from the core contributors, it was so easy to put up this template that was just in two days at your partner Sprints. And if you know these kind of things, like the entry points, for example, that’s something you really struggle to get into. If you haven’t thought of that before. I think so.
If I’ve got like, generally I already know the code that I’ve got a plugin that I wrote in just contest or just in my own file, and I want to make it a legitimate plug in that I can upload to pipe. Is that something I could use cookie cutter for then?
Yeah. So the template should be the best thing, I think, to get you going, because there is not much to publishing a Python plug in. The only thing really is you need to set up a setup to entry point and you need to specify a name for it. And that’s just one one. And that’s something that’s just happened from history, but there is no other explanation to that. And if you don’t know that, you’re pretty much screwed.
The template just takes care of these things and you can pretty much copy paste your stuff from the contest to the generated plugin module and that’s it.
Okay. I’m definitely going to try it out probably in the next couple of weeks.
Yeah. While you edit and you maybe want to write a test for your pipes plugin, the template itself gives you a baseline for that as well.
Really?
Yeah. There is a Pyte, something that’s maybe interesting for people who are interested in contributing to PYT score itself.
So there is a large test suite for Python itself and it’s written in Python, so we have an internal plugin. So in pytestcore internal it’s called pytester and it comes with a fixture called Test Deer.
And what you do, you kind of write your Python code and pass it a strings to this test dear function and it generates temporary script files for you on disk or a Conf test file or Python any file. And then it allows you to run against those temporary files and you have tiny helper functions to assert the output, for example.
Yeah, actually it’s actually pretty cool. And it ends up being like this. When you describe it, it sounds like it’s going to be a lot of code, but it’s just these little tiny functions that they’re in line tests and an inline line. Whatever you’re testing can go in and you can check it and it’s pretty clever. I like that model. I was exposed to it first when I pointed out a problem. This was two or three years ago maybe, and pointed it out and somebody told me to write a test to show that it’s failing first and then they could fix it. So I didn’t actually fix the problem, but I contributed the test to demonstrate the problem.
Cool.
That was good.
Yeah. And you can write the test code, but then again, you need to enable this plugin, and that’s something that the template takes care of as well.
Was it a Conf test or PY test? Any? I think you just need to enable the plug in Pythons.
Okay.
I don’t know template.
Well, I’ll give it a shot and try it out. One of the reasons why I wanted to get you on now or earlier than now, but. Well, we’re here is the release of Pythest Three. I think we’re up to 303 now, but there’s a whole bunch of cool stuff that came out in three days. And maybe I’m just a nerd, but I’ve been still just kind of excited about it because it’s so much easier to write. I wanted to run down through some of the features, but I was wondering if you have any favorites right off the bat that went into three.
I have a few favors. Yeah. But we can certainly just follow along your list if you want to.
Well, my list is actually. Did you write it? No, it was written by Bruno. Bruno. Yeah. Okay.
Yeah. Well, we can start with one of my favorites and the pytest Make parameterize ID hook. It’s like really simple, but it’s a new hook. So you would put it into a confedest Pi or a plugin and it receives a value and config, I think. And then you can.
How do I describe that?
You could usually specify IDs for a fixture on the parameterize marker or the fixture decorator.
Yes.
And this book is like it’s like a central place where you can put ID generation for your custom classes. Also, like, usually you would have a Mark parameterize and you have a bunch of tests and they maybe use instances of your custom classes. So your business logic, so to speak, and you want to have a standardized way of how you would translate your instance to an ID. Maybe. So if you have maybe a class of woman or man and they might have a name attribute and you want to use that as an idea and the hook is just like you can put it in there and it will be available all across the code base. So you don’t need to touch any of the other tests. We’ll use them just right off the bat.
Okay. So that’s pretty cool, but it’s actually one of the things I marked as a question Mark because that’s definitely a feature that would benefit from an example.
Do you know if there’s an example right up anywhere for that?
I haven’t had an example in my European slides.
Okay.
So we could maybe just link that.
Yeah, definitely.
It’s pretty much the same example that I just used. And the hook, again, it’s really simple. The only thing it really allows you to do is not repeat the same ID function or anything over and over again across all of your tests.
You just have this one hook and your conference and I take care of it. So that’s really just convenient. It doesn’t add any extra value.
Is this in here because you had a particular need for this or did you just think it was cool?
The hook itself?
Yeah.
It’s in the ridges.
Yeah. I mean, did you work on this or.
No, I haven’t.
The one feature that I added was so we used to have the fixtures which lists all of the fixtures which are used in your test suite.
Yeah.
So that’s pretty handy. But if you have a large code base and you maybe have like subdirectories or something and you use the same name for fixture over and over again.
Yeah.
Because you have even just like a print fixture or client maybe or something.
So you might have a general purpose client and then you have maybe some very specific clients and then you’re working on a test and you see from the positional arguments of your test function that it’s using client fixture. But the question really is how do you know where the code is for that fixture? And fixtures is like one step into the right direction because it lists the fixtures, but it doesn’t tell you where they are used. So if you’re lucky, you only have one single fixture which is called client. And then if you add a Bowser V to it, it also tells you the exact file where the fixture is defined at what line. But again, if you have the same fixed or multiple fixtures with the same name, that’s where it ends, then. So the fixture that I was added was fixtures pertestest.
Okay.
That’s also working with selection. So you maybe just want to know where all the fixtures are defined that a test with a certain marker is using. Okay.
So I can pass in a selector with that. Okay. I think that’s one of the themes for there were more features in there, but one of the themes for the fight test three, I think was making fixtures just way more usable. And that’s one of them addition, a couple of things that I around the same line is the.
Oh, there was a new flag. It’s a set up show.
And if you run that, it’ll just run all of your tests, but then show the particular fixtures as they’re running as well as your tests.
Yeah, there are three of those. I think it was set up show, set up plan and set up set up only. Yeah, right.
And then I believe those show where the fixtures are defined as well. If I’m remembering it right, maybe not.
Could be. Yeah. To be honest, I haven’t used them too much up to now, but I find them like the new command line facts. They are pretty useful.
Let’s say you’re working on your setup code so you don’t touch tests at all and you don’t really want to run them because it takes time. And then if you use I think it’s set up only. Yeah. Where the setup and the encoders actually executed so you could spot if there is some kind of issue.
I guess I didn’t even notice that. That’s so cool. So it runs all your fixtures but doesn’t run the test.
Yeah, that’s set up only. I think that’s executing set up and code set up show, which just like print out additional information while it’s running the code as usual. And setup plan is like the whole opposite. It just locks, but it doesn’t run anything.
Okay, well, both those are very useful. All those are going to be useful when you’re debugging a plug in or a fixture. There was a couple more features around fixtures. One of them was the ability to add a different name other than the name of the function.
And I like that because I always kind of wanted to name my fixtures with, as in the example of the blog post, like fixed your underscore than some other name and then not have to and then just reference the name in a test. And now you can with the name flag in the fixture.
Yeah, cool.
Yeah. And then the last one, of course, the Biggie for me is the yield, which is just brilliant.
And actually I’ve been holding off talking about that because it’s really what I’m very excited about. But have you used it or. I know you have.
Yeah.
It’s essentially just the yield underscore fixture decorator, but it’s so much easier now since you can use the usual decorator.
So maybe to explain what it does, usually if you Mark a function with a pipe test fixture decorator, you would have to return something and that would go into the test and actually run it. But there is also this yield underscore fixture which you use a yield. So you have a nice and clean way of writing your set up codes, then yield some value into the test item that will run the test. And then all the code afterwards are like below that lines. Those lines will be the title code. So that’s like super easy if you compare it to the old recurs at Finalizer. I think it was.
Yeah. Actually, I like the other method seemed fine for me, but this is just so much cleaner.
You don’t have a fixture, doesn’t have to have a request object now as a parameter, and all the code at the top is the set up, and then you have a yield and then all the code afterwards is a tear down unless there’s no yield statement. And then it just works like a normal fixture. Yeah.
I think that makes it even easier to embrace, like the whole fixture concept.
Yeah, pretty cool. I want to remind everyone that all we’re talking about so far is things that are decorated with Pinterest fixture. And so I’ve always struggled with what to call these things. They’re fixtures or pipe test fixtures because there’s also support for X unit style fixtures with the set up and tear down prefixes.
I don’t think those are included in the setup show and things like that.
The X unit. Yeah, probably not. I don’t know.
To be honest, it’s one of those things that I think I want to dig into the test code and see if I can add those, because I think that the style of there are two main reasons, I think why people jump go to Pi test. And one of them is the assert rewriting, which makes asserts really easy to write. They’re just like normal asserts, and then the other one is the really awesome fixture model. Now some people don’t care about the fixture model and think it’s complicated at first and they’re wrong. It’s not that complicated. But if they just want to use a better test system with the nicest search, they can go ahead and still use X unit style fixtures.
However, with Pythons three, it’s really hard to probably convince yourself to do that because there’s all these great debugging tools that are available for all the normal pipe test fixtures.
Yeah, that was for me also the reason why I started using Pipe Test without making this intimidating task of converting everything in one go, you could just change by test to be a test runner and see if all of your unit tests pass. And if they do, you already benefit from pretty much everything that you just said, a certain rewriting, and so on. And then you could decide if you maybe want to write your tests and high test style and make use of a fixture system that makes it kind of easy to migrate gradually rather than rushing into it.
It’s one of the great things as well is that there are some cases of pipe test fixtures that you can use with unit test classes, which I think. Is that still the case?
I don’t think so.
One of the things that I’m really excited about, which is because I tell people how to write pytest stuff so often.
I convinced Bruno to put in a fix so that you didn’t have to include any keywords in the X unit style fixtures.
So normally you would have like for setup module, you had to include a parameter. You used to have to include a parameter to specify what module. Okay, it was either like set up module or set up function.
One of them allowed not having a parameter and one of them didn’t. And I like that if you’re using those, you don’t have to have any parameters anymore if you don’t need it, which is great. But I do want to urge anybody that’s going down that road to just bite the bullet and learn by test fixtures because they’re so cool.
Yeah, I think that was for me. Also, like my goal for the you mentioned the developer Sprint. So the only thing I really had on my agenda was to revisit the documentation because when I started using Python more and more I always hit a wall where I had no idea how things work together. I think it’s maybe also important to mention that such is pretty much sure, and it’s really been around for a while. So it’s one of those projects where you support so many different interpreters. Like, you have CPU. I think J Python was supported as well. At some point there’s PyPy, and then from Python, two six to three six. So we tested already. But like three five is like two three or five. Everything is supported.
And with the maximum compatibility that comes this huge burden of carrying around a lot of old codes. Maybe that’s working and you don’t want to touch it. You have like weird constructs for compatibility reasons because some building things have been introduced in more recent versions of Python.
Yeah. Every so for projects like that with some legacy code. But I think that the openness of the Python community to basically you can a lot of these discussions are going on on the Pite test mailing list, and they complete the Pinterest development mailing list, and I’m on there. I’m just more or less a Lurker. I just kind of like listening to what everything people are working on, and it’s a pretty open environment to have anybody jump in if they wanted to. Yeah.
And especially that’s why I mentioned the documentation. That’s what I noticed. Like when you go to the doctors and they also have a new address, it’s now docs. pytest.org. And there was no dedicated section to aimed at contributors or potential contributors.
Like there was one, but it’s sometimes really hidden and you might be scared, maybe even because it’s chaotic, I would say. So that was pretty much what Brianna Andreas, who maintains the Path Jango plugin, and myself, we’ve been working on the documentation for most of those print, but obviously it’s nowhere near finished. So we at least decided on a new structure and direction that we want to head to with the documentation to make it easier for different kinds of I think we called it not personas. What did we say? Like, you could have a user which wants to write tests with pytest, then you would maybe have the advanced user who wants to get into hooks, writing on plugins, et cetera.
Yeah.
And then you have the people who want to contribute to the car itself. And all of these looked at the document, the existing documentation. I had this huge wall where we were putting up posted notes. So where do we put this kind of article that describes hooks, for example? And it certainly doesn’t belong into the beginner section because you can use by test just fine without using a single hook, and you still get most of the benefit from it.
But if you want to do complex things or your organization may have like super specific code and you could just put it into a Pytech plugin, so you would be interested in two hooks or the set up by stuff, for example.
I’m going on a different tangent. I’m looking up on some of the documentation, and I do see the new contributing contribution getting started guide, which is cool.
I’m going to shift gears a little bit because there’s one more thing I wanted to talk about, and that was I noticed that I was looking through the thanks section of the 30 release and I thought that your name showed up with a couple of the xVALE and X pass bits and the reporting.
Yeah.
I was curious if you use the juice XML in your work.
I think we use it. So when we generate we use Jenkins. But for all of my open source projects, I use Travis and Payer and we use the coverage. The combined report, I think to get coverage, especially with projects which aim at compatibility across Python versions. You have a lot of compatibility code and obviously you won’t get cold coverage on that if you just use a single coverage report.
So does the coverage report come out in the JUnit XML as well?
Yeah, I think so.
Okay.
I was guessing that most people that are using that output are using it so that they can hook into reporting system like continuous integration in Jenkins or Travis or something.
So the big feature that I’d love to find somebody to fill is somehow a reporting system like Jenkins that would correctly display xVALE and XPASS as different outputs.
Yeah.
If they do now, I don’t know how to do it.
That’s a good question. I don’t know. The change that I did for the release was I think it was added in February this year already that you could set an X fail to strict.
So if you Mark a test with pytest Mark X fail and you run the test and you have a problem in the test and it fails, your test suite wouldn’t be unsuccessful, you would still have a pass.
But you can enable the strict mode, which means if test happens to pass, although it’s decorated with next fail, it would be a failure.
Some people writer a negative form of a test if you will.
So they write the test and it’s because in a test first model I want to be able to write tests and put them in the system before the implementation is finished or before. Like if there’s a bug, I’ll write a test for the bug and market as X fail and I want very high visibility when that test now passes.
And I think that’s one of the reasons why the strict mode would be in, because I don’t want that just to slide through as a pass.
Yeah, right.
However, what I really want is not neither of those. I want to be able to see the X fail and X past levels in my Jenkins output or something else. But I’m using Jenkins at one project and the only thing I’m using it for is to collect these J unit. Xml files and display them in graphs so that I can see past fail over different versions of the software to be honest.
I don’t know if there is a CI system that is so specialized in Python that could take us into account.
There is also another plug in called Pythml and they fund and other contributors like that. So it’s pretty much like the XML, but it’s HTML, so you can maybe added support for that. I don’t know.
It’s like a funny thing with Pipelines after I gave the target your price and people ask me questions and sometimes I really just need to be honest and say, I don’t really know. It’s a really big project and there’s so many changes in this release that so hard to keep track of.
So anything you want to plug or anything about Pipe test or anything else. Before we say goodbye.
I want to give a big shout out to all the people who helped us with the fundraiser. That’s something that I really wanted to bring up.
All right.
Yeah.
Because that was for the coding Sprint, right?
Yeah, right. The developer Sprint was only possible because we were running a fundraiser and it was people from the community and a few companies who donated money and that made it possible.
So Brianna comes from Australia, Bruno Anna, they came from Brazil. We had people from Seattle. One guy was from China was his first trip outside of Asia and it was just possible because of either employers sending their employees or obviously to the fundraiser. And it was amazing to have almost 30 people there in Germany and working on the project for a full week.
So yeah, I think we as a project, I’m very thankful for that.
Yeah, that was a pretty neat support. I like that. All right.
Apart from that, I would just encourage everyone to jump onto the mailing list, maybe if there are any things that you would like to see or if you have any specific questions.
And we also have an IRC channel, it’s just pile and a lot of communication for some reason happens on Twitter nowadays, at least I think in the Python community it seems to be a common thing to share blog posts and release announcements.
Yeah, it’s definitely you miss a lot if you ignore Twitter.
Funny.
Thanks a lot for coming on and hopefully I’ll run into you sometime.
Yeah. Thanks so much for having me.
All right. Well, great podcast. Thanks.
I hope you enjoyed this episode. If you did, please tell a friend about the show.
This episode is sponsored by the people like you, contributing as little as a dollar a month through Patreon. To help support the show, go to Python testing. Net support.
Thanks for listening.