How do you test installed packages using coverage.py?
Transcript for episode 148 of the Test & Code Podcast
This transcript starts as an auto generated transcript.
PRs welcome if you want to help fix any errors.
00:00:00 Last week we talked about coverage of single file scripts. I have a couple follow ups and modifications to last week’s suggestions, and after that I’d like to talk about using coverage while testing packages.
00:00:15 Thank you Lenode for sponsoring this episode. Simplify your infrastructure and cut your cloud bills in half with Lenovo’s Linux virtual machines. Develop, deploy, and scale your modern application faster and easier. Whether you’re developing a personal project or managing larger workloads, you deserve simple, affordable and accessible cloud computing solutions. Get started on Lenode today with $100 in free credit for listeners of test and code. You can find all the details at lenode.com Test And Code Lenode has data centers around the world with the same simple and consistent pricing. Regardless of location, choose the data center nearest you. You also receive 24, 7365 human support with no tiers or handoffs. Regardless of your plan size. You can choose shared and dedicated compute instances, or you can use your $100 credit on an S three compatible object, storage, managed Kubernetes, and more. If it runs on Linux, it runs on Lenode. Visit Lenode.com testandclode and click on the Create Free Account button to get started.
00:01:33 Welcome to Test and code coverage. I love coverage and thanks to Ned Batchelder for maintaining it and keeping it. Awesome. Last week I talked about running coverage on a single file Python application with tests contained in the file. I suggested that if you need to import pytest, for instance, to make parameterization, you could put the import in possibly a try accept block or something. Carl, aka CF Bowls on Twitter suggested that the cleanest place would be in an else clause of the if name equals main block. That’s actually brilliant and I love it. I’ve added that to the show notes for last week’s episode, so check out testandcode.com for that. One more thing. I suggested that you need to put the script in its own directory to pass the directory to coverage. This is because I misunderstood the power of the source flag in coverage or the Cove flag in the pytest Cove plugin. Actually, the Cove flag for Python Cove is really the same thing as the source flag and coverage. So I’m going to try to just say the source flag for now and mean both. Okay, the source command line flag has more than one use. The docs say that the value is a comma separated list of directories or package names. If specified, only source inside these directors, directories, or packages will be measured. Okay, it also says or new lines, but if you’re using it on the command line, the dashboard, it’s a comma separated list. The new line thing is if you stick that in the coverage. Rc file. Also, for Python Cove, if you’re providing a list, you have to actually provide multiple Cove flags if you have a list of directors or packages. But I’m really getting off track now anyway. The important part is that it’s directories or packages.
00:03:44 So I figure or packages.
00:03:47 Can I just provide the single file name when measuring the coverage of a single file? I thought I already tried that and it didn’t work. So I tried it again and it didn’t work. It didn’t work if I included the dot PY extension to the file. If I drop the extension and just list the script name without the extension, it works fine. I don’t normally think of a module as a package, so I’m not sure really why this works, but it does. It’s super handy if you’ve got a bunch of scripts in a directory that you want to test. You don’t need to move them into it one at a time into a directory on their own, just so that you can test it with coverage. You just provide the file name without the extension as the source flag to coverage, and it’ll all work.
00:04:35 So in the case of episode 147, I was using pytestcov equals script something. Actually, in 147 you can just type Pi test space Cove equals script space script Pi, and that will run pytest on script Pi, the script dot Pi file and coverage will measure the coverage on that pipe file only. Anyway, I don’t know why this works. I kind of get how it works, but I’m glad that it does. So that’s cool. You don’t have to move it into the directory, you just provide the script name. This almost brings us to the topic of running coverage on packages.
00:05:16 You can see the segue, right?
00:05:19 I just want to spend a little more time talking about the term script because it came up on Twitter when I was talking about this.
00:05:27 To me, there are single file applications and there are single file scripts. It’s nuanced, but let me explain my thinking. Many applications are single files.
00:05:40 I was reminded that AC, the super cool grep alternative that can be found at beyondgrep.com, is a single file application.
00:05:50 I would not consider it a script. Why not? Well, it’s got its own repo on GitHub, for one thing, and the code may be in one file, but the repo has tons of more stuff than just that one file. But most importantly to me, you install it. That’s how you get it. You install it from somewhere. How about single file flashing applications you don’t install that? Would I consider that a script? I still consider an application because you kind of do install it. You deploy it. Well, what are scripts that I think of scripts as things that I write that I don’t install or deploy? I just run pythonspace scriptname. Py. It’s also a history thing for me, I think. I don’t write bash or Perl applications. I know Perl applications exist. Ac is one of them. I’m guessing that bash applications exist. Maybe, but I write small bash and Perl scripts. Yes, I still use bash scripts. It’s for stuff I know how to do in the command line, and I’m too lazy to figure out how to do it in Python, so I just write the commands in a bash file and voila a bash script. My use of Perl is now limited to mostly Perl PE and then a search string because I like Pearl’s regular expressions better than said, but that’s really not way off the point. If I were to take one of my bash scripts and convert it to Python, it would still be a sequential list of things that I want to get done, and probably involve stringing together calls to other programs and fiddling with the file system or whatever. I would still think of that as a script. I’d also probably not create a package or a wheel for it and package it up nicely. If I wanted to reuse it in another project, I’d probably just copy it into another repo tools or build directory or something. So long winded, I know, but I’m still going to use the term script for some Python single file applications, depending on what it’s used for mainly and how I install it. And that isn’t meant to demean or belittle applications that have to be kept in single file. Okay, anyway, enough on that.
00:08:03 What I really wanted to talk about is coverage on packages because of this thing that I learned about coverage, but I’m going to back up a little bit. I learned how to test packages from lots of people writing about how to do this.
00:08:18 One of the ones I can find that I’ve used before is a great article by Hinck called Testing and Packaging and it was written in 2015, but he keeps it up to date. So please, I’ve got it in the show notes. Please bookmark this article. It’s great. In general. Here’s the problem. You’re developing some Python code that’s intended to be Pip installed as a package. You want to test it. You want your test to test an installed packages package and not the one just sitting in your source directory or in the project directory because you want to make sure the packaging and installation things work and you don’t forget anything and whatever. Anyway, Hinix article is really good and go read it for more on using it. Talks about using the source directory, of course, and that helps, but also it ties the whole thing up. It’s a great article.
00:09:14 The important part to my conversation here is that when you’re testing the code that’s running during the test is the installed code. It’s not the one sitting in the directory.
00:09:27 So how do you run coverage on it? I’ve been told this before, but I guess I wasn’t listening. The source flag for coverage is either a path or a package. Yes, that’s it. That’s the trick. You just run your test and code the source flag to tell coverage what package you want to run coverage for. That’s it. That’s all you have to do. So for example, my little toy Cards project.
00:09:53 If you want to test Cards, if I want to test it, I’d run coverage space, runs Source equals cards and pytest spacetest that’s long winded. And then after that I’d run Coverage report. That’s a mouthful, and I don’t like to type that much. So that’s why I use the Pite test Cove plugin. So I can just type PITest Cove equals cards test that will run all the tests in the test directory and run coverage over the Cards package. And since I’m using a source layout, coverage won’t be confused and think that Cards is just a directory because there’s no directory named Cards while I’m running this, it has to be the package. So I use the source layout for that for one reason. Also, I just like it that’s all you have to do they have is in quotes there. That’s all you have to do. What you get from that is you get a report of coverage, and it’s probably got long paths in it of where your package is sitting in either a virtual environment or a talks directory in some site packages directory. So how do I get this path a little bit shorter? So that’s where some of the rest of the tips come from. So I do recommend taking one further step. If you’re using Talks, for instance, you can just follow Hennock’s advice. There’s a code example in his article of using the Path section of the coverage. Rc file and specify the source option, which is a list of equivalent paths. So the thing is basically the idea is with the coverage configuration file coverage. Rc, you can set up a whole bunch of different paths that coverage will consider as equivalent paths. So you set your local source paths path as well as the site packages path, and you don’t have to know exactly where that is. You can use Wildcards so that it’s not crazy. Again, see the testing and packaging article. It’s got a great example, and the testing and Packaging article refers to it in the Talks directory. But if you’re using virtual environments only and not Talks, you can do a similar thing, and it works great.
00:12:17 And if you do this coverage rule, report all of the files covered for your package as if they were using the first path in the list, which is usually shorter and easier to read. So again, this is really longer to explain than it is to do. Provide the package name via the source flag or the Cove flag in Pytekov during the test run, and set up the paths source option within the configuration file to make the output easy to read. But that second step is just to make the output easy to read. All you have to do is provide the package name via Source to get the information you need. Wow. Again, index article is great. Follow that. If you’re building a package using taxes, which you should talks is awesome especially if you’re building packages but otherwise if you’re just trying things out just look at the documentation for coverage. The source flag does way more than you thought it did and I love it.
00:13:26 Thanks Henry for keeping the article on testing and packaging up to date. Thanks Ned for keeping up coverage high and thank you Lenode for sponsoring the show. If it runs on Linux, it runs on Lenode. Try Lenode with a $100 credit by going to Lenode comtestcode that link is also in the show. Notes at testandcode.com 148 thank you also to Patreon supporters joining at test and code co.com support that’s all for now. Let’s go out and test something.