Code Coverage or Test Coverage is a way to measure what lines of code and branches in your code that are utilized during testing. Coverage tools are an important part of software engineering. But there’s also lots of different opinions about using it. Should you try for 100% coverage? What code can and should you exclude? What about targets?
Transcript for episode 118 of the Test & Code Podcast
This transcript starts as an auto generated transcript.
PRs welcome if you want to help fix any errors.
00:00:00 I’ve been asked many times, what do you think about code coverage and test coverage? I think it’s because they talk about testing a lot. The truth is, I have a lot of thoughts about code coverage. I mean a lot. So here’s my chance. This episode is a train of thought Brain does on what I think about code coverage. We will talk about how I use code coverage to help me write source code, what line coverage and branch coverage do for you, behavioral coverage, what that is using Test to ask and answer questions about the system under Test, how to target coverage just to the code that you care about, how to exclude code from your coverage numbers and good reasons and bad reasons to exclude code. And we also talk about the creative principle or the 8020 rule and the law of diminishing returns and how that applies or doesn’t to test coverage. These are all my opinions. Of course, if you disagree, I’m game for learning your perspective. Just contact me. This episode of Testing Code is brought to you by PyCharm. Save time, use PyCharm and by listeners like you who support the show through Patreon.
00:01:21 Welcome to Test and Code because software engineering should include more testing.
00:01:29 So let’s talk a little bit about what code coverage and test coverage really is and what it means.
00:01:36 What does 100% mean? So coverage pie is what we use in Python for code coverage. Other languages have their other tools, of course, but it’s a little deceptive to say 100% coverage because it’s 100% of what is 100% mean. But let’s specifically look into what coverage gives you. So if you run either usually use the PITest Cove plugin which loads coverage and runs coverage while pipe Test is running. So if you use that and direct it to a source, you have to give coverage to source directory of where the source files are. And this says how much of this is covered and what is covered mean. It means during the test run it will look at all of the files that get run all of your source code files. You can run it over test code files too. I like to do that. Run it over both source and test, and then it will show you what in the end, it does not give you just one number.
00:02:48 There is a summary number for how much of your code is covered, but the summary number isn’t as interesting as the rest of it. So it also looks at absolutely every file it hits and shows you what the percentages for those. And then what’s more, you can use there’s generator reports for exactly which lines of code are hit, during which lines of code are hit during a test run, and also which switches so you can coverage up high. We’ll give you both lines of code coverage and also branch coverage. So I think it’s good to turn both on because branch coverage, you want to be able to hit all those. Here’s an example, and sometimes it’s not clear what that means.
00:03:35 Really. If you’ve got 100% line coverage, won’t that mean that all your branches are hit? Well, not necessarily. So let’s say an if statement. If I say without an else. So if I say if X is true, then run some code and then I’ve got drop through code that runs no matter what. So if I run that with true and X is true, it will show that all lines of code are run. But for branch coverage, the branch coverage will detect that. We didn’t test the case that X is possibly false, so we didn’t test the code path that shows not running that line. That extra chunk. So it’s interesting and it is useful. So I always run with branch coverage turned on. But that is not the only type. It’s just the types we have to measure. Right now we’ve got line coverage and branch coverage. So why is this useful?
00:04:37 It would be lovely if it were to say during our test suite we actually have ran every line of code and we’ve hit every switch statement and branch. We’ve taken all paths. It’s not all paths through the system, but it’s at every branch point. We’ve hit each of those different paths at one point during the testing. It’s very useful for a dynamic language like Python that if there is actual Invalid code in parts of your code that aren’t run, you’re not going to know it. If it’s not run.
00:05:16 It’s possible for Invalid code to exist. As long as it doesn’t run, it won’t cause an assert causing problems. But you don’t want code like that. All right, so especially in Python, but in every language, code coverage metrics are useful.
00:05:31 I like to use them, and I have warmed to them. So my first reaction to code coverage was that it’s a silly number. There’s always going to be like I’m used to coming from the sea land.
00:05:47 In switch statements, there’s often the default at the end it says I’ll never get here. So that’s like the corner case to say a certain statement that says assert should never get here. Why do we need we’re always going to have things like that. So it’s possible you’re never going to get the coverage unless you write stupid code that can try to get to there, but you have to like mock something or try to try to do something weird. Okay, so that argument is there, but let’s counter it. If nobody were to tell you you have to hit a certain number if you’re just using this for a useful tool to help you with software development. And that’s how I use it. How is it useful? So I’m going to describe like a day in the life of me. So I’m writing some code. I’m looking at not the whole system, but I’m looking at the code that I’m writing. And I’m writing some tests also. So I’ve got code to implement some functionality, and I’m going to have tests that try to exercise that. I’m intentionally writing code that I want to be run and I want it to be run by the tests that are hitting it. So I can target just this file and say run coverage on just this file and run it from this set of tests that I’m working on. So that’s a useful way to use coverage to say, hey, just right now, run these tests and I want to make sure I hit this file.
00:07:22 And then once it’s done, it’s not just whether or not that file is 100% or not or 80% or 90%. It’s the specific lines and branches in the file that I’m working on. And if I’m writing a test that really should exercise that’s exercising an air condition, let’s say, or exercising a certain corner case, or even just the main case, and that case really isn’t getting hit by the test. There’s something wrong, and coverage analysis is a very useful thing to be able to tell me. Yes, that’s not getting hit yet. There’s something wrong with my test.
00:07:59 There was an old adage a long time ago with an early test driven development is one of the things you do is you write the test first before you write the code to make sure that your test is actually hitting your code. And not just no Ops. Okay, well, that’s a nice idea, but I’m often writing my source code first or writing tests to cover existing source code. And it’s good to see you don’t have to write the test to fail the first time you can watch the test, hit your source code and coverage is extremely useful. Okay, so on that specific file on the function I’m working on, I’m going to want all of it covered because I’m writing tests for it. Also, it is interesting to see if the parts that aren’t covered to think about, do I need a test for those parts of the code that are not covered? It isn’t that I’m trying to hit artificially hit 100%. I don’t do that, but I do every time I don’t have some code that’s getting hit by the test suite, I think, is this a test case that really needs to be that’s important to be supported and it’s important for somebody to have it work, then I should write a test case for it. There is a possibility that it’s just over engineered code. There’s a lot of dead code in software, and I believe it’s a reasonable place to also ask the question, not just should I write another test to cover this chunk of code?
00:09:31 Is this chunk of code actually not even reachable?
00:09:34 If I’ve got other safeguards in place to make sure that this code is never reachable, then one I could possibly do a more focused unit test if I really wanted to get that coverage, but it might indicate that I can just remove that chunk of code and it won’t change anything. That brings up another case that I wanted to talk about is the level of coverage testing. So I think that if you’ve listened to me for a while, you know that you understand this. But for new people, I don’t think in terms of the normal test pyramid, I think in terms of as broad of a brush as I can get. So I don’t usually write tests for a user interface or through a Gui or through a website or something web front end. I will go through an API, but I’ll go as high as an API, as close to a user interface as I can, and then I will try to test the whole system, and I’m trying to pinpoint features. So I think of a unit test as feature test. Now, I know that most of the world thinks of a unit test as testing just a little function. So I just try to avoid using the term unit test because I learned it wrong, apparently. So I focus on feature tests, and I focus on that because I’m looking to satisfy the requirement of whatever the software needs to do for the customer or for the end user. Now that doesn’t mean to say I don’t ever write small unit tests. I do or focused subsystem tests. Of course those are necessary, and those are very useful, especially for algorithmic sections where you’ve got a whole bunch of different corner cases.
00:11:24 Okay, I’ll get to this later. But there’s a whole bunch of reasons why you want to overengineer some sections of the code. Maybe not over engineer it, but really well, engineer it to make sure that it behaves appropriately, even for the future. Most of the time you’re going to want to follow yagney and don’t overengineer stuff just because you might need it in the future. But there are cases. There’s different algorithmic cases where you’re designing an algorithm that’s similar to lots of things off the shelf. And it would be weird for you to not test corner cases. So may as well. You can also for really important mission critical parts of your system, making that part bulletproof to have lots of error checking to make sure that negative numbers aren’t passed in where they’re not supposed to be, checking for ranges and all that sort of stuff. That’s great for certain parts of your code.
00:12:19 That’s a great place to put unit tests or subsystem tests to really focus, to make sure that those corner cases are really being handled. Also, if you’re going to use a chunk of code in more than one project, then the requirements of both projects, and you don’t know how those are going to change. Be thorough. But in general, I do broad brush tests if possible. So if I’m running broad brush tests, and then some Focus tests are in the mix. I’m going to run the whole thing and run coverage numbers on all of them. Now, with continuous integration systems, it makes it a lot easier because I can combine coverage reports. That’s one of the things that is covered by coverage. Pi allows you to combine reports so I can run whole bunch of like, let’s say I’ve got a Focus suite that I’m running all the time. It’s not going to be 100% coverage, possibly, but it’s a smoke test that’s really quick. But maybe it’s like 85% coverage or 90% coverage. But some of the corner cases take some of the different corner cases. Maybe they take longer and I only want to run them during continuous integration. That’s fine. I also might have operating system specific code that I’m running on a Mac, and there’s Windows stuff in there, too. I’m not going to run that all the time while I’m developing my code, but I’ll run it in CI, and then I’ll combine the Mac and the Unix and the Windows coverage numbers together to look at the report as a whole in the CI system.
00:13:54 Thank you, Pie Charm, for sponsoring this episode. I love running my tests right from Pie Charm. There are many ways to run the tests, and I use lots of them, and every one of those also has an option to run the tests with the debugger or with the profiler or with coverage. Run with coverage. And now there are a percentage of lines covered listed in the project tree right next to the file names. Select a file with low coverage, for instance, and visually you see exactly which lines are covered and which ones are not by colors in the gutter alongside the code coverage and Piper, two tools I use all the time work great together. Awesome. Running with the profiler or with code coverage are part of PyCharm Pro and is one of the reasons why I use PyCharm. Not the only reason to grab Pro, but it is super cool. Try it yourself by visiting Testingco PyCharm there. You can try PyCharm Pro free for four months and see for yourself if coverage in your editor saves you time. I think it will.
00:14:55 Let’s assume that I’m going to work on open source project, maybe. And there’s like an open source project that I’m using and I wanted to have like a new feature or there’s a bug that I found. I want to help fix that.
00:15:12 Now let’s say I’ve got some chunk of code and I want to like, I know it’s going to involve changing some code, but before I do that, I’m going to run the test tool and I’m going to run it with coverage turned on. Why am I going to do that? I want to make sure that I understand all the tests that pass or fail before I touch the code. So as I found it, hopefully everything is green. If it’s not green and some of the tests are failing, I need to understand that I configure it wrong. Am I doing something wrong? Because I really want only the failures to as I’m developing, I want the failures to be my fault, not somebody else’s fault. And odds are they’re my fault anyway. But I want it to be coding errors, not just misconfiguration errors. Okay, so let’s say we’ve got it all green, and then I want to modify some code. Hopefully that code is already covered, because if it doesn’t have any tests for coverage for it, that’s an issue. I don’t know if I’m breaking something now. Just having a test that hits it doesn’t mean that your test suite is perfect. Of course, your test suite still has to be a good test suite. It has to actually ask and answer questions about the software.
00:16:31 An example of a bad test suite is something with no asserts in it and no way to fail.
00:16:36 You can exercise especially with unit tests. You can exercise a huge amount of your system with no asserts in it. So please remember that and look for that in code reviews. If somebody writes a test that doesn’t assert anything, these aren’t necessarily terrible tests. How can it fail? There’s no asserts it can fail if your system crashes, or if your code under test throws an exception that will fail the system. That’s a reasonable kind of test. But most of my tests are testing for functionality, so they’re going to look for the correct output, the correct behavior, or something. So let’s say I’m working on an open source project, this open source project that I want to fix something on. Let’s pretend it already has 100% coverage and the tests are pretty good. I’ve looked at them and they look reasonable. So let’s say maybe if I’m doing tests first, I can write the behavior I want to have happen. I can write a test around that, maybe. But realistically, I’m probably going to go with the code and play with it.
00:17:40 And then I want to run the test to make sure that I didn’t break anything. But I also am probably going to write a new test to cover the new behavior that I’m doing or the modified behavior, or to highlight the defect that I found and then hopefully fix the defect. It’s a great way to run tests or write tests is write your defect finding test.
00:18:05 I wish it was less common. I do a lot of that. I talked a little bit about my test. I wanting my test to give me information, so I want to ask questions about the software. Does it behave like this? Does it throw this exception when I do something weird?
00:18:24 If I ask for an index that I know isn’t going to be there, is the correct exception going to be thrown, or is it supposed to return none?
00:18:33 Those sorts of behavior about my system.
00:18:36 Those are questions. And if questions and the tests can answer whether or not the behavior of the system is correct, or at least as you expected it to be when you wrote the tests. That’s one of the great things about having tests that focus on the behavior of the system and not its implementation is that that behavior is defined with the tests. Is all the behavior defined? That’s what we would call like behavior coverage. If all of the behavior of how the system is supposed to behave under different circumstances in different starting States and everything, what ending state? If you do an action, what’s the end state supposed to be? Are all the exceptions checked? Are all the features that you have tested thoroughly? Are you testing all the flags? Are you testing all the options of the flags? Are you trying to test weird things in background, like doing things out of order in an unexpected way? Those sorts of behavior descriptions are things you can test, can help you see about your code. Now, we could think of a hypothetical behavior coverage. Have I covered all of the behavior of the system? We don’t have a tool to measure that because it’s to define what your full behavior of your code is anyway.
00:20:04 Of course, I have no idea how to write like a line coverage or branch coverage tool. So maybe it does exist or could exist to have a behavioral coverage tool. I just don’t know. It seems like it would be impossible. So these are judgment calls. So we’re going to trying to test all of the behavior of the system and roughly approximate it with code inspection of our tests and also just a code coverage and branch coverage help us get there. We got a few small open source projects. I do try to get 100% coverage on there.
00:20:40 There’s a few reasons around that.
00:20:43 It’s not just because I want the little coverage badge.
00:20:48 It helps in several ways. If somebody else is going to contribute to the project, it helps me determine whether or not they’ve written tests for the code that they added. Now that’s an easy way to say, hey, if the coverage is dropped or if their code really isn’t covered at all.
00:21:11 That’s easy to tell from coverage numbers. It’s really easy if we’ve already gotten 100% coverage on everything to be able to see a difference. So like, let’s say I’ve got some project that’s 89% coverage seems reasonably good, right? Except for if somebody makes a change and we’ve gotten to 87.2% coverage, 88.3. Is that good enough? Why was 89 good enough but not 87?
00:21:42 Kind of have to defend that. And you can put a stick in the ground and say, well, no, it’s 89, I’ve got 89 coverage. I don’t want to go below that. You can do that. But it’s a lot weirder than saying I’ve already defined what code I want covered. Now when I say what code I want covered, coverage Pi has you’ve got configuration that you can set. So you can say in your configuration you can say, these are the lines, these are the source files that need to be covered. Defining it there. You can also exclude within there exclude certain parts of files if you want to. I prefer to specify which files I’m going to cover, exclude files if I need to. So there’s a possibility that you’ve got some vendors code, for instance, maybe that you’ve pulled in from a different project. It doesn’t make sense to pull it apart, but you are not responsible for it. So it doesn’t make sense for you to check those coverage numbers you’re already going to check to make sure that the behavior of using this code is going to be working. But you’re not responsible for that code. Now, we should probably should talk about vendoring code on a different episode, but that’s a pretty big topic. But it’s essentially, instead of importing a third party package, you drop the source code for the package into your code.
00:23:07 Not recommended most of the time, but in a lot of projects within companies, that happens frequently, you’ve got multiple projects that use a shared chunk of code that just gets dropped in good or bad. It is exists. And it doesn’t really help you to say, Well, I don’t have tests around this, so I just wouldn’t do that. I would probably exclude that coded one line. You can also within coverage, you can do pregnancy, you can say pregnant, no cover within hash comment block on the line, or at least on the branch area. And I’m going to give you an example of something where I just did. So I’m working on a project that had a database setting to where it picked up where the database directory was. Now most of the time it went through like when I’m just using it on the command line. I use it all the time.
00:24:08 It pulls it off of my home directory. So it uses path. Lib path home to find out where the home directory is during testing. I don’t want it to pull the database out of the home directory. I wanted to point out a temporary file or temporary directory. So I’ve got some code in place. Yes, I did modify the source code to make it more testable. I allowed it to have the feature of being able to look up the path as an environmental variable. I considered having just a test function or test API to be able to just change that directory during testing, but I thought about if I need it for testing. It’s possible that a user might need it that behavior outside. And what would be a useful thing?
00:24:58 A configuration file might work.
00:25:01 I could have like a user could have a configuration file or a dot file in their home directory. Another thing that isn’t that uncommon is to have an environmental variable that is set.
00:25:15 I chose that path that way to do things just to set an environmental variable with a long name that is very specific to this application.
00:25:27 So during the testing, I set that environment variable using the Monkey patch fixture within Pi test. We can talk about that on another episode too, but it’s basically a way to set environmental variables for a chunk of test code. So set that to the directory of a temporary directory that I’m generating and then run the code. So then when the code under test looks for where the database is, it says oh, there’s an environmental variable set. So I’ll use that instead of the home directory. So that little bit of code that uses that instead of the home directory, it’s using that for all the testing. It’s not using the home directory. So that one line of code is not being tested. That line where it says use the home and home directory instead. I’m okay with that because I’m using it all the time. I’m dog Fooding it as well this application. And I know that the home directory thing works because I’m using it all the time. I could if I wasn’t using it all the time, let’s say there was a Windows version of it that worked different.
00:26:42 Well, I probably would try to write a test so that it was tested more often.
00:26:51 So that’s one of the reasons.
00:26:53 One reason why I wouldn’t test something a line of code is to say is this, I’m dog Fooding it anyway, or it’s being tested some other way. I’m confident that this line of code isn’t going to break because of other measures in place. So I’m okay with like pragmating that out and saying we don’t need to cover that. So what does that do? If you exclude files or exclude stuff you’re focusing and you’re telling the coverage tool this is exactly what the source code that I care about. This is what I want my tests to cover. So even though I say I’ve got 100% coverage, it’s 100% coverage of the files I care about and that’s important. What are some other chunks of code if name equals main pieces that sometimes in different modules or different things? Somebody’s like throwing if name equals main. So when it’s imported, like it normally is by the rest of the system, that doesn’t get hit. But if you run the file on the command line, you get some behavior like maybe some little debug functions that you checking things out.
00:28:04 Well, personally I really look at those and go by the time I’m at the point where I’m trying to ship or test something for real and share with others, do I really want that in there? Maybe just delete it?
00:28:16 However, if it is useful to be there and it’s not a customer feature, to be able to run that by itself, I would Pragma those out. I don’t think I’d need to cover those. There’s hard to test things. Be careful with the hard to test part, though. For instance, if I’ve got multiple operating systems on there, I don’t think that’s a good reason to not test something because you’re not developing on the operating system. Because, as we’ve seen before, half of the Python users in the world are running Windows, or more than half probably. But a big chunk of the developers are running either Linux or Macs, so be careful with that.
00:29:02 If it’s important for you to support a different operating system, it’s important for you to test it. Those I would use continuous integration, make sure that you’ve got test coverage, at least for those portions that get run on every operating system, and then combine the coverage.
00:29:19 There’s different tutorials online how to combine that. What are some other reasons? Hard to test because it’s a weird corner case that you can never hit. Really, you can never hit it if you can really never hit it from the outside world. Can you delete that case? I mean, can you maybe just take that code out of your system if you’ve convinced yourself that you can’t because of various reasons, could you write a unit test for it? But I generally would.
00:29:52 If you can write a unit test or a subsystem test for it, and it’s a reasonable part of that code that you really want to keep around for some reason, you can go that route.
00:30:04 I would really lean towards trying to get rid of the code, though, or write a system test to exercise it. I’ve seen chunks of code, though, that you can’t hit from the outside world. There’s no way from the system level to set of input, like, for instance, the bulletproof concept. So if you’ve got code that checks all of its input, all of the parameters that come in and make sure they’re all valid, and there’s a bunch of if statements at the top and ranges and whatever.
00:30:36 This is defensive programming, and it used to be taught a lot. I don’t know if it still is that’s great for stuff that you don’t know who’s going to use it, like for APIs or external APIs, be very defensive on external APIs, go for it. But internal stuff, it’s just a little tool that you’re using a helper function or something within your code. Do you really need to be that defensive? If this part of the code can never get hit with a negative number, maybe you don’t need to check with it. I think it’s perfectly fine to throw a test around it and say, yeah, just throw a test around it and say, can I hit negative numbers into here if you can’t, if it’s being guarded by other parts of your system, I would argue to simplify your system and don’t put air checking through everything, error checking in all of your code just makes it more code and it makes it more brittle and it makes it I don’t know. I don’t like that. But it’s a design pattern for your choice. If you’re going to write bulletproof code in all levels, then you’ve got to write tests at all levels.
00:31:38 You should anyway.
00:31:40 Because if you’ve got those weird errors that you don’t think it will ever happen, are you sure that logic is correct? Because if you’ve got the less than your equal or the greater than equal or something a little bit wrong off by one, it really wasn’t greater than equal. It was greater than instead you got to make sure that speaking of that, when I was talking about really trying to hit behavior coverage, this could be a whole episode in itself. But I use a whole bunch of different tools for trying to hit behavior coverage.
00:32:16 I guess I personally say instead of trying to hit code coverage numbers right away, I try to mentally hit behavior coverage. And then at the end after I think I’ve Hitten mentally 100% behavioral coverage, am I really hitting everything and did I forget something? So behavioral coverage tools? I do boundary value analysis. What the inputs of different features am I doing hitting all of the different values that can go into input for there thinking about given when then as a system level and a system level in the features.
00:32:59 If I have an action that I’m testing the behavior of, what are the given States? What are the different States that I can start in? And are they all defined that’s it seems like a weird thing, but like, let’s say if you turn the keys to turn on a car, what happens if the car is already on? If it’s off, you should be able to get the car to turn on. But the beginning state is really important. If it’s already on, what happens if it’s out of gas? What happens if it’s the wrong key? What happens? I mean, there’s a whole bunch of different beginning States. So trying to hit all the beginning States, what are all the end States?
00:33:39 Is it possible for me to hit all of those in States trying to get tests around those so that’s using boundary value analysis along with given one, then I think mental finance state machines or even written down like pencil and paper state machines or state transition tables are really a great thing to say.
00:34:00 List all the beginning States, all the transitions that can happen. Can we get from A to B? Can we hit all that map with our tests? And then all the air conditions that you have in your system? Are they reasonable?
00:34:12 What are the user air conditions?
00:34:15 I’ve mentioned this before, but like index out of range.
00:34:18 I was just noticing a little tool I have that doesn’t handle things very well. I get huge trace backs if I throw in Invalid data and I got to fix that.
00:34:32 And that’s one of the reasons why dog Fooding is important as well. So after I’ve got all that, trying to get coverage up is great. So how do you get coverage up? We’ll try to do behavioral coverage. For instance, we’ll help you with code coverage as well. And then looking at it to say, can I narrow the scope? If this code really isn’t mine, it’s not our teams.
00:34:56 Why is it there if there’s a reason for it to be there, but I’m not even allowed to touch it, what’s the point of covering it?
00:35:04 Maybe that’s a good reason to cover it, but it’s a decision point. At least there is a discussion.
00:35:12 When you talk about code coverage, there’s often a discussion about the 8020 rule, the paradox rule, or the law of diminishing returns. So here’s the idea around all of that, and I’m going to mask all these ideas. So my apologies ahead of time. So Pareto principle or the Ad 20 rule essentially is your effort of anything produces 80% of the results. And people have used that for all sorts of stuff and use it because it’s handy. But let’s say I’ve got zero coverage. If I do a hypothetical amount of work would have done 100% coverage. If I do 20% of that hypothetical total amount of work, I can bring the coverage numbers up to 80%. That’s pretty cool. Does that apply again? So if I spend the double the time that’s the 40% of the hypothetical time did, I get 80% of the remaining 20%, and does it keep happening? So no matter how much time I spend trying to write tests to cover my code, I’ll never quite get to 100%.
00:36:22 Well, that’s silly one.
00:36:25 We can’t infinitely define our lines of code at some point. You’ve got lines of code. The other thing is decide what you’re going to cover. And once you get to different numbers, 80%, 90%, I don’t know what it is.
00:36:41 Look at that. Now, I did mention that it’s not just one number as well, right? I hope I did.
00:36:48 There isn’t really 80% or 90% coverage. Well, there is 100%. If you’ve got 100% covered, it’s all 100% covered. It doesn’t mean you have complete testing. It means you’re hitting every line of code. However, under that, let’s say your average is 80%. You don’t just get 180 percent number, you get percentages for every single file. And then you’ve got a way to look. So coverage itself will show you HTML output.
00:37:17 You can visualize it and open up in a web browser and look at all your code with which lines are covered or not, or within an editor. You can look at like PyCharm or something. You can look at which lines are covered and which ones are not and which files. So you’re going to look at all the files. So let’s say you’ve got the code that you’re working on. You kind of want that to not go down, right? So if it’s already at coverage before you start writing your tests and it doesn’t increase, what are you testing if you’re not covering the code? That it’s not just a percentage, though. It’s also specifically which lines. And you should be thinking about that when you’re writing tests for your code. Is the test hitting your code if it’s not in coverage can help you determine that. So the percentages are nice in a broad sense. Also, if you’re looking at a project and you can look at all the files and they’re all within the 80 or within 90 to 100 everywhere except for two or three files are down at like 10% coverage. What’s going on there?
00:38:32 What’s all that other stuff? Now, I have definitely seen files like this where I’m like, oh my gosh, this is like 30% coverage in this file. What’s going on there? And it turns out a bunch of junk in there. There’s a whole bunch of functions that aren’t being called by anything.
00:38:48 They’re just not used anymore.
00:38:51 I’m sure you’ve done this. I’ve done it. You write a function, you’ve tried it out and it doesn’t quite work, right? Or more specifically, I want to rewrite a function. I’m like taking change the name of a function to be like Foo old and Foo new or something, or just Foo and Foo old and the new function. I’m trying to slowly modify it, and I’m a dork and not using version control to keep the old one. I’ve got an older copy in the file and then I forget to delete it. It’s just there.
00:39:30 It’s never going to get run.
00:39:33 So some of these low numbers might indicate that you can clean up your code. So it’d be great to clean those things up. If they’re not getting hit by anybody, clean them up.
00:39:43 Well, I can write a unit test to hit that old function. Yes, you can, but then you’ll never know when you can delete it. So don’t do that.
00:39:52 Gosh, I know I had a whole bunch of stuff I wanted to cover, but I think I’m in a coverage of the coverage topic. Nice, but I think I’m running a little long and I would love to hear back from you.
00:40:06 How do you treat code coverage?
00:40:09 Are you trying to get 100%? I guess the last thing I want to make sure that I cover with coverage topic is it’s best driven by the software developers themselves.
00:40:23 Having a manager say Thou shalt have 100% coverage or 98 or 96% coverage is obviously silly because we’re developers and we’re smart enough to be able to write a whole bunch of tests that don’t actually assert anything and hit 100% coverage right away.
00:40:41 And we can Zoom in and do unit tests as well and hit those. I wouldn’t do that.
00:40:48 So don’t. But be careful of mandated numbers, but also be weary of numbers that are low.
00:40:58 The other thing I wanted to bring up is I had this idea that for a while that possibly open source projects really should be at 100% because it helps with new developers coming in.
00:41:11 And then after I said that once out loud I thought how is that different in a closed source project? How is that different in your code at work?
00:41:22 Why would you not want to make it easier for a newcomer to come in and look at your code and start contributing? It’s got to be just as valid in closed source projects than it is in open source projects, isn’t it? I think so.
00:41:37 But I’d like to hear what your interaction with code coverage is, what your rules of thumb are. If you’ve got some cool tools or some other ways or if you have a decent way to measure behavioral coverage that I haven’t thought about and now I do not want to measure coverage by using behavior different driven development pickle or cucumber or gurkhen or any other vegetable you throw at it. I’m not interested right now. Maybe in the future. So anyway, that’s all for today. Thank you high charm for sponsoring, this episode. Please check out their link and I really do think it would be fun for you to try Pi PyCharm pro and try out the it’s worth it when you’re talking about coverage just to try running coverage with PyCharm pro I find it handy. Maybe it’s handy for you. And with the Lincoln the show notes you can try buy Term pro for a few months for free so why not? That’s all for now. Now go ahead and test something.