Roadblocks to writing tests, and what to do about it.
Transcript for episode 78 of the Test & Code Podcast
This transcript starts as an auto generated transcript.
PRs welcome if you want to help fix any errors.
Welcome to Test and Code Code, episode 78. I can’t believe we’re at 78. What a wild ride this has been. Anyway, today’s episode. Some developers either don’t write tests or don’t like writing tests. Why not? I love writing tests. In this episode, we examine lots of roadblocks to testing and start coming up with solutions for these. I’m not going to solve all these problems, but I’m hoping that you and me and everyone that we interact with together can help start fixing some of these things. Thank you to Pantheon for sponsoring this episode. Pantheon makes building, managing, and optimizing websites simpler. Check them out at pantheon. Io Test And Code and listen to their spot later in the show.
Welcome to Test and Code, a podcast about software development, software testing, and Python.
I’ve been on projects with no automated tests where developers manually tested everything the best we could before shipping. Sometimes it took weeks of test debug fix. Now this was really early in my career, and I don’t like to work that way. I’ve also been on projects with separate QA, doing a combination of manual and automated tests. It still took a long time to fix this stuff at the end, with bug reports mentioning tests that me as a developer, I couldn’t run on my system. I would try to reproduce the issue myself and whether I could or not. I would try to fix something and examine the code. And I hope that the QA test would pass on the next report and we’d get reports daily at the end of the cycle. But yeah, I don’t really like working like that either. And I’ve also been on projects with a mix of developers developing tests as well as test engineers embedded in the test development team.
Iteratively writing tests and writing code at the same time while the code is being developed. That last one is so much better and with so much less wasted time. I like working like that. That’s my preferred method. I’ve also developed my own projects, just side projects and stuff on my own, some with very few tests and some with full coverage of tests. The ones without tests, they kind of go by the wayside and they rot.
The ones with more tests I’m more proud of and more sure of the code with tests, I like writing tests.
It’s probably not surprising to you because I wrote a book about it. But why do some people not write tests? I can guess, but I thought asking might be a better option, so I did.
I asked this on Twitter. I said I don’t write tests because blank or I don’t like writing tests because blank.
Complete this sentence, please. That’s what I put on Twitter. I got a whole bunch of answers. I’ve grouped the answers into a whole bunch of categories so that I can talk about each category.
I got all the names mixed up to start off with. So thanks to everyone who replied it’s helped a lot. It’s a long list of people, so I’m not going to thank everyone individually, but thank you for contributing. So let’s just dive in. Let’s look at the reasons and the categories and figure out what we can do about it. Here’s the first category. I don’t understand the code under test. So here’s a couple of example answers.
If I wrote test, that would require me actually understanding my code or I don’t really know how this code is supposed to work. These are very telling if you really don’t know how the code is supposed to work, how you’re supposed to write tests. But how are you writing code to begin with?
This is troubling, but I think it actually applies to a lot of people. I don’t really know how to fix this, but I think writing tests either after or during or before you write the code might help you understand your code better. So let’s give it a shot. Next one is just I don’t like a certain type of test. And this one is like, I don’t like tests that have lots of mocks. That was the comment. Well, yeah, me either. So I think this is a focus on unit tests and people thinking that unit tests are what developers were supposed to do and system or integration tests or what somebody else is supposed to do. But if you don’t like writing tests with MOX, don’t write tests with mocks.
This next one is, I think, a very common thing when you’re excited about getting the code done and you think you’re going to get back the test, but you just don’t. So here’s a couple of comments. I get excited about the problem, and I just want to jump into solving it. It’s not the right way to do things, but when I skip writing tests, that’s the reason most of the time.
And here’s another comment, just this little thing. And then I’ll test this and this. I think these are great examples of the reason why a lot of people advocate for test driven development, because this attitude of I’m going to just do a little bit more code and then a little more code.
It’s hard to get back to making sure that your code works when it’s so fun to write new code. Now, I personally don’t follow the TDD thing exactly all the time. Of course I don’t, but I do like to try to make sure. I like to think of it as I can’t remember who did this analogy, but it’s like the mountain climbing analogy.
I’m not a mountain climber either. So it’s just my mental model is you’re climbing a mountain and you’ve got hand holds and you’re not quite sure which path to take. So you try a whole bunch of different ones. But then when you do find a way to get up, maybe like another couple of feet, then you can put like one of those spiky things in the rock face and hook your rope up there, and then you can keep going. And, you know, if you fall, you’re only going to fall that far.
Actually, that sounds terrifying. That’s not why I don’t mountain climb, but for software, I think of tests like that also. So you can get a little bit farther. You can get a little bit farther ahead of your tests with your code. But it’s good to make sure to put the test in place, to make sure that as you continue on with your code that you don’t break stuff that you haven’t broken the things that you’ve sold before.
So that’s my answer, too. I’ll get back to it later.
Be safe. Be a safe climber and stick those spikes in whatever.
Thank you to Pantheon for sponsoring this episode. Pantheon Makes building, managing and Optimizing Websites Simpler as the leading DevOps platform, Pantheon features the fastest hosting on the planet, automated workflows with Dev test and live environments, plus pro development tools and easy integrations with GitHub, CircleCI, Jira, and many more. Stop putting out fires and start building sites and experiences that deliver results by using Pantheon, rated as one of Gtwo Crowds Top Ten software products of 2019. Get started for free at Pantheon. Io Test And Code. That’s pantheon. Io Test And Code.
Next up is test writing skills. This is kind of an obvious one. Maybe not obvious, but the answer is I don’t write tests because I don’t know how or I currently don’t know how, but I’m going to learn one day soon or designing tests. Good tests is a skill. Yes, it is. I think it all boils down to the same thing of how do you know it’s working manually? If you’re not going to write automated tests, if you’re going to do it manually, how do you know it’s working?
Can you codify that? I know that’s kind of a glib answer to that, but it’s very real. Here’s an interesting category. How much is enough? I can never determine or find agreement on what good tested coverage looks like. The explanations and guidance I hear are as varied as there are programmers. And another comment.
I don’t write tests because I can’t reach the level.
The real flawless code level. Yeah.
You can’t ever determine that your code is perfect, and I know it’s never enough, so what’s the point?
That’s kind of defeated.
I don’t know how to fix that.
Doing a few tests and making sure that the code works like you think it’s supposed to is definitely better than nothing if you’re really working for perfection. I don’t know if software is the right career for you, because we don’t really ever hit that. That’s why we have versions and we keep adding things and fixing bugs. But actually, I don’t think that’s really real. I think those are excuses I can’t get there anyway, so I’m not going to try.
I don’t think that’s a real excuse. I think people are making that up because they don’t feel like they can have the skills to write stuff or maybe they feel it’s below them. I don’t know.
This next category is not the right time. And actually, I think there’s something legitimate to this one.
Here’s one I’m prototyping. Yeah, I’ve totally gone through like a demo for Django demo or some other web framework demo or some other thing, and I’m writing tests for them. I’m just playing with stuff and doing what they call like a spike or something. I’m just trying stuff out. I don’t think there’s anything wrong with that. But set your spike aside or your prototype aside, and when you’re going to write the real thing, write it with tests, I think. Here’s one. My work involves programs that I will never see again after writing. What kind of job is that?
I’m always working with legacy code and working with the code that I’ve written six months ago or a year ago, even two weeks ago, that I’m like, what does this do? So, yeah, I don’t really know how to relate to that one here’s. 195 percent of what I write is proof of concept, and speed of development is top priority. It’s like prototyping. It’s proof of concept. Yeah. If you’re just testing stuff out, fine. But even that, how do you know it’s working? And I think maybe you’re testing manually and thinking that you’re not testing because you’re not writing automated tests. And for a lot of that stuff, I think manual testing is fine.
I think it’s good.
Next step, code structure.
This is another one of the excuse levels, I think, or maybe a misunderstanding. So I’m going to read a handful of these off.
My code isn’t very testable.
I don’t test because legacy code I work on, I work with is untestable. It’s blaming somebody else versus blaming myself. I don’t know how to write testable code, okay.
I don’t know how to convert my existing code so it can be tested.
This is the whole testable code thing over engineering at the beginning of the project that might be project structure.
Anyway, here’s another one. All the code was written in the waterfall days and manually tested, and now it’s completely untestable in agile short cycle days. So these are like, basically, I think my code is untestable. However, they think it’s untestable at a unit test level or an isolated function level where you can just test one function in isolation for the rest of the system. But there’s a system there. How do you know the system is working for legacy code like that? Especially if it’s a large project? I would not put that on myself of saying I’m going to try to incorporate unit tests into the entire project.
That would be a nightmare. But you can put in like, I am changing the behavior of the system. The code I’m adding is changing the behavior of the system in X ways and I know it’s going to work. If I start in this state and I run this stuff and do this thing, then I’ll see this other thing happen. If I can say that out loud, I can write a test for it and maybe it’s not like something. I’m really proud of that. I think that everything else is right, but I think that’s a great place to start. And I don’t buy the code as untestable. If it’s untestable, it must be unusable to users. How do users know it’s working? And if users don’t know it’s working, how are you selling that? How is this something that you have a job to do?
I don’t buy that argument. I do get it. If you’re really focusing on the mocked dependency injection unit test world. I don’t live in that world.
I’m a pragmatist. If you have to test from a larger scope in a subsystem or the entire system or an API or something, do it. Testing is better than no tests. It sucks without automated tests. It sucks to have all manual testing. There must be some way you’re verifying that things work at the end. You can automate that project structure.
I pulled this out separate from the code structure thing because it striked me as something sort of different. So the comment was there’s no test infrastructure in the project due to its legacy nature. The cost of the initial work on it is high due to the project complexity, and there’s no spare resources for that.
Sorry.
Yeah, I’ll bundle that into the next one.
The next category. I should have just bundled it in anyway. Next category is time and budget. There’s a whole bunch of these due dates don’t even allow enough time for feature development, let alone tests.
The clients don’t recognize the value and therefore are unwilling to pay for the test development time. It takes at least three times as long to write a useful test suite of tests than the feature itself.
That’s a time thing, but I think that’s separate. I think that’s a misunderstanding.
We’re just going to push it to production on the timeline anyway. I wish this wasn’t real, but I know this is real for a lot of people. The timeline is set. It’s going to go out anyway. What’s the point of spending extra time testing? I’m sorry if you’re there. I’ve been there.
Nobody’s yelling at me to get my test done. They want the feature by the date and not the tests.
The client didn’t want to pay for the tests. The client shrunk the test budget. We don’t have time for tests. I’m working on an outsourced project and no one cares about the test in this case. And I think this all boils down to there’s no incentives.
The incentive structure is not there to write tests. Fast coding has incentives. Good testing, not so much. And that’s where the misunderstanding I think is the crucial bit. Fast coding has incentive. Good testing, not so much.
There’s a misunderstanding that writing solid code with tests is slower than writing code without tests. This is not true. This might be true for a week, but as soon as any project more than a week, you’re going to deal with legacy code, even if the legacy code is the code you wrote last week. The code you wrote last week. If you’re sure that the code you write this week is not going to break the code you write last week, you can code faster. You can write faster. You can take more risks. It’s all about practice. I say write code with tests. It’s always together. It’s not code. Then test or test, then code it’s together. Put all of your budgets together and there isn’t an option.
If you’re working in an environment where you get in trouble for writing solid code with tests, find another job.
I know that this isn’t realistic all the time, and I know that people have budget constraints. And I know that sometimes you’re going to manually test some things because it’s kind of working right now and you need to ship it and you’ll automate it later. Just make sure the later actually happens or else your manual test stuff is going to blow up into a pain in the rear.
I’m going to link to a previous episode where I talked with somebody really in depth on this.
This next category is hilarious, but I know it’s real. I don’t write tests because I don’t want to know if I look away intensely enough. Bugs may disappear.
That’s kind of a joke. But under management pressure to release a feature, I find that failing CI is annoying. Again, I know this is real for some people. If you write a test and it fails, you’re going to be expected to fix it. If you don’t write the test, your CI will be all green. You’re cheating. You know you’re cheating.
Just talk with the powers that be and management pressure and you fight the man.
We got to make things better. And also, I think there’s no silver bullet, but Yuck. Sorry for that. We need to push management more to say test and code go together. They’re not separate. They’re the same.
Here’s one that like, I’m going to name somebody now. David Beasley. He wrote, I don’t test because customers are the tests. Now, I know that David writes tests. I know it was a joke, but this happens.
I think he joked about it because some people get good enough and send it to customers and they only fix the bugs that customers send back. Don’t be this kind of a company. It’s going to bite you eventually. And if it doesn’t, I don’t know. You’re not going to feel good about it anyway. Yuck. Okay, workflow interruptions. Here’s the next category. Writing test forces me to create and analyze at the same time. These are two different mindsets, which this shift is counterproductive, I guess. I think of writing tests as a creative activity also. So I actually have a lot more fun writing tests than I do writing the code to implement it. But that’s just me.
Workflow interruptions. It takes too long to run or the tests are slow. I know this is real. I know that some people have to deal with test suites that are annoyingly long.
Yeah, they’re a maintenance thing. It’s part of the code, and it’s one of the parts of the code that affects the workflow.
There’s ways to get around it. We will talk in future episodes on how to speed up test suites, even especially test suites for system level or integration level tests that aren’t necessarily slow. But we can make at least fixed setup times. We can reduce the amount of test cases, maybe with some boundary value analysis or some partitioning.
Anyway, there are ways to speed it up, and maybe you don’t run everything all the time. The code you’re working on right now. Run the test cases for the code that’s around that area and do a subset, and then let the computer run all the long tests all night long. I’ve never been on a system where the automated tests took all night and they weren’t done in the morning. I mean, I joke about that, but it’s real. I’ve never been on a system, even though I work on fairly complex systems where the tests and I have worked on systems where the testing does take even a couple of hours. That’s a painful process, especially if I’ve got a long compile time.
But in the middle of the night, who cares?
I’ve run it on a handful of servers and set it on different systems. I can have multiple versions running with different scenarios, and it’s totally done by the end of the night and I get in the morning and it’s good.
Split up the test suites. So anyway, we’ll talk about speeding up test suites in future episodes.
A couple of them are related to test driven development.
Test driven development is an annoying industry with too many ads associated with it. It’s kind of disgusting.
Yeah, I kind of agree with that.
That’s why I’ve been reluctant to talk about TDD too much on this podcast, because I don’t really want to be part of that complex.
But I do believe in test driven development, and I think it’s a shame that there’s a lot of people spreading misinformation.
Okay, next, one misunderstanding of TDD. Test driven Development is not for the impatient. This is a comment from somebody. It’s not for the impatient. It’s for developers who want solid results. I am a very impatient person, and I want solid results. I want both. I write tests because I want both.
I am a creative person. I want to write my source code. I want to be able to write it and figure out what it does and not have to remember stuff. That’s why I write automated tests, so that I don’t have to remember what it’s supposed to do. Once I figure it out, it’s encoded in the tests. And I’m an author too. I like writing things several times, so I want my code that way too. I’m also not that confident that I’m going to get it right the first time, so I want to be able to write a first draft that works. Might not be pretty, but it works. Put tests in place around it and then I can go back if I’ve got time. And I usually have time because I’ve done this so fast that I will rework it so that it’s prettier. Or maybe next time when I add another feature, I’ll go rework it. Refactoring maneseur the red green refactor doesn’t mean rewrite your entire system all the time. It’s when it’s ugly enough to bother you and you’re not proud of it. Make it code that you’re proud of.
I am very impatient.
I really want solid results. I think this is a misunderstanding of test driven development that testing takes longer.
I think it’s because it’s this focus on unit testing. But I don’t do test driven development with unit tests. I do it with higher level tests. I do it at whatever lower level test. I do it at system level test. I do it at whatever level of test I need to answer the question, does my code work? I have put one of the categories off for a while that I need to come back to and it is education. So I’m not sure why I put this off, but it’s something I think we need everybody’s help with. So why do people not write tests? Here’s a few comments. I’m self taught and the free learning resources that I learned from did not emphasize testing. I just graduated from code school and I haven’t learned that skill yet. As a newbie, I was never taught how to do this. It’s something I considered an advanced topic.
Now this is me. I’ve got a bachelor’s and a master’s degree in computer science. I don’t remember if we covered testing. I think we may have covered it in the software engineering class, but we never practiced it. It’s embarrassing that I got a master’s degree degree in computer science without having to write very many tests. I think this needs to change. It is not an advanced topic.
We expect electrical engineers to leave school knowing how to use a spectrum Analyzer in Oscilloscope. We should expect anybody leaving any program on how to program how to do software to know how to make sure their software is working.
It’s unacceptable that we don’t teach this. It is not an advanced skill. It’s just software that verifies that other software works. That’s it. It’s not weird. It’s not corner case.
Okay, there is a mindset difference though, and I’m going to go actually kind of intense this episode because I’m so passionate about this.
So let’s try to break my mindset a little bit by telling a few jokes. How about a couple of jokes? We do jokes on Python bites and it goes over well. So I don’t know who to credit for these, because I find a whole bunch of references to both these jokes. Okay, here’s one. It’s the car. The software engineer on the Hill thing. There was an engineer, a manager, and a software engineer driving down a steep mountain road. The brakes failed and the car careened down the road, out of control. Halfway down, the driver managed to stop the car by running it against the embankment, narrowly avoiding going over a cliff. They all got out, totally shaken by the narrow escape from death, but they’re otherwise unharmed. The manager says, to fix this problem, we need to organize a committee, have meetings, and through a process of continuous improvement, develop a solution. The engineer says, no, that would take too long. And Besides, that method never worked before. I have my trusty pin knife here, and I will take apart the brake system. I select the problem and correct it. The software engineer says, I think you’re brother. I think we should push the car back up to the top of the Hill and see if it happens again.
I love that joke. Okay, here’s another one. A QA engineer walks into a bar and orders a beer. Orders zero beers. Orders nine, nine, nine beers. Orders a lizard.
Orders minus one beers. Orders a Ueicbkjst.
The first real customer walks in the bar and asks where the bathroom is. The bar bursts into flames, killing everyone.
Actually, the second one, I did find a reference on Twitter. That version came from Brennan Keller.
I think these are funny jokes, but they’re a mind shift joke. They’re funny because we think engineering, programming and testing are different ways of thinking. But as software engineers in today’s world, we have to do all of these things. We have to be engineers, we have to be programmers, we have to be testers. And we think that that’s difficult to shift in one person to these different mindsets. And some of us have to be managers also.
So what are the mindsets? The testing is that we try to come up with weird input to crash things. We’re trying to break our software in development, really. If we get a bug that’s assigned to us, we’ll just try to reproduce it. And if we can’t reproduce it, if we can reproduce it, then we can throw it in the bugger and try to fix it. If we cannot reproduce it, I’ll just close. The issue is can’t reproduce and wait for it to come up again.
But these are both wrong. These are wrong. We don’t need to work like this.
So testing and development are part of the same thing. We have to think about what the code is supposed to do and we have to think about how we make sure that the code is doing what it’s supposed to be doing. This is what software testing this is what software development is. Just how to make the code do the thing that it’s supposed to do. That software making sure that it does that is testing. They’re the same thing. They might live in different files, but it’s the same process. It’s the same mindset. It’s the same creativity. The corner case stuff like the minus one and 999, that stuff is called input validation and load testing and stress testing.
These do apply to some fields. They don’t apply to all software and they don’t apply to all of your code. They apply to the input points, the user interface, the API, those things have to do input validation. You don’t always have to write bulletproof code throughout your code.
Maybe you do because you’re writing pacemaker code or something like that. But not all code is the same and not all testing is how do I break it? Most testing is how do I make sure it works?
Huge. Thank you to Patreon supporters for sponsoring this episode and all episodes. I really could not do this without you. Thank you thank you to pantheon for sponsoring this episode. Pantheon makes building, managing, and optimizing websites simpler. Check them out at pantheon. Io Test And Code. That link is also in the show. Notes at testandcode.com 78 this has been Test And Code episode 78 that’s it for today. Go ahead and test something.