Test Driven Development, TDD, is not easy to incorporate in your daily development.
Martin and Brian discuss TDD and testing and Martin’s experience with testing, TDD, and using it for code involved with scientific research.
Transcript for episode 144 of the Test & Code Podcast
This transcript starts as an auto generated transcript.
PRs welcome if you want to help fix any errors.
00:00:00 On today’s episode of Test and Code Code, Martin Haru comes back to the show to discuss how to apply software testing and test driven development to everyday code. Specifically, Martin works on code for academic scientific research. This is one of my favorite types of interviews, really a Q and A. It’s more of a discussion of the value and difficulties of incorporating software testing with actual work. Also, super excited to welcome Lenod as a new sponsor for the show. Please listen to their spot and check them out.
00:00:43 Welcome to Testing Code.
00:00:53 Good afternoon.
00:00:54 Hello, Brian.
00:00:55 Love that T shirt. It’s great. Thank you.
00:00:58 Yes. So Joy Division T shirt, but I guess originally it was the first sounds recorded from some radar, something in the I don’t know, 50s or Sixties. But then Joy Division English group took it in the late 70s as a cover for their album.
00:01:13 How you been?
00:01:14 I’m good. How about you?
00:01:16 A busy back to work this January. But other than that, it’s good.
00:01:22 Yeah. So we’re talking TDD today.
00:01:24 Yes, we are.
00:01:25 Exciting. So welcome back to Test and Code Martin Martin. Let’s see. Hero. Yeah.
00:01:31 Perfect.
00:01:32 Awesome. Although I really do like the heroic story. That’s one of my favorites.
00:01:37 So we ended the conversation last time and we were talking about possibly talking about test driven development. So we’ll jump into it. Yeah. Let’s just jump in to you. What is test driven development?
00:01:48 Well, the most basic understanding of it is that is that you start by writing, you don’t write any new code.
00:01:54 And again, this is not a hard and fast rule, but my basic understanding is that you write a test first, you run it, it fails, which is good.
00:02:03 And then you slowly build up functionality that it will just make that test pass.
00:02:09 And just enough, you don’t go continue on and add functionality. You just get that one test to pass.
00:02:15 And then at that point, you take a bit of a breather, you refactor, meaning that you clean up some things that you might have just done in haste to get the test to pass. And then once you’re happy with that, you kind of take the next bit of a chunk and you don’t jump in again. You start from the testing again. And that’s the most fundamental. So that red, green, refactor, red. The test fails, that you’ve just written green. You’ve done stuff in your code to make that test pass. And then refactor is where you might fix your sins for whatever you did to get that to pass. Possibly it was a bit of a quick and dirty solution, but then refactoring is where you actually get to do the clean code side of things and the maintainable side of things.
00:02:57 Yeah, definitely. So is this something you practice?
00:03:01 I started it.
00:03:02 And again, I’m new to all this. And so here I I am have a project that I do in my own time. That is for work, but it’s kind of something that I’m not rushed with. And then I have a bit of time. So I’m kind of taking it as my experience to learn TDD. So already I have some good insights and stories to tell. But just in the last week or two, I’m at work. I have to get some stuff done. With my 15 years or so of coding history.
00:03:28 My brain just knew the solution, and obviously it won’t be maintainable or testable, but I just started coding because the pressure is on each. And so there’s part of me that’s kind of like I should probably write tests for this. But yeah, it’s that fight between getting it done versus and it shouldn’t be a fight. I think last time you highlighted that, we shouldn’t see it as it’s going to take that much longer and see the, oh, I have to now add the test, but my brain is doing that right now, and I’m trying to think about adding tests. It seems like it’s going to take a lot more time, likely because I’m new to the testing framework. And I thought about that a bit after a conversation that learning to code and test at the same time, I think would be a great start versus myself. I learned to code and have my own style of coding and the way I do it for 15 years or whatever. I’m now back to learning a new framework, a new approach, and a new way of thinking. And so when the pressure is on, I revert back and I see this new thing as, oh, I haven’t struggled all that much in my 15 years. I guess I could just get it done. And so I could see why learning, testing and coding at the same time and see them as things that go hand in hand, I think would be a useful thing.
00:04:41 Thank you, Lenode, for sponsoring this episode. Simplify your infrastructure and cut your cloud bills in half with Linode’s Linux Virtual Machines Develop, deploy, and scale your modern application faster and easier. Whether you’re developing a personal project or managing larger workloads, you deserve simple, affordable and accessible cloud computing solutions. Get started on Lenovo today with $100 in free credit for listeners of test and code, you can find all the details at lenode.com. Testincode Lenode has data centers around the world with the same simple and consistent pricing. Regardless of location, choose the data center nearest you. You also receive 24, 7365 human support with no tiers or handoffs. Regardless of your plan size. You can choose shared and dedicated compute instances, or you can use your $100 credit on an S, three compatible object, storage, managed Kubernetes, and more. If it runs on Linux, it runs on Lenode. Visit Lenode.com Testandtcode and click on the Create Free Account button to get started.
00:05:47 So test driver Development I think of it differently, so I learned it early on, or I think of it as early when it was introduced during discussions around extreme programming in the Nineties, there was a notion of test first programming, I think extreme programming called that at first. And you have some notes down here that there’s the idea that test driven development is where you do the test first, but the alternative would be then doing the tests after. Right. And so the problem I mean, there is issues with doing the test afterwards.
00:06:24 One of them, like you said, is that the test might even not get written. That’s the main problem.
00:06:29 Yeah, I guess. What if, like the red green refactor, the red part of it ensures that there’s something wrong with the code will fail because there’s nothing there right now, and then I’m going to implement something that just passes that.
00:06:43 Is it possible that when you write your tests after you write a test, that in the end, in a sense, will always pass?
00:06:50 Yes, that is possible.
00:06:51 Is it more likely in the testing after, do you think?
00:06:55 I don’t know. You’re writing different tests is what I’ve seen. So I have in little tiny projects I’ve tried, like what you’ve said, something close to an issue that I think of with the way we talk about test driven development is this whole red green refactor thing, and also this notion that you can only write failing tests and you can’t write code without writing a test first. And I don’t do that. I do develop software and tests at the same time, developing in parallel. But I often jump in and think about the tests that I need to verify the functionality that needs to be to work. And I might write 510 or 20 tests all at the same time before I jump into the code. And then I work on the code until I get all the test passing or jump back and forth to me, I don’t think we’re getting the same benefit.
00:07:54 We’re getting the same benefit of allowing tests to verify that our software is working and we can build for do you think that you mentioned you started you got exposed to this in like the 90s since you’ve been testing for a long time. And possibly I guess my question to you is you kind of have an idea of what you want to be tested and how to do it.
00:08:16 Would it be possible that somebody who’s learning to test my experience from my previous project where it was the first time I implemented them, I did do it kind of after the fact as I was going, but not until the whole project was done, but after modules were done and things and I kind of didn’t really know where to go with it with the testing. I kind of did it, but I felt a bit awkward about it in the TDD stuff that I’ve been practicing just more recently. It really has changed the thinking process. And so I’m just wondering for you, is it that maybe you’ve kind of integrated all this stuff and so for you, maybe the actual TDD steps of one the other and then the other might be almost do it all, regardless of first or after you have an idea of what a good test is and how to write it is that you see that possibly for somebody who’s a bit new to testing, that might be a bit confusing and that the TDD kind of gives you a bit of a recipe that at some point you become proficient like a chef and you kind of let go of the recipe. But would it be a good way to approach testing?
00:09:14 I don’t know.
00:09:19 I hope to get somebody on that actually practices this type of test driven and thinks of themselves as efficient because I think they’re different tests. I’m trying to develop the tests that I know I’ll need at the end to verify that the software is working. There’s a segment of the TDD population, and I think it’s mostly just consultants, but it may be actually practicing software engineers also that have this rigidity and also that the D probably shouldn’t be for development. It should be for design or something like that, test driven design, because it’s going to force with this little tiny you have to isolate things and just test small things.
00:10:00 You’re going to create testable little tiny modules and then you can plug them all together or something. I don’t find tons of value in those tests in the end, and I’m looking for does my system work? And so when you say you have a time pressure and you’re trying to get things done quickly, I totally agree that writing all those little tiny tests, that might take more time, and I can’t see how it can’t take more time. But the validating that the system was working correctly, I think that has to be done or else how do you know this offers working at all?
00:10:35 It might be, though, that it’s not that we don’t test when we’re not testing, we’re not writing tests. It’s that we’re not writing the test down, that maybe we’re just manually checking things to make sure that they look okay. And that is testing. It’s just not repeatable, I guess.
00:10:51 Yeah. And that’s what I found myself doing just yesterday. I was running my thing and looking at did I create the file structure and the dummy files that I wanted. And I just went back and forth between running my code looking rather than if I had an automated test for that, I just run the test rather than always opening up my browser and looking at things. Right.
00:11:10 But we do test. I agree. Hopefully you test to some extent.
00:11:13 Yeah. The hurdle, the part that takes more time is to me, in that case, if I’m replacing the manual testing, I would have done anyway with an automated test. The extra time is often that I’m just writing it down the first time. I think it’s taking more time, but it’s not really because let’s say the manual procedure takes a minute or even 30 seconds, but it takes me five minutes to write the test. Well, then ten iterations of writing the software, and I’ve paid for the time it took to write the test. And then I can also just it’s going to help me forever because I can verify that it’s always still working. But there is extra time of things that I don’t know how to test, like a file system structure. How do I tell that the files are in the right place? I got to think about how I would check for that, and that’s both the extra time. I also think it’s like the incredible value because like you said, I’ve been doing this for a really long time.
00:12:09 Every time somebody tells me about some functionality, my brain goes, how would I verify that? How would I automatically check to see if that’s working? And then so often I get back from some other part of the team or somebody else, don’t worry about it. We will validate it ourselves.
00:12:28 But then I have to push back and go, well, we can’t do that every release. So I’ll just tell you to verify these little bits that you’ve agreed to. Let’s talk about how we can validate that this functionality is working.
00:12:41 And you get too many of those. Like, we’ll just manually validate it. That’s not going to happen all the time. So you can’t do that.
00:12:48 Yeah. And I mean, that’s simpler level was when I was doing this current project, and I’m trying to do it TDD wise, but basically even just coming up with the test. So, for example, it requires the experimenter to enter some inputs on the keyboard and it has some actions. And so I was just like, well, how do I do that? And then so Pi test. But that’s when I came into mocks. That was the first time I knew there was a mock for an input. So I did that a very useful thing for the longest time for approximation. So is this value close enough to what you would expect? Pi test has an approximate method or function. And that was the most useful thing because I came up with a more Pythonic way to do it, and I got a bit frustrated. And so sometimes, as you’re saying it’s, how do you test that? And it becomes it’s because I know what I want to do. I just don’t know how to do it just yet. So there’s the two things of just thinking in a testing mindset. That’s one thing. And then how do I implement this test? And when you’re new to a framework? So I think there’s those two. And I guess my personal experience and I apologize if the book that I would desire is out there, and I have not come across it. But your book teaches testing, but more teaches a testing framework. And by going through that, I see examples of tests, but it’s not really this is how you test computer software. There’s another book on Pi test. It’s similar. It’s about testing. How do you and I guess what I was looking for is having no experience in testing.
00:14:17 I don’t want the recipe book, but in a sense, an introduction to what makes for a good test or even historically. Some people say unit tests are a bit too fine grain and they’re not good. And then there’s functional tests and then there’s some other level of tests. I’m sure just the lingo. I’m not exactly sure what that would mean. So just an introductory how to might give me an idea. And Kent Beck’s book is good for TDD. But it’s not that book. It’s very much almost like a personal conversation with Kent that he was just kind of talking to you as he was working through these examples. So it’s a very different style of book.
00:14:51 Yes. And also Kent back, I mean, admittedly, is coming from a background where he’s working or that he was always working in a situation, I think, where it was large companies that have a QA team. So the test driven development that’s this fine grain, making sure that your software is pretty darn good. It’s always going to then be validated by another team also.
00:15:12 And that’s the fine print in a lot of the XP and test driven development literature is they’re not actually promising that you’re going to develop tests that can validate your software. That’s the part where I was like, well, that’s actually the thing I want to do.
00:15:29 So like you said, what I think would be cool is not that book that you’re talking about that we don’t know if it exists or not. That would be cool. But I’d also like to have like Kempex TDD by example, something like that, but less fine grain. But also just here’s a project.
00:15:48 How would I test that and walk through the thought process and developing the test and the whole system, not just an example of it, but the whole thing. Let’s talk about that and then maybe pick another thing. Like I want to do an e commerce site. How do I test that? Because they’re different.
00:16:07 We always talk about this extreme. But if I had some software that created animated gifs, that would be a different strictness if it was developing software for a pacemaker. Yeah, I’m not going to kill somebody if I get it wrong with gifts, but hopefully not.
00:16:25 That’s a good point, because I was thinking of that. And the whole pacemaker and sending people out into space is one side of things, either the lean startup model or I’m just doing something for myself just to help myself automate things. The tests might not be necessary. I guess part of my thinking recently has been where does science stand in that? And there’s a lot of people who do different styles of science. So personally, what I say is basically what I’ve been exposed to and what I know about. And so I apologize for the scientists out there that do very different things than myself. But I’m coming from an experimental side of things.
00:17:04 I think of an experiment, I design it, I collect data.
00:17:08 I then have to create a pipeline to analyze that to get the results and then write them up. Other people, their whole science is about trading software and tackling tough problems. That’s different, I think.
00:17:19 Right. And so I would suggest I’m thinking, I hope that the science part falls more towards the pacemaker end.
00:17:26 And that’s exactly what I was thinking about is science. It’s officially going to be staying hopefully in the literature and has a longevity. I think last time you mentioned something like but we don’t want to slow down science, which I totally agree with. But the interesting thing is that I can’t remember. I think it’s 2005 that came out. There’s a very popular paper that said pretty much most of science is false. And it’s basically based on the premise that historically we use Pvalues and Fisher type statistics, and given the number of significant results, positive results that are published, which is almost everything, it is impossible just probabilistically it’s impossible that all this science is genuine. And it’s unfortunate. I think each individual at heart wants to be correct and wants to be thorough. But the pressures that exist from the publishers, from academia, from the funding is that we have to be impactful. We have to be. And so there are these evolutionary pressures on us that lead to things. And this move towards open science is great. And I’m just seeing that, well, testing, while it might take longer and slow us down, it may actually help catch some of this stuff that is getting out that’s being called significant, that is meaningful, but just isn’t reproducible. And I’m just wondering, having said that, when I read and hear about listening to people who talk about testing, they talk about the lifespan of the code. And one of the biggest benefits of testing is that you can then refactor with confidence and that later on, when you add functionality, you can do so again with confidence. However, my type of code, which is different than other types of scientists who their code is, their project is that after I have done my project and I’ve analyzed it a lot of times, the only time I look at it is if a reviewer six months from now asks me to do more than I cringe and I have to go back to my code. But other than that, the exact pipeline for myself, I will never run again. And that code will just sit there. So is it worth the effort?
00:19:32 To add tests to that, I personally think yes, but on the other hand, it doesn’t fit with the mentality of one of the main benefits of testing is this refactoring and maintainability because there’s no maintenance once it’s published. I will probably never go back to that.
00:19:48 Can we stick there for a little bit?
00:19:50 Yeah.
00:19:50 So I don’t know the specific stages. You call it pipeline, the data pipeline. I have a hard time believing that you’re never going to go back and use similar pipelines again. Aren’t future projects going to use similar pipelines?
00:20:04 Yes. And I guess for the first ten years of my coding experience, I always heard about this. Well, reuse your code and create modules that you could reuse. And I just couldn’t figure out a useful way to do that. I’ve created at least two packages that I published are an example of. Well, these are things that now I’m kind of using regularly, so I went through the trouble of that. But just as an example, the types of data that I collect for each experiment, there might be seven files per participant. Other times there will be 56. But the file formats, one of them is coming from this type of software, that one is the other.
00:20:40 And I guess rather than creating something that will work for every possible experiment, which seems a bit of a waste of time, I tend to just code it for this type of signal, and then this time the EMG signal, which is muscle activity, needs these steps of processing, whereas in this case, because it’s a different task, I might need this other style of. So some people, I realize, do have laboratories where they do have these kind of very strict pipelines that really the experiments look very similar. And there’s just tweaks.
00:21:12 Okay.
00:21:12 I just find myself I’m a bit more of a mutt of science. I just do a bit of everything, which means that my code I’ve dealt with ultrasound data and doing kinematic analysis of what muscles do. I found myself doing perceptual things were very simple. Data comes in. I found myself doing these other, more complex experiments that are more neurophysiology because of that, each time I kind of have to. And it’s also, as you’re saying, I didn’t go out of my way to create things that I thought I could reuse. And I guess it comes down to, even if you’re working on your own project, that’s not scientific, should you implement things as a just in case? And I guess what I’ve heard generally is that, no, that’s kind of a waste code what you need and then add. But I guess at that time it’s all kind of within a project module, I guess, or a study module. I put everything in there and I just later on the hassle of extracting the bit that might be useful. I see that it’s possible now because I have done that for two packages that I’ve published, but I haven’t found it necessarily doable in my workflow. And I guess adding tests just seems like that extra thing that obviously would be incredibly useful.
00:22:23 Well, in the data that we’re kind of getting off track. But I think it’s okay on the data part, like some of the scientific stuff you’re working on in these collecting some kind of data in a certain format and then converting it into, I’m guessing a lot of that conversion code that’s converting into a different format that you can use it like into a table or into something more usable with that part of it. Is the data getting validated anyway? Do you have some way to say, like bad data is going to get kicked out or.
00:22:53 Yes, but I guess historically a lot of that has been just done by plotting. And so normally I’ve known people who run the whole pipelines and extract summary values that you would actually use to put in the statistical test and have not really looked at what does that look like? So you might convert the data, apply certain filters and signal processing filtering and extract a value, but they never actually looked at it. And sometimes people who were not signal processing, we’re not engineers. And so if you don’t look at it, sometimes you might make some mistakes. And so a lot of the validation for us is this looking, plotting, at least for myself, missing data, not enough channels. Those types of things are things that I guess I do test, not in a formal way, but I try and accept certain things to make sure they’re there.
00:23:45 And I guess it is a form of testing.
00:23:46 Yeah. There’s a data validation stage because I have heard these horror stories of somebody you’ve got a data set that comes in and there’s a bunch of missing data, but the missing data gets filled in with zeros or something.
00:23:59 Yeah. Or 999.
00:24:00 And then the analysis sees all these things hovering around zero and they go, hey, everything’s mostly around zero. But that zeros really were missing data. Not actual zeros things like that. That’s not your software, then that’s the data. So there’s data testing and then there’s like software testing, I guess.
00:24:17 Yeah, very much.
00:24:19 I think data scientists that’s a very loose definite I don’t know what that exactly is. But if I’m just handed a data set and I have to provide some sort of digested summary of that to somebody, I mean, scientists do that, but data scientists do that as well, then I think the data clean up a lot of times is one of the biggest parts of that is getting the data in shape, I guess, and being very careful for the areas that you’re talking about, which we’ve seen for some reason, some of our software or people would just enter thinking that’s a good oh, that’s obvious. Well, it’s obvious in an Excel spreadsheet when you’re typing it in. But subsequently, when people are analyzing it, unless you physically look or you plot or you have things that flag values that are nonsense. I also have heard of people who have included those types of data.
00:25:04 I don’t know if it’s real or not, but there’s a story of somebody having a null license plate.
00:25:08 Oh, yeah.
00:25:09 And then they get like all these tickets sent to them or something.
00:25:14 Yeah. So as far as where to get the test for data science, I obviously don’t know that we’re talking about test driven development. Do you feel like the hard part about the whole beginner thing of like maybe it’s easier for beginners is that testing little tiny pieces is really hard. So if you tell somebody this model of you can’t even write any software until you write a test. Well, I don’t know where to start then. Well, you write the test, assuming it’s there and it’s going to fail first, and then you have to write a little bit, you can write the interface, and then you go back and write what it’s doing that back and forth. That doesn’t work for me very well. But that sort of context switch constantly.
00:25:53 I don’t know how that would really help a beginner because I’m a beginner and at least the testing side of things.
00:25:58 I have enjoyed it in a few ways. That first doing the testing in that way. Like yourself, I’m sure that we both have enough experience that if we have a project that we’re kind of familiar of the topic, we can kind of dive in. And we almost have already a framework in our head of how this is going to end up looking like. And so I guess we can jump in at a low level, at a high level, and it’s pretty easy to see where this is going. But as a beginner or in an unfamiliar territory, I guess sometimes it just seems like such a daunting task of I want to process this data. That’s a huge topic. And where do I dive into that if you break it down? And this is how I do my work is that I have a science project. Sounds like I’m in high school and doing a science project. I do science and I want to do a study. Well, that’s too big. So what is the next step? And it’s not just what are the steps? Because if you list out all the steps, you’re going to get overwhelmed and you won’t think of them all. What’s the next doable action, because some things I can’t do until I have ethics. Okay. Well, that’s on my list first, and then what do I need? Ethics. Okay. And during that time I might need and so if you think of it that way, from a testing point of view, it forces your brain to think, I can’t see how I’m going to implement this very complicated thing because I’m just a beginner in coding and testing, or it’s a new framework for me. But I do know that, well, the person needs to I don’t do web frameworks, but the person does need to log in. And if they can’t remember their credentials, I need to remind them via email. Okay, well, that’s a small bit. Yeah. That’s kind of Kent talks about it a bit is that it forces you to think into smaller chunks of the problem. And for that, it helped me and the flow of it. What’s nice about it is it gives you a bit of confidence that and what was interesting, Kent talked about it, and I kind of didn’t believe him because I thought he was just trying to sell it a little bit, saying, look, basically, I’ve ran this whole example that he has in his book, and basically there’s only one or two tests that I wrote. That when I wrote a solution for it, I was surprised because it still failed. And what I found was that it’s true that if I had this kind of write the test, then write the code. A lot of times the code got to the point where I was confident that when I hit that button again, my test was going to pass. I kind of knew where I was going.
00:28:14 Oh, yeah.
00:28:15 And it almost made me smile. I was like, it actually did do what I expected versus the other way around when I’m just coding sometimes more often than not, I’m doing kind of like a smoke test on my own, just trying to look at an output or just see what does this give me. I see error codes, trace back messages, track back messages a lot more that way because with the testing first, the code that I wrote was so suited for a given test, just seemed to work more. And then it was like, okay, come back. I write a little bit. I broke an old test, something that was like three tests ago. Then I have to be I can just undo what I just did and then rethink it. It creates some safety there. Yeah. So I can see it as useful. And I guess part of what, from Kent Beck’s point of view is that it’s a personality thing.
00:29:04 He might be joking about when he says it, but he says he’s very insecure and he kind of condescends himself into thinking that he’s not a good coder and he needs to work in these little tiny bits and the stress that comes by living with a bunch of failing tests or a lot of tests that you don’t know whether it’s currently working. And personally, I guess I am a bit like that, that I like the security blanket of knowing that I’m doing things in small bits. And what I saw for science, obviously, it doesn’t work for everything. But if you do test first, you’re going to have tests at the end, you’re going to end up with a test suite. And Interestingly, I found myself deleting tests as I went. And so your point is true that you start by doing very small tests early on, but as the system grows, I realize what this test becomes redundant because I’m now at a higher level of abstraction and I’m now at the functionality level of my code rather than at the implementation. And so I kind of got rid of some tests and I’m like, in a sense it felt bad. I’m like, should I be deleting a test? Am I allowed? And obviously I’m allowed it’s my own test. Right. So that’s a learning thing that comes with a bit of experience, I think.
00:30:09 Yeah.
00:30:10 But then at the end, for a scientist to say, I have a test suite for the code that generates these results, that’s pretty good for me. And I think you’ve had a few guests on that are scientists. And a lot of times it’s hard to test because we don’t actually know the answer, that you don’t know the that’s true in a sense, but you can test all the bits of your code that lead to that answer. And if all those are tested and do what you expect, I would have a lot more confidence in the answer that’s generated.
00:30:40 Well, awesome. I’m glad it works for you. That’s great. I like to have these end to end things, but when you talk about like inputs and stuff, I don’t hardly ever test user interfaces. So I usually have a layer underneath, like either an API layer or something underneath, like the actual input. I try to make interact and that’s just my style. I like to have interaction with the user, be something that’s like that bit of the code is so small that I’m okay with manually testing it. And then I pass off all logic. So there should be no if statements in my user interface code. All logic is passed off to an API or some other layer. That way I can just test through there. And it’s not like, well, how do I get an input? I pass input to a function, that’s how.
00:31:29 But the other bit. So I like entire system tests that are not quite the entire system, but up to just right underneath the user interface. And then the other part that I think would really be helpful for scientists is to start with algorithmic chunks. So there’s going to be pieces like algorithms or functions that I know the process because I’ve done it. Also, I know this is supposed to work for a wide range of data input and maybe there’s like four or five different parameters, maybe even only two or three.
00:32:03 But we’re going to start with my guess how the algorithm is going to work and I’m going to throw some data into it. So this algorithmic functions to say it’s just really translating inputs to some outputs a functional thing like a mathematical function. Those are great places for tests.
00:32:22 And I always find problems. Those are things that I’m focused on. The solution it’s to get it right. You’ve kind of got to get into the zone and really understand the numbers and make sure it’s working right. And then you think about and then you can start thinking about the corner cases. What are my boundaries? It’s supposed to handle these ranges of numbers, so I better check the top and the bottom number. And then what happens if I go outside? Is it going to break right? And then I might tweak the software to handle those corner cases, but I’ll forget why I put those tweaks in place. So having the tests around those are great because somebody that comes by later and just sees this black box of code that is kind of hard to read possibly. And the variables might suck. The names might be bad. But if you’ve got tests around it, then if they go, oh, this if statement seems weird. Why are we tested for that case? That’ll never happen. Take that out and one of the tests will fail. And then that maintenance part of complicated parts.
00:33:23 Kent talks about this as well, of not testing obvious things. So initializer functions or something like that, we’re just saving well, if I create a class and then with these input, I can verify that all the inputs get stored in the right place. You kind of can look at your code and make sure that that’s doing that. So I don’t test that stuff either. But the notion of like especially algorithmic stuff and especially things that only one person on the team really understands, those are great places to throw tests around.
00:33:55 Yeah. I guess a thinking that I’ve been having from this and every time what you’re saying makes total sense. And then when I look at some and it’s not all but some of the science and scientists, graduate students, undergraduate students, are some of the biggest workforce that we have. And pretty much at least in my field, almost none of them are computer scientists. Few of them have ever coded before.
00:34:21 And so some of that thought process that you have an almost integrity and what you’re bringing to your code is just innate because of your training and your history.
00:34:32 And I guess as people who a lot of people say, well, look, it’s getting more and more difficult as a new scientist. I have to come in, I have to learn all these crazy DNA type techniques and then pipetting or in our case, all these lab things that we do. And then on top of that, I have to write and on top of that, I have to do this. And now I have to use code. It’s just too much. And so basically there was a person, I think that was pointed to an academic kind of came up with a way to say, well, at least let’s create a whole framework that tests people’s software that it runs.
00:35:08 And I’m like, okay, that’s cool. I mean, obviously it’ll run because the person likely would not publish it or put it out there. But I guess what I have is behind the scenes is it actually doing the right thing. That’s a different question. And the only person who can do that is probably the person who wrote the code, who would go in and is it test and code of the chunks that you’re talking about, the algorithms correct. Yeah. And without that, that’s kind of where I don’t know what the percentage is, but a certain and I’d say it’s probably over 50%, just as a guess is not tested at all in the way that we’re talking about. And I find the more I think about that, there’s this move towards open science and transparency.
00:35:47 And yes, putting out the code would be a start, but it would take somebody who understands that code. And because we don’t have the right training, a lot of times some of the code is a bit of a mess. It’s hard to interpret, it’s hard to read through. And so that makes it difficult. But if the person, even if it was hard code, if they had a test suite, I’d already know that. Okay, well, they’ve implemented things that are tested, and it does at least what they were expecting already. That’s a bit more confidence that I would have. But right now, we’re in a place where most of the code isn’t tested, and I have enough experience to know that I’m going to make mistakes every day that I could. And that even when something passes that my code just doesn’t throw out an error, it doesn’t mean it’s right. And so I always work with the I’m an idiot kind of mindset, and I have to just protect myself against myself.
00:36:38 But if you don’t work with that mindset, you might just be like, okay, move on, move on, move on. And I guess I wonder, how do we marry those things of the more computer science, there’s a lot of experience to learn from computer scientists and the experience they have and scientists who want to apply that knowledge. I just feel that we’re not experts thinking that, oh, great, we’re coding now, but I think we’re doing it.
00:37:05 It’s just not at the right level, especially given that we’re publishing these things and people are building on our discoveries. I think that that expression or that saying of, oh, but it’s too much for a new person to learn to do all of this and almost become a computer scientist during their graduate degree. My mindset is, well, you probably do in a focused area. And I guess coming up with a curriculum and or a way to say, well, these are the bare minimums. And if you write it and that’s where the cookie cutter approach doesn’t work because a lot of people, especially in academia, don’t like to be told what to do and how to do it.
00:37:38 That’s true. Okay. So one of the things I want to bring up is I think of the tests as a measurement for my code. I’m measuring my code with the tests. I’m verifying that it’s a definition of what it’s supposed to do. And I think that we could teach that in like small layers, little bits at a time. But that’s just part of using software. You also have to be writing software. You got to write tests. Kind of like when we teach woodshop in junior high or grade school or wherever people get like wood shop. We teach them how to use tape measures also. And you can kind of get away with building a birdhouse without a tape measure. You can just eyeball it and ballpark it.
00:38:19 But if you make something much larger than a bird house, it isn’t going to work. So teaching people how to use tape measures from the start, I think is a good thing. And so we don’t expect people even if you do learn how to use a tape measure and really do a really good birdhouse, nobody’s going to expect you to build a skyscraper. They’re different skills. So we can build them up together. Because when you’re building a large building, you’ve got to figure out how much stresses on the different walls and things like that, which you don’t have to worry about in small scale, the same as for software. If I’m writing mission critical software for a Mars mission, that’s a different level of test than something for a financial application. But for some people, it’s almost as important if you tell me I have a different balance in my account than it was really there. It really matters to me if I’m using it for my personal finances.
00:39:13 And I guess part of this, as somebody who is perpetually learning and trying to find resources, I think we both kind of acknowledge that that book that I desire isn’t really out there. And whether it be in a here’s a few small projects that we develop in the testing, we will try these tests. It’s kind of like a Kent book, but it doesn’t have to be a TDD book. It just as a how do you test book. And then for scientists or academics, it’s the same thing. That book probably doesn’t even really exist for computer scientists, and it would probably have to be even more adapted for scientists. And so I guess I could see that as one of the challenges is there’s very little out there. And if there is a lot of times it is the specialists in computer scientist theory or something that are thinking about how to but that’s not I just need the tutorial or summary review paper of how to do this. And those kinds of things aren’t there’s people upset who are thinking about this. As a gentleman in Montreal that I’m in contact a little bit on Twitter and really, I know people are thinking about this, but I think we’re the early stages, the next ten years will be very interesting of how this evolves.
00:40:26 Yeah. Do you use version control? But it takes more time?
00:40:30 Yes, it does. But for those times where I wrote an amazing paragraph of a paper later on, I was told by my supervisor that it was useless. So I deleted it. And then a month later, the supervisor came back and said, oh, yeah, you wrote something about that before. That was actually pretty good. And I didn’t save an old version. I didn’t have anything. I learnt my lesson that first hold onto your copy, and then I used to have that version one, version two, version three, and realizing that’s just inefficient and horrible. So version control is something that once you get burnt enough, you learn to use. But again, it is interesting software Carpentry, which is kind of like a two day. It has a few different flavors to it, but basically it’s an intro to computer science for scientists. They teach the Bash command line, they teach Git and an intro to could be R or Pi. But version control is in there because it’s just they see it as something so fundamental to working with code. And I actually use it for writing some of my papers and stuff because it’s actually a bit more useful than having word track changes. Once you accept that change, if you’re working not in the cloud, those changes are gone.
00:41:40 It’s definitely something that we just need to get testing to that point to where people can obviously see the benefit.
00:41:47 Exactly.
00:41:47 Because you get to the point where version control actually doesn’t take more time.
00:41:52 Very much so, yeah, it’s part of the process.
00:41:55 And the new students, I find that interesting here is talking about this open science stuff. The new students, a lot of them are just like. But isn’t everybody doing this for some of those things, registering your study and being transparent?
00:42:09 No, actually, a lot of the people above you, researchers, senior researchers, it’s not part of our workflow. And so I could see it. What you’re saying is if we do start with the young people, if we just start it that way and they don’t know anything different, they’ll do the normal learning curve regardless if there’s tests or not, and then at some point it will just be part of the process.
00:42:31 But right now, if you implement that at a Journal saying we want your code first, that’s a problem. Most people wouldn’t give it to you or would be horrified to give it to you. Then if you say, oh, and we want your code to be tested, it’s going to take a whole generation for that to become common practice.
00:42:46 Well, I also think that tests and test cases would be really helpful in scientific research because of the notion that you’ve got, like, students helping out as well. Because that’s a cool thing. You can set up a bunch of test cases and say, and by test case, I mean, like, we were talking about the algorithmic thing. You could describe a bunch of inputs and say, for these inputs, I get that output. And then also even for file formats, you could say, okay, this is the file format that’s coming in. What I want it to look like is this, and then give that to somebody else to write. But that could be a test case to make sure that it really does that. And thinking in terms of, okay, whenever I’m converting from one thing to another, I just write the test first of I want this input to look like that output. Even if you just do one test case and then work on that until it works. That’s kind of what you do anyway, right? I mean, you take one input and make work on it until it gets to the right output. And then you could just apply that to all the rest of the inputs and make sure that it’s still looking out. You do a sample of maybe like ten or 20 different files and make sure that it’s working right. And then you can throw it at the thousand files or whatever you’re using. But that might help be able to trust the algorithm that came out. So if you give this define the test cases and you have like a student write the code to do the conversion, you don’t even have to really look at the code. You can just make sure all the test pass. And you may be like, oh, cool, thanks.
00:44:16 Yeah. And I think that’s something I’ve learned just recently, like statistical software. A lot of it is point and click. There’s obviously some coding like R, but a lot of people just point and click. But regardless of the way you do those tests, a lot of times we just run them and just look at the results. And the first time we ever run the tests are on actual data. Well, now I’ve gotten into the habit of actually simulating data to put in to see what’s coming out, and I simulate different effects thinking of if this manipulation didn’t have an effect or if it had an interacting type effect, I expect this test to spit out this. So now, depending on I kind of generate known effects fake, because I’m just making up the data. But to make sure that this test is doing what I think. So I do that for statistics, and I guess it totally makes sense to think about it.
00:45:07 So that’s more of an end to end testing that you’re talking about is, well, maybe have a few tests at the big chunks of algorithms, but then also saying rather than putting in this data, that whatever comes out of it, I can never test it because it’s going to be new and novel and nobody knows what the answer is. And that’s what the science is about. But let me pump in some data that I’ve made and I know what should come out. And then if that’s correct, well, then that gives me a bit of confidence, and we have pressures to keep our papers short. We don’t want this, that and the other. But I think some of these steps are so essential to the process, and some people just don’t. A lot of people read a paper and they focus on what’s the conclusion, what’s the premise, what’s the hypothesis? Most of the time I don’t start there. I start at the low level stuff. How many people do you have? Did you do it properly? What kind of analysis did you do and build up to the point of, okay, now I can actually trust the conclusion that you’re drawing, but if along the way there’s something that just sounds fishy to me or just they missed the boat, well, there’s no point in me reading further because I think that probably there’s something that needs to be addressed lower down. But a lot of people who don’t have that experience at the dealing with all the nitty gritty, especially on the coding side, but also just the day to day running experiments over and over and over. Once you’re a bit higher up, you’re oftentimes removed from the lab. And so you kind of do float up at that. More theoretical. Oh, this is interesting. And you almost skim over some of the details versus I’m much more of a you got to convince me that it’s even worth me getting to your main conclusion and research.
00:46:40 The paper is the end of the project, but it isn’t.
00:46:45 That will have an effect on research in the future. It will have an effect on policy changes.
00:46:50 It has huge effects on society. It can. Some of them don’t, but some can. I don’t know. I think we have to get to the point where everybody thinks it’s worthwhile to make sure their software is correct for us to trust the analysis. Like you said, if somebody says I went out through the entire country and I interviewed three people, well, I’m not going to trust anything. They could just be three weirdos.
00:47:16 That’s not enough. So we need the same sort of rigor of like, okay, well, I went through the effort of collecting the data from enough samples so that it’s useful. I also trust my sampling input. Like, I trust my electronics. I trust the whole process of doing it. I trust that it was calibrated. I trust all that sort of stuff.
00:47:36 Well, we have to have that same sort of trust in the software to translate and make those decisions. I think it’s important, but I think that tester and development is a great tool. But also the reason why I’m backing off isn’t because it’s bad, it’s because I really don’t want scientific researchers and everybody around that to look at it and go, oh, I’ve just made writing software three times longer, and I want it to be like a little bit longer at first and then like version control. I want it to be a little harder at first, but not a ton harder. But then have the payoff be such that they’re going to never stop you doing it because of that one time where, man, if I hadn’t tested that, I would have totally reported that paper wrong. So it only takes once for somebody to save somebody’s hide where they’ll be like, oh, I’m always going to test for now. So I don’t know. It is a teaching thing. We have to get it into the schools. It will get there. But I don’t know if we really define what test driven development and all the benefits and downfalls are, but I like the discussion about it’s a different mindset, and it’s a different problem in the research and science space than it is in. I’m building software to sell space.
00:48:47 Oh, that’s perfect. Thank you for the every time I talk to you last time, this time I’ll go away with this thing some more. And I would say, watch this space. I think that there’s people who are thinking about this, and it is as you kind of just said, it’s a different problem. It’s not unique. There’s lessons to be learned. But I think that moving forward, we might have to adapt. And I think also resources. I think there just needs to be a little bit more out there. That’s approachable for people who are just non experts.
00:49:15 Have you thought about writing a book?
00:49:17 I have about my life, which will be very short and boring, but not about testing? No. I guess the pressure is always the more we have to publish more, we have to do this more. Everything is more. It’s very interesting in science. There’s just more journals, there’s more papers out every year. It’s just this requirement. And so for me to slow down and take the time. However, there’s a lot of books out there by academics that are kind of chapter based in terms of everybody tackles a chapter. And so maybe that’s a good idea. That because different fields do. I wouldn’t know how to do stuff in genetics, for example. I wouldn’t know how to do stuff in very different fields. Maybe a few introductory chapters. And then from there have people kind of experts who are writing for the beginning of that field. That would be a nice bottle to maybe try out.
00:50:05 Yeah. Or even like we can have a general terms. What are all the terms mean? Like you said, end to end and things like that. What does all that mean?
00:50:12 Yeah. Well, that means you’re going to have to be an author on the book because I don’t know what that is and then collaborate with different I mean.
00:50:19 It could be a collaboration, a series of books, different collaboration with different researchers to target what is the basic level of software testing mean for biologists? What does it mean for geneticists? What does it mean for astrophysics, all those sorts of things? They’re going to be different, right?
00:50:37 Yeah. And do you think just I hear a lot that testing still nowadays for some companies or some workplaces, it’s still a bit of a telling the managers that, oh, here’s this and the testing as well. And some people still have to almost argue and fight for the fact that they are one and the same. The code is the whole thing. It’s the code and the tests together.
00:50:57 I could see that in research, it’s going to be the same fight that for me. For example, if I went ahead and did this project and I worked on developing testing frameworks, educational material, and to improve testing and science, it’s definitely not going to be I’m doing a great thing and it’s going to have great impact. It’s going to be and a lot of people will see it as just too much work, probably not useful. We don’t need that kind of stuff. So I think we might face the same struggles that computer science has. And from what I understand, are still having in some places. So obviously, that’s not going to stop me. That’s not the type of person I am. But it is kind of I can see that there’ll be some pushback in terms of why are you even doing this? You’re kind of wasting your time.
00:51:41 Yeah, that happens. I mean, I don’t have a good view of what software as a whole is doing. My microcosm is high end test equipment for electronics, and a good chunk of the developers on it are not trained software developers. They are electrical engineers. Physicists got into this role because they were good at that part of it. So there’s a mix there. But there’s truth to the fact that when people pay for software, they don’t pay for the test, they don’t get the test, they get the working application.
00:52:12 So it has to be worth it to do it at the same time.
00:52:16 Well, as a final note, then, I guess, given what you’ve mentioned and if anybody, any of your listeners know of anybody or themselves are interested from a scientific point of view, to move forward with this in terms of communication and then moving towards something that might be a book or anything like that, feel free to reach out to me. As I mentioned before, last time was at Martinharu.
00:52:37 They’ll be in the show notes and stuff. But yeah, and obviously for yourself, I’ll keep on listening because I quite enjoy listening to your podcast and the pie bites as well. Python bites.
00:52:47 Thanks.
00:52:48 Yeah. So just keep up the good work and thank you very much for having me on and tackling this issue of testing and science. I think it’s been very interesting and I’ve learnt a lot so thank you.
00:52:57 Thank you and we’ll keep in touch.
00:52:59 All right.
00:52:59 Perfect.
00:52:59 Thanks, Brian.
00:53:03 Thank you, Martin. I love this discussion and I’d love to have more. I don’t think I would ever grow tired of different discussions with people and their experiences with testing in the real world. If you’d like to discuss your testing experience or add to the conversation, please reach out to me. Thank you Lenode for sponsoring the show. If it runs on Linux, it runs on Lenode. Try Lenode with a $100 credit by going to Lenode comtestincode that link is in the show notes of course at testandcode.com. 144 thank you also to Patreon supporters join them at testandcode.com support that’s all for now. Now go out and test something.