What started as a twitter disagreement carries over into this civil discussion of software testing. Brian and Paul discuss testing practices such as the testing pyramid, TDD, unit testing, system testing, and balancing test effort.


Transcript for episode 31 of the Test & Code Podcast

This transcript starts as an auto generated transcript.
PRs welcome if you want to help fix any errors.


It’s not uncommon to see arguments on Twitter, but one of the things that I think is a little unusual is to have the two people that respect each other get into an argument on Twitter and give each other a call and talk about things. That’s what happened with Paul Merrill and myself. We got into a discussion about test driven development and the Testing Pyramid, and we realized that both of us had good points.

But we disagreed with each other.

So we called each other up. Actually, this was Paul’s doing. Paul told me that he’d like to have me on his podcast, and so I went on his podcast was called Reflection as a Service. He no longer does that, but it’s a really good discussion about testing, and I think it’s worth listening to. So I’m going to reair parts of it on this episode.

But first, let me thank Nerd Lettering for sponsoring the show. Nerd Lettering has awesome Python themed Tshirts and hoodies, as well as mugs and mouse pads. If you go to the show notes for this episode at testandcode.com, you’ll be able to grab the discount code for 15% off. Now let’s hear more about this Twitter fight.

Welcome to Test and Code, a podcast about software development and software testing.

So, Brian, why don’t we get to it? Look, here’s what happened. Brian and I were talking over Twitter, and every once in a while I’ll post something and somebody will have thoughts about it. And I like interacting with people in constructive ways about all that kind of stuff. And Brian’s always good to do that. And so I really appreciate your ideas on a few things. One was there was a post that I made from someone else. So I tweeted something that someone else had written and said, you know, I have some different thoughts about this, and I think it was it the Testing pyramid.

Yeah. I think the article was something about the Testing Pyramid.

And I think your comment was, I’m so sick of the Testing Pyramid. Was that right?

Yes. I think it’s done more harm than good to our community.

Why do you think that?

Well, so essentially we’re telling people the most important thing is your unit test.

And above that, there’s some integration tests, but we’re not really going to define what that is.

But it’s something other than unit test. And then at the system level, there’s user interface. Test and code are hard, so only have a couple of those and don’t do very much.

I disagree with all of those statements, but the end result is a lot of developers here.

Okay.

Just write unit test and code about everything else. And that’s what I see. I’ve seen it in implementation in groups that I’ve been in, and I’ve seen it in other teams. I’ve talked to other people that say that they have an incredible continuous integration system, but it’s all based on unit tests. And then they send their product over to the QA team who does manual testing to determine whether or not they can release. And I think we’ve got automated testing tools that can do high level tests just fine. I don’t understand why we’re putting so much emphasis on unit tests.

Got you. So what is it about the pyramid that makes so you say the importance is placed on the unit test? Is that because it’s bigger or because that’s the foundation of the pyramid or because of something that you’ve read?

Where is that?

Well, I’m guessing that the pyramid shape is for some reason they aren’t just bubbles that are next to each other, so it’s larger than the others that I assume that means you’re supposed to write more of them.

And I don’t get why it’s a foundation because they’re very different things.

When I think of a unit test, I think of somebody, and I’m not opposed to unit tests. I like them. I like being able to say I’m going to write a function, and mentally I know what it’s supposed to do, so I can write some tests to verify that it does that, and that’s helpful to me. But that doesn’t tell me whether or not I’m meeting the requirements of the product or not. It’s just focused on this little function.

I don’t know how people tie unit test requirements to the requirements of their system.

I think those are all great points, and I probably need to go back and read Mike Cohn’s original writing about that. I did see where he revised or reiterated his initial thoughts on that, or kind of clarified his initial thoughts a while back on it in a blog post. And I’ll try to find that for a link for this show notes.

But to me, when I’ve seen this work really well and this is what I love so much about talking to other people is that I can’t possibly have the experience that Brian has. Brian, I can’t have your experience. I can’t go back and live your life and you can’t go back and live mine. When we talk to each other, we start exchanging these ideas of where you’re coming from with your experience and where mine is, and maybe it opens up new avenues. And so to me, when I look at that pyramid, I think in terms almost in volume, exclusively almost exclusively in volume.

But I do see kind of a foundational structural part to it as well, like you’re talking about. So in terms of volume, I see the width of the pyramid as being about volume, and I could completely see where one could look at the volume of that part of the pyramid and attach importance to it. I could totally understand if we thought about it in terms of structure and thinking. The fundamental part of it is the unit test. But the way that I look at it, the way that I understand it and the way that I practiced it that’s worked has been strictly in terms of volume of unit tests being high for a number of reasons. But then in the structural part of it, I see unit test is allowing you to build a testable product, one that’s more loosely coupled and more tightly cohesive. Right.

And you end up with a more testable product that way. And I find that when those unit tests are written early, we also end up with a lot of the utilities that you need to do those integration tests that you say are not well defined. And I think you’re right. I think there’s been probably less written about service level or integration level test than either the other two.

The UI part I strictly look at almost strictly as a volume issue because they tend to be so brittle and they tend to take so much maintenance over time. I don’t know if you’ve experienced any of that, though, or if your experiences are different.

Yeah, my experiences are completely different. I think if they’re brittle, they’re bad tests or you have bad software.

Okay, yeah, those are good points. I think you’re absolutely right. But for me, client after client that I sit down with, the UI tests have generally been written in such a way that they are brittle. You walk in and everything’s broken or lots of stuff is read in your CI environment or there’s not a CI environment. People are pressing a button to run them, and that’s what they’re calling test automation.

Well, I never use actually, I do hardly any user interface testing, but I think of system level testing as including a system level API.

Okay.

Even if your customers never see the API, there are people that promote and I’m one of them promote what’s called a subcutaneous layer, which is being able to hook into your system underneath the user interface and being able to test at that level. And that’s where all most of my testing is.

I also have a lot of reasons why I want tests in place. Writing tests is effort, and the effort has to have payback and it has to have value. And the thing I want the most is I want confidence that the system is going to work from the user’s perspective.

And if my user is using the user interface or my API and those tests are brittle, then anything my customer does is going to be brittle. I don’t want a system that’s brittle to my user.

So if I’ve got system level tests that are breaking all the time, you can just think of that as customer code that’s breaking all the time.

That’s a problem.

Oh, I agree completely. And I’m not saying that’s the way that UI test or top level test should be. The other thing that was interesting to me and what you said, and I think sometimes these discussions, a lot of times people who have you and I are probably at similar points in a number of years in our career. I’m going to guess we’re probably one of us may have a few years on the other.

I started in 96 out of six.

So you got a few years on me, but not very many.

I think that gap is probably more negligible than it would have been several years ago. But what I find again and again is people with similar amounts of time in a career, they’re coming to very similar practices and very similar philosophies because we’ve experienced very similar things, but we’ve experienced them in different ways. And sometimes we have different vocabulary and things like that. So when you say the word subcutaneous, I’ve never heard that word. I don’t have the Sat score for that word.

Okay. Well, I mean, like, for instance, if you got an injection, it’s just under the skin.

Okay. All right. So to me, those tests that you’re talking about, there would be integration tests because you’re working with a set of code where you’ve got more than one class working together, you’ve got more than one method working together, or you may be interacting, it sounds like with a data source in this case or lots of other systems. You may have third party applications that are called APIs that are called. Is that true, some of those tests?

Well, in my case, usually, like at work, it’s the same interface the customer is using most of the time.

Okay.

So we do have a usually it’s a test equipment environment that’s sold into production lines. These are not done by people. They’re done by other computers controlling them. So there’s everything that we can do, you can do from a remote machine. Got you.

We do have a user interface, and you can see the pictures and do the things with push buttons and stuff, but that’s in R and D environment, not in a production line or when people are just debugging things, they’ll use that. But there’s a way to design systems such that the user interface, the graphical user interface and the remote user interface come together really high up. They’re really high in the protocol stack so that you can test almost the entire system except for the graphical user interface part.

And get the most value out of that. Right?

Yeah. And they’re way easier to automate.

Yeah, I hear you. And there were several thoughts that I had when you were talking about that.

It’s a shame that I’ve seen the pyramid drawn with E to E at the top as opposed to UI. But the user interface in your case is not a graphical one. It sounds to me like you could almost substitute the layer that you’re at as the user interface that is the customer that’s going to call in. They’re going to call in that way at the level that you’re at, right?

Yeah. Well, I mean, you can take like a lot of even web applications are done the same way.

Like, for instance, Twitter. Twitter has a very extensive interface. That’s not what most people use. I mean, I’m sure they’re not testing Twitter from a cell phone application or something.

Yeah. And I’m seeing more and more web apps being tested. Like you’re talking about where we’re calling a Restful API rather than working with user interface and doing very rudimentary tests on the user interface to make sure that it hasn’t changed or that nothing’s gone crazy on it since the last time we touched it because you can get so much of the functionality there. Yeah, go ahead.

One more thing. One of the things I’ve noticed about you and I I is think there is a similar experience level. And one of the things that fascinated me is I think there was a similar reading in history. I think we’ve read all the same books and yet came up with different conclusions. Well, that’s interesting.

Well, I Act a lot more like I Act like I read a lot more than I do. I’m a really slow reader. So I have this deep appreciation for every single book that I read.

But I do try to read is I do try to read a lot. I read a lot of blog posts now more than in the past. But I think you’re probably right, because it sounds like we’re coming from very similar places. To me, that conversation and then another conversation that we had on Twitter where we were talking about, I believe it was TDD, wasn’t it?

Oh, maybe. Or Scrum. I like to bash both of those.

Right.

I don’t remember the exact part of that conversation. We’ll have to have a link on the show notes as well for this.

Do you remember that? Can you help me out? I should have prepared better for the show. Maybe I can look it up.

I can’t remember.

Yeah. Well, I think basically you were saying just now you’re bashing TDD, is that right?

Yes.

The way it’s taught, the way a lot of people teach it. I do disagree. So I actually love test driven development. It was life changing for me, but I was in a situation where it was very difficult to test units. So I thought of the unit in TDD as a functional unit, a functional piece of functionality, and then the rest of it worked just fine, just like that.

And that’s kind of how I’ve gone. And then trying to find out whether I’m doing it wrong or not, I’ve gone back to a lot of writings from Kent Back and recordings from him in interviews, and he does pretty much say his original philosophy and this comes from XP as well was a unit was never I mean, we define a unit as, like, just a function or just a class that was never part of the original writings that came by other people like Bob Martin and others.

Kent Beck believes at least I’m speaking for him, which I probably shouldn’t. But what I get from that is you can think of the tests in Test Driven Development as whatever layer makes sense to teach you information about your software and for you to have confidence in your software and to help you make the next step in implementation.

I think you’re right. I think it could be that sometimes I make sources together on some of that. I know that at the same time that I was learning about Extreme programming and learning about TDD back in the early 2000s and reading those books, I was also working with somebody that’s probably still to say the biggest impact on my ability to program.

And so it was probably his ideas about what a unit was and whatever, or it may very well have been his ideas that I’m kind of conflating with some other ones when I talk about this stuff. I mean, to me, I’ve always thought of the unit as being one class or one method or the smallest unit that you can and then knocking out everything else around it. And that being what a unit test is. And maybe there is some flexibility I should try to consider for my own benefit.

Yeah. Well, there was that thing with is TDD from Ruby on Rails guy, and there was a discussion that he talked about TDD being dead, not really, but that he really likes tests, but he was having trouble with the design issues, with people trying to follow this functional, very small scale mocked Test Driven development and saw huge problems in the architecture because of it.

There was a discussion with a handful of people, including Kent and others on this, and they’re pretty much all saying the same thing is that they never tried to push that in the first place.

Mocks weren’t part of XP. I never read about mocks when I was reading about testing during XP.

Yeah, I think that certainly was something that I found somewhere else. And at the time, there weren’t that many great tools at the time I was in Java, so we were writing our own mocks and things like that.

But the other thing that I saw was the writing and XP and the early Test Driven development. And even later there’s like a footnote or like fine print that says, by the way, you have to have a QA team to independent testing team testing your stuff also.

And so I usually don’t have an independent testing team. So how do you incorporate testing into your development project so that you correctly test everything without an independent testing team? And answering that question is kind of my pursuit for the last few years of trying to figure out where the right balance is.

And I think that’s so key is looking at the context, looking at where you are and what you need to do for that particular situation. And that’s one of the things that again and again, client after client, I sit down and it’s about where they are today, what their vision is for tomorrow, and what I can add to that, how we can close that gap together. And much of that comes down to pragmatism about what skills do we have on hand? What tools do we have on hand? How far can we go within reason? And how do we make sure that we assess risk properly and mitigate it in such a way that it makes the customer happy? Which is a whole different question than how do I add the most quality to this particular product or how do I practice a particular practice? And I guess for me, in these conversations with you, it seems like there’s so much wisdom and so much pragmatism to the way that you approach things. And I really admire that, and I think highly of it.

And yet when I walk away from some of these conversations on Twitter and I’ll just say this, I feel like you’re attacking the model, the model of TDD or the model of whatever. But then when I talk to you in more length because 140 characters isn’t enough to do anything, right. Yeah, but when I talk to you in person and I know that there’s more behind that. Like, I know that there’s more behind what you’re saying, and I want to know what it is. And that’s why one of the reasons I really wanted to talk to you is that the model is almost like a tool in the belt to me.

A lot of my inspiration came from Kent Beck. And one of the things that he said and I think a lot of the people that push models and whatever fall back on is this idea that people new to a practice need to have rules, how to practice it. And then when they get experience, they’ll know when to break the rules.

It’s the driver’s model, right? The driver’s model is what describes what you’re talking about.

Okay.

But I don’t see a lot of people changing later. I just see people just following the rule. And then you see also consultants come in, like, for instance, there’s a word called scrum, but scrum but is a way for a consultant to insult you and your team for not practicing scrum correctly, as far as I can tell. And if you read the original stuff for scrum, there is no correct way to do it. There’s a way to start doing it, and then you gradually change the practice to meet the needs of your team.

And a lot of this stuff comes from the Agile manifesto and the agile practices.

But again, there’s not an agile manifesto. There’s a manifesto for agile software development. So if your practices are increasing your agility, then they are agile.

If they’re not, like, for instance, putting in a plethora of unit tests with marks I don’t see how that can make it easy for me to change my mind and redesign a subsystem quickly in an afternoon. If I’ve got 7500 unit tests that I have to change.

I’ve seen exactly the opposite, that it’s only because those are there, that I can refactor it and change it in such a way that I know that the existing functionality continues to work or doesn’t.

Yeah, well, does it test the original functionality?

Those tests would describe what we intended the software to do originally. And if I’m going to do a refactoring, which to me means not changing the functionality, just changing the organization of the code, then I would like to know that the original intent is still there.

The other part is if I need to change it because of functional reasons and I need to change the functionality, then those tests give me a baseline to know what I’ve changed intentionally or what I may have changed unintentionally.

Well, I’ll give you like a quick example. Like let’s say I’ve got a piece of code that uses the triangle distribution statistical model and somebody wrote a function to return a random number out of a triangular distribution. And then somebody else came by and then there were tests written for it and somebody else came by and went, oh, you know, that’s already in the random module. Let’s just use the built in one and change the code.

I want to write my test such that I don’t have to change any tests for that change.

Well, what if it did something different then? Oh, I see what you’re saying to me. I would have multiple layers of the test and the unit test would say, hey look, something changed and we have to do something different with the structure of the code because we changed a library or whatever. Hopefully the functionality is still the same and we have the other layers within that test pyramid to tell us if something at a higher level has changed, like a functional level test.

But if my promises to the outside world we’re all still met.

Yes, you’ll be fine.

That’s right, I should be fine without and I shouldn’t have to change anything. The tests that are change detectors or tests that tell me that I changed my implementation, I don’t see any value in those. I see actual harm and I think they get in the way.

Yeah. And so my experience on that has been different too. So I see design emerging out of the test driven test first philosophy, test driven development philosophy. So a lot of the design decisions that I make when I’m writing code are because I’m trying to find a way to test the code before I write it and therefore I’ve got a more testable piece of code afterwards and the design ends up being more flexible later on.

In most cases, in my experience, plenty of times I get it wrong and I have to go back later and figure out the better way to do it. But that’s been my experience.

Thinking that a testing philosophy can help you design better software.

I don’t think that’s been proven.

But that’s my experience.

It may not be proven, but it’s been my experience, and my experience is all I really had to work on. Right.

But that’s what a lot of proponents of test driven development push and mock us unit test is that if you do this, your end software will be better, and it seems like a lot of work for no proof behind it. And I guess I’m not willing to do that work.

I’d rather put the effort where I really want to make my promises. I want to make my promises to the customer first and foremost. So those tests need to be designed very well, and after that, the promises get weaker as I go down, like a package within my software that I’m the only user of that package. That’s probably pretty weak. A weak promise, and within the functions within a module that work together, those are even weaker. I still want them to work. But if I change my mind, it doesn’t need to change a whole bunch of outside test and code strength of the promises and the repercussions of breaking something.

I think those need to be taken into consideration when deciding where you’re going to put the effort in writing tests.

So with each of these, like I hear you, it sounds like there are a lot of things that you’re taken out of this model of test driven development that you tend to use, but then you dislike the model in general because of that. Is that what I’m hearing?

I just like the model that as it’s taught from the bottom up testing, I think it should be the top down. I think we should start with user level tests and break those down where that isn’t giving us enough information to write the next piece of code, then write us a lower level test.

And you can say, well, I’m going to depend on this other library to do that work for me, and fine, but write a test for that, make sure that it’s going to pass if it works and it’s going to fail when it doesn’t work when I’m writing tests in that manner.

And I want tests at a system level and integration level to help development also.

And it can’t help development if it’s written afterwards because development is done, I think.

But it sounds like you’re also forgive me, but it sounds like ordering is important to you here.

Just because one wants to emphasize unit test doesn’t mean that they have to be written first or last. I know that I talked about using unit test to drive design, but that also doesn’t mean that we have to do it from the bottom up. We could also do that from the top down.

Oh, yeah, sure. I guess the thing that I really want to design well with tests is my APIs.

So the APIs to the customer, the APIs between our modules, those APIs are more important than whether or not I like the design.

Right.

I’m well aware that I’m not very good at writing software.

Who is? Right?

You think you’re so good while you’re writing, but then you come back like six months later. It’s terrible crap that you wrote.

Right.

I want to have enough leeway in my system to keep it agile so that I can rewrite chunks of it when I know more about the problem. So the person that at the end of a project, after they’ve written all of it is way more qualified to write that software than the person that’s. Sure.

Absolutely.

So there’s a chicken and egg problem. Also.

If I can’t change the design without changing the test, how do I know that I don’t break anything if I’m changing both at the same time?

That’s my issue with change detector tests. I should be able to change the software and be able to run the tests, and it still works and have confidence that I can ship it. If I’ve got a test that just says, oh, you redesigned this module or this function to use a different library function instead of the one I thought you were going to call, who cares? I’m not going to delay shipment. That’s another question of if I have a failing test that I cannot get to fail from the system level, but it fails at the unit test level, maybe that test is too picky.

For instance, corner cases. If I’m testing a function to make sure that negative numbers don’t make it crash, but in the system together, that function will never get called with negative numbers, then I’m over designing that little piece of functionality.

Yeah, and I’ve run into that conversation many times.

I guess I have two thoughts about that. So, number one, just because a user interface can’t do that right now doesn’t mean that it won’t always be able to do that. Right. So if we just allow negative numbers from coming in from whatever this interface is at a higher level, it doesn’t mean that in the future we won’t allow that. So when we do allow that, wouldn’t it be nice to have a check that’s already there? Especially if you can just very easily have a data driven test that just throws in a minus one? Right.

Okay, but how much longer do I take to write that functionality and those tests at the unit level?

My premise was that it doesn’t take long.

But if that number is getting passed through maybe six layers of software before it gets to the bottom level, do all of them have to make sure that they’re dealing with negative numbers correctly?

Well, this is very hypothetical and all that. I totally understand where you’re coming from. I guess what I’ll say is just you’re a pragmatist, and I’m a pragmatist. I believe it sounds like you are. And for me, in my experience, what I have seen that is incredibly helpful is if I have a suite of unit tests, for instance, that are written at the lowest level and they all pass, and then I have an integration level where everything passes, and then I have a UI level or interface level where a corner case like you’re talking about fails. I don’t know where that issue is. It could be anywhere within my code base. If, on the other hand, I have a UI level that does the very same test as the integration test and that particular corner case is missing in the unit level test, then I know immediately what particular part of the code that the issue is in. And I found that very helpful.

So having different tests at different levels to me, or having the same test at different levels or more pickyou and more detailed tests at the lower levels tells me at least gives me an idea of where it is and that it takes less time to find the problem.

Sure, I can totally get that.

It’s just not an argument that I usually run into.

Occasionally we change everything at the same time, but that doesn’t happen very often. Like, for instance, let’s say we upgraded to four levels of Visual Studio compiler or something, or upgraded from Python Two Seven to Python 36 or something, or some wholesale. You’re changing everything, you’ve changed the chipset or you’ve changed the operating system and you don’t know what’s going to work and what’s going to not work in those cases. It would be really great to have a lot of fine grain tests to tell you exactly where the problems are.

But in my experience, the right test to tell you exactly where the problem is isn’t going to be one that’s going to be written.

Nobody’s going to think of that.

But that doesn’t happen very much for everyday testing.

This idea that a test has to tell you exactly where the problem is, I think is ludicrous.

Well, I didn’t suggest that every test has to do that. I suggested that it is helpful when some tests do that.

Yeah, sure. But I have a great way to tell what code broke the test. It’s the code that is different from the last time I did the test and it all passed.

So if you’ve got like a nightly build or continuous integration when you check in some new code and the test failed, I don’t need to guess where that failure is. It’s that code that just got written sometimes.

Not always. I can’t tell you the number of times that I’ve sat down and I hear what you’re saying and it has to be that, right. Logically, it has to be the code that’s changed. Right. But how many times have I sat down and seen the change set and said, it must be in that code. And then I go and look at the code, and it’s not because of that. It’s because of something that is completely outside of that. And, of course, that would indicate that there’s something else wrong in the process. I totally understand that. I totally get that. But many times there is something wrong in the process. Right. Like you have test cases that depend on some type of data that’s persisted somewhere outside of your system that you don’t consider within certain test cases or whatever. But I can’t tell you the number of times that I sat down and thought, this change set will tell me exactly what’s wrong, and it will not. In fact, it’s something that happened three weeks ago. Okay.

Yeah, sure. It has similar failure points as depending on your test to tell you where the failure is. I guess different failure points, but they’re both not perfect, right?

Yeah, but I think these tools can help us identify these problems from time to time. I mean, I think that’s why the models are out there. I don’t have a new model. I don’t have new ideas to come out with in these conversations. I guess what I’m hoping with some of these conversations is that if there’s something new that comes out, maybe it’s something that we can put back out there into the listeners. Maybe there’s a different model that comes out of this. And you mentioned one. I don’t know if you were serious or not, but the column model, what were you?

Well, I’m dead serious. I mean, if the testing column. Yeah, sure. So just write the go ahead and write the testing pyramid, but just pull the pyramid sides out so that they’re straight up and down so they’re parallel.

Got you.

Yeah. So that there’s absolutely no indication from the model of how many tests are supposed to be in each level.

Yeah.

That’s different depending on what kind of software you’re writing.

Like, the software I’m writing has multiple operating systems, multiple languages. You’ve got DSPs and multiple threads. You’ve got inter process communication. You’ve got interaction with the outside user.

There’s no way you can write a unit test that makes sure that everything works. And every time that I’ve spent my evenings and weekends trying to debug something, it’s never because of a single area of the software. It’s because of an interaction of a whole bunch of different things.

Interesting.

Have you ever spent, like, a huge long debugging section that turned out to be just in one function?

Yes.

Oh, okay.

Wow.

You suck.

I don’t think so.

It always comes down to more than one place for you.

I mean, back when we did threading within methods, you never had a case where you were sitting there. And maybe it was because I was younger at the time and didn’t have as much experience or something I’m not going to say that that guy sucks because he’s me several years ago. He just didn’t know as much.

I’m sorry he insulted you.

Oh, no, that’s all right. But I mean, I guess to me, it just seems like there are plenty of times when that kind of thing happens.

Multi threading assumes that there’s multiple things going on at once.

Okay. All right.

So you’re still right.

Because how do you catch a multi threading issue with a unit test?

Okay, so what about calling into a third party? And in my definition, that wouldn’t be a unit test. But if you had a test case of some kind that found that because you’re calling to this third party and the third party can’t handle or the third party application or library can’t handle whatever it is you’re throwing at it.

I’ve spent tons of time trying to figure out a problem that ended up being in one method and a call to someone else who didn’t handle things the way that I expected them to.

Right, exactly. And I think that’s an argument to say that your tests should include calls to third party libraries.

And I’m not saying they shouldn’t. I’m saying that there are some tests that shouldn’t, some tests that shouldn’t. And that finding the right mix of those generally helps you identify where problems are.

Okay.

But this particular argument, you’re saying people spend a lot of time, all problems that take a lot of time are because of multiple things happening. And I was giving you an instance when that wasn’t the case. Yeah.

I’m just saying from my experience, I’ve never spent. Well, I probably never.

My long term memory is very short.

Fair enough. Well, look, this is a really interesting conversation. I think we focus a lot on test pyramid and test driven development, so I might have to title this something related to that. It’s always so much fun getting to know people and hearing about different ideas and different parts of life and ways that people have experienced similar things and just completely different ways from completely different point of views, points of view.

Is there any thought that you want to leave us with in terms of models or any of the ones that we’ve talked about?

Yeah, it’s something that I heard from Michael Kennedy.

He does talk Python to me, and he does Python Bytes with me, but he attributed to Steve Jobs. But I don’t know what the real quote, but essentially it’s something like all the stuff, everything that you’re using, all the software that you’re using has been written by people that are no smarter than you and me.

I think it’s the same with thoughts and everything else, too. Right.

But I think that I want people to put these models up on the same scrutiny. I think that if a model helps you, that’s great. But don’t put it above your own common sense. Think for yourself.

Thanks so much for listening. Visit the show notes at testinggo.com and grab a discount code for nerd lettering.

If you find the show useful, help the show out bye by telling your friends and colleagues about it. Thanks bye.