What tests to you write first when testing a legacy system? Or any system really?

Transcript for episode 37 of the Test & Code Podcast

This transcript starts as an auto generated transcript.
PRs welcome if you want to help fix any errors.

Welcome to Test and Code. What test do you rate first when testing a legacy system? Or really, any system, what test do you write? How do you start testing? Some variation of this question is the most common question I get.  
Of course, it depends on what your application is, what your group looks like, if you’ve got a QA team, if you’re a part of QA team, if you’re part of development; is it a web application or a command line interface application. Of course, it depends, but it depends is lame, of course.  
I’m going to tell you my strategy of how I prefer to attack the problem of putting a test suite in place for an existing system. You can decide for yourself whether this applies to you in your situation or not.  
Really, I think the question of how do you start testing— that comes from possibly an overwhelming feeling of looking at a huge pile of work and not knowing really where to start to get the most value right away. It’s one of those, “a long journey begins with one step" sort of thing.  
So let’s jump in. It’s going to be a long journey, I’m not going to cover everything in this episode of How to Get Started. Well, of course, I am going to cover how to get started but it’s not all of the test strategies. Let me tell you the approach.  
I’m going to make a series of episodes and tell you my approach to testing existing systems. I am going to cover some strategies and I am going to put names on these different strategies so we can refer to them later by name. Most of these aren’t names I made up but that I’ve heard from other people.  
I’m going to explain the strategies in general terms and explain why they’re useful, at least why they’re useful to me. And then discuss how these strategies are used in a concrete example project so you can see it in action.  
I’ve started a project up on Github that we’ll talk about later. Actually, I’m going to do this along with the episodes and so you can see the code as I’m talking about it, as we develop it. 
So today, the strategies we’re going to discuss are:  Dog Fooding Exploratory Testing Tracer Bullet tests Act Like a Customer tests  Manual Procedures and  The Initial Automated tests at two levels.  
So automated tests, of course, that’s what we use most of the time, at least I use a lot of with pytest. There are a lot of other types of testing and I think it’s important to cover those in this episode because if you’re perching the question of, “How do I start automated test suite" there may be existing tests and I think some manual tests are good. So again, dog fooding, exploratory testing, tracer bullet tests, act like a customer tests, manual procedures and Initial Automated tests at two levels. And then we’ll talk about how to apply these to a project.  
Dog fooding is just eating your own dog food. If you have an application that you’re working on that is something that you can use or your team can use on a daily basis, that’s the best situation because you can be your own customers and you can use it, you can see some of the works. 
And actually, it’s really hard to get your head into how an end user is going to use something if you don’t know how to use it yourself. But you can fake it a little bit, if it’s really not something you use every day, you can pretend to be the kind of person that would use it. The example application that I am going to talk about later is something I’m trying to use, I am having issues right now though.   
Exploratory testing goes along with dog fooding, so when you are using an application yourself or your team is, you can keep track of some of the usability stuff and it helps prioritize. Exploratory Testing is just like exploring around the application, trying different things, spinning dials and seeing what breaks.  
How I use it really is just playing around with the application and seeing what the different pieces are. And especially if you have to use an application— it’s going to be hard to just keep saying “the application" in the future. I’ve got a task application that I wrote for the Python testing book and I actually didn’t want to keep saying “tasks” because it has that S-K-S sound, and it’s rather annoying on an audio sometimes. And also if I ever put “multitasking” in, it’s going to be annoying to have “Tasks, multitask within the task”. The application, I don’t want to talk about that much.  
So the application is called Cards, it is as in like index cards. So the Cards application is up on Github under okken/cards.  
Anyway, I’ll talk about that later, so in that one, I’m going to try to use it, and I will see very quickly— I’ve already trying to use this on a day to day basis and it’s getting annoying, there are some features of it that I already want to change, because they’re just hard to use. 
That’s one of the benefits of dog fooding, we will get to automated tests but I think it’s important to consider manual tests too, because they’re part of the testing answer. 
Tracer Bullet Tests. I’ve never really heard it’s called tracer bullet tests but I got the term tracer bullet implementations, or development from the book “Pragmatic Programmers” and it talked about doing a tracer bullet feature is implementing a feature maybe not with all of the bells and whistles, making sure all the corner cases work, but a happy path from the user interface all the way down through the system into the hardware. It’s a way I like to develop applications. It’s also a way I like to develop tests. So, some of my first tests are going to be full tests that exercise everything. 

And then I’ve got Act Like a Customer tests. I think a lot of people think of the end user interface test as these act like a customer tests, they’re kind of cumbersome, they do a lot of things, pretend you’re the end user and go around and change a whole bunch of stuff. It uses a lot of features, the benefit of these is you can have just a handful, one or even two or three and you can come up with a scenario that a user might use your application for and it will test a whole bunch of the system.  
You can cram a lot of functionality in act like a customer test. The benefit of that is you just have a handful of tests that you need to maintain that exercise a huge portion of your system and it can give you a fairly good confidence that your system is working, if they pass. If they fail— completely different story, they don’t have enough granularity to tell you exactly what failed. You can look at the line number of where the test failed and get a lot of information especially if you’re using pytest, of course. But just the name of the test and having it fail doesn’t really tell you much. So these are the things when people talk about the top of a testing pyramid being small, it means these act Like a customer type tests, we don’t need very many of those.  
Manual Procedures. I used to be the kind of person to try to get rid of all manual procedures and move them into automated tests and not have any manual procedures. I think that that is a noble goal, but for new features, as you are developing new features on a product, I still think it’s kind of good to write down some procedures and how to test something as a manual procedure, even like in our example, I’m going to have an act like a customer test be a manual procedure. It does help, I think sometimes to have to write down the steps, it will show off some of the clunky parts of your system, if writing down the steps is really hard and annoying to do, it might be actually annoying for the customer do too. So you learn a lot about your system by having to write down the procedures.  
And then, of course, where we probably really wanted to go with this conversation is— the automated tests. What are the new automated tests, the first automated test that I should put into a system? Of all of the types of, I mean, the dog fooding, exploratory testing, act like a customer test, those all seem like they’re really right before manual test scenarios. But tracer bullet tests can fit either place, you can write down tracer bullet tests as manual procedures or they are great for your initial automated tests in two levels.

So for instance, I’ll just jump ahead to this application. In the Cards application I’ve got a user interface, that’s a command line interface, but your application could have a web interface or something else. I like to have start out and have duplicated tests, I don’t normally like to duplicate too many tests, but trying to do the same thing from the end user interface and from a subcutaneous layer like an API layer or some place that’s easier to test because it’s, let’s face, user interface testing is not as easy as an API test. 
All right, so where we’re at? Let’s just talk about this Cards application. I’ve got this Cards application up on Github now. I spent the last few days, I was going to start this series earlier, but I was playing with all of this stuff, so I’m learning along with you. What I am learning about is I’m learning about how to maintain an open source application and I’m pretending that this is a big thing, but it’s just this little guy, this Cards application, but as I was using a Cookiecutter template and modifying the heck out of it, there were some links to other applications. Let me bring it up from Github, here we go, github.com/okken/cards, it’s got a bunch of stuff in here.  
Actually, it’s amazing if you’ve got an open source application, how many files are around and in the guts of this file, there’s like— I don’t know if there’s a way to count files, but there’s a bunch of files in here already. In the Cards directory, there are two files, there’s a cards db and a command line interface, a CLI. So that’s the main application but there’s a whole bunch of support files. But one of the things that it gets is, that I’ve got it hooked up to read the docs,  documentation because I wanted to explore that. In the application called Pyup that looks at all of my requirements, and let’s make sure that those are up to date, it looks like I’ve got one that’s not up to date right now.  
Pushing the build off to Travis and since this is open source, all of these services are free, which is cool, and it gives a little indicator of which version is available on PyPi. And right now, I’m looking at version 0.1.2 and there’s a master branch and it looks like we have tags. On the tags there’s version 0.1.2, you can get to. That’s the state of the system I am at now. What does it do? It’s just a little to do list. If you type cards, if you have it installed, you can do pip install Cards, and run Cards–help, and it will tell you the commands, you can add things to the list, delete them and list them, stuff like that, it’s a really simple crud application. 

So let’s come for the types of testing we’re doing. Dog fooding. This is a painful application to use right now, hopefully I can get it to be more usable. The thing I get right now that I don’t like already is if I just type cards, it just shows me the help. What I’d really like is to show the list, if I just type Cards it should show me the list of tasks. But right now I have to type cards-list. So maybe I’ll work on that. So dog fooding. I am going to try to use this at work, and keep track of what I’m doing, maybe I’ll try to keep track of what my co-workers are doing and my team, we’ll see.  
Exploratory testing has not much to do around this. In dog fooding you explore a lot.  
And then, tracer bullet tests. In the app I have got two tests so far, in the test directory I’ve got a test_cli, and a test_cards file. And each one of them has a test add, and it is one test and it will add one card to the list, and then try to get it out, use list cards to get out the list of items and then make sure that the thing I added was the thing that’s in it. That’s it. 
I would consider this a tracer bullet test because it’s focused and it goes from top to bottom, it has an actual database that it’s using, There’s nothing mocked, it goes all the way up. I am duplicating, the test does the same procedure, but I’m doing it both through the command line interface and through the API. The command line interface, I’m using click for the command line interface implementation, and so, therefore, it has some testing, a test runner in place with click, this is one of the reasons why I chose click, but you can use pytest and then the click testing tools together, rather nicely to run things and look at the output. If I didn’t have something like that, and sometimes I don’t, I will use subprocess or something.  
Now if this was a web application, I couldn’t use, it wouldn’t be something like the CLI runner that comes with click, but I could use Selenium or something to do the web interface.  
Other types of interfaces— I’d like to know what you guys use, but I don’t do any other types of user interfaces, I test  APIs, I test command line tools and I test web stuff. Right now, I don’t know how to test anything else. I do test hardware but that’s through an API.  
What else? So why am I duplicating these, and why did I pick these to test? So I chose both, the user interface test and an API test as examples, so these are examples for other people when they are developing, or in a job setting, I would want some examples around, so when somebody goes and tries to test some other part of the system, they can see an example of how I did this. And it also gives me a chance to work through it, I don’t want to develop a ton of tests right away. I want to develop these two to get me— I am using the test to try to understand whether or not the system works, and if I don’t have to mock anything— actually I lied about the mocking. So the command line interface does have one monkey patch in it and it is because in the command line interface, the database is stored in your home directory and  I didn’t want to use that so I faked out the home directory and used a temporary directory instead and used monkey patch to allow me to do that. Monkey patch as part of pytest.  
Act like a customer test, I am using this add as my initial tracer bullet test through two layers, through the API and through the command line interface. I’ve got a manual procedure that’s an act like a customer test, and I don’t know if you’ve ever seen a manual procedure written in an open source application, but I stuck one in there. So under the test directory, I’ve got a manual test and act like a customer so I’m trying to come up with some way to use all of the interface, and through the user interface which is a command line interface for us, and use as much of the system as possible. So I’ve got a procedure written down that adds a bunch of stuff, updates the owner for one task, updates the done state updates the summary. And then, delete some stuff, and lists a few times to make sure everything is correct. These are very simple, but they are a reminder that when we automate it, the automation should be simple too, if your automated test is more complicated than your manual procedure, there’s something wrong, it should be as just as readable or more.  
So that’s where have it, I’ve got one manual procedure and two automated tests and I pushed out to the world, and hardly any documentation, but there you have it.  
Actually I think that’s just what I wanted to cover right now, so what are we going to cover next week? There’s a lot more to do, so I’ve just gotten started, I’ve picked one feature, add, to test a little bit, but clearly we’ve got to have more. Next time we talk about this, I am going to talk about how to choose which features— I think we’ll talk about how to choose which features to test next in your application, and in the Cards application. We only have a few features we could probably test them all, but with tracer bullet tests, it’s not too bad. But if you’ve got a bunch of features you might have to do some prioritization. So let’s talk about that next time.  
I don’t want to get this too long, one of the things I wanted to cover was a thank you for DJ Yannick again for doing the theme music for the show, I really appreciate it. I also want to thank all of my Patreon supporters, and I was reminded that the Patreon level is above 60 now, which is exciting, because I said that once it hit 60, I would start making sure that all the episodes had transcripts written for them. I’m going to have to do good on that and I will contact somebody too. I’ve got somebody lined up to help me with the transcripts, and we’ll get those written. No promises when. With these I am going to do more interviews, of course, there is a lot of people I want to talk to, but I want to try to get through all of these topics of, “How do I get started testing" because I think it’s important, I’ve got a lot of ideas about testing strategy and this should be a lot of fun.