Creating maintainable test suites for complex systems. The episode describes some complexities involved with hardware testing, then shares techniques for shifting complexity out of the test cases. The techniques are applicable to the testing of any complex system.


Transcript for episode 77 of the Test & Code Podcast

This transcript starts as an auto generated transcript.
PRs welcome if you want to help fix any errors.


On today’s episode of Test and Code, I’m going to be talking about a few things that are super important to creating maintainable test suites for complex systems. Doesn’t that sound exciting? Well, it is to me. I want to create test cases that are easy to read and debug and tell a story about what’s being tested, even if the thing I’m testing is kind of complicated. And all of that, I think will be relevant to anyone testing complex systems, whatever complex means to you. However, I want to put this into the context of a concrete example, and I’m going to use my own experience as that example, and that’s the experience of testing hardware hardware, specifically testing electronic test instruments. Since I know a lot of you might not be testing equipment sort of people, I’m going to include a super high level overview of what that means because it’s more fun to listen to the story if you kind of know what I’m talking about. But the domain actually doesn’t matter. But I thought it might be more fun to listen to if I picked a domain that I was familiar with. Special thanks to new sponsor Pantheon for sponsoring this episode. New sponsor to the show. That’s super cool. I’ll tell you more about Pantheon later. For now, let’s start the show.

Welcome to Test and Code, a podcast about software development, software testing, and Python.

I’m excited to jump into this because it’s kind of one of my passions, simplifying complex test scenarios and also testing hardware. I like it. And a lot of people at Pythons or through email and stuff have asked me questions about testing hardware and other complex systems. So I’m excited to dig into it. So to recap or I guess pre cap, we’re going to do a quick flyover of what test instruments are. Then we’re going to Zoom into what boxes that I work with, and I kind of use test instruments and boxes interchangeably.

I’m going to talk about the complexity inherent in talking with test instruments, and really, that’s the complexity I’m trying to shift out of the test cases, and I’ll cover the API involved just to give you an idea.

The domain. Again, the domain is specific, but the problems are universal, and I hope the solutions I’ve used will be helpful to you no matter what your domain is.

Also, in future episodes, I’d like to build on this personal experience example and talk about other topics around testing complex systems and actually got a lot of things like testing workflows when it’s not clear how to test anything smaller. But I got a list. I’m going to kind of list that list at the end of the episode because I’d like to jump into things.

Thank you to Pantheon for supporting this episode.

Pantheon makes building, managing, and optimizing websites Simpler as a leading Web Ops platform, Pantheon features the fastest hosting on the planet, automated workflows with Dev, test and live environments, plus Pro Dev tools and easy integration with GitHub, CircleCI, Jira and many more.

Stop putting out fires and start building sites and experiences that deliver results by using Pantheon, rated as one of G two crowd’s top ten software products of 2019.

Get started for free at Pantheon. Io Test And Code. That’s Pantheon IO Test And Code.

All right, I’m going to try a quick flyover of test equipment, although these are complicated things, so I’m going to way oversimplify them, but they’re fun.

I started out in pure computer science, and I didn’t use test equipment in College. I had my first job at HP and got jumped into communication systems and mostly doing software. But I’ve learned quite a bit, I think, about how these things work. Have you worked on a lot of different instruments?

So I don’t have any test equipment at home other than I have a voltmeter. I got a small voltmeter in the garage. I use it to check the batteries in the house to make sure they’re still good. And when I stick it on a nine volt battery, I hope that the breeding measures above 9 volts. I think that’s how it works. And the ones that are lower and they stop working. So I get rid of those and keep the good ones.

But if that’s a direct current battery, like a normal battery is a direct current thing.

But signals like going through your house or a lot of signals that are used for communication or alternating current signals. And alternating current means that it’s also the current alternates, but also the voltage alternates. And that’s often what we’re looking at, usually what we’re looking at. So if I have alternate current signal and I were to map out the voltage on the y axis and time on the X axis, and I draw it out, if it’s a perfect signal, it will look like a sine wave. And the piece of equipment that does that really good is an oscilloscope. So an oscilloscope is designed for measuring voltage over time. It’s really important for a lot of things like serial signal communications. And there’s a lot of serial communication mechanisms, and some signals do look kind of like that sine wave, but not many.

Now to talk about how we muck with those signals to do communication systems.

If I look at that sine wave and I measure the distance between a couple of the peaks, if they’re regular, that distance is the wavelength. The inverse of the wavelength is the frequency. So if I were to look at voltage over time and it looks like a spring, the sine waves getting closer together and then spreading out, that’s the frequency changing. And if the sine wave is getting smaller and then larger, that’s the amplitude changing. And if it looks like there’s a massive breaks in it, which happens, that’s a phase shift, and we can do that, too. So if I want to look at sometimes looking at springiness is hard to do on an oscilloscope if I flip that, and instead of looking over time, if I look at a signal and measure the amplitude over frequency.

So frequency is on the X axis and amplitude is on the Y axis. That’s a spectrum Analyzer. Now they come in a couple of flavors. The ones that I’m more familiar with are real time or most familiar with the swap spectrum spectrum Analyzer. And those are neat things.

I love working on those. So looking at the frequency, we can see how big things are. And we can also look at how much frequency a signal is taking up.

All of the spectrum Analyzer in Oscilloscope are pieces of test equipment, but we also don’t just measure with test equipment. We use some test equipment also to generate signals, and we can generate sine waves with function generators.

There’s also things like pulse generators and other kinds of generators to generate different looking signals. And simple signals are fairly cheap generator, but for very complex signals, we want to be able to really anything we want to do. We want to monkey with the frequency or amplitude or other things.

We can use an arbitrary waveform generator, and we can also kind of do both ends. We can send a signal out to control something and then measure what we get back. So the combination of something that generates signals and something that measures signals together. What I work with most is a one box tester that’s not only a generator and an Analyzer, but also contains a stack. Actually several protocol stacks to do lots of encoding and decoding.

It’s often referred to as a comms tester. Why is this thing useful?

So a lot of electronics you buy and they’re around you are connected. They’re connected with wireless stuff. They use Bluetooth or WiFi or cellular systems.

And all the people that make this stuff want to make sure that that stuff actually works during the design of their devices. They want to be able to test to make sure all that communication is working. Also during manufacturing, they have to make sure that all the amplifiers and all that stuff are still working. And then there’s also some FCC mandatory RF testing, as well as compliance with partner vendors and other stuff like that. The instrument that I’m working with now a lot is a CMW series instrument from Rodney Schwartz Rodriguez. Schwartz is where I work, so the CMW is a comps tester.

There also are flavors that are mostly RF testing without the protocol stack. But the full instrument also has the ability to do all this stuff, but it also has things like Internet services built in and to be able to measure throughput and Ping times and packet loss and all sorts of fun data measurements. It’s kind of like it’s pretending to be the Internet and the cellular and WiFi network system all in one box so that we can kind of spoof the device under test, thinking that we are the Internet and all of the communication around it, it’s to control the environment. Okay, so I’ve got all this test equipment and I got to drive this stuff with my test code, and my test code has to talk with these complex things, but how? All the instruments I work with have both a Gui and a remote interface. So a graphical interface where I can punch buttons and dial things in or I can send remote commands. Both of these interfaces, the GUI and the remote interface. They are documented in a user manual, which is often updated with each software release. The remote interface is a text based interface using a Skippy called syntax called Skippy. Scpi. Scpi stands for standards commands for programmable instruments. The details are really not that important, but I’ll include a link in the show notes just for the heck of it. Skippy commands and queries.

They can kind of actually get lengthy, but there is a short form that you can use, but it’s still kind of cumbersome. And the Skippy commands and queries usually map to controls or values in the GUI. So I’ve got a computer that’s trying to control these instruments, and the computer sends commands and queries to an instrument and reads the response. The mechanism of that communication is usually something called Visa or Virtual Instrument Software Architecture. It’s all wrapped up, usually in a vendor provided DLL and rodentsports has one. It’s the one I use because that’s where I work. When we think of DLLs, we think of Windows. But there also are versions available for Macs and Linux. On top of the Visa DLL, there is a project called Pi Visa that provides a resource manager which acts as a factory returning connected resource objects to finally be able to use those resource objects to send commands and queries to the instrument.

So what was the complexity that I was talking about?

Actually, a lot of it’s complex, but really actually Pi Visa hides most of that communication complexity. My test code can use Pi Visa to create a resource manager object and then use it to open connections to devices. And then when I’m done, close the connections and then disconnect from the resource manager. This is actually not too complicated. It’s only a few lines of code, but I don’t want that code in all of my test cases. So here’s we’re finally jumping into something of hiding complexity. This connecting with a resource manager and connecting to the devices. This is a perfect job for pytest fixtures because often pytest fixtures are good things to represent a resource.

So I hide a lot of this complexity in fixtures. I’ve got one fixture set up dedicated to the resource manager, and I have that set to session scope. I’ve got a resource manager fixture, and then that one just sticks around and I never actually use that directly, but I have a whole bunch of device fixtures.

I’ve got device fixtures for each device, each instrument that I’m going to be using. I’ve got those also set up to session scope, and they depend on the resource manager, and this all looks really tidy and it’s really nice. All the connection complexity is hidden in the fixtures, and I don’t need to worry about it. Now. My test code can just include a device name in a generic form like Scope or Siggen or CMW or whatever, and use that as a handle to send and get queries. So they’re just one liner for talking to the instrument, and with none of that excess crust lost in the test cases. It’s beautiful and elegant. One of the other things I like about this is if the connection fails, like if a box is turned off or something, the fixture fails and all of the dependent tests report error instead of failure. I like that lesson for other people. It’s a long story, but if you got a communication pathway that you have to set up before your test start, push them into fixtures. I could have just said that at the beginning. Now I did mention that we’re talking to the instruments with a Skippy syntax, and that it’s verbose, and sometimes simple configurations can take up lots of test code space.

For activities that are done a lot, I like to write helper functions. A great example of is anything that requires polling, and there are quite a few of these polling things that we have to do with test equipment. Some actions are known to be time consuming, and the API is set up to allow us to start the action and then sit there in a loop and ask if it’s done.

One example is if I set up a lengthy multi second measurement, I want to initialize the measurement and get it started, and then periodically query the status and wait for it’s. Usually a ready state or done state of the measurement, and then I can collect the answers later. But this setting things up and waiting, all of that clutters up the test case, and it’s way more readable. If I create a helper function and push all of that junk into a helper function. There are some gotchas with this though I’ll talk about in a moment, but for a lot of repetitive and lengthy things, pushing that complexity into helper functions is a great idea. Another place where I want to have a lot of the lengthy Skippy commands pushed out of the test is in setup sequences.

Set up sequences are also a concern because the commands needed to get a system into the state that I need to perform. I might have a little simple check that I want to check, but I need a whole bunch of statements to get the system in the right state. For that test case. I prefer to push all of that set up code into a fixture and get it out of the test function. I want the test case to highlight what is actually being tested, so it tells a nice story.

The other benefit of that is that I can do a couple of checks within the set up code, and if the setup fails, it will result in an error instead of a failure. If I have it set up like that, I can read this error as the thing that I’m testing didn’t fail it’s that I didn’t get to the point where I could test it.

Realistically, it’s still a problem I have to go investigate, but it does give me a little bit more of a pointer as to where the problem might be. So those are the benefits.

On the other side, pushing all of the setup into a fixture does split the split the function in two methods. I’ve got a setup function and a test function, and if that set of functions only used by that one test case, that does make it a little more complex for maintainability, I’ve got two functions over one. However, I might come up with another test that needs the same set up, in which case I can just reuse it. But even if I don’t reuse it, I think it’s worth it to split up just because it really helps clear up what is being tested. Now this is the bit where I talk about the Gotchas and helper functions. So I use helper functions to bundle up a bunch of API calls that are common.

And for the API calls, I like to work as much as I can with stable, documented interfaces as much as possible.

Of course, with new functionality it’s going to be a new interface. But even with testing new functionality, I have to drive a lot of stuff with old functionality. So a good stable interface is good. So I’ve pushed setup Skippy into a bunch of the setup into fixtures. I’ve used helper functions for some of the Skippy sequences, so why not go crazy and build a new Pythonic interface to the instrument so that I don’t have to deal with Skippy at all? I could wrap not use any Skippy nonsense and wrap all of the functionality into little helper functions. Have my own little API to the instrument.

I actually have seen this done. I’m not saying it’s a terrible thing, but I’m going to tell you what my reasons for not doing it. So my first reason is that the Skippy interface is the documented and supported interface, and it’s supported and documented by someone other than me. Second, if I create an intermediate layer, that intermediate layer needs to be supported and should be documented, but I’ll probably have to support it and I don’t want to document it. You see where I’m getting here. Also, when a test fails, the person debugging the failure is just reading is probably going to be reading test code and I want that test code to look like a thin wrapper around the functionality that Skippy already provides.

I want the Skippy API calls to tell a story of how we’re interacting with the instrument, and I want the decoding of that story to be in the manual, not in some API that I’ve hidden things in. That’s why I try to keep helper functions as minimal and very clear. And they better have good names, even if it’s a really long name. Tell us exactly what it is so that if somebody jumps to the implementation, they’re not surprised at the implementation. The names should need to be even more clear than normal. Third reason for using this, our customers use Skippy, so we should too. It seems obvious after you state it, but some people don’t quite get it. If it’s painful to use, it’s painful for our customers, too. Our Skippy should be nice. Fourth reason seeing lots of Skippy and learning to read it is a skill, and it’s an important skill, I think, for everyone on the team to develop again, partly to gain empathy for our customers.

So hiding it into helper functions doesn’t help. Okay, down off my soapbox. That’s why I use the publicly published interface. All right, I think that’s it. I think that’s what I was going to cover.

I talked about a quick overview of the test instruments. I talked a little bit about the CMW zoomed in some of the boxes I work on. I talked a lot about the complexity and how to communicate with these things. And I talked about pushing some of the set up complexity into fixtures and some of the other complexity into helper functions again. Also, I talked about keeping our test cases as readable as possible and making them seem like stories. I think that’s good.

I have a feeling that these ideas will transfer to other complex systems. But you tell me, is stuff like this helpful? And should I continue? Because I’ve got a whole bunch of stuff I’d like to talk about. Oh, yeah, I was going to talk about some of the other things. Not right now. I’m going to talk about things that are coming and upcoming episodes using this example. There’s a lot of things that we can talk about that relate to other complex systems. I’m going to rattle off a few that might come up in future episodes. Tell me if any of these sound interesting.

One, testing workflows when it’s not clear how to test anything smaller. Two, utilizing fixture scope to minimize long set up times. Three trade offs between test isolation and test suite speed.

Next, test. I lost count.

Test suite structure to allow both quick debugging and thorough testing.

Combining multiple test suites and test session results for one software version into a unified test report. That’s a nice one.

Using fake data to simulate events that are hard to create with real data. Balancing confidence in your system with the ability to figure out which part of the system is causing the problem or, in other words, system versus substance testing.

Yeah, I think there’s a lot of these topics that I think would be helpful to discuss on the podcast, but yeah, let me know what you think.

Thank you again to pantheon for sponsoring the show. Check them out at Pantheon IO testingco code and get started for free and that link is also in the show notes at testandcode.com 77 thank you for listening. Now go out and test something.