Exploratory testing is absolutely an essential part of a testing strategy. This episode discusses what exploratory testing is, its benefits, and how it fits within a framework of relying on automated tests for most of our testing.
Transcript for episode 179 of the Test & Code Podcast
This transcript starts as an auto generated transcript.
PRs welcome if you want to help fix any errors.
00:00:00 Exploratory testing is absolutely an essential part of your software testing strategy. This episode discusses what Exploratory testing is, its benefits, and how it fits in with a workflow that should include mostly automated tests.
00:00:18 This episode is sponsored by Rollbar. Rollbar is the leading platform that enables developers to proactively, discover and resolve issues in their code, allowing them to work on continuous code improvement throughout the software development lifecycle. Rollbar has plans for all situations, from free to large enterprise. With Rollbar, developers deploy better software faster and can quickly recover from critical errors as they happen. Learn More at rollbar.com Thanks, Rollbar this episode is sponsored by PyCharm. I use PyCharm every day for writing test code Pytest of course. I also use PyCharm for writing both the first and second edition of the pytest book. There’s some markup needed to designate code blocks, shell examples, lots of stuff like that that I don’t want to type or copy and paste. So I used Pycharm’s macros and hooked them to keyboard bindings. Huge timesaver. I save so much time with PyCharm and I know you will too. They’ve been a longtime supporter of the show, so say thanks to them by heading to testincode.com PyCharm, you’ll get a four month free trial. Pycharm save time, use PyCharm.
00:01:39 Welcome to Test and code Exploratory testing. What is it it’s thinking while using your application, your code under test coming up with new things to test, new scenarios to run through and running through those scenarios. Always on the lookout for weird stuff, confusing bits and things that don’t work. It’s black box testing and usually manual. We can be loose about it and just say, go play with it for a while. Or we can be thorough and metaphorical, and we can also be targeted and narrow. For example, new features that haven’t been battle tested yet. We also have automated tests targeting those features, of course, but we may have overlooked some test cases or not examined all the new UI elements closely during every stage of the first draft of the new feature. So spending some more time on Exploratory testing for the new and newish features makes total sense.
00:02:44 Same goes for recently changed features. Actually the same priorities listed in episode 38. Prioritizing software tests through that was way back in 2018. But those priorities are valid for Exploratory testing too, since it was way back in 2018. Let me recap what those are.
00:03:07 Crc is recent the recent features or new areas of the code core, the essential functions that must continue to work. Your products, your unique selling propositions risk. Some areas of the application pose more risk, perhaps areas important to customers but not used regularly by the development team. Configuration sensitive code that’s dependent on environmental sending or operating system specifics. Now that might be hard to do Exploratory testing with, so focusing automated on that makes sense. Repaired tests to reproduce bugs or for functionality that’s been recently repaired? Yeah, go ahead and Exploratory testing those too, and also chronic functionality that frequently breaks. Go ahead and take a look at that manually. Rcrcrc is from Karen Johnson, and it’s a great summary of both where to start your automated testing journey and where to focus your time when doing Exploratory testing. And if you haven’t listened to Episode 38, go check it out. Exploratory testing is not the same as scripted manual tests. With scripted tests, test cases are designed and written ahead of time. Then someone reads the script, does the actions, follows the procedure, and makes sure it all works. It can be boring and rote, so I recommend only using scripted tests for high priority elements that are expensive to automate. Or you don’t have time to automate it right now, but we really should aim to avoid needing scripted tests at all. Exploratory testing is different. It’s unscripted and requires a person to think, to be creative, to pay attention. When should we do it? Well, we should do it early and often, but don’t rely on it for release criteria. You’re not in the right mindset during a prerelease crunch to pay full attention with an open mind. And who should do it? Well, the entire development team should do it, and also the management team. Actually, everybody should do it.
00:05:11 I’m actually going to be a stickler on that. I think everybody should be doing it. If you’ve got dedicated QA people, great, but you should be doing it as well. For my entire software engineering career, Exploratory testing has been part of delivering quality.
00:05:25 There has been a growing reliance on automated tests, yes, and they’ve taken over most of the testing burden. But there will always be a place for manual tests, and the best place I know of is with Exploratory testing. I was on one team that relied heavily on a QA team to do most of the testing for the project. I was totally cool with that. I was working closely with the FPGA developers, writing interfaces to the hardware as higher level abstractions that would later get called by user actions. But the code I was working on day to day had no system API endpoints yet, let alone UI components, so I couldn’t play with the code other than through unit tests.
00:06:07 Then a new release was coming out and we wanted to demo it to customers.
00:06:12 That was mostly a marketing thing, but they were forming teams of three and two from marketing and one from Dev, and I wanted to be part of that.
00:06:21 You could go out and demo the box and the new features to customers, and I wanted to get out of the lab. Of course I wanted to go. I went over the demo outline and even practice it a few times before heading out. With these three person teams we’d have, like the developer, or at least in the three person team I was on. I was supposed to do a demo and the marketing people were supposed to talk to people. For the first demo, I crashed and burned. I didn’t understand the ins and outs of the interface, and I got lost and didn’t know how to get back on track. I was saved by a colleague, of course, but it was embarrassing. This was my box. Why didn’t I know how to use it? I didn’t know how because I didn’t really use it that much from the outside. I didn’t use the user interface. I used it through internal APIs and unit tests, and I hardly ever used it by actually pushing the buttons. I really didn’t want that to happen again, so I really tried to make sure I know how to operate the system, whatever system I’m on, especially new features from the UI. At first, I was surprised that I found bugs right away. When I started doing that, I shouldn’t have been surprised. Exploring and learning the UI is a great way to test the user experience problems.
00:07:33 Another team I was on divided the application functionality into a bunch of categories and then assigned different developers to each category during a testing hours session. Really, they had like a table of how many hours they wanted to hit of Exploratory testing for each release. I was new to the product line, and I figured I’d never find anything since I really didn’t know what it was supposed to do anyway.
00:07:56 At first I loathe having to do those few hours of testing a week. Really, it was just a little bit of testing. Later on, it didn’t seem like a big deal. I actually used the time to make sure I knew how different corners of the instrument worked. One day I was sitting in the test area and one of the higher ups came in and started doing his hours. It wasn’t just for Devs doing this. The management team did it as well, and he was great. He’d do things like push a bunch of buttons at once, spin dials, spin multiple dials at once, start a measurement with no signal, change things in the middle of a measurement, all sorts of things, and he’d find problems.
00:08:34 I actually think that’s when I got the notion that QA and testing was being good at that stuff, being good at coming up with weird edge cases. And yes, that’s maybe part of something that’s good to have on your team. But the value I really get from manual Exploratory testing was that I learned the user interface. I really learned the box, and when I was confused about something, I’d ask. And sometimes I was confused because there was something wrong. And now mostly I rely on automated test suites to make sure everything that was working is still working. I get value from Exploratory testing still, though, in so many ways, even if it’s never a formal part of the process.
00:09:18 Exploratory manual testing allows me to learn and understand the user interface to feel like what it’s like to use it as a customer. It allows me to experience the software as a customer. How can I have customer empathy without it? One of the times I benefit from knowing how to do things through the UI is when an automated test finds an issue. The test itself might be the easiest way for me to reproduce the bug, but the person in charge of that area of the code might not be able to run our testing system for some reason. Or maybe that’s just more complicated than it needs to be. Figuring out a way to list a manual way to reproduce an issue, if possible, is a really great thing to add to a defect ticket. Automated tests are awesome at finding Regressions exploratory testing is awesome for playing with and learning new features or changes to the user interface. It can often highlight aspects that need automated tests as well. When you’re doing the exploratory testing, you might find things that you realize aren’t being automated. Even snapshot tests which are great for detecting unexpected user interface changes and breaks. Only work if you start with a working system and someone has to look at it to see if it’s working in the first place. So even if you totally trust your automated test suite, I implore you to spend time on exploratory testing. At the least it will only make you a better developer and you may find some weird things and you can fix them and make your product better.
00:10:58 Thank you PyCharm for sponsoring visit testandcode.com/PyCharm for a four month free trial of PyCharm, save time, use PyCharm thank you Rollbar for sponsoring Rollbar enables developers to proactively, discover and resolve issues in their code so they can work on continuous code improvement throughout the software lifecycle. Learn more at rollbar.com thank you Patreon supporters join them at testandcode.com support all of those links are in the shower. Notes at testing Code.com that’s all for now. Now go out and test something.