A listener requested that I start covering some terminology. I think it’s a great idea. Covered in this episode: Test Fixtures Subcutaneous Testing End to End Testing (System Testing) I also discuss A book rewrite Progress on transcripts A story from t…

Transcript for episode 21 of the Test & Code Podcast

This transcript starts as an auto generated transcript.
PRs welcome if you want to help fix any errors.

[music] Welcome to Test and Code, a podcast about software development and software testing. On today’s podcast, we’re going to cover a few terms: test fixtures, subcutaneous testing, end-to-end testing or system testing, we’re going to cover those. And then I’m going to talk about some of my writing projects and I want to discuss the Slack channel as well. So let’s get started.

I got a reader –a listener called in or –yeah that’s not true –a listener contacted me, and let me know that there’s some terms that I use that people might not know, and he wasn’t quite sure how I was using the term “test fixtures”, and so I thought I’d start with this, and I kind of like the format, so please contact me with other terms you’d like me to cover, and if I define something that you disagree with, hey, get a hold of me there too, and let me know how you use the term.

[1:03] So first off, test fixtures. It’s kind of a weird word, and if you think of it like plumbing it’s just wrong, it’s not like a faucet or something. It’s a little bit closer if you think of it like woodworking, because woodworking has –you’ve got clamps and things like that, that go around the thing you’re working on. That’s kind of the same. A closer analogy is in electronics, where if you’re testing a device, an electronic device, you’ve got –you don’t really think of the power supply or a measurement instrument as fixtures, but you’ve got maybe something that holds the device down, you’ve got probes and cables, and all of that extra stuff that isn’t the thing you’re really testing is sometimes referred to as test fixtures. They could have problems with them, but it doesn’t indicate a problem with the thing you’re testing, so that’s a little closer. But really in software testing, a test fixture is –what I think of is, it’s got a few definitions. First off, maybe, I think of it as code that runs to get the system or data into a state before you run your test. Yeah, so, before you run your test to check the output and possibly other side effects, the stuff you have to do before that, and some cleanup after, that’s fixtures. And we often call these setup and teardown. Sometimes, fixtures is just referred to as the data used for testing. I don’t usually use it in that term. And then, also Django has this weird –I think Django’s weird for saying initial data that sets up your database is a test fixture. That just doesn’t even make sense to me. Or it’s a fixture, but I don’t use it that way because I don’t write a lot of Django apps. So, when I say fixture, I mean like setup and teardown, or Pytest fixtures. Let’s see, setup and teardown, Pytest fixtures, umm, I’ve got notes I’m reading from. Oh yeah, also the Pytest fixtures are awesome, and –but they –some people think of Pytest fixtures as –like to think of those as dependency injection type fixtures, but I don’t really use the term dependency injection anywhere else in my life, so that doesn’t really help me. But if dependency injection is something that just is obvious to you, then yeah, Pytest fixtures are kind of like that, but, yeah, doesn’t really help me.

[3:36] Umm, I’m saying umm a lot, yeah. Another thing I want to mention is, one of the reasons why I haven’t been doing a lot of podcasting lately is because the editing process is a pain in the rear, so I thought I’d try not editing so much, so you’ll probably hear a lot of umms and stutters and backups, so my apologies if that sucks, but I’m trying to get more episodes out more frequently, and since I don’t have budget to pay somebody to edit this, the fallback is just not edit it so much. So, that’s it. So, I’ve also realised I say so a lot, so a lot of umms and a lot of so’s. My apologies ahead of time. But, umm, OK, so with test fixtures it’s kind of –with Pytest I like that a fixture, if it has an error, it shows up as an error, even if you assert within a fixture it will error your test and not fail your test, that is different from Unittest. In Unittest, I actually can’t remember what Nose does, but in Unittest, any assert within a fixture or your test function will cause a test failure, and a non-assert exception that is not handled will cause an error. That’s a different model. I like the distinction of an error indicates something happened in my fixture and not necessarily something happened in my test, I like that, so that’s one of the reasons I like Pytest. If you’re not familiar with Pytest fixtures, they deserve further investigation. I’ve set up a series of blog posts intended as a gentle introduction, all the way up through almost power user level of fixtures, if you take a look, go to my pythontesting.net/starthere –there might be a dash in there, just go to the top page there’s a start here button, and there’s a –in the Pytest section there’s a Pytest fixtures part 1. Start there and you can go through everything. Of course, all of those Pytest fixtures posts along with a lot of other goodies are in the ebook that I’ve got in the sidebar. When I wrote that, I was using it on Python 2.7, and Pytest, I don’t remember what version of Pytest, it was prior to Pytest 3 which just came out, which I’m excited about. So what about Python 3 and Pytest 3? Pytest 3 is awesome, and I’ve switched to Python 3 for my side projects, so, yeah, I guess that’s what I’ve been up to in my free time, and that’s one of the reasons I haven’t been doing as many podcasts, as I’ve been completely rewriting that ebook. I didn’t intend it as an ebook when I started, it was blog posts pulled together so people could read it offline, but I don’t want to just update it, I want to try to rewrite it as a book with an editor and a timeline, and it’s just getting kind of off the ground and it’s exciting and I’m enjoying working with an editor, but it’s a slower process. So that will hopefully be out sometime in 2017, but what about the ebook now? It’s five bucks now, if you buy it now or any time up until I take it down, it still should help you. And what about that five bucks ,did you lose out on that? The answer’s no. When I was talking with the publishing house that’s going to put this new book out, I said I want to give back to the people that have supported me. So we’re going to make sure that you get at least five bucks off, if you’ve bought it through Gumroad or Leanpub, and given me your email address. If you haven’t put your email address through Leanpub, which is an option, you can just let me know, say hey I bought it through Leanpub. You can send me the receipt, or, just, you know, I’ll probably trust you. Send me your email, and I’ll make sure that you’re on the list of people that get coupons when that comes out. But it’s not going to come out until 2017, I’m still working on it, and I bring this up because I’m just going over the fixtures portion of the book right now.

[8:02] And, let’s see, what’s next? I wanted to talk –another term is subcutaneous testing. And, end-to-end testing or system testing, different levels of testing is actually quite a big topic, but I do use the term subcutaneous, I can’t remember where I got it from, I think it was, yeah, I’m not going to get it right. But anyway, end-to-end testing is testing your entire system from the main user interface that people use, all the way down to everything, just, the whole thing in place. It might –I still think of end-to-end testing even if it’s a test server, you’re not running on the real –the live system, but you’re running something very close, that still seems like end-to-end testing to me. So what is subcutaneous? Well, end-to-end testing sometimes is difficult, sometimes the end user interface, like a graphical interface or a web interface or something is difficult to test. It’s made easier with things like Selenium for web stuff, for other GUIs it’s still difficult and painful, sometimes. But often I really just want to test the functionality of the system, and I like to have an API that is available, either a REST API or just a Python API as well, or some other API, that exposes the entire functionality of the system, just like it would through the GUI, sometimes even more, and if you trust the equivalence of the functionality through the graphic interface versus the API, if that –most of the systems I work on it’s very trustworthy that these are identical or very close, then it’s often more efficient to test the API and just do less tests through the GUI, and that’s what I think of as subcutaneous testing, subcutaneous being just under the skin. Umm, let’s see, I think I’m skipping over a few things that I wanted to cover. Err, no, I think I got my notes so far. About the ebook, please continue to –I encourage you to continue to –to buy a copy, or two or three or ten, to support my writing, that would be great, because although I’m doing a rewrite, I don’t get any money from that until sometime after 2017. Umm, so, I love people buying the book, it sends an email to my phone and I get a happy feeling every time I see one of those, so thanks a lot. Let’s see. You can also support the show by becoming a Patreon supporter, more information on that at pythontesting.net/support, I’ve got links there, or you can go to Patreon, that’s P-A-T-R-E-O-N dot com slash testpodcast, all one word. Yeah, why I wanted to bring that up is that I said that I was going to do transcripts as soon as we got to the sixty dollar mark, and I haven’t gotten to that, I’m getting support for forty bucks. It says forty dollars an episode but that’s a little misleading, it’s a little less than that because you can limit it, you can say that I only want to donate at most a buck a month, or two bucks a month, or however much, and I like that. Anyway, I said I was going to do transcripts, I haven’t got to the mark but I’m going to do them anyway. I’m testing a few transcriptionists, and hopefully soon I can get some of those out. I will get the transcripts out first to the two dollars and above supporters, and then next I will get them to the one dollar supporters, and then after a reasonable delay, to give some benefit from being a supporter, I’m going to open that up on the website to everybody. Now, I don’t want to hide information from people, but I do want to thank the supporters, and I want to get them some benefit from being supporters.

[12:36] Last thing I want to talk about is something –it’s a little side thing, side thing on my side project, is the Slack channel. I’ve got, if you go to pythontesting.net, there’s a button, you can go to /slack but there’s also a button that says Slack. I haven’t really pushed this a lot. I didn’t know if it would be useful, I don’t use Slack in my day-to-day job, or much really, but I thought I wanted to play with it, I didn’t know if it would be useful, but I thought maybe it would be kind of cool to have an informal Test and Code community or an informal Python testing community. So I set up the Slack channel and I linked to it on the site, but like I said, I didn’t do any real fanfare. A few people found it, there’s right now 26 people in there so far. And I have a little note at the top, I just added this recently, that says, yeah, if you’re new, if you could tell us a little bit about yourself, like what kind of code you’re testing, and what test framework do you use, Pytest, Unittest, Nose or something else, and then, ask questions. I didn’t really know what would happen, but it’s really pretty cool, you know. Most of the time right now it’s pretty quiet, but occasionally somebody asks a question, and this was a question from a few months ago. Somebody came on, and asked a question about the best way to document tests, like document an individual test. And that kind of moved on to a discussion of given-when-then and arrange-act-assert, and, if you’re not familiar with those, there’s more information on that if you go to pythontesting.net, look at the start here page, there’s a strategies section, and there’s links to both a blog post and a podcast episode about given-when-then. But anyway, so, this discussion we talked about how to structure a test based on that, and using those tags for documenting, and then somebody posted like a little code example of a test, and the discussion moved into possibly splitting the tests so they don’t test too much per test, and how to split it into functionality versus, you know, split the functionality across, or split it by depth, doing the front end tests different from the back end tests, and maybe a combined end-to-end test, and it was a cool discussion. And then, one of the things that came up was that the person who posted the tests said “I’m new to web testing and I can’t get answers from my colleagues”, and then one of the other people mentioned that testing is unique from application to application, and I’ve seen that, I’ve seen that people don’t really know what community to check with. So the particular kind of thing that they’re testing, they don’t really know who to ask for help, and I know that there’s Stack Overflow and there’s lots of resources on the web, but this is just a resource that, who knows, it might be cool, and the more people we have in there, the better the answers can be, and it seems to happen sort of real time, and it seems to help people out. So it’s kind of a cool thing. At first I was nervous that, maybe I was going to get too many questions that I couldn’t answer, but that’s kind of the great thing about this, that I don’t need to be the expert at everything, you guys are all experts at something, and you can help people out as well, and it’s a little more private than answering Stack Overflow stuff, and it’s not for points or anything, you don’t get like, you know, Reddit gold or anything like that, you’re just helping people out in a community, it’s neat. That’s really all I got for now. I mean, I’m going to try and do more frequent, less edited, less rehearsed podcasts. I hope the quality doesn’t go down too much, but I want to up the frequency, so I got to take out some of the overhead. And I hope that next month, hopefully I can get more of these out per month. I’m really shooting back, I’d like to get 2-3 out per month, at least, cos there’s a lot to cover, I’ll do smaller –now I’m just rambling. Yeah, this is why editing is good. Anyway, come to pythontesting.net and the show notes will be fairly empty this time because I’m just kind of rambling today.

[17:42] So what did we cover? We covered –we talked about the Slack, we talked about subcutaneous testing versus end-to-end testing, and we talked about fixtures a little bit. Let me know what other terms that you’d like me to cover. I can ramble on this stuff a lot, so this is a good thing to go into some of these short episodes. Anyway, thanks a lot and I’ll talk to you next time.[music]