Corey Quinn is the Chief Cloud Economist at The Duckbill Group. He’s also a podcaster and writes a newsletter. And he also automates things with Python. But he doesn’t write tests. Let’s find out why.
Transcript for episode 149 of the Test & Code Podcast
This transcript starts as an auto generated transcript.
PRs welcome if you want to help fix any errors.
00:00:00 Corey Quinn is the chief cloud economist. Is that really a thing at the Duck Bill Group? He’s also a podcaster and writes a newsletter, and he also automates things with Python, but he doesn’t write tests. Why not? Well, let’s find out.
00:00:27 Welcome to Testing Code.
00:00:38 Well, welcome to Test in Code. I am really thrilled to have Corey Quinn on the show. So, Corey, I know a little bit about you because I’ve kind of looked around at your podcast and stuff, but I’m sure other people that listen haven’t.
00:00:51 So who are you?
00:00:53 That’s a great question. I ask myself that every morning when I stare into the mirror and try and slap on a brave face so I can basically face the mirror when it’s shaving time. I’m the chief cloud economist at the Duck Bill Group, where we fix the horrifying AWS bills. I also dabble with a newsletter last week in AWS that gathers the news from Amazon’s cloud ecosystem and then gently and lovingly makes fun of it because again, I have personality problems. And I have a couple of podcasts of Screaming in the Cloud podcast, which is an interview show with various people who are all way better at what they do than I am. But hanging out with smart people makes you look smart. And the AWS Morning Brief, which is sort of an AWS themed show every morning that has different approaches throughout the week.
00:01:37 Well, one of the questions I was going to have for you was how do you do like actually a Daily Show?
00:01:42 That seems like a lot. But you also have I mean, that’s a decent question still. But you have a team of people working with you, don’t you?
00:01:50 I do, indeed. We’re eleven people now.
00:01:52 Eleven people. And the Duckbill Group is what you’re counting as eleven people.
00:01:57 Yes, indeed. As far as the podcasting stuff goes, a couple of my colleagues handle the Friday show, so it’s great for them to wind up taking that over. But as far as the production goes, we have a company Humble Pod that we work with that does an awful lot of the nuts and bolts of handling, recording, processing, making the audio levels sound good, getting things posted, et cetera. We all have strengths. I can have a lot of conversations with folks and I can be prolific, but I also can’t really go ahead and do all of that and then remember to copy the files into the right folder. So automation goes a long way for a lot of these things.
00:02:33 Yeah, definitely. I think getting help and automation is a good thing. And also not being I’ve learned to not be a perfectionist. That helps a lot.
00:02:45 I suck at preparation and I hate going back and editing things. So I guess sort of forced myself into being better at the improv story so I can wind up being decent enough off the cuff that I don’t have to go back and edit. And also there’s this.
00:03:01 We’ve all worked with developers like this in the course of our career. I just apply it to different areas in which I take basically zero pride in my work. I’m kidding, mostly. But it’s no, honestly, close is good enough is a thing. And if you try and get something absolutely perfect, it’s never ever going to ship. And then people look at my code and they say, Well, I agree with what you just said, but have you seen this thing that you wrote? I’m not talking about good enough. Let’s just call it enough and make sure you never do this again.
00:03:31 That’s hilarious.
00:03:32 Well, it’s depressing.
00:03:34 You thought you were going to write a script. Instead you wrote a horror. What happened here?
00:03:38 Well, so how much coding do you end up doing?
00:03:41 Do you coding too much for my team’s liking? More than I ever used to. And it’s weird. My background is as a grumpy Unix systems administrator, because let’s think about it. Have you ever met a Unix systems administrator who wasn’t grumpy? You have not. It’s the only characteristic that is universal. And I would say, well, I’m not a programmer because I wasn’t. I was at best stringing together some crappy bash and working with a one liner in Pearl here or there. And I’m still not a programmer. As the world evolved and I became a Devop or an SRE or whatever we’re calling them this week. Don’t email me. And I started to teach myself Python in some weird ways, and again, in the same way that I write everything else badly. And at some point it’s writing an awful lot of code for someone who’s not a developer. And then, of course, we all ascend past a certain point to the one programming language that we all use, YAML.
00:04:35 And here we are.
00:04:38 I don’t.
00:04:40 I write mostly Python and C Plus Plus, so I haven’t ascended to YAML yet.
00:04:45 Oh, it’s the one true expression of all things computer. If you don’t believe me on that one, just ask the Kubernetes people.
00:04:53 Yeah.
00:04:54 Okay, so programming has been devolved into just configuration then?
00:05:00 Exactly. Well, yes and no. I was one of the very early developers behind the Salt Stack project once upon a time, and the configuration there was rendered as YAML combined with Ginger Templating. So instead of going full on into a world of a custom DSL, instead, we just built something horrible. And okay, we’re going to make it almost like a language, but not quite. And that really began my ongoing, I won’t say love affair relationship with YAML. Okay, pro tip. Never do this.
00:05:32 If you want people to write code, just have the right code. It’s not that hard to get people over the hump of writing some basic flow control in Ruby or Python. Things we learned in retrospect.
00:05:44 Yeah. So it’s interesting.
00:05:48 I see the resistance there sometimes, because, for instance, it’s not that hard to convince somebody to store a configuration structure in Jason or JSON, however you pronounce it.
00:06:04 But you can also just throw a dictionary or something in a Python dictionary in a Python file and just import that. But that freaks people out to consider a Python file a configuration file.
00:06:16 I have a theory on that specifically because, as I mentioned, my entire world was built around well, I’m not a developer, and I keep telling myself that even though I spent an awful lot of time doing things that look pretty close to development. Sure, it’s bad, but that’s okay. There’s no quality bar. It’s not gatekeep what development is. But if you tell people who are in that space and I know this because I was in that space, that, oh, it’s easy. Just write some Ruby or just write some Python, we freeze. We feel this isn’t something I’m actually capable of doing because we have this conception, rightly or wrongly, that, oh, that is something other people do. And you have to be at least X smart. And I’m nowhere near that smart.
00:06:57 There’s almost a stumbling block for people who are in technology but don’t conceptualize themselves as developers and then tell them to write code. It’s a sudden, monumental and fundamental shift in how they see themselves, and there is a tendency, in some cases for people to freeze up.
00:07:16 Yeah.
00:07:24 Thank you, PyCharm, for sponsoring this episode. Even if you are comfortable with another editor, I encourage you to try PyCharm for use with your test code. With PyCharm, tests are easy to run, debug, and maintain. So test functions. When you look at them in the code, they have a little green play button next to them that just begs you to click it. And then once you do run it, the output window just automatically appears. And if there’s any failures at all, PyCharm puts links in the output that you can just click on and go right to the error in your code. It’s super cool and super fast. Try it yourself by going to testimoncode.com PyCharm and sign up for a free four month trial of the Pro edition.
00:08:11 Now, one of the things I’m curious about, the Duck bill group, as far as I can tell. I mean, I haven’t really looked into it too much. Eleven people working there or twelve with you, I’m not sure.
00:08:26 Your primary goal is to just have people save money on their bill.
00:08:31 Is that what you sounds counterintuitive, but yes or no?
00:08:34 Okay.
00:08:34 That is often what people reach out to us for it’s. The AWS bill is large, and in some cases large means several hundred million dollars a year.
00:08:44 And they say we need to reduce the bill. Well, that’s often not true. What finance is saying is this bill is large and keeps getting larger and keeps blowing past our estimates. What does this mean for our projections 24, 36 months out there? I’d say oh, this is an actual problem. At some point, the bill becomes a problem no matter who you are and how much money you have, because it never gets smaller on its own. And it requires a complex interplay of different services tied to different things. Sure. There’s easy stuff.
00:09:14 There’s always the approach of turn that nonsense off, which is a great, easy first pass to do. Okay, now what? Well, the way that some applications are structured talk incredibly verboselly.
00:09:26 Between availability zones, well, that costs two cent per gigabyte that you wind up transferring back and forth. You’re only taking in one petabyte a month only. But between availability zones, you’re having 50 petabytes. Go back and forth, maybe don’t do that.
00:09:42 It turns into an architectural story. It turns into an accounting story to be very direct. Most of what we do, a lot of our engagements, is serve as marriage counseling between engineering and finance. Those two people don’t usually talk as much as they should.
00:09:57 Okay, so you really are. I mean, it’s not just you’ve got some tricks to get the bill lower. You’re working with their engineering team to try to figure out what they’re doing.
00:10:10 Exactly. And there’s usually an executive level strategic initiative behind it, because good faith citizen engineer efforts. I’m going to reduce the AWS bill. I’ve been there myself. I’m stuck there myself where I have a $600 AWS bill for my development environment. I bet I can get that down to 406 weeks later. I have, but it cost a lot more than that to my employer as I’m stepping down that path.
00:10:35 Okay, that makes sense.
00:10:37 Interesting.
00:10:39 Yeah. It’s a weird market.
00:10:41 It’s not something that is heavily served by consulting. Historically, there are tools that claim to do it, but there’s no API for that business insight.
00:10:49 Okay.
00:10:50 And there’s certainly no tests for that API that doesn’t exist.
00:10:53 So the normal listener to the podcast might be thinking, this is interesting, but why the heck did you invite Corey on the show?
00:11:00 Because I’m hilarious, obviously.
00:11:04 Well, you are, but one of the things I don’t know if this will go anywhere.
00:11:09 But I don’t know.
00:11:11 Back in January, you posed a question on Twitter that says, hey, I’d like to guess on some podcasts, what should I be on? And somebody mentioned my podcast, and your response was, But I don’t test my code. Crappy. Python is all I write, so I’d like to poke it out a little bit.
00:11:32 By all means, do you test your code?
00:11:36 I do, but not in the way that anyone would recognize as testing. For example, in my first job, I wound up being my first, I guess, computer job. I was a Unix systems administrator for a University. And the monitoring system, we had to test whether a change we had just done broke something or whether there was a problem with an existing system was the help desk. When they called and said users are asking us about this thing. Is there an issue? We would go and check and oh yeah, it looks like what we did broke something. Thanks for letting us know. And to some extent, for the internal systems that I build and write.
00:12:12 Yes, that is functionally how I wind up doing the majority of my testing.
00:12:17 Okay, when I say internal systems, I’m primarily talking about the Byzantine Rube Goldberg machine that I use to generate my newsletter that goes out every Monday. It sounds like I’m being ridiculous. I’m kidding on this, but I’m not. I have a total of something like 29 Lambda functions. Sorry. When we say Lambda functions, I’m talking AWS Lambda functions, not Python Lambda functions. Because why do I use the term when it’s new, reuse an existing term in the space to confuse absolutely everyone. And that’s bolted to a bunch of API gateways and it does a whole bunch of things with DynamoDB and the rest. It listens to various RSS feeds that AWS has, because there is no AWS RSS feed that tells you everything that has been released. There are 41 of them. And consuming all of that into a system that I can then say boring, boring, boring, boring, good, boring, boring, boring. And just get the good things into next week’s. Newsletter took a bit of work. Originally, I was doing this in Google Docs. It took me a full day every week, and then I slowly iterated forward and upgraded. But I’m the only direct customer of this thing. So my testing is I’ll go ahead and push a change, make the commit it is live in this quote unquote production system within less than a minute. And then I go and test the thing I just did. And at times this is, oh, great. So I’m just iterating forward and cycling through. And since I’m the only person in this environment, no problem.
00:13:43 When I say I’m not Test And Code, I mean that there’s no syntax checking at this. Well, you forgot a space somewhere or you misspelled a special reserve word that makes me feel dumb. Let’s not kid ourselves. And I look at this and I think, there’s got to be better ways to do this. But every time I start looking into various approaches to testing, unit testing, integration, testing, et cetera, it almost feels like I fundamentally have to restructure how a lot of that crappy Python is done. Maybe that’s accurate, maybe it’s not. But I look at this and I just think, no, and I always find something better and more interesting to do.
00:14:21 Actually, this is a great use case, so I’d like to learn a little bit more about this.
00:14:26 If you don’t mind. I’m going to ask a few questions about it.
00:14:29 Sure.
00:14:29 So you say that you’ve got a bunch of Lambda functions and other things. Basically, there’s a black box of something that you work on that you push out new versions of it and you say you just try it yourself. So what is that? Are you actually sending out the email or is all this stuff generating something that you review?
00:14:51 No, it’s all internal there. As it turns out, the email service provider I use does not offer an API. So what? I have an architectural diagram that I will send to you for inclusion in the show. Notes. There is a section of that diagram that includes a Hacksaw and a bottle of glue, because that’s the part where I copy and paste like, some kind of antiquated farm animal into a web form and then schedule it up every week. It is the most obnoxious part of this entire process. Okay, so for better or worse, even if I screw everything up massively, it’s not going to break too much because the output of this entire system from an automated place at the very end and all it can do is a web page with a click here to copy the link. Okay, so there is no risk here of. Well, I just accidentally sent out six emails this week, doesn’t it? Sucks to be you. No, it doesn’t work that way.
00:15:47 The failure motor blast radius is small. It’s just this is broken and isn’t doing what I wanted it to do. Okay. Because the infrastructure is designed as code. I do have other environments. So if I want to go off into the weeds and do something zany, I can spin this up in a separate account and not break the thing that I’m going to be using for the next issue for real.
00:16:07 Okay, so I almost missed that. So you have the ability to be testing a version of this while the other one is still live and working?
00:16:19 Yes, mostly.
00:16:20 Okay.
00:16:20 The challenge, of course, is that you talk about the natural evolution of things. One of the load bearing DynamoDB tables in this thing has the word test in its name because we iterate forward in a test environment until something works and suddenly, Holy crap, it worked. Don’t touch anything. It’s load bearing now. It’s like, well, it works on my machine. Well, back up your email, Skippy, because your laptop is going to production. You start optimizing down that path, and that’s where Docker comes from.
00:16:48 Yeah.
00:16:49 Okay.
00:16:50 I’m really glad I have it on the show.
00:16:53 Thanks. It’s a pleasure to be here. I do appreciate it. Oh, wow. People who know how to write code, they’re smart. Maybe I can learn a few things.
00:16:59 But what are the external inputs? So you’ve got your code going, but you’re reading RSS feeds.
00:17:05 Is that what you’re reading?
00:17:06 Mostly, partially. This is the joy of it. The reason why it’s 29 distinct Lambda functions is not because I believe that that is a good architecture necessarily, but that is the serverless road of the future. All right, fine. We’re going to grow with that. Do one thing and do one thing. Well, I really appreciate that Unix philosophy, and I love the microservices approach, because if a Lambda function only does one thing, I don’t need to test it because it can’t break. Because if it broke, that would be a second thing. As Matt Stratton is famous for saying, so great. That removes the need for testing. Right?
00:17:43 I don’t think that logic follows.
00:17:46 Well, that’s because you haven’t tested it, presumably. But that’s one way of doing it is gathering the official stuff from that AWS spews out into the ether. Great. I also include tools that I find that are interesting in the community. And of course, I find blog posts people have written about, wow, I wound up using AWS, and now they’re trying to foreclose on my house or whatever the heck happened. Great.
00:18:09 When I see those in the wild, I absolutely want to include them. So I want to be constructing a custom API that I can use a bookmarklet to hit, or I can use an iOS shortcut. If I’m on my phone, that winds up grabbing whatever link I’m on and stuffing it into the queue for the next newsletter’s consideration. Originally, I was using an actual service for this Pin Board, like a sensible person might, and Pin Board was working super well. I was doing it with certain Tags and then having it imported into the system. But Pin Board is a service that is made by one guy, and he’s fond of saying on Twitter that one day I’ll get hit by a bus. So make sure your BOOKMARKS are backed up.
00:18:50 Cool. I get that for a personal bookmarking site. That makes sense. But increasingly, this became a load generating business. So I can’t really just say, oh, that’s okay. I’m not going to send an email this week because the guy who winds up running the service forgot to check the fuel level and the generator at the data center or whatnot. Great. I needed something a little bit more, I guess, something I could control and could affect the meaningful uptime or downtime. So all this stuff lives on AWS services, and if AWS goes full down and I can’t write a newsletter this week, great. That’s okay. As it turns out, I will have a different newsletter that week, written by hand of what happened last week in AWS. No, really? What the hell happened?
00:19:36 Yeah. Okay, so you’ve got a lot of stuff going into let’s say one of them doesn’t work.
00:19:43 Like, I don’t know.
00:19:46 One of the RSS feeds for one of the AWS services.
00:19:51 How would you know that it stopped working?
00:19:55 That’s a great question. The short answer is I’m not entirely certain that I would.
00:19:58 Okay.
00:20:01 That’S not technically true.
00:20:04 Whenever AWS starts talking about a thing publicly that’s been released and it hasn’t hit the RSS feed, I started pinging them there. But it’s from an absence of information, not something that’s coming through that’s garbled, for example.
00:20:17 Okay.
00:20:17 And the weird thing is they release so much that if they have something break and there isn’t a particular bit of news that’s included, I actually don’t care. As strange as that sounds, the entire purpose behind this newsletter is not to give a complete list of everything AWS has done. They already have that, and it’s boring. This newsletter is sarcastic and fun. At least it is from my point of view. And it’s a curated sampling of the things that I find interesting that came out. And if they break their RSS feeds and don’t mention that this thing happens and I don’t stumble across it by other means. Well, that’s kind of a marketing fail on their part.
00:20:55 Yeah, but the output, you’re going to get an HTML page and you’re going to look at it and then you do some editing there copy and paste or whatever.
00:21:06 Yes, I actually think this is a great example because you are using code, but you’re the user of it, so you’re both the writer and the user. And so you’re looking at the output of every single one. I do get that actually everything that I’m not the user of I test.
00:21:29 But for instance, we’ve got scripts in my project that pull together the release notes from the different projects that we pull together and then generate an email for us to send out. And it’s a similar sort of thing. If it stopped working, I would know because I’m the one running the service, running the process, and I would have to fix it. So I don’t need an automated way to tell me that it’s not working because I would know right away.
00:22:03 Well, you do hit on something that I guess from that lens is considered testing. There’s an entire lending validation system that goes out pair of functions. So before this thing is rendered into HTML, it is rendered first into a custom markdown variant that we have devised called Snarkdown, because of course it is. And from there I wound up finding an existing Markdown Linting system on GitHub or GitHub. As I’m told, it’s pronounced. And from there they wind up letting you able to they give you the ability to add custom rules around this. Okay, great. Because in the early days, some links didn’t render. If there was a space somewhere there shouldn’t have been. It looks for bare URLs, it looks for unclosed Tags and things like that. And every time I screwed something up, it was, oh, awesome. I have a sponsor who’s upset with me or readers say you screwed something up again. Because if there’s one thing people love, it’s being correct with someone else has messed up and great, that’s fine, we’ll take that. I appreciate the feedback, truly. And then I add another rule and at some point you finally finish enumerating all the badness and things that are common failure modes. And great. Now whenever I run it through the system, I have a reasonable expectation that it is going to be correct, at least from a technical point of view, and render into proper HTML. The next thing it does beyond that is a link validator where it grabs a Python library and goes out to every URL that it sees in that markdown document, and validates intern that it gets an acceptable response that the 302 redirects to somewhere correct, or that the link is still up because you don’t want to send out a link that has been taken down. The most common culprit of that, as it turns out, is AWS, they’ll put out a link and it’s valid when I look at it. Great. I add my commentary at the end of the week comes along this hypothetical world where I’m working ahead. Imagine that instead of doing it all at 730 at night on a Sunday while crying, and yes, then they’ve taken the state of the link down and there’s no reason to include it. Damn it, I like catching that before it goes out.
00:24:14 I didn’t know that they were marked down, Linda.
00:24:18 I didn’t either until I started looking at how it would take to build one myself. And this seems like the sort of thing that someone might have attempted to solve before it turns out there isn’t a markdown lender. There are dozens of them, and they all have pros and cons in various ways.
00:24:34 Okay, so you have your own Snarkdown Linter, then I do.
00:24:40 My only real addition to this thing is both a configuration in which I disallow certain rules from being enforced, because I don’t care that the Snarkdown transformer does not care about those things, so I don’t care about those things. And I added custom rules as well, because I have certain blocks or Tags inside of different section headings that the system can wind up interpreting, and that is something that needs to be and that needs to be correct. But markdown itself has no conception of that.
00:25:09 Okay, is SmartDown an open source thing, or is it internal?
00:25:14 No, it’s not more or less just a custom. It’s a slight variant where I’ve added a couple of Tags. There’s not too much to it other than that, but if I call it actual markdown, people will yell at me because it does not look like typical markdown to most Editors because it doesn’t understand a tag or two.
00:25:31 Yeah, totally get that. I’ve attempted versions of markdown myself until I found out that a complete markdown system needs to be actually a language and not just a Perl script.
00:25:50 Exactly. At some point, it’s one of those. Oh, dear Lord, I’m trying to parse HTML with a regular expression again. Here we go. It’s like I should go do something better and more productive. I try to implement my own cryptography.
00:26:01 Yeah, if I were ever to do that.
00:26:05 It would be something that didn’t allow HTML inside, because that’s a hard problem.
00:26:10 Yeah, I guess what? I’m coming to realize that this is in fact a legitimate form of testing where it’s the idea of making sure an error doesn’t hit production. Now, when you talk about testing in code, my immediate response is thinking about, oh, unit tests and having a test suite that you run on checking things in. No, there’s none of that here.
00:26:27 Yeah, okay.
00:26:33 I think of testing a little more broader and also, like more narrow. I don’t really like unit tests, but I don’t write them.
00:26:43 I write more system level stuff of here’s, my inputs here’s, my outputs. Does that whole flow work still?
00:26:49 Yeah, I really should have a reference issue or something where I can wind up speeding specific events and then validating it renders correctly at the end. That’s a great idea, but I tend to view on some level the testing might almost be like the proper hierarchy of building out AWS configuration. The way we all start is using the AWS console. We click and we build things, and then you’re like, okay, this is great. Can I now get that as code? And the answer is no. Now throw it away and start over. So, okay, then we move up the chain a bit to using cloud formation or TerraForm or beyond that to the CDK, which they’re all about, or Poloomy or something like that. But then the final form, the apex of that evolution, is using the console and lying about it. And I feel like that’s sort of how testing tends to evolve on some level. Eventually we’re going to say we do unit testing because it’s easier than actually doing unit testing. I don’t know how true that is, but I can’t shake the feeling that maybe there’s something to that.
00:27:44 Well, I think there’s a horrible lack of information about what testing is in the entire education system.
00:27:58 I don’t think we need to teach testing to see us people, of course, but really, any job that involves monkeying with code at all should involve talking to people about how to validate that the thing that you’re doing works.
00:28:14 I guess that’s the week that we’re recording this. It’s funny you bring that up. I wound up writing an article after taking the AWS certification exam that’s in beta, and it was so far removed from what I actually care about when I’m working with AWS infrastructure. It’s all great. So you know how all these things work in interplay and serve each other. Great. Now we’re going to test that. You know this by asking a bunch of trivia questions like, which one of these is not an actual valid string to this argument? And it’s what is wrong with you? It’s this idea of we’re testing for things that are basically lend themselves to rote memorization rather than actual interaction with the system itself. And that is crap. And their biggest concern is because it’s a remote proctored thing. Are you looking around the room and potentially cheating. Let me see your wrist to make sure you’re not wearing a smartwatch or something that can give you answers.
00:29:10 I don’t mean to be obnoxious, but I promise you, I don’t care about your test enough to cheat on it.
00:29:16 Yeah, I don’t have any certification beyond I got through College.
00:29:24 I didn’t do that. I don’t have a high school diploma either. I hear you. Education is not at my forte. The single reason I wind up getting a certification from AWS every few years is because at their inperson events. Remember in person events. Man, I missed those on some level is that you could get access to a lounge that has better snacks and charging ports for your electronics. I view it as one of those, like, elite airport lounge style things with just a really weird entrance questionnaire.
00:29:51 You know, this is a tangent, but there’s been several times where I’ve had.
00:29:57 Like, little coupon things that I can enter one of those lounges, but they expire in six months or something, and every time I actually have one and I’m in an airport that has one of those things, I’ve got like, four minutes to get to the next gate and it’s pointless.
00:30:18 I’m one of those people that shows up at the airport sarcastically early because I live in fear of missing a flight. There are two types of people I’ve learned the people who show up there and just work in the lounge for 2 hours. That’s me. And the people who are sprinting before they close the doors to get on the flight. And that last one stresses me out.
00:30:36 I don’t want to say I travel a lot, but for a while there, I was on a first name basis with the lounge staff in three different airports, and it was I’ve got to stop doing this as much.
00:30:46 Yeah, I know what’s going on in these people’s lives, and they’re lovely people, and I really should not know this much about the bartender at an airport Lounge’s life just because I’m here too much.
00:31:01 Then I finally get off and I get on the plane, and then the flight attendant is like, oh, Corey, how are you doing? Damn it, I need to stop flying everywhere.
00:31:11 How do I do that? And then, like, the finger on the monkey’s paw curls in and here’s a pandemic. You won’t be traveling this much. Yeah. So careful what you wish for, I guess is the answer there.
00:31:21 Yeah, I guess so.
00:31:27 When you brought up testing, you also talked about unit testing. Why do you say unit test? Is that just something you think?
00:31:33 Because I can fundamentally understand what a unit test is. I can say, all right, I’ve just written a function, and if you view a function through the context of functional programming, I believe I’m using the term properly. If not, please be sure not to email me.
00:31:49 You put a thing into it. And you get a defined thing in, you get a defined thing out. At least that is the way I’ve approached things. So if you have the idea of okay, if I put X in, I will get Y out every time that feels like what a unit test is based upon people trying to teach me things and then I zone out because I spotted something shiny. I basically a giant walking raccoon and cool, I’m going to now take that idea. Now I have a test. Okay? When I put X in, Y comes out. So if I change something in that code and now something other than Y comes out, that test should fail in theory, yeah. Have I nailed the salient point of what a unit tested? You may email.
00:32:27 Well, I think that you’ve nailed this alien point for what a test is.
00:32:32 Okay, so let’s go further than what is the differentiating demarcation line between a unit test and other things like integration test or other forms of test for that.
00:32:49 I don’t think of it as too much of a difference. But the reason why I don’t like the term unit test is because people will Google how do I write a unit test? And there’s been a joke about this recently.
00:33:02 They will get 3347 answers that are all this is how you test to make sure that you can add two numbers.
00:33:13 And now that you know how to test things, you go out and test your application.
00:33:17 So there’s a couple of reasons.
00:33:25 There’s a lot of information around unit tests that talk about isolated unit tests that say if I write a function, I want to test just that function. So if that function goes out and calls AWS or talks to the database or talks to the file system or calls any other functions, I want to stub out all of the external things so that I’m only testing my function and not the rest of everything.
00:33:50 That part I just think is a waste of time bullshit. So that’s where my tendency to not use the term unit test comes from, because I don’t want people to Google that and learn to step everything out because then they’re not testing their system.
00:34:09 I don’t know how it helps you to just test your own code and not the entire application.
00:34:16 Which makes sense. And when we get back into the realm of things I do know about, like deploying applications at scale and production, I also maintain the testing and effectively QA in that sense and monitoring are the same thing.
00:34:32 And that’s something that I do feel passionately about now, how I would implement that on a code level, that is something that other people who are good at things are welcome to do. But you wind up in a scenario if you’re not careful with large scale distributed applications where if you have a specific monitoring function, sometimes the entire thing can be locked up, hung crashed, whatever. And the single component that’s working is the thing that sits there waving, saying everything’s great.
00:34:57 It has to be something more holistic. It has to be integrated in the actual code paths that get used. And that, I think, is what QA and testing needs to be for applications at significant scale. Now, how I get there from writing this silly little function that basically is doing a whole bunch of text string transformations? Not a clue. But that is philosophically where I land.
00:35:19 Well, you also mentioned using at least in the newsletter, but I know people do it outside of newsletter use as well. Is putting the changes on a test server.
00:35:33 Sure. In theory, yes. In practice, no one has time for that. So I just wind up deploying it to my actual environment. Because again, this all lives in Git. And there’s one thing I know, it is certainly not Git, but enough to get to know how to back out of things. It makes everyone feel dumb. Full stop. I don’t care who you are. I don’t care if you’re a Get committer. At some point you’re looking at it going, what the hell is this? That is the power of Get.
00:35:57 But if you have the ability to roll back great data also is one of the things you can’t really roll back in the same way. But that’s why every database I have has point in time restore enabled and continuous backups for the last 31 days.
00:36:09 Okay, so let’s explore that a little bit. So if I don’t test stuff, but I am kind of testing it, I’m pushing it out to production. And then what? You watch to see you monitor it and watch to see if it explodes in some cases.
00:36:25 Absolutely. Now, I want to be clear. This is to write a sarcastic, silly newsletter and I send it out at 730 in the morning every Monday to my audience, which is now 25,000 people and change. And that is great. But if it blows up, it doesn’t work and there’s a production failure, and I send it out at 11:00 that morning instead because I’ve been having to write it by hand. I might get a few ribbon remarks, but no one actually really cares. Even the sponsors of the newsletter are expected to go out in a rough time frame, not an exact time frame. So if you’re listening to this and you work at a hospital, don’t listen to this. This is not for you. I’m talking about things where the blast radius is minuscule and the consequences, frankly, aren’t that serious. Because the absolute full on disaster scenario in my case is that I have to apologize to a couple of sponsors and give them a refund because I just couldn’t get something out. And I’m sorry. That is not that big of a deal in the scheme of the world.
00:37:26 Yeah, well, right, exactly.
00:37:29 I talk about that a lot. Is there’s some software that can land a rocket on Mars, and there’s also stuff like automatic braking systems and heart monitors. And then there’s things that make animated gifs.
00:37:47 Most software is somewhere in between, and you have to decide how thorough do I want to make sure that this is foolproof based on if it goes horribly wrong, what’s the worst that’s going to happen?
00:38:03 And you bring that up? Like the worst that’s going to happen in your case is you just got to manually write it sucks for you for a couple of hours, but nobody else is really going to care, right.
00:38:13 And use good judgment. For example, I have placeholder terms that wind up getting replaced, like to drop, for example, when I write this stuff on the fly that will drop in the link in the title and the rest. And that’s a good way of doing stuff. But that word is the word link in all upper cases. It’s not blistering profanity or incredibly insulting terms or slurs for an obvious reason. And honestly, the reason I know that is when I was very young, before I got into Tech, I was sending out a template email for a mail merge, and I had some snarky, sarcastic joke that was basically insulting. Like, Morning Jerks was how it started. And of course it went out to 100 people and no one noticed, thank God. But it was one of those things like, that was a relatively terrifying morning and it was, oh, okay, never do that again. Got it.
00:39:06 Okay.
00:39:09 This is what it gets down to. Imagine the absolute worst case where out of a comedy of errors, the actual raw snarkdown gets emailed out to everyone.
00:39:19 It is legible, it is readable, it looks strange, and people will ask what the hell happened to me? But no one’s going to look at it and find anything insulting in it or oh my God, this is awful. Or change their view of me in any meaningful way. So as long as that is the failure mode, you aren’t going to go too wrong. And from my perspective, spending all of this time to wind up building out an entire test scaffolding around what is effectively a series of text transformations didn’t seem to make a whole lot of strategic sense.
00:39:52 Yeah. And then so let’s dial it up a notch. Let’s say it’s something that does matter to your business more.
00:39:59 But.
00:40:03 You can catch it in time. So I do know that there’s people that have rollback mechanisms, so they’ll push something out and then they’ll be monitoring, and then if suddenly, if there’s no traffic coming from half the world or something, they’ll realize something’s going wrong and they’ll roll back and you can even have ways to automate that. I think to do rollbacks based on monitoring things, it seems like a good idea, at least, but that’s the infrastructure that you would put in place if the cost of putting that infrastructure in place was less than the harm that would happen if things went wrong.
00:40:45 Oh, yeah. This entire thing cost something like $0.70 a month to run. So, yeah, I can have a duplicate infrastructure in place. The bill is absolutely. Who cares much. But let’s talk about another aspect of the business that we fix AWS bills. One of the things we do for our clients is we wind up importing their cost and usage report, which is a detailed hour by hour API call by API call, billing system, billing output. It’s generally a CSV, although we do it in parquet. And that thing winds up being tens of gigabytes every month. And companies. Rightly. View that as incredibly sensitive data. So our ingestion pipeline and how we interact with that as actual serious security controls around it. And that is very much by design. And I’ve had people build those systems who are actually good at these things because I can Yolo out some sort of testing for building an HTML newsletter. But on this sort of stuff, mistakes are going to show that people are giving you their business with sensitive data. They’re making a tremendous vote of confidence in you, that you’re not going to make them regret that decision. And we take that very seriously.
00:41:51 Yes, definitely.
00:41:53 Because again, it comes down to what is the business use case, what is the worst case scenario and how damaging is that for everyone involved here?
00:42:00 I appreciate it. Truly, I do and respect my sponsors and love their business. And there have been issues before where we send something out, where the link didn’t work through a sponsor thing. And our response has always been to reach out, apologize to them directly, and make it right for them, whatever that looks like. Because when we screw up, you have to own it. But no one is going to be enraged to the point of, well, it’s lawsuit time when we’re getting into a place of not having quite the right messaging go out, right?
00:42:31 Yeah.
00:42:32 But leaking your customer information is not an option.
00:42:37 Exactly.
00:42:38 Sending a crap email is a poor option, but still an option.
00:42:42 Right.
00:42:43 So I’m guessing your input stream for your customer data is more robust and you’ve got some tooling around making sure that that’s secure and safe.
00:42:55 Absolutely.
00:42:56 Okay.
00:42:57 And multiple controls on top of controls. And people who have themselves infosec backgrounds have built this, and that is why we do it that way. There are things that I will build myself, and I will build early prototypes of things. And very often when some of my early bash scripts that I was using here were turned into a suite of internal tools. And the way that that was done was the people that we had working on this looked at what went in, what went out, and they said, great, the bugs. Do you want those to be replicated, too? Absolutely not. Cool. So they put it up there as, like this reference thing of, I don’t know what your family situation is, but I have a daughter who’s almost four, and she draws great paintings, and I put them on my wall because they’re cute, and I love that she’s thinking of me, and it’s awesome.
00:43:45 Okay.
00:43:46 I would do that as a. This is cute. Not as a tutorial for art class.
00:43:51 Yeah. At a College level. And that is sort of how they view everything that I’ve written. Okay.
00:44:01 But you’ll write like a first draft then.
00:44:04 Yeah. Back in the early days of the first two years, I was the only person here, and I was approaching things very differently back then as far as how I would analyze things. Security was paramount. But a lot of it was stuff that I was doing in a much smaller scale. It was something that required lower levels of access, and I wasn’t consuming that giant Bill monstrosity that had the sensitive information was I was approaching it at a higher level. The humbling thing about running a business, as I learned, is that every time you hire someone, they are better at the thing you’re hiring them to do than you are by a lot. It’s a constant process of being humbled, and you get to recognize that. Look in job interviews where you’re explaining how you’re doing something in their area of expertise, and they want to shake you and say you’re not doing what you should be doing, but they also want to get the job so they don’t feel like they should. And you can see that internal conflict warring on their face a little bit.
00:44:58 That’s awesome.
00:44:59 And I tell them, oh, I know this is horrible. No one’s saying this is good. I’m just saying that we’re doing until now, if we were content with it, we wouldn’t even be opening this role. We are. It’s terrible. We know it. And then they relax.
00:45:10 That’s great. Actually, that’s the right way to hire you should be hiring people that are better at the thing you’re hiring them for than you are.
00:45:17 Oh, yeah. And for a lot of things, it’s not like there’s a second option for me. I have the worst at this time.
00:45:21 No.
00:45:21 But I didn’t even think we were going to go down this road. But I’m a strong advocate for the first version of any piece of software being answering the question, can this thing be done?
00:45:36 Not did I do it well, but is it even possible to do this in software? And so I do treat software as a draft process. There’s a first draft and a second draft, and then we can look at security and whether it’s modularized the right way or something. But sometimes for personal projects, the first draft is good enough, and I just live with it. And as far as testing goes, a lot of people I’ve heard people say, what software should you test? And what software should you not? And the answer from a lot of test experts is test after everything, of course. But that’s not really true.
00:46:18 The things that I don’t test that like when it breaks, I always think, wow, I should have written tests for that. But I never think that for the stuff that never breaks.
00:46:31 Exactly. It’s the stuff that’s finicky and broken that gets the attention. But there’s also the question assume something is going to break because everything breaks in the fullness of time. It’s what computers do, it’s what systems and processes do, it’s what people do. We are all just temporarily able. And when things that we have taken for granted breakdown, what is that going to look like? And you’re never going to be able to enumerate all of those things. Trying to enumerate badness is a Fool’s errand. Instead, figure out what does success look like and how do we define what success looks like?
00:47:02 Yeah, actually, both of those. That’s a great point.
00:47:06 I like it. Define what success looks like, test for that.
00:47:11 And also think about what the real worst case scenario is. And can we live with it?
00:47:16 Exactly. Because everything can’t be life or death. Because you’ll die lots of times.
00:47:21 Yeah. Hey, so I really enjoyed talking with you.
00:47:26 People want to hear more about get in contact with you or your podcast or something. Where do they go?
00:47:32 They can visit last week in Aws.com, which points to a variety of different places. I am. And of course, if they’re interested in seeing it amateur, but it’s approaching semiprofessional now level of ship posting. Twitter, of course, is a good place to find me at Quinny Pig Q-U-I-N-Y pig, where I aggressively call out trillion dollar companies because I have no self preservation incentive.
00:47:59 Okay, yeah, I guess. Speaking of calling people out, I’m looking at your diagram and I love that you called the ConvertKit the ConvertKit spam Canon.
00:48:10 It’s functionally what it is creditware. It’s a decent ESP. I don’t want to come across as being unfortunate or crappy to them. In fact, the reason they don’t have a broadcast API and a lot of folks don’t is the abuse potential for that is enormous. And I get that. But again, to be clear, every person in the newsletter has opted in and then confirmed that their email address works to the point where when ISPs get annoyed from time to time and start blocking email from this. Readers actively can’t complain where was it? And it gets reverted pretty quickly. Not many marketing folks have that privilege. I have the incredible privilege of an engaged audience who likes what I write, and I try not to screw up the email every week so they continue to like what I write.
00:48:51 Yeah.
00:48:52 Okay.
00:48:53 Cool.
00:48:54 Well, thanks so much for your time today.
00:48:57 Thank you. It’s a pleasure to be here.
00:49:04 Thanks, Corey, for this most enjoyable conversation. Thank you to PyCharm for sponsoring the show save time use PyCharm especially if you write tests and of course you do right check them out at testandcode.com PyCharm that link is also in the show notes at testandcode.com 149 thank you also to Patreon supporters join them@Test And Code.com support that’s all for now. Now go out and test something.