Michael Kennedy joins the show this week to share some of the tools he uses during development and maintenance. We talk about tools used for semi-automated exploratory testing. We also talk about some of the other tools and techniques he uses to keep Talk Python Training, Talk Python, and Python Bytes all up and running smoothly.
Transcript for episode 146 of the Test & Code Podcast
This transcript starts as an auto generated transcript.
PRs welcome if you want to help fix any errors.
00:00:00 Building any software, including web apps and APIs, requires testing. Of course, there’s automated testing and there’s manual testing, and in between, there’s exploratory testing that’s aided by automation tools. Michael Kennedy joins the show this week to share some of of the tools he uses during development and maintenance. We talk about tools used for semiautomated exploratory testing. We also talk about some of the other tools and techniques he uses to keep Talk Python Training, Talk Python and Python Bytes all up and running smoothly.
00:00:46 Welcome to Test and Code.
00:00:56 On today’s episode of Test and Code.
00:00:58 Michael Kennedy joins me. Michael Kennedy, of course, is my co host on Python Bites and also runs his own podcast, Talk Python to Me. And he’s got a little bit of an Empire going on with Talk Python training and all sorts of stuff that Michael is up to. So, Michael, welcome.
00:01:17 Hey, Brian, it’s really good to be on your show. Normally we get together on Python Bites, but sometimes you’re on top Python as well. So it’s great to be over here at Test and Code.
00:01:25 Yeah. And it’s wonderful to have you. One of the things I don’t really deal with websites as much as you do. I thought it’d be fun to talk about how to test websites and how to keep them, not just websites, but web services and different things and how to keep them running.
00:01:41 Yeah, absolutely.
00:01:43 I think it’ll be a lot of fun. I think there’s a lot of stuff that you hear about the straight down the line. These are the way you write your tests for websites. But there’s all these other things that you got to do to build and develop and sort of test them as you’re creating them before you can put them out in production, before you actually have a client to talk to it and things like that. So I thought it would be fun to touch on some of those ideas that kind of round out the testing story of web apps.
00:02:11 Yeah. So take it away.
00:02:13 All right. Well, the first thing that I wanted to talk about, if you’re building a web application that does HTML that’s intended for humans, it’s pretty easy to get started and look at it. Right. You fire up your browser, you click the little link when you launch the app, it says your app is running at local host, pulling 5000 or whatever. You click on it and you click around.
00:02:36 And that’s pretty straightforward. But if you’re building APIs, a lot of times that kind of exploration is harder. Right. The API is just returning something you could just put in the URL, and that probably works just as well in the browser. But many times you are using the Http verbs that are tricky to make the browser do. For example, there might be a method that has a put in Http, but how do you do an Http put in your browser? I don’t know.
00:03:05 You can technically go into the JavaScript section and start typing it, but then that’s not really working with your browser. Right. So one of the things I wanted to talk about and sort of testing out your API, getting different use cases and scenarios going before you maybe have some kind of client written in a much more exploratory way. Right. I’m developing it. I want to see if this works. I want to try that and kind of go back and forth, maybe even before I solidified this into some kind of unit test is to use something like Postman. So are you familiar with the Postman app?
00:03:39 Not really.
00:03:41 A Postman is super cool. So what Postman lets you do is it’s all about calling APIs. So what you do is you get a little dashboard and you put in a URL and there’s a drop down, it says get URL, but the drop down, you pull it down and it says things like a huge list of Http verbs. That probably would surprise many people to know there are that many there like. So get Put, Post, delete, patch, et cetera, et cetera.
00:04:07 And then you can expand out a little section of what’s going to be in your request. Instead of just sending a URL and a query string, you can actually say, well, I want to specify query parameters and maybe some off stuff. And I want to put this value in the header like this API key in the header and I’m going to put you maybe a JSON body into the body and so on.
00:04:30 And then there’s even automated testing scenarios and like scripting stuff you can set up to do. So.
00:04:36 So is Postman a paid service or a tool?
00:04:41 It’s paid, but it’s also free.
00:04:44 Okay.
00:04:45 It’s like a premium type of thing. You can get it for free and work with it, but then you can get like a team version that you pay for team sync features and higher level features and whatnot.
00:04:57 Okay, so it still seems like I have to deploy it first.
00:05:01 You’ve got to at least have it running locally as a testing. You could just put local host in there and just click away. The other thing that I end up doing a lot is I kind of want to remember these are the basic things I’m doing. So if like I get a message from something, hey, I’m trying to use your web app, right? So your mobile app. So for talk by the training, we have mobile apps to let people take courses. Like, I’m trying to use this and I had this problem. I can just fire a postman and it’ll let you create these little collections and like libraries of API calls. So I can go over there and say, okay, what I want to do is I want to try to call this function this API endpoint with this header and it says it’s this version of the application on this platform so it behaves potentially differently. And I can just have those little things and just drop in there and just go, oh, let me just try that API endpoint real quickly.
00:05:51 Push the button. It’s already got the right JSON body saved as the input data for my little scenario I want to play with and whatnot. So it’s a really cool way to diagnose stuff quickly. I can jump in like, okay, this is how this part is supposed to work and I don’t have to fire up the app and all that. I can just make sure it’s working.
00:06:06 Do you do that against the live version at that point then?
00:06:11 Yeah, probably. Sometimes. Not always. It depends if it makes some kind of change. It might be I wouldn’t do it against the live one on behalf of someone else, but a lot of times it’s like I’m just trying to log into this thing and get this data and it’s not returning what I expected or whatever. So you could just go play with that? Yeah.
00:06:27 It lets you create these cool little libraries. So, yeah, that’s the first thing I want to just throw out there is if you’re interacting with APIs, something like Postman. I have a little desktop app, but I think there’s a web browser version as well. The one thing to be aware of, I’ve had people say, Well, I didn’t want to install the app, so I just went and logged into the website and I did some stuff. The security around what you can do on the web one is stricter, like you can do less. So, for example, some sites don’t allow you to call JavaScript methods from outside the domain and stuff like that, cross site scripting protections and whatnot. There’s no restriction like that on the desktop app, but in the website, you are, of course in a web app which has more restrictions. So just heads up on that.
00:07:10 Okay.
00:07:14 Thank you, Lenode for sponsoring this episode. Simplify your infrastructure and cut your cloud bills in half with Linode’s Linux virtual Machines develop, deploy, and scale your modern application faster and easier. Whether you’re developing a personal project or managing larger workloads, you deserve simple, affordable, and accessible cloud computing solutions. Get started on Lenode today with $100 in free credit for listeners of test and code, you can find all the details at lenode.com. Testingcode Lenode has data centers around the world with the same simple and consistent pricing. Regardless of location, choose the data center nearest you. You also receive 24, 7365 human support with no tiers or handoffs. Regardless of your plan size. You can choose shared and dedicated compute instances, or you can use your $100 credit on an S three compatible object, storage, managed Kubernetes, and more. If it runs on Linux, it runs on Lenode. Visit Lenode.com testandcode and click on the Create Free Account button to get started.
00:08:20 So let me give you another scenario this is something that it’s so simple and yet it delights me so much when I see this in action.
00:08:31 I’ll start with the more benign example. If I’m developing a web app and I’ve come up with a new page like, oh, here’s the admin page I’m building that allows us to do X-Y-Z and I want to share that with the team. I’m thinking about committing it back, but I want to have a conversation with a couple of my team members about how it works. Maybe my manager.
00:08:56 How do you do that? Right.
00:08:58 If you don’t want them to completely set up maybe a change to the database and all that stuff locally, it can be kind of a pain to say just run that and quickly give me some feedback or what do you guys think of this stand up meeting? Here’s what I did yesterday, but I got stuck on this. Can anybody give me some ideas?
00:09:15 So one thing you can do is you can do screen sharing, let everyone see it, but then they all have to be present at the same time. That’s kind of annoying.
00:09:23 So you could let them connect directly to your local machine through their web browser and actually just try out that page. And that would be awesome. Except for router firewalls, net routing machine firewalls. Your local web server probably listens on local host and it doesn’t even let you have connections from the outside world, hopefully. Right. Because of the Internet as an evil place. Right. All that.
00:09:51 So what you can do is use something called Ingrock. Are you familiar with Ingred?
00:09:55 Well, I’ve heard you talk about it a few times on the show.
00:09:57 So Ingrock is super neat. So what it does is it’s a service that has like a public web server.
00:10:05 So I could go and install it and just go to the command line, type in Grok space, Httpspace, 5000, something like that. Right.
00:10:14 Okay, let me just type it in real quick so I can give you more detail. So what it does, it will come up with an Http and Https public URL that you can use, but then it uses SSH tunnels from your local machine back to their server. So if you give that URL to somebody, they do a request, it goes to Engroc. Ingrok uses an SSH tunnel to talk to your local Dev machine and send that back out to you. Basically, regardless of firewalls, you can put APIs, you can put websites and whatnot temporarily on the Internet or even permanently, I guess if we just left it running. But it’s not really, I don’t think intended for that.
00:10:58 It is super cool. So what you could do is just say, hey everyone, here’s the URL, click on this page and tell me what you think. And it’s literally your exact version running there. You don’t have to try to get some database migration set up in a staging environment or some weird thing like that. So that’s kind of the simple case, but then you get to the more interesting use cases.
00:11:17 Well, wait a second. I’m a little concerned about this. Is there security problems with me opening up basically a web service on my laptop or something?
00:11:29 Well, basically, whatever service you’re exposing, you should just assume that it is on the Internet. Right. Because it doesn’t have any authentication generally. On top of that, it’s some random URL with the huge, long UID type thing. So it’s not likely people are just going to start jumping on it right away, but at the same time.
00:11:51 People could request it and you’ve probably got some silly mock database or something running.
00:11:58 Yeah, ideally. I mean, that’s a good point. Or you could have actual authentication in your application that just happens to also kick in locally, maybe. Right. But you do want to think about that. That is definitely something.
00:12:11 And that’s interesting.
00:12:14 We didn’t want to do screen sharing. So cool. You can actually check out the thing live on your local computer because it does interesting animations or whatever. But then it goes beyond that. So one example, I was talking about the Talk Python training web apps.
00:12:29 Well, they’ve got to talk that API. And when we were developing the Talk Python apps, I was working with another guy who is more of an iOS developer, mobile developer. But I was building the API side, he was building the app side, and we needed to coordinate. Every now and then there’d be some kind of errors. Like, I’m really trying to do this thing with the API and it’s not working. I’m like, oh, man. All right, well, let’s try this and then try that. You can look at the logs on the server and stuff, and it’s just not all that helpful. So what we did is said, well, let’s run an ingrok session and go to the mobile app and change it to say that the API endpoint is that in Groc destination, not trainingapi, whatever. Right?
00:13:16 Yeah.
00:13:16 And then I would just set a break point in PyCharm. I said, okay, go click the button that was causing the problem. He click it. It would hit a break. He was in, like, Europe. It would hit a break point on my machine in America. I would look at the values, and I stepped through like, oh, you’re passing as a string. That has to be a number. It’s like quote for instead of the number four. That was a problem. And then he would fix it. And how awesome is that exchange?
00:13:40 Yeah, they’ve just got it. He’s debugging it in his app, right?
00:13:45 He’s debugging the mobile side, I’m debugging the API and point side, and we’re like stepping through it on both ends.
00:13:51 Nice.
00:13:52 Yeah, that’s really cool. And then the third one is I was doing some stuff with Twilio, and Twilio Studio has this really cool workflow designer that says if a message comes in, ask them this, and then if they say that you can do this other thing. And one of the steps is I need to call this endpoint to decide what options I have to give them. And that was coming from this app, this flask app I was working on. Yeah, same thing. Create an ingrok URL, put it into the Twilio cloud, send a message with WhatsApp it makes it to that step, and then it hits a Breakpoint back on my local machine as I’m developing it before it’s ready for production.
00:14:29 That’s so cool.
00:14:30 Yeah. So this one just delights me to know. It’s super simple idea, but it’s really well done.
00:14:35 And then you’re really just I mean, there’s not a lot of security risk there because you’re going to tear down the narrative session afterwards, right?
00:14:43 Yeah, exactly. It lasts for a few minutes and it wasn’t really sensitive data.
00:14:47 Yeah. Okay, that’s neat.
00:14:50 How about things like fast API and other things have this interactive mode. Do you know what I’m talking about?
00:14:56 Yeah. So with Fast API, if your app is at server.com you can go to server.com slash. Docs. Docs and it pulls up an API and open API documentation spec that shows you all the endpoints and then you can go in there and test it. But that would be sort of equivalent to postman. Okay, you could do that, you could go in there and type it in. But the problem is if you’re done testing, you put away your browser and walk away, or if you’ve got three scenarios to test for one end point, you’ve Typed one in and the other two are gone.
00:15:32 Yeah.
00:15:33 Well, you want to go back. I also want to pass this header and I want to say that my user agent is this versus that’s more manual. It’s more manual and it’s not permanent.
00:15:47 With Postman, it will record, like here’s a history of all the requests you’ve sent, and here are the ones that you’ve saved to your catalog, even though there could be three that go to the same URL. But this one’s anonymous, that one’s logged in and you could do the same thing, but you’ve got to redo it from typing it. It’s like working in the REPL versus working in the Python file. Like you could do it in the reptile, but you wouldn’t want to do it very long in the reple or very many times.
00:16:09 And this is something we actually haven’t explored too much on the show before on this podcast is there’s fully automated tests, there’s fully manual testing, and then there’s this in between thing of using automation to help you do sort of exploratory manual poking and being able to save things.
00:16:34 It sounds like postman falls into this middle category.
00:16:37 Right. So much of the stuff I wanted to cover was like that middle ground because I feel like it doesn’t get talked about a lot, right?
00:16:43 Yeah. Okay, so one of the things that the simple thing that you brought up before the show was this idea of testing your site map. Can we talk about that a little bit?
00:16:53 Yeah, absolutely.
00:16:58 If you were to think of the pure TDD world of type of testing, this would probably get you bad looks. They’d be like, oh, come on, Michael.
00:17:10 Right. You would get some of that. But at the same time, what have I told you? You could test like the majority of your app really, really easily and get kind of does it catch on fire and explode or does it mostly sort of work before you start answering precise questions? Like if I pass this data over, do I actually get the right location back? Proper unit test. Right. So one of the things that you can leverage is a real simple idea.
00:17:40 What if I were to request every page on my site and determine does that page load without crashing? Or does it 500 server error or four or four not found error or something like that. And it sounds really simple, but if you actually think about the ways that many websites break, they don’t break like, oh well, that little image is now hidden or there’s no data in this table, they crash. Like Wham could not load this thing. This syntax is broken.
00:18:08 This model doesn’t match the tables anymore. They break hard most of the time. And what that means when they break hard, that means the page serves up a 500 error. Right.
00:18:19 Or it’s just not found and it’s a 404. So what if you just requested every page on your site as one of your tests?
00:18:27 This is more of an integration test side of things, but it’s a really quick way to just go. Let me try this and make sure things are still alive.
00:18:35 Okay, great. Well, how are you going to do that? Maybe for a small site, super easy talk by the training site has like 4000 pages. That’s a lot of pages to request. Right. How would you know what page to request? Well, almost all of our sites have sitemaps atsitemap XML because we want to give that to Google and increase our SEO. Right. Like one of the key things to getting your site well ranked in Google is having a site map that says, hey, Google, here’s every page that I would like you to pay attention to, even if you don’t find a link to it. Right. Okay, well, here’s the test request, the site map, parse the XML for each URL requested response raise for unexpected or whatever that one function is. That raises an error if it’s not a 200 or 201 or 302.
00:19:26 Oh, yeah, right.
00:19:28 Anyway, that’s my little tip trick to sort of do a quick catch all against maybe those pages that don’t have tests or parts of the sites that you forgot to write tests for kind of an integration test more than anything but an easy one.
00:19:44 I actually love this idea, even if you’re in a real hurry. And like, for instance, if I’m doing a migration from one domain to another or to a subdomain or something like that, I can easily grab this site map, save it somewhere, maybe do a search and replace to change the domain or change the subdomain and then Bam, I can use that as a test when I finally get there to make sure everything’s still there.
00:20:18 Yeah, exactly.
00:20:19 And it’s a big hammer that probably will fail until you get it right. But it’s also really easy to generate.
00:20:27 Yeah, it’s super easy. It’s probably like 20 lines of code and you’ve tested it. You’re near 100% code coverage in a really crappy, wonky way that’s fully more integration stock and it’s probably going to talk to the database and whatever it’s going to do. But at the same time, imagine you are hired as a consultant. Like, hey Brian, I need you to come work on this project. And yeah, this is kind of a crummy site. We don’t have tests here. Please make it a little bit better and add this other feature. If you were given some piece of code, some website that was fairly complex and it didn’t have those tests and you didn’t have that much time, here’s something you could put in place to like go here’s a little tiny safety net as I tweak on it.
00:21:10 You want all the stuff for SEO anyway, right? So you probably have it there. You might as well leverage it.
00:21:16 Yeah. Okay, so that’s the easy one or semi easy?
00:21:20 Yeah, semi easy for sure.
00:21:22 But that is one of those your sites up or your services up and you’re making sure it stays that way. I’m occasionally going do maintenance on my application. What do I need to do to keep it alive?
00:21:34 One of the things that I think is really empowering about having proper unit tests, about having continuous integration.
00:21:42 If you’re working on a team or at least having some local tests, you can run yourself like this site map thing before you send it up to make sure it’s kind of hanging together and also having some simple way to deploy your site.
00:21:57 There’s deployment such a spectrum of things, but a lot of times I’ll go to places and I can’t even fathom what is happening at this team behind the scenes when they say we’re scheduling some upgrades. So we’re going to be down for 4 hours and then we’ll be back. Here’s our notice. You’re like, what do you mean you’re going to be down for 4 hours? Like this is the internet. I don’t even understand.
00:22:25 You’re on vacation. What is happening, right? Like, how could it possibly take 4 hours to do whatever you’re going to do? And so one of the things that I’ve ended up doing on my site is setting up a bunch of scripts that do various things. So for example, running this also applies to Python Bytes the server as well. But we’ve got some servers in Digital Ocean running on Ubuntu, and for the servers they’ve got the main version, but they’ve also got like a smaller, cheaper $5 server that is. Well, if I have to say, reboot the main server because the Linux kernel has got a security problem and it just has to be rebooted to fix it.
00:23:05 We can flip the IP address within the cloud hosting over to the failover server and it might be tiny bit slower, but it’s fine, right. And it’s $5 a month. Who cares? Just let it run there and do its thing. So I’ve got a script that’ll do something like that, right? So I can say switch from the main production over the other and reboot it and do the work on the machine. And whatnot case?
00:23:27 Like I said, it needs a kernel pack or something.
00:23:29 So that’s nice. Also complicated, but it’s actually Super, super easy. There’s like an API for saying Where’s the public API assigned to which server you can just like.
00:23:43 It’s like 15 lines of Python. It’s not bad. That might be less. Actually, it’s a script that just calls him the Python. Right.
00:23:52 So that’s all well and good. You might want to do something like that. If you’re working on Heroku, they’ll do that automatically. If you’re working on Kubernetes, they’ll do that automatically. That’s part of the spectrum.
00:24:03 If you’re working in a place that doesn’t let you do that, then you might just have downtime, I don’t know.
00:24:08 But then you can start to put these building blocks together like, oh, you know, that web hook idea or the narrative idea I told you about. When you check into Git, you can go and set a web hook to call back.
00:24:21 Right? So any commit that happens on GitHub, you can say for this repository, call this other place and you can check it out with ingredients and program against it. And what not. So added a little thing that says if there’s a commit to a certain branch and it’s changed a Python file, not just a text file or some other random thing, redeploy the website. Which means use that little script to switch it over, tell it to restart itself. It actually does a bunch of other things. So it starts pre warmed a lot of times. What can happen is certain pages are complicated to load up the first time, then the Caching is in place and whatnot. So it’ll do all that stuff before it brings it back online.
00:25:03 Okay, so now I’ve got a little script on my computer working in PyCharm to go to the terminal type prod P-R-O-D. It’ll switch branches, push all the changes up to the one branch that will fire the GitHub web hook, which will bring the thing over. You’ll switch it over to the failover server, which will refresh it. So it’s just all these little steps that you can take. And in the end, you end up with stuff where it’s like, oh, I just Typed this word and then with zero downtime, we have a new version up in 1 minute.
00:25:30 So there’s a lot of neat little building blocks that are sort of utilities that are useful for you. But if you piece them together, piece some of these ideas together and you end up with a cool thing. You don’t have to put messages like, Please don’t come back on Sunday because we’re updating the site.
00:25:46 So have you ever forgotten to switch away from your failover server and just left it on a slow one for a while?
00:25:53 I think I have once or twice. Not very often. I have this little dashboard thing that has a bunch of boxes and it’ll say, what’s the response time of the site? How many users have logged in today, how many videos have been watched, what was the last one? And so on, like all those kinds of things. And one of them is, what deployment mode are you in production or failover? Okay. Whenever they wait a minute that says failover, it’s been that way for a couple of days. It’s good enough. It’s just that you wouldn’t want, like, a huge burst of traffic on a $5 server, but still, it’s okay.
00:26:23 Okay, nice. But sometimes this goes wrong and it doesn’t all work, right?
00:26:28 Yeah. One of the challenges that we run into is if you’ve got to update the database server, we don’t have clusters of databases, so there’s a very small amount of downtime for that. And you also just want to know, like, for example, one of the things that happened one time was I talked about this little IP addresses, the public IP address in digitalization. It’s called the floating IP, and it can be assigned to one machine or the other machine. Well, one thing that has happened that is so hard to test, I don’t know of very many tests on the machine that I would really know how to do this because the machine is its own contained thing. It has its own IP address, but then there’s this sort of virtual IP address back to itself. Right.
00:27:12 One thing that happened is there was a problem with the API at the cloud level, the cloud hosting level, where that became disassociated with either of the servers. And both servers, they thought they were fine because they were both running and doing their thing. And they asked, how am I doing? Like, I’m fine, but from the outside, nobody could get to it because the DNS records where is this domain? Go to the public IP address. And it didn’t make its way through the networking stack at Digital Ocean. I’m just like, how was I supposed to know that this was a problem?
00:27:52 Well, you got to have cake.
00:27:55 Cake.
00:27:56 You got to have cake. Status cake.
00:27:59 So I set up statuscake at statuscake.com, which is pretty cool.
00:28:05 And another option might be better up time. I just happen to have set this up with status cake when the problem was there. And I think better up time actually looks a little bit better. I mean, they claim it in the name, but anyway, the idea is you can just set up services that will call certain things. So like status cake, I’ve got, I think, five or six different things that are continuously monitored. The homepage of training Talk, Python Bites, FM, but also the RSS feed of the two podcasts, because those are like the lifeblood of our podcast. Right. The spice. If the RSS doesn’t flow, the MP3 don’t flow. It’s a problem. So I said I want a specific alert that if anything happens to the RSS feed, let me know right now.
00:28:54 And another one I did was against this training API. And what it does is it just does a couple of things like, hey, try to go get some free assigned user from the database, try to go to the video server, let’s say in Germany, and try to watch a video and just make sure that all those things still work. And then tell status Kate. Yeah. The API is okay.
00:29:14 And then status que calls out every five minutes or ten minutes or whatever it is. And if it ever doesn’t work, it sends me a message and says, hey, can’t watch a video in Germany. That might be something you want to pay attention to.
00:29:25 Well, okay, one of my questions, there’s a whole bunch of like software services like status cake and better up time and things that you can run against your service.
00:29:44 What do you think? Something like this seems reasonable.
00:29:49 One of the questions, I’ve got a few questions around it. One of them is does it affect your it’s obviously calling stuff and requesting stuff. Does it mess up your statistics for your download counts and things like that?
00:30:02 No, because it’s not actually requesting any of the MP3 for the podcast.
00:30:07 Okay.
00:30:08 And for the video plays it’s doing it at a lower level than the stuff that does analytical tracking. So I don’t think so. It doesn’t run any JavaScript, so it wouldn’t count any like Google Analytics type stuff. We don’t even have that. But if it did, it wouldn’t do that. I think it would be okay. It is something we’re thinking about, because if what you’re having to do is triggering some kind of analytic record, like here’s a redirect to my sponsor and I want to tell them how many times it was clicked and like, oh, well, we’re testing that. Sorry, you’re really popular.
00:30:41 Maybe that’s good. Maybe you want to tell them that. No.
00:30:45 Yeah, it’s worth considering, but it’s not for me.
00:30:47 Yeah. And then you do things like crash monitoring and stuff also, right?
00:30:52 Yeah. Oh, man.
00:30:54 So I use Robar and Century. I had one and the other and they both kind of are really similar, but they’re not the same.
00:31:04 I just left them both in because I get notifications from both.
00:31:09 Yeah. So the idea is, if your website crashes, would you know about it? Because the message usually just goes back as some kind of server error to whoever requested it. And maybe they’ll send you a message. Maybe they’ll say, hey, I was trying to do this thing and it stopped working. Or maybe the reaction is, Whoa, this site is unreliable. I don’t want to log into it and create an account and I’m out of here. Right. And you just wouldn’t know.
00:31:34 So you probably want to know what you can do is set up something like roll bar or Sentry or both. They both have free levels and then paid levels if you want to capture more history, more errors and whatnot.
00:31:47 And they capture insane amounts of data. Right. So if somebody goes to a page and has a crash, it’ll say they were on this browser. It was this server, like the production server or the failure server. And the story I told you, it was this version of Python. It was this version of the Web framework.
00:32:07 Here’s the call, the trace back, here’s the call stack, here’s the headers, here’s all the cookies they exchanged. And you can go further and you can actually go as part of the API, as a part of middleware in my website, I hook into these APIs and say if there’s a user session active. Oh, hey, by the way, Century, the current user is so. And so if there’s a crash, it will put in the email like, Joe Smith had a crash trying to watch this video on Tuesday on Windows with Firefox 86. And here was the query string.
00:32:43 And more than once I’ve had somebody who was trying to buy something and there was a whole bunch of changes I made to the checkout flow for, I think for Black Friday or something stupid. I was getting myself out over my skis and anyway, I broke the team check out for a certain type of bundle. I think it’s something really esoteric that I forgot to test and it crashed. And I’m like, oh my gosh, this is like so bad. But Luckily I had all this in there and so I had the email address to the person, I’m so sorry this crashed. I saw you were trying to do this purchase earlier.
00:33:15 We fixed it and I just want to apologize and let you know it should be okay now they’re like, oh my gosh, I was going to write you later. This is so amazing. I can’t believe you wrote me a message. When was the last time you were on a website or an app and it crashed? And then they just contacted you, said, oh, sorry, you had a problem. Really sorry. We fix that for you. Like never, right?
00:33:32 No.
00:33:33 Yeah.
00:33:34 You get an email that says this user had this problem.
00:33:38 Right. And if you cut the capacity to help them, it’s relatively easy. These APIs are incredibly simple and it’s huge value.
00:33:46 That’s pretty cool. And you may have gained a sale that you may have lost.
00:33:51 Yeah, exactly. They did come back and buy it. I don’t know if they would have. Not otherwise, but certainly it worked out in the favor.
00:33:58 So something like if you’re actually providing a service that you make money off of, something like this is definitely worth whatever you’re paying for it then.
00:34:08 Yeah, exactly. Especially since I’m paying nothing, you get like 5000 errors a month before you start to pay for it. And to be honest, if you’re having more than 5000 errors a month, you’re having a problem. Right. You really just need to like stop feature development and go, okay, why are we having 5000 crashes a month? This is too many. Even if you have a lot of users, you should like log it, send them a proper response, not crash the server.
00:34:36 Well, maybe your testing strategy is deploy and pray.
00:34:40 Yeah, user testing.
00:34:45 We have beta testers. They’re called customers.
00:34:48 Exactly.
00:34:51 Cool. If you’re running Century, a roll bar or something, crash reporting, does that slow down your site?
00:34:58 Maybe, but in a very imperceivable way. So basically the way it works is when the web server, the Web framework is making a bunch of processing. So I got this URL, I’m going to hand it over to the routing system. The routing system says, okay, that’s going to go over to this view method. There’s one other function call that’s like between those two, like a decorator almost around them. Right. And what it does, it says try call the thing that’s actually happening, accept, handle the error, send it off to wherever it goes and then return the response. So it’s basically a function call and a try accept. If there’s no error, it’s just the try block. So it’s really light in terms of the overhead.
00:35:41 No, not like re instrumenting stuff or it’s nothing along those lines. It’s basically just a one function decorator wrapper around a request.
00:35:50 And you were saying you used status cake. Does that tell you response times and stuff?
00:35:57 It does actually gives you a really cool graph over time of what your response time average response time of this is. And it’ll do it from different locations. I can think. The free tier does three different servers around the world for each test. So it’s not just your site is up at the edge of your cloud or whatever. It’s like we can get to it from India, we can get to it from North America, we can get to it from Europe.
00:36:20 And I know that. I don’t know. It was some months ago you started looking at the speed of your site of your Empire and tried to speed things up.
00:36:34 Was it some of these status things that caused you to do that, or did you just what was your answer?
00:36:39 No. There were two things that made me go down this path.
00:36:44 One, we covered an article called Twelve Requests per Second on Python Bites, which made me think, well, that sounds it’s about Python frameworks. And it was about basically, like, there’s people that talk about all this amazing performance and my real web app gets twelve requests per second at the top end.
00:36:59 Should I be happy or not? Basically, I’m like, what does mine do?
00:37:04 They wanted to answer the question of, well, what is the peak number of users I could have? It’s had a lot of users from like, Pumble bundle deals and whatnot. But where was it crashed? Like, we’ve all had these experiences where if something happened on a popular day on some site and like, the site goes down, I can’t remember the last one I had. It was some consumer thing I was trying to buy. But the most egregious one of these has to be healthcare dot Gov or whatever that thing was, right?
00:37:32 Yeah.
00:37:33 So you’d like to know, can my site handle 100 users? I don’t know 100 concurrent users requesting stuff is the infrastructure we set up good. Do we need to add a Caching layer or they’re just going to handle it just fine.
00:37:49 So one of the things that I was using to answer that question was locust. Io. This is like a Python goodie for sure.
00:37:59 You and I talked about this on Python Bytes a little bit, but what a neat framework. So what it is is it allows you to do load testing of your site. There’s lots of those, but it does it in a really interesting and Python friendly way. So what you do is you create a class and a class is a bunch of methods. Each method is a thing to do to your website, and it’s very test driven feel. So it has like a set up and a tear down type of thing. Okay. The set up might be go to the login page and log in, because the next thing you’re going to do, you want to make sure that the users logged into the site because that might require authentication to get to a certain part or something, right?
00:38:37 Yeah.
00:38:38 So you could do things like that kind of set up stuff and then tear down. I guess it could be log out or delete my account or something.
00:38:44 Anyway, you would go set there’s these different methods. And you’d say on this one, I want you to request, like, let’s talk about the podcast site.
00:38:51 Right.
00:38:51 I want you to request the episode page where you see all the episodes. And then I want to have another method that’s going to request the details about the podcast. And then I want to have another one that’s going to request the RSS feed, and then another one that might try to download the MP3, and then one that goes to the home page. Who knows? Yeah. And then you can actually say, well, 70% of the time, I want you to go to the episode page, 25% of the time, go to the episode list, 5%, go and download the MP3. And you can also put in as part of that set up, you can put in a timing like, well, the average user will click a link between five to 60 seconds on average per page, because the average user isn’t like obsessively, just control a five year site or command R your site, like, as hard as they can. Right. They go into a page, they look at it, and they go to the next. So you can just set it all up into a Python class, run Locus, and it opens up a web UI with graphs and errors and real time views of all of this stuff. So you just give it that file say you tell it this server, this many concurrent clients, this many connected users go and it’ll tell you response time, number of errors. If you want to see the errors, you can log in and see, go over and see the details. You get cool graphs of performance over time. Does your response time to grade? When does it start to degrade as the number of users? So anyway, for me, I found that we could do about 125 to 135 absolute concurrent requests hammering on our server, which with a little breakdown, represented something like 20,000 to 40,000 users on the site at a time.
00:40:38 Okay.
00:40:39 Which for probably $60 in infrastructure, that felt like pretty good.
00:40:46 Yeah.
00:40:49 Conceptually, you could run this against a test server.
00:40:53 Yeah, absolutely. But the thing is, so much about the performance of Python Web apps has to do are you running in a production server or in a development server? Because the development server, there’s just one thread, one process. But like, in production, you’re running microwhisky or G Unicorn with a bunch of workers spread out in, like a multi threading. Like, the performance is really different. But, yeah, you should definitely start on your machine and then think about branching out.
00:41:19 Right. But you could also do have a test server that’s almost like a real server.
00:41:26 Exactly. That’s a good idea, because you don’t necessarily want to take down the live website for everyone.
00:41:31 But on the other hand, one of the reasons why I like talking to you about this and not like a huge company is because you’re, like, not just a DevOps kind of person, you’re the CEO and the designer and the things.
00:41:49 Yeah.
00:41:50 From top to bottom. Right.
00:41:54 Yeah. I can do this stuff and not get in trouble. I could be regretful for it, but I’m not going to get in trouble.
00:42:01 I can do these little experiments, but at the same time. One of the things that I quote a lot is you talked about something on our show together that was you’re not Facebook, you’re not LinkedIn, you’re not Google.
00:42:13 The way you build your site is not those things. And so much of this sort of like this advice and these kinds of things are like, well, here’s how we’re going to set up continuous scale at such and such insane level.
00:42:23 Well, you’re probably not at that level. So how do you still get stuff done and make these decisions when you’re at a less insane level?
00:42:31 Right. But it used to be that a single person company or maybe I just wrong about this, but I guess when I just had a little website or something, WordPress site or something, that if I could build up some sort of revenue flow, but I would not do any of this extra stuff around it because I either didn’t know how to or it was expensive to run some sort of service. But now, like you said, some of these things, they have a free tier and you can just run them for free to monitor your stuff.
00:43:05 Absolutely. I talked about it being $60 in infrastructure. That’s a $40 database server and a $20 web server. But I’m pretty sure it would be actually totally good to go $10 on the web server, $10 on the database server. It just happens to be there’s other stuff going on that I want to account for. There’s nothing else in this whole conversation that I’ve paid any money for. I mean, that’s $20 a month, right? It’s incredible. I remember in the early.com days, I’m guessing, what is this, 99 2000, I went to this conference that was put on by because Oracle, Microsoft, and one other thing, they invited me over to this, like setting up an ecommerce website. It’s just so easy. We’ll get it all set up for you. It predates a lot of the clock computing. So they’d sell you the hardware and then the SSL load balancers and then all the things. And they’re like, at the end, it was like for just $100,000, you can have an ecommerce site.
00:44:10 It’s just so insanely different than now, right? It’s so crazy. Also, it’s worth pointing out that a lot of this stuff hasn’t kind of been building up over four or five years. Right. When it started out, it was much simpler. And I’m like, this thing is kind of a pain point. How can we address that?
00:44:27 It was down. I didn’t notice it. How can we address that?
00:44:30 Right.
00:44:30 All these things have been layered on slowly over time, but it’s kind of like a look back.
00:44:35 Right?
00:44:35 I wouldn’t do this from day one. Like, I would start I would just get something working and like, slowly build towards it.
00:44:41 I like the idea of having something like crash reporting and stuff because I assume you can take a weekend off and not worry about stuff without like, if your livelihood depended on it, even if it’s just your side gig and your vacation money depended on it.
00:45:03 I’ve noticed that I check stuff. I’m like, is it still working? And I’ll go around and manually check things. You don’t want to be doing that while you’re on your weekend. So some monitoring is a good idea.
00:45:14 Yeah, it absolutely is. And it definitely gives you that sort of comfort level. Like, okay, I haven’t got any alerts from any of the systems. Everything must be fine.
00:45:25 Actually. Thanks for coming on. And I’ve also enjoyed actually just coming along for the ride. I remember a lot of these changes. I remember you setting up just a simple site and then starting adding videos and the complications with all that. And I remember getting a message from you saying, I just am now deploying video from three different locations on the Earth.
00:45:53 Yeah. It’s like twelve different video servers that just automatically replicates to I’ve got it working on now. It’s pretty sweet.
00:46:00 Yeah, that’s pretty cool. And when we were talking about the $60 thing, that’s something I think about. Not everybody is deploying and serving so much video content.
00:46:11 Since you’re doing a lot of video content and you’re serving that, that’s a lot of bandwidth that most people aren’t doing with their sites.
00:46:18 Yeah.
00:46:22 It would be probably a busy month, but it wouldn’t be outrageous to say we had $1,000 bandwidth bill for the month.
00:46:28 Wow.
00:46:30 At AWS. Yeah.
00:46:32 But that’s not now, right?
00:46:34 Yeah, that’s still now.
00:46:36 Okay.
00:46:37 All the other stuff I’m talking like servers and database servers and all that. But to actually deliver the bandwidth, not the podcast, the podcast is for free, but the video stuff, because I want it, like super close to people in the way it set up. Yeah, it’s like that. But it’s okay. People are paying for it. So it’s fine.
00:46:55 If people don’t consume a lot, that means they haven’t bought a lot. But if they consume a lot means they bought a lot of courses. So it scales perfectly with income. So it’s fine.
00:47:03 Okay.
00:47:05 That sounds terrifying to somebody making zero off of their content.
00:47:09 Yeah. But again, it starts with smaller numbers and it goes up. You’re like, okay, there are many alerts. There are many tiers at AWS. If it goes over $50, send me a notice. It’s over $100, send me a notice.
00:47:21 There’s all these billing bells that go off.
00:47:24 Okay.
00:47:26 My heart is slowing down now. Again, thanks.
00:47:30 Not mine, but it’s okay.
00:47:32 So this has been fun. And I’m sure I’d like to have people generate questions. So I’m sure this answered some questions and generated questions. So please reach out and we could do a follow up if we need to do some more questions. Thanks a lot, Michael. And we’ll talk to you later.
00:47:52 Yeah. Thanks for having me. And it’s always great to be here Brian.
00:47:59 Thank you, Michael. There were a lot of cool tools to check out. Thank you Lenode for sponsoring the show. If it runs on Linux it runs on Lenode. Try lenode with a $100 credit by going to Lenode.com testingco code that link is also in the show notes at testincode.com 146 thank you also to Patreon supporters join them at testandcode.com support that’s all for now, go out and test something.