I asked people on twitter to fill in “How do I test _____?” to find out what people want to know how to test. Lots of responses. David Lord agreed to answer them with me. In the process, we come up with lots of great general advice on how to test just about anything.


Transcript for episode 129 of the Test & Code Podcast

This transcript starts as an auto generated transcript.
PRs welcome if you want to help fix any errors.


00:00:00 I sent out a question on Twitter saying, Please fill in the question. How do I test blank to find out what people want to know how to test? I got a lot of responses. David Lord agreed to answer them with me, and in the process, we come up with lots of great general advice on how to test just about anything.

00:00:19 Do You Want to Get better at Python? Now is an excellent time to take an online course, whether you’re just learning Python or need to go deep into things like APIs and async our friends at Talk Python Training have a top notch course for you.

00:00:33 Visit Talk.

00:00:33 Python. Fm test to find your next level that’s Talk Python. Fm test. Stick around until the end of the show. Also for a chance to win a free course.

00:00:59 Welcome to Testing Code.

00:01:08 Welcome to Test and Code. I am really excited to have David Laurie on the show. David, this is the first time you’ve been on, right?

00:01:15 Yes. You finally got me on. It’s been like two or three years since we met at a conference, and he said, hey, you should come on.

00:01:21 Yeah. Right. I think I met David by sitting next to him at dinner at one of the Pythons’and. Then we chatted for a little while. But, David, I know you mostly as one of the people that helps keep Flask alive, but is that true or is there more to David than that?

00:01:38 Is that true? Yeah, I guess I’d say that’s true. I’m the lead maintainer for Pallets, which is a community organization that maintains Flask and all the other libraries related to Flesh, and I’ve been doing that for about five years now. I have slowly been adding more maintainers as I meet people at conferences or get people interested on GitHub. But it usually comes down to me to be doing the big reviews and making the releases and everything.

00:02:04 Okay. So under Palettes, there’s Flask, but there’s also everybody should be familiar with. But there’s also Click and Ginger and Work Zigzagsig.

00:02:15 I’m probably pronouncing it wrong also.

00:02:17 Okay. Those are the main big hitters, right?

00:02:20 Yeah. Okay, so Flask is the web application framework that’s definitely the most popular, but the other libraries are all used a ton. Also, Ginja is a template language that can be used for any type of application. It Flask uses it to provide HTML templates if you want to render your website. Other completely different sorts of applications, like Ansible or Salt, use it for configuration management, sort of a configuration language. Click is kind of the equivalent to Flask, but for command line instead of web applications. So it lets you plug together command line commands and groups and options and arguments and all that sort of stuff. So it’s framework for building CLI’s.

00:03:02 Yeah. I’m a longtime Click user and recently trying to switch to Typer because it’s fun. But typers built on top of Click also.

00:03:11 Yeah. Typer is pretty cool. One of the big things we’re working on right now that I think specials aware of, too, is click provides kind of tab completion abilities.

00:03:21 If you install a completion script for your command, you can hit tab to see what options are available or what values for the options.

00:03:30 Just really basic right now. And click and we’re completely rewriting the completion system so that it’s overrideable at every level and customizable so extensions can add new shelves and all sorts of stuff. It’s going to be pretty exciting. Okay. That should carry forward to typer also.

00:03:48 Cool. Well, this is going to be an interesting episode. One of the things that David agreed to do is I sent out a message on Twitter a couple of weeks ago, I guess, and the question really was how do I test blank and wanting to know what people want to know how to test. And I got a lot of answers. What I’m going to do is I’m going to run through a lot of these, and David’s going to help me answer them, I guess. Let’s just do the first one.

00:04:13 Sure.

00:04:13 Josh Peak asked. He said, I’m trying to teach interns and colleagues what is a good test. So I was curious what your opinions are and what the Flask philosophy is for testing.

00:04:25 Yeah. So I think my opinions are basically the Flask philosophy. That’s where I got my opinions from.

00:04:31 Okay.

00:04:31 Before I started really working with Flask and maintaining Flask, I had kind of worked with tests a little bit in College courses and all that sort of stuff, but never really did a good job at them writing tests or anything. So my first exposure to fairly thorough test was Flask. And this is kind of my interpretation, I guess, of what I saw there. But kind of what I’ve done going forward since I started maintaining was testing. Like not worrying about testing every false positive or false negative. You can not trying to think of every corner case. Usually in my experience, it’s been enough to test the thing you’re building. So if you’re creating a feature, you know, your feature, you want your feature to do test those things about the feature. Right. And if you miss a corner case, somebody will report a bug eventually, and then you add another test for that use case as well. So it’s kind of like an iterative process. You start with something you want. People say, oh, I need to see this and kind of add on and add on as opposed to trying to think of everything ahead of time.

00:05:37 So, like, if I didn’t have automated tests and I was just doing manual testing, that’s sort of generally how people use manual tests. As a developer, you’re trying to get some feature to work so you could some stuff up and try it and see if it works and move on. Is that kind of line up with how you’re writing automated tests, then?

00:05:57 I guess so. I don’t want it to sound like we’re only manually testing or we’re not thoroughly testing.

00:06:02 But we’re not trying to describe I guess I think of this as functionality testing. So we’re just essentially testing make sure that the expected functionality works as expected, yes.

00:06:13 Does the thing work as documented? I usually go a little beyond that. Like, most of the time I’ll be reviewing a PR. I’ll see what the initial issue was. I can read that. I can see what the code is that changed. And I can look at the tests and say, I’ve just gotten good at this every time. But I can just say, oh yeah, these tests look pretty good for they seem to cover this stuff or oh, wait, there’s this huge block of code that clearly you can get into, but we don’t test at all. Okay. So I haven’t really used coverage itself too much in a while.

00:06:42 Flask has the ability to do some to test. It’s a web framework. So we’re bringing up a web browser to test for the automated tests.

00:06:50 Yeah. So when I was referring to tests in Flask, I meant like testing the internals, like the code of Flask, which most of the time looks like actually testing an application written in Flask, too. A lot of tests will be here. Create this little sample application that uses this feature, then run a request against it. But Flask does provide a test client that doesn’t actually require you to run a server.

00:07:13 Okay.

00:07:13 So it just simulates making a request with the right, like WSGI environment and passing that to the Flask application and just letting it dispatch all that.

00:07:23 And does a lot of the internal Flask testing use that then?

00:07:26 Yeah.

00:07:28 What I was saying was the Flask tests kind of fall into two categories. Either there’s some internal function we’re testing the function directly, or we’re like testing some sort of application, like some sort of flow through the framework that we want to result in something. So we’ll actually write an application that would use that and then run a test request against it.

00:07:47 Okay. Is there like a litmus test for how to decide what’s what, or is it just whatever somebody thinks of to test a particular feature? Yeah.

00:07:55 It really just depends on what’s being written. I think in Flask nowadays, mostly it’s going to be like, let’s see, the last big feature we added was so you have a view function, which is you had a request in the Flask dispatches to that view function, and then you return something from the view function to have Flask turned into a response. And so previously you could return a couple of different things from that function. You could turn like a string and that would automatically be interpreted as an HTML response. You could return a tuple, which could have a different status code Besides the default of 200, and it would kind of recognize these different types and turn them into a standard response. Something that a lot of people are using Flask for now is writing APIs. And so we have this jsonify function you can pass in some data, creates a JSON response, and as a shortcut to that, you can now return a dictionary from a view, and that’s automatically turned into a JSON response, just like a string is turned into an HTML response. And so when we wanted to write a test for that, we literally just wrote a little application with a single view that returns a dictionary. And we made a request to it with the test client and said, okay, is the response type application JSON? Does the response decoded as JSON have the same data? So we actually did put together a little application.

00:09:13 Okay.

00:09:13 I’m sure there were other tests, too, but yeah, most of them end up being like, here, create an application that does this feature.

00:09:18 Okay, cool.

00:09:22 Thank you, Datadog, for sponsoring this episode. Are you having trouble visualizing bottlenecks and latency in your apps and not sure where the issue is coming from or how to solve it? With Datadog’s end to end monitoring platform, you can use their customizable built in dashboard to collect metrics and visualize a performance in real time. Datadog automatically correlates logs and traces at the level of individual requests, allowing you to quickly troubleshoot your Python applications. Plus, their service map automatically plots the flow of requests across your app architecture so you can understand dependencies and proactively monitor the performance of your apps. Start tracking the performance of your apps, sign up for free and install the agent, and Datadog will send you a free T shirt to get started, visit Test And Code. Comdateddog.

00:10:13 Well, I don’t know if we answered what is a good test? Let’s just jump ahead. The next one is Nicholasminzer, how do you test web app performance?

00:10:22 You had a couple of follow ups to that. Like what kind of performance is the real question?

00:10:27 Yeah, it could be single user speed or under load or throughput.

00:10:31 Yeah.

00:10:32 This one I don’t have a good answer for. I guess it’s because we don’t really know what kind of performance you’re testing. But I know there are tools like I’m trying to remember now there’s an Apache tool, JMeter, and there’s some other basically like load testing tools. So you can point this tool at your running application and it will make a ton of requests and tell you what the performance is.

00:10:57 How useful that is, I’m not sure.

00:10:59 I don’t personally write giant applications that have to perform under huge load.

00:11:04 So I don’t really do these sorts of things. Usually my applications that I write for work just scale, maybe the two servers.

00:11:11 Okay, well, we’re going to walk through a lot of these, and I’m okay with the answer being like that. Not really sure, because I’d like to invite other people to come on. I know that there are some people that really do care about testing Web performance. I think that’d be fun. So please, if you’re out there and listening and you know about this and you’re like, oh, my God, why didn’t you recommend X? Well, then come on.

00:11:33 I haven’t recommended it. I haven’t heard about it yet.

00:11:36 Yeah, well, I’ve heard of things like swarm testing and stuff and ways to try to have artificially build upload. And then there’s a lot of people that say just do segment testing and test in the field, and then there’s your performance.

00:11:49 It also really depends on like, do you expect your web application to be your bottleneck? In my experience, at least the web application itself hasn’t been the problem. It’s like, have I optimized my sequel enough, or am I dispatching tasks to a job queue instead of holding up workers, that sort of thing things I guess you can identify by doing load testing, but the load testing itself wouldn’t really be interesting.

00:12:14 So kind of looking at the Web app performance as a whole for your entire application and realizing that the bottleneck might not be your web server, it’s probably database or something.

00:12:26 Yeah. Some other service that you might be talking to that’s doing the real thing.

00:12:29 Interesting. And there’s probably shortcuts for that. If your load testing is really on your database, you could probably test that separate maybe. I don’t know. Who knows?

00:12:38 Next, we’ve got Waylon Walker says, I just learned how to test cookie cutter templates. This would have been my question two days ago. So I wanted to know. I asked him how and he pointed me to a couple of examples. I think I lost the links, but there’s a Python plugin for cookie cutter templates.

00:12:56 And then also looks like he pointed to cookie cutter data science.

00:13:01 So has some tests in place.

00:13:05 I’ve never tested cookie cutter templates.

00:13:08 Yeah, I was going to look at there’s a cookie cutter for flask called cookie cutter flask. And I was going to see if they have tests. But if they do, I don’t know. They’re not called test stop High or anything. But yeah, this is kind of generally just cookie cutter. And I don’t know anything about pytest cookie cutter. But if I don’t know how to test something, I go and look at how that thing is tested. So if I don’t know how to test cookie cutter, I would go and look and see. Well, I would hope cookie cutter is testing itself. Right. So it’s probably setting up something that would test its functionality and then looking at it somehow. So you can kind of replicate that if nothing like that.

00:13:43 Yeah.

00:13:44 I bring this up partly because I think it would be fun to talk about testing cookie cutter stuff with somebody that knows what they’re talking about. But also this example is he went out and looked at two examples of other projects using cookie cutter and looking at how their testing is working, which is neat.

00:14:02 Yeah. Same sort of thing like find a project that’s using it already and see how they.

00:14:06 Yeah. So I’m looking at it and it looks like there’s some common. I’m looking at one of the data science one, there’s some tests to see running the cookie cutter and making sure cookie cutter will go and has this base project or this cookie cutter project that it populates a new project for you. So a lot of the tests are this file is supposed to exist as a file show up the right way.

00:14:31 It’s an interesting concept because I’ve never even tried, but making sure that the licensing shows up correct and everything. So, yeah, I would recommend this. Also, go look at other examples. That’s a lesson for everybody. Not just cookie cutter, but other how to test whatever. It’s one of the beauties of open source. You can look at a lot of projects that have done it already.

00:14:53 Occasionally I’ve had to look at the tests just to figure out how to use a library, too.

00:14:58 Yeah, that’s my favorite kind of documentation, right? Test, definitely.

00:15:03 I like it. So then the next one is Israel Ruktor, maybe. Sorry if I got your name wrong. Israel. The question is how do I test my test framework? Aka who will guard the guards? Well, I use Pipes and Pipes has a whole bunch of tests inside of it, and we can look at those.

00:15:22 Yeah, you can run the tests. You could just clone the repo and follow the instructions to set up your test environment and go wonder if pytest test is solved. pytest.

00:15:33 It’s an interesting chicken and egg problem, though, but at the same time, for instance, I was plugins have to run to test a Pipe test plugin, you have to run Pipes.

00:15:45 You have to run Pipe Test to run the test that ends up running Pipe test. It’s definitely the harder case. And once you learn how to do that on a little tiny Pipe Test plugin project, you can learn how most of pytest is being tested. Do you use pytest or do you use Unit Test for Flask?

00:16:03 We use pytest. That conversion. All the projects were originally written using Unit test. And then I think about a year before I started being a maintainer, another maintainer went through and just kind of did a fairly surface level rewrite of taking out Unit tests and making it runnable with pytest, which doesn’t require that many changes, to be honest, because Pytest kind of understands Unit test.

00:16:27 So I’m still today, there’s plenty of old weird artifacts from those days and I’ll rewrite into like, oh wait, we have this giant test to test these like 20 different cases in a for loop that should be parameterized. Nice, right?

00:16:40 So I still pull those today. If anybody wants to contribute to flashed just go through the tests and turn them into better parameterized tests.

00:16:48 That would be fun. Yeah. Sounds like a fun job or fun volunteer opportunity.

00:16:54 So neat. Next up, Will Robinson says, how do I test permission management in an application for a public facing application? Do you know what permission management is?

00:17:06 Yeah, I was trying to figure this one out. I was thinking they were maybe mean like role based authentication or something like that. If this user has this role, they can access this resource, something like that, which to me just seems like any other sort of test. Right. You’re setting up some sort of precondition, you’re running some, you’re doing an action and then you’re looking did I get the result I expected? So you set up like here’s a user with these permissions if I request this thing.

00:17:33 Oh, yeah. So assuming different parts of your application or different APIs to API frameworks have permissions, I assume they do.

00:17:42 It really depends. Like Flask doesn’t come with anything. Then you get like on the other end, Django, which does have a permission framework built in, so it has users and permission.

00:17:52 There’s a permission systems that you can build on top of Flask, though, right?

00:17:56 Yeah, I think the new one I saw somebody working on was called Flask Pretorian or something like that. But there’s also older ones like Flask principles.

00:18:07 And this is different. I guess there’s a couple angles that if that’s like roles. So user role, like maybe Admin versus Reader or for different applications. You could imagine if it was a CMS system, you might have editor permissions and writer permissions and things like that. Yeah, that makes sense to me. You set up a user with particular permissions and then see if the activities that they’re allowed to do work and then attempt some activities they’re not supposed to do and make sure those don’t work.

00:18:37 I guess now that I think about it, the Flask tutorial walks you through creating a little blog application that has login, single user login. You just preset username and password, but it still requires you to be logged in to make a new post. And so the tests in the tutorial show creating a little fixture that will let you perform login or perform login automatically before a test so that you will be logged in and can do an action or that you won’t be logged in, you’ll see an error. Okay.

00:19:07 Now on the other side of it, I was thinking, I don’t think this is part of the real question, but user authentication with different author authentication systems, that seems like a related but completely different thing.

00:19:20 You mean like if I have off providers, like if I have Google Login or something like that. How do you test that?

00:19:27 Yeah, I don’t know how you would test that. I mean, if you really want to test it, you probably have to create some sort of live server environment to actually make an OAuth flow. But for applications, everything in the past, I just don’t test that part because the OAuth library I’m using is tested.

00:19:43 And I would hope that whatever OAuth provider I’m using is up and doesn’t stop using OAuth all of a sudden or something. But you’re kind of making an assumption that this is OAuth. So for those I just faked like, yes, I just force the user to be logged in without going through the flow.

00:19:58 And then I just yeah, I’m glad I’m asking because I think that’s a good point. Not just with authentication, but other things. Is this idea of if you’re relying on a separate plug in or something, some other system to do part of your workflow, like authentication or something else? You don’t necessarily I guess it’s a decision based for your project, but it’s a reasonable decision to not test that part. At least in automated tests.

00:20:27 You might want to test what happens if you can’t get to that service. So, like, what if Google off goes down for the day? Does your whole application fail also? Or can people who are logged in just keep using it?

00:20:40 You might want to make sure your app doesn’t blow up if the other thing goes away. But other than that, really know what you’re doing.

00:20:46 Yeah. And if you were testing it, that’d be a good place to stick some, like, a fake service or something, because I wouldn’t want to be hitting Google for my automated system. Anyway, here’s the big answer. So the next question. Felipe Bid, you I should have had you try to do the names.

00:21:05 No, I prefer you to do it.

00:21:06 Yesterday, I was struggling with the philosophy fee behind testing SQL Alchemy models and by Dante Schemas in a Fast API app, at which point testing something like that becomes vacuous. And what should I test before that happens? I don’t know what that means, really, but what is the.

00:21:27 How far should I go down to what level should I test before it’s not meaningful to test?

00:21:32 Yeah, and nicely. Sebastian Ramirez, the Fast API person himself, came back and replied, do you understand Sebastian’s answer? And can you summarize.

00:21:46 Let’S see.

00:21:48 So, yeah, he’s saying basically the same thing that I would have said.

00:21:52 And I think we’ve kind of touched on this already. But you’re not testing SQL Alchemy or Fast API. You’re testing the assumptions about your code and your data. Right. So instead of testing that, like, oh, does Fast API decode JSON data correctly? Well, yes, hopefully Fast API decodes JSON correctly. Otherwise Fast API wouldn’t be working at all. Right. That’s already tested by Fast API itself. So instead you want to be testing like, oh, if I send data in this shape, did I get the custom validation area that I wrote? Or if I send this data, does it end up as an object in the database that I can retrieve later.

00:22:35 So you don’t need to test things that fast. Api itself or SQL. Fm itself would already be testing. You can test the side effects of those things. You don’t have to completely avoid them either. I mean, you might want to test like, hey, if I send this request, does it get validated even if I’m just using the standard validation?

00:22:53 Okay, cool. And especially focus on what you were adding to your application.

00:22:59 Right. Test your code first, and then if you find that you’re still missing coverage or you want to like, I think if you test your code first, you’ll start to discover the other things you might want to do.

00:23:08 No, you said the magic word. You said coverage.

00:23:13 Now we can hit on it later. But I was curious what your views on code coverage was. Is Flask 100% covered?

00:23:21 No, none of the palace projects are 100% covered. There’s been a good task at any conference, Francis. Always pick something and add more coverage to it. But I kind of stopped running coverage as part of the test suite because it just tended to slow down the tests. And being 100% covered wasn’t difficult for any of the PRS we were getting. So it wasn’t like we were missing anything in any new code. It was that there’s just all this old code where there’s these weird pathways that weren’t tested or something.

00:23:51 So, yeah, I just kind of turned it off for performance reasons and because it got kind of noisy and people were never sure. New contributors were never sure. Wait, am I supposed to fix this or was this coverage because of some other thing that wasn’t covered before? All the important bits are covered for the most part.

00:24:09 Let’s take a pull request, for instance. So if somebody has a feature adding a flask or click or something, you’re going to have to evaluate by looking like, really just looking at it and looking at what’s being tested and what’s being added.

00:24:24 Yeah, I know that sounds really ridiculous and like a lot of work, but it’s never actually been a problem. It’s never been actually something I feel like I consciously thought about. But it turns out that every time I’ve gone back and looked later, it hasn’t been the new code. It’s not covered, it’s just part of the review process. I look at the test and code tests look good and the code looks good. It’s usually getting about covered.

00:24:45 Okay, fair enough.

00:24:47 Also, nowadays all the libraries are stable for the most part. So it’s not like we’re adding huge new functionality where there’s lots of complex, different code paths. We might be adding like a significant feature, but it’s one thing.

00:25:01 So you test the one thing and you can’t really go wrong with that. But I do like coverage. It’s just I don’t run it day to day. You can still run it against all the projects, and it’s a good way to find something to contribute. If you want to say like a time to go figure out how to 100% cover the blueprint tests and flask, I think somebody did that.

00:25:20 Okay. There are a couple of points that are interesting. The notion that if a project isn’t 100% covered already, there is some newcomer stress of if they’re adding something, how do they tell if they’ve covered their stuff if the rest of it isn’t covered, but also that it doesn’t guarantee even if you are 100% covered, it doesn’t guarantee that it’s good enough testing.

00:25:44 Oh yeah, we’ve had things which have been clearly 100% covered, and then we’ll get a bug report on them anyway somebody will not have because it doesn’t cover the things you can’t think of. Right.

00:25:55 Okay, this next one complete change of subject. How do I test a warehouse ETL or how do I test data warehouse ETL code? The ETL being extract transform load, which I completely know I’ve memorized. Now what ETL stands for.

00:26:13 I did not know it for. I’ve seen it a bunch.

00:26:15 But I don’t know how to do this. Any thoughts?

00:26:19 Well, I don’t really know what the circumstances are. I mean, are you writing this ETL, then it’s a test like any other, right? You set up some preconditions, you set up your data, you run some function on it, did it transform it the way you wanted to transform? Did it get loaded into the database? Okay, test pass rate?

00:26:37 I don’t know. That seems very reductionist, but that’s usually my testing philosophy is set up the preconditions, run the function, get the result. It doesn’t really matter what the thing you’re testing is. So then it more comes out. This is why I was saying it’s not really clear what the question is because are they using some external service to store their data and they want to make sure their data is stored correctly or what?

00:26:59 I forgot to give credit that came from Max Hofer, so thanks, Max. Well, I’m sure I can try to hit up some data science people, but actually I’ve talked about that recently on the podcast and that really was the gist of it’s. Just like any other code, you should know if you have a behavior, you should expect what it’s supposed to do. Some of the real question often isn’t how do I write a test for it, but how do you think about how it should work? What are the preconditions? What is a small sample set to when you’re expecting large data sets? How do you test it on a smaller data set? Things like that?

00:27:34 Yeah.

00:27:35 Okay. This one’s kind of fun. US Hills says how to test and mock GPIO pins on hardware for code written on Python, but running micro Python on the device?

00:27:47 I have no answer for this one. I need to do more micro Python. I have a bunch of those little different boards that I’ve gotten from conferences and stuff.

00:27:56 But yeah, actually I would love to hear how people are doing their testing on things like micro Python and circuit Python. If they’re writing tests for those, how are they writing test form and including?

00:28:08 This is kind of a hardware software thing. Now if you want to test if your Pins are working, sometimes you actually have to hook up data, something collection device to actually look to see if the signals are coming out.

00:28:22 Yeah. Are you testing the actual hardware or are you like testing just that if a signal came in, you did the right thing. Right. And then at that point, hopefully your GPO library has some sort of test client where you can fake signals or something. Just like Plastic has a test client where you can fake requests.

00:28:40 Interesting rabbit hole to go down sometime.

00:28:43 Yeah.

00:28:43 I don’t know if I’m going to be able to answer the next question.

00:28:46 Yeah, well, I’ll take it out of there anyway. Nicholas George asks Piqt Apps so my current hack is to start the QAA application in Comfdest Pi, but I still get occasional SEG faults. Pi test QT doesn’t seem to help, so don’t know. I do know a few people that have written QT apps, so maybe I’ll take a look at some of those applications or hit them up.

00:29:11 Yeah. I don’t even know how. I really don’t do Gui’s at all. Gui’s and graphics for my two weaknesses, and I’m kind of wondering now how do you test like a Tk enter app, even something that the Tk enter that’s built into Python? If you wrote a Gui, there, is there a way to test it?

00:29:29 I don’t know. Yeah, I’m not sure. I would point anybody that wants to do QT testing, though, to a project called Cute Browser. It’s Q-U-T-E Browser, and it’s written in Cute by Cute. But also the compiler who is the person responsible for the project, is a contributor to pytest as well. And so I know that a lot. Some of the pipes changes were done to help with acute testing, so I guess I would check out how that’s kind of nice. It looks like the tests are split up into unit tests, helper test and code to end tests.

00:30:06 One thing I’m aware of is I haven’t used it in a long time, but I remember another group at work using this is a project called Sakuli. I think it’s Siculi or something like that.

00:30:20 And I think the idea behind it is that it takes screenshots or it can actually interact with. So it’s kind of like Selenium where it will interact with the actual components in your web front end, except it will go a step further and kind of actually move the mouse around and click things based on compare screenshots to figure out things.

00:30:41 Maybe there’s something there, but I don’t know enough about it to recommend.

00:30:46 Yeah, okay. Well, I guess I do have something to add then. In general, when testing, a lot of this has to do with the design, your software design. So for instance, like Flask or other web frameworks, if you have to go through the web interface at all, you don’t really want to for most of your testing. So if you can test underneath that interface, you’re better off. So I would do the same with Pi QT or with QT or Cute. It’s a long habit of mind to pronounce the T separate.

00:31:21 Yeah, I learned recently that it was pronounced cute and I still can’t see Q.

00:31:26 Anyway, I would try to architect their application so that most of the work can be tested, not using the GUI test most your application without it. Do an API layer just underneath and try to have the GUI as minimal as possible with the logic. And then in that case, if you have to just even dog Fooding it or playing with it once in a while might be enough. You might not have to test that the end user test the user interface testing. Or at that point, maybe the Pytech will be enough to test that for all GUI applications. I recommend that it’s try to do as little actual GUI testing as possible.

00:32:05 I would not want to rely on like Pixel capture sort of testing.

00:32:11 No, I can’t imagine it’s the most robust.

00:32:14 Actually, I would love to. Maybe things have changed and there’s really advances in that. So if you’re listening and you’re like, oh, my gosh, the optic testing has advanced so much. You need to know about all of this stuff. Contact me. I’ll bring you on the show next one. Jason H. Crow. How do you test web scrapers?

00:32:34 This one, I think I would assume you’re hoping the web scraper itself is already tested, right? You’re testing your code about like, okay, the web scraper has connected to this site and downloaded the HTML. Did I now extract the right stuff from that? And so again, you’re testing your code, not the web scraper code. Right. So here’s a document I would expect to get if I load that up in the web scraper. Does it go through the pipeline correctly?

00:32:59 That’s an interesting thing to talk about. So part of what you were just talking about was you were splitting up is splitting up the problem set. So that part of it is a web scraper takes information from a website and then does something with it, comes up with parses the information, gets different data. So there’s really two ends of that problem. There might be more parts of that pipeline. You can test each one of them individually. A big chunk of that would be, I would think, faking the data. So starting with known data, putting information in a web page. But like you said, the web scraping part, you’d have hope that already works. You could mock up an actual web page, but that seems like overkill the rest of the tool chain might be what you really want to test.

00:33:45 Yeah, this isn’t just for web scrapers, but you can. Like, if you’re using something like httpx or the older requests, there’s libraries that will let you mock Http or requests so that when you make requests with them, they actually return these mock responses. So httpx has respects request has responses.

00:34:07 They kind of integrate with Pythons, I think, and they let you say like, okay, if httpx makes a get request to this URL, then return a response that looks like this without actually making the request.

00:34:20 So that you’re kind of like saying I expect to get a response like this if the website is working correctly.

00:34:26 Right.

00:34:27 So then you wire that up, let the scraper go as if it was making a real request.

00:34:32 And then there I didn’t know there was a similar thing for http. That’s cool.

00:34:37 Yeah, I’ve started using it on my libraries.

00:34:39 Okay, let’s see. Ben Hancock, let’s see. What does he ask? Is it the best practice to put static HTML data in your test directory or to test against or just snippets stored in string variables? I guess this is a similar sort of thing as the web scraper thing.

00:34:57 But there’s no hard and fast rule. I think the Flask tests have some things like we want to test that we can serve a file correctly, and that when you point the send file function at an HTML file, you get a response with a text HTML content type as opposed to pointing a text file, and you get a text plus plane. And so we actually do have some resource files sitting in the test directory. But then sometimes you just want to say like, oh, if I return this HTML, is this HTML visible in the response? And that HTML could just be a string like that content kind of depends on how big it gets, because if it gets really big, it’s kind of annoying to read the test files themselves. But if you do break it up into separate files, they’re now separated from your test file. So it’s kind of hard to see the relationship between the two. I don’t think it’s bad to do like.

00:35:46 For instance, it’s fairly traditional. This is a different direction. It’s not a web scraper, but converter tests like, for instance, markdown converts markdown to HTML. It’s fairly traditional for a lot of the tools to have like input and output directories. Of example, this input makes this output, and then the tests have to read those directories and convert that stuff.

00:36:10 Yeah, I think the black code formatter does that. Also, it has a big folder full of test input and test output, so it can run the code for matter against the input and see if it matches the output and those big Python files.

00:36:23 Yes, those are interesting ways to get around it. I would say. I don’t think there’s a hard and fast rule either. It’s what is convenient and what’s more, maintainable. Ryan Morsaid asks what’s the best way to test server client API contracts, especially if the server and client code are kept in two separate repos.

00:36:42 So I think he’s saying that the API isn’t necessarily the server or the client because the server and the client have both agreed that I will send requests this way and the server is that I will send responses this way to these requests.

00:36:57 I don’t know how to test that. I guess it kind of gets into like, okay, if I change the server, I want to make sure that client tests are running periodically in addition to on every commit or something, so that I can make sure like, oh, if I committed something to the server but didn’t make any changes to the client, do the client test still pass anyway, even though nothing happened to trigger those tests? So probably just kind of having a better scheduled testing set up so you can see things failed.

00:37:25 Yeah, I don’t really have much to add, so let’s move on. Oh, this is a cool name, Capille. Mature.

00:37:31 I probably got that one wrong, too. Sorry. How do I test a monitoring tool like Naglos or Naman? I have never heard of either of these.

00:37:41 Never heard of them.

00:37:42 What are you testing about the tool?

00:37:46 I think these monitoring tools are things you run on your server next, and they’ll look at your Apache logs or other things to aggregate information.

00:37:56 That’s not what they do. There are other tools to do that, stuff that monitors other things. But what are you testing about those hopefully those tools are already tested. Right. And you’re using them because you want their features, not because you want to use something specific about them. Or if you are using something specific about them, then you know what your input is, you know what you expect from them. So again, now you have the two parts of the test you sign up for preconditions run it.

00:38:19 Yeah.

00:38:20 We get kind of going with a theme here.

00:38:25 Yeah. I think a lot of people are just intimidated by, like, I have this giant tool.

00:38:30 They’re just not sure do I need to test it to begin with? You need to be kind of like at some point you kind of have to say I am making assumptions about the tools I’m using. I’m hoping they’re tested right. I need to trust that those tools will do the things I’m using for.

00:38:45 Yeah. So I’m going to actually cut it off here. We’ve got a few more, but I don’t think they’re going to add to the conversation. So we’ve heard, I guess, a lot of things.

00:38:53 I think a lot of this is just sort of a testing strategy of a testing mindset. And some of it is, like you said, a common philosophy or a common strategy of a test of there’s a thing happening, I want to set up a precondition, make the thing happen. What do I expect to happen afterwards?

00:39:11 And that’s a lot of it. There’s other things of like your process that you’re testing is really a process. It’s a multi stage thing. Test the pieces. And sometimes if you’re depending on something that is undependable, like a service that might change, then we’ll mock that or fix that or something or put your own HTML in. And then also don’t worry so much about testing the parts of your system that are written by others.

00:39:37 I would say, yeah, don’t worry about it too much. I mean, definitely be aware of the things you’re using. Maybe go look like, hey, are the things I’m using tested? Because that probably should factor in your decision to use that thing a little bit at least, especially if you’re going to put that in production. But yes, at the end of day, test your code.

00:39:55 Yeah. And then also that if you’ve got multiple moving parts, fix most of the parts. Not that they’re broken, but make a lot of the pieces not moveable, like, for instance, the client server stuff. It’s actually kind of great that they’re in two different repos because one of the things you can do is you can update the server, but test against an old version of the client, make sure that still works, and do stair step sort of things. Don’t change them both at the same time. If you do need to change the API, that’s a known to break and you’re going to have to change both at the same time. So we kind of do that with the libraries, too. Libraries have to sometimes break their API.

00:40:34 Hopefully with deprecation warnings first.

00:40:37 Yeah.

00:40:38 So we were discussing before we started recording, like some of these can be grouped together. But in the end, I think all of these kind of boils down to that same sort of philosophy that you just said, nail down as many parts as you can, identify your inputs and your outputs and what function you’re actually testing. Do that test and avoid testing, people’s guesses.

00:40:59 Also, it’s not easy. Nobody said it was going to be easy, but we’ll definitely have to have you on some more to talk about Flask at some point. But thank you so much for going through this list with me, David.

00:41:10 Yeah, this was fun. We can definitely talk about Flask another time.

00:41:14 Okay, cool. Thanks.

00:41:16 Yeah. Thanks, Ryan.

00:41:21 Thank you, David, for coming on the show and for all you do with the palace project. And thank you Datadog for sponsoring. Visit Test And Code. Comdatog to get started. Thank you, Talk Pythonsraining for sponsoring. Check it out at talk. Python. Fmtest to level up your skills and enter to win a free course by joining the show mailing list at Test And Code. Comsubscribe. And thank you to all the listeners that support the show through Patreon join them by going to testandcode.com support. All of those links are in the show notes at testandcode.com. One, two. Nine. That’s all for now. Now. Go out and test something.