Playwright is an end to end automated testing framework for web apps with Python support and even a pytest plugin.
Transcript for episode 153 of the Test & Code Podcast
This transcript starts as an auto generated transcript.
PRs welcome if you want to help fix any errors.
00:00:00 In this episode, we talked with Ryan Howard about playwright. Playwright is an end to end automated testing framework for web apps with Python support and even a Pi test plug in.
00:00:24 Welcome to Test and Code.
00:00:34 Welcome to Test and Code. I am thrilled today to have Ryan Howard on the show, and we’re going to talk about Playwright. So, Ryan, tell us a bit about yourself.
00:00:43 Hi. My name is Ryan.
00:00:45 I live in the UK and I am a software tester and the founder of Healthcare, which is a software testing consultancy over here.
00:00:59 You founded a consultancy?
00:01:02 Yeah. So I work with mostly kind of small companies and startups providing kind of ad hoc and short term testing consultancy and support.
00:01:15 Okay.
00:01:17 Do you usually get involved kind of after they think they’re done, or is it during development or when do you get involved?
00:01:28 It’s changing slightly. Historically, it’s been towards the end of the project when they realize they might need someone to test it.
00:01:37 But there’s definitely been a kind of shift recently to getting people on board earlier on and actually starting testing right at the start of the life cycle or even testing before you even coded anything, whether that means looking at requirements, testing requirements, or if you’re taking a sort of TDD approach, how QA and software testers can help develop those tests up front before you’ve even written anything.
00:02:06 Yeah. Or even just even just getting started and doing some skeleton tests at first and then have you jump in periodically as they have new features to check out things like that seems like the right way to do it instead of, okay, we’re ready to ship. We need to test it.
00:02:24 Yeah. It makes a big difference. It makes the testing side of things go a bit smoother.
00:02:30 If you’ve done the work upfront.
00:02:32 How did you get into this?
00:02:35 Bizarrely. I kind of just fell into it. I was working in retail about ten years ago, no experience with software at all.
00:02:46 And I just happened to know someone who worked for a software company who asked if I’d be interested in going to an interview.
00:02:54 Zero idea that software testing was even a career, but yeah. So I joined as a junior QA analyst, and I’ve been doing it for ten years, starting out.
00:03:10 Was it manual stuff or did you start automating right away?
00:03:14 It was mostly manual to begin with. I was probably maybe a year or two into my testing career, and I still quite haven’t worked out what testing really was, I don’t think. But then I started picking up some code and working with developers and understanding how you can kind of add test automation and write tests in code a bit later on.
00:03:41 Okay. Do you do any manual testing anymore?
00:03:44 Yeah.
00:03:47 I feel like it’s still quite an important part of the role.
00:03:50 Again, there’s been a shift from sort of traditional manual testing, as I call it, where you used to spend a long time writing step by step test scripts, enter username and password. Click Submit check. User is logged in, click on this button.
00:04:12 And that’s still quite useful.
00:04:15 But what tends to happen now is we do use a process called exploratory testing a bit more.
00:04:24 And the idea is there that instead of having these sort of robust scripts that you follow to test, you have an idea of an area of functionality you’d like to explore and some rough guidelines about the sort of things you’d like to find out.
00:04:43 And then it’s up to you to kind of go away and you usually time box these. So you maybe say, I want 30 minutes to explore the login functionality.
00:04:52 And then instead of following these scripts, you can sort of go off piece to bit, so you can wonder what happens if I enter a password which is incorrect or I keep pressing the submit button. So it’s more about kind of exploring the application rather than just following a script.
00:05:15 And that works really well.
00:05:17 That was my introduction of two Exploratory testing. I was on a project where we didn’t have any formal test people. We just had the development team. But we had like, for instance, we had a big lab with lots of different instruments and people working on different ones. And we would say, we want to get so many hours of exploratory QA and you didn’t have to know anything about the instrument. They would say it would be something like, well, I don’t know, we changed this particular measurement. So exploring that would be great. But then also looking at other stuff and it would just be playing around and see if it makes sense and just trying to use it carefully and seeing weird things like why is the trace going up and off the screen and stuff like that.
00:06:08 I think that’s the part that actually gets missed a lot. This exploratory just playing with it.
00:06:14 I think that’s an important thing to keep even with the push to automation.
00:06:19 Yeah, definitely. It’s really useful. Like, you say just as much to test an application as it is to kind of understand it and understand how it works and what happens when you do things. And even, like you say, just picking up on things which aren’t necessarily bugs, they’re just is this the right way of doing it? Actually, it’s a little bit annoying if the way it’s working functionally, it might work, but actually and then you can provide a feedback at the end of your session to say, these are the things I found and these are the things I’ve realized I need to explore a bit more.
00:07:02 So it’s useful and it works really well in conjunction with automation. Because if you can automate those kind of regression test and code structured tests that you know must work, those user flows, those journeys.
00:07:21 If you can automate all that stuff that gives you kind of a bit more free time to focus on exploring new features and understanding the application that way. So it works really well together.
00:07:33 Yeah. Plus, I mean, maybe some people enjoy it, but the whole reading the written script of what I’m supposed to do and typing things in that’s boring, especially if you got to do it like once a month or something like that.
00:07:45 Yeah, it’s boring, and it takes a long time to it’s like maintaining code.
00:07:51 It might be written in plain text, but you still got to maintain it. It’s still going to match the code. So every time the code changes, you then got to go back to those test and code the steps, which invariably no one does. So they become awfully out of date, and then they’re difficult to run.
00:08:08 Sometimes they’re like hints at the indication of things that you’re supposed to kind of test, and the people running the test know what that means. But then you hand it off to somebody new because somebody’s on vacation and they’re like, I don’t know what this means at all.
00:08:25 Anyway. Well, we wanted to talk about Playwright, and so what is Playwright and what’s your involvement with it?
00:08:36 Okay, so Playwright is a Python package for end to end testing of web user interfaces.
00:08:48 So it’s basically a tool set for an API for automating mostly web browser interactions. So navigating to a page entering text, clicking buttons, that sort of thing. All the things you’d expect that a web page should be doing or an application should be doing you’re able to do using Playwright.
00:09:17 And the idea is that using Playwright, you can automate your, like we mentioned a minute ago, your end to end journeys or end to end flows of your application.
00:09:28 Okay.
00:09:30 It’s fairly new, and it came out last year, went to version one last year.
00:09:38 So it’s kind of the kind of new kid on the block, as it were, but it’s growing in popularity.
00:09:44 I think people might be used to Selenium. Is this fit into a similar space of Selenium then?
00:09:52 Yeah, absolutely. So Selenium is very similar in the sense that it’s a toolkit which provides you all the things you need to do to automate browser interactions. So Playwright, you could consider a competitor, or probably more so an alternative to Selenium.
00:10:15 Situations where Selenium is a better fit. There might be situations where Playwright is a better fit. The really great thing in terms of Python is that we haven’t really had many options.
00:10:28 You kind of had to use Selenium.
00:10:32 If you’re writing tests in JavaScript, for example, you’ve got a bit more freedom. There’s a few more options for you, but we haven’t really had that in Python, so it’s quite nice to have something else, another option to use.
00:10:47 Definitely. Okay.
00:10:54 This episode is brought to you by PyCharm. I love how easy it is to run tests from PyCharm. Really easy super powerful. And then when my code is ready to commit and merge, that’s another place where Pytram really shines. There’s a nice green checkbox icon for committing. Hit that and you’ll see all the files you’ve changed and a dip on the highlighted file. You can revert files you didn’t mean to change or simply Uncheck them, and they won’t be committed. You can even do that within a file. Say you added a print debug statement you forgot to take out. Just Uncheck that part of the file and it won’t be part of the commit. I can’t count how much time PyCharm gets. Support has saved me. Try PyCharm yourself at testandcode.com. Pycharm SaveTime. Use PyCharm now your involvement. Do you use it day to day or is it?
00:11:45 I’m a user, so I spend quite a lot of time automating websites, writing automated tests websites, and I quite like using if I get an opportunity to use Python, I like to use it.
00:12:02 So Player itself was started as test and code library node. Js library, but they bought out bindings for Java and Python and C Sharp as well, I think.
00:12:18 So I’ve been using it when I can. I’ve been using the Python language bindings.
00:12:26 Cool and playing with it.
00:12:28 Yeah, and it does look fairly full. We’ve got support for Test, Firefox, Chrome, and WebKit. That kind of crosses the gamut, doesn’t it?
00:12:38 Yeah, I think that’s one of its biggest kind of selling features is that it’s cross browser like you mentioned.
00:12:48 You can test on Chromium, WebKit and Firefox browsers. And the really cool thing is that’s part of the installation process. So there’s no need to have Web Kit for Safari, for example, installed on your machine. So if you’re running a Windows machine or a Linux machine, all you need to do is pick, install, and then you get the instances of those browsers as part of the package. So you can then go away and test on Chromium, Firefox, and WebKit without having to manage external browsers and web drives.
00:13:26 There’s a headless option.
00:13:28 I don’t know if it’s defaulted on or not.
00:13:31 It is, yeah.
00:13:32 Okay.
00:13:32 Yes, defaulted headless.
00:13:35 So you have to pass in if you want to see what’s going on, you have to pass in a headphone.
00:13:42 When you’re launching the browser, you see nothing. But again, we kind of see the shift in the market, especially with UI tests, which have traditionally been a bit slower than maybe kind of unit tests and API layer service layer tests.
00:13:59 So we’ve seen a shift to kind of make those faster. And one of the ways you can do that is obviously run headlessly, which works perfectly well, especially if you’re running kind of CI CD. You’re not going to be looking at the screen anyway, so you may as well run them.
00:14:16 Headlessly I imagine it’s probably the only time you do not want to do headless is maybe when you’re developing something and it’s not working you can kind of watch it run and see where it’s going wrong or something.
00:14:30 Yeah, exactly.
00:14:35 As you say, when you’re creating the test, it’s quite useful to have it headfall. And that flow Mo option is really useful because even if you’re watching it, it’s super quick sometimes. And if you’re closing the browser at the end of your test, all of a sudden it flicks up, you can’t see anything and it closes again. Just to be able to slow it down is quite useful.
00:14:59 Are people using things like this for really testing the entire application, or are they using it to test the user interface part?
00:15:10 Do you know what I mean?
00:15:12 Yeah, it’s the user interface sort of thing. So this is the kind of whole efoss behind Playwright is end to end user journeys.
00:15:21 So it’s really focused on being able to write end to end test. So the idea is that you can take a full user flow or journey and test the entire process in a single test.
00:15:37 And so these are probably going to be in conjunction with other tests, like if you’re really beating up on a particular algorithm or something, you may be able to do that at an API level or something, right?
00:15:51 Yeah, definitely. There’s kind of this concept of a test pyramid, and the idea is that at the lowest level, you have as many unit tests as you can, and then above that, everything you can test at API level.
00:16:08 And as you go up the pyramid, you tend to find that you have fewer and fewer tests. And right at the top of that, you might have some end to end tests.
00:16:18 So even before your end to end test, you might have some tests which leverage the UI but might not be end to end. They might be kind of a bit more focused, a bit more specific about a certain element in the page.
00:16:37 And then on top of that, your end to end test, which you might not have many of, are really focused, as I say, on driving those flows.
00:16:48 Those big work flows. Okay. Have you used Selenium before?
00:16:52 Yeah.
00:16:53 Okay. So what are some of the things that stand out to you that maybe Playwright offers that you can’t get from Selenium? Or why would you use that over Selenium?
00:17:02 Yeah, I think it’s kind of what Selenium has got in its favor is it’s incredibly stable and it’s been around for a decade, so there’s lots of documentation.
00:17:16 But what has happened is that actually kind of Web developments changed quite a lot in the last decade.
00:17:24 We’re seeing a lot more single page applications and a lot more going on web applications Loading, a lot more behind the scenes.
00:17:35 So there’s a couple of features that Playwright have which are really useful.
00:17:41 So it also waits for elements in the application to be ready. So one of the challenges with Selenium has always been how do I manage it’s often quicker than the web application, so it starts typing in fields before they’re visible, or tries to click a button which hasn’t finished Loading yet.
00:18:09 And then you get lots of errors and can’t find this element Playwright uses is built in, waiting to wait for elements to load, so it won’t try and enter text or press a button before it goes through a checklist of is it visible? Is it enabled? Has it stopped animating it does that checklist before it even attempts to interact with that element. Okay, so that tends to make test a little bit less flaky.
00:18:44 The other thing it’s got in its favor is you can interact with the networking behind the scenes. So again, you might have a web page which fires off as soon as it loads up, fires off a whole ton of Http requests or XHR requests, and you can monitor that network traffic using Playwright, and you can do things like stop certain APIs being called. So block them from returning information to the page, or even mock them out.
00:19:25 So if you’ve got a long list on an application on a web page that you don’t want to return because it’s time consuming, you could mock out the sort of JSON response of the request, and then you could send that back to the front end instead.
00:19:47 Or if there were particular Http status codes you want to test. So it’s often difficult to test those error scenarios. If you want to produce a service unavailable or something like that to see how the web application handles that, you can mock that out. So you could send back a Http 500 or additional header in a response and just see how the application interacts handles that.
00:20:21 So that’s really useful. It’s good, again for games, for performance that you could do things like stop every request for an image being returned to your page. So again, if there’s something very specific you want to check on your page and you were running Headphones for some reason, then you could block every API request which returns an image, and again, that just stops the whole page Loading and speeds things up.
00:20:52 Are you normally running locally, or is the application running on a test server or something?
00:20:59 It varies.
00:21:01 I think kind of the more modern scenario maybe, is that you have something running everything. Maybe it’s through CI CD. So maybe every time your application gets built, it gets automatically deployed on a test server, or maybe even Docker, and then at the point it’s built, your tests are triggered and they hit that wherever that endpoint has been deployed.
00:21:34 Actually, Playwright, there’s a Docker image for Playwright, so you can run your tests on a Docker image.
00:21:44 And there’s also a pre built GitHub action as well. So you can hook it all up through GitHub and trigger set up your GitHub action to run your Playwright test.
00:21:58 Okay, this looks neat. Now one of the things it looks like it has support for is Async stuff. Does that help with testing and where would you use that?
00:22:09 So I wouldn’t use it.
00:22:12 Okay.
00:22:12 I don’t think there’s necessarily any benefit. So I think part of the reason is because it started off as a node library. It’s all async in node. Js, so I don’t know whether it’s a kind of attempt to keep things in line and familiar.
00:22:34 Secondly, I’ve heard of people using it if their application is heavily reliant on Async. Io as just a way of keeping that level of continuity. And if you’re a developer writing Async code all day and you’re involved in writing EndToEnd test as well, then maybe it’s a sort of familiarity thing that is kind of consistent across the two.
00:23:05 Typically, I wouldn’t unless you have a use case, I don’t see any specific value.
00:23:11 Well, I was looking at the GitHub pages for both Playwright Python, and I’m guessing that’s the Python bindings for Playwright and the Playwright pytest plugin, and both of them are under the Microsoft umbrella. So is Playwright a Microsoft thing?
00:23:31 Yeah. So there’s a team at Microsoft maintaining it. I think the story goes that there’s a node library called Puppeteer, which I think is a similar sort of thing, and I think that was developed by a team at Google. It’s open source, and Playwright is a fork of that original Puppeteer library.
00:24:00 And I think even the team that worked on Puppeteer at Google, maybe the same team who are now at Microsoft working on it.
00:24:12 All right. It’s cool that there’s a pipes plugin. Do you use the pipe test bindings for it?
00:24:18 Yeah, I’ve started playing with it a little bit.
00:24:22 It’s nice.
00:24:24 I need to look at it a little bit more and explore it. But it’s nice that there is one, and it cuts out some of the setup.
00:24:35 I mean, great that there’s plenty of things, but what would it provide? And one of the examples is on the page was kind of neat. Was setting up a fixture to a session scope fixture that automatically sets the viewport so that you have, like, width and height already set or something.
00:24:55 I guess that makes sense to set some of that stuff up at once and not have to worry about it later.
00:25:01 Yeah.
00:25:04 It’s nice to kind of leverage the plug in for that kind of pytest for that kind of thing. And yeah, it just cuts out some of the not that there’s much boilerplate stuff, but it cuts out some of the having to create the driver.
00:25:23 There’s an example on the Bytes plugin page of modifying the run test make report hook so that it grabs a screenshot on test failure. That’s a pretty cool idea.
00:25:36 Yeah, that’s nice.
00:25:38 Playwright makes it this is just a single command page, dot screenshot or something like that to grab a screenshot. So it’s quite nice. You can create screenshots really easy. Again, that’s quite useful.
00:25:55 If you’re not watching the screen the whole time that you can go back.
00:26:00 And if I’m running Headless, does the screenshot just capture a random portion of my screen, or does it?
00:26:07 Yeah, I don’t think it will capture anything worth seeing.
00:26:13 Cool. Any other cool stuff about Playwrights that you want to share?
00:26:20 What’s quite nice about it is that it uses what they refer to as context, so it’s quite easy. And what a context is is kind of like an instance, almost like a tab or incognito instance of a browser.
00:26:44 So when you’re running tests, what can be time consuming sometimes is spinning up a browser for each test.
00:26:52 But with Pi test, with Playwright, what you can do is spin up the browser once and then use context.
00:27:02 And you can spin up as many contexts as you like.
00:27:05 And each of those context is quite quick to build and then to destroy at the end.
00:27:15 It’s just kind of an attempt to make tests run a bit quicker without having to launch a browser each time.
00:27:23 And each of those contexts can then have multiple pages on them. So if you want to test an EndToEnd journey which covers two different pages, you can launch multiple pages and interact with them both.
00:27:40 Or maybe on top of that, you can launch multiple iframes and things like that. You could have a single context which has got a couple of pages which have each got a couple of iframes on. And you can really drill down into the different levels and manage it all under a single browser.
00:28:01 That’s cool, especially with like a user interaction thing.
00:28:05 Like, for instance, I might want to go through in the workflow of something you want the user to log in, of course, and then go to a certain page, and then you kind of want to do a parameterized thing where you’re hitting maybe 15 different workflows through a little section of the workflow and having all of the set up before that be something you can just refer to. That’d be very handy.
00:28:34 Am I getting that right? Would you use a card?
00:28:36 Yeah, that’s right.
00:28:38 And it’s good for scenarios. I heard someone mention a scenario where they needed to open up a file on an application or page on an application.
00:28:50 Whilst that user had that page open, it had to be locked for everyone else. So it’s quite useful.
00:28:56 You can navigate to that page as one user, then spin up another context, login as another user, then try and access that same page all in the same test and check that it does what it needs to do for the second user.
00:29:13 Oh, nice.
00:29:14 Yeah, it’s nice because you can share things between those contacts as well. So one of the things they’ve got in the documentation is about kind of sharing authentication.
00:29:29 So if you log in to an application using one context, not having to do those login steps again, but being able to access the cookies and essentially copy over those cookies into a second context.
00:29:46 Okay, then don’t have to log in again.
00:29:50 Oh, that’s nice. Cool.
00:29:52 It’s kind of just a different play on the whole driving a web page automatically.
00:29:58 Yeah, just another alternative. It’s good. Tell me again about your consultancy.
00:30:05 What’s it called?
00:30:06 It’s called Hal QA howqa. Okay.
00:30:09 And if people wanted to hurry, that’s where they would go.
00:30:17 Yeah. So we do mostly kind of functional testing of applications, mobile web applications, back end things.
00:30:30 I’m trying to branch out a bit more and do a bit more performance testing, accessibility testing and that sort of thing as well. But yeah, nice. That’s where you can find me.
00:30:41 Well, thanks so much for teaching us about playwright today. I’m excited to give it a test run.
00:30:49 Yeah, you should.
00:30:50 All right. Well, thanks a lot. And we’ll keep in touch.
00:30:54 Cool.
00:30:54 Great. Thank you.
00:31:01 Thanks, Ryan. That was a fun talk. Thank you. Also to Patreon supporters, join them at testandcode.com support. Thank you, PyCharm, for sponsoring the show. Save time. Use PyCharm. Check them out at testingco.com. Pycharm that link is also in the show notes@testingco.com. One, five, three. That’s all for now. Now go out and test something.