Visual Testing has come a long way from the early days of x,y mouse clicks and pixel comparisons. Angie Jones joins the show to discuss how modern visual testing tools work and how to incorporate visual testing into a complete testing strategy.
Transcript for episode 141 of the Test & Code Podcast
This transcript starts as an auto generated transcript.
PRs welcome if you want to help fix any errors.
00:00:00 Visual testing has come a long way from the early days of XY, mouse clicks and Pixel comparisons. Angie Jones joins the show to discuss how modern visual testing tools work and how to incorporate visual testing into a complete testing strategy.
00:00:28 Welcome to Testin Code.
00:00:39 Welcome to Test and Code. I am thrilled today to have on the podcast, Angie Jones. So welcome, Angie.
00:00:46 Thank you.
00:00:47 How are you?
00:00:48 I’m really good. It’s a nice, cool day. Heated up for a while, but it’s being nice now.
00:00:52 Nice.
00:00:53 I’m in Oregon, but where are you at?
00:00:55 I’m in Oakland, California. Bay Area.
00:00:57 Well, you’re kind of a force in the testing realm lately, or in the last several years.
00:01:04 Yeah.
00:01:05 For people that don’t know who you are, can you tell people what you do and who you are?
00:01:09 Yeah, sure. So I’m a developer advocate in Java Champion.
00:01:14 I work at Apple Tools, which is a start up that does visual testing.
00:01:21 I am an automation architect, so I’ve been in test automation for the majority of my career. That’s where my passion is. But I also do full stack development as well. So little taste of everything. But my passion is in testing. Yes.
00:01:38 Okay. Did you say Java?
00:01:40 Yes.
00:01:41 Okay. Is most of your testing in the Java space then?
00:01:45 So I do back in testing. I do front end testing. So, yeah, it doesn’t really matter what your application is built in, but a lot of teams tend to align with whatever they’re using for production development. So if it’s written in JavaScript, then you might want to use JavaScript or whatever. So I dabble in a bunch of languages. I used to work at Twitter as an automation engineer and we would support multiple different teams. And so I would have to code in probably like two to three different languages every single day. So, yeah, it’s a mix.
00:02:23 Okay. I know we wanted to get into visual testing, but you brought up the term automation engineer and I guess I don’t really know what that is. What is an automation engineer?
00:02:31 Yeah. So an automation engineer is a developer whose test is not necessarily writing production features, but automating things. So automating tests or automating processes, very common in DevOps sort of arenas. And so my focus has been mostly on automating tests, again, like UI test or API test, component level, test unit test, all the different types of tests. But yeah, automation engineer would be someone who’s not necessarily focusing on the exploratory or manual testing, but automating those tests.
00:03:13 Okay. And then also I think it’s associated with Apple Tools as well. You’ve got some sort of testing school thing going on, right?
00:03:21 Yeah. Test Automation University. So Test Automation University is sponsored by Apple Tools and I lead this project. It’s essentially an online learning platform with courses on all things test automation. So there are courses on UI Automation mobile API code list as well. And there are courses in all of the different. Well, not all, but the popular languages such as Java, JavaScript, Python, Ruby, those things. What’s great about this initiative is that the instructors are leading experts in the testing space. So it’s not just me teaching, but I’ve basically reached out to a lot of my peers and they’ve offered these courses as well. And since the whole initiative is sponsored by Apple Tools, the courses are available absolutely free to the public.
00:04:24 Okay.
00:04:25 Yeah.
00:04:26 Where do I find that testautomationu.com?
00:04:29 Okay, I’ll have to check it out more.
00:04:32 Yeah.
00:04:33 Also, one of the things that in the automation and test automation space, there’s kind of I don’t know if it depends on the company or the team that you’re on or something, but a lot of times there’s just, I don’t know, the end product, the shippable product developers, they’re doing the testing as well while they’re developing the features.
00:04:55 And then on other teams, there’s like, independent people that are doing the test automation, and those might not be people with engineering backgrounds. Is the test automation you something applicable to both groups?
00:05:10 Yeah. So we have a range of different types of people who study at Tesla University. So it’s been around, I guess about a year and a half now we have 60,000 students, so it’s really popular. And I took a look at just a small sample of who made up like the top 1% of students there. And I was surprised by the range of people. So it was developers.
00:05:44 They have the coding skills, but not necessarily the testing skills. So they want to learn how to better test their software. There were testers, of course, like you said, don’t have the engineering background but want to get into coding or automation. And so lots of those. But there were also managers who were taking it, I guess, trying to evaluate different tools and things like this, keeping their skills up to date. There’s people who switch from other disciplines. So one of my favorite stories is one of a woman who was working in HR, and so she’s using Test Automation University to learn how to get into like a more technical role. And we don’t just provide the test automation courses themselves. But in order to do test automation, you have to know how to code. So there’s a Java course that I taught. There’s JavaScript, there’s Python. So there’s also these programming courses for people who are interested in learning how to get started in coding.
00:06:52 That’s cool.
00:06:53 Yeah.
00:06:54 It wasn’t what I was going to focus on, but I see some familiar faces. You have Boss Dijkstra, Andy Knight, Paul Meryl. Lots of familiar faces in there. That’s cool.
00:07:05 Yeah.
00:07:06 At least a Crispin even. That’s neat.
00:07:11 Thank you pie chart for sponsoring this episode. I first tried PyCharm when they started supporting pytest many years ago. Their support for pytest is now amazing. I was a long time vim user, so next I needed to test the idea. Vim plugin. So all of my finger muscle memory still worked while editing. Check. It works great. There’s lots of reasons to love Pie chart, but for me it is because they have the absolute best user interface for test automation. Then I learned many more ways by charm can save me time. Like really great support for editing, markdown HTML CSS, JavaScript, remote connections to database, and amazing version control support. Really, it’s the best Get diff tool I’ve ever used. And now version 2023 is out and the Shift Shift, the find anything key sequence even lets you search, get commit messages. What even that is so awesome. Tons of other cool features have been added in 2023. Check it out and I hope you enjoy it at testandcode.com. Pycharm when you said you were interested in talking about visual testing, I have to admit I kind of cringed because I guess I just remember, I guess the last couple of decades we’ve had a lot of products come out in the last couple of decades that promised easy visual test automation and they’ve been train wrecks.
00:08:37 And a lot of it is because a lot of them were like tracing mouse clicks, X, y coordinates, and like Pixel comparisons and stuff like that. But I think it’s moved on. It’s a different world now, right?
00:08:53 Yes, absolutely correct. So the Pixel tools, those folks are still out there. The way I describe it is it’s kind of using a data technology. Just as you mentioned, it’s cringe worthy. It really wasn’t built like those sorts of technologies weren’t built for testing software.
00:09:14 And so you have a lot of flaky test and code negatives and you’re being pinged because all the cursor is in the wrong spot. So the whole thing blew up.
00:09:29 However, the company I work for, Aptitude, is using visual AI in order to mimic the human brain and eye to compare those pictures. So it’s a much different experience. I often tell people about like, you know, those little spots of different games. So I ran one of those where the two images had some differences. I ran that through like a Pixel to Pixel comparison tool versus the AI driven tool, and the results were startling. The Pixel to Pixel comparison, just as you said, highly sensitive. It was picking up every little white Shift change. It just was not conducive for testing versus running it through. The AI spotted the differences, just as I would have as a human being. So, yes, visual testing has gotten much more mature.
00:10:25 Yeah, there are some tools, and I think Apple Tools is one of them that also does Dom comparison.
00:10:32 Yeah, Apple Tools is not one of those.
00:10:34 Okay, so there’s like three classes of visual testing tools, right? So the first class is the one we described that does the Pixel to Pixel. I promise you that’s not what you want unless, you know, you have like a really basic app with nothing much going on on it, but anything more advanced, you start getting into trouble. The second class does the Dom comparisons like you specify. And that’s kind of similar to what the normal functional test automation tools do. When you think about those, when you’re asking something like is this text present on the screen? It’s not looking at the screen, it’s querying the Dom to see if that’s there. So those visual testing tools kind of fall into that same category where it’s comparing things like the CSS to see if there have been any changes and then alerting you with that. And then there’s the class, the AI class, which we’re in a class of our own, really, we’re the only ones using this approach, but it’s essentially looking at the surface. So doesn’t look in the Dome. It compares the pictures, like I said, using machine learning. But there’s also like root cause analysis built into. So it kind of couples both of those approaches where it uses the machine learning to compare the photos. But if there are any differences, it’ll then go and start querying like the CSS to see what changes were made so that it could give you a nice report to say, oh, this is why it failed.
00:12:11 Because X. Yeah, I actually watched a little presentation you did where you had an example where, I don’t know, there’s some text that was just like off the screen or in the wrong place and a person would totally see it right off the bat and say, this is ridiculous, but that’s hard to test from a Dom perspective.
00:12:34 Yeah, that’s where those visual bugs live in those viewport changes. Right? So right now we’re in a responsive world where everyone is expecting your application to work on every single device, whatever device you happen to have your app that will work on it. And that’s really hard from a development and a testing standpoint. And those visual bugs, that’s where they live.
00:13:00 That’s when you start seeing your app get a little wonky when you start changing the size of the screen. And you didn’t really prepare for that.
00:13:10 Okay, so now we’ve got a whole bunch of tools then. So if I take this seriously with end testing and API test, let’s say let’s pick a particular application. So I’ve got an application that does have a web front end, but there’s also maybe, let’s say I’ve structured it fairly well and there’s a Rest API or something also that I can get most of the functionality from. So now I’ve got decisions. Do I test locally? Do I test against a server against the Rest API? Do I test against the Dom? Do I test? And then do I do visual testing or possibly all of them, but in certain percentages. Do you have any advice around that?
00:13:52 Yeah, I typically follow the most common models out there, like the test pyramid, for example, to say like, okay, bulk of my test should be done at the unit test level. But I’m also not naive enough to think that that covers everything I need covered. Right. So yeah, there may also be some API or when you start getting into your integration test, that gets really interesting as well. And even if I test things from the API level or back end level. Right. Let’s say I’m just going to test all my data and stuff here. There’s still this presentation layer that I believe should be tested. Right. And that does not mean I need to run like, all of my tests at this level. No, but I think that the visual testing really helps here because I don’t have to do much except like, oh, go and make sure my app looks good and I can bypass probably a lot of the navigational things and the entire, like, in the end scenario or whatever. I could run some API calls, get my application and state it’s supposed to be in. And hey, just make sure that looks good. Cool.
00:15:09 And then verify the rest on the back end.
00:15:11 I’m still envisioning. And maybe it’s just my own history that a lot of the I guess unit tests are very focused by definition, but even I like to focus like API test as well at a narrow bit of functionality.
00:15:26 But often like, because if we had like a manual scenario, manual test script or something, I want to try to be cognizant of the testers time. So have a workflow that does maybe a common user scenario that hits a whole bunch of my application, but it’s easy and to follow and it kind of hits a lot of stuff. Do I do the same sort of thing when I’m doing something like a visual testing to try to have like one or two or three tests that hit a whole bunch of my application?
00:16:00 Or do I still want to try to do a focused thing?
00:16:05 Yeah, that’s a good question. I think. Of course, it depends. But I’ll say this, when you do the visual testing, you get much more bang for the buck. Right. And the reason for that is because your pictures are worth a thousand assertions. Let’s say, for example, that let’s take a scenario where I want to test that I can decrease the quantity of an item that’s in my shopping cart. All right, so I’m on the shopping cart page and I decrease that item, and then I’m ready to assert on that. Now if I were to write this functional assertion, I would probably make sure that the number decreased by one. Right. So it went from three to two, for example, right?
00:16:50 Yeah.
00:16:51 Okay, that’s pretty good. But I bet you can imagine a whole bunch of other things that could go wrong there that we’re not accounting for. Like did the price update. Oh, yes, snap. Let me write an assertion for that, too, right? Yeah, did it affect anything else in the cart? What if there are other things and everything went down by one or something like that? So there’s a whole bunch of stuff. And then you start saying, I don’t want to write like, am I being paranoid here? Am I going overboard?
00:17:20 So you’re on this fine line where you don’t know if you’ve written too little or too much, but when you have that visual testing, you basically say, okay, let me decrease it by one and take the picture. And now you have all of these assertions kind of built in already where it’s going to look at everything there to determine if everything is correct there. Right. And that’s like a single line where I just say, yeah, Cert, you know, and it’s going to make sure all the products are right, the quantity is right, the title didn’t change, nothing is overlapping each other. So you kind of get functional assertions built in as well, which is an amazing side effect.
00:18:03 So a picture is worth 1000 assertions. Did you make that up or did you make that up? That’s awesome.
00:18:09 That’s good on a T shirt.
00:18:13 You deserve a bonus just for that.
00:18:16 Anyway, actually. So I went into this thinking, yeah, I don’t think anybody can convince me that visual testing is worth it. But I got to tell you, I don’t know, their sales pitch is good.
00:18:29 It totally makes sense. Okay, so when I’m teaching people just even like at an interface layer testing or an API level testing, there is this, okay, I’m not addressing what you said, but the idea of a workflow test is a similar sort of thing that they’re horrible when they go wrong because you don’t know what failed because like a million things have to go right for it to work. But when they pass, you know, a lot.
00:19:05 But the visual testing is kind of neat. And I like how the highlighting happens and the comparisons and stuff. So the snapshot sort of stuff, I’m kind of brushing by it because I know it already, but there’s an API, so I still have like these text, or at least I can have these text based tests that call the Apple Tools thing API and say, hey, I’m at this part of the workflow, I want to take a picture here, and then you just sprinkle those through in reasonable, knowable parts.
00:19:39 Right, right.
00:19:39 And then do you have like a baseline where you just say, okay, run it against a good version, and then we’re just going to keep that tool running again.
00:19:48 Yeah.
00:19:49 A lot of the tedious stuff, like taking a picture and managing all those images happens behind the scenes. So we as engineers don’t have to worry about that. Just like you said, you just make the calls wherever you want. Right? So I do take a picture. Okay, now I want to do take a picture. And those pictures are then sent up to the Aptitude cloud and they’re stored in a dashboard for you. So everything is there and it knows what to compare it to based on a number of parameters. Right. So when you say that you want to take a picture, you’re giving it the name of your test and the name of your application that you’re testing against. Right? So it takes those, but then it also behind the scenes, combines that with, okay, what browser and version is this running against? What OS and what are the dimensions that they’re running against? And all of those things together make like this unique identifier. So if I were to run that same test, let’s say I ran the test on Chrome and then I want to run that same test on Firefox where it’s not going to compare the Firefox image to the Chrome image because browsers rendered differently, but it will create a new image. Like, oh, she hasn’t run this one against Firefox. So this is a new test and it’s automatically going to create that as a new baseline.
00:21:25 So how do I know that? I guess it’s the primary part that I’m still confused about. How do you know that it’s working in the first place?
00:21:31 Like that initial one?
00:21:32 Yeah.
00:21:33 Okay, so you don’t have to check it. But you’re like me, right? We have a testing mindset. So you’re like, how do you know that it works? So you go to the dashboard after your test, if you want to, you go to the dashboard and then you look at the baseline image that was captured and you can verify that, just like we do when we write any other automated tests. Right. You’re making sure everything actually looks okay before you commit this to being an automated test. That’s going to be run several times a day and gate deployments.
00:22:07 So same kind of thing. You can go to the dashboard when you run it that first time, see what was there, and you say, yes, that looks beautiful. Nothing to do, right? If you hate it, you delete it. But if it’s fine, you just kind of leave it there.
00:22:21 Okay. So I can kind of like have that open and then have my code open and then walk through and go, oh, there’s, that snapshot.
00:22:29 Nice. That makes sense.
00:22:32 Cool. I thought maybe you just had to know it was good to start with or something.
00:22:38 Okay, cool. How long have you been working with visual testing?
00:22:41 I started working with it when I was working at Twitter. We used it.
00:22:47 It’s funny because the Twitter test, some of them were like dynamic type of tests because we didn’t have these stage tweets. Right. So you don’t know what the tweet is going to say. And you almost think, Well, I can’t use visual testing if it’s going to be dynamic like that. But that’s not true. So you can use this mode. There’s different modes with the Apple TV and the Apple’s API is called Eyes, which is amazing. Right. So with the eyes, you can set like these different match levels. So by default, it runs on strict. There’s also, like one that does Pixel to Pixel. But of course, I told you that’s not the way to go. So don’t use that one. There’s one that goes colorblind and just verifies the content. And then there’s one that verifies the layout of your page. Right. So it’s using machine learning in a different way to detect the pattern on your page. Right. Oh, you have let’s take a tweet. It has different components that are standard for a tweet. So there should be a profile picture in the left corner followed by your name and then a username, and then some context, which could include media, et cetera, time and date stamp, et cetera. Right. So I don’t want to verify what all of this says exactly. I don’t need to know that it’s October 15 at twelve noon. But I want to make sure those tweets are not overlapping or my ads are not bleeding off the page or something like that. So that’s where I started using visual testing in the dynamic mode to test the tweets. And I would say that’s been, I guess about three or four years now.
00:24:37 Okay. You said you’ve been doing test automation a lot or automation of lots of kinds. A lot of the testing. Now you’re a developer advocate for Apple Tools, is that right?
00:24:48 Yeah.
00:24:48 You’ve been speaking a lot, too. Is this speaking something that started with your developer role, or is that something you’ve done before?
00:24:54 Yeah, I started speaking in 2016. I was like, oh, I want to speak at a conference, just kind of like one.
00:25:05 And I did it and I enjoyed it.
00:25:09 Before I started speaking, I was blogging a little bit, and so I started kind of gaining a following that way. And when I started speaking and I was getting requests to do more of that, so I started taking them up on that, and it just kind of grew like out of control.
00:25:29 I’m serious.
00:25:32 This was even before my Twitter days. This was at another company. And it got to the point that I was gone more than I was in the office. So when I went to work at Twitter, I’m like, hey, this is a big part of my life. It’s that cool because I don’t want to go anywhere where people are going to have problems with it. Like, no, it’s fine, it’s fine. It even grew more, I guess, when you start working at one of these notable companies or whatever, people want you on their agendas and stuff. And so the invitation started coming in even more when I was there to the point that I couldn’t sustain it, Brian, it was like working two full time jobs. I had to do my Twitter work.
00:26:13 You’re like working in the hotels and stuff.
00:26:15 Then, oh, my God, I would be like in Europe and I would give it an eight hour workshop, and then I would have to go back to my hotel room and put in 8 hours at Twitter, like working through the night and stuff. It was impossible. So it got to a point. I’m like, okay, I got to pick one of these things. And I was like, but these conferences aren’t paying you, so how can I make a living off of this? And that’s when I learned about this debris job where it’s essentially where you produce content, you do talks, you do blog posts, you make videos and stuff like that, where you’re educating people on tech, and you get paid to do that. So I’m like, that’s where I need to be. I didn’t even know that was a thing. So that was great. And because I was already familiar with Aptitudes and I knew them, the founders from the conference circuit and stuff, they expressed that they had a need for this type of role. So this might be like a match made in heaven. And it was so that worked out great.
00:27:22 So that’s awesome. I’m glad that that’s working for you.
00:27:25 Yeah.
00:27:25 The conferences have sort of dried up. Are you doing a lot of virtual conferences now, Brian?
00:27:30 Listen, they dried up in person, but everybody is doing a virtual conference now, so now I’m doing one to two of those a day now.
00:27:42 Are you serious?
00:27:43 Yeah, it’s kind of ridiculous.
00:27:46 So I’m trying to slow up on that front, but yeah, they’re still like, I think people are like, oh, we didn’t have enough conferences in real life, and now we can do them virtually. So everybody’s doing a conference.
00:28:00 Yeah. And it used to not be possible to do a conference in two different locations in the same day, but possible now.
00:28:09 True.
00:28:10 Yeah.
00:28:11 Wow. One to two days.
00:28:13 If I look for you’re probably all over YouTube, then, yeah. Wow.
00:28:18 I mean, if you’re doing you’re speaking a lot, what’s really exciting to you about whatever in the future now, do you have anything that you’re thinking might be fun to do personally or work related or any sort of test stuff that excites you in the future?
00:28:35 Yeah, I think Test Automation University was really exciting for me, so I’m really happy with how that turned out and how that’s going now, as far as what’s next, what’s in the future, I’m not sure. I’m one of those people who I don’t really plan a lot.
00:28:55 I just kind of go with the flow and keep my eyes open to what’s going on and things like that. So I’m not sure. But you never know with me. I just bust out with something like really dope out the blue, so you just never know.
00:29:10 Well, I’m excited to get that. We get more and more people talking about talking about automated test and code to do it in a smart way and you’re definitely one of the people doing that. So thank you for all of your effort. I think it’s great.
00:29:24 Thank you.
00:29:25 I do want to let’s see, we tell people to look for test automation. You if they are curious about other stuff you’re doing or pursuing any other places you want to send people.
00:29:35 Yeah, I live on Twitter so definitely check me out there. My handle is TechGirl 1908 and so that’s where I post like a lot of the stuff that I’m working on, any blog posts or upcoming talks and things like that. And I also have a website, Angiejones Tech, that you can check out where I write about test automation strategies and techniques.
00:30:00 Angiejones tech.
00:30:03 Hey, you’ve got your little logo thing, some artwork. Did you have somebody do that?
00:30:11 Yeah, I had an artist created for me.
00:30:13 Oh, that’s very nice.
00:30:14 Thank you.
00:30:15 Nice look. Cool.
00:30:16 Thank you.
00:30:17 Thank you for taking the time to talk with us today and wish you well in the future.
00:30:22 Okay. Thanks so much for having me, Brian.
00:30:29 Thank you, Angie. I thought that was a very interesting discussion also. Thank you Pie Charm for sponsoring the show. Try pycharmourself at testandcode.com PyCharm thank you, Patreon. Supporters join them at testandcode.com support the show notes for the episode are@Test And Code.com 141. That’s all for now. Now go out and test something.