Interview with Andy Knight, the Automation Panda.


Transcript for episode 47 of the Test & Code Podcast

This transcript starts as an auto generated transcript.
PRs welcome if you want to help fix any errors.


Welcome to Test and Code, a podcast about software development and software testing.

On today’s episode we have Andy Knight, also known as the Automation Panda. We talk about quite a few things, for instance Selenium and WebDriver headless, Chrome BDD with its Kirkin and Given Wind, and I think it’s a good conversation. I think you’ll enjoy it.

This episode is sponsored by JetBrains and PyCharm. My team uses PyCharm for Python development because it speeds up so many of our daily tasks. It’s the pytest support that really sold us code completion with fixtures and parameterization, and the ability to run a single parameterized test case saves us tons of time and make PyCharm an essential tool for test development. Using Get from within PyCharm allows for easy diffs against previous commits, makes partial commits trivial, and really makes using Get on the team on a large team really easy. The virtual environment support is great and keeps getting better. I don’t know which of the features of PyCharm are going to be the features that save your team time, but if you value your time, you owe it to yourself to try PyCharm. And you can try PyCharm Professional for free for four months with a four month subscription by going to testncode PyCharm. That link will also be in the show notes for episode 47.

Hey Andy, welcome to Test and Code. I’m super excited to have you here. Before we get into a bunch of stuff, you are not only known as Andy, you’re known as Automation Panda, but would you introduce yourself?

Sure. Thanks, Brian. I’m really excited to be on the podcast today. Yes, my name is Andy Knight. I live in Raleigh, North Carolina and I am the Automation Panda. You can follow me on Twitter at that handle. I also write a blog at automationpanda.com, so just a little bit about me. I am a software engineer in test. I specialize in testing automation with a developer’s heart.

My main specialty is building test automation solutions from the ground up, including frameworks, design patterns, even to the point of mentoring teams, doing some consulting, all that kind of good stuff.

Okay, well, what’s your background? How did you get into doing what you’re doing now?

Long story. So the TLDR is when I first entered software industry, I did computer science in College for my bachelor’s and Masters and I had an internship with IBM here in North Carolina, and my first internship back in 2007. They actually had me doing automation for the product that they were working on was an Eclipse plugin for the Rational suite of software. That was my first taste to test automation. After that, I did a whole bunch other things. I did some developments and databases, programming languages, yada, yada, all the good stuff.

But I really came back to testing and automation in a powerful way. When I worked at NetApp starting in 2011, I was brought onto a team. I was the most junior member at the time. And they thought, well, Andy’s fresh out of College, he’ll be good at doing all this programming type stuff. Ha ha. And so that became my first project, get the test automation scripts working in our lab. And then from there, I realized they were kind of icky a lot of problems. So I started rewriting them, making them better. And that’s when I really got the passion for seeing test automation as its own special domain within software development. And I just stuck with it. Since then, I’ve worked at many places now. I’ve worked with many frameworks and many languages. I’ve done a lot of tests, a lot of infrastructure type things to set up for tests, continuous integration.

And that’s how I got where I am today.

Okay. And then what made you start wanting to write on Automation Panda?

Yeah.

In software, we flip jobs a lot, unfortunately, right. It’s a good and bad.

It gives us a lot of exposure to different companies and areas and ideas. But then everything we do at one company kind of has to get left behind.

As I was working on stuff through NetApp and then later at Max Point and LexisNexis, I always tended to write a lot of Wiki documentation for my teams and for those companies, just because that’s just how it happened. And I always like to document what I do as well. My code is full of comments. I try to be helpful to other people. And I realized after joining LexisNexis, I was like, man, I just lost all my wikidoc again. And a lot of that stuff, it was really public domain knowledge, things like, hey, this is how you use this framework. Here’s an example project, here’s my thoughts on behavior driven development, stuff like that. And I realized this would be helpful for me to kind of keep my own notes and do a good job of that. And then I also thought, well, it’s probably also helpful to other people, too. And I saw other people writing blogs, and I thought to myself, I can do that. So I started Automation Panda in January of 2017. I actually registered it at the end of 2016, in December. But I made it my personal professional goal for 2017 to try blogging, put it out there, see what happens. And the quick lesson I learned from that was I write primarily for myself. I write what I want to write, and I’m going to do a good job of writing what I feel is right.

And with that, I just started putting content out there, things about ten things you lose without test automation or the benefits of BDD. And I even put like a whole BDD one on one series to get people ramped up little things like that. And at first, as with anything like blog is slow, you don’t get too many hits. But then over time, it kind of grows and grows.

You’ve got a lot of articles on here, though.

Thank you.

Do you have a count?

Yes. I have 102 articles as of today, I believe.

And you started this when?

January of 2017.

That’s not that long to get that many articles out.

Thank you. Yeah, I try to keep up with it. Anytime a new idea pops an article.

That’s great.

I love it. And you’ve got a lot of stuff covered. And I also like that it’s pretty opinionated, or at least it’s your opinion, right?

True.

No, it’s good. I don’t agree with a lot of.

Well, okay. I just default that I probably won’t agree with some stuff because I’m an opinionated person also. But sometimes I forget to write that way. Often I’ll write something down and it’ll be kind of trying to not it’s not that I’m afraid of offending people, but I also want to try to give a clear picture of what a topic is in the world, and then I’m okay with giving my opinion. But sometimes if I don’t know the bigger picture outside of my own opinion of something, sometimes I’ll just not write about it. And I think I need to take a hint from you and write about more stuff, even if I’ve just got a glimpse of what the big picture is.

Anyway. Sure.

Absolutely.

Good job.

Thank you. Yeah.

Being humbly opinionated is very helpful, I think. Right. Saying I have an opinion of this and I’m respectively putting it out there, and I’m going to be respectful of other people’s opinions as I can learn myself.

I’m glad you brought that up, because that does definitely come out in your writing that this is what you think of it, and it isn’t necessarily this is the truth. And if you don’t think this way, then you’re not a professional.

That’s something that really turns me off is people saying my way is the only way, and if you disagree with it, well, then you’re an idiot.

Yeah, that’s bad. It’s okay to be opinionated, but realize that there’s more than one opinion, even if most of them are wrong.

So right now you write this and you’re also, at least according to your LinkedIn, you are a software engineer in test, and you have in the past been a software quality engineer.

Yes.

So I’m excited because I never actually talked with a real live software engineering test before.

I guess there’s a couple of people that work for me that probably fit that description, but they don’t have that title. So is there a difference between a quality test engineer and a software engineering test?

Yes and no.

Unfortunately, a lot of companies, I’ve noticed don’t really nuance the title difference too much.

At least when I was a software quality test engineer or a QA specialist, I was doing very similar kind of work to being a software engineer and test. But I personally prefer the title software engineer and test for what I do because I think I mentioned before, I like to have the developers heart when I approach problems. I very much am a software developer. I’m in the mode of thinking about design and implementing solutions and treating things as features for the end user.

And with test automation, that would be your internal team, the developers who make it, the product managers, all those kinds of folks.

So that tends to be my approach to test automation. I see. Test automation development is software development. We use the same practices. It just so happens that instead of making a web app or a micro service, what am I looking for? A micro service framework.

I’m making tests and I’m making test frameworks, and I’m making automation solutions.

And I think a lot of more historically, the focus for testing teams and organizations has been more siloed and separate from development, where people historically have been more focused on just manual testing, test to break, more exploratory type testing. And while that may involve some programming or scripting, a lot of people in those roles don’t always have that background. And so those types of roles tend to have more of the QA or the tester type titles versus a software engineer title.

Yeah, I don’t know. Is over half of your day usually spent working with automated tests then?

Yes, I would say depending on the day, of course, but I’d say easily at least half of my work I’m fingers to the keyboard coding or at a whiteboard designing something.

Okay.

There’s a bunch of things I want to jump through. We could probably get off on a lot of tangents.

You’re testing primarily web applications or what kind of applications?

Yes. For the past two companies I’ve worked for, it’s been web applications. So Web UI on the front end usually driving that through selenium WebDriver and also testing services a little bit on the Rest API layer underneath of that.

Can you describe this Selenium and Web driver thing?

Sure.

Selenium Web driver has been around for over a decade or so. Now it’s the de facto way of doing black box web browser automation. There are some other tools, but WebDriver is the big standard that most people use, and Selenium as a project is the one who made the web driver standard, which has now actually been adopted by W Three C as a recommendation. So all browsers for the foreseeable feature should implement this interface. How it works is there’s this programming package called Selenium WebDriver, and it’s in all the major languages Java, C Sharp, Python, Ruby, you name it, boom. And you’ll claim that as a dependency on your test project, and then on the machine upon which the test will run, you also have to install an executable web driver.

Executable. You have to install a WebDriver executable for the target browser, be that Chrome or Safari is actually bundled like Firefox is the Marionette driver. Now, Ie has its own driver what happens is you will write your tests using the Selenium web driver interface in your native programming language. So if it’s Python, you would create the Selenium WebDriver for, say, Chrome and say, browser navigate to Www.whatever.com. And then you would say, okay, driver, click this button using this ID or this XP. And that’s how you model the interactions.

So you can write all your tests. You write all your tests just in like a test script or test function, and it drives the web browser.

Bingo, bingo. Because those calls will then kind of kick the little executable process, which actually acts as a proxy server. And that thing gets the thing from your code of your script, and that’ll kind of kick the browser into action. So you’ll see Boop, Chrome browser pops up, and then you magically see the mouse move and keys being entered and buttons being clicked and no hands, right? It’s all automated. And so, yeah, you basically control it from your code so you can put it in a framework like Pi test or it doesn’t even have to be a test framework. It could be anything, right?

You can just automate any web interaction with Selenium.

One of the questions I had about this is I guess I’ve got a lot, but usually utilize several drivers to use both like Chrome and Firefox and Safari. Or do you just primarily go through one.

Wonderful question depends on the needs and intensity of your testing effort.

Cross browser testing where you’re trying to test your application against multiple browsers. And not only that, but multiple versions and even multiple operating systems, all those combinations there that can get really sticky really fast.

For most cases, testing exclusively on Chrome or Firefox is good enough, and those tend to be the two easiest browsers. So most teams will start off there, and then once they hit bigger scale, they may decide they want to add, say, Ie or Safari as extra browsers.

My best practice that I recommend is you should only ever be using one web driver instance at a time per test case.

And so if you want to run the same test case across multiple browsers, there should be separate test runs.

For example, I’ll usually customize my test suites to say, okay, the browser you choose as an input, it’s a configuration option. And so when you launch the test suite, they’ll all be launched against one browser, let’s say Chrome, and then maybe the next day you can change it and launch them against Firefox and get coverage that way.

I think all the browsers now have a headless option. Do you use headless?

Oh, gosh, headless Chrome is the Bee’s knees.

Oh, I love headless Chrome. The reason why I like headless Chrome is not so much that it’s more efficient or more performance, because honestly, it’s about the same as regular Chrome, at least based on what I’ve seen on my setups. Just doing local testing what I love about Headless Chrome is when I’m developing my tests locally, web testing can be kind of intense, right?

It’ll pop up the extra browser and it’s kind of slow. I mean, order of magnitude, you’re looking at between 30 seconds to a minute per test, usually even locally. Like, right now, the tests I’ve been doing have been really short. They’re about 15 seconds, but still that’s long. Headless Chrome is awesome because you can set the Headless option and then fire off the tests, let them go on your machine, on the side, and just keep doing whatever else you need to do on your machine without being interrupted.

From a development standpoint, that’s cool.

Are you using usually Python with this or what do you use?

For the past two jobs I’ve had, my current company is Precision Lender here in Kerry, North Carolina. And then before that I was at Lexusnexis, also here in North Carolina.

Both of those shops have been Microsoft. And so I’ve been doing a lot of this web testing using C Sharp with Selenium.

Okay.

But I have done Python, and I love Python.

All right. Guessing the test frameworks within C Sharp or you got something cucumberish spectacle. Okay. And then do something unit test like.

Yeah. So if I’m doing unit testing with C Sharp, I’m going to gravitate towards NUnit or Xunit. Net.

Okay. What did you call the Gurkin version?

Spec Flow. Spec Flow is cucumber. For. Net.

It’s the officially branded version of cucumber. For.

Net. Is that primarily you’re writing a lot of your tests in Spec Flow then?

Yes, for web testing. I love using Spec Flow because I like using Kurt to describe my tests in plain language.

Yeah. So this is interesting. You’ve got a lot of stuff on your blog about behavior driven development and Gerkin.

One of the things about behavior driven development where it kind of took itself is to this communicating with other people.

Is that part of what you use it for? I mean, are you writing tests that other people, the nontechnical people are reading?

Yes and no.

Right now on my current team, I’m kind of a team of one. So I’m kind of writing tests in a bubble and then sharing with my developers. So that has not happened so much yet. When I worked at a previous company, that was more the case where we had a full team of test automation engineers sent out to Agile Scrum teams. And during the Sprints, their job would be to collaborate with the product owner and the developers on their teams, and they would write Gherkin together and kind of review it together and say, okay, is this what everybody has in mind for this feature? Yes. No, maybe so.

Okay.

Is it a hard thing for me to get into is this whole Gerken thing? And so I’m going to actually try to read some of your stuff to try to at least have an open mind and not shut it down directly, because I think there’s probably just something I’m missing because I like the idea. So I totally stole the Given Win then model from Behavior driven Development because I love it. Because there’s another model that’s almost the same thing, which is arranged act, assert, designer tests. It’s the same thing. It’s just different names.

I don’t really talk in terms of a range and act, but Given when then makes more sense. To me that is just a language thing, but I usually use it with BYTest and with just comments. So I haven’t done like Gerken stuff.

Is there a way to do this Gerken syntax in Python or pytest or anything like that?

Yes. So there are some Python behavior driven development frameworks. The one that I recommend most these days is actually pytestbd, the plugin for pytest.

Okay.

And it works the same way as Cucumber does. You write feature files with the given when, then Gurkins syntax.

And then what’s beautiful about pytestbd? It has all the power of Pi test underneath it. You want to do parallelization use pytest exist. You want to do coverage, Pi test, cover. You want nice, pretty reports, HTML.

The list goes on. Everything that you know about pytest is immediately available within a BDD context.

And right after Python 2018 JetBrain’s Pie Charm finally added support for pytestbd. So to me, that’s just like done, we’re good. We’ve arrived at a further notice. Some of the other frameworks you did mention behave. In fact, I gave a talk on behave at that. Pycon, that’s a good one.

Older ones like Lettuce and Freshen, they’re around.

Personally, I would not use them these days, given Python BDD being available.

Okay, well, I’ll definitely go ahead and just try that and walk through some of your stuff.

So what’s controversial about your view of BDD versus other people’s?

Okay, when it comes to BDD, actually, it’s kind of two sided. For one. I tend to be rather purist in how I recommend we write our Gerken steps. Right. Writing Gerken is fairly easy. It’s just given one then with plain language. But I found that writing goodgirken is hard.

People get writer’s block.

And then when you’ve got these multiple teams, everybody has their own writing style. And so you hit a lot of Englishy issues like should we be using first person or third person? What should we be parameterizing in our steps? How long should steps be? Is it okay to say given when then? Or should we be purely ordered in Given when then? And I’ve put a lot of articles out there on my recommendations and they are all opinionated because I found certain things to work better than others.

Some things make me cringe, some things make me happy. And so I put the things that make me happy as good, and the things that make me cringe as not good, along with the consequences. I see of each of those.

Okay, so that would be the purest side. On the more pragmatic side, where I tend to differ from most of the BDD practitioners out there is if you read stuff from Cucumbers blog or you go to Cucumfest or you listen to the big thought leaders for BDD, a lot of them are adamant that BDD is more about collaboration than test automation.

The purest focus on that side is that we want to have practices around BDB for things like example mapping, to come up with what the things and the features are, and the three Amigos meetings.

It’s not to the discredit of the test automation, but when you see the conferences and the talks and see the articles, at least my vibe recently has been the focus of that community has shifted very heavily towards that side. I still maintain that BDD based test frameworks are useful, even apart from the collaboration that goes on.

Right.

Even if I’m a team of one, as I kind of am right now a little bit at my current company, I still find value in a BDD style test framework, even if I’m not collaborating to other people. When I write test cases, I’m usually at a whiteboard. First I’m describing a test in plain English and then if I need to talk to a developer or product owner about it, I’m going to be talking with them in English. So I want the English there for all of us to see. And I’ll write out because I’m so trained now in my own girkinness that I just write the scenarios out in the plain language and then I’ll have the test formalized before I even touch code.

And then when I go to touch code, I can reuse a lot of the same steps without repeating code. Okay, that’s wonderful.

I’m actually looking at your BDD 101 writing good Gerkin article. And one of the things I noticed right off the bat is I don’t see some of the things that bugged me about other tutorials is a lot of the other tutorials that have started with given that I am a user, especially if I’m just testing an API or something like that, that doesn’t make sense. I mean, I’m the calling code, so I don’t have to do I am a user of specific role.

That’s not something I have to do with Kirkin. Is that correct?

Correct. The steps are totally free form. And I agree with you.

I’ll even broaden what you said. I think using firstperson and Gurkin is not a good practice for the precise reason you just mentioned. If I’m testing a service, I’m a service caller. Am I really an end user? Not really. It could be anything in the system that’s calling out to an API. Right?

Using. I mean, I does not fit in this context. And so you’re smashing something into bad language that should never be considered that way in the first place.

Okay, my original thought when looking at this, not at your stuff, but at Gerken altogether is isn’t this just hiding another API and I’m just making an API on top of the actual API? That’s this English language API, and it’s another layer between the test and the actual code.

That better buy me something big for all that extra work.

Do you think it does it’s enough of a payoff that it’s worth that extra layer?

Yes, and yes, you’re absolutely right. It is an abstraction. It’s an extra thing. There’s nothing you can do in Gurkin and a cucumber like framework that you can’t do with any other standard functional test framework. It does add that extra layer.

I like that extra layer because I like a separation of concerns.

There is my concern of what is the right test case. And then there is the concern of how do I implement the test case? And that layer lets me cleanly separate those so I can discuss what is the right test case with nontechnical people or technical people in a more plain language that’s easier to communicate. And then from that I have my test scenario, then I can focus on the concern of the implementation.

Okay, cool.

You’re a good salesperson for BDD and you convinced me. But I was actually kind of convinced at least to give it a shot because a lot of people asked me about it and I haven’t really Lee, I read enough to go, hey, give them when then is an awesome idea. I’ll use that and I ran with that and forgot about the rest of it because the rest of it just seemed to me like silly stuff. Well, I’ve also heard people say, even proponents of behavior driven development, that if they were not communicating tests with other people, then it’s a waste of time and you should just use something like pytest.

Yeah, you’ve heard that a lot.

Okay, I don’t know, it’s worth a shot. Maybe it will help design stuff I am afraid of a little bit afraid of using it on a team where possibly it could confuse the issue of if we’re using both pytest tests and something like PYT.

When do you use one over the other?

Yeah, so I can actually give some advice on that.

The higher level or the closer to the user your test is, the better you’ll find the BDD frameworks are.

That’s why I personally like to use them for web testing because it’s very natural for web testing.

I kind of step back from DDD type frameworks when I’m looking at, say, rest API testing.

You can do it, but it may not be the best usage of it. Or you may want to look at another tool that already specialized and specializes in that like karate.

Okay, specializes in what?

So karate is a test automation framework right now focusing on rest API testing, but they use a given when then type language. In order to do that their sales pitches. They’ve already implemented all the steps you would need for Rest API testing. And so you don’t even need to know Java or code. You can just use our karate domain specific language, but it’s the same kind of given when then type stuff but applied for Rest APIs.

Is that a commercial thing or open source?

Free open source.

Cool.

Neat.

And if I can shout it out, one more thing that you mentioned with pytest, it’s really cool about pytest and Pytestbd.

You can have a folder with Pi test test right next to a folder of pytestbd tests and run them all from the same command and even have the same markers, filter them.

Okay, so you said I can do a lot of things with pipes BDD. So can I do things like attach fixtures too?

I’m pretty sure. Okay, they’d have to come in through hooks, but I think you can. It’s been a while since I dug deep. Okay to it. Since I don’t use it every day, I think you might be pleasantly surprised.

I’ll give it a shot.

Cool man. Here to help you.

Well, you’re like a wealth of knowledge with this stuff, so it’s great.

I know that since my day to day job, I’m testing hardwareish things and not websites.

And I know that a lot of people that listen to this are listening to do test web things. And so if we rush by anything, this is talking to the listener. Now, if we rush by anything that you really want to jump deeper into, let us know. And we’ll try to put together a show that jumps into something specific, like maybe an entire episode on karate or something. I don’t know.

Cool. Yeah, it’d be fun. You said you gave some talks. Did you give a talk at Python?

Yes.

You said you started the blog because you were writing down stuff that you wanted to share between jobs or things that are open to everybody. Anyway, so how did you get into speaking?

Yeah.

So the blog was 2017 school.

Talking at conferences was 2018 school.

I just wanted to see how far I could take things. My blog traffic had really picked up. I was getting requests from people for advice, even some side consulting gigs. It’s like, wow, this is really cool. So I thought maybe speaking at conferences would be the next natural step in progression.

And because I had loved Python, I write some Django websites for my wife’s small businesses to help her out with things.

I thought, hey, it would be really cool to speak at a Python conference. I’d never been engaged in the Python community at all before and thought, let me just give a shot. So I went to the Googles and I searched Python conference and I saw Python and I was like, okay, here shot in the dark. Boom. I submitted about five proposals. One of them got accepted. And that was the talk on Behave test framework.

That’s awesome. That you got in rather quickly.

Thank you.

How do you think it went? Oh, gosh.

Oh, Python or the talk?

The talk.

I think the talk went pretty well. It was fairly well attended. I had some people come up to me afterwards and ask questions and like, good questions. They’re like, I want to learn stuff. And I think it was a good experience for me, just getting up there in front of a whole ton of people and talking about something I love. So I think it went well. Pycon is the whole conference. That was freaking amazing.

Yes. Well, I have to agree with that. It’s like, whatever I have to do to get myself the Python Amen. We talked about a lot of things. So if you could send me all those links and we’ll put them in the show notes, that would be awesome.

Absolutely.

And so, again, if people want to get a hold of you, you are automationpanda on Twitter, correct?

Yes.

And your website is just automationpanda.com, right?

You got it.

Okay. Is there a story about why Panda?

Oh, my wife is from China, so I’ve been to China a number of times. I’ve always loved Panda bears. So I was trying to think of a name for my blog that would be kind of fitting for me, something I like, something I could own and just went with Automation Panda.

Well, it’s catchy. And you get lots of cute artwork that you can do with it, I guess.

Marketing man.

So what are your goals for 2019?

Yes, good question. My goal for 2019 is to write my first software book.

Nice.

So I’ve been swirling around ideas for a while. I really want to write a book about the art of test automation.

So many of the books we have today are really good. Like, for example, you’ve written a good book on Pi test. And I’ve seen others on different other frameworks, but I’ve noticed that a lot of them for the past decade have focused very, very much on, like, here’s a tool, here’s how you use it, or here’s a framework, here’s how you use it. I want to address test automation as a discipline, as a domain within software development, and how we as software engineers and test, approach solutions and approach development and approach the different types of testing that we will need to do. Right.

That sort of topic is what I’m really trying to address.

Yeah. Again, there’s definitely a need there. So I wish you luck with that and hope that you succeed with that, because I definitely see the need. I thought of my book as teaching the mechanics, but I get a lot of questions of, okay, that’s how to write the test, but what test should I write?

And I’m like, well, that’s the million dollar question there of what to test, because there is no way to do 100% behavioral coverage of a system so you have to pick and choose what you do. Plus I haven’t found a decent way to measure behavioral coverage.

You can do code coverage but that’s not the same as behavior coverage. All the things that you want to do anyway. Well, this is like a blast talking with you about this. Thanks for coming on.

Thanks for having me.

Any extra call to actions you want to shout out or anything before we sign off?

I just want to say thank you so much. I’ve really enjoyed our talk. I hope we can have more and just happy testing and automating and developing everybody.

Well, thanks for coming on, man. We’ll talk to everybody later.

Thanks again to PyCharm for sponsoring the show. Remember you can get that four month free subscription by going to Test And Code PyCharm to get trypai charmprofessional and check out the show notes at testandcode.com.

Also at testing Code.com is a menu item that says donate. What that does is it takes you to our Patreon page and I’ve got a lot of Patreon supporters and I want to thank a handful of them right now so thank you Evan and Andrew Diedrich and George Rink and all of our best Walter and Steven Oates and Steve Holden. You’re donating at the super hacker level and I really appreciate it. Thank you.

That’s it for today. Get out there and go test something.