The story of how I discovered the current best GUI to run pytest.
Transcript for episode 48 of the Test & Code Podcast
This transcript starts as an auto generated transcript.
PRs welcome if you want to help fix any errors.
Welcome to Test and Code Code, a podcast about software development and software testing.
The story of how I came to find a good user interface for running and debugging automated tests. Is it interleaved with a multi year effort of mine to have a test workflow that works smoothly with product development and actually speeds things up? It’s also interleaved with the origins of the blog Python Testing.net and this podcast and the pytest book I wrote with Pragmatic. It’s not a long story, and it has a happy ending.
Actually, it’s not over, but I’m happy with where we are now.
I’m hoping that this tale of my dedication to or perhaps obsession with quality and developer efficiency helps you in your efforts to make your daily workflow better and to encourage you to try to increase the efficiency of those you work with. But first, thank you to those of you supporting me through Patreon and through the purchase of the Bytes book. If you want to help support the show, click the link in the show notes. At Testingcode Comfort 48, I use the funds to do things like pay for transcriptions of my hosting fee and my hosting fees, and take my wife out for dinner occasionally because she puts up with all this podcasting craziness and whatnot.
I also want to thank those of you who have purchased the book Python Testing with pytest.
Just this morning on Twitter, I received some feedback from Elizabeth McGraw and this is what she said. Excellent book, Brian Hawkin. The example program was perfect and the link was right on. It has just the right amount of info to be both a good tutorial and a reference guide to the most used things when writing and running by test. Thanks, Elizabeth.
So what are we talking about today? We’re talking about a Gui for pytest, and the story starts in 2010. In 2010, I switched jobs. Actually, I switched companies and jobs. In fact, both companies were electronic test and measurement companies, and both of the teams I worked with were using Python for automated testing of the instruments. However, they had different frameworks.
They were both running custom inhouse built tools that were attempting to be kind of like Ides with test runners tacked on and include abilities to store information about the equipment under test and help with the communication between the test code and the instruments and keep a log of all the communication transfer. All of this is really important and stuff, and mostly they were Gui’s. They were user interfaces to run all of this stuff because there’s a lot of people that don’t like command lines and that’s pretty much it. They were GUI so that you don’t have to use the command line.
Now I don’t mind the command line. I actually kind of like it. However, I don’t work alone. I understand that GUI are convenience and a lot of people just prefer them. And there are some things that are just nicer with GUI.
These tools weren’t terrible tools, but the problem with inhouse tools is there’s no way that an inhouse tool often developed by a small team or even an individual is going to keep up with the best in class tools developed by many people and fine tune in many, many companies and individuals.
Both the tools I had used were tied to Python 2.7 also, and I really wanted to try starting to work with Python three.
The big difference when I shifted companies was that I was in a smaller team and therefore I had a proportionally larger influence, which is awesome.
And I am an automated testing fanatic, so I kind of took over the direction of the testing part.
I added a new test runner to the in house system that could be more gracefully deal with the larger test suites that we had. I wanted to encourage smaller tests, so that means larger test suites.
I also added decorators to be able to skip tests on certain hardware configuration and generate reports in June XML so that we could view the reports in Jenkins.
The reason we didn’t switch to Unit Test Nose or pytest were multifold for me. I just didn’t know enough about them, any of them to strongly recommend anything. And also, these are all command line tools. Great for driving from Jenkins. Not so great to be able to run subsets of tests or even one test at a time when debugging failures or developing tests, especially for team members that were used to driving tests from agoie. This is a hard sell.
The Python Testing. Net blog was an offshoot of me trying to solve half of the problem. My own knowledge of the available off the shelf test frameworks, and that effort led to me falling in love with Pi.
Test efficiency is important. It’s even more important when combined with hardware and RF signals and settling times and client devices, and the list goes on and on.
pytest fixtures help with test times by allowing us to layer fixture code in different scopes.
Connecting to an instrument can happen once over many tests, and multiple tests requiring the same waveforms can run together.
The other huge help is the separation of failures.
Fixture failures are errors in Pi test so we can separate real failures from couldn’t get to the right state to test errors.
Brilliant price test wins, fixture fixture failures or errors test failures or failures they’ve separated awesome.
About that same time though, came along the podcast and then the pytest book because I wanted to share with everyone how best to get your head around Python so that I can help you with efficient and effect to be efficient and effective in your testing in the fastest time possible.
Many of people have told me that they’ve read it in a couple of evenings, which is cool.
So did our team switch to Pipest? No. Gui? I wanted to switch and I solved the other issues like communicating with the instruments, keeping track of instrument information, logging the communication, et cetera. But still no GUI, and I didn’t want to build an in house solution. I really didn’t want to support it, and I knew I couldn’t compete with off the shelf things.
During all of this research into testing frameworks, and while writing the book, I had teamed up with another podcast or Michael Kennedy to produce Python Bytes podcast.
Michael uses my Term and at the time I was a Vim user converted to Sublime with Vim Emulation.
I didn’t see any reason to switch to PyCharm until Michael intervened. In April of 2016, Paul Everett did a video about testing in PyCharm and used pytest. Michael saw this video, knew Paul, and sent an email introducing me to Paul.
Pychimp support for Pythest intrigued me, but I was skeptical.
Efficient workflow is super important to me, and switching Editors is a big deal. It’s a big time hit to switch Editors. However, Paul was kind enough to talk with me personally about the Van Emulation, the pipeline support, and the future plans of Pi PyCharm with respect to Pi test.
Then I got excited and I tried it out in earnest. The support for Pytist wasn’t seller at first, but it was better than any other GUI solution I had at the time. So I started the transition in my team towards using Pyte for the test framework and PyCharm for test development and test debugging.
The killer features of PyCharm for testing are one the ability to run tests at any level. All tests a subdirectory, a single file, a single test, or even a single parameterization of a test.
They also support running a class of tests, but we really don’t use that much. You can also store a template set of command line options so that the on the fly run configurations that get created have the same command line options.
Pycharm correctly parses fixtures from installed, plugins fixtures from comptest PY files and file local fixtures so that you can jump to fixture code from the test code.
You can rerun failed tests. You can debug the tests, meaning that you can run a debugger while you’re running tests. This is very cool. So while during the test run you can look at the different variables, the different results that get returned from different calls. It’s just awesome.
Also, did I mention the ability to run a single parameterization of a test? This is really important for us. We have huge parameterization lists for long running tests.
Debugging a failure that occurs in only one parameterization is sped up so much by being able to just rerun the one that you care about at the time. And I would be remiss if I didn’t mention the awesome responsiveness of the PyCharm development team. When I found issues with Python support, they jumped on it quickly. It was so cool. I even got a chance to meet them at the last Python and at least a few of them and they’re really cool people.
By the way this episode is not sponsored by Python. If you’re looking for the discount code see the previous or next episode. I just wanted to share with you kind of the story about how I got into using PyCharm. I’m not a PyCharm zealot. If a solution comes up for a Gui for running and debugging pytest that works better than PyCharm or even attempts to I’ll check it out. It doesn’t even have to be an idea but right now to get my team writing running and debugging Pi test tests pie PyCharm is the answer and not just the people writing the test. I’ve got embedded C people that use other Editors for C Code. Sorry JetBrains, C plus plus people are even harder to get the switch Editors than Python people.
Even they are using Python when they debug test failures and help with test creation. It’s awesome.
Well some of them use the command line but there’s old Unix hackers everywhere.
Like I said this isn’t an ad for PyCharm it’s just a story about how I finally got to use pytest at work because I can use pie PyCharm as a test GUI.
Next week we have an interview with my good friend Oliver best Walter about talks. It’s an awesome interview. I hope you tune in for it. Talks is simpler than you think it is and more powerful than you think it is.
Thanks again for Patreon supporters and the people reading my pytest book Python testing with pytest I think it’s the fastest way to get you’re head around efficient testing with Python.
Talk to you.
I’ll talk to you next week. Now go test something.