Deep dive with Brett Cannon into changes in to Python packaging such as pyproject.toml, PEP 517, 518, and the implications of these changes.
Transcript for episode 52 of the Test & Code Podcast
This transcript starts as an auto generated transcript.
PRs welcome if you want to help fix any errors.
Welcome to Test and Code, a podcast about software development and software testing.
Thank you to Digital Ocean for sponsoring this episode. On today’s episode, Bret Ken and I discussed the changes in football it in Python packaging as a result of Pep five one seven and five one eight. Starting with how did we get here and where are we going? We discussed Flit poetry talks, continuous integration, the demise of setup, Pi manifest in and requirements text. How does this affect Piping and will Pipe ever use the Pipe projectoject tumble? What are locked files and will lock files ever be standardized? We talk about workflows. We talk about applications versus libraries, dependency resolution, deployment, dependencies versus development dependencies, and all sorts of packaging. Fun like that. I think it’s a fun conversation and I hope you get a lot out of it.
Hello, hello.
Welcome to Testing Code and I’m excited to have you here. Thanks. Before we get into it too much, who is Brett Cannon?
Who is Brett Cannon? It’s a complicated question.
So in terms of a day job, I am the Dev lead for the Python extension for Vs code in terms of a spare time thing.
I am a core developer on Python and have been for over 15 years now. I got my commitment rights the same month as the first Python US back in 2003 when at least the first conference named Python.
If you really want to go deep. I have a PhD in Computer Science from the University of British Columbia. I got my Master’s degree from Cal Poly San Luis Pispo down the central coast of California, and I have a Bachelor’s in Philosophy from UC Berkeley. So I managed to get two of my alma maters to be covered by the two cofounders of Jupiter notebooks. So that’s kind of handy, I guess.
Phd is cool. The philosophy to computer science.
How did that happen?
So I went to junior College in the States, Pasadena City College specifically. And when I started, my mother made me promised her that I would take a class in philosophy and the class of computer science, knowing she knew that I would love the subjects. So I did, and she was completely right. I love both subjects and I actually tried to double major, but Unfortunately I was given some bad information about how the UC system has it. I wasn’t told that the UC system has a unit cap. So after I did a double major workload in junior College and I applied to get into the University of California school system, I found out I had so many units trying to make the double major work that if I applied to the CS Department and didn’t get in, I actually would end up not having enough unit space to graduate with a degree or I got kicked out of the school. Okay, so it was either major in computer science or major in philosophy. And this was in January of 2000 when I started. And so it’s like, okay. And so boom time. So it’s like, okay, either I can get a degree in philosophy just for fun and then still take some CS courses under the unit cap and then still get a job or go for CS, hope I get in. And if I don’t, I’m in deep trouble. So I took the safer route at the time.
Obviously, the boom did not last.
And I decided, yeah, you know what? I come from a family of teachers, and so I thought, all right, maybe I want to be a professor. Let’s go do the PhD thing.
So that’s how I ended up doing a master’s degree. I kind of treated as a junior College towards the PhD because not a single PhD program would accept me.
But Luckily, Cal Poly San Luis Obispo’s application date was late enough that I had already become a core developer.
And so I was able to put that on my application, and then they accepted me. And then so I went there, got my Masters, and with that, I was able to apply to PhD program at UBC and get in there.
Nice.
Yeah.
So you didn’t really switch from philosophy to computer science. You were doing the both all kind of all along?
Yeah. And it was actually helpful. I mean, philosophy teaches you how to think logically.
And in terms of open source, it’s surprisingly helpful to be able to understand where there’s potential fallacies and people’s arguments when you’re having to discuss things, how to have informed, insightful discussions without yelling.
There’s some good actual people skills that you can pick up in a philosophy degree that have turned out to be surprisingly useful in open source.
Yeah, I’m always surprised.
I don’t know why I’m surprised, but the amount of people skills required to program a computer is amazing.
It is definitely way above zero. That’s right.
Yeah.
I used to joke that I didn’t get into computer science because I like to be around people.
Yeah. Jokes on you, right?
Yeah, definitely.
When did you get into Microsoft?
So I’ve been at Microsoft now since July 2015. It’s been about three years. Okay.
Prior to that, I was at Google for about four and a half years, and then prior to that was grad school. Okay.
Timeline wise, you were already a core contributor for Python before your master’s program then. So you’ve been doing Python for a long time.
Yeah, actually a long time. So, yeah, I actually got my commit rights. I took a year off between my undergrad and grad school, figuring I needed to kind of develop a programming CV, as it were, to kind of show grad programs that, hey, I might not have a bachelor’s in computer science, but I do know how to program.
And so I just figured open source would be a good way to kind of prove that. And that’s how one of the ways I ended up getting bold with Python.
I got into the Python Cookbook and I had a recipe I wanted to upstream and so I emailed them, email Python asked how do I do this?
They explained it, I did it, I got in and I had the year off and I enjoyed the experience and I wanted to get more open source. So I just stayed on, got more and more involved. Started Python Dev summaries which let me know about issues that needed a fix before anyone else. And I had the time to do the fixes with basically no job. And so I just was able to get more and more involved and just get to know the code base more and more to the point where I said, hey, I can fix this. I just need someone to merge the patch. I think we’re just like, oh well, do you want commit rights?
Sure. I said, okay, someone will flip the bit and then they got flipped.
Cool. I got my commitment to rest is history.
Rest is history.
This episode is sponsored by Digital Ocean. Digital Ocean is the preferred cloud platform of hundreds of thousands of innovative companies. Digital Ocean makes it easy to deploy, manage and scale applications with an intuitive control panel and API designed for developers. Get started with a free $100 credit towards your first project on Digital Ocean and experience everything the platform has to offer such as cloud firewalls, real time monitoring and alerts, global data centers, object storage and the best support anywhere. Join the over 1500 businesses already creating amazing things on Digital Ocean. Claim your credit today at Dot CoTest And Code so you were one of the guests on our 100th episode of Python Bites. And thank you for doing that.
It was a lot of fun.
It was a lot of fun. And one of the things I brought up was poetry, and you filled us in a little bit on there and we don’t really need to cover all that again. But I also don’t want to assume that people have listened to that episode, even though they probably have.
But anyway, I guess when I stumbled into poetry is a tool for packaging, I guess it’s more than that. But there’s this bigger story, and this bigger story is there are changes afoot in what’s going on with packaging with Python. And I guess the learning about that actually opened up way more questions that I had that had answered to them. So I don’t even really know how to crack this open. So there’s both Poetry and Fled. And as far as I can tell, there may be other packages or projects, but I don’t know of them that use a piproject Tamil file instead of a setup pie file.
But it’s more than that.
The pipeproject Tamil is not just a replacement for the setup pie, it’s also, I think, what else does it get rid of?
So probably the best way to jump into this is to kind of explain the way the world used to be and basically how we want to change that to make it into the world that we want it to be.
And that’s usually how I break into this conversation. Okay, so the way the world everyone’s used to is set up tools and set up PY file, right?
Yeah.
And at least at this point, everyone’s familiar with wheels eggs seem to have finally gone away enough that I don’t really need to get into that.
But everyone thinks the way you build your package into a wheel is you basically go Python Setup PY BD under wheel, and that’s it. And it’s all set of tools. And that’s how it is.
The problem with that story is a couple of issues.
One is it’s the only store you set up tools. Right. That’s very restrictive for the ecosystem where anything anyone wants has to somehow work with Setup tools or it has to be in set of tools or else tough. It just won’t work. People just don’t expect or work with any other tool. And so if subtools can’t do what you want, you’re just in a real tight bind.
The other problem is setup. Pi is not really a structured thing. It’s just a convention of naming a file the Python will execute that will take commands that we once again on convention will accept that will somehow lead to a wheel being produced, which is also restrictive in terms of okay, well, how do I introspect that? Well, you can’t. The only way to really introspect that is either parse the AST of that file or Shiman set up tools, import that module, fake out what Setup tool setup does, and then introspect on what gets passed in as the argument to that huge function.
So basically the packaging world ended up in this really restricted position by accident of everything has to be set up tools, and thus everything has to be set up top pie.
And it doesn’t really allow for improvements. Right. Like Dustin Ingram just posted his slides to a talk he gave that was really good. We basically said like, yeah, we have it really good. We just forget it because it’s been such a long slog to get here from the early 2000s to today.
But there are things that we have to do in order to allow for evolution, to allow things to improve. So we’re definitely not at our best necessarily, but we’re not horribly off. But how do we get from pretty good to great? And one of the things you have to do is you have to allow for innovation and break the Setup tool shackles as you were.
And so that’s what led to Pep five one eight and five one seven. So Pep 51 eight. What defines Pipe TOML, and what that did was all it did was allow you to specify your build dependencies for your project, and that’s all it did.
And it happened to bring in this concept of Pyproject Tamil as a file format and it left it open in terms of how other tools could add their own metadata to the file. But really its only goal was to be able to say, you know what, you can now specify what is required to build your wheel.
Okay.
So because before this you just assume Setup Tools, right? It will literally just inject Setup Tools when it runs Setup Pi because it just knows almost everyone’s going to do that.
But there’s no really other option. How do you take an Estis with Pip and say hey, make me a wheel with Pip wheel?
Well before Pipe Project there wasn’t really a waste. Actually, funny enough, Setup Tools has an argument to say build requires.
But how do you tell what bill requirements you have unless you already have installed set up tools to run it to then tell you what the bill requirements are. Yeah, right. It’s a problem. So there needs to be a way to statically specify up front that hey, my project happens to use Flit or Poetry or what have you ensconce is another one.
How do I say that? And so with Pipe Project Tomo you can specify in that file that hey, my build requirements is I depend on Flip to build that’s basically all Pip five, one eight did okay. And then we built on top of that. And when I say we this is the Royal we of the members of the Dis Util, Sigma list and Pet 517 came about where we took that the next step and said okay, here is the API that a build tool must implement and you get to specify the entry point.
And what can happen is your build tool can then or like Tip for instance can go like okay with Tip five, one eight, here’s what I have to install to build. And then with Pep .517 it looks in the same Pipe Project tumble and it goes okay, now I know what code I have to call in this build tool to actually make your wheel.
And that breaks the dependency on set of tools. Now that’s not saying you can’t keep set up tools, right? You could still say hey, my build dependency is set up tools. And then all you need to do in your Piproject Tomoe would be to say okay, call Setup Tools, Pipe Project Setup or what have you. And it would work. But it also means I can use Flip if I want or poetry or anything else. And it still will lead to that ultimate wheel, which is the true end artifact that you really want to be installing anyway.
So I guess to just clarify a little bit so it’s not flat that’s completely creating the wheel. It’s an interaction between Flit and Pip for whatever your tool is.
And Pip Pip is driving the building. But Flip is doing the actual production.
Okay.
If you’re running Pip ten or newer, what happens is if you do Pip wheel and your project, what happens is Pip will readuppriceproject Tamil, go. Okay. You depend on Flip. It will create a virtual environment, install Flit into it, and then read the section of the project Tamil that specifies the interface for the build tool. And go like, okay, I want you to build me a wheel. Okay. That requires calling Flit dot main with its main function or something.
Okay. But you specify that in the Tomo file.
Exactly. And then Pip will literally just call that function in your virtual environment and then that will emit a wheel. And then Pip is then able to just install that wheel.
Okay.
And all those API and the extensions that allow for Flit to exist, it could be anything then. So I could create my own like, Branch of Flit or Branch of Poetry or Insponse or even just make up my own.
Okay, that’s cool.
Yeah. As long as you provide an API that is compatible with what Pip 517 outlines tools like Pip that need to go like, all right, I have an Estis. Right. I download a tar. Gz file from Pip. How do I make that into a wheel that I can install?
And you said something during our conversation in Python Bites. That a wheel. Okay. So that’s how to build a wheel. But when somebody wants to do, like, Pip, install the wheel once it’s built, that doesn’t need like a packager to do that other than Pip.
Correct. So this is all about building, right? This is all about how do you get to the final build artifact that’s a wheel. Installing a wheel is a completely separate project.
This is a key point that trips a lot of people up and it takes a while to I know it took me a while to get separated in my head because really what you’re hearing about is there’s the packager. Like you as the person who’s going to distribute this package on Pipe, you’re the one that cares about making a wheel to make available to users.
But your users shouldn’t have to care about any of that. All they care about is the actual wheel they’re going to install.
Okay.
So Piproject Tommy does not extend beyond actually build step of your project.
Okay. I’m a little confused, though.
Not a problem.
One of the things that you specify is your package dependencies.
And so somewhere in the wheel those are listed somewhere that they can be pulled out later by Pip.
Yes. So the way it works is your tool lets you specify the tools like Flip, for instance, have you specify your dependencies in their own way, right. Like Flip, for instance, has its own Flip section in the Pi project.
And you can specify your requires there. And that’s the normal thing, just like in your requirements text file or something where you would say like, okay, I depend on Brian’s Cool Pi test plugin version something.
When the wheel gets built, wheels have metadata in them. And one of the pieces of metadata, I can’t remember which file it’s in right off the top of my head.
But there’s a file where those dependencies get specified, as I believe requires key. And in that metadata it says, okay, here are my dependencies. It’s on Ryan’s awesome PY test plug in version blank. And it supports all the normal markers and everything you could put in there in your dependencies, such as Python version dependencies and all that stuff and all that gets packaged in and put into the wheel and its metadata. So that’s why it’s a separation.
So the build tool has you specify how the build tool wants, but the wheel because the wheel format is completely standardized, all the build tools will know how to write that appropriate metadata into the wheel. Felt so that when Pip or whatever install you’re using it’s your wheel, it can open up and go like, okay, I’ve got the wheel now, and I know what files it has and where those files need to go. But what other things do I have to grab? And they can read the metadata and go, okay, this thing wants this and this, and then that’s when it go back up to Pipe and start doing dependency resolution and all that stuff. But that’s all a separate specification just for wheels.
Okay, so wheels don’t have all the dependencies built into them. They just have references to what their dependencies are. And those are grabbed later during install, right?
Typically. I mean, you can technically build a massive wheel just for yourself to deploy all your dependencies at once. I’ve heard people do that as that’s their deployment story is they actually build a one massive wheel that represents their code plus all the dependencies, and they push that up to their server and then the server just installed that one massive wheel. Okay, correct. Wheels are designed such that they can just contain the code for that package that it represents and then just lists what it depends on. And then Pip or whatever other tools you’re using to do the install pipe, they can do dependency resolution as necessary to figure out, okay, well, you specified you depend on that. So we’ll go figure out what the best version for that is. Grab that wheel, then that wheel is going to have its own dependencies. You just keep doing it to figure out all the dependencies you need, and then differences will just install all of them.
Okay, well, this is cool. It has ripple effects that kind of go all over the place that I was noticing as I was trying to play with all this after we recorded there’s some hiccups that people are still trying to figure out. Like talks, for instance, is just recently supported. I think just recently supported being able to use non setup tools packages and because it’s got to go off. And one of the things that Talks does is package your wheel for you to test that part.
So it has to know how to do that.
I’m assuming there’s other ripples when I look in other places.
We’ve had set up Pie for a long time, so people just kind of assume it’s there and it might not be.
I don’t know how this affects like continuous integration servers or Jenkins or not Jenkins, but like Travis, do you know if they’re dealing with those already?
So it really depends on the tools, right. So for instance, on Travis it’s not too much of an issue because on Travis you’re usually just running Pip install and as long as your Pip install is new enough and you can always just do a quick Python M pipinstall U Pip and just have Pip update itself that will take care of this potential issue.
Now Talks, I believe they may have just landed support for this. I know they’ve been working on it and looking at it and they’re aware of it.
But yes, this is much like anything else at this scale because the Python can be just so big at this point and set up the Pie has been used for so long, it’s going to take some time. Now the one thing that can be said is it is possible to actually generate set up high shims like Flip for instance, tries to do this and it will actually spit out a setup Pi that you can actually just call for tooling that has been updated. Because if you have a shell script, for instance, that’s just going to call set up by blindly and just haven’t been bothered to update it to call Pip install or Pip Wheel or what have you.
Their tools can theoretically spit out a set of Pi that will provide backwards compatibility as necessary to make that work. But you’re right, there are knock on effects and it’s going to take time to flow through people’s tool chains and get all the tools updated that are doing build steps to start using this and developing libraries to help with this and all that.
So it’s very much a multi, multi year process to get this to a point where it becomes the thing everyone uses. But it’s been interesting tools have picked up on this pretty quickly.
For instance, Black, the code formatter already has you specify rules in a Pi project Automo file town. Crier, which is a chain log file generator, has you specified settings in your Pipe project tumble. So it’s gotten pretty quick leaf usage of the people’s build tool chains and that kind of level. It’s just taking time to build up to the tools like talks and stuff where they’re actually doing the actual build calls and getting them to start doing the appropriate thing when they’re faced with the Pipe project Tomo.
It’s good to have all this stuff allowed in the Pipe Project Tomo, because we were kind of getting a proliferation of configuration files.
Yeah.
And being able to use one with people just having a different section. That’s awesome. And I’m all for it. One of the things that we used to have with setup tools is like extra files like the manifest file that could specify, like to package your README with it in your documentation or something. Is that part of this also does the Pipe Project Tom will allow that.
So it allows it to match up to the bills to choose how you’re going to specify that.
So remember, the file only specifies lets you specify what your build tool dependencies are and then how to call your build tool. But how your build tool asks you to specify like how to include the license file, for instance, or the README that’s between you and the build tool.
So as you said, set up tools happens to have their manifest info, and maybe set of tools will keep that when they go over for prior project tumble completely or set up CFG or who knows? But Flip, for instance, has no concept of this with the way Flit does it, at least with their ests is Flip will package everything that is actually in version control.
So Flit asks yet to tell it, okay, what’s been committed, because that’s probably what you want to end up in your St, which is where you really want these files to end up.
And then it’s able to then have you say, okay, well, here’s what I want, my wheel. Usually you just specify the package and it just knows, all right, well, the package is this directory. So pulling everything in that directory and that’s how it makes the decision. So Flip says, yeah, you know what? Forget manifest in and having to remember to update that. So you don’t forget to include your license file because someone is bound to ask you, hey, I need the license file in here. It just says, all right, anything that’s committed that goes into the Est.
Okay, so can you use Flip with non get projects then?
I honestly don’t know what its support is for something like material or subversion.
Okay, not that it matters too much, but I did notice that when I tried to play with it just with some files that weren’t checked in into anything, it said, oh, yeah, I can’t do that because there’s nothing committed. It surprised me that there was a version control dependency.
Yeah. So that’s the exact reason it basically allowed Thomas Clover, the creator and contributor to Jupiter, to basically say, yeah, you know what? I don’t like manifest in and the way I run all my projects, everything that’s committed is really what should be in the estate anyway.
So you know what, I’m just going to make that the rule is if you don’t want it to be in, add it to your get, ignore, and otherwise it’s going into the estist and that role itself, I mean, on all the projects that you split on, it’s actually worked out really well.
It’s actually kind of helpful to me as well. It cleans out some Cruft aside, thing is, how does this interact or does it interact with Pip files and Pip file lock files and stuff like that?
So it doesn’t.
So when we first created Pep five, one, eight, and the sole concept of Pi Project Tamil, it was actually originally scoped to just libraries.
So just stuff you would put up on Pi basically is the way to differentiate versus an app where you would deploy it to your server, for instance. Okay, right. So basically, originally Piproject TM was designed specifically for the use case of Shareable Redistributable artifacts, specifically Wheels.
And then Pip End was created with its Pip file concept for apps. Right. Things you’re not going to put up on Pip and Redistribute, it’s just going to be something you deploy somewhere.
And so originally we’re designed for two separate use cases. Now we have subsequently based on how all the tools basically started to ignore the fact that Pyproject Tamil was specifically designed for libraries and started to use it for apps as well. Specifically, Poetry completely ignored that. And then other tools that you would potentially want to use with apps, and not just libraries such as Town, Crier and Black, because you might run Black on your libraries, but you might also want to run it on your app. Code made us loosen that and just make it just say project. And it’s no longer specified to that. But that’s the key reason Pip file for Pip End exists and is separate from Pyproject Tumble because it was originally designed for a separate use case that Piprotoject.com was kind of pulled into. Also supporting.
Okay, well, I’ve never really quite understood the separation, to be honest.
I get it for this is like a Django application versus a library that’s only used for import. But then we’ve already specified some things like Poetry, Flit, Black. These are things that you do Pip install, but you use them as a tool.
Right. But they’re still something that would go up on Pi.
Okay.
Originally the breakup was Library is something that goes up on pipes or stuff that you wouldn’t put up there.
Okay. That’s the general rule of thumb I use in my head to differentiate between the two because it is obviously very funny. There’s no real clear cut definition, but it’s basically, am I going to be sharing this with ten different people on the planet and that’s up on Pipe, or is this just for me or just for this specific case? And I’m never going to put it up on Pip? That’s the difference in my head between library and app.
Okay.
Now, having said all that, there has been talk about what the future is. I’ve had informal discussions on Twitter between the maintainer of Poetry and the maintainers of Pipem, because in my brain, the artifact that a build tool will emit for a library is a wheel file. Because that is standardized. It specifies as dependencies. It’s got all your code. It’s fast to install. Like I wish every project out there built a wheel file that was going to be a library up on Pipi, to the point that I sometimes wish we had a requirement that you have to upload wheel files.
I didn’t know that it wasn’t.
No, it’s not. You can totally just upload us to us.
Okay.
Now, having said that, for an app, that’s typically that’s not what you really want. What you really want, at least in my experience, is you really want that lock file like in Pipim’s case, right? It’s not the Pip file you want. That Pipfile lock that specifies okay, when I take my code and I deploy it, here all my exact dependencies I need you to install on the server. Or here are all the dependencies I need to have with Pie installer to pull in to go with my code, to then distribute out to the world as a desktop app, for instance.
Okay, if people new to the conversation, a lock file is some sort of you don’t actually want to read it. It’s some sort of format that a tool can read that specifies the exact version of the dependencies that I need to install. Is that right?
Exactly. So if you’re using Pip, for instance, that’s what Pipfrees generates. It generates a requirements text file that lists every single thing you have installed so that when you go try to install all that stuff that you’ve built up into your virtual environment, there is no questioning of what version to grab.
Tip freeze will have written their requirements text file that specifies down to the bug fixed version exactly what you want pulled in and installed.
And then in pipeline’s case, it’s a pitfall dot lock. In Poetry’s case, it’s a pipe project lock. If you’re from NPM, it’s the same as a package lock. Json file. Pretty much every community has kind of come to the conclusion that lock files are good for this kind of use case. The problem we have is we haven’t standardized that lock file yet.
Yeah, and I’d like to get us there.
Okay.
I did notice that Poetry does a pipe project lock, and that sounds all official because it’s just like Piproject Tom will. It’s just a lock. But that’s not a standard specification. That’s the Poetry’s definition.
Yeah, the maintainer for Trend. I’m sorry, his name escapes me right now.
So he knew he needed a lock file for apps because Poetry supports both libraries and applications. And since there was no standard, he decided to basically make another file another Tamil file that specified all the dependencies for your app. That way, Pip file because it started with a separate file created their Pip file lock file. But there is no specification anywhere in the Python community as to a specification or through a PAP specifying. This is how we will specify our lock files. And I’ve had this discussion on Twitter.
People seem amenable to the idea if we can meet people’s needs. It’s kind of tricky because Python is so supportive cross platform to go like, okay, generate me a lock file. Okay, well, this might be your lock file here on Windows, but that won’t necessarily be your lock file like on your Linux VM.
Oh, really?
Okay.
Because they’re going to resolve differently, right? Because some things are going to say, hey, this is an optional dependency on Windows to get you something faster. This is like a weird accelerated NTFS support thing while over here, this will get you fast or something on Linux. And so it can totally vary, right. As soon as you have like or pieversing, right? Like you might be running three seven locally, but your cloud might be three six or something weird.
So you can have multiple lock files.
Who knows? That’s the real question.
Do you do them separate based on Python version or OS? Do you do it in one and you just have to potentially specify the target you’re building a lock file for? Do you try to build the penultimate lock file that covers all potential cases so that it doesn’t matter what platform you’re on, you’re going to pull them all in. Okay, it’s an unknown question.
It’s a question with no answer yet because no one has started this work yet. It’s just kind of sitting there waiting for someone to have the time and motivation to start really having that conversation, to start working towards that. But in my head, Wheels give us the artifact we want for libraries.
But for me, the artifact for apps is this lock file.
Okay, we haven’t taken it to standardized that yet because we just got libraries in a good place in terms of getting them out of the restriction of only considering a set of tools.
Okay, I got a few more questions, but I just looked up so I don’t have to place it in later. It looks like it’s Sebastian Eustace, is that right? Okay.
With kind of an entertaining black and white picture. I love black and white pictures on his GitHub profile. Anyway, the lock file who uses the lock file. So does poetry use it or when you’re using poetry? So if it’s tool dependent, how do you use it?
Yeah. So the way it typically works is you start specifying your dependencies. Right. So in your prior project, Tommy, you would specify or Pip file for pipeline, you specify where dependencies are you download all your depend, it’ll download all your dependencies, and then that will give you a state of the world that you develop against.
Okay.
And so what then happens is you want to freeze those dependencies as is, right? You don’t want to accidentally.
Let’s say you and I are working on a project together. Brian, you first have the dependency specified in your Pip file, you then run pipelines and it’s going to generate this pitfall dot lock based on version two of Pi test and etcetera. Etcetera for all the dependencies. Now we want to make sure we have some reproducibility in our environment, right? Yeah, because we all know testing against shifting dependencies is not fun. You really want to get those dependencies pinned and consistent so that your test results are going to be the same across the board.
That means you need to lock your dependencies down into your pipel lock so that when I then download the latest version of our code and I run get pull. If I run Pip install, it’s not going to look at the Pip file, it’s going to look at the Pipfile lock and go okay, will you pin your dependencies to pytest, version three, seven one, and et cetera. I’m going to download those. I’m not going to look at your Pip file and try to recalculate them all because you’ve basically agreed to as a team to this Pip file lock. Now you can always update it later by updating things in your pipeline or tell Pipe, hey, can you update this dependency because there’s been a bug fix or what have you? But the key thing is the pipeline lock is what you as a developer agree to with the rest of your team to make sure everyone’s on the exact same page in terms of your dependencies.
Okay. And Pepin and Poetry both support and I don’t know about Flat or not. Actually not sure Flit does this anyway.
A separation of the dependencies that are needed for the application to run or module to run and the dependencies needed for development. Like if I wanted to add Pi test or something to those dependencies so that’s separation and Flip does disport that. Okay, is there a notion of that with Hipfile lock? Okay.
I believe you say Pipam installdev and then Pipe will then install based on what gets flagged as a Dev dependency.
And same with Flip and same with any of these other tools. Basically all the tools seem to have at least come down to an agreement that you have your deployment dependencies.
In the Django scenario, let’s say you’re deploying out to Azure to support my employer.
You’re going to want to say here are my dependencies for when this application runs, period. And you get those. But then, as you said, your debt depends. Like I don’t need to deploy Pi test out to the cloud, I just need it here locally on my machine to run my tests. Okay, so you want to make sure that when you deploy to the cloud, you’re doing pipe install and just the key things you have to have to run but on your local machine, Dev will then look at the lock file, look for anything that has the optional Dev true or Dev false setting, and then that way it’ll pull down exactly what you need to get your work done.
Well, I really like that the tools are starting.
I know that we’re not there yet.
Like you said, we were in a bad place. We’re in a better place. We can envision that there might be even better places, and we need to put things in place to make that happen. And that’s cool. I also like that that’s including the notion of workflows that people when you’re developing it’s different than when you’re testing or different when you’re deploying and possibly thinking about, well, what if you’ve got dependencies that are generic in one case, but they might be different for different operating system deployments or something.
And keeping in mind those workflows, that’s actually what I’m trying to get my head around now, and we can’t do this in this conversation. But having the entire community kind of think about those workflows of how do you actually use these tools to get stuff done in your day?
And having it better supported, I think is great because the requirements that text always seemed like a little bit of a side hack to me.
I know we didn’t even bring that up.
But these things like the Pipe Project Tumor and Pip End are trying to kind of replace in Poetry replace things like requirements text. Right?
Also, yeah, exactly.
Okay.
No, they are just because requirements text files, as you pointed out, they just don’t quite fit into modern day workflows.
They were a solution to a problem people had and people still do have. But it just turned out we’ve realized what the real underlying problem is, which is having a way to specify your dependencies. But B, then have a way to specify your dependencies. Exactly. For reproducibility and requirements. That text file didn’t have that concept built in. It became convention. Right. This is why we have fit freeze, because Pipfree is all this is just spit out yet another requirement text file, but there’s no notion of that is definitely a lock file. That is definitely the one you want to install. It’s just potentially convention that you might call it requirements dot freeze or requirements frozen dot text or who knows what. Well, these other tools like Poetry and Pitfam have just gone down the road of defining their own convention of saying that this is how you very much separate the concept of just listing what your dependencies are and specifying what they happen to be exactly based on what you want to deploy against.
So do you think that we will have in conversations with these other tool people that are building these things? Do you think there will be a standardized lock file format?
Yes, purely based on the fact that I want it and I will find the time some day in my life to make it happen.
Yeah. I think everyone’s kind of we’re slowly realizing. I think we want people to have flexibility in their development workflows, but we want specifications in the resulting final artifact we want out of those workflows. Right. Like getting us Wheels as the final artifact for libraries has been really freeing because it’s allowed us to specify everything we need in a standardized way. Well, Pipe Project Automo has given us the freedom to have people have whatever workflow they want. So really the key thing is all we all have to agree on is what’s the final thing tools like Pip and such are going to consume. And having that standardized on Wheels has been great for libraries. The next step is going to be, I think, applications and having Poetry and Pitfam both coexist is actually, I think, going to be helpful because it’s allowed the community to try different approaches, see which ones work and which don’t see how people want things to go, and then see basically how the dust settles, and then it’s just going to be a matter of someone putting in the time and effort. But I personally do think that a lock file format is going to be what we all care about in the end, and thus what we all want to work towards standardizing on somehow. Okay.
And here’s a possibly contentious question.
Do you have any chance of Pip End moving towards just using the Pipe project tomorrow and lock files?
That I don’t know.
They were the Dan Ryan and I can’t unfortunately remember the other piffin Co maintainer right now when I was talking about this on Twitter. They were both very amenable to the idea of coming up to the standard, especially for the lock file. So whether or not they’re willing to move get rid of Pip file and move to Pirate Project Tamil, I’m not sure they might be if that’s the way the wind blows in the community, but they were definitely open to the idea of we’re not married to Pip file lock. If a spec came out in a pet that said this is how we’re going to do lock files, they were very willing to move towards that.
Because one of the things I’m realizing is not only can different projects choose whatever they want to use as their packager to make their wheel even within one project, you can have different maintainers of the project like to work differently. I’m not really a Pip End person, but my one semi active project so far on GitHub, there’s been people involved that said, I’d like to help out with this, but you don’t have a Pip file published, could you do that? And I don’t have any problem with that. But that means if I want to use Flood and the people using Piping probably don’t care whether I’m using Flood or Poetry, as long as they can use Pip end also, but they all kind of have to stay in sync.
Yeah, this is one of the reasons pipam, for instance, allows you to export to a requirements text file So that they’re using that as their kind of universal standard of requirements format. Okay, you’re right. Usually at the tool level, typically you kind of have to standardize. You have to go like, okay, as a library, do I want to use slit or do I want to set up tools as an app? Do I want to use poetry or do I want to use tip M? And you just kind of have to at this point, choose one.
Or honestly, you can just go, I’m just going to require a text files and we’re just always going to export to that.
But yeah, typically you have to, at least within a project, standardized, but at least as a wider community, you’re given the freedom or per project. Choose. But I would imagine trying to keep things in sync in a single project between multiple tools like that, that might be probably a little hard.
Okay.
I’m sure you can write a tool to do it if you really want to make sure everyone ran that tool to update the Pipfile and the pyproject.com appropriately. But I don’t know if that headache is worth it for you.
I know we’ve answered a lot of questions, and by we, I mean you. So thanks a lot for being subjected to all these questions.
I’m quite happy to help out.
And I’m sure that there will be more questions that this raises as well, but I’m excited to see where we go with this.
Yeah, I’m really excited, too.
I want to see what the community does with this new found freedom and see what tools people develop and what workflows people come up with and see how it hopefully helps them work better.
Well, this has been fun, so thanks a lot for coming on and we’ll talk to you later.
Thanks for having me.
Thanks again to digital ocean for sponsoring this episode. Claim your $100 credit at dot coTest And Code. Not only does that let you try their amazing service, but it also tells them that sponsoring this show is a good thing. That link is also on the show notes, page at testandcode.com 52. Thank you to Brett for taking his time in researching and recording this episode. And thank you thank you for listening, for sharing the show with friends and colleagues, for supporting the show through Patreon, and for using the link in the show notes to try out digital ocean. That’s all for now. Maybe go try flitter poetry on a new project.
Or if you don’t know what to do next, write somewhere. Test.