I always learn a lot when I talk to Brett, and this episode is no exception. We talk about the packaging workflow, tools, changes, pyproject.toml, flit, setuptools, and so much more. I hope you learn as much as I did in this great discussion.
Transcript for episode 152 of the Test & Code Podcast
This transcript starts as an auto generated transcript.
PRs welcome if you want to help fix any errors.
00:00:00 This is the future of packaging. There is this benefit. You want to do this. Trust me, it’s not a waste of time. It’s not going anywhere.
00:00:08 On today’s episode of Test and Code, we talk about packaging with Brett Canon. I always learn a lot when I talk to Brett and this episode is no exception. We talk about packaging, workflows, tools, changes, Pyroject, Tumml, Flit, setup, tools, and so much more. I hope you learned as much as I did in this great discussion.
00:00:41 Welcome to Test and Code.
00:00:51 Welcome to Test and Code. Am pleased to have Brett Cannon on again. Brett is my go to resource for all things, actually, everything.
00:01:01 I’m sorry, but mostly like stuff that’s going on within the Python world, but also packaging.
00:01:10 I think you care about packaging. Why do you care about packaging?
00:01:14 Good question. Because it does take up a lot of my time.
00:01:19 Basically, I think once the whole Python two three situation was taken care of to the best of my abilities, I was kind of done with major projects at that point and I was kind of just looking around like, okay, what’s the next big thing that the community kind of seems to want or need help with? And packaging was moving, but I felt there was something there that I could contribute, partly just because I am very familiar with writing PEPs, for instance, and dealing with all of that, and both just personalities that inevitably come in these kinds of discussions that can be somewhat difficult to deal with, to just helping drive consensus, to just be willing to write the words and all that, and kind of just drive discussions to closure. And so I just started to get involved with it and I perpetually been involved with it since obviously it’s not done and that’s basically it. It was just I saw an opportunity to help out and so I just stepped in and been trying to help out ever since.
00:02:21 Okay, but you said it’s not done.
00:02:26 There’s rough edges still. But is there like an obvious next thing that needs fixed?
00:02:32 Well, if you want to start off with long term plans, we can start with that.
00:02:39 Yeah.
00:02:42 When I started to get involved with this, one of the things that really kind of kicked it off for me was I looked around and went, okay, if I were to write an installer that only used standards specified either at Packaging Python.org, which is actually the official place for packaging specs, I was going to say, or path. Honestly, even those paths get linked from Packaging Python.org as recognized as the official standard.
00:03:10 I looked at it and I went, Could I actually re implement a fully standards only installer somewhat like Pip from the bottom up?
00:03:20 And I realized you couldn’t. Okay, there were certain holes. And by the way, I should also say not only just from a standard perspective, but also from a library perspective, because one of the things I really always appreciate with the Python community is we seem to, as a, as a group, come together around projects and libraries and kind of put our effort behind singular projects and libraries that do a really good core job at something really well without having 50,000 options. We’d prefer to have more like three, which I like because it’s the NPM approach versus our approach as a community. Right.
00:03:56 Yeah, I guess I don’t know anything about NPM.
00:04:00 That’s fine. I don’t need to go down that route.
00:04:04 Okay.
00:04:05 I deal with that for a living anyway.
00:04:09 So for me, what that does is it makes me want to have libraries that back up every single spec in the packaging ecosystem, and there just wasn’t for various reasons. But what happens is at least to this situation where every single tool in the packaging tool chain implements the same Pep differently.
00:04:29 For instance, the best example I have and from personal experience was Wheel Tags, right. I noticed that every single project they had to deal with Wheel Tags had their own vendor cargo quilted copy of a module that I think came from either Pip or build out Pep. 425 Tags was the name of the file. Everyone had their own copy, everyone had hacked it to their own needs over time and they’d all diverged.
00:04:57 And so what happened was I came in and I went, no, you know what? This should all be one module, right? And what happened was there’s the packaging project, which is a little confusing to discuss with people when you’re talking about packaging. So the packaging project for packaging, you can go to GitHub. Compataging and that’s what I’m talking about.
00:05:18 I decided to put in the effort to create a tags module for that project and was able to work with Pip, set up tools, and I don’t know, build out what directly participate, but I did at least look at their source code.
00:05:43 Okay.
00:05:43 And we all work together. I worked with all these projects in the Pep to try to create a singular implementation of Wheelchair, and I did and actually turned out to be a really good thing to do because I’ve discovered, for instance, Pipe was getting screwed. Like there were a massive amount of Tags that under PyPy were not being listed as compatible, even though they were. And there were a couple of other places where there were actually wheel Tags that existed and combinations that were possible that were not being listed as a potential for other implementations of Python. It doesn’t see Python.
00:06:21 This episode of Test and Code brought to you by Data Dog. Do you have an application that is performing slower than you like? Do you know why requests have high latency with Data Dog? You can find the root cause fast troubleshoot your app’s performance with Data Dogs, distributed tracing and continuous code profiling to quickly detect what happened and why down to the line of code, and in one click you can correlate individual requests with related logs and infrastructure metrics. To get full stack observability with zero contact switching, start tracking the performance of your apps with a free trial at Test And Code. Comdatog and Datadog will send you a free Tshirt. And to be honest, my data dog Tshirt is one of my favorites. I wear it all the time.
00:07:14 Just wheel Tag What is an example of a wheel tag? Is that like the description or something?
00:07:21 So if you ever look at a file name for a wheel, and if you don’t know what a wheel is for the audience, it’s basically the binary artifact that you install typically. So basically, in packaging there’s Estis or source distributions. Those are the Targz files. Or on a rare occasion, the dot zip files you find on Pipe.
00:07:42 That’s what you build and you create a wheel for. And then the wheel is going to contain all the code. And by the way, even for people who maintain Python only projects, please build wheels. They’re literally just zip files. And for Pip and any other installer tool to install them, it’s literally an unzip and copy the files to the right place. Everything else requires that compilation step and slows down installs, and you also can’t rely on the metadata in those files, so they still have to build the wheel anyway to find out your dependencies. So a quick PSA. If you maintain a project on Pipe, please make sure there are wheels tangent done.
00:08:21 If you ever look at a wheel file, you’ll notice that it’s got a very specific structure. It starts with the project name, normalized dash version number, and then potentially an optional build number.
00:08:36 And then after that it has what are called the wheel Tags, and it breaks down to three parts. So one is the interpreter version that you’re using. Right? So that can be as specific as like, CP three. Nine means C Python 3.9 or generic as Pi three to any Python three compatible interpreter.
00:08:56 The next part is what’s called the Abi tag. So for C Python, that’s usually like CP. 39 to specify that is the CPython 3.9 C Abi. And if you don’t know Abi versus API, a by is the application binary interface. So when you link things at the C level, things have to be compatible at a binary perspective, not from an API perspective. Right. So if you know C, a good example is macros macros are API compatible but not Abi compatible, because I have to be able to link against it knowing the function call. The way you call is completely the same, but I don’t need to worry about the name so much. Well, at the API level, you can completely change how stuff works and hide things behind Macars and new funky stuff. But as long as the way you call, the thing still is compatible, that’s API compatible.
00:09:50 Okay, yeah, it’s funky complicated but there’s a very clear distinction and key thing that basically if you compile anything you want Abi compatibility, API compatibility. Okay, and then that CP 39. You can also specify Abi three, which means you’re compatible with Python’s stable Abi, which they have, or you can actually end up seeing none, which means there is no API needs. And that’s what pure Python code specifies. And then the last part is your platform, and then that one is where you specify both your OS and your CPU level. So for instance, if you’re on Linux, that’ll be the mini Linux X 64 one to say. Or actually it’s mini Linux G libc version X 86 or X 64 for the CPU architecture. If you’re on Mac, you’ll look and you’ll notice some wheels, for instance, have huge lists of Tags on the platform section because it comes all the way from Mac 10.9 up to modern Mac versions. And then it specifies whether it’s a universal universal two binary which is brand new thanks to Apple Silicon, or whether it’s X 86, X 64 or Arm Power PC is not really a problem anymore.
00:11:10 And then WHL, which actually is a zip file. So that’s what a wheel tag is. And standardizing the code for all projects to use in one place was one of the first things I did in the packaging world. And it’s been really helpful because actually how we were able to get Apple Silicon support out for the entire ecosystem in one go, because Apple came to us in July on the packaging project and submitted a PR to actually add Apple Silicon support.
00:11:37 There were some discussions higher up about Abi’s and how to handle that part of the tag, like should it be universal two or should we list, oh hey, if it’s a fat binary, should we break out X 64 and Arm in the end with universal two? We got that in in November there were some bugs that came up. We got those fixed that got pushed out in one of the Pip 20.9. I believe I got the last of the Apples Silicon bugs fixed and pushed out, which is also the very last the reason that’s the last version with Python 2.7 compatibility.
00:12:09 We wanted to make sure that the last version of Pip that had two seven also works on Apple Silicon.
00:12:15 Okay.
00:12:17 But because we centralized all this and now have a single library, we’re able to do this once and all libraries get the fixes, and all libraries now are compatible with Apple Silicon. So it worked out. Honestly, it’s a lot. It took me months to write this module of just research alone. I had spreadsheets. It was crazy.
00:12:37 I want to walk through a couple of examples.
00:12:40 One example is, let’s say I just have a Python only thing that a new project that I want to share with the world. I want to push it to Pipe and I just did this last week or recently for Python only stuff. At least I’m leaning towards Flit, usually for my packaging.
00:12:59 Same here.
00:13:02 All these Tags that you mentioned. I didn’t have to care about them.
00:13:06 You still don’t.
00:13:08 Okay, so I just built and I did the deploy thing, although I really should have done the deploy to the test server first. But I forgot to do that.
00:13:19 We won’t tell anyone you took a testing shortcut.
00:13:22 Well, actually, I think it should be. It should be more obvious. I think maybe Flit should just warn you and go, dude, you haven’t done the test first or something. I don’t know, whatever.
00:13:32 But when I look at I look at pipe.org and look up pytest sourcepaths SRC paths is the thing I pushed up.
00:13:43 And then there’s a file button or download files. I can look at that. And I see the pipe test underscore source path SRC paths. And then my dash version number 102, and then Pi two, Pi three. So apparently it’s Pi two and Pi three compatible. But I didn’t decide that. Did I decide that somehow?
00:14:06 Did you specify your requires Python in your metadata?
00:14:11 Let me go look.
00:14:17 Because if you didn’t specify that, I could see flat inferring that you’re two and three compatible.
00:14:27 So that would be my prime project Tomo file, right?
00:14:30 Correct. In your Flip metadata section.
00:14:38 Now I’ve got disk name, module author, author, email homepage requires just what it requires. Pipes, of course, classifiers and a description file. That’s it.
00:14:51 Right? So that’ll be why. So what you should do and another PSA everyone should do is in your metadata, there’s a requires Python field, and the naming might be slightly tweaked on every project, depending on or for every build tool. But they all support it because it’s metadata pipes. And it’s actually really important to Pip because what happens is when you specify your requirements Python, you’re specifying the version requirements for what version of Python you support. So for instance, Brian, if you only test against 36 and newer, you probably just list 36 and newer as the requires Python. Right? So greater than 3.6. Okay, if you don’t specify that, Flip is basically taking a guess. And like, well, if you didn’t say it, you can be compatible with anything. So I’m going to say you’re compatible with anything.
00:15:39 But what will happen is that also means someone on Pip, for instance, tries to install something, it’ll just be considered successful and install it regardless of whether it actually could be used or not.
00:15:51 Right.
00:15:52 So someone on Seven would import your code. And if you happen to have anything that’s Python three, they’ll magically get a syntax there when they first try to import your code. But if you said it’s only compatible three, six, and newer, people just say, sorry, you can’t install this, it’s not compatible with your version of Python that you’re trying to install.
00:16:07 For which that’s probably what I want.
00:16:09 Exactly. And then when you do the build, it’s going to be smart enough to say, oh, this is not Python two compatible. So it’ll just use a Pi three interpreter tag to say this is Python three compatible. And then for very specific, what versions of Python three, that’s what that metadata that requires Python specifies will be used for.
00:16:29 Okay, so the thing to think of Wheel Tags is basically they’re used by Pip and other install tools to quickly figure out what wheel should I try to install for? And then it does a slightly deeper dive when it downloads it, because then it looks at the metadata. Okay, does this exact version work? Because you’re not going to typically have if you’re going to have multiple versions on wheels. Like if you look at NumPy, for instance, they have 23 wheels because they cover all the major versions of Python on all the major OS. Okay, so if you’re on three nine, it’ll only grab the 39 wheel because there will be a CP 39. But when you’re generic like this, people go like, chances are it’s going to work, I’m going to grab it and then when I open it, I’ll double check. And then that’s when it’s going to double check.
00:17:11 Okay, so I probably should add that requires yeah.
00:17:14 It’S definitely worth it because that will also surface on Pip UI.
00:17:20 If you look in the left hand column, it’ll be shown there saying what the version specification is. So people will also be able to look and go, I’m still stuck on three five. Can I use Brian’s Cool Source Path project? Darn it, I need to go talk to management and upgrade because I really need to use this project or something.
00:17:37 My rule of thumb is whatever you run in CI for your testing is basically what you’re guaranteeing compatibility with. So that’s what I always try to match against.
00:17:47 Oh, okay, yeah.
00:17:51 Then I claimed 35.
00:17:56 That’s a good rule of thumb. And I’m going to try to find I don’t really want to push somebody just to the metadata section of the Flip project.
00:18:05 Is there some place that talks about this requires Python that you know of?
00:18:12 I’ll try to find a link later.
00:18:14 Every build tool will have the link to what metadata they support and then Flip will say in their docs that, hey, this is a field you can specify as to the meaning of that field.
00:18:24 You could link to the spec that specifically outlines where this gets written out. As to just general guidance like this, there’s hopefully something on packaging Python.org, but I will fully admit that outside of the specs, that website is fully understaffed by the volunteers are just really Swan because they’re pipe folk who also help manage Pip and pipelines other stuff. So keeping that website up to date is a struggle.
00:18:54 Okay, so I probably should fix that. So it’s not Pi two, Pi three, it’ll be Pi three only then after I fix that and then it says none, and the none is what the none is the Abi.
00:19:12 So because you’re not linking against any C Code or anything like that, there’s no Abi to care about. So you’re just saying I don’t care.
00:19:19 Okay?
00:19:20 There is no API restriction for me.
00:19:22 Okay? And then it says any. And that’s the what’s the any again, that’s platform.
00:19:28 So any OS, any CPU architecture. Once again, because there’s no C Code. You’re running Python. Python is cross compatible on the OS and any CPU. Okay, you don’t have to care anymore.
00:19:38 And it’s smaller. So this is 2.7K. And then also there’s the Tarball.
00:19:48 That’s the source.
00:19:50 So that’s 3.7 kwh. Why is that bigger?
00:19:55 For Flip? Specifically, what Flit does is to build your estate. What it does is it sticks into that Tarball. Everything you have committed into Get, the Wheel only has things that you need to install for Tip or any other install tool. So if you open up that Tarball, it’s going to have you read me and your docs and your tests and your source code and everything else that’s in your repo. Basically it’s a Tarball of your repo without the Get history. If you look at the Wheel, the Wheel will only have your source code, your license file, and maybe if you have any entry points specified, that kind of stuff. But otherwise that’s going to be it. So there’s literally less files because one is very much just kind of like a freeze of the code used to build the Wheel and the Wheels. Literally just what you have to have to install and make the thing work.
00:20:47 Yeah, I’m recommending for people even within a company or something, if they’re not pushing it to pipe, but they just want to share between teams or something. Flat is a great way to just share code.
00:20:58 Python code flip is great. I mean, if you have pure Python code and you’re not like on the poetry or Pipeline camp in terms of how that’s how you manage stuff. I totally recommend for any pure Python project. It’s fast, it’s simple, it totally just does what you need.
00:21:16 You never have to worry about a manifest infile ever again, which is usually how I sell people on this. There’s no setup CFG or set up PY file.
00:21:23 Well, what I like about it is the workflow. So I’m thinking about a workflow of somebody saying I got some code and I want to share it. Oh, I should add some tests and then stick a test directory in and start testing against it.
00:21:38 And then I want to share this with people. What do I do? Do I just stick the whole thing on a shared drive or something?
00:21:48 At that point, if you want to go the packaging route, like with setup tools, it’s kind of a big hurdle to go. Like how do I do this? Whereas with Flip you can just say flipping it and it like looks at your code and just says okay, I’m done, check your stuff in.
00:22:07 Usually it works. Okay.
00:22:10 Thomas Clover has done, I think, a great job of Flip because he very purposely made it just for Python source projects and setup tools. Complexity is a combination of both. Just being the first tool that tried to do packaging in Python because of that, it’s age, but also because it handles extension modules. Right. It does compilation of C Code for Python.
00:22:35 Flip doesn’t have to care about any of that. So because of that, Thomas has been able to do a great job of keeping the whole workflow and the project itself very small and contained such that he’s been very quick on implementing new standards when they come out and all that kind of stuff. And he’s been able to keep it nice and simple and fast. And as you said, workflow is nice and simple. It’s gotten rid of, I think, a lot of the nasty sharp edges that set up tools just has once again due to how it was initially designed, that they have to keep going forward compatibility, but also just does not apply if you don’t have C Code like keep your life simple.
00:23:15 And it does have a few guardrails on it though, which I actually appreciate.
00:23:20 It assumes you’re using a Git repo. I don’t know if it works on other source control or not, but I’m always using Git and I think that’s a safe assumption now.
00:23:33 So I just wanted to mention that because if people are using like ClearCase or something RTC.
00:23:40 Oh yeah, that would probably be a more obvious one.
00:23:46 Your mileage may vary, but the other part of it is and it checks you. If you try to build something and you have stuff checked in, it says you probably want to commit this stuff first or something. I can’t remember it gives you some messages, but it also allows you to run talks. So if I do talks on this, I can run my talks on checked out stuff. I don’t have to commit at first cause so that’s cool getting away from that. Okay, so that’s like a source only you’re talking about like NumPy or something that’s compiled with some C Code.
00:24:22 What’s the workflow like there? Do they have to build the C Code stuff on different machines first to build a wheel? Build all the wheels.
00:24:32 Yeah.
00:24:33 So the way this workflow typically works for extension modules is you get it all set up with set up tools. So you install set up tools and you install the wheel project. This is actually why the default if you use Pep five one.
00:24:46 I think the last time I was on we talked about Pep 517 and Pep 518. Right. So the reason Pep 518 defaults if you don’t have the build system dependencies specified as laid out in Pep 518 and Piper Tumble the default is set of tools and will because that’s what’s required to build the vast majority of the world into a wheel file, and hence why that’s done for Pet 517 benefit.
00:25:13 So basically you set up set up tools and you have wheel and then this is where you typically end up getting a machine for the platform you want and you run on Pythonsetup PY. Bdist.
00:25:27 And then that’s how you build your wheel. You can do SDS pretty much anywhere, unless you have custom code. Crazy custom code somewhere that has to run on a certain OS in order to package everything up appropriately.
00:25:37 And this is why source distributions are so nasty to deal with as an installer for Pip, because it can’t trust anything who knows what was specified and how it was run and everything. This is why static metadata is such a big push and important thing going on in the packaging ecosystem and trying to get everyone to move to if you’re on setup tools, please use Setup. Cfg.
00:25:57 I can’t officially speak for the setup tools developers, but I bet they will say please use Setup. Cfg over Setup. Pie. Static metadata is really useful and important and allows automation and other things and just makes everyone’s lives easier. So please use that.
00:26:15 Oh, I did. I thought it was. In addition, you don’t need a Setup pie if you have a Setup CFG.
00:26:21 Yes, if you use Pep five, one, seven, you can skip the Setup. Pie and use Setup. Cft.
00:26:28 I don’t know how editable installs will work if that’s a thing for you, but you can actually, with PEPs 517 with a pipeline tumor, you can just have a Setup. Cfg and you can ditch the Setup pie.
00:26:44 Okay, but the key thing is specify as much as you can in your Setup. Cfg.
00:26:49 But yeah. So basically the way this works is like, let’s say you set up GitHub actions for your project on GitHub.
00:26:54 Okay?
00:26:54 You would then have one of your workflows or one of the actions, build your wheel on Mac, build your wheel on Windows, and build your wheel on Linux. And then you’d end up with three separate wheels, assuming they’re all the same version of Python. If you have to build for other versions of Python, you repeat the build matrix for the version of Python you need to support.
00:27:15 Okay?
00:27:15 And then you’d end up with the individual wheels. This is basically because to the best of my knowledge, set of tools isn’t really structured for cross compiling. So there’s no way to have like your Docker container or something that’s all set up appropriately and say, hey, go build me a Mac wheel, build me a Windows wheel, and build me all of these other wheels for all these other versions of Python.
00:27:38 It’s just a project. I don’t think it’s structured that way, and it’s unfortunate. It’d be cool if we could get there. I don’t know how much work it would be to add the set of tools that might honestly require a new build project to come about that supports this kind of workflow. But as of right now, there isn’t a good story for that in Python.
00:27:55 I mean, conceptually, you could do it yourself, right?
00:27:58 Instead of a CI system, you could have Docker images on your machine or something to do that.
00:28:06 Yeah.
00:28:07 And then do you have to get all these pieces together first before you push it to Pipe? You have to push it all at once? Or can you put them one version like macOS first and then work on Linux?
00:28:24 I believe the API will let you push up as long as you’ve already done it and it’s not overwriting your pre existing file. You can push up more files if you add more wheels.
00:28:35 Okay.
00:28:36 But I mean, typically I just do it all at once.
00:28:38 Yeah. Okay. That’s probably what everybody does.
00:28:41 And then one of the things it’s funny. So my little source paths package that I put up it’s at, I haven’t made any changes to it. It’s it’s the original thing. The reason why it’s not 10 zero is because I had changes to the README or something. And you can’t even push a new read me to pipe without changing the version.
00:29:05 So actually a trick to know is there’s actually a way to do that without bumping the version number? Bumping the bug fixed version is their support for post versions specifically for this. So instead of pushing 1.0 .1 and 1.0 .2, you could have pushed 1.0 .0. I can’t remember if it normally, I think, normalized to a point. Post one and 1.0 .0 dot post two, and then what would Pip would do is go, oh, this is probably fixed to read me or something. I don’t need to download a new version because post versions are just basically bookkeeping of there’s something that changed, but the actual source code didn’t, and so it won’t bother to redownload and install it.
00:29:52 Okay. And I’m sure Flat supports post versions then.
00:29:57 Yeah, because it’s just part of the packaging spec on version specification. That’s for 40. I think maybe you know how you can specify Alpha, beta, AB, and C as C stands for release candidate, but you can also do post. Okay, that’s the easiest way to do. If you literally have zero code change, you can use the post and then Pip will still install the newest version with the highest version of the post number. But if it has a version installed already, the only difference by a post, it just won’t bother because the assumption is the code didn’t change. It was just like read me or some metadata. That’s not important and we can totally skip it.
00:30:39 Yeah, I guess we talked about Python requires. But the one thing I also wanted to change that wasn’t the default with Flit was the disk name, because Python has this weird thing.
00:30:57 I don’t know if it’s convention or required or something, that if you’ve got dashes or underscore. So I need to underscore for the directory name. Like, for instance, Pipest searchpath is underscore search paths, but the package name. Most people are looking for Pipest plugins by searching for pytest dash. That’s just the convention. So when I pushed it up at first, it assumed the same name. So it assumed my package name distribution name would be pipes underscore source paths.
00:31:35 So I had to add that metadata. I had to look that up.
00:31:38 So there’s just a couple of new metadata things that aren’t by default.
00:31:43 Probably do a blog post on this or something.
00:31:45 Yeah. So the reason that happened is actually remember how we’re talking about how real file names are structured and they’re dash to separate and everything. You totally break all the algorithms in the world if we let you have a package name with the dash in it.
00:32:00 But the distribution name can have a dash in it.
00:32:03 Yeah, because source distributions literally just so for now, source distributions just got standardized kind of implicitly in 517. And we kind of hardened that up a bit to now normalize that name as well.
00:32:15 Okay.
00:32:16 So your metadata can specify that the project name and Pip has the dash. But the normalization is to do an underscore because we use dashes to separate version numbers and source distributions, or dashes to separate all the parts of the wheelchair and everything. That’s why you see that happen.
00:32:34 Well, the disk name also shows up.
00:32:37 It just shows up as the name I search for in Pip and what I Pip install.
00:32:44 You shouldn’t have to worry about this. Pipeline handles the normalization in the look up with Pip. So if you want it to be searchable and show up with the dash, that’s totally fine. As you said, Pi test plugins, for instance, almost always pytest something.
00:32:56 The file normalization is a technical thing you should never have to worry about. This is all what your build tool should be doing for you and taking care of it. And it should not be your concern. Basically, it’s a file, and it’s not how Pipe does its searching. It’s just a searching based on the metadata.
00:33:14 And the reason you do that for flight is because Flip automatically assumes the directory name you specify is going to be the name of the project. So you had to overwrite it because you wanted that dash specifically, even though you put in the underscore because.
00:33:25 Okay, that’s where I was confused. Because I guess there should be a project name that’s different. Maybe because Fled is using disk name for project name.
00:33:38 Yeah. For distribution. That’s what they’re considering the project name.
00:33:43 Okay.
00:33:44 And then you have to specify the directory of where the source is. So you’re just telling me don’t use the name of the directory as the project name. Use the one I’m telling you to use. And it’s the one with the dash so clear as mud.
00:33:55 I think if I’m doing a Python only I don’t have to care about Apple Silicon or anything like that.
00:34:00 Yeah.
00:34:02 But like you said, if I have some C Code that needs built, what did you call it?
00:34:10 Universal two.
00:34:14 Why is it universal two?
00:34:16 Because there was universal one when they moved from Power PC to intel.
00:34:22 Okay.
00:34:22 Universal two is an Apple thing.
00:34:24 Yes. Okay. It’s their version of that binary. So when Apple moved off of Power PC and over to intel architecture specifically to X 86 and then to X 64, eventually they developed this fat binary format where you can basically ship versions of files that contain both a Power PC version and an intel version.
00:34:50 Okay.
00:34:50 Basically they’ve just continued on with that format because after the Power PC, they still had X 86 and X 64.
00:34:58 Right.
00:35:00 And then now that they got into this point with Arm, they’ve just said, all right, well now you can ship. And they dropped 32 bit. So now it’s just X 64. Well, we’ll just now just bring an Arm. And so they did. And the decision was made instead of having a really ridiculously long wheel file name of saying, well, this is X 64 compatible and Arm compatible for all versions between OS Ten version 10.9 to Mac OS 15, the decision was just call it universal two. It’s going to be a fat binary with all the stuff. Just say it’s compatible with this version of Python. And we just introduced that name. And as long as you update to the newest version of all your tools, everything’s taken care of, anything will just work.
00:35:42 You have to do a rebuild for your wheels to get the new stuff.
00:35:46 But that’s basically it just new. Update the latest versions.
00:35:49 Okay.
00:35:50 Do your Mac builds again and you’re covered now as you can do that, by the way, on Arm hardware, once again, consider tools to do Cross compilation. You have to have an Arm machine to do those builds.
00:36:01 Okay. But I have to have a CI system that has our stuff.
00:36:09 But on the Python side or whatever my source code side you called that extension modules is extension modules. That just means if I have some C Code. Okay. Now when I look up extending on Python.org docs, Python.org, it has a page called extending and embedding the Python interpreter. That’s something different, right?
00:36:39 Yeah. So that’s actually when you want to embed the Python interpreter into an app, kind of like how Blender and Maya and RGIs all embed the Python interpreter for scripting purposes. That section of the Doc is all about how do you.
00:36:51 Okay, that’s something completely different.
00:36:53 Yeah.
00:36:54 Cool.
00:36:58 So I think that I’ve got myself into a place where I think I just really like Wheels, I like Flit, and I really don’t want to write extension modules.
00:37:08 Yes. That’s a good place to end up in.
00:37:11 And if you want to continue to avoid that, this is where people start to use Python or CFI or Pi Bind. Eleven. All of those tools that help you write wrap any of that CNC plus code you have, it’s definitely worth using these higher level tools that have come along from various places to do it versus doing it by hand. Like, I literally got carpal tunnel in my first internship at Google due to writing an extension module by hand.
00:37:37 Okay, right.
00:37:41 But you also don’t have to care about it if you’re using work that other people did. Like NumPy or something.
00:37:49 Yeah, because they’ve done the job of building the wheels for you. Like the whole point of Wheels, the reason Wheels came into existence was specifically to try to tackle the problem of, well, not everyone’s going to have a C compiler, especially on Windows. If you don’t bother to install the tool chain, you’re not going to have the compiler. So how do we get it?
00:38:09 So the package creators don’t have to have their users have the tool chain already set up and ready to go. And this is what birth wheels. And this is also, by the way, why Conduct came into being. Right? Because they got to it first and then they took a slightly different tact in terms of standardizing the build tool chain and all that crazy stuff. And that’s why there’s this difference between wheels and Conda and kind of how there’s been this permanent bifurcation in terms of approach.
00:38:34 Okay, well, you cracked that door open.
00:38:36 I did.
00:38:37 So one of the things that gets me is it just I don’t use condo if you don’t have to, except for I teach people how to do stuff. And so whenever I talk about virtual environments, they say, well, is that the same for Honda environments? And I have no idea.
00:38:59 It’s fairly similar, actually.
00:39:03 Environments designed to isolate themselves from other environments on your machine. The biggest difference. So the key difference between the P-A-P-I. World and the condo world is the way Conduct packaging and approaches and stuff is they standardize the entire build tool chain across all packages for versions of Python. So if you build a project for Python 3.9, every kind of package is going to use the exact same build tools, the exact same version of GCC, the exact same visuals, C compiler, et cetera, et cetera. Everything is compiled with the exact same stuff across the board to guarantee this Abi compatibility as much as possible. Because, I mean, this is the reason conditions used by the science world and came out of the science world because they have crazy stuff like Fortran code. Right. Like they got nutty stuff that most people don’t have to deal with. So they went out and figured out their own solution to this problem before Pipe came up with the wheel concept.
00:40:09 Okay.
00:40:10 So that’s the reason they did that. The other key thing is because they do that, they also typically have you install a specific version of Python that once again was built with the exact same tool chain. So one of the interesting things you can do with condo environments is they use environment. Yaml as the file name in your environment. Yml you can specify the version of Python that you depend on. And so your environment will not only have all the packages you specified and have them all be compatible, but they will also install the virtual Python that your environment is dependent on. So there’s one little extra bit of metadata they have that will do an install for you that we do not have with virtual environments, and that’s be able to specify the exact Python version you want.
00:40:54 Okay. But if you’re using Condo, you install through the condo world.
00:41:04 Basically, they have Pip compatibility now, so you can also install Pip if necessary.
00:41:09 Okay.
00:41:10 But typically, if you’re in the kind of world you prefer kind of packages because they’re way more guaranteed to be compatible with each other than some random wheel pulled off.
00:41:18 Do they have a way for people to just package some Python stuff?
00:41:24 Yeah, they use Honda.
00:41:26 I can’t remember the exact terms they use. I don’t think it’s definition Conda build files. I can’t remember the terms that they use, but yeah, they have a similar here’s the metadata you specify in terms of your requirements of how to build it. And then there are projects like Honda Forge on GitHub whose entire purpose in life is to take projects like NumPy and Pandas and stuff. And Auto run across multiple CI systems to get as many compatible, compatible Condo packages as possible. And they maintain this big piece of infrastructure that Auto updates all the build rules and all this stuff and get some building across all the CI systems that they need to package all the stuff up so that when you install from Condo Forge, you’ll have the packages you want.
00:42:11 Okay, well, this is not the talk I thought we were going to have, but I enjoyed it. And actually, one of the things I’d really like to have, and I might have to build it myself if nobody else does. But somebody else could go ahead and do it. And I’d probably feature it is a I want to share some code. How do I do that? Because it changes all the time. Like you were saying, we don’t really want set up. Even if setup tools. We don’t really want setup pie. We want people to use set up CFG. I think people are using setup pie just because there’s so many examples out there of how to do that.
00:42:47 Yeah. And honestly, your packaging code is usually the last thing anyone ever changes.
00:42:53 It becomes habitual. It’s like, oh, I made this work in my last project. How did I do it. Oh, I did it with the setup Pi. I’ll just copy that over, change the names and a couple of things here and there. And people just literally copy and paste so much of their stuff from previous times that it’s actually a real problem that we have. When we try to push out new things and we try to update like Pi project Tamil, for instance, there’s been resistance to it, not only just because some people just don’t like the idea that we went with Tamil instead of Ini files or something, but also just because why it works for me, I don’t need to switch. So for me, for instance, it’s a constant thing of trying to find that carrot to get people to want to switch. There’s this benefit. You want to do this. Trust me, it’s not a waste of time. It’s not going anywhere. This is the future of packaging. It’s the way to do it.
00:43:46 Please look at it.
00:43:47 Yeah. But I’d like something that says, okay, I want to just share some code. Here’s how to do it in setup tools. Here’s how to do it in flit. Here’s how to do it in poetry.
00:43:58 And then we’re just going to keep that updated.
00:44:01 Yeah. It’s a little tricky just because it’s one of those opinionated things. Like there was massive Flack when Packaging Python.org recommended Pipe amp, for instance, people got really upset that a specific tool was listed. Now, it was very clearly labeled a tutorial. It was not saying this is the tool to use, period. Here’s a tutorial on how to package up an app. We’re going to use Pipep. And so they did. But people did not like that. So it’s tricky because as soon as you give I mean, if people disagree with your opinion, they’re not going to be happy. And by the way, this is one of the reasons we don’t have a single tool. Right. Like, if you look at some of the communities like Node or Rust or pretty much a lot of other ecosystems, they started off with a single tool. And so they just kind of said, we’re going to have packaging. It’s going to be this tool, and this is the way we’re going to have workflows go. And they went, okay, that was the start, right. For us because packaging was such an organic thing that came about in the Python world.
00:45:05 Right.
00:45:05 I remember the vaults of Pronassus back when everyone just downloaded the zip file off of one random website with some animated gifs. And that was how you install Python projects, right?
00:45:16 Yes.
00:45:17 So we kind of organic. You grew packaging and we learned lessons and we clean things up. And then people now go like, well, why don’t we have centers in one tool?
00:45:27 Well, guess what? I can guarantee you if we chose a single tool, there are going to be other people who disagree with that choice of that tool. And they go, no, that’s not the tool I meant. I meant my tool.
00:45:35 Yeah, exactly right.
00:45:37 So we’re kind of just basically too far along to go, hey, here’s the canonical list of commands and things to do, because inevitably someone’s going to disagree. Why didn’t you use this? I like this more. It solved my need better than this one will. Okay, so it’s this weird spot where we try to list tools. I believe if you go to packaging Python.org, there’s a list of tools you can use, but we try not to have opinions on which tools best per se. Like, we might say this tool is really good if you’re only like Flit is good if you only have pure Python code. Same with Poetry and Pipeline. Right? All three of those tools solve it. But having that website say Flit is the one you want over Poetry and Pipeline is going to not only cause some consternation amongst some people, but also Flip doesn’t manage your virtual environments like Pipe and Poetry do.
00:46:30 That’s why I use it.
00:46:31 Exactly. Well, same here.
00:46:33 But some people do want that as part of it. So even getting Apples to Apples comparison is hard.
00:46:39 Yes.
00:46:40 So unfortunately it’s one of these things where basically you just have to have people do blog posts that just are somewhat opinion on this. I will fully admit I’m not really ready to share or announce anything, but I’m very slowly trying to build up almost like an example repo to try to figure out where there’s some rough edges in the whole getting a project off the ground and that’s packaging, that’s testing, that CI and minimizing all the metadata. Like right now, for instance, it drives me a little nuts that I have to specify requires a Python in my package metadata. But then I have to go tell talks and knocks what versions of Python I want tested. And then I also have to then go tell my CI system what versions of Python to install to then run. Like I should specify this once and all the other tools should just be able to do the right thing, whether it’s auto Gent or reading from your metadata.
00:47:32 Yeah.
00:47:32 So I think there’s work in general in the community to try to smooth out some of the rough edges of how do we kind of just make everything coherent and just kind of easy to set up? And I think part of this is, as you said, figuring out an easy way for all these factors to work out. And I mean, this is one of the reasons I worked on Pep 621, which tries to standardize specifying the metadata for packages in a way that’s standardized across all tools and pipelines tumble because it means you can write that one tutorial. This is here’s how you write out, like your dependencies choirs, Python, all that stuff. And then for each of the tools, you just have this small, small section that’s specific of. All right, here’s how you specify what source code to include or build versus all the standardized stuff and that way it’s actually easier to evaluate tools, it’s easier to write that tutorial and it becomes less and less of a problem.
00:48:26 Yeah, that’d be great.
00:48:27 I hear you and it’s a life goal. We’ll see if I get to it before I die.
00:48:32 Okay, well, I think we should wrap it up. Actually, we didn’t really get to 621 much, so maybe we could kick that down the road and also trove classifiers, but that might be a big can of worms.
00:48:46 Trove classifiers are their own story. Some people actually want to see them just go away.
00:48:50 Their usefulness is slowly dying because of requires Python and stuff.
00:48:55 Anyway, that’s another story, but yeah.
00:48:57 Okay. Well, cool. Well, thanks for your time today and I learned a lot.
00:49:01 Anytime.
00:49:08 Thanks, Brett. I always learn a lot when we talk. Thank you also to Patreon supporters join them at testandcode.com support thank you Datadog for sponsoring this show. Check them out at testandcode.com Datadog and start tracking the performance of your ads apps with a free trial and get a T shirt. That link is also in the show notes@Test And Code.com 152 that’s all for now.
00:49:31 Now go out and test something.