Len Wanger works on industrial 3D printers. And I was pleased to find out that there’s a bunch of Python in those printers as well. We talk about 3D printing as well as finite state machines and testing software and hardware.
Transcript for episode 121 of the Test & Code Podcast
This transcript starts as an auto generated transcript.
PRs welcome if you want to help fix any errors.
00:00:01 A couple years ago, I bought a Python Staff of Enlightenment from a company in Australia, and they made it using 3D printing. I know many of you have used 3D printers for fun inside projects. Even my neighborhood craft store is a small 3D printer for sale for a few hundred dollars. I don’t have one, though, but I would love to play one with one. What I didn’t know was 3D printing is also used in industry to make super strong and high temperature parts for automotive and airline and other industries. And that’s pretty cool. I didn’t know that until I started talking with Len. That is. Len Wanger works on industrial 3D printers, and I was pleased to find out that there’s a bunch of Python in those printers as well.
00:00:48 In this episode, Leon and I talked about 3D printers. What are the different types of 3D printers? Where are 3D printed industrial parts being used? Why would you use one type of additive manufacturing over another? And because this isn’t just that kind of a podcast. We also talk about the Python that’s used in 3D printing hardware. We’ll talk about finance state machines, the benefits of FSM for testing, logging and breaking a complex behavior into small testable parts, and also the benefits of simulation and writing and testing software to control hardware. It’s a really fun episode, I think I think you’ll get a lot out of it, even if you’re not into the hardware software thing. This episode of Test and Code is brought to you by Data Dog and Circle CI, and I want to thank them, and I’d like you to thank them by listening to their segments in the show and to check out their offerings with the links that I’ve provided.
00:01:45 This episode is also brought to you by listeners just like you that support the show through Patreon, thank you so much for helping to support the show.
00:02:07 Welcome to Test and Code, because software engineering should include more testing.
00:02:19 On today’s episode of Test and Code, have Len Langer, you and I exchange some emails on the state machines, but I then found out that you work for a company that does 3D printing, and I’m fascinated by the sort of work that you’re doing because of course, I know about 3D printing, but I thought it was things like people building mock parts. And I’ve heard about people doing rapid design of skateboard pieces and things like that. And I know it’s done in lots of industries, but you’re kind of at the high end industrial type printing, is that correct?
00:03:02 That’s right.
00:03:03 What are the different types of 3D printing that are around?
00:03:06 Well, happy to talk about that. And thanks for having me on your podcast here.
00:03:11 So 3D printing, it’s about a 40 year old technology at this point, and it’s really blossomed into a lot of different categories.
00:03:19 You’re right. When most people think about it, they think about an FDM or fused filament printer. These are the plastic filaments that look kind of like big rolls of plastic spaghetti, and it’s kind of like a hot glue gun. What it does is it melts the plastic and puts a small layer of it up and then drops the bed down a little bit and puts the next layer on top of the previous one and builds it up. So it’s something we call additive manufacturing because opposed to subtractive techniques that a lot of people are used to, like carving things or drilling things, you’re building it up layer by layer. And it turns out there’s a lot of different ways of doing additive manufacturing at this point, and it’s really hit all over the place. So we talked about FDM. There a little bit the fuse deposition types, but there are also things like sterile lithography where you take pools of photoactivated resin. And what you’ll do with the laser is you’ll coat a small layer on top, Harden it with ultraviolet light or other techniques, and then drop it down a little bit and recoat it with another piece. There are pieces with beds of powders. Those powders can either be polymers, plastics or metal. So they’ll coat a small layer of powder, and then they’ll melt it with a high energy source such as laser. And you can make plastic parts, or there’s a large category now direct metal parts. You can make that way. We have a process that makes composite parts, things like carbon fiber composites and fiberglass composites. That’s called seabam composite based additive manufacturing. We take layers of fabric. We coat it then with Palmer’s in the place where you want the part, we stack these sheets up together, put them in a Press and consolidate. I’ll talk a little more in detail, I’m sure, about this later, and can make very strong, very light composite parts. But people doing all kinds of things. People are printing with chocolate. People are making organs. They’re doing bio printing to print cells. There’s a whole proliferation. What I’d really do is I’d split it into two main categories. They’re really sort of the consumer categories. These are the types of things you see in schools, which are generally these FDM or Filament printers. And then there’s a whole industrial category. And that’s where we play is making end use parts for manufacturers. And really it’s really blossomed to where we’re making things for some of the largest, most sophisticated manufacturers in the world, Ford Motor Company, the United States Air Force Jabel, which is a large contract manufacturer. It’s going in airplanes and cars and medical devices and everything you can imagine in the industrial space.
00:06:05 Where are these things showing up, for instance, an airplane or something? What part might be made in this process?
00:06:12 Right. That’s a terrific question.
00:06:14 And airplanes and cars are also a very difficult area because these are highly regulated areas where safety is of a big concern. So it can take a long time to get a part qualified on a plane like that. To give you an example, a commercial airplane right now might have three to 403 D printed parts in it. This is, for instance, a Boeing or an Airbus jet that you’d fly in or precove it. You’d be flying. No one’s flying right now, but the 300, 400 parts. Now, this platform might have 100 or 120,000 unique parts on it. So it’s still a very small number of parts where the low hanging fruit for the parts that you’re seeing are these are generally very high complexity parts, geometric complexity. So a large, flat surface like what you have on your wing is generally not a great candidate. But GE, for instance, has a new model of engine where they’re doing fuel injectors with it. And what they’re finding with it is they can consolidate from having 16 parts in the previous version of it to doing one 3D printed part, because you can do a lot of complexity of your part without adding a lot of cost to it. In traditional manufacturing, when you add complexity, you add cost in add to manufacturing, the complexity is free is what they like to say. And that’s basically true. So they’ll find that they get 30% increase in efficiency and a 15% reduction of weight by doing these fuel injectors. What you’re tending to find are these smaller, very complex parts and what we call non rotating parts. They’re generally not the structural parts that you’re depending on, because additive manufacturing tends not to be as strong a part, for instance, as a cast metal part.
00:08:02 But there are a lot of places where it’s very useful, where you’re seeing even bigger proliferation of it. Also those where you don’t have as big regulatory issue with it. So, for instance, an unmanned drone where there are not people flying in it. So you’re not as worried about the safety.
00:08:19 You can see many more parts and you can move much more quickly, but it’s really going all over the place, and it’s hitting in all kinds of funny areas on top of it.
00:08:28 I can name lots of different applications, but one of them that we’re seeing and that’s very interesting, for instance, is just for the manufacturing of circuit boards, where when you’re doing Wave soldering, you put it on a board to hold the components onto your green circuit board. And we’re making a lot of these printed circuit board carriers, for instance, something that no one, including us, would have ever thought about using Ford. But it turns out to be very good for 3D printing. So there are all kinds of things you can do, but it generally tends to be lower volume, high complexity parts.
00:09:01 Okay. Yeah.
00:09:03 And those are things that are already expensive to make and traditional processes, then.
00:09:10 Yeah. The low volume is an important aspect of that. When you’re doing manufacturing, you’re often making tooling and the tooling can be very expensive. So when you’re making a low volume that you’re advertising out the cost of that tooling with every part you make. So if you’re only making 100 and your tooling costs $1,000, you’d be adding $10 per part for the tooling. Now, by the time you get up to tens of thousands, hundreds of thousands, you’ve gone to a point where it’s really de minimist for that tooling amount. But. So if you’re making low volume, making one or five or 100 of it, you can afford to spend more for the apart from a 3D manufacturing because you might be saving from these other pieces like the tooling.
00:09:54 Okay.
00:09:55 An additional piece to where you could justify cost often is the time to market and the speed that you could move to make a redesign. Get it out there. There’s a lot of value in being able to get a part out there quickly.
00:10:08 Yeah, definitely.
00:10:12 This episode is sponsored by CircleCI. Designed for modern software teams, CircleCI continuous integration and delivery platform helps developers push code with confidence. Trusted by thousands of companies, from four person startups to Fortune 500 businesses, CircleCI helps teams take their software from idea to delivery quickly, safely and at scale. Visit Circle CI Testingcode to learn why highperforming DevOps teams use CircleCI to automate and accelerate their CI CD pipelines. That link, again, is Circle CI Test And Code. And that link will also be in our show notes. And you will definitely want to go and check that out before July because you can enter to win a pair of Sony noise canceling headphones. Very cool, Circle CI. And thank you for sponsoring this episode.
00:11:08 You work at a company with a really cool name, right. Remind me what it is again.
00:11:13 The company is Impossible Objects.
00:11:16 It’s a little bit of an oversell here. It’s probably improbable or Difficult Objects. It would probably be a different name.
00:11:22 But Impossible Objects, it’s a great day. Thank you. But I was looking at the site, and some of the composites have a melting point of, like, over 300 degrees.
00:11:33 I didn’t think you could do that with plastics, so that’s pretty cool.
00:11:37 Well, Celsius, not Fahrenheit. I mean, these are very high.
00:11:43 When you get into the world of composites, you can get into some very interesting material properties. You can get to very high melting points. You can get to a lot of chemical resistances that can be used in harsh environments. One of the reasons we’re being used in the soldering pallet environment is first you hit this with liquid metal with liquid solder. Right. So it has to be able to take the thermal shock of that, and then you’ll have to clean it off with some fairly nasty chemicals afterwards. So the material properties really matter. And that’s one of the big lessons. I named several different categories of add to manufacturing. And the real question is, why would you use one versus another? And why would you use that versus any of the other ways of manufacturing an object? Stamping and casting and drilling. And it goes on and on, and a big piece of it is marrying the specifications of the materials you can get. We tend to use carbon fiber or fiberglass as a substrate and then marry it with one of different polymers. One of the polymers we use is Peak. That’s peek ether, ketone, I believe it stands for. It’s kind of a mouthful. And this is a very high melting point plastic that’s used wisely in the medical and aerospace industries.
00:13:06 If you have an airplane, you’re very concerned about a fire because people can die of smoke inhalation before you can get to the ground. So they’ve got very rigorous smoke and fire regulations on it. And something like Peak has a very high melting point and also doesn’t put out a lot of smoke. So there’s a lot of technology that goes into these things. And, yeah, there’s a little surprising you can make things out of these. It’s kind of neat.
00:13:31 It’s not surprising that it’s going into things like drones, because these high strength, high melting point parts are also lighter than metal. Right.
00:13:42 You have a smaller engine on your drone and less battery if you have lighter parts.
00:13:49 This is a really interesting conversation, but we are the podcast that talks about testing and Python.
00:13:56 How are we going to get back there? Right.
00:13:59 You’re a software developer, right?
00:14:00 Yeah. My background is all in software and computer graphics, actually, which led into simulation and which led to 3D printing and. Yeah. So what’s interesting is we actually have developed our machines that we sell. The software that runs them is based on Python, which is kind of a curious choice for hardware, but it’s worked out very well for us.
00:14:29 Okay. How long have you been doing this?
00:14:31 Well, I graduated College in the mid 80s, 87, and went off to work for Hewlett Packard. And those days they used to make graphics workstations at Hewlett Packard, similar to what Silicon Graphics later did, which now basically looks like $100,000 workstation that has been put into a $200 Nvidia card, except it’s about 2 million times faster now.
00:15:01 So I’ve been doing software for quite a while and a lot of different incarnations there.
00:15:06 The computer graphics I’ve done, though, has always been on not the picture making side of it, but the software that makes the pictures of the rendering algorithms, et cetera. So if you’ve used technologies like OpenGL, more on that side of the implementation there.
00:15:22 And I got into Python early on. I was a researcher at the San Diego Supercomputer Center for a while, and we were working on climate simulations for NASA, and one of our competitors was implementing a visualization system.
00:15:40 A guy named Randy Powell ended up being kind of an interesting character.
00:15:45 Very sadly, he died quite young, but before he did, he wrote a book and was on Oprah and it was quite a story.
00:15:53 He had used Python in a technology called Pandas 3D that’s different from the data frame Pandas. It was a 3D visualization piece and so I started playing around with Python around then it was probably Python 1.8 at that point and started using it really in earnest about Python 2.1 for some projects we were doing 2000 or so. I can’t remember the dates.
00:16:19 You’ve been using Python for quite a while then I have and I’ve really enjoyed it.
00:16:25 It’s really a fascinating language.
00:16:29 I’m sure all your users have heard a lot of these Python podcasts, but the ability to leverage developer time and the productivity you can get out of it is remarkable. And then on top of it, with the packages available in the community, it’s just a remarkable tool to use. And that is one of the secrets where it’s been so useful for us on the implementing 3D printers.
00:16:51 Developer time is way more costly than computer time. I got into it because of test automation of testing hardware, the communication costs and latencies for things like just the communication from your computer to the device, and then things like waiting for settling time, those sorts of delays. The communication was the bottleneck. It was never Python. So developer time was ideal to be able to write in whatever you can. That makes it easier to read.
00:17:22 Completely agree, right?
00:17:24 I guess it’s not really that surprising to me knowing that the strengths of Python that Python could be effective on a 3D printer because you’re in similar situations where there’s physical processes going on, right?
00:17:39 Yeah. I mean, we might be printing ten pages a minute our printer, since we’re using sheets of fabric, we actually talk about it in pages and if you think about it from a processor period of time, you’ve got 6 seconds in between pages. That’s an attorney to a processor on top of. When you start using multicores and multi processing, you just have a real luxury. Their compute time. And the Python, it tends to be really for top level message passing between all of our systems, so it can be quite slow there. And the things it’s talking to tend to be tightly rolled. See whether it’s a NumPy or a pill or pillow or a library that we write or talking to a microcontroller.
00:18:26 The Python slowness is really not the issue for us, and if it does, there are lots and lots of techniques to pull some tricks, multi processing or writing a little bit of C Code to get around it number all kinds of different things and we generally haven’t had the resort to too much that the standard C packages out there like the Numpies and the Pills and some intelligent Python has gotten us quite far.
00:18:56 Are you having trouble visualizing bottlenecks and latency in your apps and are not sure where the issue is coming from? Or how to solve it with Datadogs and monitoring platform. You can use their customizable built in dashboard to collect metrics and visualize our performance in real time. Data Dog automatically correlates logs and traces at the level of individual requests, allowing you to quickly troubleshoot your Python applications. Plus, their service map automatically plots the flow of requests across your app architecture so you can understand, dependencies and proactively monitor the performance of your apps. Start tracking the performance of your apps with a free trial at testandcode.com. Datadog. If you sign up for a trial and install the agent, Datadog will send you a free T shirt.
00:19:47 When you say fabric, I think of like a Bolt of fabric to make T shirts and stuff.
00:19:53 It’s not much different. Instead of your cotton thread, though, you’d use a thread of glass for fiberglass or thread of carbon almost looks like hair. And they use similar to a paper making process to lay it down and specify thickness. Our fibers tend to be about half inch to an inch long. They refer to them as long fiber. They’re different types of carbon fiber. You’ll also hear of woven fabrics or continuous fibers, but we actually do get bolts, thousand yard long bolts of this carbon fiber fabric. And we cut sheets out of it because we have a sheet based system. There also would be role based or web based systems that you could make, but it is very similar to what you’d get with your cotton T shirt. In fact, we have used different natural fibers such as silk and cotton to print on.
00:20:43 We haven’t commercialized that, but it is a sheet very similar to what you’re thinking about with that type of cloth.
00:20:49 I assume. Then there’s leftover stuff that you’re not using, right?
00:20:53 Yes. Well, waste is one of the big issues here because there’s leftover in a number of places. When we stack our sheets up, you get something that looks kind of like a brick of material with the part embedded inside it, and you blast away the part of the sheet where there’s no part and the part is revealed in there. So that is a waste product there. We have some plans to do recycling right now. We don’t have enough volume that it’s making sense, but we’re working on that one. And then. Yeah, there are bits of fabric leftover.
00:21:24 Okay.
00:21:25 But you have waste in manufacturing in the different processes. If you are using a CNC milling machine, which looks kind of like a drill on a six axis arm, you can end up with 90% of the metal and shavings on the floor, so you have to pick your poison there. But waste is clearly one of the factors you need to look at.
00:21:49 Interesting. Okay, well, yeah, definitely. All manufacturing has to deal with those same sorts of issues, right?
00:21:57 Absolutely.
00:21:58 Within your Python, within this stuff, you’ve got finance state machines.
00:22:02 Yes.
00:22:02 That’s how we started our conversation to somebody that’s never heard of it. Can you describe what an FSM is?
00:22:09 Sure. And it’s used quite widely in the hardware world. So a finite state machine, it’s a data structure, it’s a directed graph structure. So what you have is nodes, and you’ve got connections between these nodes and each node represents a state.
00:22:28 So in our printer I’ll use that as an example. As we go along, I might have a state that’s initializing. And after initializing, I might have waiting for a print job from my print server. And then I might have waiting to print the first page and then printing the first page and then go back to waiting to print the next page. I might have another state. That’s an error state. What you have in the States then is you have a way of processing and dealing with inputs.
00:23:00 Inputs could be, in our case of a printer, a sensor value. It could be a click from the GUI telling you to do something.
00:23:09 It could be a status from the hardware. And what the state then is it knows how to respond to these inputs for the particular state it’s in.
00:23:19 And it also has another event that it can do, which is transitioning to another state.
00:23:26 So what you have is if we were to try to understand what to do with all these different inputs in all these different conditions, we’d have just a little rat’s nest of if then statements, and it gets unmanageable very quickly. But it’s much easier to say if I’m in the printing state and I get a button press in my GUI that says pause. What I’ll do is I’ll transition to the pause state. And what I can do then from the pause state, maybe I can open the door without tripping the emergency stop. Right? Because now I’m in a stopped state there. Maybe I can interrogate something or set something.
00:24:01 And if I hit the UN pause button, I go back into the running state. So it’s a very nice abstraction to be able to deal with complicated States and how to deal with their inputs. And you find quite often in the hardware world, if you were designing a Coke machine, for instance, you put a coin into the coin slot and how it’s dealing with the counter. And eventually the account gets high enough that you could purchase a drink. Well, when you press the button before that gets to that total, it’s not going to let any can come out afterwards. What it’s going to do is it’s going to produce a can and produce your change, then go back into the waiting state for the coin. So you do see it quite often in hardware implementations.
00:24:45 There’s a lot of places, even within software, where you have fairly complicated behavior when you get all the details. But in concept, the concept is not that hard to understand.
00:24:58 Just a few lines of code. That’s right. Whenever you’re dealing with a graphic user interface. A state machine is a very natural thing to do, for instance, because you tend to have modes in these and what buttons are on and off, and what happens when you press things.
00:25:14 Finite steam machines are all over the place. They use very heavily in simulation.
00:25:19 It’s a major concept in computer science, and if you haven’t used it, they’re fun to play with. They’re very easy to do. And there’s some good packages we can talk about in a minute, but it can really simplify your code.
00:25:31 One of the things I love about them is it makes it so you can test these behaviors easily because you need to hook up actions to the events if they’re attached to physical interaction or actually driving hardware. That’s tough to test. But you can replace all these callbacks with something you can test, like a print statement or something else.
00:25:52 Yes.
00:25:52 And then you can test the whole thing really fast.
00:25:55 I think that’s right. It’s also very difficult to understand what’s going on in a complex system at times. So using these state machines to drive entries into your log can be very powerful, because then you can say, okay, what was going on? I see. I was printing and the sensor came and gave me a bad reading and I went into an error and I got caught in a loop and I never came out of my air. So logging these things can be very powerful and you can actually play them back like a recorder if you do it right, too, because you’re right. The state machine, you can attach a lot of things to these actions that are very useful.
00:26:28 Yeah, I forgot about that. Logging directly. Having a log that just tracks the state machine transitions is very helpful.
00:26:35 And we just did another thing that was very useful. This is kind of a silly trick, but I’m really enamored with our log files. What we do is we drop chunks of JSON into log entries. And what that does is it makes it really easy to parse these log files back for tracking things and visualization, because you can pick out those values very easily because they just come out as a dictionary. So what we do is we’ll read back these log files, we’ll look at when events occurred, we’ll plot different aspects of the machine, whether it’s what was going on with the sensor or how often something happened. So, yeah, we use the log files very extensively for kind of reproducing what happened in the run of the machine.
00:27:15 Oh, nice. You talked about some different FSM packages. Do you have a favorite one you use?
00:27:20 Well, we took the FSM package that we use in our machine. We made it open source, so I’ll just call that one. It’s called Pi State.
00:27:28 You can Pip install P-Y-S-T-A-T-E. What’s fun about that one is we actually use kind of an odd feature called coroutines to drive our States. And it’s kind of a fun example for coroutines.
00:27:43 Everyone is probably used to the yield statement for a generator, and what you can kind of do is treat that in reverse. What the generator does is it creates a function that after it yields, it stays in memory with its state, and then when you go back to it, it does a little processing and yields another value. Well, if you think about a state and state machine, it’s really kind of the same thing. It’s kind of its own little processor. It gets an event that says, here’s what I do with the event, I spit something out or I transition to another state. And so it can just sort of explicitly sit there in hibernation until the state machine dispatches the next event to it and then go on its Merry way. So we use the pie state to do that. It’s kind of a neat use of coroutines. One of the nice things about using a coroutine to do this also is we tend to have a lot of persistent connections in our machine. So, for instance, we might have to open up a Port to the bus to talk to another part of the hardware, a serial Port or USB Port. So keeping that open and initialized and just in hibernation means you don’t have to re initialize that every time you get into your state. So that saves you a lot of work.
00:29:00 Okay, cool. Also, check this out. Yeah, there’s been a million different implementations of state machines, so it’s fun to see different ways people solve a similar problem. There’s a lot of stuff going on where you have to kind of learn the framework quirks to get into it, but I think it is worth doing financial state machines, they are definitely a useful thing and definitely used a lot in hardware, the logging, the testing ability, and just the ability to simplify a design. One of the things that’s tough is any sort of generic description of a state machine ends up talking about events and a B, transitions and things and graph theory, and it quickly becomes something that looks way more complicated than it is.
00:29:50 I think that’s absolutely right. If you download Pi state, there’s a very simple example. I think it’s a turnstile. It’s similar to the Coke machine example where you put coins in and it can unlock the turn style really what it is. If you look at the code here, you set up a couple of States, state A and state B, and you say, and you pass it events. An event could be a string, and it could be just that simple. And the state machine code passes that event, the string street into the state. And the state is just a function, right? It’s just a callable there. It takes that string, it says, okay, what do I do with the string. If the string says go to B and I’m in A, my transition to be if I’m in B and it says go to A, I go to A. If it says N, I go to my end state and end the thing. It sounds very complicated. When you look at the code, really, you create these States, you hook them together and register them with this dispatcher and start sending in events. It’s very simple.
00:30:57 Yeah. And that’s one of the things I love about them, is the constructs around it can be easily thoroughly tested. And then the individual little components that you’re using to put in, like your transition functions and stuff, can be tested on their own. And then you can build up this a fairly complex system and be confident that it’s going to work well, because we’ve tested all the pieces.
00:31:22 That’s right. And where it really shines also is it would be lovely if we could design systems that don’t change over time. But where you really get in trouble is when you need to extend systems and adding new States to a state machine is really where it shines because you take your Coke machine and now someone says, okay, well, I want to be able to insert a credit card. Well, to do that, you can do it by just inserting more States into that existing system. And that abstraction can grow if you think about doing it without a state machine, again, your rats nest of if thens and else ifs just grows and grows and grows and gets Uglier and uglier. So it’s really a nice abstraction as these systems evolve over time.
00:32:06 You’ve got this 3D printer stuff. Which part of that are you working on most of the time or recently?
00:32:13 Well, a lot.
00:32:15 One of the things that you’re never done with when you’re doing piece of hardware is trying to improve the reliability and the repeatability and the quality. So I’m always trying to make it print faster. I’m trying to make it have more lines of reliability for the equivalent of paper jams, for instance. How to reduce that more and more? One of the areas I’ve been spending a lot of time, though, lately is actually probably interesting for this podcast is simulating the hardware for testing. So what I’ve been working on is a fake version of the hardware that we can run for a test environment. So I worked on the original GPS project many, many years ago when I was in College and learned the lesson there. That what you did in GPS, you had a vehicle, and that might be an aircraft carrier or a tank or a person. You had a satellite, which was your GPS satellite. And then you had the radio, which was installed in the vehicle, which is actually in those days was a big radio. Now it’s a chip in your phone. And we would simulate all three pieces and you could plug and play the simulation for 1230f the pieces in order for testing. And why do you do that? Well, the answer was renting an aircraft carrier at the time in about 1985 costs $2 million a day. So you’d prefer to do it with simulation and by renting an aircraft carrier and you don’t even know what it takes to go fix a satellite once it’s up in the air, if it’s not right. So simulate it, get it right on the ground, and then send it up was the answer. But we’re doing something similar with the hardware where if I’ve got a particular component, I can write a simulation back into it and then I can test it. One of the nice things for this, in addition to being able to test without having your expensive printer in front of you, is you can simulate hard to replicate errors. That way, if there’s something that happens infrequently, let’s say once every 100 pages, instead of waiting every 100 pages, you can have the simulation, force the air condition and make sure your testing covers the case.
00:34:23 A simulation can run faster than the hardware can.
00:34:26 Yeah.
00:34:26 Are you trying to simulate the delays as well, or are you allowing your simulation to run as fast as I can?
00:34:32 It’s a case by case basis. There’s clearly timing that you have to worry about in cases. One of the nice things about simulation is you can do things much faster. And if you wanted to simulate 100,000 years of forest growth, you could do it in a day of computer time instead of 100,000 years. Similarly, if you wanted to simulate a really large amount of printing, we could do it quickly. But there are also times where the timing is dependent. So there are places in the simulation where you get an event and you say sleep for whatever amount of time, and it just sits there. But there are a lot of time and related conditions in running a printer like this. So there are places where that’s an important thing to do.
00:35:12 How are you simulating this stuff? Do you have, like a fairly cut and dry layer where your API to your hardware is that you can just write some code that pretends to be interacting with the API?
00:35:26 Yeah, that’s exactly right. The first thing is the architecture of the printer itself. There’s a PC, which is the main brains of it, and this PC connects to a couple of different buses depending on what system. But there’s one main bus that all the different microprocessors for running certain modules of the printer hang off. And we send messages down this bus, and there’s a protocol packet structure that we have got a message header, which is the command, and a data packet, and various things for basically check sums or CRCs to make sure that the message wasn’t corrupted.
00:36:08 And so what we can do then we have a driver level. It’s very similar to the driver you’d have for a printer on your computer where you have a standard set of messages, it goes through that driver, and then every printer manufacturer can write how to respond to that particular message for their hardware. Well, we do the same thing internally, so that if we go from module version A to module version B, we can just do it with a small software change that’s actually the versions of it are just in configuration files. One of the nice things, again about Python, you can dynamically load these things at runtime.
00:36:44 And so what I tend to do, what I do is I write a class that pretends it’s a serial portfolio simulating a serial Port. And when you write to the serial Port instead of going over the real Port, what it does is it goes to my layer there and it might just have transmit command and receive command and a buffer in it and the transmit command will do its processing and push something on the buffer.
00:37:13 So we tend to just simulate it with classes that look like these physical entities. We do have this driver level. There’s another trick that we tend to do also though, which is sometimes it’s nice to have a Gui on these things. It’s nice to be able to see visually what the state is, maybe even have buttons where you can force things.
00:37:33 We have an emergency stop button and you want to make sure that you can recover from the emergency stop. Having a button where you can press it to force an emergency stop is kind of nice. So another thing that I often do here is we can create a pipe and the pipe. Then we’ll start a process, which is a Gui process here, and then the GUI is receiving these messages and responding to it. So instead of using that serial Port, it’s the pipe. And the reason we use the pipe is that way it’s non blocking and your GUI doesn’t lock up all tweeting for things because the pipe wakes up.
00:38:08 It basically ends up acting like a queue for your input and output. And so you press a button, the button responds immediately and then it will have the effect when needed.
00:38:18 That’s cool.
00:38:19 Yeah. It’s kind of a nice trick. We do that in our main GUI because nothing is more annoying than you hit the button and it waits 3 seconds before it sees the button click, and particularly something like pause, you’ve hit it three more times and it’s queued them up and it’s Unpaused on you. So we use this pipe trick quite a bit.
00:38:38 That’s something that all hardware software systems have to deal with is event overflows as well.
00:38:43 Yeah, I hope I answered the question on it, I think I meandered a little bit, but we do have a standard driver structure and at that point we can simulate just a few calls underneath it. And what you tend to do in that class that it’s calling, that’s processing this stuff is you keep some state that’s similar to the hardware state.
00:39:01 So what the temperature is on this thing, or whether it’s paused or what it thinks its name is, or it’s a firmware ID, whatever. And so it’s just some fairly simple classes with some state data in them. And what would a pause do? It receives the message. It sets the pause from false to true.
00:39:21 It returns its status back and goes on its Merry way for the rest of the software on the other end, the control software and whatever else that you’re using to control your printers. Are you using the fake hardware as part of the system level tests?
00:39:41 We’re just starting to that’s the goal. It’s a fairly new thing. It’s been on my to do list for a very long time, and the idea is to do that that should allow us to do module level testing, unit testing on the different pieces of it. And do that because our GUI is connected to a pipe. Also, you can script the GUI. Right. Instead of having a person pressing a button, you can send the same messages just from a command line tool, basically, or a testing tool.
00:40:17 So we are starting to do that a little bit. But we’re not going to win any awards for advanced testing strategy.
00:40:27 In my environment, everything that you can do to an instrument also can be done remotely because that’s where our customers are using it. So we have to build it in. It’s a good idea anyway, even if your customers don’t use it. So that it helps with testing to be able to send in the event without having to actually push a button.
00:40:49 Right. And then we have to do a lot of other testing on top of it. We’re manufacturing items, and we have to test the quality of the manufactured item. So we do Metrology, for instance, where we’ll do physical measurements on the part made to see how our tolerances are.
00:41:05 Was the hole that was supposed to be round, actually round if it was supposed to be 2 mm in diameter, was it actually 2 mm in diameter? Was the height of the part. Right. Was the strength the part we do destructive testing where we rip them apart and we test the tensile strength of these things. We’re not done testing when the hardware works either.
00:41:25 Oh, wow. I’m sure. Yeah.
00:41:27 You have to test the actual output.
00:41:29 Right.
00:41:31 People flying in airplanes don’t want them to fall out of the sky. You got to make it work.
00:41:37 There is drift that happens. Like, for instance, in measurement equipment, there is accuracy when it leaves our plant, but it can drift overtime, and we have tolerances for that. I imagine there’s something similar with your stuff to where when you install the hardware in the first place. Yes. We can verify that the whole is exactly this size, give or take some tolerance. Is that something that can be measured in the field over time to make sure that it’s still accurate?
00:42:10 It absolutely can. And there’s a lot of work. First of all, when we start the machine, we’ll home it, and you’ve got various sensors physically on the machine that will be limit switches, so you know where the zero point is. And the maximum point is you can do that. We also have a fair amount of work for doing calibration and testing on it. We will, for instance, run test sheets through and measure the amount of powder that gets deposited on a page. And we’ll actually measure that with a microgram scale.
00:42:40 We’ll look at things under microscopes. We’ll do various things to calibrate and test, and you log this data along with the parts that you know what was going on.
00:42:51 There is quite a bit of drift, and you also have variances in the materials.
00:42:56 If you’re using a powder or all the particles of powder the same size, did your chemical manufacturer change something?
00:43:02 Are all sheets the same? There’s a lot of variants you can have. So over time, one of the things that you tend to do also is you add more and more sensors to these systems so that you can detect more of them in real time and adjust to it. One of the things that we’ve been adding in this latest version of the printer cameras to do inspection. So we’re using a package called Open CD to do various image processing at different points in the process to do both calibration as well as detection of errors.
00:43:35 I imagine also some sort of logging of that, too, so that if somebody finds an error later in the process, they can back up and try to find out where that happened and which parts were susceptible and things like that.
00:43:49 It’s absolutely right.
00:43:51 Traceability is very important. So you want to log all this information, include we can log each layer that comes through, and we also want to be able to know, for instance, which bag of powder or bottle of ink was used or batch of fabric. And if you had a box of fabric that you’d loaded in the machine, what role was it cut from and which batch from the manufacturer it was. So I have a whole set of things that we do from our misinformation systems that’s also in Python for dealing with this stuff.
00:44:22 One of the great things people should play with if they have the barcode scanners, they’re incredibly easy to use. And what they do is they just return strings of text to simulate a keyboard. So you put a barcode scanner on, you scan your bucket of powder that goes to a database record and says, okay, this came from lot A and batch B from vendor X. And if you had a problem, you now know which other ten bottles in your database were made from the same bad batch and to pull them. But you’re absolutely right. When you have a failure, you want to be able to go back and trace through it.
00:44:58 We keep it down to the level also of knowing which part that was produced came from, which print shop in the printer and where it was in that block and everything else, because you get all kinds of funny problems. Maybe in your oven you’ve got a cold spot, and the things in the upper left of the block aren’t getting cooked quite the same temperature profile. So this traceability becomes very important, and you want that audit trail, and it becomes more and more important if you’re prototyping something. It’s not a big deal if you’re manufacturing things, very important.
00:45:36 If you make a part for a ventilator, you really don’t want that ventilator to fail. The results can be catastrophic.
00:45:43 This entire field seems fascinating to be in.
00:45:47 There’s a lot of cool. It’s moving very quickly. It’s growing fast. It’s very friendly.
00:45:52 It’s great fun. And there are a lot of opportunities for folks to learn about or get involved if they want to play with these machines. And I highly encourage that. Manufacturing is not necessarily considered very sexy, but it’s a really interesting area to be in and very important. And it’s absolutely an enormous market.
00:46:12 Just fascinating that there’s, like this crossover between Python and software and fabrics and making things. And it’s pretty cool. I like it.
00:46:23 I’m glad we got your interest. Come over and visit anytime you want.
00:46:27 I’m assuming I can’t buy one of these things from my house. These are probably expensive things.
00:46:32 Yeah. These are not office friendly printers.
00:46:36 Some of the low end spaghetti printers. Right. Can be as low as 2300, $400. Better ones can be $1000 to $3,000. The industrial printers that we’re selling, they’re about a quarter million dollars, and they’re about 25ft long and 6ft tall. They’re not office friendly.
00:46:57 If you’ve got a very large house with a lot of electricity, give me a call. But no, this is really something that you’d have on a manufacturing floor. Some large direct metal printers can get into the in excess of a million dollars.
00:47:15 These are serious pieces of equipment, but there’s a lot on the consumer side and the prosumer side that people can deal with. And there are also a lot available at museums and schools and community centers and makerspaces where you can use someone else’s machine to get involved. You don’t have to make a large investment to play with it.
00:47:35 Yeah, well, this has been a ton of fun. So thanks for coming on the show and sharing all this with us. So your company remind me again, impossibleobjects.
00:47:45 It’s impossibleobjects with an Sburl.com.
00:47:50 Okay. And if people have questions for you, can people reach you?
00:47:54 Sure. My email address is Lwanger, and that’s Lwan at impossibleobjects.com. And yeah, I’d love to hear from folks.
00:48:07 Cool. And I will definitely check out this we’ll leave a link to the state machine that you have worked on and some other stuff.
00:48:18 Cool. Thanks a lot.
00:48:19 Terrific. Thanks, Brian.
00:48:23 Thank you. Lyn that was a fascinating talk. Thank you also to Datadog for sponsoring check them out at Test And Code. Comdatog and thank youcircleci for sponsoring check them out at circle. Ci testcode and thank you to all of the listeners that support the show through Patreon join them by going to testandcode.com support all of those links including links about some of the stuff we talked about in the episode are at testandcode.com 121 wow, 121 episodes. That’s so cool. Thank you for sticking by. I know there’s a lot of great back catalog stuff so check that out and you can follow me on Twitter at Brian ockin B-R-I-A-N-O-K-K-E-N. That’s all for now. Now go out and test something.