Topics covered in this episode: Pandas Should Go Extinct Pydantic-pint puts real-world units in your Pydantic models How Libraries Run Rust Inside Python (With PyO3) AWS acquires DuckLabs Extras Joke Watch on YouTube Sponsored by Logfire from Pydantic: pythonbytes.fm/logfire Connect with the hosts Michael: Mastodon / BlueSky / X / LinkedIn Calvin: Mastodon / BlueSky / X / LinkedIn Show: Mastodon / BlueSky / X Join us on YouTube at pythonbytes.fm/live to be part of the audience. Usually Tuesday at 7am PT. Older video versions available there too. Finally, if you want an artisanal digest of every week of the show notes in email form? Add your name and email to our friends of the show list, we'll never share it. Calvin #1: Pandas Should Go Extinct Pandas' slowness pushes teams toward "Big Data" tools (Spark, Databricks) they don't actually need - most workloads never hit true Big Data scale Amazon Redshift telemetry: ~95% of tables are under 100GB, ~87% of queries touch 80GB or less - that's "Medium Data," not Big Data Polars and DuckDB fill that gap: single-machine, fast, no cluster required 1 Billion Row Challenge benchmark: Pandas took 4m28s vs. Polars 5.04s and DuckDB 5.19s - DuckDB also used 19x less memory On a real-world NYC taxi dataset (3GB parquet), pure DuckDB ran 2x faster than pure Pandas while using a fraction of the RAM Bonus: Apache Arrow lets you pass data between Pandas/Polars/DuckDB with zero copying, so trying them out doesn't mean a full rewrite Michael #2: Pydantic-pint puts real-world units in your Pydantic models Pydantic-pint bridges Pydantic and Pint so models can validate physical quantities like 4m or 12 meters instead of bare floats. Fields annotated with PydanticPintQuantity parse user input, convert between compatible units, and serialize quantities back out as strings. That closes a real gap for anything consuming API payloads, config files, or sensor data with measurements, letting you enforce units at the validation boundary instead of hoping every caller remembered them. via PyCoder's Weekly newsletter Unit mix-ups have literally crashed spacecraft; now your Pydantic models can refuse them at the door. Annotate a field as Annotated[Quantity, PydanticPintQuantity('km')] and inputs like 12 meters arrive auto-converted to kilometers Validation covers string, numeric, and quantity inputs, and model_dump_json serializes quantities as readable unit strings Installable from PyPI as pydantic-pint, MIT licensed, with docs at pydantic-pint.readthedocs.io Early-stage solo project at version 0.4, so API stability and maintenance are open questions worth discussing Calvin #3: How Libraries Run Rust Inside Python (With PyO3) Pydantic v2's validation core (pydantic-core) is Rust under the hood, built with PyO3 - this post shows how that bridge actually works via a small hand-built JSON parser Four steps to get Rust into Python: write a normal Rust module, annotate with PyO3 macros (#[pyfunction], #[pymodule]), compile/install with maturin, then just import it The parser builds a Rust tree first - Python never touches it until the boundary crossing Key insight: converting the Rust result into Python objects (.into_pyobject) is often the expensive part, not the parsing - 100,000 JSON values means ~100,000 Python objects built after parsing's already done Errors cross the boundary too: Rust's typed errors convert into real Python exceptions (ValueError, FileNotFoundError) via From/?, so callers get clean Python semantics Takeaway for anyone porting Rust in: if you're returning a scalar, don't sweat it; if you're returning a big structure, profile the boundary - that's the real cost, not the algorithm Michael #4: AWS acquires DuckLabs Thank you Dylan McConnell. What does this mean for the DuckDB ecosystem? DuckDB is the open-source in-process analytical SQL engine. MIT licensed. The IP is not owned by any company - it's held by the nonprofit DuckDB Foundation, which was created when the team spun out of CWI Amsterdam. Peter Boncz, the CWI representative on the Foundation board, describes it as the entity that holds all IP of open-source DuckDB. DuckLabs (ducklabs.com) is the company, formerly branded DuckDB Labs. Founded a little over five years ago by Hannes Mühleisen and Mark Raasveldt to give the DuckDB team a stable long-term home, bootstrapped deliberately instead of taking VC, grown to 30+ people in Amsterdam, funded by support and feature-prioritization contracts. It employs the core devs. It does not own DuckDB. DuckLake is one of three projects DuckLabs builds, what they call the Duck Stack: DuckDB, DuckLake, and Quack. DuckLake is the lakehouse format that puts catalog metadata in a SQL database instead of in files on object storage. Quack is newer - an RPC-style protocol that turns DuckDB into a client-server system where both ends are DuckDB instances, slated to stabilize in DuckDB v2.0 in September 2026. MotherDuck is a separate Seattle company, Jordan Tigani's, selling serverless hosted DuckDB. It was started in partnership with DuckDB Labs and has worked closely with Hannes and Mark for four years. It contracted DuckLabs for engineering work and contributes heavily upstream - three of its engineers are among the top 10 outside contributors to DuckDB. It also sells its own DuckLake offering. Customer and collaborator, never owner. What the AWS post changes. Amazon bought the company, not the project. DuckLabs joined AWS effective September 1, with the process concluding August 31, 2026. Hannes and Mark keep leading the team and the project's technical direction, the team stays in Amsterdam, and DuckDB stays MIT under the Foundation. AWS gets the people and a direct line to the roadmap. The license protects your code, not your priorities. Three second-order effects worth tracking: The Foundation board is the real question. It has three directors: Mühleisen, Raasveldt, and Boncz. Two now work for AWS. Commentary on the deal has focused on exactly this - the license protects the code, not the roadmap. The announced counterweight is governance: a technical advisory board on the Foundation, and opening the extension stack so extensions signed by other developers can run in DuckDB. MotherDuck immediately moved into the business DuckLabs vacated. It now sells DuckDB enterprise support, which it had avoided because it didn't want to compete with DuckLabs' business model, and says it has explicit blessing from Hannes and Mark now that they're joining Amazon. It also bought Tower.dev the day before the AWS announcement. Everyone expects an AWS DuckDB service. Tigani says Amazon will likely release one eventually, and welcomes the competition, citing Redshift's failure to slow Snowflake on AWS. The groundwork is already visible: Amazon Quick uses DuckDB to query S3 Tables and has processed over 2.5B queries with it since launching in October 2025. The DuckLake angle is the one to watch. AWS is heavily committed to Iceberg through S3 Tables, and it just acquired the team behind a competing lakehouse format. The stated plan is to use DuckDB, DuckLake, and Quack together to power a new generation of data services, but which format wins internal priority is unannounced. Extras Calvin: astral-sh/uv 0.12.12: code-signed release binaries 🥳 Michael: My MacBook power supply rebooted to install updates (?!?) The Story of VS Code | Official Documentary Amazon/AWS acquires DuckLabs (see recent episode on DuckLake) Joke: We’re agentic now
Audio and video play straight from the show's own hosting. PodSubs stores and serves no media.
Transcript
From the show. Tap a time to play from there.
Michael KennedyHello and welcome to Python Bytes, where we deliver Python news and headlines directly to your earbuds. This is episode 496, recorded Tuesday, September 15th. I'm Michael Kennedy.
Michael KennedyThis episode is brought to you by Logfire from Pydantic. If you want observability for your apps and your AI agents, Logfire is the business. I will be telling you more about them later. Find the link at the top of the show notes. Follow us on the socials. All the various things you can think of are there on the episode page as well. And sign up for the newsletter. I just sent out the most recent one a couple days ago. It was a little bit late. Sorry, folks. But really cool stuff that we add, like extra information that doesn't even appear in the show that helps you get a little more out of the show.
Calvin Hendryx-ParkerYep. I love all the context it adds.
Michael KennedyI do, too. I do, too. I'm like, well, that's pretty good. We found some good stuff here. I would say that this newsletter that we're writing here, it shouldn't go extinct.
Calvin Hendryx-ParkerBut some things might need to go extinct. What's going on here? So I found, so this is a blog post from, what's Eddie's last name? Hold on, it's down here at the bottom of his copyright. Eddie Atkinson. He gave a talk at the most recent latency conference. So it's actually a talk from last year, but I think he kind of brought it back a little, evergreened it into a blog post last week about pandas that should go extinct. and we're not talking about the cute little fluffy things that are used for international diplomacy,
Calvin Hendryx-Parkerbut the Python data frame library. Michael, how many times have you thought you had big data, only to find out you were ready to defunstrate your laptop because pandas was the problem?
Michael KennedyYou know what? It's happened.
Calvin Hendryx-ParkerIt's happened.
Michael KennedyI'm going to get my dictionary real quick, and then I'm going to know that that happened.
Calvin Hendryx-ParkerI think the issue is a lot of folks really don't have truly big data problems. I mean, we've done some big data projects in the past, which were 10,000 tables, petabytes of data. That's truly big data. Most folks probably lie in the medium-sized data, but Pandas definitely tops out. I mean, he does some interesting benchmarks in here, gives a couple good code examples, actually shows a really interesting post from Amazon Redshift team, where they were looking at the composition of many of the tables that are out there in the Redshift environment.
Calvin Hendryx-ParkerIf anybody's going to have a good view on what the size of data is and what big data could be, they're probably the ones to look at that. But if you look at this chart, they basically say on a continuum of data size, most folks start over here in Excel. You've got like under a gigabyte, around a gigabyte of data. About that point in time, Excel falls over. It's probably time to pick up another tool to handle that. And a lot of people reach for pandas because I think there's just a lot of built up inertia or momentum in the community around the pandas and data frames and it's an easy
Calvin Hendryx-Parkerui it's been taught in a lot of universities so there's just not a lot of like need to kind of move out of that space because there's a lot of good code examples a lot of blog posts have been produced a lot of data science is based on pandas but they're really based on data frames and there's more than one library out there to handle data frames and probably do it more efficient so if you actually looked at the chart here they're basically saying when you get up into like the 10 10 gigabyte range for data sizes. Pandas is probably still pretty good, but then there's a gap. It falls off somewhere between 10 and 100 gigabytes of data.
Calvin Hendryx-ParkerAnd 100 gigabytes of data these days is not unfathomable. You can easily go find sample data sets that are in that realm, in that range, in that size. And so the next thing they reach for is typically a commercial tool like Databricks, Snowflake, Dask, or some of these other things that are like Spark. So you're distributing the memory of that data set across many machines or maybe even across one very large machine, but doing it in a distributed manner. Most people probably don't need to go that far. Like most people probably are still sitting in the range where you can see on this chart that Polars handles.
Calvin Hendryx-ParkerPolars can handle straight up into 100 gigabytes of data easily on a single machine. And then DuckDB takes it to the next step, which actually kind of fitting for this episode. I think this will be an interesting episode because there's a lot of information here about DuckDB later on in the show. But he kind of goes on again and shows that basically the average size of a row in Redshift is about a kilobyte. Every Redshift cluster has like 10 machines in it. They're capable of guzzling 8 gigabytes per second from S3, but really in actuality,
Calvin Hendryx-Parkeralmost 95% of the tables in Redshift contain fewer than 100 gigabytes of data. Most people are still in the range of just using a single machine with DuckDB or even just Polars, which is probably similar, simpler to maintain and manage, but it's above the reach of pandas. That's why the post is kind of going on about pandas needing to go extinct. Another interesting bit, again, kind of good code examples in here. When we get down into some of the tables for the performance, what strikes you here, Michael, on their memory usage?
Calvin Hendryx-ParkerThe pandas library, we're talking about 30, this is one of those examples, I can't remember which one, but four minutes basically for the duration of the processing, 38 gigabytes of RAM. If you get into pullers, that gets halved, 18 gig of RAM. And if you go into DuckDB to do the same operation, five seconds at 1.93 gigabytes of RAM. So even the-
Michael KennedyIt's like nine to 20 times as much, yeah.
Calvin Hendryx-ParkerYeah, I mean, an iPhone could do this operation against 100 gigabytes of data.
Michael KennedyIt's fine though, 'cause you can just get more memory, memory's cheap these days.
Calvin Hendryx-ParkerMemory's cheap. Totally cheap, totally, totally cheap. So I just think folks need to put this one to bed. Pandas was probably a good way to start, but you'll notice in here, like the Polars notation or syntax, really, really similar. I like the DuckDB syntax. I think he's got some examples in here where he reads in and does some more operations. He does a couple against some larger machines, then falls back into an older laptop, like a framework 13, to show that this is still useful as a developer tool.
Calvin Hendryx-ParkerSo I think here, Pollard versus DuckDB kind of comes down to your workload, your experience, and your preference. It's a good post. I really liked all the code snippets he links over into the GitHub where you can actually try it yourself. It goes against the New York City taxi data set. It's got a ton of data in there, so it's fun to play with. And you can see that we want to wait minutes or do you want to wait seconds? And would you want to use all your memory for this? And actually, another thing that Pollard and DuckDB did much better was utilizing the CPU. PurePandas in this case was using like a multi-core machine,
Calvin Hendryx-Parker146% CPU, where if you go to PureDuckDB, there were over 800% CPU. So obviously eight to 10 cores are being fully utilized as opposed to basically one and one and a half cores.
Michael KennedyYeah, that's awesome. I feel like this is a pretty data heavy episode for the data science crew out there. It wasn't on purpose, but we ended up that way. I have a little bit of a real-time follow-up for you, Calvin, for people who... Yeah,
Calvin Hendryx-ParkerI was gonna ask the one last bit in here. He does mention Apache Arrow, and if you've not played with it, it allowed him to switch back and forth between pandas, Polars, and DuckDB without having to reload or copy the data into RAM. So you could actually do the same operation with each of the libraries without actually having to take the data back out of RAM. So check out Arrow, which is really cool. It's kind of a little bonus side bit that was in the blog post. So data folks who got medium-sized data, This is going to be a godsend for you.
Michael KennedyYeah. I think Arrow is the foundation of pandas too, if I remember correctly, and also a polar. So that's pretty sweet. Yeah. My real-time follow-up here.
Calvin Hendryx-ParkerIt was a real-time follow-up here. Oh.
Michael KennedyIf you were working with one of these and you want to switch to the other, I had Marco Garale on, really on Talk Python a while ago to talk about narwhals. And narwhals is a facade adaptive layer that speaks native polars, but also talks pandas. So if you want to try like, oh, let's see what we're doing. You know, you could use this as a intermediate layer to kind of swap that out a little more easily than rewrite and everything.
Calvin Hendryx-ParkerYeah. I think people just need to drop pandas. I mean, it was great. It was great 10 years ago.
Michael KennedyYeah. Yeah. I have some funny jokes, but let's carry it. Let's move on. Let's move on to Pydantic Pint. So Pydantic Pint is really interesting. Do you know Pint? Are you familiar with Pint? actually i've never i've never used pint so pint we've covered that on the show back in the day and pint is interesting because if you're i mean all you got to do is say mars lander sample return whatever and it's like the 100 million dollar plus fail because somebody used feet and somebody used
Michael Kennedyneighbors or something like that right and so pint lets you do math in python with units attached which is pretty cool right so i can say instead of just having a distance i have 42 i have 42 kilometers and you can say like two miles to whatever and so on so it it basically mean forces you to work in units right a lot of times we don't do this as regular programmers but if you do anything scientific well there you go right so that's the background on pint but what i want to talk about is actually not pint it's called pidantic pint because pidantic is an awesome
Michael Kennedylibrary that lets you validate the inputs and parse them and everything whenever you read some sort of JSON, right? Like if it's FastAPI or just a JSON file or whatever, you need database, SQL model, all those things. Yeah. So Pydantic Pint takes this idea and adds units to your data validation libraries. So instead of saying I have a box that has a length and a width, I could say I have a box that has a length and a width that is a Pint quantity and the validation is to convert it to
Michael Kennedymeters. So even if you parse something that says feet or centimeters or whatever, it will show up correctly. What do you think?
Michael KennedyIt's the kind of thing that's like, you don't have to, you're not going to use it a lot unless you're really in, you know, some kind of engineering or something. Oh my God, this is so good. It's like so perfect.
Calvin Hendryx-ParkerThis has got to solve so many, like you said, small mistakes that end up in huge damage. A hundred percent. So yeah. And to be able to validate with it too.
Michael KennedyYeah. Just automatically, right. Just all the Pydantic validations have a Pydantic base model And it parses over to whatever it is. And if you put, I don't know, liters into the length, well, liters, you can't convert liters to meters. So I don't know.
Calvin Hendryx-ParkerAnd it kind of fits perfectly under the Pydantic scope of the data validation and serialization. Like it just, there's natural, like this should exist and they made it exist.
Michael KennedyYeah, it's really cool. So you can have say a FastAPI endpoint that just automatically just takes units and automatically converts units. And yeah, it's a really nice one there. Speaking of really nice, now this transition here has nothing to do with the sponsorship, the previous one.
Calvin Hendryx-ParkerThey just happen to do great stuff in open source code too.
Michael KennedyBut Pydantic also happens to create Logfire, which I told you about at the beginning. So let me go ahead and tell you about our sponsorship offer, Logfire, not Pydantic Pint, which is not even from them, but it's based on Pydantic. So here's the deal. It's 2 a.m. Your AI agent failed. Was it the model? A tool call? the database, just the general unreliability of, hey, I think a new model is coming. So the current one starts breaking periodically. So most observability tools, they can't tell you because they only see part of your stack.
Michael KennedyPyDentaglogfire sees all of it. One trace across your agents, LMs, APIs, and databases down to the infrastructure, services, Kubernetes, hosts. It's built on open telemetry with SDKs for Python, TypeScript, and Rust. And it works with any OTEL compatible language. Every prompt, token count, and cost right next to your vector searches and API calls. You query everything with Postgres compatible SQL to understand what your app is doing, and so can your coding agent. It can use the same way because it talks SQL really well.
Michael KennedyIf you connect it to the MCP server, your agent can also figure out what is going on. So stop guessing, read the trace, Pydenic, Logfire. AI, it is still just engineering, even if it's weird engineering these days. So visit pythonbytes.fm/logfire today and sign up. Get 10 million records free every month, no credit card required. You can even click, and I really like this, there's a little copy of this text, onboard you with your agent. Click that and it gives you a prompt you can drop in to Claude Code or Codex or whatever and it automatically knows what to do to set up Logfire
Michael Kennedyin your app. So thank you to Pydantic for supporting the show. Calvin, I know you're a big fan of the visibility into the token.
Calvin Hendryx-ParkerYeah. I was just curious now if copy the setup prompt is the new pipe to bash, like pipe some curl to bash. Yes. This is the replacing that. I think it is. That's why I was getting it. Yeah. No,
Michael Kennedyno. I think it is. And it's amazing. I have some stuff that I'm working on. I'm like, oh, this is like, this idea is perfect. I love it so much. So yeah, pretty cool. Thanks to Pydantic for sponsoring the show. Thank you. And let's jump over to your topic next. You've got in the wrong order in the other. So what's next?
Calvin Hendryx-ParkerWell, speaking of Pydantic, this comes from Bob Builderboss, a friend of the show. I know you've had him on numerous times for other events and things, but this one is about how to run, how Rust code becomes something you can import. I think it's interesting that we can, if people are complaining about performance, the first news article I had about getting rid of pandas and bringing back in with polars and DuckTV was about performance. This is similarly veined.
Calvin Hendryx-ParkerLike if I've got a very computationally intense data structure or function that's happening in my program, it'd be sure be nice if I could maybe replace it out with a Rust version of that, but have it act natively inside of my Python code. So this blog post from Bob goes over basically what Pydantic V2 does, which is a data validation library that most Python apps are using these days. It's actually a Rust extension under the covers that does the work. Its core, Pydantic core, is all built with Py03,
Calvin Hendryx-Parkerthe same tool chain we're going to use here in this example. So it goes over some examples of basically you write a normal Rust module, you annotate it with some specific macros, and then you'll actually be able to import that into your Python code fairly naturally. I think there's basically the Rust parser is incredibly fast. Python never touches anything until the boundary crossing. So if you call for some data, that all happens over in Rust. The thing that I think the article covers that's really important is that
Calvin Hendryx-Parkerif you pull back that data across the boundary, those Rust results get turned into Python objects. And so you're going to want to think carefully about how you bring back parts of that, because maybe you're only interested in a small piece of what is coming back and you don't need to populate. Say, for example, 100,000 JSON values means that you're going to get 100,000 Python dictionary objects after the parsing is all done and everything gets passed back. You don't incur the penalty until you cross that threshold back into Python land. Maybe you don't need all 100,000,
Calvin Hendryx-Parkerbut there's some other operation you can do to get down to just the pieces you need. So what's nice is errors cross a boundary too. So if Rust runs into errors, those come back as Python exceptions. So it makes it easy to debug and figure out what's going on. So you get clean Python semantics while still leveraging Rust. So basically for anyone porting Rust, if you're returning a scalar, don't sweat it. If you're returning a big structure, profile the boundary. And if that's a real cost, you want to switch over and maybe do more of the algorithm on the Rust side.
Calvin Hendryx-ParkerSo much like you can use C or other languages in, I don't know if you can, can you use Ruby to do this kind of thing? I don't know if you can or not, but you definitely use Rust. And I'm kind of excited about that. I know you've been doing some coursework on it. And it sounds like Bob has also made some learning materials to lead folks through. It just feels like there's a really nice friendship between the Rust communities and the Python communities and all the niceties that have been put in place to allow us to use Rust almost natively
Calvin Hendryx-Parkerover in the Python world. So thanks, Bob, for the awesome post about that. He's, again, code examples in here, kind of explains to the Python folks who have never touched Rust what the function signatures look like, which I appreciate because breaking it down and telling me what each of those pieces means that I can now pretty easily read some Rust code and understand what's going on because it doesn't look terribly foreign to me, but it's just different enough, but this goes over a good usage of what each of those pieces mean for you.
Michael KennedyYeah, it's pretty surprisingly similar to Python, honestly. Yeah,
Calvin Hendryx-Parkeryeah, and incredibly fast, but you get to think about things a little differently because of the way it manages memory, and I think that's the big differentiator.
Michael KennedyWhat was that cartoon with the guys? Like, I'd gladly pay you on Tuesday. It was Popeye. Popeye, that was Popeye, right? Yeah, it was. I mean, I was thinking borrow checkers.
Calvin Hendryx-ParkerIt was Wimpy, who will glad you pay you Tuesday for a hamburger today. Yeah,
Michael Kennedythat's the difference of Rust is you've got the borrow checker. Always checking. Yep, yep. Always checking.
Calvin Hendryx-ParkerSo yeah, I don't know. You've been doing a little more with Rust and Python and teaching some folks these things. Yeah,
Michael Kennedyyeah, a little bit, a little bit. I have two follow-ups here. So you talked about DuckDB, but the question is, do you have a lake house? Do you have a lake? I mean, a lake house. So we've heard of data lakes, which is a place you just kind of dump a ton of like an insane. This is like back to your big data thing, DuckDB thing. You just dump a bunch of data into this data lake and you figure it out. Well, that's grown up a little bit. And now there's this thing called DuckDB,
Michael Kennedybut it's an implementation, an example of what's called an open lake format.
Calvin Hendryx-ParkerWho knew?
Michael KennedyDo you know?
Calvin Hendryx-ParkerI didn't know that. We've done lake house implementations. I didn't know there was an open lake format now.
Michael KennedySo the story is, what if we could use S3 to scale our data access? S3 scale is pretty large. If you can read stuff off the file system instead of out of memory, you can scale that tremendously large. In the open lake story is, well, if you put file formats in S3 that everything could read, like maybe JSON files that tell you what the files mean, you could read them first. Here's where the data lives in each piece. And then parquet files. Yeah. Or zipped CSV. don't know take your pick right it could be whatever so i just had the folks from duck lake on which is
Michael Kennedya duck db implementation story of uh this open lake format well then i get this message here saying yep guess what aws i'm sure you know this is a i did i did see this one come by yeah yeah and in duck labs just offered basically the h1 is bad let me read let me read the first sentence today we are announcing that Amazon has signed an agreement to acquire Duck Labs, the Amsterdam based company behind the open source analytical database DuckDB.
Michael KennedyI'll put a link to the announcement and I thought, well, hmm, what does this even mean? And I didn't, I wasn't entirely sure. So I, I like, I went and did some looking here. I'm like, there's, there's actually a lot of pieces in play. So let me lay it out and I'll tell you what part AWS acquired, what part didn't. Okay. So first of Well, thanks to Dylan McConnell who sent this in. What does this mean for the DuckDB ecosystem? So first of all, DuckDB, which you gave a shout out before, twice really, is the open source in-pros analytical SQL engine,
Michael KennedyMIT licensed. It's like SQLite, but for columnar data, which if you're And
Calvin Hendryx-Parkerway more, and way, way more. Yeah. Anything you pointed at becomes SQL queryable. It's amazing.
Michael KennedyYeah, yeah, yeah. So you can say pointed at a pandas data frame and then do SQL queries against your Panda. Like there's a bunch of plugins. It's far beyond just a database, but it's an in-process sort of data processing engine, much like that. So the IP of this is not owned by any company. It's held in a DuckDB foundation.
Calvin Hendryx-ParkerOh, good. Thank goodness.
Michael KennedyThat sounds good. That is good. It was spun out of CWI Amsterdam from the folks who mentioned that article. Good, but we'll come back to it. Then there's Duck Labs. And the story is Amazon AWS has acquired Duck Labs. This is the company formerly branded DuckDB Labs, founded over five years ago by Hannes Mullison and Mark Roosevelt to give DuckDB a stable home, bootstrapped, grew to 30 people. Now they can go chill on their island,
Michael Kennedywhich congrats to them. That's awesome. Because DuckDB really has taken over, right? Then there's DuckLake, which I mentioned earlier. It's one of three projects by DuckLab. And there's DuckDB, there's DuckLake. And then there's an API for working with this called Quack. Check out the Talk Python episode. But it's an open lake, lake house format.
Calvin Hendryx-ParkerAll the waterfowl puns are great.
Michael KennedyIt is. And we actually on the podcast had a fun conversation about like, do you need a more serious name? Like RPC for your data lake. Like, no, they're like, we're calling it Quack. Come on now. And then also we have MotherDuck, which I think I thought Mother Duck was online version of DuckDB. But no, that is a separate Seattle company selling serverless hosted DuckDB. And originally it was started in partnership with DuckDB Labs and they worked closely with Hannes and Mark for years and even contracted Duck Labs for some of the engineering.
Michael KennedySo now, what does this all mean? So the foundation owning DuckDB is awesome, but it has three directors, the people who own Duck Labs. So there's a bit of a, how much independence is it really going to have? There are some other folks, other governance and so on there. But, you know, it's cool. There's a foundation. It's not super independent of DuckDB at the moment. Maybe it will be, though, after this. Mother Duck immediately moved into the business that Duck Labs vacated. They now sell enterprise support for DuckDB and so on.
Michael KennedyEveryone expects an AWS DuckDB service, probably a Duck Lake as well. already used s3 right but maybe just a little more formal yeah there's s3 query and some adjacent
Calvin Hendryx-Parkerlike technologies that sound like this may maybe this will augment or replace what's really nice
Michael Kennedyabout ducklapse is it runs a local duckdb or a local postgres server and a lot of the chatty api that would come from an open table format and the metadata now all happen in the database and then it just fetches and reads the files yeah that's pretty cool yeah we can't beat physics if we can
Calvin Hendryx-Parkerkeep the data, the bits where they're at physically and bring the compute to it. That's the win. Yeah.
Michael KennedySo the duck lake angle actually is probably the most interesting one because AWS heavily committed to iceberg through S3 tables, which is a competitor, at least a competitor competing concept to a duck lake. So yeah, check it out. I think AWS just got better. We'll see what that means for the rest of the world. What do you think? I mean, you're on the inside of this a little bit.
Calvin Hendryx-ParkerYeah, a little bit, but you know, they've, we've had mixed reviews on their handling of open source, but luckily they don't have any control over the open source other than they've just bought out the founders who are on the board of the open source foundation. It's still separate enough of an entity. I don't think there's a conflict here. I mean, it's going to be good for the project. The project is already incredible. Like the DuckDB stuff is, it's just like, if someone had thought about SQLite and said, I need to grow all these other features that handle all kinds of crazy data and do give me native like JSON access and functions. And it's a really great platform for building cool little utilities or talking to giant chunks of data,
Calvin Hendryx-Parkeras we saw in the first article. Yeah, yeah, yeah, absolutely. And use barely any memory. I mean, that's why this will run and work.
Michael KennedyYeah, the DuckDB part is really interesting on that. And then the Duck Lake is like insane. You know, you could have terabytes of Parquet files all broken in little bits. No big deal. No big deal. NBD, NBD. Yeah, I love it. Well, how about some extras?
Calvin Hendryx-ParkerWell, I will continue on my beating the uv drum. The latest release, every week we got something new, the latest release from the uv folks, we get code signing on Mac and Windows. So the binaries are now officially signed, code signed. Again, I think this is all coming together, ensuring we can secure the software supply chain part of this. So I'm excited to see that release too. I was hoping I wouldn't see a uv thing this week, but sure enough, it popped up in my feed.
Calvin Hendryx-ParkerAnd I was like, I have to mention it because they just keep making everything better and better and better. So uv is now code signed. So you can trust that it came from the right source on your own machine if you're on Mac and Linux.
Michael KennedyYeah, that's excellent. Excellent, excellent. Yeah. I see it's still going. Oh my gosh, the code signing is such a pain these days. It used to be you could just build an EXE or a.app and you could just go here, try my app.
Calvin Hendryx-ParkerHow dangerous is that?
Michael KennedyI know. That's how the world used to be, though.
Calvin Hendryx-ParkerWell, we used to have, what, R login with no password to remote machines? What could go wrong? It's fine.
Michael KennedyIt's fine. Trust. You've got to have a lot of trust. Why would somebody do something mean to computers?
Calvin Hendryx-ParkerI don't know.
Michael KennedyI don't know. I remember in Windows 95, we had a bunch of them at a university I worked at. Plugged straight in to Ethernet. And the Ethernet, everyone got its own IP address. And guess what? That thing got taken over pretty quickly.
Calvin Hendryx-ParkerThe university I was at, they all had public IP addresses in the labs too.
Michael KennedyYeah, it didn't go well. No. It did not go well. Speaking of things that need to be patched and updated, check this out. So there's two things that involve restarts. But I have a MacBook Pro, M5 Pro. Very nice. Love it. I got it this summer or earlier, maybe spring. I don't know, whenever I got it. And it came with a power supply. One of those power bricks. the power brick had to reboot the other day to to update itself i was sitting there working and i saw this article come up come by say apple releases a firmware update for the 140 watt usbc power
Michael Kennedyadapter which is the one that runs the macbook and it's almost that that's almost actual size right there it's so big yeah it's it's a beefy boy it is uh yeah i think this is even a little small this big old picture of it it's it's it's heavy but i was i read this article and i was sitting there working and my Mac, when it comes off of power, it dims the monitor.
Calvin Hendryx-ParkerYeah.
Michael KennedySo I'm, I'm just, you know, and I was an hour or two later, I was sitting there and everything goes dim for a second. The little power just disconnects. And then two, two, three, five seconds later, something like that, power comes back, brightness comes back. I'm like, I just, my power break just rebooted. What in the world is going on here?
Calvin Hendryx-ParkerThat's crazy. I've, I've, I've replaced all my Mac power books. I've got, not that I'm trying to be an ad for Anchor or anything like that, but the Anchor Prime has 160 watt, like itty bitty little power brick that, because I travel quite a bit, it has four USB-C ports on it and they can all deliver a combined sum of 160 watts. So I can full bore charge my MacBook Pro M4 Max and my iPad and my phone all at the same time. That's beautiful.
Calvin Hendryx-ParkerAnd it's smaller than that brick.
Michael KennedyYeah, I'm also a fan of the Anchor stuff. This one, since I had it anyway, I just plugged it into the wall part of the house. And just if I'm in that part of the house, I just grabbed that cord. But yeah, normally if I travel, I have an Anker that's actually a power brick, a little battery. And it has two USB things and it'll do not quite as high as yours, but pretty high. And it's super nice because it's also a power brick, right? So if I need to charge up, it'll even charge the MacBook, but then just get it and just plug it into the wall. And then it just becomes a power thing. I'm also a fan of this Anker stuff. Okay, a couple more extras really quick here.
Michael Kennedywe've got do not play yeah so the story of vs code the official documentary is out have you watched this no i have not watched this it's an hour and 38 minutes and i'm here for it okay all right it's got a lot of people that uh maybe you didn't see coming like Eric gamma for example you know thinking back to the gang of four patterns and all that kind of stuff because he was apparently involved in the early days um so yeah cool we had uh cult repo do the python documentary we had them do the jet brains document or intellij documentary and here's the vs code one
Michael Kennedyi'm really loving these like high quality production i mean these are these are nice little nice videos they yeah a lot of people behind it um i mean there's such there's an
Calvin Hendryx-Parkeraudience for all these things i'm here for it too i love the fact that like the underdogs can feel they are important for us and we can now hear more of the story about how some of these things came
Michael Kennedyabout yeah it's really interesting i mean vs code has taken over so much and then yeah anyway and the origins are way way more uh less ambitious let's say it's cool to check out so it also has over half a million views so there is an audience for this apparently there's absolutely audience okay think speaking of things that make you reboot yesterday last night yesterday mac os golden gate ios golden gate watch os 27 golden gate all those things came out so uh-oh did you upgrade i I did. Why wouldn't I mean, I'm like, oh, let's go. I'm not yet upgraded on this.
Calvin Hendryx-ParkerI'm usually of that opinion, but lately I may wait a month or until a.1 or.1 to come out.
Michael KennedyI spent one day working with it and so far it's okay. Okay.
Calvin Hendryx-ParkerI'm going to upgrade then on your full recommendation.
Michael KennedyWell, I've not upgraded my MacBook or my streaming computer. I only recorded my main desktop. So we'll see. Ask me next week. Ask me how I feel about it. You're right. Honestly, the one thing to be a little careful about as developers is the Rosetta 2. Yeah. That's going away. So your ability to run Intel compiled stuff, you might think, Michael, why would I run Intel compiled stuff? Like, ooh, Docker. Certain Docker things only have Intel versions. So that's going to be a mega hassle. I mean, that's pretty rare.
Calvin Hendryx-ParkerPeople have cross-compiled most of this stuff. Because when was the last time an Intel Mac was released?
Michael KennedyYeah, but let's suppose I'm deploying to an x86 server. Yeah. And I want to test something. I think it's gotten a lot better. It definitely has gotten better. But it used to be, the thing is, I don't know. It used to be certain stuff would only work in an x86 version of Linux. Oh, I remember this.
Calvin Hendryx-ParkerYeah. But that was like three, four, five years ago when I was really dealing with that. Actually, it was when I was dealing with like Databricks and trying to coordinate that stuff. Nice.
Michael KennedySo this one still has it, but the one after it, whatever that's called, won't. So this is like your last safe upgrade if you're worried about the other thing.
Calvin Hendryx-ParkerSo did Apple actually deliver some AI features this time?
Michael KennedyWell, I'll tell you what. the new Siri caught me off guard. I'm like, oh yeah, I did. I did actually upgrade the phone. And I guess it has the new Siri because it sounded, I asked it something like set a timer and it said something completely different that I'm used to. And it sounded better. I'm like, oh wait, I haven't had a chance to test it though. All right. It did set the timer like a champ. Let me tell you.
Calvin Hendryx-ParkerWell done. Way to go.
Michael KennedyAll right. Let's talk a joke. Okay. Speaking of, you know, the new Siri is supposed to be agentic. So the joke is we're agentic now. We're an agentic startup. You ready? This is how you, there's certain things you've got to position yourself. I was just watching an ad because I started watching football yesterday. And normally ads are excluded from my life, but apparently not in football, American football. And there's some ad for Zoom that Zoom is an AI company. They're not about meetings anymore. Nope. They can book the thing so that your dry cleaning gets picked up.
Michael KennedyThey can do a slideshow. Like what? Okay. So everyone's got to be some kind of AI thing now. So here's the joke. i changed all of our loading dot dot dot states to thinking dot dot dot we're an agentic startup now perfect i'm gonna get right on that yeah get right in there like you can there's so much vc money to be had from this go for it discombobulating i'm thinking oh oh i hate that so much about Claude Code it drives me crazy that it's got all these random little words yeah the reason i don't
Michael Kennedylike it is I don't, it feels like it's made for someone with ADHD who just can't possibly let it just be for like five seconds. And so if I'm like doing something else, I'd look over and like word starts. I'm like, Oh, maybe it's no, it's not done. It's like, Oh, maybe it's done. Oh no, no. It's just still like randomly. Like, could it just have the little icon go? No, no, no, no. It's thinking is combobulating. It's wording. I don't know. What is it doing?
Calvin Hendryx-ParkerThat's why you need her to exactly.
Michael KennedyThat's, that's, that's the story for another episode.
Calvin Hendryx-ParkerAll right. Sounds good. All right. Well, thanks as always for being here, Calvin. And thank you everyone for listening. We'll talk to you soon. Yeah. Bye.
Topics covered in this episode: tinyjobs: a SQLite job queue goes free-threaded funkycache: cache Python functions in SQLite Ghostman - a keyboard-driven terminal API client diskcache-oxide: A shared, durable disk cache...
Topics covered in this episode: PyPI's redesigned project pages now warn when provenance changes gilguard catches the import that turns the GIL back on pip 26.3: pylock extras, and pip says plainly that untrusted wheels...
Topics covered in this episode: Your GitHub language bar is lying (how to fix it) Python Typing Survey 2026: developers mix type checkers, and AI is writing the hints Flask Vibe - Build Apps You Can Understand Python on...
Topics covered in this episode: PHP wasn’t the only problem last week chameleon-rs Bringing uv and ty together Stlite runs Streamlit apps entirely in the browser Extras Joke Watch on YouTube About the show Connect with...
Topics covered in this episode: Nod: pause your Python program and open a notebook at any line Polars 2.0 ships: streaming engine by default, spill-to-disk, and a row-order gotcha django-cachalot caches your ORM queries...