Nik and Michael are joined by Qian Li and Peter Kraft to talk about DBOS, an open source platform for durable workflows built on Postgres. Here are some links to things they mentioned: Qian Li https://postgres.fm/people/qian-li Peter Kraft https://postgres.fm/people/peter-kraft DBOS https://www.dbos.dev Temporal https://temporal.io Gadget's use of Postgres https://postgres.fm/episodes/gadgets-use-of-postgres Postgres LISTEN/NOTIFY blog post from Recall AI https://www.recall.ai/blog/postgres-listen-notify-does-not-scale Postgres 19 commit that improves LISTEN/NOTIFY, avoiding waking backends that have no need to process the notification messages https://git.postgresql.org/gitweb/?p=postgresql.git;a=commitdiff;h=282b1cde9 PgQue https://github.com/NikolayS/pgque pg_durable https://github.com/microsoft/pg_durable Absurd https://github.com/earendil-works/absurd LISTEN/NOTIFY performance considerations (docs patch submitted by Nik) https://www.postgresql.org/message-id/flat/CAM527d8oDVb2K%2BtTjpjLkd7g14HLDiwwnvv%3Drcn4BMEs%3DMatkQ%40mail.gmail.com#1b6fa752a96aad22bd8ef794fbaf3ea1 ~~~ What did you like or not like? What should we discuss next time? Let us know via a YouTube comment, on social media, or by commenting on our Google doc! ~~~ Postgres FM is produced by: Michael Christofides, founder of pgMustard Nikolay Samokhvalov, founder of Postgres.ai With credit to: Jessie Draws for the elephant artwork
Audio and video play straight from the show's own hosting. PodSubs stores and serves no media.
Transcript
From the show. Tap a time to play from there.
Michael: Hello and welcome to Postgres.FM, a weekly show about all things PostgreSQL. I am Michael, founder of pgMustard. I'm joined as always by Nik, founder of Postgres.ai. Hey, Nik Nik: Hi, Michael Michael: And we have two special guests with us today. We have Qian, who is CEO and co-founder of DBOS, and Peter, CTO and co-founder of DBOS. Welcome both Qian: Thank you Michael: I wonder if one of you could give us a bit of the backstory of DBOS.
I think it's quite a cool story Qian: Yeah. So we started DBOS as a research project. I think it was started quite in the middle of pandemic in summer 2020, when Professor Mike Stonebraker heard a talk from Matei Zaharia, the profes- our advisors and also co-founder of Databricks, where they were talking about how Databricks were u- was using Postgres to do some scheduling work because, distributed scheduling and cluster management was a mess.
And they were talking about how to use a database to simplify that. And then I think during that summer we talked and discussed, like, how we can u- extend that idea to make database more helpful to build, reliable systems to even replace the operating system. So that's how we started the original research project called DBOS. And then Peter and I were the co-leads PhD students working on that research project. We published several papers, did a bunch of work.
And then what we found was that, the most interesting thing was actually to use the database as a foundation for distributed systems, to build distributed systems. And that's why when we both graduated in 2023 we decided to do, to commercialize this idea, to do a startup. And now we are working on DBOS Inc., the company. Nik: that's cool And I must admit I observed queues in Postgres for so long, like 20 plus years.
And somehow when DBOS DBOS was announced, of course it it attracted my attention, but I didn't quite get it like Because like w-w-workflow, like what i-what is it, right? But then I started to realize, actually, to realize that in any business there are workflows, and to keep the state and all the history and so on in database, it's very natural. So this is a very good idea. And I think it's, it's-- it revives the idea that queue in that database
is not only possible, but it's possible in quite efficient way. But you need to… With Postgres, you need to be very careful, right? Because of some, some things that might happen if you implement queues and workflows in naive way. But I don't quite get it still why operational system. I understand when Andrei Karpaty talks about like AI as a CPU in the center, like memory, like every p-peripheral devices, everything. I understand this analogy, but here I don't quite get the analogy why
is there is some intent or idea to implement whole works operational system in, in da-database. Like what's your view? Peter: Yeah. So what we're doing as a company is using the database as a foundation for distributed workflows, and then for things around the distributed workflows like queues, like messaging, like notifications, and so on. I think the core idea here is that as your program runs, you regularly
checkpoint the state of that program in a Postgres database so that if the program ever fails, if anything happens to it, if it crashes, if it's interrupted, it can reload those checkpoints to recover from exactly where it left off. And that, durable workflows idea backed by Postgres is the core idea of DBOS, using Postgres as an orchestrator to make durable workflows that run really w- that run with high performance and that are really simple because all they have is, all they have, all they rely on is your database
Nik: Yeah, this makes sense. I understand that. I'm just still processing the idea. It's like whole operational system. Qian: I think our goal is that when we think about operating system, we should not be limited to the operating system we're running our, on our laptop or desktop. Thinking like anything operational, I think Kubernetes can be called as a distributed operating system. Thinking about like anything that can run your application as like a meta name for like operating system.
So I think nowadays, especially with AI, operating system is like cool again. Everyone is calling themselves an AI operating system or, like something AI-powered operating system for engineers, for sales, for whatsoever. So I think operating system now, like the idea is more about how to properly run your applications in any environment, like in a distributed environment, in a highly available environment. It's about how to provide the guarantee and provide kind of an
illusion that you are ru- like normal, like modern developers no longer directly write or interact with low-level operating system primitives. They usually interact with some- something like Kubernetes or cloud resources. So like you can think of the cloud as an entire operating system. So like I think the name like DBOS right now really focuses on how to operate those, applications at scale and reliably. So it's I would say it's eventually at a grand vision, we hope everything
will be backed by a database, Nik: Yeah. Qian: then that will be like a super operating system. Yeah. Nik: Yeah this makes sense in, in broader meaning operating system. Yeah. Not just like Linux or Windows or something, Right. I get this. And also like there are even broader systems like there, there are companies like which are building like an operational-- operating system for businesses in broader meaning, right? I also remember when Anthropic released I think Ultracode with workflows.
I remember on Hacker News top both topics were present. One was about their workflows, another was about some blog post from your blog, actually. I remember. It was like-- And for me, it was like this is two different things. And I remember internal discussion in our team actually, that we use these workflows in Claude Code, but workflows which are like saving some state in database is something different. And then I remember an aha moment when we realized we actually have agentic
workflows ins-internally already working, and we save state in Postgres. So I realized it's very close topics. I think you also use this resonating moment when they released it and you like to discuss workflows in database, right? So I have one specific technical question though because Peter, you mentioned that if something crashes, right? And ch- this checkpoint and database were all good, like we ha- we have stayed safe. I don't like fully understand why checkpoint is needed because we always
can replay more after checkpoint. There are WALs and so on, right? But anyway, we Michael: checkpoint Nik: Ah, okay, a checkpoint there. But what wonders me, if, for example, we se-send an email but we don't know the result of that yet, right? I'm like moving us slowly to the topic of two-phase commit or something, right? How do you handle that? Peter: Yeah. So if your step is taking an action in an external system and it fails in the middle of the step, then we're going to have to retry that step, and
that, that's unavoidable in any kind of workflow or durability system. So our recommendation then is for individual steps that take-- have side effects, external systems to make those steps idempotent. And, we provide help for that. Every workflow has a unique ID that you can use as an idempotency key in systems that support that. Many email systems do support idempotency keys. Some problems are fundamental to distributed systems. If you're taking a side effect somewhere else and it fails and you don't know whether it succeeded, you're gonna have to retry it. And so you should really try to make sure that, that action's idempotent
Michael: And another thing I didn't fully appreciate till recently, I kept focusing on the word durable I see durable workflows mentioned quite often. I'm thinking about reliability and retries, and that all makes sense. The thing I didn't really appreciate was actions that quite-- could be quite costly. If you've got a five-step workflow and you've spent a lot of tokens or a lot of money, or you've sent physical goods or something like that in steps one, two, and three, and it fails at step four, you can avoid the c- the cost of redoing steps one, two, and three.
Like it, the-- that hadn't really clicked for me, that sometimes it's about other side effects, like might just be cost or it might be safety or it might be something else that you don't have to risk re-replaying is that a big part of it at the moment in terms of… or are there parts of that might be helpful to explain? Qian: Yeah, definitely. I think especially nowadays, everyone's building AI agents and those new models are becom- becoming more and more expensive. So if you have already asked the model about something, you want to
use that output as much as possible. And if something crashes, you want to resume from where it left off. Say, like you already asked the model what to do next, then you should continue from that step instead of asking the models again. Because otherwise you have to burn all the tokens and this will quickly become too costly if you're having like processing thousands of documents and then you want to do it in a more efficient way. That's where DBOS is used a lot, like building agents and building AI-powered pipelines.
I think cost saving is definitely… It's like becoming more and more important, I'd say. Yeah. Michael: Makes sense. Interesting Qian: And also compute resources, right? Even if you don't have to pay those, Even for some people who will self-host their GPUs and - self-host their model inference, then you don't want to burn all the compute resources again. It's all about efficiency. I think yeah, durability itself sounds like not, doesn't sound like a very useful word, but it indicates a lot about cost savings and also
in- indicates about after you've done something, you don't repeat it again. So it's indicates about correctness as well. Yeah Nik: I have a question about why should it be inside Postgres? We know there is-- there are systems which do the same outside, right? Temporal is one of the most popular systems. I even from my personal experience, I even had a moment when one of our clients reimplemented our Postgres major upgrade workflow in Temporal. it was, It was Gadget Michael: They spoke about it on the episode.
Nik: Yeah, we had an episode about that, and it was like we always used Ansible and so on. Of course it's very different, right? When you have workflow, you save everything, it's great. But you move further, let's bring the whole thing inside Postgres. What are pros and cons and, why do you think this is a winning strategy? Peter: Yeah. So I'd, what I'd ask is why not Postgres? The temporal model is that you are taking… is that you're outsourcing the control flow of your program to an external orchestrator, and that's
a lot of additional infrastructure. You can either self-host it, in which case to run your workflows, you have to spin up highly available temporal cluster, a database for it, which is Cassan- and its database is Cassandra and Elasticsearch. And now you're spinning up three new distributed systems to run your workflows, and all three of those systems are totally on your critical path. If anything happens to them, Nothing will work. Your workflows will not function. Your business grinds to a halt. So you're putting three complex distributed systems on your critical path. Or - you use their cloud service. Now you're putting an external SaaS on your critical path.
You don't need to do that. You just need a database. And what we're trying to do in DBOS is totally replace all of those distributed systems, all those external orchestrators with your Postgres database so that you can use your database as an orchestrator. It's already on your critical path. It's already something that you're making sure is highly available. It's already critical for your application to work. Why not also use it as your workflow orchestrator so that you have less infrastructure, so that your operations are simpler, so that there are fewer things that can break and bring down everything you have
Nik: Two things here. One is who will-- like Postgres doesn't have scheduler, right? You can use pg_cron or something, or you need to still need external help to tick, right? To move forward. And another thing is that if it's all running on the primary and primary CPU, first of all, is not infinite, right? So you need to avoid heavy things and have them outside still, right? Peter: Yeah, and that's the reason Postgres is
the orchestrator. The actual business logic of your workflows is still running in your application. The scheduler is running in your application, for example, and then Postgres is the orchestrator. So your applications are running a scheduler, and your applications will enqueue when the sched- when the cron fires, your, one of your application servers will enqueue a scheduled workflow into Postgres. And then Postgres there does orchestration. It coordinates multiple servers to make sure that only one of those servers actually executes the scheduled workflow when its cron tick fires. So your applications your regular servers are doing the work, and
Postgres is orchestrating them to make sure that every, all the work is being done totally reliably, that everything happens exactly once. Nik: Makes sense. So this is when you say critical path, you don't include things that like sending email can be done from multiple servers and they are replaceable like stateless and so on. Yeah, I get that. It feels slightly awkward for me to try to criticize the idea. I'm a big fan of myself, like maybe Michael should join
and find some other questions Michael: I was actually gonna ask a question that probably goes even more positive. M- or maybe not, but I would expect there to be performance benefits to using your already h- used database for this rather than an external service. So I was only gonna say What are, what are the performance trade-offs and what do you do to minimize the impact you're having on primary CPU and that kind of thing? Peter: Yeah, absolutely. So latency in particular gets a lot better because now the latency of executing
a step is just one write to your local Postgres as opposed to a async dispatch from somebody else's cloud service. So definitely latency improves. Now throughput then becomes database bounded. You can process as many workflows per second as, as your Postgres allows if you want. And if you need to scale beyond that, you need to shard across multiple Postgres databases. The good news is that these operations are fairly light. So our rule of thumb is that four Postgres cores can handle 1,000 actions per second.
So for a large database, that means you can vertically scale for a large Postgres database up to… we've benchmarked around 40,000 actions per second on a single Postgres database. And then of course, you need more than that, you can shard or you can use multiple Postgres databases. So your throughput does become bounded by your database, but the good news is that your database can handle a lot of throughput Nik: Yeah. Yeah. Michael: and we're talking… so let's say we've got a workflow with six steps. We've got one write at the beginning, one write per step, and one write at the end.
So we're still only talking eight writes total for a si- and this workflow's having to do things in the meantime, it's not gonna be running through them instantly. So yeah, great. Okay, cool. Peter: And you can handle 40,000 of those actions per second on one database Nik: Yeah, I'm curious. I researched a little bit and I noticed y-that you started with repeatable read isolation l-level and then reduced it to default read committed, right? I'm very curious what was behind that. Is it the throughput question or anything else?
Peter: So we use isolation levels there's a lot of different parts of our system that use isolation levels for different reason. we've designed the system so that almost everything is a single statement. So it can just be a, it can just execute a recommitted, and that makes things far more efficient. As many things as possible execute as a single Postgres statement, so they can just be atomic and you don't have to worry about isolation. You set it to default recommitted. The one thing that we can't do that for is the main DQ query. this is probably the single most important s- query in the entire system,
and it's the query that every worker runs periodically on every queue to dequeue the latest work off the queue And that's a very complex query we've put a ton of effort in optimizing. That query has a, actually has a conditional isolation level, s- depending on what, how you've configured your queue. So if your queue uses global flow control primitives, if your queue, for example, has a global concurrency limit or a global rate limit, so you say, for example,
that this queue can only execute one operation at a time across every worker, then that requires global coordination, which means we have to use a high isolation level like repeatable read. And that has coordination costs, but if you have those global limits, you're probably fine paying the cost because you're only executing one workflow at a time across all your servers. th- your throughput was limited anyways in practice though, what we see is that high throughput queues usually only use local limits.
They might say only execute 10 workflows at a time per worker. And those local limits don't require global coordination. And that means that it's perfectly safe to run the queues query at a very low isolation level at at read committed. And that way you can have this query running across hundreds of servers at the same time, or thousands of servers at the same time without them conflicting because they're all running read committed. And that lets you get this thirty, forty thousand dequeues, dequeued workflows per second, actions per second throughput.
And it really caps out at thirty thousand dequeues per second on a single Postgres across thousands of workers because we've optimized this query Nik: And in case of sharding solution re-repeat-repeatable read wouldn't be available anyway, right? So because like global limits it's… Yeah. You Peter: But in practice, global limits tend to go with relatively low throughputs. Nik: And it's flexible now, right? So u-u-user can control it and decide what, what's in. That's great. And I also noticed listen to file was recently added, right?
Or not recently, maybe I'm wrong. Qian: We've been having listen notify, but recently we've been optimizing it Nik: yeah. We ha-- Yeah, I think it was last year, Recall AI they discovered this awful global lock on commit for Notifier, right? What's your deal with this right now? Peter: Listen/notify is really powerful but has multiple performance issues. There was that issue that… I forget. There was that issue that reached top of Hacker News last year. That was actually not the issue we ran into.
That issue was solved in Postgres 19. That is not the issue we ran into. that issue is that notify had terrible perform- notify's broadcast mode had terrible performance for large numbers of sparse readers. Uh, that's actually an unrelated issue. We do not have that. We have dense readers. We don't have sparse readers. Nik: Can you explain please the sparse readers versus dense readers? Peter: Oh, sorry. The problem that everybody was talking about last year was that if you're broadcasting to a huge number of readers, each of which is only reading a small fraction of all messages, that had terrible performance.
So that's actually not the issue we ran into. The way DBOS is set up, pretty much all workers are listening to all messages and that performs fine. The issue we ran into was the global lock at notify. When you commit a transaction that performs a notify, it has to acquire a global lock for the entire duration of the commit, and that includes the fsync. And what that means is that these transactions can't be group committed. They have to commit sequentially, which essentially bounds the throughput of
any operations that perform notify. You can't perform more than a certain number of them per second because you can't group commit. So you're stuck waiting out the entire duration of the fsync. the optimization that we settled on there to make listen notify, because we really want to use listen notify. It provides incredibly low latency, and when it works, it provides incredibly low latency. So w- the optimization we settled on was notify batching. So instead of sending a notify, originally, we were sending notifies in a trigger every single time you, for example, wrote to a stream.
And the optimization was instead it was removing the trigger and instead buffering stream writes in a background thread and then sending all the notifies in the batch and send that batch e-every ten milliseconds or so. A huge batch of notifies is actually very fast because you only have to hold the global lock for the duration of that commit. that's fast. And what this does is this allows us to use listen notify without re- running it, worrying about the global lock because we're batching commits and the global lock's only required for commits, and still getting the benefits of low latency because we can still emit these every, every few milliseconds
Qian: I think, the overall observation is that we don't really need that global o- ordering when we're doing the application level streaming, like streaming, for example, results to the client. Then what do you-- we're leveraging the listen/notify to do is to notify like, "Hey, we have some results available." So we don't really need that strict ordering. And I think that was the main aha moment for us that, okay, we don't need to limit ourselves to that exclusive locks. We can do batching. Yeah.
Nik: And yeah, and this is… okay. So and also the problem is happening. You, Peter mentioned that it was fixed in 19, and it's happening when commit has a lot to write, right? So like it's this fsync. Basically, transaction is significant in terms of how much it's writing, right? So th-this increases the problem. But in your case, you have only like brief writes in transaction. Not a lot, right? Not like a lot of JSONs to be written and fsynced during commit,
So the problem shouldn't be huge a-already if I'm not mistaken. and you decided to group it. This is interesting optimization. I, I wanted to raise this question, of course. You use skip locked updates and what's happening if we have some xmin horizon blockers, so long transactions or lagging logical slots? Obviously, a lot of dead tuples should be accumulated, right? Is it a requirement if you use DBOS at, like at high rates, at a lot
of events per second and so on? Is it a requirement I need to watch closely what is my xmin horizon? Don't allow long-running transactions, don't allow logical… lagging logical replication slots long transactions, so on standbys with hot standby feedback on and so on. Is it a requirement or you solve this somehow I don't know, like maybe mitigate-mitigating bloat with repacking, reindexing and so on? Peter: Got it. So just to be clear, we're only using skip locked on our own workflow status
table, and that's typically isolated within its own Postgres database. So our recommendation is Don't take out ultra long-running transactions inside the DBOS database, and you probably weren't doing that anyways because DBOS certainly doesn't do that. And then the rest should be fine. The DBOS operations are isolated here Nik: So you don't use skip locked for like events t- tables where events or workflow data is stored?
Peter: It's only used for deques off of the main workflow status table, and that's usually pretty isolated from user data Nik: But still, it's you need a s- single update on every status change, right? Peter: Yes, and that does lead to dead tuple accumulation. but that's a separate issue that y-yes, at ma- at massive, at really high throughputs, like when we were doing these benchmarks at, 40,000 workflows per second, we saw high CPU usage due to dead
tuple cleanup and autovacuum because we're turning on these indexes. And we've done a lot of work to optimize indexes, so most, almost every index is partial. So most indexes are only being maintained either for a small subset of workflows or only for a small subset of workflow lifetime. But there's still a fair amount of index churn on the order of… And this, of course, not as hot a dead tuple, and it's going to create a bunch of dead index. It's going to create two or three dead index entries that are
going to have to be cleaned up. At really high throughputs, this can become the bottleneck but that sets in around 20 or 30,000 actions per second is when we start seeing the tuple accumulation really becoming a problem, because there really aren't that… Tuple accumulation become a problem, because we have optimized this as much as we can. Nik: Yeah, I see it very differently. I see it like when you say 20, 30,000 per second, it means for me it becomes 2,000, 3,000 or 200, 300 per second if we talk about really long
transaction happening in parallel because they block vacuum, uh, work. We cannot delete the tuples, and at very high throughput with shorter transactions will be-become the same if you have much lower throughput, Peter: Yeah, which is why our recommendation is isolate the DBOS system database from the database we're using, we're using long-running transactions. These don't have to be in the same, in the… These don't have to be in the same Postgres database, and they probably shouldn't be, especially for long-running
transactions that could block Autovacuum Nik: Yeah, this is what my… Yeah, th-this is exactly what I was asking. It's not like like requirement, but it definitely if you have long-running transactions or any other explain horizon blockers performance of DB-DBOS will, will suffer eventually. Peter: Yeah, but these don't, these can be isolated from each other. They don't have to run in the same place, and they probably shouldn't run in the same place, especially at scale Nik: This is interesting. We have a lot of common things. I agree with putting everything to Postgres open source.
I w- didn't mention that the thing you're building is open source, which is great, and like this is super cool. And and like we don't need extra things on critical path. That's great. Also I guess you have self-served option as well, right? If it is open source, you can self-host yourself. Like freedom, right? It's great. And you run also y-you can run DBOS in, on RDS. Absolute fr-freedom. But when you say let's split it, it's against the first principle. Like first principle, let's have it everything in one place.
Because if we have, this thing and we have other tables, we can organize code right inside database. Sometimes it's a bad idea if it's very long, if CPU-consuming. But if it's something simple, I want to update everything inside, like using triggers, PostgreSQL code functions, and so on. I don't want to split. But once I have database, say terabyte in size, and I start some analytical queries right there because I'm not I'm not grown enough to have ClickHouse or something. I have only Postgres, right?
Those long-running queries will take time, right? And in this case updates and data, MVCC in Postgres actually is not a good thing to deal with, as we all know, right? This is I'm j- I'm asking because, like you m- you know that I'm trying to revive this PGQ from Skype, which is bloat-free. No, almost no updates. For event table, there are no updates there. But I don't see how it will… I don't see clearly how it will be
used to implement workflows yet. But for many tasks, it's great, and we don't have problems with bloat there. And I'm trying to understand how our operate properly. Like in, in case of updates, we need to establish some rules. Like for example, we should stop allowing long-running transactions. I'm I'm glad I participated in building transaction timeout. So we set transaction timeout, say like 10 minutes. That's it. And I'm very curious, like maybe there is a way to define some
boundaries and rules when it's time to move away and to like to split to different Postgres cluster, right? To split these workloads. M- And there are no good rules yet. Th-This is a problem Peter: One is that you don't have to split multiple Postgres clusters or servers. You can, if I remember how Autovacuum works correctly, it is isolated across multiple databases on the same cluster. Nik: Logical databases. Peter: Yes. So like the xmin horizon is set per database. So so you don't have to split your physical infrastructure.
You can share physical infrastructure Nik: Yeah that's a good point. I agree. Still, I feel like it would be great to have some benchmarks or something to define when it's time, when it's time to split, when it's fine, and which transactions should be disallowed. I think it-- you're very right. If you have 30,000 workflows per second it-- your requirements become much stricter in terms of what means long-running transaction, right? How much xmin horizon is lagging. Anyway, thank you for diving into this topic.
I'm also trying to learn this myself Qian: Yeah. I think that's actually a good, really good point because you are right that if you put… Y- there are benefits of co-locating the workflow tables with your actual application data. One thing we were talking about in some of our blog posts is this transactional outbox pattern. If you say someone, like we updated the customer data, now we want to kickstart some of the workflows, then by co-locating them in the same
database, you can do it in a trigger. Say, whenever you update this customer data, automatically start or enqueue a DBOS workflow in the same database, and that's really convenient and that, that's a way that you can guarantee whenever there's updates in this table, you'll kickstart a workflow that is durable, that will, that DBOS will guarantee will finish. And that's a pattern that is really difficult to do with, say, like external orchestrator because there will always be a gap. And by co-locating in the same database, there's a way to close this gap.
But everything has a trade-off. Like you said, if you put in the same database, you have to be really careful about what you, what queries you put in that database, like how to avoid head of line blocking, how to avoid bloating in that database. So I think that's a topic we're continuously, benchmarking and also by talking to users and see what kind of workflows and what kind of workloads they're running. And we're, we are also trying to categorize like a more detailed, guidelines for people as well. Nik: Yeah, interesting.
But any-anyway I'm glad that this, this topic is becoming more and more popular and other companies started to join t- like M-Microsoft recently, right? With pgDurable. What do you think about that system? Qian: Yeah. So this is great. I think pgDurable is mostly for pure SQLs, right? It's, it doesn't really work for if you want to send a email or interact with external systems. So we'll definitely, take a look and see if there are any ways to collaborate or borrow ideas from pgDurable. But I think in general it's another validation of the Postgres
Nik: yeah. yeah. Everything more and more people start looking in the same direction. I think yeah, the topic is warming up. I agree with you here. Also, there was a system called Absurd, which got my attention. Single SQL file as well. It… I think it was released last November. And yeah. I noticed in pgdurable is that the tendency to have a SQL PL, PostgreSQL code quite ef-ephemeral way. And this is w- like you're just passing it as a parameter, like to lump the functions, right? And this concerns me because I want my code still to be in Git, right?
To be versionable, fix, reviewable, and so on, right? So this leads me to question, what are the best use cases, case studies you ha- you had already, you where you observed debor… I'm sorry, like I pronounce it DBOS for myself. I don't know why. What are the best use case studies, use cases you have, like where it shines in terms of how like, like it suits really well, right? Sending emails or like processing orders in e-commerce or something like this.
What are the best cases? Peter: Yeah. the thing about workflows is that they're really powerful f- they're really powerful for a lot of things. So of course, the, the really hype new use case we see is AI agents and operations around them. So people using workflows both to make their actual agents durable. This is particularly important for long-running interactive agents that might, talk to a person or do multiple rounds of conversation with a person or require approvals for a person over the course of hours or days.
Similarly, data pipelines like document processing pipelines for agents often require durability, and, you know, workflows can really help there. So those are kind of the, the new use cases. And then there's this whole host of traditional business workflows that have always needed durability and where Postgres backed durability just makes it easier to provide it. And that's things like logistics. It's things like, like a- like lab automation. It's things like new user registration, making sure new users are properly updated in every single one of your cor- of your corporation's databases.
So we see this really broad spectrum of use cases for workflows because really any important operation that touches customer data, touches important data, it probably should be a workflow if you're really operating at scale and you really care about reliability Qian: I think in summary it's we really view Postgres as the central orchestrator to connect things together. You can connect external systems to your local database, or you can do event processing to connect multiple systems like your other
databases or your email server, payment system, all sorts of things. So we re- try to make sure your Postgres is the glue for everything. Yeah Nik: And speaking of Postgres maybe last question from me. So now Postgres 19 beta 2 just released, Postgres 20 development has started already. What what do you think should be changed in Postgres or brought to Postgres to make the product you are building even better? Peter: I think we do have to go… I would like to go back to notify.
Removing that global lock is not easy, and I'm not sure if it's doable with the way notify is currently implemented, but that area of the code hasn't really been worked on much since Postgres 9 and, maybe there's something that could be done there to remove the footguns from notify. And even if you don't remove them, I think notify's really complicated performance characteristics should probably be, should probably be actually
documented instead of having to figure this out like through by… I mean, we figured this out basically by, by reading the Postgres source code. That, that's one thing. Qian: Yeah, I think another thing is about, like, how to ho- better s- even better support for larger larger blobs or larger columns. when you-- when people are using DBOS, sometimes they want to checkpoint, say, a PDF or something as part of the output. And still I know Postgres has some good support for larger blobs right now. But I think most of the time, we still recommend people to say "Upload your
large documents to S3, then store a pointer in the database." But that still leaves, a little bit gap in the durable checkpointing, because there could be the case where you upload something but you don't checkpoint or vice versa. It'll be great if we can really put more, like, larger data into Postgres. Michael: There is a nice change, I think, coming in 19 around that in that the, I think the default TOAST compression is changing from
like a, an older PGLZ, like a Postgres specific compression algorithm to, I think it's LZ4. but yeah only minor on, on the grand scheme of things. nice one Nik: Yeah. And speaking of notify, I think, yes, global lock is still there, not removed. I remember some work last year in PostgreSQL Peter: There was work in 19 on improving the performance of broadcast, of the notify broadcast when not all listeners are listening
for all messages, which is great. That's a real performance improvement. The global lock is much harder to get rid of, and I'm not sure if it's possible. But that is… Nik: work on removing that lock. I remember that, that work. I also remember I just founded my email to document, as you said, the limitations from July 10, 2025. Nobody answered my email. So there is a patch proposed. Let's document it at least. Yeah, Peter: Yeah, the I, certainly 19 doesn't include any work on the global lock. Maybe there's some in development for 20, I'm not
Nik: Yeah. Yeah. Yeah. It's worth checking. And that documentation should be changed. I agree with you. So maybe someone listening to us will go and comment on that proposal, and we will get it documented at least. Great. Thank you so much. I enjoyed it a lot. I wish you all the best with this wonderful product. I'm going to try it in a couple of places, and I encourage our listeners to try it out. Especially it's open source, so it's so easy to start, right? Thank you Michael: Nice one.
Thanks very much, both of you Peter: Yeah, thanks a lot Nik: Yeah, have a great week. Qian: You too. Nik: Bye-bye
Tip: watch on YouTube. Since episode 193, the video shows charts and illustrations as we talk. https://youtu.be/VcDU-NRE3hI Michael, Nik, and Kirk discuss Postgres 19: why the release came in November and not in...
Tip: watch on YouTube. Since episode 193, the video shows charts and illustrations as we talk. https://www.youtube.com/watch?v=gAdhpYewLZ8 Nik and Kirk discuss the hidden costs of CREATE TEMP TABLE: dead tuples and...
Tip: watch on YouTube. Since episode 193, the video shows charts and illustrations as we talk. https://www.youtube.com/watch?v=7oeAAqU00zk Nik and Kirk are joined by Álvaro Hernández to discuss how Postgres in...
Tip: watch on YouTube. Since episode 193, the video shows charts and illustrations as we talk. https://www.youtube.com/watch?v=gf8e3JkQ_8M Nik is joined by Kirk Wolak to discuss what a random UUIDv4 primary key really...
Nik is joined by Kirk Wolak, Software Architect at KiraSoft, to discuss what a database still needs after pg_restore finishes and before it takes production traffic - what pg_dump and pg_restore leave behind, why...