TL;DR
Microsoft has announced the open source release of pg_durable, a PostgreSQL extension for fault-tolerant, durable workflow execution. This allows SQL-defined workflows to survive crashes and restarts without external orchestration tools, streamlining data pipelines and operational automation.
Microsoft has officially open sourced pg_durable, a PostgreSQL extension that facilitates fault-tolerant, in-database workflow execution, available now on Azure HorizonDB. This development introduces a new pattern for building resilient data pipelines and operational workflows directly within PostgreSQL, eliminating the need for external orchestrators or complex app-tier logic.
pg_durable allows defining workflows as SQL graphs that are executed with checkpointing at each step. If the database crashes or a step fails, execution resumes from the last durable checkpoint, simplifying recovery and ensuring reliability. The extension is designed to integrate seamlessly with PostgreSQL, requiring no additional infrastructure beyond the extension itself.
Microsoft’s announcement states that pg_durable is suited for use cases such as vector embedding pipelines, ingestion workflows, scheduled maintenance tasks, parallel query aggregation, and external API calls. It replaces traditional reliance on external schedulers like Airflow or custom app logic, centralizing workflow management within the database.
The extension is implemented as a PostgreSQL extension, supporting SQL-native workflow definitions using operators like ~> and |=>, with execution managed via functions such as df.start(). It is compatible with PostgreSQL 17 and 18, with prebuilt packages available for Debian-based systems, and source code for custom builds.
Why It Matters
This release marks a significant step in modern database operations by embedding durable workflow execution directly inside PostgreSQL. It reduces complexity, improves reliability, and enhances operational visibility by keeping workflow state within the database, aligning with industry trends toward compute close to data. For organizations managing large-scale data pipelines, this could streamline operations and reduce dependency on external orchestration systems.
Additionally, by open sourcing the extension, Microsoft encourages broader adoption and community development, potentially influencing best practices for reliable data workflows across the industry.
PostgreSQL durable workflow extension
As an affiliate, we earn on qualifying purchases.
As an affiliate, we earn on qualifying purchases.
Background
Prior to this release, building fault-tolerant workflows in PostgreSQL often involved external tools like Airflow, Temporal, or custom app logic with queues and status tables. These approaches can be complex, fragile, and difficult to maintain at scale. Microsoft’s introduction of pg_durable builds on existing trends toward SQL-native, database-resident orchestration, similar to features seen in other modern data systems but now integrated directly into PostgreSQL.
This development follows Microsoft’s broader initiative to enhance PostgreSQL’s capabilities on Azure HorizonDB, their managed cloud service. The extension aligns with recent industry efforts to bring compute closer to data, reducing latency and operational overhead.
“pg_durable introduces fault-tolerant, in-database workflow execution, simplifying reliable data pipelines without external dependencies.”
— Microsoft spokesperson
“Embedding durable execution within PostgreSQL could transform how organizations manage complex workflows, reducing reliance on external orchestrators.”
— Open source community analyst
fault-tolerant database workflow tools
As an affiliate, we earn on qualifying purchases.
As an affiliate, we earn on qualifying purchases.
What Remains Unclear
It is not yet clear how widely adopted pg_durable will become or how it will perform in large-scale, production environments. Details about security, multi-tenant support, and integration with existing workflow tools remain to be seen. Additionally, the extent of community contributions and future feature development is still developing.
PostgreSQL extension for data pipelines
As an affiliate, we earn on qualifying purchases.
As an affiliate, we earn on qualifying purchases.
What’s Next
Next steps include broader community testing, feedback from early adopters, and potential integration into other PostgreSQL distributions. Microsoft is likely to continue enhancing pg_durable, with possible future features such as richer control flow, broader SDK support, and tighter cloud-native integrations. Monitoring user adoption and performance benchmarks will be key indicators of its industry impact.
Azure HorizonDB PostgreSQL tools
As an affiliate, we earn on qualifying purchases.
As an affiliate, we earn on qualifying purchases.
Key Questions
What is pg_durable?
pg_durable is a PostgreSQL extension that enables defining and executing fault-tolerant, durable workflows directly within the database using SQL, ensuring workflows survive crashes and restarts.
Who can benefit from pg_durable?
Backend and data engineers, DBAs, SREs, and teams building data pipelines or AI workflows that require reliable, persistent execution within PostgreSQL.
How does pg_durable compare to external orchestrators?
It embeds workflow management inside PostgreSQL, reducing the need for external tools like Airflow or Temporal, and simplifies operational visibility and recovery.
Is pg_durable available outside Azure HorizonDB?
Yes, the extension is open source and can be installed on PostgreSQL 17 and 18 environments, though it is currently promoted through Azure HorizonDB for cloud deployment.
What are the limitations of pg_durable?
It is designed for SQL-based workflows; complex logic requiring arbitrary code or external systems may still need external orchestration. Also, certain environments may face restrictions on extension installation or background worker use.
Source: Hacker News
