TL;DR
DuckDB has announced Quack, a new protocol allowing its instances to connect and communicate as clients and servers. This development enables multi-process, client-server architectures, enhancing DuckDB’s versatility beyond in-process use.
The DuckDB team announced the release of Quack, a new protocol that enables multiple DuckDB instances to communicate over a client-server architecture, a feature previously absent from the database system. This development expands DuckDB’s capabilities, allowing it to support workloads involving concurrent writers and multi-process environments, which are critical for many real-world applications.
Quack is a lightweight, simple-to-set-up protocol built on proven technologies such as HTTP, designed from scratch without legacy constraints, and integrated into DuckDB version 1.5.2. It allows two or more DuckDB instances to connect, regardless of whether they are on the same machine or distributed across different networks, facilitating multi-process data workflows.
To use Quack, users need to install the extension in both instances, load it, and then establish connections via a specified URL and token. Once connected, the instances can perform operations such as creating tables, inserting data, executing queries, and copying data between instances, effectively enabling a client-server architecture within DuckDB.
Why It Matters
This development matters because it addresses a longstanding limitation of DuckDB, which traditionally operated as an in-process database optimized for single-process, interactive use cases. By enabling client-server communication, DuckDB can now support more complex, concurrent workloads, including multi-process data collection, real-time dashboards, and distributed data processing, broadening its appeal and applicability across different domains.
DuckDB client-server extension
As an affiliate, we earn on qualifying purchases.
As an affiliate, we earn on qualifying purchases.
Background
Since its release in 2019, DuckDB has been known for its in-process architecture, favored for data science and embedded applications. Prior to Quack, users relied on workarounds such as custom RPC solutions or integrating with other databases like PostgreSQL to enable multi-process access. The introduction of Quack represents a significant architectural shift, inspired by the client-server models of traditional databases, but designed to be simple and fast.
“We see DuckDB as a universal data wrangling tool. If this means having a client-server protocol in addition to the in-process capabilities – fine. If this ends up unlocking a vast new set of cases in which DuckDB can be useful – excellent!”
— DuckDB team
“Quack is simple to set up and builds on proven technologies such as HTTP, making it accessible and performant.”
— DuckDB team
DuckDB HTTP protocol setup
As an affiliate, we earn on qualifying purchases.
As an affiliate, we earn on qualifying purchases.
What Remains Unclear
It is not yet clear how Quack will perform under very high concurrency workloads or with extremely large data volumes. Details about security features, authentication mechanisms beyond simple tokens, and long-term stability are still emerging.
DuckDB multi-process database
As an affiliate, we earn on qualifying purchases.
As an affiliate, we earn on qualifying purchases.
What’s Next
Next steps include broader testing of Quack in real-world scenarios, integration with additional tools and workflows, and potential development of advanced features such as enhanced security, load balancing, and failover support. The DuckDB team may also release updates or extensions to further optimize and document the protocol.
DUCKDB: EMBEDDED ANALYTICS FOR DATA SCIENTISTS AND PYTHON DEVELOPERS: Process Gigabytes Locally with SQL, Pandas Integration and Real-World Projects
As an affiliate, we earn on qualifying purchases.
As an affiliate, we earn on qualifying purchases.
Key Questions
How does Quack compare to existing client-server solutions for DuckDB?
Quack is designed to be lightweight, simple, and built from scratch, unlike workarounds like Arrow Flight SQL or custom RPCs. It aims to provide fast, easy-to-setup communication between DuckDB instances without the overhead of traditional protocols.
Can I use Quack with multiple remote instances across different networks?
Yes, Quack supports distributed setups, allowing instances on different computers or networks to connect, provided network configurations permit communication.
What are the security implications of using Quack?
Currently, authentication relies on tokens, but detailed security features and best practices are still being developed. Users should consider network security when deploying Quack in sensitive environments.
Will Quack support high concurrency workloads?
Performance under high concurrency is still under evaluation. The protocol is designed for speed, but real-world testing will determine its limits.
Is Quack available in stable releases?
Quack is available in DuckDB version 1.5.2, which is a current release. Future updates may enhance its stability and features.
