Events

Events

[Andy at Sea] Silnik: Re-deciding the Plan at Every Stage, Adaptive Distributed Execution on DataFusion

Speaker:
Artur Borycki
Date:
Mon Nov 30, 2026 @ 04:30pm EST
Date:
Mon Nov 30, 2026
Time:
04:30pm EST
Location:
https://cmu.zoom.us/j/92568565952?pwd=N8VKf5xfF5aisG9VoEwdNsCp0dCwel.1Zoom
Title:
Silnik: Re-deciding the Plan at Every Stage, Adaptive Distributed Execution on DataFusion
System:
TeraData

Talk Info:

Silnik (Polish for “engine”) is a distributed SQL query engine from Teradata Labs, built in Rust inspired by Apache DataFusion Ballista. It is designed to cover a wide scale range with one engine: gigabytes on a single node, terabytes across a cluster. It reads open table formats like Apache Iceberg directly.

The talk starts from upstream Ballista’s clean scheduler–executor design and walks through what had to change to make it production-grade. Upstream Ballista had adaptive execution off by default, no catalogs, and no SQL frontend. Silnik keeps the same skeleton but rebuilds almost every component inside it. The main changes are:

  • Adaptive planning at every stage boundary. The plan is not fixed at submission. When a stage finishes, the planner re-plans the subtree that hasn’t launched yet, using the measured statistics from that stage.
  • Faster data movement. Dynamic filters carry predicates from one side of a join to the other. Pipelined shuffle lets downstream stages launch together.
  • A redesigned storage path. Shuffle goes to local NVMe, with hedged S3 reads.
  • Fault handling. The engine uses speculative execution, recovers lost shuffle output, and runs grace hash joins within a spill budget.
  • Standard interfaces. BI tools connect through an Arrow Flight SQL frontend. A multi-catalog layer talks to Glue, REST, SQL and HMS catalogs.

This talk is part of the Whatever Andy Found Interesting While Out At Sea.

Bio:

Artur Borycki is a Senior Director of Innovation and Advanced Development at Teradata, where he explores emerging technologies and develops next-generation data platforms. With over 20 years of IT experience, Artur has extensive expertise in designing and building large-scale distributed data systems, database architectures, and massively parallel processing engines. His recent work focuses on database internals, query execution, vector search, and integrating artificial intelligence with modern data platforms. He has collaborated with industry partners, academic researchers, and enterprise customers to push the boundaries of data management technology. Most importantly, Artur knows a pile about databases.