The architecture diagram of a typical data and AI infrastructure setup often features a primary compute engine (e.g., Apache Spark) alongside an array of supplementary tools for observability, AI integration, streaming support, memory management, interactivity, and more. While this modular architecture can be effective, it also introduces challenges around performance bottlenecks, maintenance costs, and integration complexity. In this talk, we will explore whether it is possible to simplify such complex architectures by addressing some of the core engine-level limitations that ultimately necessitate this complex picture. Drawing from Synnada’s experience with Apache DataFusion, we will explore the concept of “unified” compute engines, highlighting both the opportunities this approach presents and the obstacles that remain.

This talk is part of the Database Building Blocks Seminar Series.

Zoom Link: https://cmu.zoom.us/j/95283696582 (Passcode 787637)

Bio:
Mehmet is a Apache Datafusion PMC Member, co-founder and CEO of Synnada.