News & Events

[Fall 2020] A Peek into Snowflake’s Scalable Architecture

Snowflake is an analytic data warehouse offered as a fully-managed service in the cloud. It is faster, easier to use, and far more scalable than traditional on-premise data warehouse offerings and is used by thousands of customers around the world. Snowflake's data warehouse is not built on an existing database or "big data" software platform such as Hadoop—it uses a new SQL database engine with a unique architecture designed for the cloud. Snowflake operates three engineering centers in San Mateo, Read More

Self-Driving Database Management Systems: Forecasting, Modeling, and Planning (Lin Ma)

Database management systems (DBMSs) are an important part of modern data-driven applications. However, they are notoriously difficult to deploy and administer. There are existing methods that recommend physical design or knob configurations for DBMSs. But most of them require humans to make final decisions and decide when to apply changes. Furthermore, they either (1) only focus on a single aspect of the DBMS, (2) are reactionary to the workload patterns and shifts, (3) require expensive exploratory testing on data copies, Read More

Quarantine DB Talk 2020: ksqlDB: A Stream-Relational Database System

ksqlDB is a distributed event streaming database system that allows users to express SQL queries over relational tables and event streams. The project was released by Confluent in 2017 and is hosted on Github and developed with an open-source spirit. ksqlDB is built on top of Apache Kafka, a distributed event streaming platform. In this talk, we discuss ksqlDB's architecture that is influenced by Apache Kafka and its stream processing library, Kafka Streams. We explain how ksqlDB executes continuous queries, Read More

Quarantine DB Talk 2020: TiDB – On the Long Journey of HTAP

Due to the rising demand for real-time analytics and insights on fresh data, the term HTAP becomes hot in recent years. From the very beginning, TiDB was designed for pure TP workload. But gradually as we adapt to users' requirements, TiDB evolves into an HTAP database based on Raft. We will introduce TiDB's design, internals, and HTAP architectural evolvement. This talk is part of the Quarantine Database Tech Talk Seminar Series. Zoom Link: https://cmu.zoom.us/j/562649242 (Password 264771) Read More

Quarantine DB Talk 2020: The Cascades Framework for Query Optimization at Microsoft

The Cascades framework was an academic project introduced 25 years ago as a foundation for modern query optimizers. It provides extensibility, memoization-based dynamic programming, an algebraic representation of logical and physical operator trees, and manipulation of such trees using transformation rules to enable cost-based query optimization. Cascades provides a clean framework/skeleton for optimizer development, but it needs to be instantiated with domain-knowledge and augmented in several directions to cope with real-world workloads in an industrial setting. We will describe some Read More

Quarantine DB Talk 2020: PlanetScale: Query Planning for a Sharded System like Vitess

Traditional query planning involves parsing of an input SQL into an AST, and then transforming it into primitives which can later be sent through an optimizer. However, in a sharded system, each leaf node is a full relational engine that is capable of doing its own optimizations. So, the traditional approach may not work for such a system. And who knows if the finally reconstructed query would be correctly optimized by the underlying engine? The Vitess VTGate proxy uses a Read More

Quarantine DB Talk 2020: Databricks: A Deep Dive into Spark SQL’s Catalyst Optimizer

Catalyst is the SQL query optimizer in Spark SQL. It is one of the most important components of Apache Spark, as it powers major Spark APIs like SQL, DataFrames/Datasets, as well as Structured Streaming. Unlike many traditional SQL systems, Spark enables users to query data in arbitrary formats stored in arbitrary locations at scale. While being powerful, this also imposes extra query planning challenges such as statistics collection and cost estimation, which further affect performance negatively. In this talk, we Read More

Quarantine DB Talk 2020: Refactoring Query Processing in MySQL

MySQL is often called the world's most popular open source DBMS, and it's certainly one of the most used. MySQL grew up with the open source movement and the public Internet and became a part of the famous LAMP stack. Today, MySQL server are still powering a huge number of web sites. A lot has changed in MySQL in the 25 years since the initial release, but a lot of the core code has also remained almost unchanged. The query Read More

Fall 2020: Prashanth Menon (CMU)

Just-in-time (JIT) query compilation is a technique to improve analytical query performance in database management systems (DBMSs). But the cost of compiling each query can be significant relative to its execution time. This overhead prohibits the DBMS from employing well-known adaptive query processing (AQP) methods to generate a new plan for a query if data distributions do not match the optimizer's estimations. The optimizer could eagerly generate multiple sub-plans for a query, but it can only include a few alternatives Read More

Quarantine DB Talk 2020: CrocodileDB: Resource Efficient Database Execution

The coming end of Moore’s law requires that data systems be more judicious with computation and resources as the growth in data outpaces the availability of computational resources. Current database systems are eager and aggressively consume resources to immediately and quickly complete the task at hand. Intelligently deferring a task to a later point in time can increase result reuse, reduce work that might later be invalidated, or avoid unnecessary work altogether. In this talk I will introduce CrocodileDB, a Read More