[Vaccination 2021] The Design of InfluxDB IOx: An In-Memory Columnar Database Written in Rust with Apache Arrow (Paul Dix)

I’ll talk about the design of InfluxDB IOx, the future core of InfluxDB, an open source time series database. It’s an in-memory columnar database that uses object storage for persistence. It’s written in Rust and is built on top of Apache Arrow. Unlike previous versions of InfluxDB, IOx supports standards compliant SQL and the Postgres dialect in particular. This is in addition to backwards compatibility with InfluxQL and Flux, our other two query languages.

InfluxDB IOx is a project that’s still under early prototyping and development so I’ll talk about some of the design decisions we’ve made and the challenges we’re working on. This includes managing the data lifecycle from real-time ingest to compressed in-memory storage, to persistence via Parquet files in object store. The choice of making object storage the primary durability layer leads to interesting opportunities for building a production system, which I’ll also dig into.

This talk is part of the Vaccination Database Tech Talk Seminar Series.

Bio:
Paul is the creator of InfluxDB and the founder and CTO of InfluxData. He has helped build software for startups, large companies and organizations like Microsoft, Google, McAfee, Thomson Reuters, and Air Force Space Command. He is the series editor for Addison Wesley’s Data & Analytics book and video series. In 2010 Paul wrote the book Service Oriented Design with Ruby and Rails for Addison Wesley. In 2009 he started the NYC Machine Learning Meetup. Paul holds a degree in computer science from Columbia University.

More Info: https://db.cs.cmu.edu/seminar2021/#db15