LSM (Log-Structured Merge) trees are now the bedrock of many storage engines and datastores like RocksDB, HBase, Cassandra etc. They provide the ability to avoid random-writes, and provide immutability. Data is organized in multiple-levels that are exponentially increasing in size. Each data mutation writes a new version of an object, and background processes named merge/compaction continuously remove the unused versions, while moving the data across the layers of the LSM tree and maintain its shape.

This talk will describe how the immutability of LSMs can be used to provide data virtualization. Since the underlying data in persistence in an LSM never changes, it provides an opportunity to build a virtualization layer over it. In this talk, we will describe a mechanism to use metadata in the form of many to 1 references over data files in the LSM. This metadata can then be used to create constant-time clones of data-sets without physically copying the data. These clones can then be used for testing, and experimentation and also for taking back-ups. The cloning can also be used for fast data-migration between multiple datastores with LSM based-storage, running on a common distributed storage layer.

PDL Summer 2024 Talk Series
Zoom: https://cmu.zoom.us/j/97530013271
Contact Karen Lindenfelser for password

Bio:
Vaibhav works in Salesforce where he works on various large-scale data management problems including LSM, multi-tenancy, transaction processing, indexing and other access layer related challenges in the database engine. Prior to joining Salesforce, he did his PhD in Computer Science from University of California, Santa Barbara, where he worked on problems related to data variety - both in structure and access. Aside from that, he has worked on various distributed database problems during his time at Amazon, Microsoft Research, HP Labs and Yahoo!.

More Info: https://pdl.cmu.edu/SDI/index.shtml