Current OLTP database designs, which date largely from the 1970s, are based on several assumptions about the architecture of database applications and hardware that are less true today than they were 30 years ago. For example, all but the very largest OLTP applications can fit in main memory of a modern shared-nothing cluster of server machines. On a single node with a memory resident database, OLTP transactions take only a few microseconds to execute. Additionally, many applications carefully construct database transactions so they have no user stalls.

Taken together, both of these points mean there is a large class of OLTP applications for which a single-threaded execution engine with no concurrency control performs very well, avoiding the need for high overhead, locking-based pessimistic concurrency control protocols designed to keep CPUs busy during disk and user stalls. Further, the cost of computers has dropped so dramatically in the past thirty years that paying for a dedicated database administrator has become one of the dominant costs in running a database system, such that tools that automate design and tuning have great value. Finally, the architecture of a server node has also shifted — the number of cores available to process data is proliferating.

The goal of the H-Store project is to investigate how these architectural and application shifts affect the performance of OLTP databases, and to study what performance benefits would be possible with a complete redesign of OLTP systems in light of these trends. Our early results show that a simple prototype built from scratch using modern assumptions can outperform current commercial DBMS offerings by around a factor of 80 on OLTP workloads.

People

• Andy Pavlo
• Joy Arulraj
• Lin Ma

Acknowledgements

Part of this work was sponsored by the NSF under grants IIS-0845643 and IIS-0704424, the Natural Sciences and Engineering Research Council of Canada, VoltDB, and the Intel Science and Technology Center for Big Data.

Publications

1. L. Ma, J. Arulraj, S. Zhao, A. Pavlo, S. R. Dulloor, M. J. Giardino, J. Parkhurst, J. L. Gardner, K. Doshi, and S. Zdonik, "Larger-than-memory Data Management on Modern Storage Hardware for In-memory OLTP Database Systems," in Proceedings of the 12th International Workshop on Data Management on New Hardware, 2016, p. 9:1-9:7. PDF Bibtex

   @inproceedings{ma16,
      author = {Ma, Lin and Arulraj, Joy and Zhao, Sam and Pavlo, Andrew and Dulloor, Subramanya R. and Giardino, Michael J. and Parkhurst, Jeff and Gardner, Jason L. and Doshi, Kshitij and Zdonik, Stanley},
      title = {Larger-than-memory Data Management on Modern Storage Hardware for In-memory OLTP Database Systems},
      booktitle = {Proceedings of the 12th International Workshop on Data Management on New Hardware},
      series = {DaMoN '16},
      year = {2016},
      pages = {9:1--9:7},
      articleno = {9},
      numpages = {7},
      url = {https://db.cs.cmu.edu/papers/2016/ma-damon2016.pdf},
    }

2. M. Serafini, R. Taft, A. J. Elmore, A. Pavlo, A. Aboulnaga, and M. Stonebraker, "Clay: Fine-Grained Adaptive Partitioning for General Database Schemas," Proc. VLDB Endow., vol. 10, iss. 4, pp. 445-456, 2016. PDF Bibtex

   @article{serafini16,
      author = {Serafini, Marco and Taft, Rebecca and Elmore, Aaron J and Pavlo, Andrew and Aboulnaga, Ashraf and Stonebraker, Michael},
      title = {Clay: Fine-Grained Adaptive Partitioning for General Database Schemas},
      journal = {Proc. VLDB Endow.},
      volume = {10},
      number = {4},
      month = {December},
      year = {2016},
      pages = {445--456},
      publisher = {VLDB Endowment},
      url = {http://hstore.cs.brown.edu/papers/hstore-clay.pdf},
    }

3. H. Zhang, D. G. Andersen, A. Pavlo, M. Kaminsky, L. Ma, and R. Shen, "Reducing the Storage Overhead of Main-Memory OLTP Databases with Hybrid Indexes," in Proceedings of the 2016 International Conference on Management of Data, 2016, pp. 1567-1581. PDF Bibtex

   @inproceedings{zhang16,
      author = {Zhang, Huanchen and Andersen, David G. and Pavlo, Andrew and Kaminsky, Michael and Ma, Lin and Shen, Rui},
      title = {Reducing the Storage Overhead of Main-Memory OLTP Databases with Hybrid Indexes},
      booktitle = {Proceedings of the 2016 International Conference on Management of Data},
      series = {SIGMOD '16},
      year = {2016},
      pages = {1567--1581},
      numpages = {15},
      url = {https://db.cs.cmu.edu/papers/2016/zhang-sigmod2016.pdf},
    }

4. A. J. Elmore, V. Arora, R. Taft, A. Pavlo, D. Agrawal, and A. El Abbadi, "Squall: Fine-Grained Live Reconfiguration for Partitioned Main Memory Databases," in Proceedings of the 2015 ACM SIGMOD International Conference on Management of Data, 2015, pp. 299-313. PDF Bibtex

   @inproceedings{elmore15,
      author = {Elmore, Aaron J. and Arora, Vaibhav and Taft, Rebecca and Pavlo, Andrew and Agrawal, Divyakant and El Abbadi, Amr},
      title = {Squall: Fine-Grained Live Reconfiguration for Partitioned Main Memory Databases},
      booktitle = {Proceedings of the 2015 ACM SIGMOD International Conference on Management of Data},
      series = {SIGMOD '15},
      year = {2015},
      pages = {299--313},
      url = {https://db.cs.cmu.edu/papers/2015/p299-elmore.pdf},
    }

5. J. Meehan, N. Tatbul, S. Zdonik, C. Aslantas, U. Çetintemel, J. Du, T. Kraska, S. Madden, D. Maier, A. Pavlo, M. Stonebraker, K. Tufte, and H. Wang, "S-Store: Streaming Meets Transaction Processing," PVLDB, vol. 8, iss. 13, pp. 2134-2145, 2015. PDF Bibtex

   @article{meehan15,
      author = {John Meehan and Nesime Tatbul and Stan Zdonik and Cansu Aslantas and Ugur {\c{C}}etintemel and Jiang Du and Tim Kraska and Samuel Madden and David Maier and Andrew Pavlo and Michael Stonebraker and Kristin Tufte and Hao Wang},
      title = {S-Store: Streaming Meets Transaction Processing},
      journal = {{PVLDB}},
      volume = {8},
      number = {13},
      pages = {2134--2145},
      year = {2015},
      url = {https://db.cs.cmu.edu/papers/2015/p2134-meehan.pdf},
    }

6. U. Cetintemel, J. Du, T. Kraska, S. Madden, D. Maier, J. Meehan, A. Pavlo, M. Stonebraker, E. Sutherland, N. Tatbul, K. Tufte, H. Wang, and S. Zdonik, "S-Store: A Streaming NewSQL System for Big Velocity Applications," Proc. VLDB Endow., vol. 7, iss. 13, 2014. PDF Bibtex

   @article{cetintemel14,
      author = {Ugur Cetintemel and Jiang Du and Tim Kraska and Samuel Madden and David Maier and John Meehan and Andrew Pavlo and Michael Stonebraker and Erik Sutherland and Nesime Tatbul and Kristin Tufte and Hao Wang and Stanley Zdonik},
      title = {S-Store: A Streaming NewSQL System for Big Velocity Applications},
      journal = {Proc. VLDB Endow.},
      volume = {7},
      number = {13},
      year = {2014},
      url = {http://www.cs.cmu.edu/~pavlo/static/papers/sstore_vldb14.pdf},
    }

7. J. DeBrabant, J. Arulraj, A. Pavlo, M. Stonebraker, S. Zdonik, and S. Dulloor, "A Prolegomenon on OLTP Database Systems for Non-Volatile Memory," in ADMS@VLDB, 2014, pp. 57-63. PDF Bibtex

   @inproceedings{debrabant14,
      author = {DeBrabant, Justin and Arulraj, Joy and Pavlo, Andrew and Stonebraker, Michael and Zdonik, Stan and Dulloor, Subramanya},
      title = {A Prolegomenon on OLTP Database Systems for Non-Volatile Memory},
      booktitle = {ADMS@VLDB},
      pages = {57--63},
      year = {2014},
      url = {http://hstore.cs.brown.edu/papers/hstore-nvm.pdf},
    }

8. R. Taft, E. Mansour, M. Serafini, J. Duggan, A. J. Elmore, A. Aboulnaga, A. Pavlo, and M. Stonebraker, "E-Store: Fine-Grained Elastic Partitioning for Distributed Transaction Processing," Proc. VLDB Endow., vol. 8, iss. 3, pp. 245-256, 2014. PDF Bibtex

   @article{taft2014,
      author = {Taft, Rebecca and Mansour, Essam and Serafini, Marco and Duggan, Jennie and Elmore, Aaron J and Aboulnaga, Ashraf and Pavlo, Andrew and Stonebraker, Michael},
      title = {{E}-Store: Fine-Grained Elastic Partitioning for Distributed Transaction Processing},
      journal = {Proc. VLDB Endow.},
      volume = {8},
      number = {3},
      month = {November},
      year = {2014},
      pages = {245--256},
      publisher = {VLDB Endowment},
      url = {http://hstore.cs.brown.edu/papers/hstore-elastic.pdf},
    }

9. J. DeBrabant, A. Pavlo, S. Tu, M. Stonebraker, and S. Zdonik, "Anti-Caching: A New Approach to Database Management System Architecture," Proc. VLDB Endow., vol. 6, pp. 1942-1953, 2013. PDF Bibtex

   @article{debrabant13,
      author = {DeBrabant, Justin and Pavlo, Andrew and Tu, Stephen and Stonebraker, Michael and Zdonik, Stan},
      title = {{A}nti-{C}aching: A New Approach to Database Management System Architecture},
      journal = {Proc. VLDB Endow.},
      volume = {6},
      issue = {14},
      month = {September},
      year = {2013},
      pages = {1942--1953},
      publisher = {VLDB Endowment},
      url = {http://hstore.cs.brown.edu/papers/hstore-anticaching.pdf},
    }