Benchmarking is an important, yet often overlooked, aspect of any database management system (DBMS) research and development effort. Despite several advancements over the last decades, the deployment of a comprehensive testing platform with a diverse set of data sets and workloads is still non-trivial. In many cases, researchers and developers are limited to a small number of workloads to evaluate the performance characteristics of their work. This is due to the lack of a universal benchmarking infrastructure, and because it is typically hard to gain access to real data and workloads due to privacy and security concerns. This results in many unnecessary engineering efforts, limits availability of reference workloads, and makes scientific results difficult to compare.

To remedy these problems, we present BenchBase, an extensible “batteries included” DBMS benchmarking testbed that is tailored for on-line transaction processing (OLTP) and Web-oriented workloads.

Some of the key features are:

• Precise rate control (allows to define and change over time the rate at which requests are submitted)
• Precise transactional mixture control (allows to define and change over time % of each transaction type)
• Access Distribution control (allows to emulate evolving hot-spots, temporal skew, etc..)
• Support trace-based execution (ideal to handle real data)
• Extensible design
• Support for statistics collection (microseconds latency and throughput precision, seconds precision for OS resource utilization)
• Automatic rendering via Javascript (and many available and gnuplot matlab scripts)
• Elegant management of SQL Dialect translations (to target various DBMSs)
• Targeting all major relational DBMSs and DBaaS offerings via the standard JDBC interface (tested on MySQL, Postgres, Oracle, SQLServer, DB2, HSQLDB, Amazon RDS MySQL, Amazon RDS Oracle, SQL Azure)
• Store-Procedure friendly architecture

The key contributions of OLTP-Bench are its ease of use and extensibility, support for tight control of transaction mixtures, request rates, and access distributions over time, as well as the ability to support all major DBMSs and database-as-a-service (DBaaS) platforms.

People

• Andy Pavlo
• Wan Shen Lim
• Tim Veil (Cockroach Labs)
• Dana Van Aken
• Carlo Curino (MSR)
• Djellel Eddine Difallah (University of Fribourg)
• Philippe Cudre-Maroux (University of Fribourg)

Acknowledgements

This research was funded (in part) by the National Science Foundation (III-1423210) and the YourKit Java Profiler.

Publications

1. D. Van Aken, D. E. Difallah, A. Pavlo, C. Curino, and P. Cudré-Mauroux, "BenchPress: Dynamic Workload Control in the OLTP-Bench Testbed," in Proceedings of the 2015 ACM SIGMOD International Conference on Management of Data, 2015, pp. 1069-1073. PDF Bibtex

   @inproceedings{vanaken15,
      author = {Van Aken, Dana and Difallah, Djellel E. and Pavlo, Andrew and Curino, Carlo and Cudr{\'e}-Mauroux, Philippe},
      title = {BenchPress: Dynamic Workload Control in the OLTP-Bench Testbed},
      booktitle = {Proceedings of the 2015 ACM SIGMOD International Conference on Management of Data},
      series = {SIGMOD '15},
      year = {2015},
      pages = {1069--1073},
      url = {https://db.cs.cmu.edu/papers/2015/p1069-vanaken.pdf},
    }

2. D. E. Difallah, A. Pavlo, C. Curino, and P. Cudré-Mauroux, "OLTP-Bench: An Extensible Testbed for Benchmarking Relational Databases," PVLDB, vol. 7, iss. 4, pp. 277-288, 2013. PDF Bibtex

   @article{DifallahPCC13,
      author = {Djellel Eddine Difallah and Andrew Pavlo and Carlo Curino and Philippe Cudr{\'e}-Mauroux},
      title = {OLTP-Bench: An Extensible Testbed for Benchmarking Relational Databases},
      journal = {PVLDB},
      volume = {7},
      number = {4},
      year = {2013},
      pages = {277--288},
      url = {http://www.cs.cmu.edu/~pavlo/static/papers/oltpbench-vldb.pdf},
    }

3. C. A. Curino, D. E. Difallah, A. Pavlo, and P. Cudre-Mauroux, "Benchmarking OLTP/Web Databases in the Cloud: The OLTP-bench Framework," in Proceedings of the Fourth International Workshop on Cloud Data Management, 2012, pp. 17-20. PDF Bibtex

   @inproceedings{curino2012,
      author = {Curino, Carlo A. and Difallah, Djellel E. and Pavlo, Andrew and Cudre-Mauroux, Philippe},
      title = {Benchmarking OLTP/Web Databases in the Cloud: The OLTP-bench Framework},
      booktitle = {Proceedings of the Fourth International Workshop on Cloud Data Management},
      series = {CloudDB '12},
      year = {2012},
      pages = {17--20},
      url = {http://www.cs.cmu.edu/~pavlo/static/papers/oltpbench.pdf},
    }