It is important for developers to understand the performance of a software project as they develop new features, fix bugs, and try to generally improve the product. While it is simple to state that requirement, it can be hard to do in practice. There are a lot of choices an organization faces when trying to understand the performance of the software, with a lot of opportunities to waste money (execution resources), or worse, time. We have run full speed into a number of those opportunities, and we like to think we have learned from those opportunities. The challenges include: false positives, false negatives, reproducibility of tests, noise in the results, tests execution time, insufficient workload coverage, insufficient configuration coverage, cost, identifying regressions in the presence of noise, diagnosing those regressions, one regression hiding another, etc.

In this talk we describe our performance testing environment at MongoDB, and what we have built into that environment in order to address those challenges. The core of our environment is a focus on automating everything, integrating into our continuous integration (CI) system (Evergreen), controlling as many factors as possible, and making everything as repeatable and consistent as possible. We have built our performance testing infrastructure over the last 5-6 years and it has had a huge impact on the product we release. We are able to quickly identify changes in performance, triage them, and assign them out to developers. The developers are able to exactly reproduce those regressions and are eager to fix them. And at the end of the day, we have a much better understanding of the performance of the software we release and ship to customers.

This talk is part of the Vaccination Database Tech Talk Seminar Series.

Bio:
David is a staff engineer at MongoDB focused on server performance. He focuses on increasing the understanding of how MongoDB's software performs for its customers. In practice this includes: Asking hard questions about MongoDB performance and then trying to answer them (or having someone else try to answer them); Challenging assumptions and commonly accepted wisdom around MongoDB performance; Encouraging everyone at MongoDB to think about performance, including adding new performance tests relevant to their ongoing work (e.g., adding new performance tests for new features or refactors); And explaining the current state of performance to others. He helped build and design MongoDB's performance testing infrastructure from the bottom up. At various times this required focusing on complete end-to-end automation, control of test noise and variability, working around test noise, and building processes to make sure that issues identified by the infrastructure were properly recognized and addressed.

More Info: https://db.cs.cmu.edu/seminar2021/#db2