Automated Generation, Evaluation, and Enhancement of JMH Microbenchmark Suites from Unit Tests

Abstract

Ensuring the performance of software systems is a cornerstone of modern software engineering, directly influencing user satisfaction and reliability. Despite its critical role, performance testing remains resource-intensive and difficult to scale, particularly in large projects, due to the complexity of microbenchmark creation and execution. Microbenchmarking frameworks like the Java Microbenchmark Harness (JMH) offer precise performance insights but require significant expertise, limiting their adoption. This thesis addresses these challenges by introducing ju2jmh, a novel framework that automates the transformation of JUnit tests into JMH microbenchmarks, bridging the gap between functional and performance testing.

The contributions of this thesis are threefold. First, ju2jmh automates the generation of high-quality JMH microbenchmarks from widely used JUnit test suites, enabling developers to adopt performance microbenchmarking with minimal manual effort. Results demonstrate that the generated microbenchmarks exhibit stability comparable to manually crafted ones and effectively detect real-world performance bugs. Second, the Performance Mutation Testing (PMT) framework is developed to systematically evaluate the robustness of microbenchmarks in detecting artificial performance bugs, achieving competitive mutation scores. Third, a clustering approach is proposed to optimize the execution of microbenchmarks by grouping functionally similar tests based on code coverage information. This strategy reduces execution time by 81.2% to 86.2% across three large-scale projects while preserving accuracy and reliability.

Evaluated on three diverse open-source Java projects, the proposed solutions address stability, detection capabilities, and scalability challenges in performance testing workflows. The findings highlight the potential of ju2jmh and its associated methodologies to transform performance microbenchmarking practices, providing developers with practical tools to integrate reliable and efficient performance testing into modern software development pipelines. These advancements pave the way for future research into extending automated performance testing across different programming languages and development ecosystems.