Title: Improving System Performance via Design and Configuration Space Exploration
Abstract: The runtime performance of complex software systems often depends on the settings of a large number of static system configuration and design parameters. Many of them influence runtime performance, and some interact in complex ways, which make the configuration spaces large and complex. It is hard to find high-performing designs and configurations that achieve available performance. The result is that engineers often just accept default settings or decisions made by tools, leading such systems to underperform significantly relative to their potential. To improve the performance of such systems, we addressed three sub-problems.
The first is that we lack adequate concepts, methods, and tools to find high-performing designs in a tradeoff space. To address this problem, we conducted a systematic evaluation of an approach to finding Pareto-optimal system designs. This approach starts with relational logic specification of a system, which is conjoined with a separate specification of the mapping from such specifications to tradeoff spaces. Second, all designs in the space is implemented as a running system. We developed a framework based on Spark to parallelize the profiling such implementations for their performance. Finally, we select and return Pareto-optimal designs. We evaluated our approach in the object-relational mapping (ORM) domain. The results show that we can find schemas for ORM models (with tens of classes and relationships) that significantly outperform those produced by widely used ORM tools. This work demonstrates the potential for formal tradeoff space synthesis and profiling to significantly improve the performance of component-level design problems in industrially meaningful systems.
Second, we lack approaches for developing accurate performance models for highly configurable systems. Having such models could significantly reduce the cost of finding good configurations. This is a hard problem because mappings from configurations to performance are not straightforward and learning such mappings tends to produce inaccurate models. Our approach is to exploit semantic meanings of configurations parameters, such as dependency structure, to eliminate parameter values made irrelevant by the particular settings of other parameters. We evaluated our approach by building predictive performance models for the MapReduce framework, with and without the use of our semantic data cleaning approach. In our experiments, the use of our approach improved model accuracy from under $0.75$ to over $0.88$. These results suggest that semantic cleaning of data provides real value for training performance prediction models for large systems with complex configuration spaces.
The third problem is that find high-performing configurations for systems without performance models because learning such predictive models is costly. Our approach is to employ a direct meta-heuristic search for desired configurations. To save the search cost, we 1) benchmark configurations using \textit{small} datasets as proxies for much larger ones, and 2) validating the usefulness of configurations found for one job on similar ones. To evaluate our approach, we applied it in the domain of big-data systems, particularly Hadoop and Spark. We measured performance improvements achieved by our approach and by baseline approaches: 1) a naive random search strategy; 2) a basic genetic algorithm; 3) and a cutting-edge model-learning-based approach. Our approach improved the performance of five canonical Hadoop jobs by 14\% to 25\%, and 2\% to 17\% for another five Spark jobs, and significantly outperforms the three competing approaches. These experimental results support the claim that our approach has significant potential to improve the performance of industrial big-data systems.
This dissertation solves three problems to improve system performance with novel formal synthesis + large-scale profiling, learning with parameters' semantic meanings, and meta-heuristic searching combined with cost-saving tactics. The results show that we can significantly improve system performance in critical engineering domains, like ORM design and big-data infrastructures tuning.
Committee Members:
Advisors: Kevin Sullivan and Baishakhi Ray
Chair: Alfred Weaver
Other Members: Mary Lou Soffa, Joanne Bechta Dugan, Jonathan Bell (George Mason University)