Data-efficient performance learning for configurable systems

Jianmei Guo, Dingyu Yang, Norbert Siegmund, Sven Apel, Atri Sarkar, Pavel Valov, Krzysztof Czarnecki, Andrzej Wasowski, Huiqun Yu

Research output: Journal Article or Conference Article in JournalJournal articleResearchpeer-review

Abstract

Many software systems today are configurable, offering customization of functionality by feature selection. Understanding how performance varies in terms of feature selection is key for selecting appropriate configurations that meet a set of given requirements. Due to a huge configuration space and the possibly high cost of performance measurement, it is usually not feasible to explore the entire configuration space of a configurable system exhaustively. It is thus a major challenge to accurately predict performance based on a small sample of measured system variants. To address this challenge, we propose a data-efficient learning approach, called DECART, that combines several techniques of machine learning and statistics for performance prediction of configurable systems. DECART builds, validates, and determines a prediction model based on an available sample of measured system variants. Empirical results on 10 real-world configurable systems demonstrate the effectiveness and practicality of DECART. In particular, DECART achieves a prediction accuracy of 90% or higher based on a small sample, whose size is linear in the number of features. In addition, we propose a sample quality metric and introduce a quantitative analysis of the quality of a sample for performance prediction.
Original languageEnglish
JournalEmpirical Software Engineering
Number of pages42
ISSN1382-3256
DOIs
Publication statusPublished - Nov 2017

Keywords

  • Parameter tuning
  • Model selection
  • Regression
  • Configurable systems
  • Performance prediction

Fingerprint

Dive into the research topics of 'Data-efficient performance learning for configurable systems'. Together they form a unique fingerprint.

Cite this