Variability Bugs in Highly Configurable Systems: A Qualitative Analysis

Variability-sensitive verification pursues effective analysis of the exponentially many variants in number of features of a program family. Several variability-aware techniques have been proposed, but researchers still lack examples of concrete bugs induced by variability, occurring in real large-scale systems. A collection of real world bugs is needed to evaluate tool implementations of variability-sensitive analyses by testing them on real bugs. We present a qualitative study of 98 diverse variability bugs collected from bug-fixing commits in the Apache, BusyBox, Linux kernel and Marlin repositories. We analyze each of the bugs, and record the results in a database. For each bug, we create a self-contained simplified C99 version and a simplified patch, in order to help researchers who are not experts on these subject studies to understand them, so that they can use it for evaluation of their tools. In addition, we provide single-function versions of the bugs, which are useful for evaluating intra-procedural analyses. A Web-based User Interface for the database allows to conveniently browse and visualize the collection of bugs. Our study provides insights into the nature and occurrence of variability bugs in four highly-configurable systems implemented in C, and shows in what ways variability affects and increases the complexity of software bugs. Software bugs appear across different kinds of components, features, and bug categories. Variability affects complexity of bugs by feature interactions, that may span multiple subsystems; through hidden configuration dependent definitions of functions, variables, and types; extending the bug locations from code to also feature models and code mappings, and to modeling languages in which these are expressed. We also find many bugs that are caused by disabling a feature (negative feature interaction), which leads us to suggest the one-disabled strategy for configuration sampling.