Knowledge Overlap and Iterative Development in ML Projects: An Information Processing View

Machine Learning (ML) projects encounter significant uncertainty due to the search for potential use cases and the opacity of ML models, which may challenge project efficiency and model effectiveness. Taking an Information Processing (IP) View, we examine how projects can counter these sources of uncertainty with appropriate sources of IP capacity, including iterative development and knowledge overlap between data scientists and domain experts. Survey data from 141 ML project teams shows that iterative development and knowledge overlap in the form of domain experts’ data science knowledge can significantly enhance ML project efficiency. Our interaction analysis shows that iterative development and domain experts’ data science knowledge helps address uncertainty in the business sphere (i.e., requirements uncertainty), while data scientists’ domain knowledge helps address uncertainty in the technical sphere (i.e., inscrutability). We conclude by providing implications for the IS ML literature and practice.