Developing Theory Using Machine Learning Methods
We describe how to employ machine learning (ML) methods in theory development. Compared to traditional causal inference methods, ML methods make far fewer a priori assumptions about the functional form of the underlying model that best represents the data. Thus researchers could use such methods to explore novel and robust patterns in data, allowing the researcher to engage in inductive or abductive theory building. ML’s strengths include replicable identification of novel patterns in data. ML methods also address several issues raised by scholars pertinent to the norms of empirical research in the fields of strategy and management (such as “p-hacking” and confounding local effects with global effects). We provide a step-by-step roadmap that illustrates how to use four ML methods (decision trees, random forests, K-nearest neighbors, and neural networks) to reveal patterns in data that could be used for theory building. We also illustrate how ML methods can illuminate interactions and non-linear effects better than traditional methods do. In summary, ML methods could serve as a complement to existing theory-creating methods, such as multiple-case inductive studies, as well as traditional methods of causal inference.
Keywords: Machine learning; theory building; induction; Decision Trees; random forests; k-nearest neighbors; neural network; p-hacking;
Citation:
Choudhury, Prithwiraj, Ryan Allen, and Michael G. Endres. "Developing Theory Using Machine Learning Methods." Harvard Business School Working Paper, No. 19-032, September 2018. (Revised October 2018.) View Details