2020
Working Paper
HBS Working Paper Series

Digital Experimentation and Startup Performance: Evidence from A/B Testing

By: Rembrand Koning, Sharique Hasan and Aaron Chatterji

Format:Print
| Language:English
| Pages:77

Abstract

Recent work argues that experimentation is the appropriate framework for entrepreneurial strategy. We investigate this proposition by exploiting the time-varying adoption of A/B testing technology, which has drastically reduced the cost of experimentally testing business ideas. This paper provides the first evidence on how digital experimentation affects a large sample of high-technology startups using data that tracks their growth, technology use, and product launches. Despite its prominence in the business press, relatively few firms have adopted A/B testing tools. However, among those that do, we find increased performance on several critical dimensions, including page views, code changes, and new product features. These results are robust to our use of instrumental variables and synthetic control models, where we leverage Google’s launch of a new A/B testing tool. Our results also show that the value of A/B testing increases over time and is positively related to tail outcomes, with adopting firms both scaling and failing faster. Our results inform the emerging literature on entrepreneurial strategy and illustrate the growing importance of digitization and data-driven decision-making.

Keywords

Experimentation; A/B Testing; Data-driven Decision-making; Entrepreneurship; Strategy; Business Startups; Technology; Performance

Citation

Koning, Rembrand, Sharique Hasan, and Aaron Chatterji. "Digital Experimentation and Startup Performance: Evidence from A/B Testing." Harvard Business School Working Paper, No. 20-018, August 2019. (Revised September 2020. SSRN Working Paper Series, No. 3440291, August 2019)