Technology & Operations Management
-
- 2023
- Working Paper
Design-Based Confidence Sequences: A General Approach to Risk Mitigation in Online Experimentation
By: Dae Woong Ham, Michael Lindon, Martin Tingley and Iavor I. BojinovRandomized experiments have become the standard method for companies to evaluate the performance of new products or services. In addition to augmenting managers’ decision-making, experimentation mitigates risk by limiting the proportion of customers exposed to innovation. Since many experiments are on customers arriving sequentially, a potential solution is to allow managers to “peek” at the results when new data becomes available and stop the test if the results are statistically significant. Unfortunately, peeking invalidates the statistical guarantees for standard statistical analysis and leads to uncontrolled type-1 error. Our paper provides valid design-based confidence sequences, sequences of confidence intervals with uniform type-1 error guarantees over time for various sequential experiments in an assumption-light manner. In particular, we focus on finite-sample estimands defined on the study participants as a direct measure of the incurred risks by companies. Our proposed confidence sequences are valid for a large class of experiments, including multi-armbandits, time series, and panel experiments. We further provide a variance reduction technique incorporating modeling assumptions and covariates. Finally, we demonstrate the effectiveness of our proposed approach through a simulation study and three real-world applications from Netflix. Our results show that by using our confidence sequence, harmful experiments could be stopped after only observing a handful of units; for instance, an experiment that Netflix ran on its sign-up page on 30,000 potential customers would have been stopped by our method on the first day before 100 observations.
- 2023
- Working Paper
Design-Based Confidence Sequences: A General Approach to Risk Mitigation in Online Experimentation
By: Dae Woong Ham, Michael Lindon, Martin Tingley and Iavor I. BojinovRandomized experiments have become the standard method for companies to evaluate the performance of new products or services. In addition to augmenting managers’ decision-making, experimentation mitigates risk by limiting the proportion of customers exposed to innovation. Since many experiments are on customers arriving sequentially, a potential...
-
- 2023
- Working Paper
Location-Specificity and Geographic Competition for Remote Workers
By: Thomaz Teodorovicz, Prithwiraj Choudhury and Evan StarrThe precipitous growth of remote work has given rise to a new phenomenon: geographic competition between localities for the physical presence of remote workers. Remote workers with high general human capital may create value for their new destinations and reverse net talent outflow from smaller cities in middle America and globally. However, localities seeking to attract, retain, and create value from so-called “digital nomads” face significant challenges because such workers may have a low attachment to their new destination. Analogizing these challenges to the problem of creating and capturing value from workers with general human capital, we argue that localities can compete for remote workers by leveraging location-specific attributes which create value for the individual and the locality. We examined these ideas in the context of Tulsa Remote, a program that provides relocation incentives and a bundle of services to increase engagement and embeddedness in Tulsa, Oklahoma. We found that Tulsa Remote increased community engagement, real income, and entrepreneurship of remote workers, benefiting both the community and the individual. Tulsa Remote increased worker’s willingness to stay, and local community engagement is a key driver of this relationship. This work thus suggests that location-specificity enables localities to both create and capture value from remote workers.
- 2023
- Working Paper
Location-Specificity and Geographic Competition for Remote Workers
By: Thomaz Teodorovicz, Prithwiraj Choudhury and Evan StarrThe precipitous growth of remote work has given rise to a new phenomenon: geographic competition between localities for the physical presence of remote workers. Remote workers with high general human capital may create value for their new destinations and reverse net talent outflow from smaller cities in middle America and globally. However,...
-
- 2023
- Article
Exploiting Discovered Regression Discontinuities to Debias Conditioned-on-observable Estimators
By: Benjamin Jakubowski, Siram Somanchi, Edward McFowland III and Daniel B. NeillRegression discontinuity (RD) designs are widely used to estimate causal effects in the absence of a randomized experiment. However, standard approaches to RD analysis face two significant limitations. First, they require a priori knowledge of discontinuities in treatment. Second, they yield doubly-local treatment effect estimates, and fail to provide more general causal effect estimates away from the discontinuity. To address these limitations, we introduce a novel method for automatically detecting RDs at scale, integrating information from multiple discovered discontinuities with an observational estimator, and extrapolating away from discovered, local RDs. We demonstrate the performance of our method on two synthetic datasets, showing improved performance compared to direct use of an observational estimator, direct extrapolation of RD estimates, and existing methods for combining multiple causal effect estimates. Finally, we apply our novel method to estimate spatially heterogeneous treatment effects in the context of a recent economic development problem.
- 2023
- Article
Exploiting Discovered Regression Discontinuities to Debias Conditioned-on-observable Estimators
By: Benjamin Jakubowski, Siram Somanchi, Edward McFowland III and Daniel B. NeillRegression discontinuity (RD) designs are widely used to estimate causal effects in the absence of a randomized experiment. However, standard approaches to RD analysis face two significant limitations. First, they require a priori knowledge of discontinuities in treatment. Second, they yield doubly-local treatment effect estimates, and fail to...
About the Unit
As the world of operations has changed, so have interests and priorities within the Unit. Historically, the TOM Unit focused on manufacturing and the development of physical products. Over the past several years, we have expanded our research, course development, and course offerings to encompass new issues in information technology, supply chains, and service industries.
The field of TOM is concerned with the design, management, and improvement of operating systems and processes. As we seek to understand the challenges confronting firms competing in today's demanding environment, the focus of our work has broadened to include the multiple activities comprising a firm's "operating core":
- the multi-function, multi-firm system that includes basic research, design, engineering, product and process development and production of goods and services within individual operating units;
- the networks of information and material flows that tie operating units together and the systems that support these networks;
- the distribution and delivery of goods and services to customers.
Recent Publications
Design-Based Confidence Sequences: A General Approach to Risk Mitigation in Online Experimentation
- 2023 |
- Working Paper |
- Faculty Research
Location-Specificity and Geographic Competition for Remote Workers
- 2023 |
- Working Paper |
- Faculty Research
Exploiting Discovered Regression Discontinuities to Debias Conditioned-on-observable Estimators
- 2023 |
- Article |
- Journal of Machine Learning Research
CMA CGM: Reducing the Carbon Footprint of Container Shipping
- May 2023 |
- Case |
- Faculty Research
Sian Flowers: Fresher by Sea - Video Supplement
- May 2023 |
- Supplement |
- Faculty Research
Regulatory Submission Characteristics and Recalls of Medical Devices Receiving 510(k) Clearance
- May 9, 2023 |
- Response |
- JAMA, the Journal of the American Medical Association
Setting Gendered Expectations? Recruiter Outreach Bias in Online Tech Training Programs
- 2023 |
- Working Paper |
- Faculty Research
The Hidden Cost of Coordination: Evidence from Last-Mile Delivery Services
- 2023 |
- Working Paper |
- Faculty Research
Harvard Business Publishing
Seminars & Conferences
There are no upcoming events.