• 2023
• Working Paper

Randomized experiments have become the standard method for companies to evaluate the performance of new products or services. In addition to augmenting managers’ decision-making, experimentation mitigates risk by limiting the proportion of customers exposed to innovation. Since many experiments are on customers arriving sequentially, a potential solution is to allow managers to “peek” at the results when new data becomes available and stop the test if the results are statistically significant. Unfortunately, peeking invalidates the statistical guarantees for standard statistical analysis and leads to uncontrolled type-1 error. Our paper provides valid design-based confidence sequences, sequences of confidence intervals with uniform type-1 error guarantees over time for various sequential experiments in an assumption-light manner. In particular, we focus on finite-sample estimands defined on the study participants as a direct measure of the incurred risks by companies. Our proposed confidence sequences are valid for a large class of experiments, including multi-armbandits, time series, and panel experiments. We further provide a variance reduction technique incorporating modeling assumptions and covariates. Finally, we demonstrate the effectiveness of our proposed approach through a simulation study and three real-world applications from Netflix. Our results show that by using our confidence sequence, harmful experiments could be stopped after only observing a handful of units; for instance, an experiment that Netflix ran on its sign-up page on 30,000 potential customers would have been stopped by our method on the first day before 100 observations.

• 2023
• Working Paper

The precipitous growth of remote work has given rise to a new phenomenon: geographic competition between localities for the physical presence of remote workers. Remote workers with high general human capital may create value for their new destinations and reverse net talent outflow from smaller cities in middle America and globally. However, localities seeking to attract, retain, and create value from so-called “digital nomads” face significant challenges because such workers may have a low attachment to their new destination. Analogizing these challenges to the problem of creating and capturing value from workers with general human capital, we argue that localities can compete for remote workers by leveraging location-specific attributes which create value for the individual and the locality. We examined these ideas in the context of Tulsa Remote, a program that provides relocation incentives and a bundle of services to increase engagement and embeddedness in Tulsa, Oklahoma. We found that Tulsa Remote increased community engagement, real income, and entrepreneurship of remote workers, benefiting both the community and the individual. Tulsa Remote increased worker’s willingness to stay, and local community engagement is a key driver of this relationship. This work thus suggests that location-specificity enables localities to both create and capture value from remote workers.

• 2023
• Article

Regression discontinuity (RD) designs are widely used to estimate causal effects in the absence of a randomized experiment. However, standard approaches to RD analysis face two significant limitations. First, they require a priori knowledge of discontinuities in treatment. Second, they yield doubly-local treatment effect estimates, and fail to provide more general causal effect estimates away from the discontinuity. To address these limitations, we introduce a novel method for automatically detecting RDs at scale, integrating information from multiple discovered discontinuities with an observational estimator, and extrapolating away from discovered, local RDs. We demonstrate the performance of our method on two synthetic datasets, showing improved performance compared to direct use of an observational estimator, direct extrapolation of RD estimates, and existing methods for combining multiple causal effect estimates. Finally, we apply our novel method to estimate spatially heterogeneous treatment effects in the context of a recent economic development problem.