Skip to Main Content
HBS Home
  • About
  • Academic Programs
  • Alumni
  • Faculty & Research
  • Baker Library
  • Giving
  • Harvard Business Review
  • Initiatives
  • News
  • Recruit
  • Map / Directions
Faculty & Research
  • Faculty
  • Research
  • Featured Topics
  • Academic Units
  • …→
  • Harvard Business School→
  • Faculty & Research→
  • Academic Units
    • Academic Units
    • Accounting & Management
    • Business, Government & the International Economy
    • Entrepreneurial Management
    • Finance
    • General Management
    • Marketing
    • Negotiation, Organizations & Markets
    • Organizational Behavior
    • Strategy
    • Technology & Operations Management
    →
  • Technology & Operations Management
    • Technology & Operations Management
    • Faculty
    • Curriculum
    • Seminars & Conferences
    • Awards & Honors
    • Doctoral Students
    →

Technology & Operations Management

Technology & Operations Management

  • Faculty
  • Curriculum
  • Seminars & Conferences
  • Awards & Honors
  • Doctoral Students
Overview Faculty Curriculum Seminars & Conferences Awards & Honors Doctoral Students
    • 2023
    • Working Paper

    Design-Based Confidence Sequences: A General Approach to Risk Mitigation in Online Experimentation

    By: Dae Woong Ham, Michael Lindon, Martin Tingley and Iavor I. Bojinov

    Randomized experiments have become the standard method for companies to evaluate the performance of new products or services. In addition to augmenting managers’ decision-making, experimentation mitigates risk by limiting the proportion of customers exposed to innovation. Since many experiments are on customers arriving sequentially, a potential solution is to allow managers to “peek” at the results when new data becomes available and stop the test if the results are statistically significant. Unfortunately, peeking invalidates the statistical guarantees for standard statistical analysis and leads to uncontrolled type-1 error. Our paper provides valid design-based confidence sequences, sequences of confidence intervals with uniform type-1 error guarantees over time for various sequential experiments in an assumption-light manner. In particular, we focus on finite-sample estimands defined on the study participants as a direct measure of the incurred risks by companies. Our proposed confidence sequences are valid for a large class of experiments, including multi-armbandits, time series, and panel experiments. We further provide a variance reduction technique incorporating modeling assumptions and covariates. Finally, we demonstrate the effectiveness of our proposed approach through a simulation study and three real-world applications from Netflix. Our results show that by using our confidence sequence, harmful experiments could be stopped after only observing a handful of units; for instance, an experiment that Netflix ran on its sign-up page on 30,000 potential customers would have been stopped by our method on the first day before 100 observations.

    • 2023
    • Working Paper

    Design-Based Confidence Sequences: A General Approach to Risk Mitigation in Online Experimentation

    By: Dae Woong Ham, Michael Lindon, Martin Tingley and Iavor I. Bojinov

    Randomized experiments have become the standard method for companies to evaluate the performance of new products or services. In addition to augmenting managers’ decision-making, experimentation mitigates risk by limiting the proportion of customers exposed to innovation. Since many experiments are on customers arriving sequentially, a potential...

    • 2023
    • Working Paper

    Location-Specificity and Geographic Competition for Remote Workers

    By: Thomaz Teodorovicz, Prithwiraj Choudhury and Evan Starr

    The precipitous growth of remote work has given rise to a new phenomenon: geographic competition between localities for the physical presence of remote workers. Remote workers with high general human capital may create value for their new destinations and reverse net talent outflow from smaller cities in middle America and globally. However, localities seeking to attract, retain, and create value from so-called “digital nomads” face significant challenges because such workers may have a low attachment to their new destination. Analogizing these challenges to the problem of creating and capturing value from workers with general human capital, we argue that localities can compete for remote workers by leveraging location-specific attributes which create value for the individual and the locality. We examined these ideas in the context of Tulsa Remote, a program that provides relocation incentives and a bundle of services to increase engagement and embeddedness in Tulsa, Oklahoma. We found that Tulsa Remote increased community engagement, real income, and entrepreneurship of remote workers, benefiting both the community and the individual. Tulsa Remote increased worker’s willingness to stay, and local community engagement is a key driver of this relationship. This work thus suggests that location-specificity enables localities to both create and capture value from remote workers.

    • 2023
    • Working Paper

    Location-Specificity and Geographic Competition for Remote Workers

    By: Thomaz Teodorovicz, Prithwiraj Choudhury and Evan Starr

    The precipitous growth of remote work has given rise to a new phenomenon: geographic competition between localities for the physical presence of remote workers. Remote workers with high general human capital may create value for their new destinations and reverse net talent outflow from smaller cities in middle America and globally. However,...

    • 2023
    • Article

    Exploiting Discovered Regression Discontinuities to Debias Conditioned-on-observable Estimators

    By: Benjamin Jakubowski, Siram Somanchi, Edward McFowland III and Daniel B. Neill

    Regression discontinuity (RD) designs are widely used to estimate causal effects in the absence of a randomized experiment. However, standard approaches to RD analysis face two significant limitations. First, they require a priori knowledge of discontinuities in treatment. Second, they yield doubly-local treatment effect estimates, and fail to provide more general causal effect estimates away from the discontinuity. To address these limitations, we introduce a novel method for automatically detecting RDs at scale, integrating information from multiple discovered discontinuities with an observational estimator, and extrapolating away from discovered, local RDs. We demonstrate the performance of our method on two synthetic datasets, showing improved performance compared to direct use of an observational estimator, direct extrapolation of RD estimates, and existing methods for combining multiple causal effect estimates. Finally, we apply our novel method to estimate spatially heterogeneous treatment effects in the context of a recent economic development problem.

    • 2023
    • Article

    Exploiting Discovered Regression Discontinuities to Debias Conditioned-on-observable Estimators

    By: Benjamin Jakubowski, Siram Somanchi, Edward McFowland III and Daniel B. Neill

    Regression discontinuity (RD) designs are widely used to estimate causal effects in the absence of a randomized experiment. However, standard approaches to RD analysis face two significant limitations. First, they require a priori knowledge of discontinuities in treatment. Second, they yield doubly-local treatment effect estimates, and fail to...

About the Unit

As the world of operations has changed, so have interests and priorities within the Unit. Historically, the TOM Unit focused on manufacturing and the development of physical products. Over the past several years, we have expanded our research, course development, and course offerings to encompass new issues in information technology, supply chains, and service industries.

The field of TOM is concerned with the design, management, and improvement of operating systems and processes. As we seek to understand the challenges confronting firms competing in today's demanding environment, the focus of our work has broadened to include the multiple activities comprising a firm's "operating core":

  • the multi-function, multi-firm system that includes basic research, design, engineering, product and process development and production of goods and services within individual operating units;
  • the networks of information and material flows that tie operating units together and the systems that support these networks;
  • the distribution and delivery of goods and services to customers.

Recent Publications

Design-Based Confidence Sequences: A General Approach to Risk Mitigation in Online Experimentation

By: Dae Woong Ham, Michael Lindon, Martin Tingley and Iavor I. Bojinov
  • 2023 |
  • Working Paper |
  • Faculty Research
Randomized experiments have become the standard method for companies to evaluate the performance of new products or services. In addition to augmenting managers’ decision-making, experimentation mitigates risk by limiting the proportion of customers exposed to innovation. Since many experiments are on customers arriving sequentially, a potential solution is to allow managers to “peek” at the results when new data becomes available and stop the test if the results are statistically significant. Unfortunately, peeking invalidates the statistical guarantees for standard statistical analysis and leads to uncontrolled type-1 error. Our paper provides valid design-based confidence sequences, sequences of confidence intervals with uniform type-1 error guarantees over time for various sequential experiments in an assumption-light manner. In particular, we focus on finite-sample estimands defined on the study participants as a direct measure of the incurred risks by companies. Our proposed confidence sequences are valid for a large class of experiments, including multi-armbandits, time series, and panel experiments. We further provide a variance reduction technique incorporating modeling assumptions and covariates. Finally, we demonstrate the effectiveness of our proposed approach through a simulation study and three real-world applications from Netflix. Our results show that by using our confidence sequence, harmful experiments could be stopped after only observing a handful of units; for instance, an experiment that Netflix ran on its sign-up page on 30,000 potential customers would have been stopped by our method on the first day before 100 observations.
Keywords: Performance Evaluation; Research and Development; Analytics and Data Science; Consumer Behavior
Citation
Read Now
Related
Ham, Dae Woong, Michael Lindon, Martin Tingley, and Iavor I. Bojinov. "Design-Based Confidence Sequences: A General Approach to Risk Mitigation in Online Experimentation." Harvard Business School Working Paper, No. 23-070, May 2023.

Location-Specificity and Geographic Competition for Remote Workers

By: Thomaz Teodorovicz, Prithwiraj Choudhury and Evan Starr
  • 2023 |
  • Working Paper |
  • Faculty Research
The precipitous growth of remote work has given rise to a new phenomenon: geographic competition between localities for the physical presence of remote workers. Remote workers with high general human capital may create value for their new destinations and reverse net talent outflow from smaller cities in middle America and globally. However, localities seeking to attract, retain, and create value from so-called “digital nomads” face significant challenges because such workers may have a low attachment to their new destination. Analogizing these challenges to the problem of creating and capturing value from workers with general human capital, we argue that localities can compete for remote workers by leveraging location-specific attributes which create value for the individual and the locality. We examined these ideas in the context of Tulsa Remote, a program that provides relocation incentives and a bundle of services to increase engagement and embeddedness in Tulsa, Oklahoma. We found that Tulsa Remote increased community engagement, real income, and entrepreneurship of remote workers, benefiting both the community and the individual. Tulsa Remote increased worker’s willingness to stay, and local community engagement is a key driver of this relationship. This work thus suggests that location-specificity enables localities to both create and capture value from remote workers.
Keywords: Remote Work; Human Capital; Geographic Location; Civil Society or Community; Motivation and Incentives
Citation
Read Now
Related
Teodorovicz, Thomaz, Prithwiraj Choudhury, and Evan Starr. "Location-Specificity and Geographic Competition for Remote Workers." Harvard Business School Working Paper, No. 23-071, May 2023.

Exploiting Discovered Regression Discontinuities to Debias Conditioned-on-observable Estimators

By: Benjamin Jakubowski, Siram Somanchi, Edward McFowland III and Daniel B. Neill
  • 2023 |
  • Article |
  • Journal of Machine Learning Research
Regression discontinuity (RD) designs are widely used to estimate causal effects in the absence of a randomized experiment. However, standard approaches to RD analysis face two significant limitations. First, they require a priori knowledge of discontinuities in treatment. Second, they yield doubly-local treatment effect estimates, and fail to provide more general causal effect estimates away from the discontinuity. To address these limitations, we introduce a novel method for automatically detecting RDs at scale, integrating information from multiple discovered discontinuities with an observational estimator, and extrapolating away from discovered, local RDs. We demonstrate the performance of our method on two synthetic datasets, showing improved performance compared to direct use of an observational estimator, direct extrapolation of RD estimates, and existing methods for combining multiple causal effect estimates. Finally, we apply our novel method to estimate spatially heterogeneous treatment effects in the context of a recent economic development problem.
Keywords: Regression Discontinuity Design; Analytics and Data Science; AI and Machine Learning
Citation
Read Now
Related
Jakubowski, Benjamin, Siram Somanchi, Edward McFowland III, and Daniel B. Neill. "Exploiting Discovered Regression Discontinuities to Debias Conditioned-on-observable Estimators." Journal of Machine Learning Research 24, no. 133 (2023): 1–57.

CMA CGM: Reducing the Carbon Footprint of Container Shipping

By: Willy C. Shih and Emilie Billaud
  • May 2023 |
  • Case |
  • Faculty Research
Marine transport is the most cost-effective way to move large volumes over long distances, and container shipping is the backbone of international trade in goods. Yet shipping contributed 3% of worldwide greenhouse gas emissions, and the deep-sea segment, which included long distance trade lanes such as Asia to Northern Europe and Asia to North America, warranted special focus because they accounted for 80% of maritime transport’s total emissions in 2019. New International Maritime Organization regulations that came into force in January 2023 mandated the annual calculation and grading of each ship of more than 5,000 deadweight tons. Vessels that received a grade of A, B, or C were compliant, while those graded D or E had time limits for getting back into compliance or removal from service. More significantly, the standards for grading required annual improvements in efficiency. This mean that a brand-new vessel built with the latest technology that was initially graded A could over time become graded E and no longer be legally operable if no upgrades were made. This case afford students the opportunity to consider different fuel and operational choices and calculate their impact on greenhouse gas emissions and ship grading. It exposes some of the choices that an operator might choose to make.
Keywords: Container Shipping; Logistic Regression; Supply Chain; Trade Links; Decarbonization; Environmental Strategies; Environmental Impact; Globalization; Trade; Environmental Regulation; Shipping Industry; European Union; Asia; North America
Citation
Educators
Related
Shih, Willy C., and Emilie Billaud. "CMA CGM: Reducing the Carbon Footprint of Container Shipping." Harvard Business School Case 623-006, May 2023.

Sian Flowers: Fresher by Sea - Video Supplement

By: Willy C. Shih
  • May 2023 |
  • Supplement |
  • Faculty Research
The setting for this case is the Sian Flowers, a company headquartered in Kitengela, Kenya that exports roses to predominantly Europe. Because cut flowers have a limited shelf life and consumers want them to retain their appearance for as long as possible, Sian or its distributors used international air cargo to transport them to Amsterdam, where they were sold at auction or trucked to markets across Europe. The Covid-19 pandemic caused huge increases in the cost of shipping, so Sian launched experiments to ship roses by ocean using refrigerated containers. Chris Kulei, the Executive Director, was interested in not only the potential costs savings, but whether he could also market the reduced carbon footprint.
Keywords: Supply Chain; Supply Chains; Sustainability; Sustainable Agriculture; Sustainability Reporting; Carbon Emissions; Supply Chain Management; Quality; Ship Transportation; Cost Management; Agriculture and Agribusiness Industry; Africa; Kenya; Netherlands; Europe
Citation
Purchase
Related
Shih, Willy C. "Sian Flowers: Fresher by Sea - Video Supplement." Harvard Business School Multimedia/Video Supplement 623-713, May 2023.

Regulatory Submission Characteristics and Recalls of Medical Devices Receiving 510(k) Clearance

By: Alexander O. Everhart, Yi Zhu and Ariel D. Stern
  • May 9, 2023 |
  • Response |
  • JAMA, the Journal of the American Medical Association
Keywords: Product Design; Safety; Medical Devices and Supplies Industry
Citation
Find at Harvard
Read Now
Purchase
Related
Everhart, Alexander O., Yi Zhu, and Ariel D. Stern. "Regulatory Submission Characteristics and Recalls of Medical Devices Receiving 510(k) Clearance." JAMA, the Journal of the American Medical Association 329, no. 18 (May 9, 2023): 1609–1610. (Reply to original paper.)

Setting Gendered Expectations? Recruiter Outreach Bias in Online Tech Training Programs

By: Jacqueline N. Lane, Karim R. Lakhani and Roberto Fernandez
  • 2023 |
  • Working Paper |
  • Faculty Research
Competence development in digital technologies, analytics, and artificial intelligence is increasingly important to all types of organizations and their workforce. Universities and corporations are investing heavily in developing training programs, at all tenure levels, to meet the new skills needs. However, there is a risk that the new set of lucrative opportunities for employees in these tech-heavy fields will be biased against diverse demographic groups like women. Although much research has examined the experiences of women in science, technology, engineering, and mathematics (STEM) fields and occupations, less understood is the extent to which gender stereotypes influence recruiters’ perceptions and evaluations of individuals who are deciding whether to apply to STEM training programs. These behaviors are typically unobserved because they occur prior to the application interface. We address this question by investigating recruiters’ initial outreach decisions to over 166,000 prospective students who have expressed interest in applying to a mid-career level online tech training program in business analytics. Using data on the recruiters’ communications, our results indicate that recruiters are less likely to initiate contact with female than male prospects and search for additional signals of quality from female prospects before contacting them. We also find evidence that recruiters are more likely to base initial outreach activities on prospect gender when they have higher workloads and limited attention. We conclude with a discussion of the implications of this research for our understanding of how screening and selection decisions prior to the application interface may undermine organizational efforts to achieve gender equality and diversity as well as the potential for demand-side interventions to mitigate these gender disparities.
Keywords: STEM; Selection and Staffing; Gender; Prejudice and Bias; Training; Equality and Inequality; Competency and Skills
Citation
Read Now
Related
Lane, Jacqueline N., Karim R. Lakhani, and Roberto Fernandez. "Setting Gendered Expectations? Recruiter Outreach Bias in Online Tech Training Programs." Harvard Business School Working Paper, No. 23-066, April 2023. (Accepted by Organization Science.)

The Hidden Cost of Coordination: Evidence from Last-Mile Delivery Services

By: Natalie Epstein, Santiago Gallino and Antonio Moreno
  • 2023 |
  • Working Paper |
  • Faculty Research
Problem definition: Communication and customer interaction design have been used as elements to improve customer satisfaction and future purchasing behavior, but little is known about how they can be used as levers to improve operational efficiency. Methodology/results: We partner with a last-mile delivery company and, using natural and field experiments, explore the effects that communication channels and customer interfaces have on a key metric of the process: failed deliveries. We show that communication can be used as a lever to improve operational outcomes -- when no communication is possible, failed deliveries increase. We find that using more salient communication channels increases the reach of the information. However, against expectations, it increases the likelihood of failed deliveries. The reason is that customers try, unsuccessfully, to coordinate the delivery times and locations. We explore the value of self-service information gathering. While we find no effect on the process outcomes, we show that these channels can be used to identify customers that are harder to serve. Managerial implications: Our work shows the relevance of communication channels as levers of operational performance. We provide evidence that these channels can be used to manage customers’ expectations and improve operational processes. We also show the importance of alignment between the communication channel and the operational process, deciding the type of channel is key to shape customers behavior and expectations. Understanding and supporting channel norms and expectations is essential to achieve such alignment.
Keywords: Customer Satisfaction; Consumer Behavior; Logistics; Communication
Citation
Related
Epstein, Natalie, Santiago Gallino, and Antonio Moreno. "The Hidden Cost of Coordination: Evidence from Last-Mile Delivery Services." Working Paper, May 2023.
More Publications

In the News

    • 07 Jun 2023
    • Harvard Business School

    My One Case: MBA Class of 2023 Looks Back

    Re: Ryan Buell, Anjali M. Bhatt, Jeffrey Rayport, Ting Zhang & Henry McGee
    • 02 Jun 2023
    • MarketWatch: Best New Ideas in Money

    Podcast: Would $10,000 convince you to move to a new city?

    Re: Prithwiraj Choudhury
    • 31 May 2023
    • Conde Nast Traveler

    Digital Nomads: Are They Crowding Cities or Reviving Them?

    Re: Prithwiraj Choudhury
→More Faculty News

HBS Working Knowledge

    • 01 Jun 2023

    A Nike Executive Hid His Criminal Past to Turn His Life Around. What If He Didn't Have To?

    by Dina Gerdeman
    • 31 May 2023

    Why Business Leaders Need to Hear Larry Miller's Story

    Re: Hise O. Gibson
    • 31 May 2023

    From Prison Cell to Nike’s C-Suite: The Journey of Larry Miller

    by Jamal Meneide
→More Working Knowledge Articles

Harvard Business Publishing

    • July–August 2013
    • Article

    Leadership Lessons from the Chilean Mine Rescue

    By: Faaiza Rashid, Amy C. Edmondson and Herman B. Leonard
    • April 2023
    • Case

    Fizzy Fusion: When Data-Driven Decision Making Failed

    By: Michael Parzen, Eddie Lin, Douglas Ng and Jessie Li
    • 2012
    • Book

    Producing Prosperity: Why America Needs a Manufacturing Renaissance

    By: Gary P. Pisano and Willy Shih
→More Harvard Business Publishing

Seminars & Conferences

There are no upcoming events.

→More Seminars & Conferences

Faculty Positions

Harvard Business School seeks candidates in all fields for full time positions. Candidates with outstanding records in PhD or DBA programs are encouraged to apply.
→Learn More

Contact Information

Technology & Operations Management Unit
Harvard Business School
Morgan Hall
Soldiers Field
Boston, MA 02163
tomunit@hbs.edu

ǁ
Campus Map
Harvard Business School
Soldiers Field
Boston, MA 02163
→Map & Directions
→More Contact Information
  • Make a Gift
  • Site Map
  • Jobs
  • Harvard University
  • Trademarks
  • Policies
  • Accessibility
  • Digital Accessibility
Copyright © President & Fellows of Harvard College