Publications

Publications

Filter Results : (34) Arrow Down

Show Results For

Experimental Methods Remove Experimental Methods →

Page 1 of 34 Results →

Are you looking for?

→ Search All HBS Web
  • 2023
  • Working Paper

Design-Based Confidence Sequences: A General Approach to Risk Mitigation in Online Experimentation
By: Dae Woong Ham, Michael Lindon, Martin Tingley and Iavor I Bojinov
Randomized experiments have become the standard method for companies to evaluate the performance of new products or services. In addition to augmenting managers’ decision-making, experimentation mitigates risk by limiting the proportion of customers exposed to...  View Details
Related
  • 2023
  • Working Paper

Nailing Prediction: Experimental Evidence on the Value of Tools in Predictive Model Development
By: Daniel Yue, Paul Hamilton and Iavor Bojinov
Predictive model development is understudied despite its centrality in modern artificial intelligence and machine learning business applications. Although prior discussions highlight advances in methods (along the dimensions of data, computing power, and algorithms)...  View Details
Keywords: Analytics and Data Science
Related
  • 2022
  • Article

Data Poisoning Attacks on Off-Policy Evaluation Methods
By: Elita Lobo, Harvineet Singh, Marek Petrik, Cynthia Rudin and Himabindu Lakkaraju
Off-policy Evaluation (OPE) methods are a crucial tool for evaluating policies in high-stakes domains such as healthcare, where exploration is often infeasible, unethical, or expensive. However, the extent to which such methods can be trusted under adversarial threats...  View Details
Keywords: Analytics and Data Science; Cybersecurity; Mathematical Methods
Related
  • 2022
  • Article

Towards Robust Off-Policy Evaluation via Human Inputs
By: Harvineet Singh, Shalmali Joshi, Finale Doshi-Velez and Himabindu Lakkaraju
Off-policy Evaluation (OPE) methods are crucial tools for evaluating policies in high-stakes domains such as healthcare, where direct deployment is often infeasible, unethical, or expensive. When deployment environments are expected to undergo changes (that is, dataset...  View Details
Keywords: Analytics and Data Science; Research
Related
  • 2022
  • Conference Presentation

Towards the Unification and Robustness of Post hoc Explanation Methods
By: Sushant Agarwal, Shahin Jabbari, Chirag Agarwal, Sohini Upadhyay, Steven Wu and Himabindu Lakkaraju
As machine learning black boxes are increasingly being deployed in critical domains such as healthcare and criminal justice, there has been a growing emphasis on developing techniques for explaining these black boxes in a post hoc manner. In this work, we analyze two...  View Details
Keywords: AI and Machine Learning
Related
  • Article

Reliable Post hoc Explanations: Modeling Uncertainty in Explainability
By: Dylan Slack, Sophie Hilgard, Sameer Singh and Himabindu Lakkaraju
As black box explanations are increasingly being employed to establish model credibility in high stakes settings, it is important to ensure that these explanations are accurate and reliable. However, prior work demonstrates that explanations generated by...  View Details
Keywords: Black Box Explanations; Bayesian Modeling; Decision Making; Risk and Uncertainty; Information Technology
Related
  • Article

Towards the Unification and Robustness of Perturbation and Gradient Based Explanations
By: Sushant Agarwal, Shahin Jabbari, Chirag Agarwal, Sohini Upadhyay, Steven Wu and Himabindu Lakkaraju
As machine learning black boxes are increasingly being deployed in critical domains such as healthcare and criminal justice, there has been a growing emphasis on developing techniques for explaining these black boxes in a post hoc manner. In this work, we analyze two...  View Details
Keywords: Machine Learning; Black Box Explanations; Decision Making; Forecasting and Prediction; Information Technology
Related
  • 2021
  • Working Paper

Population Interference in Panel Experiments
By: Iavor I Bojinov, Kevin Wu Han and Guillaume Basse
The phenomenon of population interference, where a treatment assigned to one experimental unit affects another experimental unit's outcome, has received considerable attention in standard randomized experiments. The complications produced by population interference in...  View Details
Keywords: Finite Population; Potential Outcomes; Dynamic Causal Effects; Mathematical Methods
Related
  • Article

Beyond Individualized Recourse: Interpretable and Interactive Summaries of Actionable Recourses
By: Kaivalya Rawal and Himabindu Lakkaraju
As predictive models are increasingly being deployed in high-stakes decision-making, there has been a lot of interest in developing algorithms which can provide recourses to affected individuals. While developing such tools is important, it is even more critical to...  View Details
Keywords: Predictive Models; Decision Making; Framework; Mathematical Methods
Related
  • 2020
  • Working Paper

Design and Analysis of Switchback Experiments
By: Iavor I Bojinov, David Simchi-Levi and Jinglong Zhao
In switchback experiments, a firm sequentially exposes an experimental unit to a random treatment, measures its response, and repeats the procedure for several periods to determine which treatment leads to the best outcome. Although practitioners have widely adopted...  View Details
Keywords: Switchback Experiments; Design; Analysis; Mathematical Methods
Related
  • August 2020 (Revised September 2020)
  • Technical Note

Assessing Prediction Accuracy of Machine Learning Models
By: Michael W. Toffel, Natalie Epstein, Kris Ferreira and Yael Grushka-Cockayne
The note introduces a variety of methods to assess the accuracy of machine learning prediction models. The note begins by briefly introducing machine learning, overfitting, training versus test datasets, and cross validation. The following accuracy metrics and tools...  View Details
Keywords: Machine Learning; Statistics; Econometric Analyses; Experimental Methods; Data Analysis; Data Analytics; Forecasting and Prediction; Analytics and Data Science; Analysis; Mathematical Methods
Related
  • August 2020
  • Technical Note

Comparing Two Groups: Sampling and t-Testing
By: Iavor I Bojinov, Chiara Farronato, Yael Grushka-Cockayne, Willy C. Shih and Michael W. Toffel
This note describes sampling and t-tests, two fundamental statistical concepts.  View Details
Keywords: Statistics; Econometric Analyses; Experimental Methods; Data Analysis; Data Analytics; Analytics and Data Science; Analysis; Surveys; Mathematical Methods
Related
  • April 2020
  • Article

Field Comparisons of Incentive-Compatible Preference Elicitation Techniques
By: Shawn A. Cole, A. Nilesh Fernando, Daniel Stein and Jeremy Tobacman
Knowledge of consumer demand is important for firms, policy makers, and economists. One common tool for incentive-compatible demand elicitation, the Becker-DeGroot-Marschak (BDM) mechanism, has been widely used in laboratory settings but rarely evaluated for...  View Details
Keywords: Incentive-compatible Elicitation; Experimental Methods; Weather Insurance; Rainfall Insurance; Agricultural Extension; Demand and Consumers
Related
  • March 2020 (Revised March 2023)
  • Module Note

The Role of Experiments in Organizations
By: Michael Luca
This note outlines the structure and content of a four-class module—The Role of Experiments in Organizations—that is designed to introduce students to the role of experimental methods in managerial decisions.  View Details
Keywords: Experiments; Experimental Methods; Decision Making; Information
Related
  • Article

Statistical Physics of Human Cooperation
By: Matjaž Perc, Jillian J. Jordan, David G. Rand, Zhen Wang, Stefano Boccaletti and Attila Szolnoki
Extensive cooperation among unrelated individuals is unique to humans, who often sacrifice personal benefits for the common good and work together to achieve what they are unable to execute alone. The evolutionary success of our species is indeed due, to a large...  View Details
Keywords: Human Cooperation; Evolutionary Game Theory; Public Goods; Reward; Punishment; Tolerance; Self-organization; Pattern Formation; Cooperation; Behavior; Game Theory
Related
  • December 2016
  • Article

The Effects of Endowment Size and Strategy Method on Third Party Punishment
By: Jillian J. Jordan, Katherine McAuliffe and David G. Rand
Numerous experiments have shown that people often engage in third-party punishment (3PP) of selfish behavior. This evidence has been used to argue that people respond to selfishness with anger, and get utility from punishing those who mistreat others. Elements of the...  View Details
Keywords: Third-party Punishment; Norm-enforcement; Strategy Method; Economic Games; Cooperation; Emotions; Fairness
Related
  • Article

Gathering Data for Archival, Field, Survey, and Experimental Accounting Research
By: Robert Bloomfield, Mark W. Nelson and Eugene F. Soltes
In the published proceedings of the first Journal of Accounting Research Conference, Vatter (1966) lamented that “Gathering direct and original facts is a tedious and difficult task, and it is not surprising that such work is avoided.” For the 50th JAR Conference,...  View Details
Keywords: Archival; Data; Experiment; Empirical Methods; Field Study; Analytics and Data Science; Surveys; Financial Reporting
Related
  • March 2016 (Revised January 2020)
  • Teaching Note

Behavioural Insights Team (A) and (B)
By: Michael Luca and Patrick Rooney
The Behavioural Insights Team case introduces students to the concept of choice architecture and the value of experimental methods (sometimes called A/B testing) within organizational contexts. The exercise provides an opportunity for students to apply these principles...  View Details
Keywords: Behavioral Economics; Experiments; Choice Architecture; Public Entrepreneurship; Decision Choices and Conditions; Mathematical Methods; United Kingdom
Related
  • March 2016
  • Supplement

Advertising Experiments at RestaurantGrades
By: Weijia Dai, Hyunjin Kim and Michael Luca
This exercise provides students with a data set consisting of results from a hypothetical experiment, and asks students to make recommendations based on the data. Through this process, the exercise teaches students to analyze, design, and interpret experiments. The...  View Details
Keywords: Experimental Methods; Analytics; Web Technology; Marketing; Digital Marketing; Analysis; Performance Effectiveness; Social Media
Related
  • March 2015 (Revised January 2020)
  • Case

Behavioural Insights Team (A)
By: Michael Luca and Patrick Rooney
The Behavioural Insights Team case introduces students to the concept of choice architecture and the value of experimental methods (sometimes called A/B testing) within organizational contexts. The exercise provides an opportunity for students to apply these principles...  View Details
Keywords: Behavioral Economics; Experiments; Choice Architecture; Public Entrepreneurship; Decision Choices and Conditions; Consumer Behavior; Taxation; Economics; Public Administration Industry; United Kingdom
Related

Are you looking for?

→ Search All HBS Web