Skip to Main Content
HBS Home
  • About
  • Academic Programs
  • Alumni
  • Faculty & Research
  • Baker Library
  • Giving
  • Harvard Business Review
  • Initiatives
  • News
  • Recruit
  • Map / Directions
Faculty & Research
  • Faculty
  • Research
  • Featured Topics
  • Academic Units
  • …→
  • Harvard Business School→
  • Faculty & Research→
Publications
Publications
  • March 2022
  • Article
  • Computational and Mathematical Organization Theory

Sensitivity Analysis of Agent-based Models: A New Protocol

By: Emanuele Borgonovo, Marco Pangallo, Jan Rivkin, Leonardo Rizzo and Nicolaj Siggelkow
  • Format:Print
ShareBar

Abstract

Agent-based models (ABMs) are increasingly used in the management sciences. Though useful, ABMs are often critiqued: it is hard to discern why they produce the results they do and whether other assumptions would yield similar results. To help researchers address such critiques, we propose a systematic approach to conducting sensitivity analyses of ABMs. Our approach deals with a feature that can complicate sensitivity analyses: most ABMs include important non-parametric elements, while most sensitivity analysis methods are designed for parametric elements only. The approach moves from charting out the elements of an ABM through identifying the goal of the sensitivity analysis to specifying a method for the analysis. We focus on four common goals of sensitivity analysis: determining whether results are robust, which elements have the greatest impact on outcomes, how elements interact to shape outcomes, and which direction outcomes move when elements change. For the first three goals, we suggest a combination of randomized finite change indices calculation through a factorial design. For direction of change, we propose a modification of individual conditional expectation (ICE) plots to account for the stochastic nature of the ABM response. We illustrate our approach using the Garbage Can Model, a classic ABM that examines how organizations make decisions.

Keywords

Agent-based Modeling; Sensitivity Analysis; Design Of Experiments; Total Order Sensitivity Indices; Organizations; Behavior; Decision Making; Mathematical Methods

Citation

Borgonovo, Emanuele, Marco Pangallo, Jan Rivkin, Leonardo Rizzo, and Nicolaj Siggelkow. "Sensitivity Analysis of Agent-based Models: A New Protocol." Computational and Mathematical Organization Theory 28, no. 1 (March 2022): 52–94.
  • Read Now

About The Author

Jan W. Rivkin

Strategy
→More Publications

More from the Authors

    • June 2025
    • Strategic Management Journal

    Outcome and Process Frames: Strategic Renewal and Capability Reprioritization at the Federal Bureau of Investigation

    By: Ryan Raffaelli, Tiona Zuzul, Ranjay Gulati and Jan Rivkin
    • May 2025
    • Faculty Research

    BlackRock (A): Selling the Systems?

    By: Jan Rivkin
    • May 2025
    • Faculty Research

    Community First! Village: Scaling Goodness?

    By: Jan Rivkin and Coelin P. Scibetta
More from the Authors
  • Outcome and Process Frames: Strategic Renewal and Capability Reprioritization at the Federal Bureau of Investigation By: Ryan Raffaelli, Tiona Zuzul, Ranjay Gulati and Jan Rivkin
  • BlackRock (A): Selling the Systems? By: Jan Rivkin
  • Community First! Village: Scaling Goodness? By: Jan Rivkin and Coelin P. Scibetta
ǁ
Campus Map
Harvard Business School
Soldiers Field
Boston, MA 02163
→Map & Directions
→More Contact Information
  • Make a Gift
  • Site Map
  • Jobs
  • Harvard University
  • Trademarks
  • Policies
  • Accessibility
  • Digital Accessibility
Copyright © President & Fellows of Harvard College.