Skip to Main Content
HBS Home
  • About
  • Academic Programs
  • Alumni
  • Faculty & Research
  • Baker Library
  • Giving
  • Harvard Business Review
  • Initiatives
  • News
  • Recruit
  • Map / Directions
Faculty & Research
  • Faculty
  • Research
  • Featured Topics
  • Academic Units
  • …→
  • Harvard Business School→
  • Faculty & Research→
Publications
Publications
  • 2022
  • Working Paper

The Disagreement Problem in Explainable Machine Learning: A Practitioner's Perspective

By: Satyapriya Krishna, Tessa Han, Alex Gu, Javin Pombra, Shahin Jabbari, Steven Wu and Himabindu Lakkaraju
  • Format:Print
  • | Language:English
  • | Pages:46
ShareBar

Abstract

As various post hoc explanation methods are increasingly being leveraged to explain complex models in high-stakes settings, it becomes critical to develop a deeper understanding of if and when the explanations output by these methods disagree with each other, and how such disagreements are resolved in practice. However, there is little to no research that provides answers to these critical questions. In this work, we introduce and study the disagreement problem in explainable machine learning. More specifically, we formalize the notion of disagreement between explanations, analyze how often such disagreements occur in practice, and how do practitioners resolve these disagreements. To this end, we first conduct interviews with data scientists to understand what constitutes disagreement between explanations (feature attributions) generated by different methods for the same model prediction, and introduce a novel quantitative framework to formalize this understanding. We then leverage this framework to carry out a rigorous empirical analysis with four real-world datasets, six state-of-the-art post hoc explanation methods, and eight different predictive models, to measure the extent of disagreement between the explanations generated by various popular post hoc explanation methods. In addition, we carry out an online user study with data scientists to understand how they resolve the aforementioned disagreements. Our results indicate that state-of-the-art explanation methods often disagree in terms of the explanations they output. Worse yet, there do not seem to be any principled, well-established approaches that machine learning practitioners employ to resolve these disagreements, which in turn implies that they may be relying on misleading explanations to make critical decisions such as which models to deploy in the real world. Our findings underscore the importance of developing principled evaluation metrics that enable practitioners to effectively compare explanations.

Keywords

AI and Machine Learning; Analytics and Data Science; Mathematical Methods

Citation

Krishna, Satyapriya, Tessa Han, Alex Gu, Javin Pombra, Shahin Jabbari, Steven Wu, and Himabindu Lakkaraju. "The Disagreement Problem in Explainable Machine Learning: A Practitioner's Perspective." Working Paper, 2022.
  • Read Now

About The Author

Himabindu Lakkaraju

Technology and Operations Management
→More Publications

More from the Authors

    • June 2023
    • Transactions on Machine Learning Research (TMLR)

    When Does Uncertainty Matter? Understanding the Impact of Predictive Uncertainty in ML Assisted Decision Making

    By: Sean McGrath, Parth Mehta, Alexandra Zytek, Isaac Lage and Himabindu Lakkaraju
    • 2023
    • Proceedings of the International Conference on Learning Representations (ICLR)

    Probabilistically Robust Recourse: Navigating the Trade-offs between Costs and Robustness in Algorithmic Recourse

    By: Martin Pawelczyk, Teresa Datta, Johannes van-den-Heuvel, Gjergji Kasneci and Himabindu Lakkaraju
    • April 2023
    • Proceedings of the International Conference on Artificial Intelligence and Statistics (AISTATS)

    On the Privacy Risks of Algorithmic Recourse

    By: Martin Pawelczyk, Himabindu Lakkaraju and Seth Neel
More from the Authors
  • When Does Uncertainty Matter? Understanding the Impact of Predictive Uncertainty in ML Assisted Decision Making By: Sean McGrath, Parth Mehta, Alexandra Zytek, Isaac Lage and Himabindu Lakkaraju
  • Probabilistically Robust Recourse: Navigating the Trade-offs between Costs and Robustness in Algorithmic Recourse By: Martin Pawelczyk, Teresa Datta, Johannes van-den-Heuvel, Gjergji Kasneci and Himabindu Lakkaraju
  • On the Privacy Risks of Algorithmic Recourse By: Martin Pawelczyk, Himabindu Lakkaraju and Seth Neel
ǁ
Campus Map
Harvard Business School
Soldiers Field
Boston, MA 02163
→Map & Directions
→More Contact Information
  • Make a Gift
  • Site Map
  • Jobs
  • Harvard University
  • Trademarks
  • Policies
  • Accessibility
  • Digital Accessibility
Copyright © President & Fellows of Harvard College