This CREST report explores the role of explanation in human engagement with artificial intelligence and machine learning.

Executive summary

For artificial intelligence/machine learning (AI/ML) to augment human intelligence (in terms of extending a human’s cognitive capabilities through the provision of sophisticated analysis on massive data sets), there needs to be sufficient common ground in the way humans and AI/ML communicate.

In this report, we assume that interactions between humans and AI/ML occur in a system in which cooperation between humans and AI/ML is one interaction among many, e.g. humans cooperate with other humans; humans programme the AI/ML; humans could be involved in selecting and preparing the data that the algorithms use; the AI/ML could interact with other algorithms etc.

Not only is it important that humans and AI/ML establish common ground, but also that humans who communicate with each other using AI/ML share this common ground.

Explanation is the process by which common ground between interactions is established and maintained

From this perspective, the term ‘explanation’ is the process by which common ground between interactions is established and maintained.

We have developed a framework to highlight this concept, and this is instantiated to show how different types of explanation can occur, each of which requires different means of support.

Primarily, an explanation involves an agreement on the features (in data sets or a situation) which the ‘explainer’ and ‘explainee’ pay attention to and why these features are relevant.

We propose three levels of relevance:

  • ‘Cluster’ – In which a group of features typically occur together
  • ‘Belief’ – which defines a reason as to why such a cluster will occur
  • ‘Policy’ – which justifies the belief and relates this to action.

Agreement (on features and relevance) depends on the knowledge and experience of the explainer and ‘explainee’, and much of the process of the explanation involves ensuring alignment between parties in terms of knowledge and experience.

We relate the concept of explanation developed here to concepts such as intelligibility and transparency in the AI / ML literature and provide guidelines that can inform decisions on the development, deployment, and use of AI/ML in operational settings.

From the framework of explanation developed in this report, we propose the following guidelines:

  1. Explanations should include relevant causes. Explanations should relate to beliefs in the relationship between features of a situation and the causes that can directly affect the event being explained (probability) or can explain most of the event (explanatory power); are plausible (construct validity); and if the cause was instigated by a person, deliberative.
  2. Explanations should include relevant features. Explanations should relate to the key features of the situation and the goals of the explainer and explainee.
  3. Explanations should be framed to suit the audience. Explainers should fit the explanation to suit the explainee’s understanding of the topic and what it is they wish to gain from the explanation (their mental model and goals).
  4. Explanations should be interactive. Explainers should involve explainees in the explanation.
  5. Explanations should be (where necessary) actionable. Explainees should be given information that can be used to perform and/or improve future actions and behaviours. 
Read more
  • Acharya, A., Howes, A., Baber, C. and Marshall, T. (2018) Automation reliability and decision strategy: a sequential decision model for automation interaction, Proceedings of the Human Factors and Ergonomics Society 2018 Annual Meeting, Santa Monica, CA: HFES, 144–148
  • Amir, D., Amir, O.: Highlights: summarizing agent behavior to people. In: Proceedings of the 17th International Conference on Autonomous Agents and MultiAgent Systems, pp. 1168–1176. International Foundation for Autonomous Agents and Multiagent Systems (2018)
  • Anjomshoae, S., Najjar, A., Calvaresi, D., Framling, K.: Explainable agents and robots: results from a systematic literature review. In: Proceedings of the 18th International Conference on Autonomous Agents and MultiAgent Systems, pp. 1078–1088. International Foundation for Autonomous Agents and Multiagent Systems (2019)
  • Arias, A. M., Davis, E. A., Marino, J. C., Kademian, S. M., & Palincsar, A. S. (2016). Teachers’ use of educative curriculum materials to engage students in science practices. International Journal of Science Education, 38(9), 1504–1526
  • Aronson, J. L. (1971). On the Grammar of 'Cause'. Synthese, 414–430
  • Baber, C., Morar, M.S. and McCabe, F. (2019) Ecological interface design, the proximity compatibility principle, and automation reliability in road traffic management, IEEE Transactions on Human-Machine Systems, 49, 241–249
  • Bellotti, V., & Edwards, K. (2001). Intelligibility and accountability: human considerations in context-aware systems. Human–Computer Interaction, 16(2–4), 193–212
  • Bird, A. (1999) Explanation and Laws, Synthese, 120, 1–18
  • Borgo, R., Cashmore, M., Magazzeni, D.: Towards providing explanations for AI planner decisions. arXiv preprint arXiv:1810.06338 (2018)
  • Braaten, M., & Windschitl, M. (2011). Working toward a stronger conceptualization of scientific explanation for science education. Science education, 95(4), 639–669
  • Broekens, J., Harbers, M., Hindriks, K., van den Bosch, K., Jonker, C., Meyer, J.-J.: Do you get it? User-evaluated explainable BDI agents. In: Dix, J., Witteveen, C. (eds.) MATES 2010. LNCS (LNAI), vol. 6251, pp. 28–39. Springer, Heidelberg (2010)
  • Chakraborti, T., Sreedharan, S., Grover, S., Kambhampati, S.: Plan explanations as model reconciliation. In: 2019 14th ACM/IEEE International Conference on Human-Robot Interaction (HRI), pp. 258–266. IEEE (2019)
  • Chen, J.Y., Procci, K., Boyce, M., Wright, J., Garcia, A., Barnes, M.: Situation awareness-based agent transparency. Technical report, Army Research Lab Aberdeen Proving Ground MD Human Research and Engineering (2014)
  • Chen, X., Starke, S.D., Baber, C. and Howes, A. (2017) A cognitive model of how people make decisions through interaction with visual displays, In CHI’17: Proceedings of the 32nd annual ACM conference on Human factors in computing systems, New York: ACM
  • Clark, H. H., & Brennan, S. E. (1991). Grounding in communication. In L. B. Resnick, J. M. Levine, & S. D. Teasley (Eds.), Perspectives on socially shared cognition (p. 127–149). American Psychological Association
  • Clark, H.H. (1991) Using Language, Cambridge: Cambridge University Press
  • Cramer, H., Evers, V., Ramlal, S., Van Someren, M., Rutledge, L., Stash, N., & Wielinga, B. (2008). The effects of transparency on trust in and acceptance of a content-based art recommender. User Modeling and User-Adapted Interaction, 18(5), 455
  • De Fauw, J. et al., 2018, Clinically applicable deep learning for diagnosis and referral in retinal disease. Nat Med 24, 1342–1350
  • Dell, G. S. (1986). A spreading-activation theory of retrieval in sentence production. Psychological review, 93(3), 283
  • Dell, G. S., Chang, F., & Griffin, Z. M. (1999). Connectionist models of language production: Lexical access and grammatical encoding. Cognitive Science, 23(4), 517–542
  • Dowe, P. (1992). Wesley Salmon's process theory of causality and the conserved quantity theory. Philosophy of science, 59(2), 195–216
  • Endsley, M.: Measurement of situation awareness in dynamic systems. Hum. Factors 37, 65–84 (1995)
  • Fair, D. (1979). Causation and the Flow of Energy. Erkenntnis, 14(3), 219–250
  • Floyd, M.W., Aha, D.W.: Incorporating transparency during trust-guided behavior adaptation. In: Goel, A., D´ıaz-Agudo, M.B., Roth-Berghofer, T. (eds.) ICCBR 2016. LNCS (LNAI), vol. 9969, pp. 124–138. Springer, Cham (2016)
  • Fox, M., Long, D., Magazzeni, D.: Explainable planning. arXiv preprint arXiv:1709.10256 (2017)
  • Fromkin, V. A. (1973). Speech Errors as Linguistic Evidence. The Hague, Netherlands: Mouton
  • Funder, D.C. (1995) On the accuracy of personality judgment: a realistic approach, Psychological Review, 102, 652–670
  • Greydanus, S., Koul, A., Dodge, J. and Fern, A. (2018) Visualizing and understanding Atari agents,
  • Grice, H. P. (1975). Logic and conversation. In Speech acts (pp. 41–58). Brill
  • Gunning, D., & Aha, D. W. (2019). DARPA's Explainable Artificial Intelligence Program. AI Magazine, 40(2), 44–58
  • Hacibeyoglu, M. and Ibrahim, M.H. (2018). The Effect of Over-sampling and Undersampling Techniques in Medical Datasets)
  • Harbers, M., Bradshaw, J.M., Johnson, M., Feltovich, P., van den Bosch, K., Meyer, J.-J.: Explanation in human-agent teamwork. In: Cranefield, S., van Riemsdijk, M.B., V´azquez-Salceda, J., Noriega, P. (eds.) COIN -2011. LNCS (LNAI), vol. 7254, pp. 21–37. Springer, Heidelberg (2012)
  • Halpern, J. Y., & Pearl, J. (2005). Causes and explanations: A structural-model approach. Part I: Causes. The British journal for the philosophy of science, 56(4), 843–887
  • Halpern, J. Y., & Pearl, J. (2005b). Causes and explanations: A structural-model approach. Part II: Explanations. The British journal for the philosophy of science, 56(4), 889–911
  • Hartsuiker, R. J., Corley, M., & Martensen, H. (2005). The lexical bias effect is modulated by context, but the standard monitoring account doesn’t fly: Related reply to Baars et al. (1975). Journal of Memory and Language, 52(1), 58–70
  • Harley, T. (2008). The Psychology of language. 4th ed. New York: Psychology Press, pp.397–451
  • Hellström, T., Bensch, S.: Understandable robots-what, why, and how. Paladyn J. Behav. Robot. 9(1), 110–123 (2018)
  • Hempel, C. G., & Oppenheim, P. (1948). Studies in the Logic of Explanation. Philosophy of science, 15(2), 135–175
  • Hepenstal, S., Wong, B.L.W., Zhang, L. and Kodogoda, N. (2019) How analysts think: a preliminary study of human needs and demands for AI-based conversational agents, Proceedings of the Human Factors and Ergonomics Society 2019 Annual Meeting, Santa Monica, CA: HFES, 178–182
  • Herlocker, J. L., Konstan, J. A., & Riedl, J. (2000, December). Explaining collaborative filtering recommendations. In Proceedings of the 2000 ACM conference on Computer supported cooperative work (pp. 241–250). ACM
  • Hilton, D. J. (1990). Conversational processes and causal explanation. Psychological Bulletin, 107(1), 65
  • Hilton, D. J. (1996). Mental models and causal explanation: Judgements of probable cause and explanatory relevance. Thinking & Reasoning, 2(4), 273–308
  • Hoffman, R. R., & Klein, G. (2017). Explaining explanation, part 1: theoretical foundations. IEEE Intelligent Systems, 32(3), 68–73
  • Hoffman, R., Miller, T., Mueller, S. T., Klein, G., & Clancey, W. J. (2018). Explaining explanation, part 4: a deep dive on deep nets. IEEE Intelligent Systems, 33(3), 87–95.
  • Holzinger, A., Carrington, A. and Müller, H. (2020) Measuring the Quality of Explanations: the Systems Causability Scale (SCS), Künstliche Intelligenz, 34, 193–198
  • Holzinger, A.,Plass, M., Kickmeier-Rust, M., Holzinger, K., Crisan, G.-C., Perita, C.-M., and Palade, V. (2018) Interactive machine learning: experimental evidence for the human in the algorithmic loop, Applied Intelligence, 49, 2401–2414
  • Hume, D. (2000). An enquiry concerning human understanding. In Seven Masterpieces of Philosophy (pp. 191–284). Routledge
  • Hutchins, E., 1995. How a cockpit remembers its speeds. Cognitive science, 19(3), pp.265–288
  • Hutchins, E., 2014. The technology of team navigation. In Intellectual teamwork (pp. 205–234). Psychology Press
  • Jaspars, J. M., & Hilton, D. J. (1988). Mental models of causal reasoning.
  • Kaur, H., Nori, H., Jenkins, S., Caruana, R., Wallach, H., Vaughan, J.W. (2020) Interpreting interpretability: understanding data scientists’ use of interpretability tools for machine learning, CHI 2020, New York: ACM, paper 92
  • Kelemen, D. (2019). The Magic of Mechanism: Explanation-Based Instruction on Counterintuitive Concepts in Early Childhood. Perspectives on Psychological Science, 14(4), 510–522
  • Kim, B., et al.: Interpretability beyond feature attribution: quantitative testing with concept activation vectors (TCAV). arXiv preprint arXiv:1711.11279 (2017)
  • Klein, G.A. (1989) Recognition-primeddecisions,” in Advances in Man-Machine Systems Research, ed. W.B. Rouse, Greenwich, CT: JAI Press, Inc., 47-
  • Klein, G. (2018). Explaining explanation, part 3: The causal landscape. IEEE Intelligent Systems, 33(2), 83–88
  • Klein, G., Moon, B., & Hoffman, R. R. (2006). Making sense of sensemaking 1: Alternative perspectives. IEEE intelligent systems, 21(4), 70–73
  • Leddo, J., Abelson, R. P., & Gross, P. H. (1984). Conjunctive explanations: When two reasons are better than one. Journal of Personality and Social Psychology, 47(5), 933
  • Leite, R.A., Gschwandtner, T., Miksch, S., Kriglstein, S., Pohl, M., Gstrein, E. and Kuntner, J., 2017. Eva: Visual analytics to identify fraudulent events. IEEE transactions on visualization and computer graphics, 24, 330–339
  • Lewis, D. (1974). Causation. The journal of philosophy, 70(17), 556–567
  • Lipton, Z.C.: The mythos of model interpretability. arXiv preprint arXiv:1606.03490 (2016)
  • Livengood, J., & Sytsma, J. (2020). Actual causation and compositionality. Philosophy of Science, 87(1), 43–69
  • Lomas, M., Chevalier, R., Cross, E.V., Garrett, R.C., Hoare, J., Kopack, M.: Explaining robot actions. In: Proceedings of the Seventh Annual ACM/IEEE International Conference on Human-Robot Interaction, pp. 187–188 (2012)
  • Lombrozo, T. (2007). Simplicity and probability in causal explanation. Cognitive psychology, 55(3), 232–257
  • Lombrozo, T. (2010). Causal–explanatory pluralism: How intentions, functions, and mechanisms influence causal ascriptions. Cognitive Psychology, 61(4), 303–332
  • McClure, J. (2002). Goal-based explanations of actions and outcomes. European review of social psychology, 12(1), 201–235
  • Malle, B. F. (2006). How the mind explains behavior: Folk explanations, meaning, and social interaction. Mit Press
  • Meyer, J. (2004). Conceptual issues in the study of dynamic hazard warnings. Human Factors, 46(2), 196-204
  • Miller, T. (2019). Explanation in artificial intelligence: Insights from the social sciences. Artificial Intelligence, 267, 1–38
  • Miller, T., Howe, P., Sonenberg, L.: Explainable AI: beware of inmates running the asylum or: how i learnt to stop worrying and love the social and behavioural sciences. arXiv preprint arXiv:1712.00547
  • Mitchell, T. (1997) Machine Learning, New York: McGraw-Hill
  • Morar, N. and Baber, C. (2017) Joint human-automation decision making in road traffic management, Proceedings of the Human Factors and Ergonomics Society 2017 Annual Meeting, Santa Monica, CA: HFES, 385–389
  • Neerincx, M.A., van der Waa, J., Kaptein, F., van Diggelen, J.: Using perceptual and cognitive explanations for enhanced human-agent team performance. In: Harris, D. (ed.) EPCE 2018. LNCS (LNAI), vol. 10906, pp. 204–214. Springer, Cham (2018)
  • Nunes, I. and Jannach, D. (2017) A systematic review and taxonomy of explanations in decision support and recommender systems, User Modeling and User-Adapted Interaction, 27, 393–444
  • Paula, E.L., Ladeira, M., Carvalho, R.N. and Marzagao, T., 2016, December. Deep learning anomaly detection as support fraud investigation in brazilian exports and anti-money laundering. In 2016 15th IEEE International Conference on Machine Learning and Applications (ICMLA) (pp. 954-960). IEEE
  • Pickering, M. J., & Garrod, S. (2013). An integrated theory of language production and comprehension. Behavioral and brain sciences, 36(4), 329–347
  • Rapp, B., & Goldrick, M. (2000). Discreteness and interactivity in spoken word production. Psychological review, 107(3), 460
  • Rapp, B., & Goldrick, M. (2004). Feedback by Any Other Name Is Still Interactivity: A Reply to Roelofs (2004)
  • Rehder, B. (2003). A causal-model theory of conceptual representation and categorization. Journal of Experimental Psychology: Learning, Memory, and Cognition, 29(6), 1141
  • Ribeiro, M.T., Singh, S. and Guestrin, C., 2016. Model-agnostic interpretability of machine learning. arXiv preprint arXiv:1606.05386
  • Roelofs, A. (2004a, April). Error biases in spoken word planning and monitoring by aphasic and nonaphasic speakers: comment on Rapp and Goldrick (2000). In XIX Workshop on Cognitive Neuropsychology, Jan, 2001, Bressanone, Italy; Part of this work was presented at the aforementioned conference. (Vol. 111, No. 2, p. 561). American Psychological Association
  • Roelofs, A. (2004b). Comprehension-based versus production-internal feedback in planning spoken words: a Rejoinder to Rapp and Goldrick (2004)
  • Rosenfeld, A., & Richardson, A. (2019). Explainability in human–agent systems. Autonomous Agents and Multi-Agent Systems, 33(6), 673–705
  • Roundtree, K.A., Goodrich, M.A. and Adams, J.A. (2019) Transparency: transitioning from human-machine systems to human-swarm systems, Journal of Cognitive Engineering and Decision Making, 13, 171–195
  • Salmon, W. C. (1971). Statistical explanation and statistical relevance (Vol. 69). University of Pittsburgh Pre
  • Sanneman, L. and Shah, J.A. (2020) A Situation-Awareness based framework for design and evaluation of explainable AI, In D. Calvaresi et al. (Eds.): EXTRAAMAS 2020, LNAI 12175, pp. 94–110
  • Sachan, S., Yang, J.B., Xu, D.L., Benavides, D.E. and Li, Y., 2020. An explainable AI decision-support-system to automate loan underwriting. Expert Systems with Applications, 144, Article 113100
  • Sheh, R., Monteath, I.: Introspectively assessing failures through explainable artificial intelligence. In: IROS Workshop on Introspective Methods for Reliable Autonomy (2017)
  • Sheridan, T.B. (1992). Telerobotics, automation, and human supervisory control. MIT press
  • Sobel, J. (2020). Lying and deception in games. Journal of Political Economy, 128(3), 907–947
  • Spiegelhalter, D., 2020, Should we trust algorithms, Harvard Data Science Review, 2
  • Sreedharan, S., Srivastava, S., Kambhampati, S.: Hierarchical expertise level modeling for user specific contrastive explanations. In: IJCAI, pp. 4829–4836 (2018)
  • Starke, S.D. and Baber, C. (2018) The effect of four user interface concepts on visual scan pattern similarity and information foraging in a complex decision making task, Applied Ergonomics, 70, 6–17
  • Starke, S.D. and Baber, C. (2020) The effect of known decision support reliability on outcome quality and visual information foraging in joint decision making, Applied Ergonomics, 86
  • Swartout, W. R. (1983). XPLAIN: A system for creating and explaining expert consulting programs. Artificial intelligence, 21(3), 285–325
  • Tversky, A., & Kahneman, D. (1983). Extensional versus intuitive reasoning: The conjunction fallacy in probability judgment. Psychological review, 90(4), 293
  • van Melle, W., Shortliffe, E. H., & Buchanan, B. G. (1984). EMYCIN: A knowledge engineer’s tool for constructing rule-based expert systems. Rule-based expert systems: The MYCIN experiments of the Stanford Heuristic Programming Project, 302–313
  • Vasilyeva, N., Wilkenfeld, D. A., & Lombrozo, T. (2015). Goals Affect the Perceived Quality of Explanations. In CogSci
  • Vogogias, A., Kennedy, J., Archambault, D., Bach, B., Smith, V.A. and Currant, H., 2018. BayesPiles: Visualisation Support for Bayesian Network Structure Learning. ACM Transactions on Intelligent Systems and Technology (TIST), 10(1), pp.1–23
  • Vygotsky, L. S. (1980). Mind in society: The development of higher psychological processes. Harvard university press
  • Wang, D., Yang, Q., Abdul, A., & Lim, B. Y. (2019, May). Designing theory-driven user-centric explainable AI. In Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems (pp. 1–15)
  • Wilson, D., & Sperber, D. (2002). Relevance theory
  • Windschitl, M., Thompson, J., & Braaten, M. (2008a). How novice science teachers appropriate epistemic discourses around model-based inquiry for use in classrooms. Cognition and Instruction, 26(3), 310­–378
  • Windschitl, M., Thompson, J., & Braaten, M. (2008b). Beyond the scientific method: Model‐based inquiry as a new paradigm of preference for school science investigations. Science education, 92(5), 941–967