Zerilli2022TransparencyModulates

Related: interpretability

John Zerilli, Umag Bhatt, and Adrian Weller - How transparency modulates trust in artificial intelligence

[Zerilli, Bhatt, and Weller 2022 How transparency modulates trust in artificial intelligence]

Bibliographic info

Zerilli, J., Bhatt, U., & Weller, A. (2022). How transparency modulates trust in artificial intelligence. Patterns.

Commentary

This paper systematically discusses the role played by various forms of algorithmic transparency in the process of trust. I find it interesting that Zerilli et al. not only argue that transparency is important for trust in AI systems, but they discuss the potential dangers of incorrect or misleading transparency information and what features AI systems should have to foster trust.

They take a different approach than I have seen so far by saying that there is a spectrum of trust in AI systems, from loafing to opposition at the ends and there is an optimal level of trust in between. They also organize transparency into four different categories that can influence trust differently (which I will discuss in the quotes below). Additionally, I like the section with open questions that illustrates everything that is still unknown in this field.

Excerpts & Key Quotes

Explanation as a form of transparency

"Too much transparency can cause people to incorrectly follow a model when it makes a mistake, due to information overload. On other occasions, poor or confusing explanations can lead to algorithm aversion."

Comment:

This shows that explanations are important in establishing appropriate levels of trust, but that they can backfire easily. It is easy to imagine that when explanations about an AI system are very thorough, and possibly difficult to understand, people might be more inclined to be 1) overloaded with information 2) suspicious that all that information might 'hide' troublesome aspects.

Performance metrics as a form of transparency

"There is reason to believe that a better calibration of trust to a system’s actual level of accuracy can be achieved by providing .. not just cumulative performance feedback, but continuous performance feedback that allows the user to maintain a better picture of the system’s relative superiority in real time"

Comment:

To increase trust in AI systems, it would be good to consistently provide performance metrics. This is linked to the reliability and accuracy aspect of trust (see conceptnote of trust).

User control as a form of transparency

"provided that they can modify its forecasts, users are apparently willing to take an algorithm seriously even after seeing it make occasional mistakes. What is more, the precise degree of control seems to be irrelevant: the ability to modify a forecast even slightly may be sufficient to induce appropriate reliance"

Comment:

The quote above shows that people are willing to accept mistakes occasionally when they have control to modify a forecast. A possibly dangerous aspect is that users only need to feel as if they are still somewhat in control, when they in fact are not. This has not been mentioned in the article, but I think this could be an interesting future research direction as well: do they really need to be in control or is the feeling of control sufficient for trusting the system more?

Confidence information as a form of transparency

"A different form of transparency involves presenting users with system confidence information. There is growing evidence that suitably formatted confidence data (e.g., in the form of uncertainty estimates, confidence intervals, confidence levels, etc.) may improve trust calibration."

Comment:

The authors mention that giving confidence information (how confident is the system of the predictions/output), people are able to think about their level of trust: high confidence leads to more trust. One downside of this is that humans are often not great at handling numeric (confidence) information and also that it is challenging to provide reliable uncertainty estimates. Usually AI systems are prone to being overconfident where they might perform poorly, so this would increase trust unjustly.

Open questions

"First, it is unclear what effects the size, frequency, type, and distribution of errors have in the loss and recovery of trust after users witness automation errors. Second, we know little about how different forms of transparency compare in the course of rebuilding that trust."

Comment:

The quote above shortly summarizes interesting aspects that should be researched further. What is the impact of the size of an error on the loss and recovery of trust? How often can a human forgive a machine for making mistakes? Overall, this paper presents a lot of interesting research topics in this field (Human-AI teams) that other researchers can build on.