Artificial intelligence models that spot patterns in images can often perform better than the human eye, but not always. If a radiologist uses an AI model to determine whether a patient’s x-rays show signs of pneumonia, when should the model’s advice be trusted and when should it be ignored?
A personalized onboarding process can help radiologists answer this question, according to MIT researchers and the MIT-IBM Watson AI Lab. They designed a system that tells users when to collaborate with an AI assistant.
In this case, the training method can find situations in which radiologists trust the model’s advice. But you shouldn’t do that because your model is wrong. The system automatically learns the rules for how to cooperate with the AI and explains them in natural language.
During onboarding, radiologists practice collaborating with the AI through training exercises based on these rules and receive feedback on their performance and the AI’s performance.
Researchers found that when humans and AI collaborate on image prediction tasks, this onboarding process improves accuracy by about 5%. Their results also show that simply telling users when to trust the AI, without training, reduces performance.
Importantly, the researchers’ system is fully automated, learning to create an onboarding process based on data from humans and AI performing specific tasks. And because it’s adaptable to a variety of tasks, it can be used scalably in a variety of situations where humans and AI models work together, including social media content moderation, writing, and programming.
“People can often use these AI tools without any training to help them figure out when they might be helpful. We don’t do this for almost every other tool that people use. There is almost always some kind of tutorial that comes with the tool. But for AI this seems to be missing. We are trying to solve this problem from a methodological and behavioral perspective,” says Hussein Mozannar, a graduate student in the PhD program in Social and Engineering Systems at the Institute for Data, Systems and Society (IDSS) and lead author of the paper. training course.
Researchers believe such onboarding will become an important part of training for healthcare professionals.
“For example, one could imagine that doctors making treatment decisions with the help of AI would first have to conduct training similar to what we are proposing. We may need to rethink everything from continuing medical education to how we design clinical trials,” says lead author David Sontag, EECS professor and leader of the MIT-IBM Watson AI Lab and MIT Jameel Clinic. This is the Clinical Machine Learning Group at the Computer Science and Artificial Intelligence Laboratory (CSAIL).
Mozannar, who is also a research associate in the Clinical Machine Learning Group, was joined on the paper by Jimin J. Lee, an electrical engineering and computer science undergraduate student. Dennis Wei, senior researcher at IBM Research; Prasanna Sattigeri and Subhro Das, researchers at the MIT-IBM Watson AI Lab. This paper will be presented at the Neural Information Processing Systems Conference.
Evolving Education
Existing onboarding methods for human-AI collaboration often consist of training materials produced by human experts for specific use cases, making them difficult to scale. Some related technologies rely on explanations, where AI tells users how confident they are about each decision, but research shows that explanations are of little help.
“As the capabilities of AI models continue to evolve, the use cases where humans can potentially benefit are also increasing over time. At the same time, users’ perception of the model continues to change. “Therefore, we need training procedures that evolve over time,” he added.
To achieve this, onboarding methods are automatically learned from data. It is built on a dataset containing many instances of a task, such as detecting the presence of a traffic light in a blurry image.
The first step of the system is to collect data about the humans and AI performing this task. In this case, humans, with the help of AI, try to predict whether a blurry image contains a traffic light.
The system inserts these data points into a latent space where similar data points represent data that are closer together. We use algorithms to discover areas in this space where humans collaborate poorly with AI. These areas capture cases where humans trusted the AI’s predictions, but the predictions were wrong, and vice versa.
Perhaps humans mistakenly trusted AI when images showed highways at night.
After discovering the regions, the second algorithm leverages a large-scale language model to describe each region with rules using natural language. The algorithm iteratively fine-tunes its rules by finding contrasting examples. This area could also be described as “ignoring AI when on the highway at night.”
These rules are used to build training exercises. In addition to the AI’s predictions, the onboarding system shows humans examples such as a blurry highway scene at night and asks the user if the image shows traffic lights. Users can answer yes, no, or use AI’s predictions.
If the human is incorrect, the correct answer and performance statistics for the human and AI for that task instance are displayed. The system does this for each area and, at the end of the training process, repeats the exercises where the human got it wrong.
“Humans have since learned something about these areas that they can take with them to make more accurate predictions in the future,” Mozannar said.
Improve Accuracy with Onboarding
The researchers tested the system with users on two tasks: detecting traffic lights in blurry images and answering multiple-choice questions in various domains (e.g. biology, philosophy, computer science, etc.).
They first showed users a card containing information about the AI model, its training method, and an analysis of its performance in broad categories. Users were divided into five groups. Some just showed the card, some went through the researcher’s onboarding process, some went through the basic onboarding process, and some went through the researcher’s onboarding process and received recommendations on what to do and when not to do it. Others trusted the AI and were only given recommendations.
The researchers’ onboarding procedure alone, without any recommendations, significantly improved users’ accuracy, improving performance on the traffic light prediction task by approximately 5% without slowing down. However, onboarding was not effective in answering questions. Researchers believe this is because the AI model, ChatGPT, provided each answer with a description that conveyed whether it was trustworthy or not.
However, providing referrals without onboarding had the opposite effect. This means that not only did users perform worse, but it also took longer to predict.
“Whenever I recommend something to someone, they seem confused and don’t know what to do. It derails their process. People also don’t like being told what to do, so that’s a factor,” says Mozannar.
He added that just providing recommendations can cause harm to users if those recommendations are wrong. On the other hand, the biggest limitation of onboarding is the amount of data available. If you don’t have enough data, the onboarding phase won’t be effective, he says.
In the future, he and his colleagues want to conduct larger studies to evaluate the short- and long-term effects of onboarding. They also want to leverage unlabeled data in their onboarding process and find ways to effectively reduce the number of regions without omitting important examples.
“People are adopting AI systems arbitrarily, and indeed AI offers great potential, but these AI agents still make mistakes sometimes. “It is therefore important for AI developers to devise ways to help humans know when it is safe to rely on AI’s suggestions,” says Dan Weld, professor emeritus at the Paul G. Allen School of Computer Science and Engineering at the University of Washington. did not participate in this study. “Mozannar et al. We have created an innovative way for AI to identify trustworthy situations and (importantly) explain situations to people in a way that leads to better human-AI team interactions.”
This work is partially funded by the MIT-IBM Watson AI Lab.