What are the chances of dying in a plane crash? According to the 2022 report published by the International Air Transport Association, the industry’s mortality risk is 0.11. In other words, an average person would have to fly every day for 25,214 years to have a 100% chance of being in a fatal accident. The aviation industry, long considered one of the safest forms of transportation, is highly regulated, and MIT scientists believe it may hold the key to regulating artificial intelligence in healthcare.
Marzyeh Ghassemi, an assistant professor in MIT’s Department of Electrical Engineering and Computer Science (EECS) and the Institute of Medical Engineering Sciences, and Julie Shah, HN Slater Professor of Aeronautics and Astronautics at MIT, share an interest in transparency issues. AI model. After their conversation in early 2023, they realized that aviation could serve as a model to ensure that marginalized patients are not harmed by biased AI models.
Ghassemi and Shah, who are also principal investigators at the MIT Abdul Latif Jameel Health Machine Learning Clinic (Jameel Clinic) and the Computer Science and Artificial Intelligence Laboratory (CSAIL), are researchers, lawyers, and fellows at MIT, Stanford University, the Federation of American Scientists, Emory University, and Adelaide. Policy analysts from the University, Microsoft, and the University of California, San Francisco came together to launch a research project, the results of which were recently accepted by Equity and Access. In Algorithms, Mechanisms and Optimization Conference.
“I think many of my co-authors are excited about the potential of AI to have a positive societal impact, especially the recent developments,” said lead author and current assistant professor of EECS at the University of Michigan and a postdoctoral researcher in Ghassemi’s lab. Elizabeth Bondi-Kelly says: The project has started. “But we are also cautious and hoping to develop a framework to manage potential risks as deployment begins, so we were looking for inspiration for such a framework.”
AI in health today is similar to the aviation industry 100 years ago, says co-author Lindsay Sanneman, a doctoral student in MIT’s Department of Aeronautics and Astronautics. Despite being known as the “golden age of aviation,” the 1920s were “surprisingly high” in fatal accidents, according to the Mackinac Center for Public Policy.
Jeff Marcus, now director of the National Transportation Safety Board’s (NTSB) Safety Recommendations Division, recently published a National Aviation Month blog post stating that although many fatal accidents occurred in the 1920s, 1929 is still considered “the worst year on record.” I mentioned that it remains. 51 accidents were reported, making it the deadliest aviation accident in history. By today’s standards, there are 7,000 accidents per year, or 20 per day. After a number of fatal accidents occurred in the 1920s, President Calvin Coolidge passed landmark legislation called the Air Commerce Act of 1926, which regulated air travel through the Department of Commerce.
But the similarities don’t end there. Aviation’s subsequent path toward automation is similar to that of AI. AI explainability has been a controversial topic given AI’s notorious “black box” problem. This problem has led AI researchers to debate how much an AI model should “explain” its results to users before they are potentially biased into blindly following the model’s instructions.
“The 1970s saw an increase in the amount of autopilot systems that processed warnings to pilots of hazards,” Sanneman adds. “There have been some growing pains with the entry of automation into aviation in terms of human interaction with automated systems. The confusion that can arise when pilots do not have a keen awareness of what the automation is doing.”
Today, becoming a commercial airline captain requires 1,500 hours of flight history along with equipment training. This rigorous and comprehensive process takes about 15 years, including a bachelor’s degree and a co-pilot’s degree, according to the researchers’ paper. The researchers believe the success of the extensive pilot training could serve as a potential model for training doctors to use AI tools in clinical settings.
The paper also suggests encouraging reporting of unsafe health AI tools, like the Federal Aviation Administration (FAA) does for pilots. That is, through “limited immunity,” which allows pilots to keep their licenses even after doing something dangerous, as long as they didn’t intend to do so.
According to a 2023 report published by the World Health Organization (WHO), on average, one in 10 patients suffers an adverse event (i.e. a ‘medical error’) while receiving hospital treatment in high-income countries.
However, in current medical practice, clinicians and health care workers are often afraid to report medical errors, not only because of concerns about guilt and self-criticism, but also because of negative consequences that emphasize personal punishment, such as revocation of a medical license. , instead of reforming a system that is more prone to medical errors.
“When the hammer misses on health, patients suffer,” Ghassemi wrote in a recently published commentary. natural human behavior. “This reality presents unacceptable ethical risks to the healthcare AI community, which is already struggling with complex care issues, staffing shortages, and overburdened systems.”
Grace Wickerson, co-author and health equity policy manager at the Federation of American Scientists, believes the new paper is an important addition to the broader governance framework that has not yet been established. “I think there is a lot that can be done with existing government powers,” they say. “There are a variety of ways to pay for health AI that will require Medicare and Medicaid to consider equity in their purchasing or reimbursement techniques. NIH [National Institute of Health] We can fund more research to make algorithms more equitable and build standards for these algorithms that the FDA can use. [Food and Drug Administration] “They are trying to figure out what health equity means and how it is currently regulated within the authorities.”
Among others, the paper covers the FDA, the Federal Trade Commission (FTC), the recently created Agency for Advanced Study in Health Care, the Agency for Healthcare Research and Quality, the Centers for Medicare and Medicaid, the Department of Health and Human Services, and the Office for Civil Rights (OCR).
But Wickerson says more needs to be done. In Wickerson’s view, the most difficult part of writing the paper was “imagining what we don’t already have.”
The paper proposes creating an independent audit body similar to the NTSB that would allow for safety audits of malfunctioning medical AI systems, rather than relying solely on existing regulators.
“I think this is a current question about technology governance. There really hasn’t been an institution that has been assessing the impact of technology since the ’90s,” Wickerson added. “We used to have an Office of Technology Assessment. That office existed before the digital age even began, and the federal government allowed that to be abolished.”
Co-author and recent Stanford Law School graduate Zach Harned believes the main challenge for emerging technologies is that the pace of technological development outpaces regulation. “However, the importance of AI technology and the potential benefits and risks it poses, particularly in healthcare, are driving a flurry of regulatory efforts,” Harned says. “FDA clearly has a key role to play here and has continued to issue guidance and white papers to explain its evolving position on AI. However, privacy will be another important area to focus on, along with OCR enforcement for HIPAA. [Health Insurance Portability and Accountability Act] It’s the FTC and you who enforce privacy violations against entities not covered by HIPAA.”
Harned notes that the field is evolving rapidly, including developments such as the recent White House Executive Order 14110 on the Safe and Trustworthy Development of AI, as well as regulatory activity in the European Union (EU), including the finalization of EU AI law. . We’re getting closer to the finish. “It is certainly an exciting time to see this important technology being developed and regulated to ensure safety without stifling innovation,” he says.
Beyond regulatory activity, the paper suggests other opportunities for insurance companies to create incentives for safer health AI tools, such as pay-for-performance programs that reward hospitals for good performance. Be fair).
So how long do researchers think it will take to create a regulatory system that works for health AI? “The NTSB and FAA systems, in which investigation and enforcement are spread across two different agencies, were created by Congress over several decades,” according to the paper.
Bondi-Kelly hopes this paper will be a piece of the AI regulatory puzzle. In her mind, “the dream scenario is that we all read the paper, are inspired, and apply some of the useful lessons from aviation to help AI prevent potential AI harm during deployment.”
In addition to Ghassemi, Shah, Bondi-Kelly and Sanneman, co-authors from MIT include senior research scientist Leo Anthony Celi and former postdoctoral fellows Thomas Hartvigsen and Swami Sankaranarayanan. Funding for this research came in part through the MIT CSAIL METEOR Fellowship, Quanta Computing, the Volkswagen Foundation, the National Institutes of Health, the Herman LF von Helmholtz Career Development Professorship, and the CIFAR Azrieli Global Scholar award.