A widely used algorithm that predicts which patients will benefit from extra medical care dramatically underestimates the health needs of the sickest black patients, amplifying long-standing racial disparities in medicine, researchers have found.
The problem was caught in an algorithm sold by a leading health services company, called Optum, to guide health care decision-making for millions of people. But the same issue almost certainly exists in other tools used by other private companies, nonprofit health systems and government agencies to manage the health care of about 200 million people in the United States each year, the scientists reported in the journal Science .
Correcting the bias would more than double the number of black patients flagged as at risk of complicated medical needs within the health system the researchers studied, and they are already working with Optum on a fix. When the company replicated the analysis on a national data set of 3.7 million patients, they found that black patients who were ranked by the algorithm as equally as in need of extra care as white patients were much sicker: They collectively suffered from 48,772 additional chronic diseases.
“It’s truly inconceivable to me that anyone else’s algorithm doesn’t suffer from this,” said Sendhil Mullainathan, a professor of computation and behavioral science at the University of Chicago Booth School of Business, who oversaw the work. “I’m hopeful that this causes the entire industry to say, ‘Oh, my, we’ve got to fix this.’”
The algorithm wasn’t intentionally racist — in fact, it specifically excluded race. Instead, to identify patients who would benefit from more medical support, the algorithm used a seemingly race-blind measure: how much patients would cost the health care system in the future. But cost isn’t a race-neutral measure of health care need. Black patients incurred about $1,800 less in medical costs per year than white patients with the same number of chronic conditions; thus the algorithm scored white patients as equally at risk of future health problems as black patients who had many more diseases.
Machines increasingly make decisions that affect human life, and big organizations — particularly in health care — are trying to leverage massive data sets to improve how they operate. They utilize data that may not appear to be racist or biased, but may have been heavily influenced by longstanding social, cultural and institutional biases — such as health care costs. As computer systems determine which job candidates should be interviewed, who should receive a loan or how to triage sick people, the proprietary algorithms that power them run the risk of automating racism or other human biases. […]
