Artificial Intelligence systems are not objective or unbiased according to The Washington Post. Many of the inequalities that exist in society today, though, are being copied through these systems.
When AI models are trained on mostly one demographic they may introduce bias. Even if software developers strip out explicit demographic markers such as race or gender from training data, other features can be used by the system to introduce these traits. For example, zip codes, student organization participation or past spending habits of people can be used to determine race or gender.
According to the magazine Science, in an October 2019 article titled “Racial Biases in Healthcare Algorithms,” the demographics of data in which a machine learning model is trained have a significant effect on the model’s performance. For example, if a model is trained on a dataset of images including people with lighter skin, then the model will perform with images of people with light skin rather than on images of people with dark skin.
“A couple of years ago, every time the general user would ask an AI prompt to generate a picture of a CEO or doctor it would be a white male, but just recently it would bring someone who looks almost just like you, to mask it,” stated computer science teacher Victor Rinaldi.
Removing race and gender tags from training data for automated systems does not necessarily result in a neutral system, however. Many algorithms are able to reconstruct location, education level and behavior from other variables present in the training data.
Computer vision systems have varying levels of accuracy depending on the demographics of people in the images.
Machine learning models are known to have differing error rates for different demographics. In a study called the Gender Shades project conducted by the Massachusetts Institute of Technology Media Lab, commercial facial recognition software found that the software had an error rate of 34.7% for dark-skinned women and less than 1% for light-skinned men.
These systems are being used by many of the country’s largest hospital networks to assign clinical risk scores to patients to help determine which patients are most in need of hospital resources. However, in the Science article published in 2019, it was revealed that the systems use past healthcare spending as a backup and because on average Black patients have historically spent less on healthcare than white patients, the systems misclassify Black patients as being healthier than equally-ill white patients.
Many medical imaging software programs and dermatology training data sets contain few, if any, images of people of color. According to The Lancet Digital Health paper of September 2022, automated systems designed to diagnose skin conditions, for example, can perform poorly on people of color, misdiagnosing skin conditions and even misidentifying tumors.
Published in the Proceedings of Machine Learning Research in February 2018, MIT Media Lab findings showed that facial recognition models consistently show the highest failure rates when analyzing women with darker skin tones.
These systems can generate text that contains racial slurs, gender stereotypes and other systemic prejudices unless aggressive reinforcement filters are applied.
Larger language models can reproduce structure biases, hate speech and other stereotypes that they have learned from large web data. To prevent this, developers have to apply strong post-training adjustment measures, such as Reinforcement Learning from Human Feedback (RLHF), to correct toxic behavior and strengthen safety standards.
