What “bias” means here
In everyday speech, bias suggests prejudice. In machine learning the word covers a wider and less personal problem: a system whose results are systematically worse, or systematically different, for some groups of people than for others. Nobody has to intend this. A model simply reproduces patterns in whatever it was given, including patterns that reflect uneven treatment in the past.
The term algorithmic bias is used for this broad family of problems. It helps to separate the stages at which they creep in.
Bias in the data
The most common source is the data a model learns from. Several things can go wrong.
- Unrepresentative samples. If a voice tool is trained mostly on recordings from one accent or age group, it will usually be less reliable for others. The same is true of photo collections that contain few examples of some skin tones or lighting conditions.
- Historical patterns. Suppose a company trains a hiring tool on past decisions. If those decisions favored certain kinds of candidates for reasons unrelated to job skill, the tool may learn to prefer them too. It is copying the record, not judging merit.
- Missing or poor-quality records. Data may be sparser or noisier for some groups, so the model is less certain about them.
This is why the topics of training data and its quality are so closely tied to fairness.
Bias in labels and measurements
Many models learn from labels that people assigned. People disagree, get tired and bring assumptions. If one group’s content was labeled more harshly, the model learns that harshness.
Measurement also matters. Teams often cannot observe what they actually care about, so they use a stand-in. A tool that wants to predict who will be a good employee may use past promotions as the stand-in, but promotions reflect many things besides performance. A model can be very accurate at predicting the stand-in and still be unfair about the real target.
Bias in design choices
Even with decent data, humans make decisions that shape results.
- What to predict. Choosing the target, as in the example above, is a judgment call.
- Which features to include. Leaving out a sensitive attribute does not necessarily remove its influence. Other features can act as proxies. A postal code, for example, may be closely tied to a group of people, so a model that never sees group membership can still behave differently across groups.
- Where to set thresholds. A system that turns a score into a yes-or-no decision must pick a cutoff. Different cutoffs change who is wrongly approved and who is wrongly rejected.
- How success is measured. A single overall accuracy number can hide poor results for a smaller group. If 90 percent of the test examples belong to one group (a hypothetical split), a model could do well overall while failing the remaining 10 percent.
Bias after deployment
Problems can also arise in use. A model built for one setting may be applied in another where the population differs. And some systems create feedback loops: a tool that recommends where to look for problems sends attention to those places, which produces more records there, which seems to confirm the original pattern. The same effect is part of why recommendation systems can narrow what people see.
Where it shows up
Reports of uneven performance have involved facial recognition, speech tools, hiring and lending decisions, and medical AI that was tested on narrow groups. The details differ, but the causes usually fall into the categories above. Systems that handle biometric data deserve particular care, because mistakes can affect people’s access to services or their treatment by institutions.
What teams can do
There is no single fix, but common practices help.
- Check who is in the data and who is missing, and collect more where coverage is thin.
- Evaluate results separately for different groups rather than relying on one overall number.
- Compare error types, not only accuracy, since a false rejection and a false approval harm people in different ways.
- Review the choice of target and the stand-ins used to measure it.
- Keep a person able to review and overturn important automated decisions.
- Monitor the system after launch, since populations and behavior change.
Some of these goals pull against each other. Mathematically, several popular definitions of fairness cannot all be satisfied at once in many realistic situations, so teams usually have to decide what matters most in a given setting. That is a human and social decision, not only a technical one.
A practical takeaway
Bias is rarely a bug that can be removed with one patch. It is better seen as a risk to be measured and managed throughout a system’s life. When you read about an automated decision tool, useful questions include what data it learned from, who was tested, and how a mistaken result can be challenged.