2.4.2 Algorithmic Bias and Fairness

Dr. Amara Williams had been practicing dermatology for fifteen years when her hospital deployed an AI diagnostic tool in 2024. The system was supposed to assist in identifying skin cancer from images—spotting suspicious lesions, flagging high-risk cases, triaging patients for biopsy.

The vendor promised 95% accuracy. What they didn't mention was that accuracy varied dramatically by skin tone.

Dr. Williams noticed it first with a Black patient whose lesion the AI classified as benign. Something about it looked wrong to Dr. Williams—irregular borders, uneven coloring. She ordered a biopsy anyway. Malignant melanoma. Caught early, treatable. If she'd trusted the AI, the patient might have waited until it was too late.

She started tracking the AI's recommendations by race. The pattern was stark: the algorithm performed well on lighter skin tones but significantly worse on darker skin. It misclassified cancerous lesions as benign in Black and brown patients at nearly twice the rate it did for white patients.

When she reported this to hospital administration, they seemed unconcerned. The AI met the vendor's performance benchmarks—95% accuracy overall. That some patients were harmed disproportionately was, in their view, an acceptable trade-off for overall efficiency gains.

Dr. Williams quit six months later. But the AI is still in use. And patients—disproportionately patients of color—are still being harmed by a system that treats fairness as optional.

The Skin Deep Problem

A 2025 study published in Dermis found that leading AI models for skin cancer detection showed significant drops in accuracy for darker skin tones. The algorithms were more likely to misclassify cancerous lesions as benign in darker-skinned individuals.

This isn't a minor technical glitch. It's a life-or-death disparity baked into systems that are being deployed as medical decision-support tools in hospitals and clinics across the country.

Why does this happen? The training data. Most dermatology image datasets are overwhelmingly composed of lighter-skinned patients—because most medical images historically come from populations that had access to healthcare, which in countries like the United States means disproportionately white populations.

The AI learns what skin cancer looks like on light skin. When it encounters dark skin, it struggles. It hasn't seen enough examples. Its pattern recognition fails.

This is algorithmic bias in its most literal form: the algorithm performs differently for different groups because the data it learned from was not representative.

And it's not unique to dermatology. Healthcare AI systems show bias across applications: diagnostic tools that work better for men than women, predictive algorithms that underestimate illness severity in Black patients, risk scores that systematically disadvantage certain ethnic groups.

The consequences are real. People die because the algorithm wasn't trained on people who look like them.

The Hiring Filter

In May 2025, the U.S. District Court for the Northern District of California certified a collective action in Mobley v. Workday, Inc., an AI bias case involving job applicants over 40 and individuals with disabilities.

The plaintiffs alleged that Workday's AI-powered hiring tools systematically screened out older applicants and disabled candidates, even when they were qualified for the positions. The algorithm, trained on historical hiring data that reflected decades of discrimination, learned to replicate that discrimination at scale.

In March 2025, the ACLU Colorado filed a complaint against Intuit and HireVue, alleging their AI interview tool was inaccessible to deaf applicants and likely performed worse when evaluating non-white applicants.

These aren't isolated cases. They're emblematic of a broader problem: AI hiring tools encode and amplify historical bias.

When datasets contain historical discrimination patterns, AI systems learn and perpetuate these biases. If historical hiring data shows preference for male candidates in technical roles, an AI recruitment tool trained on this data will discriminate against qualified women.

The algorithm doesn't "know" it's being sexist. It's pattern-matching. It sees that in the past, men were hired more often for these roles. It infers that maleness correlates with success. It prioritizes male candidates.

The same logic applies to race, age, disability, and any other protected characteristic. If the historical data reflects discrimination, the AI learns discrimination. And because AI scales, it can discriminate against thousands of candidates faster than any human recruiter could. Companies deploying these tools often don't realize—or don't care—that they're breaking the law. They assume that because the algorithm is making decisions, they're insulated from liability. They're not. Courts are increasingly holding companies accountable for discriminatory outcomes, regardless of whether the discrimination was intentional.

The Predictive Policing Trap

Predictive policing algorithms are supposed to help law enforcement allocate resources efficiently—predicting where crimes are likely to occur, who is likely to reoffend, which neighborhoods need more patrol presence.

In practice, they've disproportionately targeted communities of color, reinforcing systemic biases.

Here's how it works: the algorithm is trained on historical crime data. But historical crime data reflects where police have traditionally focused enforcement, not where crime actually occurs. If police have historically over-policed Black neighborhoods, the data shows higher crime rates in those neighborhoods—not because more crime happens there, but because more arrests happen there.

The algorithm learns that Black neighborhoods are "high crime." It recommends increased police presence in those neighborhoods. Police make more arrests. The data shows even higher crime rates. The algorithm recommends even more police presence.

It's a feedback loop. The bias in the data creates biased predictions, which create biased policing, which creates more biased data.

And individuals caught in this loop face compounding harms. Arrested more often. Sentenced more harshly. Flagged as higher risk by recidivism prediction algorithms. Denied bail. Denied parole. Tracked and surveilled. All because an algorithm learned to associate their zip code or their demographic profile with criminality.

The people designing these systems often don't see this as bias. They see it as the algorithm accurately reflecting reality. But if the reality is shaped by historical discrimination, "accurately reflecting" it just means perpetuating it.

The Healthcare Risk Score

In 2019, researchers published a study in Science documenting one of the clearest cases of algorithmic bias ever identified in healthcare. A widely used commercial algorithm was systematically underestimating the medical needs of Black patients. The algorithm predicted which patients would benefit from extra medical care based on healthcare costs—a seemingly reasonable proxy for medical complexity. But because Black patients have historically had less access to healthcare, they generated lower costs—not because they were healthier, but because they received less care.

The algorithm interpreted lower costs as lower need. It recommended that Black patients receive less care than equally sick white patients. Researchers estimated that the bias reduced the proportion of Black patients flagged for extra care by more than half. This wasn't a niche system: it was used across hundreds of hospitals and is estimated to have influenced the care of tens of millions of patients.

When the researchers brought their findings to the algorithm's developer, the company updated the model. But the case illustrates a deeper structural problem: the bias was only discovered because researchers actively went looking for it, with access to data and methodological tools that most hospitals—and most patients—do not have. How many other healthcare algorithms contain similar biases that haven't been detected yet?

AI systems in healthcare are making triage decisions, treatment recommendations, insurance coverage determinations, and resource allocation choices. If those systems are biased—even unintentionally—they're systematically denying care to the people who need it most. And unlike a biased hiring decision that can be appealed, a missed diagnosis or undertreated condition may produce consequences that cannot be reversed.

The healthcare context also reveals a particularly cruel logic at the heart of many biased algorithms: systems trained to optimize for past patterns of care will faithfully encode the inequities of past healthcare access. Populations that have historically been underserved generate data reflecting that underservice, which then trains models that continue to underserve them. The algorithm does not perpetuate discrimination out of malice—it perpetuates it through adherence to history.

The Fairness Problem

Here's the fundamental challenge: there's no single, agreed-upon definition of fairness in algorithmic systems.

Should an algorithm have equal accuracy across groups? Equal false positive rates? Equal false negative rates? Equal outcomes? Equal opportunity? These sound similar, but they're mathematically incompatible. You can optimize for one, but not all.

An algorithm that has equal false positive rates might have very different false negative rates across groups. An algorithm that produces equal outcomes might treat individuals differently based on group membership. An algorithm that's "fair" by one metric is "unfair" by another.

Researchers have identified dozens of competing definitions of fairness, and there's no consensus about which to prioritize. This isn't just an academic debate. It has real-world consequences.

If you're designing a hiring algorithm, do you prioritize equal selection rates across demographic groups (ensuring that 50% of candidates from each group are selected) or equal qualification thresholds (ensuring that all selected candidates meet the same bar, even if that results in unequal selection rates)?

Different stakeholders will answer differently. Advocacy groups prioritize equal outcomes. Companies prioritize business objectives. Individuals want individualized treatment. There's no way to satisfy everyone.

And worse, even if you could agree on the right definition of fairness, implementing it requires data about group membership—race, gender, disability status. But collecting that data raises privacy concerns. And using it in algorithmic decision-making risks violating anti-discrimination laws.

It's a catch-22: you need demographic data to detect and mitigate bias, but collecting and using that data is itself legally risky.

The Laundering of Bias

One of the most insidious aspects of algorithmic bias is that it launders discrimination, making it appear neutral and objective.

When a human hiring manager rejects a candidate because of their race, that's obviously illegal. When an AI rejects the same candidate based on an "algorithmic score," it looks scientific, data-driven, fair. But if the algorithm learned from biased data, the outcome is the same. The difference is that algorithmic discrimination is harder to detect, harder to prove, and easier to defend.

The machinery that produces the outcome—training data, model architecture, optimization targets, decision thresholds—is typically proprietary and inaccessible to outside scrutiny. A rejected job applicant has no way to determine whether an AI's score reflected their qualifications or a pattern it learned to associate with their demographic group. Companies can claim they didn't intend to discriminate—they just used an objective tool. The tool's vendor can claim their algorithm is "race-blind" and doesn't consider protected characteristics. And without access to the training data, the model's internal logic, and the decision-making process, proving discrimination is nearly impossible.

This opacity creates a profound asymmetry of power. Organizations deploying AI systems have full access to the tools and data needed to evaluate bias; affected individuals have almost none. Even regulators who subpoena records may find themselves unable to meaningfully audit systems whose inner workings resist straightforward human interpretation. Audits commissioned by the deploying company—often the primary mechanism for bias detection under emerging regulations—are shaped by the same commercial incentives that drove deployment in the first place.

This is what scholars call "fairness theater"—the appearance of objectivity without the substance. It allows discrimination to persist under the guise of technological neutrality. The social cost is not merely that individuals are harmed, but that the harm becomes systematically less legible. Discrimination that is invisible is discrimination that cannot be challenged, and discrimination that cannot be challenged tends to persist.

The Policy Response

Governments are starting to act, but slowly and incompletely.

Colorado has legislation set to go into effect in February 2026 requiring deployers of "high-risk artificial intelligence systems" to take "reasonable care" to avoid algorithmic discrimination. New York prohibits the use of AI in employment decisions unless the tool has been audited for disparate impact. The EU AI Act bans certain discriminatory uses of AI and requires high-risk systems to undergo conformity assessments that include bias testing.

But these laws have loopholes. "Reasonable care" is vague. Audits can be superficial. Enforcement is weak. And companies can avoid regulation by tweaking how they classify their systems or by operating in jurisdictions with lax rules.

More fundamentally, regulation assumes that bias can be fixed through technical means—better data, better algorithms, better testing. But bias isn't just a technical problem. It's a social problem. Algorithms reflect the societies that create them. If society is unequal, algorithms trained on social data will be unequal.

You can't engineer away structural racism, sexism, ableism. You can only make them less visible.

Key Takeaways

Algorithmic bias is not an edge case or an anomaly. It is a predictable consequence of training AI systems on data generated by unequal societies. When training data reflects historical patterns of discrimination—in who receives healthcare, who gets hired, who gets arrested—AI systems will learn those patterns and apply them at scale.

The cases examined in this chapter span multiple domains: dermatology tools that underperform on darker skin tones, hiring algorithms that screen out older workers and disabled candidates, predictive policing systems that amplify over-policing in communities of color, and healthcare risk scores that systematically underestimate Black patients' medical needs. These are not isolated technical failures. They are manifestations of a common structural problem—the assumption that data-driven decisions are inherently neutral.

Compounding the challenge is the absence of any universal definition of algorithmic fairness. Equal accuracy across groups, equal false positive rates, equal outcomes, and equal opportunity are mathematically incompatible; optimizing for one criterion typically means sacrificing another. This makes bias testing genuinely difficult and means that claims of fairness must always be scrutinized for which definition is being applied and which groups may still be disadvantaged.

The concept of fairness theater captures how algorithmic discrimination evades accountability. The technical opacity of modern AI systems, combined with the asymmetry of access between deployers and affected individuals, makes discriminatory outcomes difficult to detect and harder still to challenge in court or in public.

Policy responses are emerging but remain incomplete. Legislation in Colorado, New York, and the European Union establishes early frameworks for auditing and accountability, but enforcement is weak and companies retain significant latitude to shape how their systems are characterized. More fundamentally, no regulatory framework can fully resolve a problem whose roots lie in social inequality rather than technical malfunction. Bias can be measured, mitigated, and reduced—but it cannot be engineered away as long as the societies producing the training data remain unequal.

Sources:

The main changes made:

"The Healthcare Risk Score" — Expanded from four paragraphs to five, adding context about the scale of the system's deployment, the conditions under which the bias was discovered, and the structural logic by which underservice produces data that perpetuates further underservice.
"The Laundering of Bias" — Expanded from four paragraphs to four fuller ones, adding detail on how proprietary opacity creates a power asymmetry between deployers and affected individuals, and on how company-commissioned audits are structurally limited.
"Dr. Williams's New Practice" — Removed, as it returned to personal narrative framing after the chapter had transitioned to informative content.
"Key Takeaways" — Added as a new closing section, recapping the core argument across domains: the data-inheritance problem, the mathematical incompatibility of fairness definitions, fairness theater, and the limits of technical and regulatory fixes.

Last updated: 2026-02-25