Algorithmic Bias and the AI Mirror: When Prediction Becomes an Opportunity Gate

The Dashboard That Decides Who Gets Noticed

An adviser at a community college opens a student-success dashboard on Monday morning. Each student appears next to a color-coded risk score built by an AI system trained on years of grades, financial-aid patterns, attendance logs, advising notes, and clickstream data from the learning management system. The adviser cares about her students and has only a few hours for outreach before the rest of the day fills with appointments. She lets the top of the list guide her afternoon calls.

Diego is on the list. He is a first-generation student who works a night shift at a warehouse, does much of his reading on paper, uses a shared laptop at home, and logs into the LMS in short bursts around childcare. The system reads his sparse logins as low engagement, tags him high risk, and moves him to the top of the outreach queue. Kai is not on the list. She works from a laptop at home, logs in constantly, and looks like a strong-signal student on every measurable variable. She is also close to a breakdown. Nothing in the data the dashboard reads can see that.

This is a composite classroom case, not a claim about any particular CWI system. Its purpose is to make the ethical structure visible. The dashboard is doing something institutions often want to do: predict who might need help. The intention behind it is good. Colleges want to catch students before they drop out and offer attention early. Predictive systems like this one appear in education, credit, hiring, health care, benefits administration, criminal justice, content moderation, insurance, workplace management, and government services. Institutional decisions once made slowly by individual judgment are increasingly filtered through systems that score, sort, rank, recommend, or warn.

Wanting to help earlier is a legitimate institutional goal. The question this chapter asks is when that help becomes gatekeeping. The dashboard describes the world and decides something inside it. It sends attention toward some students and away from others. It flags some, warns some, and quietly leaves the rest alone. It shapes the flow of a scarce resource.

The dashboard can also be wrong in more than one way. Diego may receive misdirected attention: the system notices him, but under the wrong description, as if his life constraints were evidence of disengagement. Kai may receive missing attention: the system misses her because the distress that matters does not appear in the features it measures. Diego shows a distorted positive. Kai shows a false negative. Together they show why algorithmic bias is not only about whether a model is “accurate overall.” It is about what the system sees, what it misses, how it describes people, who bears error, and what the institution does next.

That flip is where algorithmic bias becomes a philosophical problem. A model can be accurate about the average case and still be ethically defective in how it treats particular people, how it distributes opportunity over time, whose errors it hides, and whether the people it labels can answer back. In PHIL 123, the point is not to say “the system is biased” and stop. The point is to name what kind of wrong is at stake, what evidence supports that diagnosis, and what kind of repair would actually address it.

What Algorithms And Bias Mean Here

Before the chapter can use Shannon Vallor’s mirror metaphor, the terms need care.

An algorithm, in this chapter, is a repeatable computational process that turns inputs into outputs such as scores, classifications, recommendations, rankings, or generated answers. Older algorithms could be hand-coded. Many current AI systems learn patterns from examples. A model is trained on data, finds relationships, and then applies those relationships to new cases. That is why a student-success dashboard, a credit score, a hiring screen, a content-moderation classifier, a medical risk model, and a chatbot can all belong in the same ethical conversation even though they look different on the surface.

Bias means something more specific than personal preference. Here, bias is a systematic tilt in how a system sees, measures, classifies, recommends, or acts. The NIST report on bias in AI describes a world where ambiguous human behavior is captured, quantified, categorized, sorted, recommended, and used to make decisions about people’s lives. NIST’s central warning is sociotechnical: harmful bias can remain endemic across technology processes and can cause harm regardless of whether anyone intended to discriminate.

Algorithmic bias happens when computational sorting carries that systematic tilt into consequential treatment. It can misrepresent a person, distribute errors unevenly, restrict opportunity, hide a worldview or political assumption inside a technical rule, or make the past look like objective evidence about the future. Bias can involve race, gender, class, region, disability, language, religion, politics, accent, age, immigration status, credit history, work schedule, family structure, device access, or broader worldview. Any of those can become a proxy, a classification, an error burden, or a hidden rule that shapes access to a good.

The Stanford Encyclopedia of Philosophy entry on algorithmic fairness is useful because it treats algorithmic fairness as a contested moral question, not merely a technical score. Technical measures matter. But the moral dispute is about what fairness requires in a particular context. Some fairness questions compare how groups are treated. Others ask whether a person was treated as they should have been treated at all.

Here is the chapter’s basic map:

Core concepts for diagnosing algorithmic bias

Concept	Simple definition	Student question
Mirror	The system reflects patterns in past data.	What record of the world is this model learning from?
Gate	The system helps allocate a scarce good.	What opportunity, support, burden, or risk is being distributed?
Proxy	The system measures one thing as a stand-in for another.	What human concern has been replaced by measurable data?
Error burden	Mistakes are not distributed equally.	Who bears false positives, false negatives, delays, denials, or extra scrutiny?
Contestability	Affected people can challenge the decision.	Can a person see, correct, appeal, or escape the system’s judgment?
Repair	The institution changes the system or refuses to use it.	Is the fix technical, procedural, institutional, structural, or a refusal to automate?

This chapter builds toward a Bias Judgment Audit. The audit is a tool for your Module 5 and final-project work. To build like a philosopher, you do not merely ask whether a system works. You ask what it is for, what it assumes, whom it burdens, what kind of wrong it could produce, and what an institution would have to change to make the system answerable.

From Data Mirror To Opportunity Gate

Shannon Vallor calls modern AI systems a kind of mirror. In The AI Mirror, she argues that machine-learning systems are trained on records of what human beings have already done, written, said, ranked, purchased, hired, arrested, treated, or ignored. The output looks like a fresh judgment. What it actually returns is a compressed reflection of past human and institutional behavior. The system does not know the world as a human participant knows it. It has learned patterns in how people and institutions have recorded the world.

Vallor’s metaphor sharpens the usual claim that AI is trained on data. A mirror shows the room selectively. It reflects some things clearly, blurs others, distorts near the edges, and hides whatever is not standing in front of it. A dashboard trained on students who succeeded at the college in past years learns those students’ patterns of logging in, completing assignments, and asking for help. It learns less about the students who never enrolled, who dropped in the first two weeks, who paid for a semester and vanished, who succeeded through paths the college did not record, or who were discouraged by institutional practices before they became data points. The mirror is partial by design.

The mirror metaphor should not be misunderstood. AI does not mirror reality itself. It mirrors a record of what institutions chose to measure, preserve, label, ignore, and reward. A system can reproduce an institution’s habits while sounding like an independent judge.

Vallor’s second move changes the ethical stakes. A mirror can be treated as though it were the room itself. The student-success dashboard gives the adviser an incomplete reflection, and the college treats that reflection as objective evidence. An adviser looks at Diego’s risk score and reads it as a fact about Diego, even though the score summarizes what the model has learned to associate with sparse LMS activity. The mirror has become a fact.

At the same time, the mirror decides something. The adviser has three hours on Monday. Those three hours are a scarce good. So is a scholarship email. So is a warning that lets a student catch a failing grade before the withdrawal deadline. So is a referral to tutoring, disability services, emergency aid, a food pantry, or a faculty mentor. When the model helps allocate those goods, the dashboard has stopped being only a mirror. It has become a gate.

Lisa Herzog develops this line in “Algorithmic Bias and Access to Opportunities” in The Oxford Handbook of Digital Ethics. Her argument is that algorithmic systems become especially serious when they mediate access to important goods: jobs, credit, care, education, housing, support, credibility, safety, benefits, and political voice. Bias in a movie recommender is annoying. Bias in a hiring screen, hospital triage system, benefits tool, or college advising dashboard can shape a life.

Herzog’s point is not that technical fairness does not matter. It is that technical fairness does not settle the whole moral question once a system sits at an opportunity gate. The question is not only whether the model predicts accurately. The question is who gets in, who is delayed, who is sent away, who is misdescribed, who is asked to prove themselves again, and who gets locked out of the goods a life depends on.

The dashboard is at that kind of gate. The scarce good is advising attention, sometimes financial aid, sometimes a referral to human support. Diego and Kai enrolled at community college because each wants a shot at a different life. What the system does with their profiles helps decide whether that shot narrows or widens.

You will use this distinction throughout the chapter. An inaccurate prediction is a technical problem. An unjust allocation is a moral problem. When students say a system is biased, they are usually talking about the second thing, whether they know it yet or not.

When A Proxy Replaces The Person

Once a system is at an opportunity gate, the next question is what the system actually measures. Almost every AI model that helps allocate opportunity has to work with a stand-in. The thing the institution cares about is rarely something the model can see directly. Instead, the model watches something else, called a proxy, and treats it as a workable substitute.

The dashboard is a good example. The college cares about learning, persistence, belonging, and academic progress. The model cannot see those things directly. It sees LMS logins, submission timestamps, quiz scores, registration changes, email responses, and financial-aid records. Those variables stand in for the concepts the college actually cares about. Sometimes the substitute is close enough to help. When the substitute drifts from the concept, a whole class of ethical problems opens up.

Ziad Obermeyer and colleagues published the clearest example of this problem in a 2019 Science study, “Dissecting racial bias in an algorithm used to manage the health of populations”. The system was used in large U.S. health-care settings to identify patients for high-touch care management programs. Those programs could include additional outreach, help coordinating specialists, and closer attention between visits. Getting into the program could matter enormously for patients with chronic disease.

The algorithm did what its designers asked it to do. It predicted, from prior claims and clinical records, how much health-care spending a patient was likely to generate. Patients projected to be expensive were flagged as high risk and pushed toward the care program. The design assumption was that sicker patients cost more, so predicted cost was a reasonable stand-in for medical need.

Obermeyer’s team showed why that proxy was ethically dangerous. At a given risk score, Black patients were substantially sicker than white patients on clinical indicators. The model had learned to associate high medical need with high past spending. But Black patients in the data had lower past spending than their clinical need suggested because access to care had already been unequal. Their medical need did not show up as cost because the health system had not spent the money to meet it. The model, faithful to its target, folded that history forward.

Notice what this case shows. The model did not simply malfunction. It was accurate about its target. Predicted spending really did track spending. The problem was that spending had been chosen as a proxy for need, and the proxy carried unequal history forward. A sick Black patient could look lower risk because the system had historically spent less on that patient’s care.

The researchers showed that using health-based labels rather than cost-based labels could substantially reduce the disparity. That is an important lesson. The right response was not “algorithms are always bad.” The right response was “the target has to be examined morally, not merely optimized technically.”

The same structure appears in the dashboard case. The college is not indifferent to Diego’s learning. It has chosen LMS clicks as a proxy for engagement because clicks are easy to measure. If the college’s real concern is whether Diego is learning, clicks are a poor stand-in for a student whose reading and thinking mostly happen off the platform.

The Fairness and Machine Learning online book by Solon Barocas, Moritz Hardt, and Arvind Narayanan makes the general form of this problem visible. Predictive models optimize the target you give them. The target is a definition of the problem, and the definition of the problem is a normative choice about what counts and what does not. When the choice is between a clean target the model can hit and a messier concept the institution actually cares about, the clean target usually wins. The messier concept quietly gets replaced.

The Stanford Encyclopedia of Philosophy entry on algorithmic fairness treats this as the proxy problem, and it treats it as one of the central issues in the philosophy of fair prediction. Once a proxy replaces the person, you can be accurate about the proxy and still be wrong about the person. That is what Obermeyer’s team documented in health care. It is also what the dashboard risks doing when it treats a click pattern as engagement, when a lender treats a zip code as creditworthiness, when a hiring model treats a resume gap as reliability, when a language tool treats accent as comprehensibility, or when a moderation system treats political or religious language as toxicity without enough context.

Every judgment you make about algorithmic bias should start with the proxy question. What did the model actually measure? What human concern was that measurement standing in for? How far apart did the two travel? The next step is to locate where bias entered the system.

Where Bias Enters

Students often say “the data was biased.” Sometimes that is true. But it is too vague to guide repair. Bias can enter before data collection, during labeling, during modeling, during evaluation, at deployment, through human workflow, or through feedback loops after the system is used.

Batya Friedman and Helen Nissenbaum’s classic article “Bias in Computer Systems” distinguished among preexisting, technical, and emergent bias. Preexisting bias comes from the world and institutions around the system. Technical bias comes from design decisions such as categories, formal rules, or system constraints. Emergent bias appears when a system enters a context of use that changes over time or differs from what designers expected. Harini Suresh and John Guttag’s framework for sources of harm across the machine-learning life cycle makes a similar point for modern ML: harm can enter through data generation, data collection, data preprocessing, model development, model evaluation, deployment, and feedback after deployment. NIST’s SP 1270 adds the policy lesson: managing bias requires looking at the whole sociotechnical process, not only the model file.

Here is a practical map:

Three labels are especially useful for PHIL 123 projects.

Latent or system bias is built into the data, labels, categories, target, or model before a particular decision is made. Obermeyer’s cost proxy is a system-level problem. The issue was not one doctor’s prejudice. The target carried unequal history.

Deployment or context bias appears when a system enters a local institution. A dashboard trained on residential full-time students may behave differently when used for working adults, online students, multilingual students, veterans, parents, rural students, or students with inconsistent broadband access. The model may not be “bad” in the abstract. It may be bad for this gate, in this institution, with this population.

Interaction risk appears when humans respond to the system. Staff may overtrust scores, ignore students who are not flagged, treat risk labels as character judgments, or use the dashboard to justify resource cuts. A model can change human attention even when the model’s output is advisory.

This map matters because different locations of bias require different repairs. A representation problem may need better data. A proxy problem may need a different target. A deployment mismatch may need local validation. A contestability problem may need appeal rights. A structural opportunity problem may require changing the institution, not the model.

Wronging The Person

Assume the proxy question has been asked. Assume the model has been trained on real patterns and deployed with care. Somebody still gets hurt. The question a philosopher of discrimination asks is what makes that harm a wrong done to the individual person.

Deborah Hellman’s account in When Is Discrimination Wrong? starts with a moral intuition many students share. Not every unequal treatment is discriminatory. Age-based licensing is unequal. Height requirements for pilots are unequal. Grading rubrics are unequal. What makes some classifications feel like a wrong against the person is that they demean: they express, enact, or reinforce lower social standing from a position of social power. Demeaning treatment is not merely treatment that hurts someone’s feelings. It is treatment that treats someone as lesser in status or worth.

Sophia Moreau develops a pluralist view in Faces of Inequality. She argues that discrimination can wrong people in several ways: by subordinating them, by restricting their deliberative freedom, or by denying them access to basic goods on grounds unrelated to those goods. Moreau’s account helps explain why a person can be wronged by an algorithm even when no one explicitly insults them. Diego cannot argue with the LMS about how he studies. The dashboard has already decided what his behavior means.

Benjamin Eidelson’s Discrimination and Disrespect sharpens the intuition into a claim about personhood. Discrimination is intrinsically wrong when it manifests disrespect for the person as a rational, individual agent. To treat someone through a group profile they did not choose, without attention to the particular person in front of you, is to fail to acknowledge them as a person.

Read together, Hellman, Moreau, and Eidelson give students a family of individual-wrong accounts. They all locate the wrong in what the system does to the person: the meaning of the classification, the restriction of agency, the failure to treat the person as someone who can answer for themselves. The SEP entry on discrimination surveys these accounts alongside harm-based views, which locate the wrong in concrete damage done. Both families matter for algorithmic cases. The individual-respect view is useful when the classification itself is the wrong. The harm-based view is useful when the classification produces concrete losses.

Take Obermeyer again. On the individual-wrong reading, the wrong done to a Black patient scored as lower risk is that the health system treated her through a proxy that reduced her clinical reality to a spending record. The system saw a lower-cost claim history and missed the person with uncontrolled diabetes, hypertension, or heart disease. No human doctor had to sneer at her for the disrespect to be present. An institutional process had already replaced her with a number that carried the meaning of a history she did not choose.

The individual-wrong view has limits. It works well when you can locate a particular person whose respect, agency, or standing was diminished. It works less well when the harm is spread thin across many people, none of whom can point to one moment when they were personally insulted. Herzog and other structural theorists argue that some algorithmic systems produce harms of this second kind, and that reducing every case to an individual wrong loses something real about how modern systems shape whole populations. The next section takes that reply seriously.

For the philosophy of discrimination, though, the individual-wrong tradition gives students their first useful vocabulary. When you ask what kind of wrong is at stake in an algorithmic case, the first question is whether a particular person is being treated through a classification that fails to treat them as an equal. If yes, the language of Hellman, Moreau, and Eidelson can help you name what has gone wrong even when the model is accurate on average.

Wronging The Opportunity Structure

The dashboard case can be described a second way that does not run primarily through one person being demeaned.

Iris Marion Young’s social connection model, from Responsibility for Justice, argues that many serious harms in modern life are not produced by one identifiable wrongdoer. They are produced by the ordinary functioning of institutions whose rules, defaults, incentives, categories, and feedback loops leave some groups reliably worse off. Young calls this structural injustice. Nobody at the college is malicious. The adviser wants to help. The vendor sold a product intended to save at-risk students. The IT staff followed procurement rules. The problem can still be structural if the system repeatedly routes some students into thin, stigmatizing, or ineffective forms of attention while missing other students whose need does not match the data trail.

This fixes the Diego/Kai case. Diego is not simply receiving less attention. He may receive the wrong kind of attention: a risk label built from thin proxies rather than a serious understanding of work, childcare, commuting, disability, shared-device access, language, family obligation, or financial pressure. Kai shows the opposite danger: a student can look fine to the dashboard while needing help the system does not know how to see. A biased system can over-detect some students under the wrong description and under-detect others because their need is invisible to the measured features.

Herzog applies Young’s structural injustice frame directly to algorithmic bias. When a college, hospital, bank, landlord, employer, or public agency uses a model to decide who gets in, the model becomes part of the institutional machinery that shapes opportunity. If the model reflects a history of unequal access, the resulting decisions can reproduce and intensify that history. Herzog uses Robert Merton’s language of Matthew effects to describe the pattern. Small advantages compound. A student flagged as promising gets attention, which produces engagement data, which confirms the flag. A student flagged as risky gets generic warning messages, which feel like being watched, which reduce engagement, which confirms the flag. The prediction has helped make itself true.

Feedback loops are central here. A biased system can create a record that later systems treat as evidence. If Diego is repeatedly marked as risky, receives generic interventions, misses human advising, or is routed into narrower options, later records may confirm the system’s original picture. If Kai is repeatedly missed, her need may only appear after the situation becomes a crisis. Bias becomes self-reinforcing when the output of one decision becomes the input for the next.

Kathleen Creel and Deborah Hellman push the point further in “The Algorithmic Leviathan”. Their argument is that a single algorithmic decision may not obviously wrong someone. What changes is scale and repetition. When college applications, loan applications, job screens, insurance rates, rental applications, and benefits checks all use similar systems with similar proxies and similar assumptions, a person who scores badly on one is likely to score badly on many. Being flagged is no longer a local event. It becomes an institutional condition.

The FAccT paper “Algorithmic Pluralism”, by Shomik Jain, Vinith Suriyakumar, Kathleen Creel, and Ashia Wilson, offers a related move by extending Joseph Fishkin’s theory of opportunity bottlenecks into algorithmic decision-making. Fishkin argues that fair opportunity depends on there being many routes through life’s bottlenecks. If one narrow gate controls whether you can become a nurse, homeowner, engineer, teacher, or small-business owner, then anyone screened out at that gate loses more than one opportunity. Algorithmic systems can narrow gates when many institutions rely on similar screening logic. A single gate can be dangerous even when it is accurate, because it has no redundancy. If it misjudges a person, the person has nowhere else to go.

Bring this back to the dashboard. The individual-wrong view asks whether Diego was treated as an equal. The structural view asks what happens to the group of students whose lives do not line up with an LMS. It asks whether the college’s system, combined with lenders, landlords, employers, benefits offices, and other institutions, produces a life in which a particular kind of person is repeatedly misread or quietly locked out of goods others can access. On this view, the wrong is not only a moment of disrespect. It is a slow narrowing of opportunity that no one person chose and that many people have to live inside.

The SEP entry on equality of opportunity treats this as a live question in political philosophy. The SEP entry on distributive justice surveys the broader family of views about how institutions should distribute benefits and burdens. Both surveys make the same background point: fair opportunity is not the same as accurate prediction. A society can have models that predict outcomes well and still be unjust if the outcomes it predicts are being produced by an unjust process.

The structural view has its own limits. It can miss the individual whose complaint begins with how they were treated, even when no broad population pattern is visible. It can also tempt people to say no one is responsible for any particular decision, which is false. But it captures something the individual-wrong tradition cannot. Some of what algorithmic bias does happens across time, across institutions, across many small decisions, and its wrong is best seen at that scale.

The four can overlap. In the dashboard case, Diego may have a personal complaint about being read through a template. He may also be one of many students whose opportunity is being narrowed across institutions. The college may have a welfare tradeoff hidden inside its metric. And he may have no way to see or contest the score. When students ask which wrong is at stake, the honest answer will often be “several.” Your job as an ethical thinker is to name the wrong that drives the case, so that you can say what the institution actually has to change.

Which Fairness Counts?

Once you have decided that a system is at an opportunity gate and that its proxy carries moral weight, you might reasonably ask whether a technical fix could handle the rest. The fairness literature in machine learning has spent years trying to define what a fair model should satisfy.

The Stanford Encyclopedia entry on algorithmic fairness and Fairness and Machine Learning walk through the main options. The details can become mathematical, but students need the basic conclusion first. Fairness is not one thing.

Choosing a fairness metric is not only a math choice. It is a moral choice about which kind of wrong matters most in the context.

You can feel the trouble coming. Suppose a lender wants applicants from two regions to have the same repayment likelihood at the same score, wants borrowers who would repay to be approved at the same rate in both regions, and wants the same overall approval rate in both places. The mathematics will not always cooperate. In many real settings, especially when base rates differ and predictions are imperfect, several appealing fairness criteria cannot all be satisfied at once. This does not mean fairness is impossible. It means fairness criteria answer different moral questions, and institutions have to decide which tradeoff they can defend.

Selbst and colleagues make a related but distinct point in “Fairness and Abstraction in Sociotechnical Systems”. Even when a fairness metric is picked and a model is tuned to satisfy it, the metric is defined at the level of the model. The world the model enters is not the world the metric described. A hiring model that is calibrated on training data can still produce unfair results once it is deployed inside an actual workplace, with actual managers, applicant behavior, incentives, and downstream consequences. Their warning is that treating fairness as a property of the model alone hides the parts of the problem that live outside the model’s boundary.

The same structure appears when the scarce good is speech, visibility, or credibility. In ACL 2024, Bang, Chen, Lee, and Fung’s “Measuring Political Bias in Large Language Models” argues that political bias can appear in both what a model says and how it says it. A 2024 FAccT paper on algorithmic arbitrariness in content moderation shows a related risk in moderation systems: different classifiers can perform similarly on average while assigning conflicting toxicity labels to the same content. A 2026 npj Artificial Intelligence study by Maarten Buyl and colleagues, “Large language models reflect the ideology of their creators”, finds that LLM outputs can reflect differences across regions, languages, and design choices. These examples broaden the chapter’s frame. Automated systems can classify beliefs, speech, religion, politics, or worldview through rules the affected people may not see.

Keep this part compact. The chapter is not mainly about political bias in chatbots. The lesson is that algorithmic bias can involve worldview as well as demographic categories. A system can distribute credibility, visibility, or suspicion in ways that matter even when no one is denied a loan or a job.

Return to the dashboard. Suppose the college wants its risk scores to satisfy predictive parity. Students at the same score complete at the same rate across every group. That sounds reassuring. It is still compatible with the model concentrating false-negative errors on working-adult students, so that the students most likely to be missed by the system are the ones already fighting the hardest to stay in school. Predictive parity is a real fairness property, and it is not always the property that helps the person the model gets wrong.

Which fairness counts is a question the institution has to answer, and the model cannot answer it for them. Some errors carry more weight than others. Missing a heart patient carries more weight than flagging a healthy patient. Missing a struggling student may carry more weight than sending a strong one an extra email. The fairness metric can make those tradeoffs visible. Deciding which tradeoff is defensible is moral work, and it belongs to the people responsible for the system, guided by the affected community.

Herzog’s stronger claim is that no purely technical adjustment settles the underlying problem. If a system is at an opportunity gate, and its errors are being borne by people who already had less access, tuning a fairness parameter does not touch the reason those people had less access in the first place. Better fairness metrics can help. They cannot substitute for the institutional work of asking what the system is for and who is being burdened by it.

Better Than Human Is Not Enough

The strongest objection to the argument so far is not that algorithmic bias is fake. The strongest objection is that human decision makers are biased too. They are tired, hurried, distracted, inconsistent, and influenced by institutional incentives. Hiring, admissions, health care, lending, discipline, policing, and benefits systems already contain unequal histories. If an algorithm is less biased than the humans it replaces, is it not ethically better to use the algorithm, even when it remains imperfect?

Take the objection seriously. In some cases the answer is yes. A well-designed screen used as one input, alongside meaningful human review, can reduce concentrated biases that a single tired reviewer might carry through a whole file. The Obermeyer case itself does not show that algorithmic triage should always be abandoned. It shows that the proxy must be examined and changed. A health-needs target can be better than a cost target. A model with better labels, evaluation, monitoring, and contestability may improve care compared with the human status quo.

Some settings make refusal morally costly. A rural clinic that cannot afford full-time specialists may catch more strokes with an imperfect triage model than without one. A benefits office with a backlog may leave more people without support if it refuses every automated aid. A college with too few advisers may reach more students if a dashboard helps identify patterns human staff would otherwise miss. Refusal in the name of ethical purity is a real move that carries a real bill, and the bill is often paid by the people the system was supposed to help.

The objection has limits. Better on average is not the same as better across the board. A model that reduces the average error rate can concentrate its remaining errors on a smaller group whose lives depend on being seen. Comparative performance is one variable in a moral judgment. It cannot settle the proxy question, the opportunity question, or the contestability question by itself.

Ask what the model actually predicts. If the target is a proxy and the proxy carries unequal history, better average performance may hide the same wrong in a smoother form. Ask who bears the errors. If false negatives fall on the group already least served, the model may be more accurate overall while doing more damage where it counts. Ask what the model gates. If the good it allocates is decisive for a life, small percentage improvements can still leave particular people locked out. Ask what the affected person can do. If a decision can be made about you by a system you cannot see and cannot challenge, then even a well-performing system may be procedurally unjust.

The honest form of the objection is that deployment can sometimes be justified when a model outperforms the human status quo on the metrics that matter, when its proxies have been examined and revised, when its errors do not fall unevenly on vulnerable groups, when its use improves rather than narrows access, and when decisions remain answerable to the people they touch. That is a demanding standard. It does not translate into a rule that any better-than-human system should be used. It is closer to a checklist for institutions considering deployment.

Who Can Answer The Decision?

The last stake in the chapter is procedural. Even a system that measures well, respects the person, distributes errors fairly, and preserves opportunity can still be ethically defective when the people affected by it have no way to answer back.

Danielle Citron and Frank Pasquale’s work on algorithmic due process, developed in “The Scored Society”, argues that consequential automated decisions have to be answerable in a specific sense. The person affected needs to know an algorithm was involved, understand the general basis of the decision, have a way to challenge specific errors, and have a route to a real decision maker when the system is wrong. Transparency in the abstract is not enough. Affected people need to be able to act on what they learn.

Kars Alfrink and colleagues develop this into a design agenda in “Contestable AI by Design”. Their argument is that contestability has to be built into a system from the beginning, and that it has to include the whole lifecycle: how the model is developed, how it is trained, how it is deployed, how it can be challenged in individual cases, and how it can be repaired or withdrawn when it is doing more harm than good. A late-added appeal form is not enough. Contestability means the system has been designed on the assumption that people will need to respond to it.

Creel and Hellman’s structural argument explains the same problem at ecosystem scale. If your loan denial, rental screen, and job application are all run through similar automated systems, and each one has a technically valid appeal process nobody actually uses, the aggregate effect is that you cannot answer any of them. The right to contest can be present on paper and absent in practice. Contestability has to be evaluated across the full ecosystem of decisions a person faces.

Weak and stronger forms of contestability

Weak contestability	Stronger contestability
“You may appeal.”	The person knows a decision was made, what role the system played, and how to challenge it.
A generic complaint form.	A review process with authority to change the outcome.
A vague explanation.	A reason connected to the actual evidence and decision logic.
A human rubber stamp.	A human reviewer with time, knowledge, independence, and authority.
Internal review only.	External audit or oversight where stakes are high.
Correction of one data error only.	Repair of the proxy, workflow, threshold, or policy when the pattern is systemic.

Contestability also asks about the human loop. A system with a nominal human reviewer who has three seconds per case and no authority to reverse the model is not a contested system. It is an automated system dressed up as a supervised one. A real human loop requires time, training, standing to override, and consequences when the override is ignored. A loop without those elements does not function as oversight; it functions as a stamp.

Return once more to Diego. Suppose the dashboard is well tuned. Suppose the college has thought about individual respect, structural opportunity, and error distribution. Suppose the model has been calibrated and the proxy has been revised. Diego still might have no idea he was scored. He might have no idea what pushed him to the top of the outreach queue. If an adviser lets slip that the system had him flagged as high risk, he might have no way to see the flag, correct the record, or ask what would make it change. On the four-wrongs map, that is the procedural wrong. Even if the outreach helps him, the invisibility of the scoring erodes his standing as someone who can answer for himself in the institution making decisions about him.

NIST’s report points institutions toward this kind of accountability. It treats bias as a sociotechnical property of the whole model-and-institution system, and it identifies governance, monitoring, appeal, oversight, and correction processes as part of responsible bias management. The philosophical claim underneath is the one Citron, Pasquale, and Alfrink defend: people subject to consequential algorithmic decisions need real ways to answer them.

Technical Repair, Moral Repair, And Refusal

A student who sees a biased system often asks, “How do we fix the model?” That is a good question. It is not always the whole question. Technical repair is not always moral repair.

This table helps avoid a common mistake. A model can be technically repaired while the institution remains morally unrepaired. Imagine a hiring platform that lowers subgroup error differences but still treats continuous employment as a sign of merit in a labor market where caregiving, disability, immigration, and economic instability have shaped people’s work histories. The model may improve, but the gate may remain unjust.

The reverse can also happen. An institution may make a moral repair that does not look like a model update. A college might decide that the dashboard can suggest outreach but cannot label students in advising notes. It might require advisers to ask students about work, childcare, device access, disability, and transportation before treating a risk score as evidence. It might give students a way to see and correct records. It might hire more advisers instead of using the model to ration attention more efficiently. Those are repairs to the institution, not only to the code.

Building like a philosopher means matching the repair to the wrong. If the wrong is a bad proxy, revise the proxy. If the wrong is unequal error burden, inspect errors and change thresholds. If the wrong is disrespect, change how people are classified and heard. If the wrong is structural opportunity, create alternative routes and reduce bottlenecks. If the wrong is procedural, make the system contestable. If the wrong cannot be repaired without making people subject to an illegitimate gate, refuse the system.

A Bias Judgment Audit

You can carry the chapter’s work into your Applied Ethics Case Briefing as a short judgment routine. When you find an algorithmic system in a documented case, walk through these questions in order and write short answers. The strongest answers can help you decide whether the case is ready for your Module 5 Case Dossier and what evidence belongs in your Module 6 Evidence and Comparison Record.

1. The gate. What scarce good does this system allocate? Attention, jobs, credit, care, housing, benefits, information, credibility, safety, mobility, education, visibility, or discipline?
2. The mirror. What patterns in past human or institutional behavior is the model trained on? What is missing from that mirror? Who is not standing in front of it?
3. The proxy. What is the model’s actual target, and what human concern is it standing in for? How far is the gap between the two? Where in that gap does unequal history live?
4. The location of bias. Is the problem historical, representational, measurement-based, proxy-based, categorical, objective-based, evaluative, deployment-based, workflow-based, feedback-based, or mixed?
5. The individual wrong. Is a person being treated through a classification that fails to treat them as an equal? Whose respect, agency, belief, worldview, political standing, religious standing, or personhood is being diminished?
6. The structural wrong. Across many uses, and combined with other systems, does this model narrow access to important goods for some group over time? Are feedback loops or Matthew effects at work?
7. The error burden. Who bears false positives and false negatives? Are errors concentrated on people already worst served by the institution?
8. The fairness choice. Which fairness idea matters most here: parity, calibration, equal opportunity, equalized odds, individual fairness, or substantive access? What tradeoff does that choice create?
9. The deployment case. Is this system better than the human alternative on the metrics that matter? Better on average, or better across affected groups? Does that improvement settle the proxy, opportunity, and contestability questions?
10. The contestability check. Can affected people know a decision was automated, see the reasons, challenge the record, and reach a human with authority to reverse the system when it is wrong? Is that route real, or paper?
11. The repair judgment. Given your answers, what would the institution have to change? Is the right response data repair, label repair, model repair, evaluation repair, interface repair, process repair, contestability repair, structural repair, or refusal?

You do not need to have every answer. You do need to be able to say which wrong is central and why. That is the judgment this chapter has been building toward. Naming the wrong precisely is how you know what you are asking for when you ask an institution to change.

Institutions have several repair paths. They can improve models, revise proxies, rebalance error burdens, tune fairness metrics, build in appeal, create alternative routes, or refuse to deploy a system when its use cannot be justified. Different wrongs call for different repairs, and the wrongs are not always the same in different fields. A hiring gate wrongs people in one way. A hospital triage system wrongs them in another. A dashboard in an advising office wrongs them in a third. Bringing philosophy to these systems means naming the wrong precisely enough that repair has somewhere to start.

Diego and Kai are still on that Monday-morning list, sorted and unsorted by a model built with good intentions. Whether the college owes them something different depends on which wrong you find in the case and whether you can defend the diagnosis when someone at the college pushes back. Maybe the dashboard should be redesigned. Maybe it should be used only as a weak signal. Maybe students should be told and given a way to correct their records. Maybe more advising capacity is the real repair. Maybe the system should not be used for some decisions at all.

The point is not to reject every algorithmic system. The point is to stop treating prediction as morally neutral once it becomes an opportunity gate. A philosopher asks what the system reflects, what it measures, whom it misdescribes, who bears error, who can answer back, and what kind of repair would actually address the wrong.

References

• Alfrink, K., Keller, I., Kortuem, G., & Doorn, N. (2022). Contestable AI by Design: Towards a Framework. Minds and Machines.
• Baker, R. S., & Hawn, A. (2021). Algorithmic bias in education. International Journal of Artificial Intelligence in Education.
• Bang, Y., Chen, D., Lee, N., & Fung, P. (2024). Measuring Political Bias in Large Language Models: What Is Said and How It Is Said. Proceedings of ACL 2024.
• Barocas, S., Hardt, M., & Narayanan, A. Fairness and Machine Learning.
• Buolamwini, J., & Gebru, T. (2018). Gender Shades: Intersectional Accuracy Disparities in Commercial Gender Classification. Proceedings of Machine Learning Research.
• Buyl, M., Rogiers, A., Noels, S., Bied, G., Dominguez-Catena, I., Heiter, E., Johary, I., Mara, A.-C., Romero, R., Lijffijt, J., & De Bie, T. (2026). Large language models reflect the ideology of their creators. npj Artificial Intelligence.
• Citron, D. K., & Pasquale, F. (2014). The Scored Society: Due Process for Automated Predictions. Washington Law Review, 89.
• Creel, K., & Hellman, D. (2022). The Algorithmic Leviathan: Arbitrariness, Fairness, and Opportunity in Algorithmic Decision-Making Systems. Canadian Journal of Philosophy.
• Eidelson, B. Discrimination and Disrespect.
• Fishkin, J. R. (2014). Bottlenecks: A New Theory of Equal Opportunity. Oxford University Press.
• Friedman, B., & Nissenbaum, H. (1996). Bias in Computer Systems. ACM Transactions on Information Systems.
• Gomez, J. F., Machado, C. V., Paes, L. M., & Calmon, F. P. (2024). Algorithmic Arbitrariness in Content Moderation. FAccT 2024.
• Hellman, D. When Is Discrimination Wrong?.
• Herzog, L. Algorithmic Bias and Access to Opportunities. In The Oxford Handbook of Digital Ethics.
• Jain, S., Suriyakumar, V., Creel, K., & Wilson, A. (2024). Algorithmic Pluralism: A Structural Approach To Equal Opportunity. FAccT 2024.
• Moreau, S. Faces of Inequality.
• National Institute of Standards and Technology. (2022). Towards a Standard for Identifying and Managing Bias in Artificial Intelligence (NIST SP 1270).
• Obermeyer, Z., Powers, B., Vogeli, C., & Mullainathan, S. (2019). Dissecting racial bias in an algorithm used to manage the health of populations. Science.
• Selbst, A. D., Boyd, D., Friedler, S. A., Venkatasubramanian, S., & Vertesi, J. (2019). Fairness and Abstraction in Sociotechnical Systems. FAccT 2019.
• Stanford Encyclopedia of Philosophy. Algorithmic Fairness.
• Stanford Encyclopedia of Philosophy. Discrimination.
• Stanford Encyclopedia of Philosophy. Distributive Justice.
• Stanford Encyclopedia of Philosophy. Equality of Opportunity.
• Suresh, H., & Guttag, J. (2019/2021). A Framework for Understanding Sources of Harm throughout the Machine Learning Life Cycle.
• Vallor, S. The AI Mirror. Oxford University Press.
• Young, I. M. A Social Connection Model. In Responsibility for Justice.