You sent the résumé. They didn't call. You asked for a mortgage and they denied it "on scoring." You applied for a social benefit and your file was flagged for review for fourteen months. In none of those moments was there a person on the other side. There was a model. And the model wasn't called AI.
It was called selection algorithm, risk engine, anti-fraud system. Nobody explained which input penalized you, because the company doesn't know, because the provider considers it a trade secret, and because the law that in theory protects you is almost never applied.
This is the AI that matters. The one with no face, no interface, the one that doesn't open a conversation on Twitter every time a new model is announced. The public conversation has spent three years obsessed with the chatbot, the plush toy the industry pushes to the center of the table so the public will argue about whether it understands, whether it lies, whether it's conscious, whether it's going to take our jobs. While the plush toy is being argued about, the models that decide over concrete lives operate in silence. With no commercial name. No demo. No presentation in Las Vegas.
The Catalog Nobody Has Written in Headlines
In 2016, the mathematician Cathy O'Neil published Weapons of Math Destruction. The book did work no newspaper had bothered to do. Cataloging the opaque systems that already decided, at that point, who got into which university, who got which job, who received which credit, who went to which prison and for how long.
The technical thesis, in one sentence, is this. The models that most affect people are the ones that receive the least audit, and the ones that receive the least audit are the ones that protect the most opacity by contract. The asymmetry isn't a flaw of the system. It's the system.
Virginia Eubanks, in Automating Inequality (2018), covered the same ground from the other side of the file: that of the families in social services who watch an algorithm decide whether their child enters a child-protection risk list. Her thesis was even cruder. When these systems are installed in welfare administration, they aren't distributed evenly: they fall on the poor. Frank Pasquale, in The Black Box Society (2015), had set the legal frame: the combination of machine learning and trade secret produces an opacity that classical law wasn't prepared to open. Three books, a decade, the same finger pointing at the same spot. And no catalog from the major media has incorporated that finger into its routine.
Ten years later, the catalog has fattened and nobody has renamed it. Credit scoring (automated credit scoring) decides your mortgage with dozens of variables the bank won't list for you, and some of those variables are proxies (substitute variables that correlate with another without naming it) for race, postal code and social network. The ATS filters (Applicant Tracking System, candidate-screening software used by HR departments) sort and score the résumés by their lexical match with the job ad's keywords, not by the candidate's suitability. The figure for how many applications get discarded before a human reads a line circulates everywhere with no published study behind it, and it's best distrusted: what's verifiable is the criterion, not the percentage.
The predictive policing models concentrate patrols in neighborhoods that were already over-policed, generate more arrests in those neighborhoods, feed the model's data back, and produce a statistically self-fulfilling prophecy. Ad targeting decides which audience is shown a housing offer and which isn't, which in the United States led Meta to a settlement with the Department of Justice in 2022 over discrimination in real-estate ads. Route optimization decides in what order a home-care nurse visits their patients, which in practice decides which patients she reaches in time and which she doesn't.
None of those systems is sold as AI. They're sold as scoring solutions, as decision engines, as business intelligence tools (literally "business intelligence," an industrial euphemism for corporate data-analysis software). The term artificial intelligence is reserved for the product that appears in the press. What actually decides is called something else, and it's called something else for a reason.
The Word That Avoids Scrutiny
When an HR department buys a résumé filter, the contract doesn't say AI. It says talent acquisition platform. When a public administration installs a fraud-detection system in social benefits, the tender doesn't mention AI. It mentions a risk-scoring model. When a police force deploys predictive software, the press release talks about a decision-support tool, not an algorithm that decides anything.
There's a carefully maintained linguistic asymmetry.
The conversational chatbot, which in terms of individual harm is practically irrelevant for most of the population, is called AI and occupies eighty percent of the public conversation about AI. The models that actually concentrate life decisions over hundreds of millions of people are called algorithms, systems, engines, tools. Anything but the one thing they are. Machine learning applied to decisions about humans, with measurable material consequences, in production for a decade.
The reason for the asymmetry isn't semantic. It's regulatory and reputational. Calling it AI attracts press attention, demands explainability, activates clauses of the GDPR (the EU's General Data Protection Regulation, in force since 2018), triggers obligations of the EU AI Act. Calling it a "risk model" keeps the system in the category of boring corporate software, where nobody asks anything. Language is the first filter. The invisibility isn't an emergent property. It's a choice of reverse marketing.
Four Cases Where the Harm Left a Trail
COMPAS is an algorithm developed by Northpointe that assigns each defendant in the United States a recidivism-probability score. Judges use it to decide bail, parole, sentence length. In 2016, ProPublica published an analysis of more than seven thousand cases in Broward County, Florida. The system classified as high-risk Black defendants who didn't reoffend at a rate almost double that of white defendants in identical situations. Northpointe responded that its model didn't use the race variable. It didn't need to. It used twenty proxies. The company never published the model. It never submitted to external audit. The score is still in use. Some judges still cite it as an objective element.
Apple Card and the Disparity Without a Variable
In 2019, several users of the just-launched Apple Card reported on social media that their wives received much lower credit limits, with identical wealth, income and history. The developer David Heinemeier Hansson denounced that his limit was twenty times higher than his wife's; Steve Wozniak, Apple co-founder, described a difference of about ten times relative to his. Goldman Sachs, the issuing bank, responded that the algorithm didn't use gender. The New York State Department of Financial Services opened an investigation. In March 2021 it concluded that it had found no unlawful gender discrimination, which is not equivalent to proving the absence of bias. The difference matters. The model was never published. Nobody knows which variables generated the disparity. Nobody can repeat the experiment.
Robodebt and the Dead of an Annual Average
In Australia, between 2016 and 2020, the federal government used an automated system colloquially called Robodebt to detect improper payments in social benefits. The system cross-checked declared income against tax-agency data through an annual averaging that, applied to workers with irregular income, generated fictitious debts. Four hundred and seventy thousand people received letters demanding money they didn't owe. Several suicide cases among people who received those demands were publicly linked to the program. In 2023, a Royal Commission concluded that the system was illegal from day one, that its architects knew it, and that the internal legal warnings flagging it were buried. The final report runs to three volumes of almost a thousand pages. The prime minister of the time apologized. Nobody went to prison.
SyRI and the Ruling Almost Nobody Cites
In the Netherlands, the SyRI system (Systeem Risico Indicatie) did something similar. Cross-checking data from several administrations to identify probable fraudsters in social benefits. It was applied almost exclusively in poor, majority-migrant neighborhoods. In 2020 the Hague Court declared it contrary to Article 8 of the European Convention on Human Rights. The ruling is one of the few European precedents where a court has dismantled a state automated-decision system for incompatibility with fundamental rights. The system was withdrawn. Its successor, slightly reformed, remains in use under another name.
Four cases, four countries, four sectors. The only thing they have in common is that the harm was documented because someone with time, money and legal training pushed for years to get it documented. The vast majority of the invisible AI's harms aren't documented, because the harmed person doesn't know they've been harmed, doesn't know who to point at, and even if they knew wouldn't have the technical recourse to prove it. The asymmetry is total. You don't know the decision existed. The decision knew everything about you.
The Article 22 Nobody Applies
The GDPR, approved in 2016 and applicable since 2018, includes an Article 22 that on paper should have solved a good part of the problem. The text says the citizen has the right not to be subject to a decision based solely on automated processing that produces legal effects on them. If you read the sentence calmly, it seems solid. If you read it with a corporate lawyer beside you, you discover where the holes are.
"Solely" is the word that empties the article.
Any company can argue that the decision isn't solely automated because a human reviewed the result before communicating it. In practice, that human has ninety seconds per file, sees the score on screen, clicks accept. That's human review according to the majority case law. It's a rubber stamp signing what the machine already decided, and that's enough for Article 22 not to apply. The safeguard clause was designed with the right intention and drafted with enough vagueness for the industry to nullify it by procedure.
Meaningful Information About the Logic Applied
On top of that, Article 22 guarantees a right to obtain human intervention, to express the affected party's point of view and to contest the decision. It doesn't guarantee a right to understand. Effective explainability, knowing which specific variable sank your application, doesn't appear in the text. The Article 29 Working Party (the European advisory body of data-protection authorities, predecessor of the current Board) adopted in October 2017 guidelines insisting on meaningful information about the logic applied. Those guidelines aren't binding. National regulators almost never require them. And when they do require them, the companies respond with generic explanations drafted by their compliance department, which meet the letter and inform of nothing.
The EU AI Act (the European Artificial Intelligence Regulation, approved in 2024 and applied in stages through 2027) introduces the category of high-risk systems and requires registering many of these models in a European database. It's a real advance. It's also the next clause the industry will spend a decade learning to circumvent. The big companies will hire lawyers specialized in reclassifying their systems as lower-risk. The consultancies will sell certifications the industry itself will self-manage. The national regulators will have staff to audit two percent of the registered systems. The rest will operate without effective oversight, with a compliance seal on the cover. It's not defeatism. It's what has happened with every digital regulatory framework from the cookie directive to the GDPR itself.
The Conversation the Chatbot Covers Up
While people argue over whether GPT understands, whether Claude hallucinates, whether Gemini should be able to generate images of historical figures, the systems that actually concentrate decisional power remain nameless and unaudited. This isn't an informational coordination failure. It's a success of attention redirection.
The industry that sells chatbots and the industry that sells opaque scoring systems aren't the same, but the first works as a curtain for the second. The reputational load is eaten by the chatbot, which is noisy, visible and arguable. The decisional load is carried by the risk engine, which makes no noise, has no press account and doesn't answer emails.
Who Audits What Nobody Sees
The question of who audits the invisible layer has a short answer. Nobody with real power. The layer is audited, at best, by investigative journalists with philanthropic funding, academic groups with no access to the data, NGOs that take three years to obtain a ruling. They audit it out of time, once the harm has already occurred in thousands of files. They audit it on the technical crumbs the companies are forced to drop under judicial pressure. To audit for real would imply access to the model, to the training data, to the history of real decisions, to the error metrics broken down by group. None of that is obtained without a lawsuit, and the lawsuits are almost always lost by the party that isn't the company.
The debate about the AI that's visible is entertainment. There are interesting pieces in it, there are design decisions that deserve scrutiny. But it occupies a disproportionate position in public attention compared to the other debate, which is where everything is being played out. If the next ten years of conversation about AI are spent on whether generative models are creative or are parrots, while the discriminative models keep deciding in silence who accesses what and with what sentence, it won't be an accident. It will be the natural result of a system where attention is channeled toward the arguable and the important is kept outside the frame.
You don't know which model evaluated your last application. You don't know which provider trained it. You don't know which variables penalized you. You don't know who to complain to, and if you did, you couldn't prove anything because the model is a trade secret. The system works exactly as it was designed to work. The invisibility isn't a flaw. It's the product's main feature.
Definitions
Credit scoring. A credit score generated by an automated model that cross-checks the applicant's financial, demographic and behavioral variables to assign a risk level. The exact variables and their weights are the provider's trade secret.
ATS. Applicant Tracking System, a platform for automated screening of applications used by HR departments. It filters résumés by lexical match with the job ad before a human sees any.
Predictive policing. A family of models that estimates the probability of crime by geographic area or by individual, from historical data of reports and arrests. It tends to generate feedback loops in already over-policed neighborhoods.
Proxy. In applied statistics, a substitute variable that correlates with another that one doesn't want or can't use directly. A postal code can act as a proxy for race without the model ever naming race.
GDPR. The European Union's General Data Protection Regulation (Regulation EU 2016/679), applicable since May 2018. Its Article 22 regulates decisions based on automated processing.
EU AI Act. The European Artificial Intelligence Regulation, approved in 2024 and applied in stages through 2027. It classifies AI systems by risk levels and requires the registration of high-risk ones in a European database.
Explainability. The capacity of an automated system to offer an understandable justification of an individual decision. It isn't equivalent to publishing the model. The industry usually provides generic explanations that don't allow reconstructing the reason for the specific rejection.
COMPAS. Correctional Offender Management Profiling for Alternative Sanctions. A recidivism-risk assessment algorithm developed by the company Northpointe (today Equivant), used in courts of several US states.
SyRI. Systeem Risico Indicatie. A Dutch system for detecting fraud in social benefits by cross-checking data from several administrations, declared contrary to human rights by the Hague Court in 2020.
Robodebt. The colloquial name of the Online Compliance Intervention, an automated system of the Australian government (2016-2020) that claimed improper debts from some 470,000 people using a wrongly applied annual-average calculation.
References
O'Neil, Cathy — Weapons of Math Destruction (Crown, 2016). The founding work that catalogs the opaque models in universities, HR, insurance, banking and criminal justice. The conceptual basis of the article.
Eubanks, Virginia — Automating Inequality (St. Martin's Press, 2018). Research on automated systems in social services in the United States. The reference for the treatment of the informational asymmetry.
Pasquale, Frank — The Black Box Society (Harvard University Press, 2015). A legal analysis of algorithmic opacity in finance and information. The conceptual frame on trade secret and the affected party's rights.
Angwin, J., Larson, J., Mattu, S. and Kirchner, L. — Machine Bias (ProPublica, 23 May 2016). An empirical analysis of the COMPAS algorithm in Broward County, Florida. The source of the COMPAS case.
Apple Card investigated by NY regulator after gender discrimination claims — Bloomberg, 9 November 2019. Coverage of the Apple Card case and of the complaints by David Heinemeier Hansson and Steve Wozniak.
New York DFS clears Goldman Sachs of gender bias in Apple Card algorithm — Banking Dive, March 2021, https://www.bankingdive.com/news/goldman-sachs-gender-bias-claims-apple-card-women-new-york-dfs/597273/. The source for the New York Department of Financial Services resolution.
Royal Commission into the Robodebt Scheme — Final Report — Commonwealth of Australia, July 2023. The official report on the Robodebt system. The source of the Australian case.
The Hague District Court — Ruling NJCM and others v. the Dutch State (SyRI case), 5 February 2020. The source of the Dutch case.
European Parliament and Council of the EU — Regulation (EU) 2016/679, General Data Protection Regulation, Article 22. The legal text cited when dealing with the right not to be subject to solely automated decisions.
Article 29 Working Party (WP29) — Guidelines on Automated individual decision-making and Profiling for the purposes of Regulation 2016/679, adopted in October 2017 and revised in February 2018, https://ec.europa.eu/newsroom/article29/items/612053. The guidelines cited on meaningful information about the logic applied.
European Parliament and Council of the EU — Regulation (EU) 2024/1689, Artificial Intelligence Regulation (EU AI Act). Available at https://artificialintelligenceact.eu. Cited when dealing with the risk classification and the European registry of systems.
US Department of Justice — Settlement with Meta Platforms Inc. on discriminatory housing advertising, 21 June 2022. Cited in the catalog of the invisible AI.
Para profundizar
Crawford, K. (2021). Atlas of AI. Power, Politics, and the Planetary Costs of Artificial Intelligence. Yale University Press. A general read on AI's political materiality, a natural frame for the argument about operational invisibility.
Noble, S. U. (2018). Algorithms of Oppression. How Search Engines Reinforce Racism. NYU Press. Documentation of aggregate bias in algorithmic systems, extensible to the scoring and filtering engines described in the article.
También te interesa
- Cathy O'Neil and the Mathematical Weapons
- Invisible Biases. The Declared Bias Gets Filtered, the Underlying One Stays Intact
- The Floors of Public AI. The Facade and the Floors Below
- Why Shoshana Zuboff Is Right About AI
No comments yet
No comments yet. Be the first.
Leave a comment