Contents 4
Sufficient consensus
Algorithmic bias describes systematic distortions in automated decisions, arising from training data, feature choice, label definitions and feedback loops. It is documented across retrieval, recommendation and generation.
It is a property of the pipeline rather than an error in it.
If the classifiers worked equally well on every face, the four rows would show the same error rate.
| Faces in the benchmark | Error rate, Microsoft | Error rate, IBM |
|---|---|---|
| Darker-skinned women | 20.8 % | 34.7 % |
| Darker-skinned men | 6.0 % | 12.0 % |
| Lighter-skinned women | 1.7 % | 7.1 % |
| Lighter-skinned men | 0.0 % | 0.3 % |
Buolamwini, J. and Gebru, T., Gender Shades: Intersectional accuracy disparities in commercial gender classification, Proceedings of Machine Learning Research 81 (Conference on Fairness, Accountability and Transparency), 2018. Commercial gender classifiers tested on 1,270 faces of parliamentarians.
What the result pages leave out
These belong to the subject. People ask about them. They are missing from the agreed coverage.
- Who was never in the data. Groups and subjects absent from the corpus cannot be disadvantaged in any measurable way, because they generate no measurements.
- Feedback as amplification. Small initial advantages are reinforced by the interaction they produce, which is then used as evidence of quality.
- Bias towards the well funded. Producing enough material to look authoritative is a budget question, and the ranking reads budget as authority.
- Language. Coverage, quality and evaluation are all strongest in English, and the resulting gap is rarely stated to the people affected by it.
- Nobody to appeal to. A publisher who is systematically disadvantaged has no venue and no record of the decision.
What people actually want to know
- Who is missing from these results entirely?
- Why do the same organizations appear for every query in this field?
- Would this answer be different in another language?
- If the ranking is wrong about me, who do I talk to?
A ranking is an opinion produced at scale. Treating it as a measurement is the part that causes damage.