Contents 4
Sufficient consensus
Mean average precision averages precision across recall levels and across queries. With NDCG and MRR it is the standard way to compare retrieval systems on a benchmark.
It made the field comparable and it is the reason progress can be claimed at all.
If relevance were a fixed property of a document, trained assessors judging the same documents would agree almost fully. None of the comparisons reaches half.
| Assessors compared | Overlap of documents judged relevant |
|---|---|
| Topic author and assessor A | 42.1 % |
| Topic author and assessor B | 49.4 % |
| Assessor A and assessor B | 42.6 % |
| All three assessors | 30.1 % |
Voorhees, E. M., Variations in relevance judgments and the measurement of retrieval effectiveness, Information Processing and Management 36(5), 2000 (NIST). 49 TREC-4 topics, each judged by three NIST assessors; overlap is the intersection of the relevant sets divided by their union, mean over topics.
What the result pages leave out
These belong to the subject. People ask about them. They are missing from the agreed coverage.
- The relevance judgement. Every score rests on a human decision about what counts as relevant for a query. That decision carries the values of whoever made it.
- Unjudged documents. Documents nobody assessed count as irrelevant. A genuinely new answer is scored as noise.
- Averages hide the failures. A high mean can contain queries where the system returns nothing useful at all, and those are the queries people remember.
- What the user wanted to achieve. No standard metric contains the outcome the person needed. It measures the list, not the life.
- Benchmarks age. Systems are tuned until the test set is exhausted, and the score keeps rising after the improvement has stopped.
What people actually want to know
- Relevant to whom, decided by whom?
- Which queries does this system fail completely, and are they mine?
- Does a better score mean I will find better answers?
- What is not in the test set?
A metric is a shortened definition of good. Reading the definition is more informative than reading the number.