Sufficient consensus
Mean average precision averages precision across recall levels and across queries. With NDCG and MRR it is the standard way to compare retrieval systems on a benchmark.
It made the field comparable and it is the reason progress can be claimed at all.
What the result pages leave out
These belong to the subject. People ask about them. They are missing from the agreed coverage.
- The relevance judgement. Every score rests on a human decision about what counts as relevant for a query. That decision carries the values of whoever made it.
- Unjudged documents. Documents nobody assessed count as irrelevant. A genuinely new answer is scored as noise.
- Averages hide the failures. A high mean can contain queries where the system returns nothing useful at all, and those are the queries people remember.
- What the user wanted to achieve. No standard metric contains the outcome the person needed. It measures the list, not the life.
- Benchmarks age. Systems are tuned until the test set is exhausted, and the score keeps rising after the improvement has stopped.
What people actually want to know
- Relevant to whom, decided by whom?
- Which queries does this system fail completely, and are they mine?
- Does a better score mean I will find better answers?
- What is not in the test set?
A metric is a shortened definition of good. Reading the definition is more informative than reading the number.
More in information retrieval
Learning to rankThe system is trained on the behaviour it caused.Snippet generationIt is written by a machine, from your text, for a query you did not know about.Query expansionYou are answering a question that was quietly edited before it reached you.Relevance feedbackAbsence produces no signal.Query performance predictionA page of ten results looks the same whether the answer exists or not.Passage retrievalYour page is being read in fragments by a system that never sees the whole.Vector space modelIf the web never described your case, the vector for it is somebody else's.Inverted indexThe web is the part of knowledge that somebody bothered to publish and a crawler managed to reach.