Contents 4
Sufficient consensus
Learning to rank trains a model to order documents by relevance, using features from the query, the document and past interactions. Pointwise, pairwise and listwise approaches are the standard families.
It replaced hand-tuned formulas in every large search system.
If clicks followed relevance alone, the second row would mirror the first. When the better abstract sat in second place, most single clicks still went to the first result.
| More relevant of the top two abstracts | Google order | Top two results swapped |
|---|---|---|
| Shown first | 19 of 20 clicks | 15 of 16 clicks |
| Shown second | 2 of 7 clicks | 7 of 17 clicks |
Joachims, T., Granka, L., Pan, B., Hembrooke, H. and Gay, G., Accurately interpreting clickthrough data as implicit feedback, SIGIR 2005. Eye-tracking study with Google results, relevance of abstracts judged manually; counts are result pages where exactly one of the top two results was clicked, share going to the more relevant one.
What the result pages leave out
These belong to the subject. People ask about them. They are missing from the agreed coverage.
- The feedback loop. Documents that were never shown generate no clicks, so they stay unlearnable. Position creates the data that confirms the position.
- What counts as a good document. Relevance labels come from raters working to a definition. The definition is a decision, and it is rarely published next to the result.
- New documents. A page with no interaction history starts with the weakest possible evidence, regardless of quality.
- Why a page lost. The model produces an order. It does not produce a reason that a publisher could act on.
- Optimization against the model. Once features are known, they are targeted. The model then measures the targeting rather than the quality.
What people actually want to know
- If my page has never been shown, how is it supposed to earn the data that would get it shown?
- Who wrote the definition of relevance that decides my visibility?
- What did the pages above me do that I can see, and what did they do that I cannot?
- How long does a new page need before the system has an opinion about it?
A learned ranking is a record of what was already visible. Publishing something that was never visible is the one move it cannot anticipate.