Contents 4
Sufficient consensus
Relevance feedback refines a query from judgements about results, explicitly from the user or implicitly from clicks and dwell. It reliably improves the next round of results.
Implicit feedback is what large systems actually use.
If a search without a click always meant failure, these shares would be close to zero. Up to half of abandoned queries asked for something the result page itself could answer.
| Country | Mobile search | PC search |
|---|---|---|
| United States | 54.8 % | 31.8 % |
| China | 49.8 % | 23.3 % |
| Japan | 32.3 % | 19.0 % |
Li, J., Huffman, S. B. and Tokuda, A., Good abandonment in mobile and PC internet search, SIGIR 2009 (Google). Abandoned queries sampled from Google logs, September to October 2008; upper-bound estimate of queries whose need a result page could answer.
What the result pages leave out
These belong to the subject. People ask about them. They are missing from the agreed coverage.
- The abandoned search. Someone who gives up and closes the tab has delivered the most important judgement and the least recorded one.
- Satisfaction that is invisible. A person who reads an answer, closes the page and acts on it looks identical to a person who bounced.
- Feedback from the wrong person. Aggregated signals mix the expert and the beginner into one average preference that fits neither.
- The result that was never offered. Feedback can only rank what was shown. A better document that stayed on page four is not part of the conversation.
- Slow satisfaction. Some answers prove themselves weeks later. No feedback mechanism reaches that far.
What people actually want to know
- What were people looking for on my site and not finding?
- Which of my pages ended a search well and looks like a failure in the data?
- How would I ever learn about the answer I should have written?
- Who is the average user my numbers describe, and does that person exist?
Ask the people who left. That is the only channel through which the missing answer can reach you.