Sufficient consensus
Relevance feedback refines a query from judgements about results, explicitly from the user or implicitly from clicks and dwell. It reliably improves the next round of results.
Implicit feedback is what large systems actually use.
What the result pages leave out
These belong to the subject. People ask about them. They are missing from the agreed coverage.
- The abandoned search. Someone who gives up and closes the tab has delivered the most important judgement and the least recorded one.
- Satisfaction that is invisible. A person who reads an answer, closes the page and acts on it looks identical to a person who bounced.
- Feedback from the wrong person. Aggregated signals mix the expert and the beginner into one average preference that fits neither.
- The result that was never offered. Feedback can only rank what was shown. A better document that stayed on page four is not part of the conversation.
- Slow satisfaction. Some answers prove themselves weeks later. No feedback mechanism reaches that far.
What people actually want to know
- What were people looking for on my site and not finding?
- Which of my pages ended a search well and looks like a failure in the data?
- How would I ever learn about the answer I should have written?
- Who is the average user my numbers describe, and does that person exist?
Ask the people who left. That is the only channel through which the missing answer can reach you.
More in information retrieval
Learning to rankThe system is trained on the behaviour it caused.Snippet generationIt is written by a machine, from your text, for a query you did not know about.Query expansionYou are answering a question that was quietly edited before it reached you.Query performance predictionA page of ten results looks the same whether the answer exists or not.Passage retrievalYour page is being read in fragments by a system that never sees the whole.Mean average precisionA score of 0.87 tells you how well the ranking matched a list somebody wrote down.Vector space modelIf the web never described your case, the vector for it is somebody else's.Inverted indexThe web is the part of knowledge that somebody bothered to publish and a crawler managed to reach.