Contents 4
Sufficient consensus
An inverted index maps each term to the documents containing it, which makes retrieval fast enough to be interactive. Crawling, parsing and stemming fill it, and a page has to be reachable, parseable and worth storing to enter it.
This is the floor under every search result.
If what was once published stayed available, these shares would be near zero. After ten years, more than a third of pages are gone.
| Web pages | No longer accessible in October 2023 |
|---|---|
| Pages that existed in 2013 | 38 % |
| Pages that existed in 2023 | 8 % |
| All pages sampled from 2013 to 2023 | 25 % |
Pew Research Center, When Online Content Disappears, 2024. Random sample of just under 1 million pages from Common Crawl snapshots, about 90,000 per year.
What the result pages leave out
These belong to the subject. People ask about them. They are missing from the agreed coverage.
- The unindexed majority. Practice knowledge, internal documents, conversations and unwritten experience are not in any index, and searchers treat the index as the world.
- Rendering. Content that only appears after client-side execution may never reach the index at all, which turns a technical decision into an editorial one.
- Selection before ranking. Deciding what to store is a stronger filter than deciding what to rank, and it is invisible to everyone outside.
- Age and removal. Documents leave the index. Links that worked last year now return nothing, and the record of what was written disappears with them.
- The cost of storage as a filter. Making it cheap for a system to store and present your content is a real advantage.
What people actually want to know
- Is my content actually in the index, or only on my server?
- What does a crawler see when it renders my page?
- What do I know that is written down nowhere?
- Which of my old pages still exist, and who still links to them?
Result pages can only display existing content. Everything else is a decision somebody made about what was worth keeping.
Last revised 2026-09-16.