News
Perplexity opens Q2D-Web, a web-scale first-stage retrieval benchmark
Perplexity’s Q2D-Web pairs 190M web docs with 70k agent queries and a public leaderboard for first-stage retrievers in agentic RAG.
Searcher → Analyst → Writer → Editor · subagentic-20260909-2000
Perplexity has opened Q2D-Web (Query2Doc-Web), a private first-stage retrieval benchmark with a public leaderboard for agentic RAG—not a new search product. The eval pairs 190 million web documents with 69,721 agent-reformulated queries in ten languages and scores retrievers on three judgment sets instead of one ground truth.
Construction details are on Perplexity’s engineering blog. A matching technical report, Q2D-Web: A Large-Scale Benchmark for Retrieval in Agentic RAG Systems (arXiv:2609.08887), was posted on 8 September 2026.
First-stage recall is the ceiling
A production retriever searches a huge corpus and returns thousands of candidates. Those candidates are the only sources an agent can cite. Perplexity’s case for Q2D-Web is that evaluation has to grow along three axes at once: corpus size (hard negatives), query count (stable estimates), and labels per query (fewer false negatives). Most public IR sets grow only one or two of those, and they usually test human-written queries. Agentic systems search with reformulations that differ in wording, structure, and specificity.
The query set comes from 23,000 PII-filtered production searches collected over nine months. Each search is a bundle of tool calls: one primary query that restates the user need, plus optional support queries for alternate phrasings, background, or related entities. Q2D-Web judges every query on its own. English accounts for 65.8% of queries; the rest covers Spanish, Russian, German, French, Portuguese, Italian, Korean, Japanese, and Chinese.
The corpus is the union of the top 5,000 production hits per query, then MinHash–LSH deduplicated. It is not a random crawl. Every document was already a plausible result, which packs the collection with distractors that match topic or language but miss a date, entity, version, or aspect. The combined judgment set averages 99.6 positives per query. Perplexity’s comparison point is MS MARCO Web Search: 100.9 million documents, 9,374 test queries, and one click-derived positive per query.
Three label stacks
At this scale, any single labeling pipeline is biased and incomplete. Q2D-Web therefore ships three relevance sets:
- Citation — a document is relevant if an agent cited it. Closest to downstream use, but high precision and low recall by design.
- Web Ranking — up to 50 documents per query (43.1 on average) from Perplexity’s production ranking stack.
- Combined + LLM-judged — the union of those two, plus binary DeepSeek-V4-Flash judgments on previously unjudged candidates pooled with reciprocal rank fusion.
The headline metric is Recall@1000. First-stage retrieval only has to land relevant documents in the candidate pool; rerankers handle order. nDCG@10 is reported alongside.
On thirteen publicly released retrievers, no model wins every column. pplx-embed-v1-4b leads Combined Recall@1000 at 69.11 and Web Ranking Recall@1000 at 65.73. Nemotron-3-Embed-8B leads Citation Recall@1000 at 61.68 and Combined nDCG@10 at 47.44. The paper reports that relative model order is largely stable across judgment sets, while scores still diverge by domain, language, and query type.
Perplexity runs its own embedding models through the same pipeline as everyone else, and says eval queries and corpus were held out of training. It still flags a possible in-distribution advantage because Q2D-Web is built from Perplexity traffic. The corpus is private; the board is public. Read the scores with that caveat in mind.
Public board, private collection
Results appear on a Hugging Face leaderboard. Teams request an evaluation for a public checkpoint that loads with standard transformers or sentence-transformers APIs. Queries and documents are truncated to 512 tokens with the model’s own tokenizer.
Full-corpus scoring is expensive: 4,608 H200 GPU-hours for pplx-embed-v1-4b. Every submission is first run on an RRF-sampled subcorpus that keeps 31.7% of the documents and, in Perplexity’s ablation, inflates mean Recall@1000 by 4.5 points. Models are split into up-to-1B and above-1B bands. Only the top ten in each band get a full-corpus eval.
Read the methodology post and the arXiv paper next, then compare how those thirteen retrievers trade citation recall against combined nDCG on the public leaderboard.