The Seven Moves

“Never is the hard word. I’ll assume you mean no claim without a citation that resolves to a retrieved span. Not zero errors, which is unachievable. So: 95% of answers carry a verifiable citation, and the system is allowed to say I don’t know. What’s the abstention budget? If refusing 10% of questions is unacceptable, that changes the design. I’ll also assume p95 under 4 seconds and a corpus that changes daily.”

“The dumbest version is: stuff the top 10 keyword hits into a prompt and ask for citations. At 10M documents that fails on recall, and ‘ask for citations’ is an instruction rather than a control, so the model can fabricate one. So I need retrieval better than keyword, and a verifier outside the model. That’s the reason for everything I add next.”

“Query → hybrid retrieval, dense plus BM25 → rerank top 100 to top 10 with a cross-encoder → generate with the spans, requiring span IDs in the output → verify each cited span actually supports its sentence → return, or abstain. The verifier is what makes never mean something. Ingestion runs offline: parse, chunk on structure, embed, index, with a delete path for retracted documents.”

“Three hundred questions labelled by their subject-matter experts, including a hundred with no answer in the corpus. Those test abstention, which is what they actually asked for. Offline: citation-support rate via an NLI check, not an LLM judging its own work. Online: abstention rate and citation click-through, both of which move within hours when retrieval degrades. Complaints arrive three weeks later, so they are not a metric. Deploys gate on the labelled set.”

“Retrieval returns nothing relevant. Detect it on low rerank scores and degrade to I don’t know plus the closest documents. A retrieved document contains injected instructions, so treat retrieved text as data, never as instructions. The provider goes down, so fall back to extractive answering and return the passage verbatim with no generation. Worse, but never unsourced, which is the invariant they bought.”

“10M documents, roughly 500 tokens a chunk, call it 40M chunks. Embedding once is low thousands of dollars, so not a blocker. Per query I’m sending about 8k tokens of context; at a million queries a month that dominates, so tiering matters: cheap model for extraction, expensive one only when the verifier disagrees. Latency: retrieval 200ms, rerank 300ms, generation 2s. The 4-second budget holds with room.”

“Week one I’d ship retrieval plus citations with no verifier, and measure how often citations are wrong. That number decides how much verifier I need. The weakest part of this design is chunking. If their documents are tables and forms rather than prose, my assumption breaks, and I’d want to see real documents before committing.”