FDEBench Join us live

Q
13 / 159

Retrieval & RAG

How do you handle stale context in a RAG pipeline where the knowledge base updates every 5 minutes?

question bank FDEBench bank Source

01
Ask

What a strong candidate asks first

Before designing anything. Skipping this is the most common way the round is lost.

  • How stale is too stale? Five minutes, an hour, a day?
  • Do all documents change at the same rate, or is it a small hot subset?
  • Is a wrong-but-old answer worse than no answer?

02
Rubric

What earns the points

6 criteria, 100 points total. Scored independently, partial credit allowed.

Asks how fresh is fresh enough 20 pts

Asks for or states an acceptable staleness window, and notes it may differ by query type. Full credit requires treating freshness as a requirement to be elicited, not assumed.

If you miss it You never asked how fresh is fresh enough. "Updates every five minutes" is the write rate, not the requirement — the requirement is how stale an answer is allowed to be.
Re-embeds only changed docs 20 pts

Re-embeds only changed documents, driven by a change feed / CDC / update queue. Rejects full periodic re-indexing on cost grounds.

If you miss it You re-indexed the whole corpus. At a five-minute cadence that's enormously wasteful — drive re-embedding off a change feed so only modified documents are touched.
Sets a target for indexing lag 15 pts

Reasons about the lag between a source document changing and the new embedding being queryable, and states a target for it.

If you miss it You didn't account for indexing lag. The gap between a document changing and its embedding being searchable is where stale answers actually come from.
Ranks newer content higher 15 pts

Uses recency as a ranking signal or filter at retrieval time, not just similarity.

If you miss it You ranked purely on semantic similarity. An old chunk and a new chunk about the same topic score nearly identically — recency has to enter the ranking.
Clears out deleted documents 15 pts

Handles documents that are deleted or superseded, not just added and updated. Vector stores commonly keep serving deleted content.

If you miss it You handled additions and updates but not deletions. A retracted document that's still in the index is the worst failure mode here — the model cites something that no longer exists.
Alerts when content goes stale 15 pts

Measures freshness in production — age of retrieved chunks, indexing lag, or an alert when lag exceeds the tolerance.

If you miss it Nothing in your design tells you when freshness has broken. Track the age of what you retrieve and alert on the indexing lag.

03
Flags

Red flags

Reported separately, not subtracted — this is the "what would have ended the interview" note.

  • Proposes re-embedding the full corpus every five minutes.
  • Treats "stale" and "irrelevant" as the same problem.

—