FDEBench Join us live

Q
12 / 159

Retrieval & RAG

A multi-tenant RAG assistant is returning documents from other customers' accounts. Diagnose the design flaw and tell me how you would fix it.

question bank FDEBench bank

01
Ask

What a strong candidate asks first

Before designing anything. Skipping this is the most common way the round is lost.

  • Is the leak across tenants, or within a tenant across permission levels? Different fixes.
  • Is it one shared index or an index per tenant today?
  • Where does the tenant identity currently come from — the session, or the request body?
  • Has any customer data already been exposed? That changes this from a design task to an incident.

02
Rubric

What earns the points

7 criteria, 100 points total. Scored independently, partial credit allowed.

Keeps access out of the prompt 25 pts

States that authorization cannot live in the model — no system prompt, instruction, or fine-tune makes unfiltered retrieval safe. Full credit requires them to say the enforcement point is code the model cannot influence, not just that "the prompt isn't enough".

If you miss it You left the model somewhere in the authorization path. A model is a probabilistic text function; it is not an access-control boundary. The filter has to be applied in code the model cannot reach or argue with.
Customer ID from the login 20 pts

Derives the tenant id from the authenticated session or token server-side. Full credit requires them to explicitly reject taking it from the request body, the model's tool arguments, or anything else the caller controls.

If you miss it You never said where the tenant id comes from. If it arrives in the request payload or as a model-supplied tool argument, you have built a leak with extra steps — it must come from the authenticated session.
Filters before searching 15 pts

Applies the tenant scope as a pre-filter on the vector query, not a post-filter on the results. Credit for naming either metadata filtering or per-tenant namespaces/collections. Post-filtering earns half at most — recall silently collapses and the leak returns whenever top-k fills with other tenants' rows.

If you miss it You filtered after retrieval instead of before it. Post-filtering both leaks (anything that reaches the context is already exposed if a downstream step slips) and quietly destroys recall, because top-k was spent on documents the user can't see.
One index per customer 15 pts

Argues for hard isolation — separate namespace, collection, or index per tenant — where tenant count allows, on the grounds that a *forgotten* filter then returns nothing rather than everything. Credit for naming the tradeoff against per-tenant overhead at high tenant counts.

If you miss it Your design fails open. With one shared index, the day someone adds a new query path and forgets the filter, it leaks. Per-tenant namespaces make the same mistake return an empty result instead.
Checks who can see each doc 10 pts

Notices that tenant-level filtering is insufficient and carries per-document ACLs into the index, intersected with the caller's entitlements at query time.

If you miss it You stopped at tenant boundaries. Inside one tenant, HR documents and the intern's documents are not the same permission — the index needs document ACLs, or you have solved the customer-visible half only.
Updates when access changes 8 pts

Raises that permissions change after indexing — revocation, deletion, and document re-scoping must invalidate or update the affected vectors, and that a stale index is an authorization bug.

If you miss it You treated the index as static. Permissions change and documents get deleted; without an invalidation path, your index is an authorization snapshot that keeps serving revoked access.
Logs lookups, tests for leaks 7 pts

Adds retrieval logging with tenant and document ids plus an alert on any cross-tenant hit, and proposes a deliberate adversarial-tenant test rather than trusting the fix.

If you miss it You gave no way to know whether the fix worked. Log tenant id and document id on every retrieval, alert on any cross-tenant hit, and test with a tenant that is actively trying to reach someone else's data.

03
Flags

Red flags

Reported separately, not subtracted — this is the "what would have ended the interview" note.

  • Proposes solving it with a system-prompt instruction not to leak.
  • Filters results after generation rather than before retrieval.
  • Takes the tenant id from a model-supplied tool argument.
  • Never asks whether data has already been exposed.

—