FDEBench Join us live

Q
16 / 159

Serving & inference

How would you design a prompt caching system for 1 billion daily LLM API calls while optimizing for cost and latency?

question bank FDEBench bank Source

01
Ask

What a strong candidate asks first

Before designing anything. Skipping this is the most common way the round is lost.

  • What's the expected repeat rate in the traffic? A cache is worthless below a few percent.
  • Are stale answers acceptable, and for how long?
  • Is this cache shared across tenants, or per-tenant?

02
Rubric

What earns the points

6 criteria, 100 points total. Scored independently, partial credit allowed.

Asks the hit rate first 20 pts

Establishes or asks about the expected hit rate before designing, and ties the savings to it. Any explicit reasoning that a cache's value is entirely a function of hit rate earns full credit.

If you miss it You designed the cache without establishing the hit rate. Every number in this problem — cost saved, latency saved, whether the cache is worth building — follows from it.
Strips noise from the cache key 20 pts

Normalizes the key: strips timestamps, user IDs, session data, and other per-request noise before hashing. Must recognize that raw prompt text rarely repeats verbatim.

If you miss it You keyed on the raw prompt. Real prompts carry timestamps, user IDs, and session context that make every one unique — normalize those out of the key or you'll never get a hit.
Also catches reworded prompts 15 pts

Proposes a second tier for near-duplicate prompts (embedding similarity), or explicitly argues against it on cost grounds. Both earn full credit if reasoned.

If you miss it You only handled exact matches. Paraphrases of the same question are common; an embedding-similarity tier catches them, at the cost of an extra lookup.
Knows a close match can be wrong 15 pts

Acknowledges that semantic cache hits can return a wrong answer, and proposes a guard — a conservative threshold, verification, or scoping it to low-stakes traffic.

If you miss it A semantic cache can serve the answer to a *different* question. You didn't address that risk — it's the reason semantic caching is dangerous where exact caching isn't.
Expires entries on model change 15 pts

Handles invalidation on model version change, prompt template change, and via TTL. At least two of the three.

If you miss it You didn't say when entries expire. A cached answer from an older model version is a correctness bug, not just a stale one.
Separate cache per customer 15 pts

Notices that a shared cache can leak one customer's answer to another, and scopes keys per tenant or per trust boundary.

If you miss it Your cache is shared across callers with no isolation. If two customers send similar prompts, one can be served the other's cached answer — that's a data leak, not a cache miss.

03
Flags

Red flags

Reported separately, not subtracted — this is the "what would have ended the interview" note.

  • "Cache everything" with no repeat-rate reasoning.
  • No mention of what happens when the model is upgraded.

—