FDEBench Join us live

Q
17 / 159

Serving & inference

Walk me through the request path for a conversational AI product serving hundreds of millions of users, and tell me which parts genuinely need exotic infrastructure.

question bank FDEBench bank

01
Ask

What a strong candidate asks first

Before designing anything. Skipping this is the most common way the round is lost.

  • Hundreds of millions registered, or concurrent? Those are different systems by two orders of magnitude.
  • Is the constraint GPU capacity or cost per request?
  • What's the latency target — time to first token, or completion?
  • Is conversation history queried, or only appended and read back?

02
Rubric

What earns the points

6 criteria, 100 points total. Scored independently, partial credit allowed.

Traces the full request path 20 pts

Traces the full path: client, edge/anycast, regional routing, gateway with auth and quota, inference scheduler, GPU pool, streamed response. Full credit needs the inference tier as a distinct scheduled resource, not "call the model".

If you miss it Your path skipped the inference tier's own structure. GPUs are a scheduled, queued, capacity-constrained resource — that is the interesting box in this diagram, not the load balancer.
Finds the hard part is GPUs 25 pts

Puts the difficulty in GPU capacity — batching, KV-cache reuse, queueing, admission control — rather than in the CRUD layer. Full credit requires at least two of those named specifically.

If you miss it You spent the answer on the web tier. At this scale the web tier is solved and the hard problem is inference capacity: how requests are batched, how KV cache is reused, what happens to the queue when GPUs are full.
Plain Postgres, split by user 20 pts

Argues the relational store can be ordinary Postgres, partitioned on user id, because access is per-user keyed with no global joins. Full credit requires the *reason* the workload shards cleanly, not just the assertion that Postgres scales.

If you miss it You reached for an exotic datastore without checking the access pattern. Everything here is keyed by user — it partitions trivially and never needs a cross-user join, which is exactly when a boring relational database is the right answer.
Caches the reads that dominate 15 pts

Puts a cache in front of metadata reads and says what dominates read volume — conversation lists, settings, entitlements — so the hit rate claim is grounded.

If you miss it You added a cache without saying what it holds. Name the reads that dominate — conversation lists, settings, entitlements — otherwise the hit rate is a guess.
Streams for a fast first token 10 pts

Streams tokens and frames the SLO as time-to-first-token rather than total completion.

If you miss it You treated response time as one number. With generation, first token and last token are different SLOs and streaming is what makes the product feel fast.
Stores messages outside the DB 10 pts

Keeps message bodies in object storage or an append-only store with pointers in the database, so the hot relational tables stay small.

If you miss it You put full message history in the primary database. Message bodies are append-only and never queried relationally — keep them in object storage and keep the hot tables small.

03
Flags

Red flags

Reported separately, not subtracted — this is the "what would have ended the interview" note.

  • Proposes a globally distributed NewSQL database with no access-pattern justification.
  • Never mentions GPUs or inference capacity.
  • Treats the answer as a generic web-scale template.

—