FDEBench Join us live

Q
4 / 159

Cost & scale

You are designing quota enforcement for a paid LLM API. A single shared account can consume 140 hours of GPU time over a week while never looking abnormal in any one-minute window. Design the limiting scheme and defend the number of limits you chose.

question bank FDEBench bank

01
Ask

What a strong candidate asks first

Before designing anything. Skipping this is the most common way the round is lost.

  • Are we trying to control cost, prevent abuse, or ration scarce GPU capacity? They imply different limits.
  • Is account sharing against the terms, or just expensive? Enforcement versus pricing.
  • What's the legitimate burst pattern — do real users spike on deadlines?
  • Per-user, per-org, or per-API-key? A shared account means the key is the wrong unit.

02
Rubric

What earns the points

7 criteria, 100 points total. Scored independently, partial credit allowed.

One window per abuse pattern 25 pts

Explains that the abuse patterns have different durations, so no single window can catch them all, and maps each proposed window to a named pattern — burst script, deadline spike, always-on agent or shared account. Full credit needs the mapping, not just "use several limits".

If you miss it You proposed limits without saying what each one catches. The whole reason for more than one window is that a burst script and a 24/7 shared account are invisible to each other's timescale — name the pattern each limit exists for.
Limits tokens, not requests 20 pts

Limits on tokens, GPU-seconds, or cost rather than request count, because request cost varies by orders of magnitude. Credit for noting output tokens dominate and that a limit must be checkable before the cost is incurred (estimate then reconcile).

If you miss it You counted requests. One request can be 200 tokens or 200,000 — a request counter does not bound cost at all. Meter the thing you are actually paying for, and say how you estimate it before the call rather than after.
Uses rolling windows, not fixed 15 pts

Uses rolling windows or token buckets and explains the fixed-window boundary problem — a caller spending the full allowance at the end of one window and again at the start of the next.

If you miss it Fixed clock windows let a caller spend twice the limit across a boundary. Use a rolling window or a refilling bucket so the constraint tracks behaviour rather than the wall clock.
Shares counters across servers 15 pts

Puts counters in shared storage so the limit holds across API nodes, and addresses the cost of that — atomic updates, and either local approximate counting at the edge or accepting the round trip. Credit for stating the fail-open/fail-closed decision when the store is down.

If you miss it You kept counters per node, which multiplies the effective limit by your node count. Say where the counter lives, how it is updated atomically, and what happens to requests when that store is unavailable.
Limits the account, not the key 10 pts

Picks the entity being limited deliberately — user, key, org, or billing account — and notices that the 140-hour case is specifically an *account sharing* problem, so a per-key limit is the wrong unit.

If you miss it You never said who the limit applies to. The scenario is one account used by many people; a per-key limit is exactly what that abuse routes around. Limit the billing entity, or detect concurrency per account.
Says which limit hit and when 8 pts

Returns a distinguishable signal per limit with retry-after and remaining quota, so a caller can tell which limit it hit and when to come back.

If you miss it A bare 429 with no indication of which limit fired or when to retry turns every rate-limit event into a support ticket. Surface the limit name, the reset time, and the remaining allowance.
Justifies every extra limit 7 pts

Names the downside honestly — each additional window is more state, more support load, and more user confusion — and sets the rule that a window must justify itself with a named pattern.

If you miss it You added limits without pricing the complexity. Every window is state to maintain and a new way for a legitimate user to be confused about why they were blocked — say why each one earns its place.

03
Flags

Red flags

Reported separately, not subtracted — this is the "what would have ended the interview" note.

  • One global requests-per-minute limit and nothing else.
  • Counts requests, never tokens or compute.
  • Stores counters in process memory across a fleet.
  • Never distinguishes legitimate bursts from abuse.

—