LLM cost and latency interview questions: caching, routing and the math to say out loud

Cost and latency questions sound like operations trivia. They are design questions in disguise. "Your app gets 1M LLM queries a day. How do you optimize cost?" has no right answer until you know what those queries look like, and the interviewer is waiting to see whether you ask.

This guide covers the Cost & scale and Serving & inference topics in the FDEBench bank. For where they fit in the loop, see the round-by-round guide.

What do interviewers ask about LLM cost and latency?

They ask you to cut a bill, hit a latency target, or survive growth. Concepts come up too, such as "What is a KV cache, and how does it help LLM inference?", reported at Apple.

• Cut the bill: "How would you design a prompt caching system for 1 billion daily LLM API calls while optimizing for cost and latency?"
• Hit a target: "How would you design a GenAI system that rewrites a user’s resume based on a job description with a latency of under 2 seconds?" And "How do you measure LLM latency in a chat app, and which percentile matters?"
• Survive growth: "Your AI traffic grows 100x. What breaks first and what do you change?" Or the request path for hundreds of millions of users.
• Ration capacity: quota enforcement for a shared account that burns 140 hours of GPU time in a week without looking abnormal in any one minute.

What should you ask before optimizing cost or latency?

Ask what repeats, what the latency target measures, and what share of traffic is easy. Every number after that follows from these answers.

• Repeat rate. How often does the same or a similar request come back? The bank-39 page is blunt: "A cache is worthless below a few percent." Its first criterion, 20 of 100 points, is establishing the hit rate before designing.
• Which latency. Is 2 seconds to the first token or to the full answer? At p50 or p99? A generative system has two clocks. Streaming helps the first. Only a faster model or a shorter answer helps the second.
• Which users. Hundreds of millions registered, or concurrent? The bank-116 page calls those "different systems by two orders of magnitude".
• Traffic mix. What share of requests could a small model handle? That caps what routing can save.
• Tenancy. Is a cache shared across customers? Then one customer can be served another's answer.

Which caching layer saves what?

There are four layers, and each saves something different. Keeping them apart is a quick way to show depth.

A semantic cache embeds each query and returns a stored answer when a new one is close enough in meaning, as Redis explains in its guide. The same guide suggests starting at a similarity threshold of 0.90 to 0.95 and watching false positives. Keep personalized and time-sensitive queries out of it.

The bank-39 rubric also rewards a normalized key, with timestamps and user ids stripped before hashing, and invalidation when the model or the prompt template changes. A cached answer from an older model is a correctness bug.

When does routing to cheaper models pay off?

When a real share of your traffic is easy. A router scores each request's difficulty and sends it to the cheapest model that can handle it. Below a confidence threshold, it escalates to a stronger model instead of gambling.

The saving depends on the traffic. Anyscale's router write-up, routing between Mixtral 8x7B and GPT-4, reported up to 70% lower cost on MT Bench at the same performance as its baselines, but 30% on MMLU and 40% on GSM8K. Same approach, different mix of questions, different saving. Hand-label a few hundred real prompts before you promise a number.

Say what routing costs, too. It adds a classification hop before inference, and every tier needs its own evals. Routers also drift as models launch, prices change and traffic shifts. Name an owner and rerun the evals when any of those happen.

A cascade is the same idea at scale. For moderating 500 million posts a day, the rubric's biggest criterion, 25 points, is a cheap-to-expensive cascade where most content exits early.

What numbers should you estimate out loud?

Do the token math in the room: tokens per call, times calls, times price. Say "illustrative" and use round prices.

A daily bill (illustrative). 1M queries a day, each with 2,000 input tokens and 300 output tokens. At $3 per million input tokens and $15 per million output, input is 2B tokens, or $6,000 a day. Output is 300M tokens, or $4,500. That is about $10,500 a day.

After prompt caching (illustrative). Suppose 1,500 of the 2,000 input tokens are a shared prefix billed at a tenth of the price. Input falls to about $1,950 a day, and output doesn't move. Output is now about 70% of the bill, so the next levers are shorter answers and routing, not more caching.

Throughput (illustrative). 500 million posts a day is about 5,800 a second on average, and peaks run higher. That number is why the rubric marks down sending every post to a large model.

What should you measure?

Measure cost per successful request, hit rates per layer, and latency as a distribution.

• Cost per successful request. A cheap model that fails and retries is not cheap.
• Hit rate per cache layer: semantic hits, billed versus gross input tokens, prefix hits on the server. One blended hit rate hides the weak layer.
• Time to first token and time to last token, at p95 or p99, not the average.
• Quality per routing tier on a held-out set, plus misroute and escalation rates.
• Queue depth and GPU utilization if you host the model. At scale, the bank-116 rubric puts the hard problem there, not in the web tier.

Which mistakes lose points in a cost and latency round?

These come from the red flags and criteria on FDEBench's rubrics for these questions.

• "Cache everything" with no repeat-rate reasoning.
• No plan for the cache when the model is upgraded.
• One cache shared across tenants.
• Chaining model calls in sequence inside a 2-second budget.
• One global requests-per-minute limit, when one request can be 200 tokens or 200,000. Meter tokens or GPU time.
• Sending all 500 million posts to a large model.
• Treating response time as one number.

The same arithmetic sits behind RAG designs and agent designs, and the evaluation guide shows how to prove a cheaper path didn't cost quality. On the FDEBench board, Updated 2 Nov 2026, you can search roles that mention LLMs. To build and run agents yourself, try the free course Claude Code for Forward Deployed Engineering.