AI evaluation interview questions: evals, metrics and production readiness

Every AI design question has an evaluation question inside it. Some interviewers ask it outright: how would you know it works? Others wait to see whether you raise it. In forward deployed engineer loops, this is often where strong and average answers split.

The scoring backs that up. Across FDEBench's first 10 rubrics, 63 criteria worth 1,000 points, evaluation carries 16.5% of the points and failure modes 31%, as the seven design moves on the prep page show. Those moves give the eval step 8 of 45 minutes. The round-by-round guide shows where that sits in the loop.

What do AI evaluation interview questions look like?

They come in two forms: a question about evaluation itself, or a design question with a measurement clause at the end. The Evaluation topic in the bank holds the first kind.

• "Your eval scores are high but users still complain. What is wrong with your evaluation?"
• "Which offline and online metrics would you use for a search ranking system, and what edge cases break the online ones?"
• A math-solving model, reported at Cohere, that you walk through from data collection to evaluation.

In the bank, the second kind is more common. Agent build tasks end "with accuracy evaluation" or "including quality evaluation metrics". One unverified Reddit report from an Eightfold.ai round asks for a 15-minute RAG pitch with evals, tracing and a rollback story.

The rubrics score measurement inside design questions, too. The billion-item catalog scores recall against ground truth. The several-hundred-tool agent scores task success alongside token savings.

What should you define before choosing metrics?

Define what done means. Without it, every metric measures progress toward an undefined goal.

A completion definition names the output format, the facts that must be present, the constraints that can't be broken, and who decides it's done. "Summarize the issue and draft a reply" is a description. "A reply under 150 words that cites the order number, offers nothing beyond policy, and that a support lead would send unchanged" is a definition.

Then ask four more questions.

• What does a wrong answer cost? A wrong refund and a clumsy sentence are different failures.
• Who can judge correctness? In specialized domains, a domain expert, not an engineer.
• Is there a cheap verifier? Tests, a schema or a compiler beat any model judge.
• What does real traffic look like? The eval set has to match it. That is usually the answer to web-075.

Which evaluation methods should you name, and what does each miss?

Name several, because each catches what the others miss. A strong answer layers them.

Interviewers listen for two things here. First, a judge model has to be checked against human scores on a random sample. If the two disagree, fix the rubric before you trust the numbers.

Second, online metrics break in known ways. For web-061, name a few: clicks follow position, new items have no history, and a metric that rewards engagement can reward the wrong result.

What numbers should you estimate out loud?

Two calculations show you treat evals as statistics, not as a single score.

Compounding steps (illustrative). If each step of an agent is right 95% of the time, a six-step task succeeds about 74% of the time end to end. A ten-step task succeeds about 60% of the time. A per-step accuracy number hides that.

Sample size (illustrative). On a golden set of 100 tasks, an 80% pass rate has a standard error of about 4 points. A 95% interval runs from roughly 72% to 88%. So a two-point gain between versions can easily be noise. Grow the set, or test the change on the tasks it was meant to fix.

How do you answer "is it ready for production?"

Treat readiness as pass or fail gates, each with a named owner, not as a good demo. A demo shows the agent can do the task. Production asks whether it should act on its own in a live system.

1. Security scoping. Every API, credential and data path the agent touches is documented.
2. Permission minimization. Least privilege, not the broad access it had in development.
3. Human approval, tested. The override path works under realistic load in staging, not only on a diagram.
4. Observability, tested. Alerts have fired, and the rollback has actually run in staging.
5. LLM failure modes. An adversarial suite for prompt injection, invented action chains and tool misuse, because ordinary QA won't surface them.

A failed gate blocks promotion. A strong pass on one gate can't average out a fail on another. That framing also answers the rollback story in web-088.

Which mistakes lose points in an evaluation answer?

Most come down to measuring what is easy instead of what matters.

• An eval set built from invented examples, not real traffic, so scores stay high while users complain.
• Final-answer accuracy only, with no look at the path.
• A model judge nobody checked against humans.
• Metrics chosen before anyone defined done.
• Tuning prompts until the eval set passes, then shipping on that score.
• Treating a successful demo as sign-off.

Evaluation runs through the other rounds too: recall in RAG, task success for agents, and quality per tier when you route to cheaper models. On the FDEBench board, Updated 2 Nov 2026, browse Applied AI / ML roles or roles that mention observability. To build an agent you can then practice evaluating, start the free course Claude Code for Forward Deployed Engineering.