FDEBench Join us live

Q
2 / 159

Agents

What is an agent harness, and what specifically does it add over a well-engineered single prompt for a long-running task?

question bank FDEBench bank

01
Ask

What a strong candidate asks first

Before designing anything. Skipping this is the most common way the round is lost.

  • How long is the task — minutes or hours? That decides whether context management dominates.
  • Is there a cheap verifier available (tests, a compiler, a schema)? Harnesses are far stronger when there is.
  • Are any steps irreversible? That changes where approval gates go.

02
Rubric

What earns the points

7 criteria, 100 points total. Scored independently, partial credit allowed.

The runtime around the model 20 pts

Defines the harness as everything around the model — the loop, tool implementations, context management, verification — rather than as a prompt technique or a framework choice.

If you miss it You described a harness as a prompting style. It is the runtime around the model: the loop, the tools, what enters and leaves the context, and the checks between steps.
Says why one prompt falls short 15 pts

Explains that a single prompt cannot observe results, cannot recover from a wrong step, and must hold the entire task in one window.

If you miss it You never said what a single prompt structurally cannot do. It cannot see the outcome of its own step, so it cannot correct one — that is the gap the loop fills.
Spells out the act-observe loop 15 pts

Specifies the cycle concretely: decompose, execute one step, observe the real result, update context, decide next.

If you miss it Spell out the loop. Decompose, act, observe, update, decide again — without the observe step it is just a chain of prompts.
Loads and drops context per step 15 pts

Manages context per iteration — loads what the current sub-task needs and drops what it does not — rather than accumulating everything.

If you miss it You let context accumulate. On a long task that fills the window with history and starves the current step; the harness decides what to load and what to drop each round.
Checks work inside the loop 15 pts

Puts a check inside the loop — tests, linters, assertions, a re-read — so errors surface one step from where they were introduced. Credit for noting the quality of the verifier bounds the whole system.

If you miss it Nothing in your loop checks the work. Without in-loop verification, errors compound silently for twenty steps and you debug the wreckage instead of the mistake.
Sets step limits and approvals 12 pts

Names the controls the harness owns: step and cost ceilings, retry policy, and human approval for irreversible actions.

If you miss it You gave the loop no stopping condition. Step limits, cost limits, and approval gates on irreversible actions are the harness's job — a loop without them is an unbounded spend.
Fixes the harness, not the model 8 pts

Concludes that most agent failures are harness failures — unhelpful tool errors, lost context, no verification — not model failures, and that this is where the engineering effort goes.

If you miss it You attributed failure to the model. In practice the loop breaks: tool errors the model can't act on, context dropped at the wrong moment, no verifier. That is where the work is.

03
Flags

Red flags

Reported separately, not subtracted — this is the "what would have ended the interview" note.

  • Answers with a framework name instead of a mechanism.
  • Proposes a multi-agent swarm before establishing a single working loop.
  • No termination condition anywhere in the design.

—