FDEBench Join us live

Q
1 / 159

Agents

An agent has access to several hundred tools. Their definitions consume most of the context window before the user's request is even read. Design a fix.

question bank FDEBench bank

01
Ask

What a strong candidate asks first

Before designing anything. Skipping this is the most common way the round is lost.

  • How many tools does a typical task actually use? If it's three, this is a retrieval problem.
  • Are we optimising cost per turn, or is the window genuinely overflowing?
  • Is prompt caching in play today? That changes which fixes are cheap.
  • Can tools be grouped, or is the long tail genuinely flat?

02
Rubric

What earns the points

7 criteria, 100 points total. Scored independently, partial credit allowed.

Searches for tools on demand 25 pts

Reframes the problem as retrieval over tools rather than enumeration — expose a search or lookup tool, defer the rest, load only what the task needs. Credit for any equivalent lazy-loading scheme.

If you miss it You tried to make hundreds of definitions cheaper instead of not sending them. The task uses a handful of tools; make discovery a retrieval step and load definitions on demand.
Appends tools to keep the cache 25 pts

Recognises prompt caching is prefix-based and appends newly loaded definitions at the *end* of the prompt, so the cached prefix survives. Full credit requires them to say why: an insertion near the front invalidates everything after it and erases the saving.

If you miss it You inserted the loaded tools into the existing tool block near the top of the prompt. Caching keys on the prefix, so that invalidates the cache on every load and the token win you just made is paid back immediately. Append at the end.
Counts the tokens saved 15 pts

Puts numbers on it — tool tokens before and after, roughly an order of magnitude — and says which metric they would watch.

If you miss it You proposed a fix without sizing it. Say how many tokens the tool block costs now, what it costs after, and what you'd measure to confirm.
Names the cost of the extra step 12 pts

Names the cost imported: a search round trip before the first useful tool call, added latency, and the failure mode where the model searches for the wrong thing.

If you miss it Deferred loading adds a round trip before any real work happens. Say what that costs in latency and what happens when the search returns the wrong tool.
Keeps common tools always loaded 10 pts

Keeps a small always-present set of high-frequency tools so common tasks skip the search hop entirely, and explains how that set is chosen.

If you miss it You deferred everything. The tools used on most turns should stay resident — otherwise you have traded a token cost for a latency cost on every single task.
Makes tool descriptions count 8 pts

Notices that search quality now depends entirely on tool names and descriptions, making them a first-class artefact rather than documentation.

If you miss it Once discovery is retrieval, the tool description *is* the index. Vague descriptions become silent capability loss — the model simply never finds the tool.
Tracks task success too 5 pts

Measures task success rate alongside token savings, and states that a token win which lowers success is not a win.

If you miss it You optimised tokens without watching task success. Deferred tools can cut cost and quietly cut completion rate — track both or you will not notice.

03
Flags

Red flags

Reported separately, not subtracted — this is the "what would have ended the interview" note.

  • Proposes summarising or truncating tool schemas until they fit.
  • Ignores prompt caching entirely.
  • Inserts loaded definitions at the front of the prompt.
  • Suggests a bigger context window as the fix.

—