FDEBench Join us live

Q
11 / 159

Product & scoping

A stakeholder believes the model 'connects to the internet' to answer questions with fresh information. Correct the picture and describe the actual data flow.

question bank FDEBench bank

01
Ask

What a strong candidate asks first

Before designing anything. Skipping this is the most common way the round is lost.

  • What decision is riding on this? If it's a security review, the answer emphasises different parts.
  • Are they asking about a hosted product's behaviour or a system we're building?

02
Rubric

What earns the points

7 criteria, 100 points total. Scored independently, partial credit allowed.

The model asks, the app fetches 25 pts

States plainly that the model has no network access and emits a structured request; the application parses and executes it. Full credit requires the boundary to be explicit — the model proposes, the harness acts.

If you miss it You left the model doing the fetching. The model produces text describing a call it would like made. Everything that touches the network is your application. That boundary is the whole answer.
Walks through the whole loop 20 pts

Traces the complete cycle: request emitted, application calls the API with its own credentials, result formatted and appended to context, model reads it on the next turn and may call again, until it answers.

If you miss it You described one hop. Walk the loop — emit, execute, append, re-read, decide again — because the iteration is what makes it useful and what makes it expensive.
The app holds keys and limits 15 pts

Places auth, secrets, rate limiting, and allow-listing in the application layer, and notes the model never holds a credential.

If you miss it You never said who holds the API key. The application does — which is also where rate limiting, domain allow-lists, and spend controls belong. The model is not a principal.
Treats web results as untrusted 15 pts

Identifies that fetched content is attacker-controlled and can carry prompt injection, and names a mitigation — delimiting, treating results as data, or constraining what the model may do after reading them.

If you miss it You treated search results as trustworthy. Anything fetched from the web is attacker-controlled text entering your prompt. Say how you contain it, especially if the agent can act after reading.
Trims results to fit context 10 pts

Notes that results consume context, so they must be ranked, trimmed, or summarised, and that truncation has consequences.

If you miss it You appended results without a budget. Search results are large; what you keep and what you cut determines both cost and whether the model sees the part that mattered.
Same pattern for every tool 10 pts

Says this is the same pattern for every tool — code execution, database queries, file access — not a web-search special case.

If you miss it You framed this as how web search works. It is how every tool call works; saying so is what makes the explanation useful beyond this one question.
Ties it to the symptoms seen 5 pts

Uses the model to explain real symptoms — stale answers when no tool fires, latency dominated by the tool round trip, fabricated citations when results were truncated.

If you miss it Tie the mechanism to what the stakeholder has actually seen. Stale answers, slow responses, and wrong citations all fall out of this loop — that is what makes the correction land.

03
Flags

Red flags

Reported separately, not subtracted — this is the "what would have ended the interview" note.

  • Says the model "has browsing enabled" as if it were a model capability.
  • Puts the API key in the prompt.
  • Never mentions prompt injection.

—