FDEBench Join us live

Q
7 / 159

Grounding & safety

What are the key steps to design an AI moderation system for removing harmful content from 500 million posts per day in under 1 second?

question bank FDEBench bank Source

01
Ask

What a strong candidate asks first

Before designing anything. Skipping this is the most common way the round is lost.

  • What's the acceptable false-positive rate? Removing good content has a cost too.
  • Is 1s the decision latency, or time until the post is actually taken down?
  • Are we blocking on publish, or moderating after the post is live?

02
Rubric

What earns the points

7 criteria, 100 points total. Scored independently, partial credit allowed.

Works out posts per second 15 pts

Converts 500M/day into a per-second rate (~6k/s average) and notes peak is higher than average. Order-of-magnitude arithmetic is fine.

If you miss it You never turned 500M/day into a per-second rate. That number — about 6,000/s average, more at peak — is what determines whether your design is affordable.
Cheap checks first, costly last 25 pts

Proposes a cheap-to-expensive cascade where most content exits early and only a small fraction reaches the expensive model. The specific tiers do not matter; the escalation principle does.

If you miss it You ran every post through the same classifier. At this volume the only affordable shape is a cascade — a very cheap filter that handles the overwhelming majority, escalating a small fraction to expensive models.
Estimates the cost of each tier 15 pts

Reasons about what fraction of traffic reaches each tier and what that costs. Explicitly ties tier hit-rates to the compute bill.

If you miss it You designed the tiers but never estimated what share of traffic reaches each one. Without that, you can't tell whether the cascade actually saves anything.
Matches known bad posts by hash 10 pts

Handles previously-seen content without re-running inference — hash matching, perceptual hashing, or a known-bad store.

If you miss it You re-classified content you'd already seen. A large share of harmful content is re-uploads — hash matching catches those for almost nothing.
Sends unsure cases to humans 15 pts

Routes on confidence rather than a single threshold: auto-remove when confident, auto-allow when confident, human queue in the ambiguous band. Must acknowledge the ambiguous band exists.

If you miss it You used one threshold for a binary decision. Classifiers are confident about most content and unsure about some — send the unsure band to human review instead of guessing.
Picks allow or block when down 10 pts

States what happens when the pipeline is down or overloaded, and picks fail-open or fail-closed with a reason.

If you miss it You didn't say what happens when moderation is degraded. Fail-open publishes harmful content; fail-closed blocks everything. Pick one out loud and justify it.
Learns from human reviewers 10 pts

Uses human review decisions or appeals as training signal to move work down into cheaper tiers over time.

If you miss it Your human reviewers produce labels and you threw them away. That's the cheapest training data in the system.

03
Flags

Red flags

Reported separately, not subtracted — this is the "what would have ended the interview" note.

  • Sends all 500M posts to a large transformer.
  • Never mentions false positives or appeals.

—