FDEBench Join us live

Q
14 / 159

Retrieval & RAG

What are the key steps to design an embedding pipeline that maintains accuracy in vector search when scaling a product catalog from 1M to 1B items?

question bank FDEBench bank Source

01
Ask

What a strong candidate asks first

Before designing anything. Skipping this is the most common way the round is lost.

  • What's the query latency budget, and what recall are we willing to trade for it?
  • What's the write rate — how many items change per day?
  • Is this one catalog, or partitionable by category/region/tenant?

02
Rubric

What earns the points

6 criteria, 100 points total. Scored independently, partial credit allowed.

Says what fast search gives up 20 pts

States that exact nearest-neighbour search is infeasible at this scale and that approximate search trades recall for latency. Must frame it as a deliberate tradeoff, not a free win.

If you miss it You moved to approximate search without naming what it costs. ANN buys latency by giving up recall — say how much recall you're spending.
Keeps measuring search recall 20 pts

Defines how recall will actually be measured against ground truth, continuously, not just at launch.

If you miss it You have no way to know if search quality degraded. Measure recall against an exact-search ground truth on a sample, continuously — it drifts silently otherwise.
Embeds only what changed 15 pts

Embeds only new and changed items via a change feed, rather than batch re-embedding the catalog.

If you miss it You re-embedded the full catalog on a schedule. At a billion items that's a multi-day job — drive it incrementally off changes.
Says how the index is split 15 pts

Partitions the index and says on what — random, semantic, or by a business dimension — and notes the query-fanout consequence of the choice.

If you miss it You didn't say how the index is partitioned. At a billion vectors it doesn't fit on one machine, and how you shard determines whether a query hits one shard or all of them.
Plans for a new embedding model 15 pts

Handles swapping the embedding model — recognizes old and new vectors are not comparable and proposes a dual-index or backfill migration.

If you miss it You didn't plan for changing the embedding model. Vectors from two models don't live in the same space — you can't mix them, and re-embedding a billion items needs a migration path.
Sizes the index in memory 15 pts

Estimates storage or memory footprint at a billion vectors, or reduces it deliberately via dimensionality reduction / quantization.

If you miss it You never sized the index. A billion vectors at typical dimensions is terabytes of memory — that number drives the whole architecture.

03
Flags

Red flags

Reported separately, not subtracted — this is the "what would have ended the interview" note.

  • Proposes brute-force similarity at a billion items.
  • Never mentions recall.

—