FDEBench Join us live

Q
152 / 159

General design

Your primary database is down and a replica is available for promotion. Give me the order of operations and justify each step.

question bank FDEBench bank

01
Sketch

Reference sketch

A short answer key carried over from the source. Not yet a graded rubric.

1. Establish the state first: is the primary dead, unreachable, or slow — because promoting against a live primary is the failure mode you are trying to avoid. 2. Note that reads keep serving from replicas throughout, so the outage is write-only and you have more time than the panic suggests. 3. Measure replication lag to quantify exactly which writes are at risk, and decide whether to wait briefly for the replica to catch up. 4. Fence the old primary before promoting anything — revoke its access, block its network path, or stop the instance — so a reboot cannot resume accepting writes. 5. Explain the split-brain scenario concretely: an unfenced primary that comes back believes it is still primary, and now two nodes accept conflicting writes with no reconciliation. 6. Promote the most caught-up replica, and re-point the remaining replicas at the new primary. 7. Switch traffic in one place — a proxy or virtual endpoint the application connects to — rather than redeploying every client with a new host. 8. Rebuild the old primary as a replica from the new one rather than reattaching it, since its divergent tail cannot be trusted. 9. State the RPO consequence of the lag you accepted, and record it for the incident review. 10. Automate with a quorum-based manager so a single observer's view of 'down' cannot trigger a failover.

—