FDEBench Join us live

Q
153 / 159

Memory & state

A Kubernetes node fails. Stateless pods reschedule fine, but the StatefulSet pods stay in Terminating indefinitely. Explain why, and give me the safe recovery procedure.

question bank FDEBench bank

01
Sketch

Reference sketch

A short answer key carried over from the source. Not yet a graded rubric.

1. Explain the intentional behaviour: the control plane cannot distinguish a dead node from an unreachable one, and rescheduling a stateful pod whose original may still be running risks two writers on one volume. 2. Point at the concrete blocker: the volume attachment cannot be safely detached while the node's state is unknown, so the pod cannot start elsewhere. 3. Rule out blind force deletion and state the consequence: it removes the API object without stopping the process, producing split-brain and silent corruption in a database. 4. Establish ground truth first — confirm the node is genuinely powered off or fenced at the infrastructure layer. 5. Apply the out-of-service taint to the node, which tells the control plane it is safe to force-detach volumes and reschedule the workload. 6. Verify the volume detaches and reattaches to the new node, and that the database comes up consistent. 7. Prevent the manual step next time: node health automation that fences before tainting, plus replicated storage or a database that handles member loss itself. 8. Note the general principle — for stateful workloads, availability actions must be gated on a fencing guarantee, never on a timeout alone.

—