Control Plane vs Data Plane Failures
Some outages are worse because the wrong part of the system was allowed to be important at the wrong time. A config service hiccups and request serving stops. A scheduler is down and already accepted work cannot continue. A policy fetch times out and workers lose the ability to finish tasks that were approved earlier.
The failure is not only that a dependency broke. The failure is that the data plane needed a live control plane for work that should already have had enough authority to proceed.
The thesis
Control plane failures should not casually take down the data plane.
The control plane decides what should run, where, under which policy, and with which configuration. The data plane performs the user-facing or effectful work that has already been admitted. If those responsibilities are tangled, management problems become execution problems and execution incidents become management incidents.
The separation does not mean the data plane ignores policy. It means the system is explicit about which decisions must be fresh and which can proceed from durable, bounded, last-known-good authority.
The production pattern
A service handles user traffic but checks a central configuration path on every request. The config system slows down. User traffic slows down with it. Another system accepts jobs but requires the scheduler to be reachable at completion time. The scheduler is unavailable, so workers pile up after doing expensive work. An agent runtime approves a tool action, but the write path reconsults a policy service that is temporarily unreachable and leaves the operation in an ambiguous state.
In each case, the control plane is performing a reasonable function. Configuration, scheduling, ownership, policy, and approvals matter. The design problem is that the data plane has no degraded mode for already admitted work.
The model
I separate questions by plane:
- Control plane: who may act, what policy applies, what version is current, where work should be placed, and what limits exist?
- Data plane: given accepted authority, how does the system execute, record outcomes, reject stale owners, and recover partial work?
Then I ask which control decisions must be synchronous. Some must be fresh: disabling a compromised credential, enforcing a legal hold, or blocking a dangerous action may need fail-closed behavior. Others can tolerate bounded staleness: display configuration, routing weights, worker placement, or noncritical feature choices.
The important design artifact is the authority token or accepted operation record. It should capture the policy version, owner, deadline, and scope that were approved. The data plane can then continue within that bounded authority even if the control plane is temporarily impaired.
Where this goes wrong
The first mistake is fetching configuration or policy on every hot-path operation without a local decision about failure behavior. When the fetch fails, the service invents policy under pressure: fail open, fail closed, retry, or hang.
The second mistake is letting deploy systems, schedulers, or admin APIs share fate with serving or execution paths. A management outage should be inconvenient. It should not automatically prevent already accepted work from reaching a safe terminal state.
The third mistake is using the control plane as the only source of current ownership. If workers need to ask a central scheduler whether they still own every write, scheduler latency becomes write latency. Fencing tokens and conditional writes usually belong closer to the data plane.
The fourth mistake is ignoring recovery direction. After a control plane outage, the system needs to know what work was accepted, what continued under cached authority, what stopped, and what requires reconciliation. Without that, recovery becomes guesswork.
There is a counterpoint. Some actions should stop when fresh control is unavailable. Security-sensitive writes, destructive operations, and policy changes with external obligations may need fail-closed behavior. The point is to decide that deliberately and represent it in the operation state, not discover it through timeouts.
What I do now
I ask what happens to already accepted work if the control plane is unreachable. Can it finish? Can it pause safely? Does it have a deadline? Does it hold a fenced ownership token? Can it explain its state later?
I prefer last-known-good configuration with explicit expiry for noncritical decisions. The expiry is important because stale configuration should be a bounded choice, not an eternal shadow system.
I keep admission and execution records durable. If the control plane approved an operation, the data plane should have a record of that approval with enough scope to act or stop safely. A transcript, cache entry, or in-memory planner state is not enough.
I design recovery views by plane. Control plane health should show whether scheduling, policy updates, ownership transfer, and configuration publication are working. Data plane health should show whether accepted work is executing, stuck, stale, or reconciling. Mixing those signals makes incidents harder to read.
For agentic systems, the split is unavoidable. The model may propose actions, a controller may approve them, and tools may execute them. If those layers share no durable boundary, a planner retry or policy timeout can produce duplicate writes, lost approvals, or stalled recovery.
Closing takeaway
Keep control decisions explicit and bounded so the data plane can either finish accepted work or stop with a truthful state when management systems fail.