Planned restarts are not hitless: drain before config-flip, gateway-restart and agent shutdown

[berendt]

Problem

Planned lifecycle events — a config rollout (config-flip), a container restart (gateway-restart), a SIGTERM to the agent (agent-terminate) — cost 6–19 s of probe downtime in the E2E Chaos runs, because the gateway goes down without handing mastership away first. A planned event should be (near-)hitless: drain, let the standby take over, then restart.

Measured across the last four E2E Chaos runs (30288259170, 30333584301, 30427197931, 30518214683):

action	events with downtime	worst-probe downtime
config-flip	3 of 5	6.7–12.4 s (30288259170 tick 2: all probes 11.4–12.4 s)
gateway-restart	3 of 7	8.2–13.8 s (30518214683 tick 9, gateway-1: fip-vlan* 13.8 s)
agent-terminate	5 of 6	14.3–19.3 s (30333584301 tick 5, gateway-1)

Mechanism

The lab deploys with drain_on_shutdown: false (test/e2e/gwnode-config.yaml:39); among the chaos profiles only heterogeneous's gateway-2 drains (test/e2e/chaos/profiles.go:239). So a SIGTERM'd agent exits without handing the CR ports away, and config-flip/gateway-restart restart the container into the same unplanned-failover path as a crash — inheriting the no-failover-until-restore blackout.

Proposed measures

1. Exercise the drain path under chaos: run more chaos profiles (or a dedicated profile dimension) with drain_on_shutdown: true, so the drain path's smoothness is measured nightly and regressions in it are caught. Today it is effectively untested under randomized faults.
2. Verify a drained shutdown is actually hitless under load, and fix what it misses (the drain-hitless scenario asserts the mechanics; the chaos probes measure the loss).
3. Check whether config-flip needs a restart at all: if the flipped settings are reloadable, the applier (and production rollouts) could apply them without a container restart; if not, document which settings require a restart and drain first.

Items 1–2 are measurement + product posture; item 3 may spawn a follow-up if a reload path is worth building.

Replay

make e2e-chaos CHAOS_FLAGS="-seed 30288259170 -profile everything-on -duration 10m -out /tmp/chaos"

Tick 2 is the config-flip on gateway-3 with all probes down 11.4–12.4 s.

Acceptance criteria

• With drain enabled, a planned restart (config-flip, gateway-restart, agent-terminate) causes < 1 s worst-probe loss in the chaos report.
• At least one nightly chaos profile exercises drain_on_shutdown: true on more than one gateway, so the drained numbers appear in every nightly report.

Found by the chaos smoothness analysis of the last 4 E2E Chaos runs (.claude/skills/chaos-analysis/).