Demoted gateway keeps attracting ingress: withdraw announcements when mastership moves away

[berendt]

Problem

When controller-restart hits the chassis that owns a network's CR port, the probes of that network stay dark for the entire fault hold and recover only ~1 s after the controller returns — even though the CR claim is released cleanly, the standby takes it over within ~1 s, and both the agent and FRR on the demoted gateway keep running the whole time. Ingress traffic keeps flowing to the demoted gateway until restore.

Measured across the last four E2E Chaos runs (30288259170, 30333584301, 30427197931, 30518214683 — 2026-07-27..30, all green):

run	tick	target	hold	worst-probe downtime (from inject)	residual (after restore)
30518214683 vlan-no-dnat	2	gateway-1	13.5 s	fip-vlan101/102 15.2 s	1.4 s
30518214683 everything-on	2	gateway-1	13.5 s	fip-vlan101/102 14.7 s, pf-vip 14.4 s (fip-vm* 1.2 s)	~1.0 s
30333584301 everything-on	2	gateway-3	16.5 s	all probes 17.3–18.3 s	0.5–1.6 s
30333584301 flat-dnat	2	gateway-3	16.5 s	fip-vm*/pf-vip 17.2–17.8 s	0.5–1.0 s

The fip-vm* 1.2 s recovery on 30518214683 shows the OVN side works: the CR port moves fast. What does not move is the ingress path.

Suspected mechanism

stopController runs ovn-ctl stop_controller, a clean shutdown that releases the claim on the CR port (test/e2e/chaos/lab.go:167). The new owner's agent announces its FRR /32 routes. But the demoted gateway's bgpd session to the upstream stays established and its previously announced routes apparently stay in place, so the upstream's best path (eBGP tie-break: oldest path) keeps pointing at the demoted gateway until its session resets at restore.

Candidate causes, to be separated by the investigation:

1. The agent on the demoted gateway does not delete its FRR /32 static routes (and VIP plumbing) when the SB shows the CR port claimed by another chassis — or does so only on some transitions / too slowly. (fix: re-plumb the VIP routes even when the master never moved #234 fixed a related re-plumb gap on the gaining side.)
2. The withdrawal happens but something downstream (prefix-list handling, redistribute static latency) delays the BGP UPDATE.

Note fast dead-path detection (BFD) would not help here: the demoted gateway's BGP session is alive and healthy. Only an actual withdrawal by the demoted agent can move the upstream's best path.

Replay

make e2e-up
make e2e-chaos CHAOS_FLAGS="-seed 30518214683 -profile vlan-no-dnat -duration 10m -out /tmp/chaos"

Tick 2 is the controller-restart on gateway-1. During the hold, check on the upstream (vtysh -c "show bgp vrf all ipv4 unicast") which gateway announces the affected /32s and which path is best; on the demoted gateway, check whether the agent removed its FRR static routes.

Acceptance criteria

• When controller-restart (or any fault that moves mastership while the demoted node's agent and FRR stay up) hits the owning chassis, worst-probe downtime ends at the CR move, not at restore: < 5 s from inject, instead of hold + ~1 s today.
• The nightly chaos report's recovery table shows controller-restart probe loss bounded by failover time across all profiles.

Found by the chaos smoothness analysis of the last 4 E2E Chaos runs (.claude/skills/chaos-analysis/).