Skip to content

[Perf] Report every 10 seconds by default - #33

Merged
bluayer merged 1 commit into
mainfrom
feature/faster-member-reports
Oct 5, 2026
Merged

bluayer merged 1 commit into
mainfrom
feature/faster-member-reports

Conversation

@bluayer

@bluayer bluayer commented Oct 4, 2026 •

Copy link
Copy Markdown
Owner

What and why

Members reported every 30 seconds, and two things followed from that:

  • Late shortages. A shortage seen only in signals (latency, saturation) reached the hub up to 30 seconds late. A Deployment change, such as KEDA scaling up, already triggered a report immediately.
  • Slow traffic steps. The hub moves traffic a step only after both members have reported since the last step. This guard keeps traffic from bouncing on stale readings. Since Prometheus signals refresh only on the interval, traffic moved about one step every 30 seconds, whatever cooldown said.

In the AWS comparison, a queue-based controller with a 10-second loop recovered sooner where a ready replica was waiting in the other cluster.

Change

  • The default --interval (Helm agent.interval) goes from 30s to 10s, the same as the hub's.
  • The wait for fresh reports stays. Members just report more often.
  • Cost: each report runs the policy's Prometheus queries and writes its status once. That is one status patch per policy per member every 10 seconds.
  • Docs: docs/configuration.md (chart value and flag rows; the value row now explains that it paces traffic steps) and the comment in values.yaml.

Not a duplicate

This came from comparing reaction times with the experiment's queue-based controller. No issue or PR covers it.

How it was tested

  • go test ./cmd/... ./internal/controller/ passes, make lint reports 0 issues, and make chart passes.
  • No planning code changes. The e2e suite already runs members at a 1-second interval.

Checklist

  • make test lint verify-generate verify-mod passes
  • Every commit is signed off (git commit -s, DCO)
  • AI assistance, if any, is stated here and in commit trailers, and I reviewed every changed line. Written with Claude Code; the commit carries a Co-authored-by: Claude trailer.
  • New behavior works in shadow mode first, and hub decisions are recorded in the decision log (default change only)
  • Strings from other projects (Karpenter, KEDA, Gateway API, model hosts, …) are copied from source with a comment naming it (n/a)
  • Docs updated (docs/, chart values, CRD field comments)

🤖 Generated with Claude Code

Members reported every 30 seconds. Signal-based shortages (latency,
saturation) were seen up to 30 seconds late, and since the hub moves
traffic a step only once both members have reported after the last step,
traffic moved about a step every 30 seconds whatever cooldown said. In the
AWS comparison a queue-based controller with a 10-second loop recovered
sooner where a ready replica was waiting in the other cluster.

The default `--interval` (Helm `agent.interval`) is now 10 seconds, as the
hub's. The wait for fresh reports stays; members just report more often.
Each report runs the policy's Prometheus queries and writes its status once.

Co-authored-by: Claude <noreply@anthropic.com>
Signed-off-by: Jungwoo Song <bluayer@gmail.com>
@bluayer
bluayer merged commit 5fddcb5 into main Oct 5, 2026
5 of 6 checks passed
@bluayer
bluayer deleted the feature/faster-member-reports branch October 5, 2026 02:29
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant