Skip to content

fix(server): atomic replay boundary - #3613

Open
letv1nnn wants to merge 5 commits into
NVIDIA:mainfrom
letv1nnn:fix/server-atomic-replay-boundary
Open

letv1nnn wants to merge 5 commits into
NVIDIA:mainfrom
letv1nnn:fix/server-atomic-replay-boundary

Conversation

@letv1nnn

Copy link
Copy Markdown
Contributor

Summary

Follow-up A from #3209: closes two remaining WatchSandbox resume gaps, a lock race between the log and platform tail reads, and a coverage gap where an asymmetric replay depth between the two sources could let a resumable cursor imply coverage a shallower source never actually delivered, silently dropping events on the next reconnect with no OUT_OF_RANGE.

Related Issue

Refs #3055 (partial; addresses the interleaving concern @varshaprasad96 raised on #3209, tracked there as follow-up A. Issue stays open until the SDK helpers (B/C/D) and e2e coverage (E) also land.)

Changes

  • Atomic replay snapshot. TracingLogBus::snapshot_after reads the log and platform buses under one lock hold instead of two independent tail_after calls, so a publish can no longer land between them and desync their high-water marks. Removes the now-redundant post-read epoch re-check this race previously required.
  • Cross-source coverage floor. tail_with_floor reports the newest event a bounded tail read excluded. On connect, the server takes the smallest nonzero floor across followed sources and withholds any event, from either source, at or above it, a log event that clears its own bus's floor can still sit past the platform bus's floor, and handing it out would let the client's single shared cursor outrun platform's unreplayed backlog. This can withhold the entire initial batch when a followed sibling has any backlog outside its requested tail depth (e.g. event_tail left at its default of 0); that's intentional, an emptier connect beats a resume that silently and permanently drops events.
  • Docs. Documents the coverage-floor contract in proto/openshell.proto (log_tail_lines, event_tail, resume_after_cursor), architecture/gateway.md (new Coverage floor subsection, rewritten atomic-snapshot description), and a short SDK-facing note in docs/observability/accessing-logs.mdx (openshell logs itself is unaffected, it doesn't expose asymmetric tail depths).

Testing

  • mise run pre-commit passes
  • Unit tests added/updated
  • E2E tests added/updated (if applicable)

Checklist

  • Follows Conventional Commits
  • Commits are signed off (DCO)
  • Architecture docs updated (if applicable)

Signed-off-by: Artem Lytvyn <alytvyn@redhat.com>
Signed-off-by: Artem Lytvyn <alytvyn@redhat.com>
Signed-off-by: Artem Lytvyn <alytvyn@redhat.com>
Signed-off-by: Artem Lytvyn <alytvyn@redhat.com>
Signed-off-by: Artem Lytvyn <alytvyn@redhat.com>
@copy-pr-bot

copy-pr-bot Bot commented Sep 23, 2026

Copy link
Copy Markdown

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant