Skip to content

ConsensusOrderedCollection duplicates jobs during snapshot catch-up reconciliation #28040

Description

Describe the bug

ConsensusOrderedCollection can retain duplicate/inconsistent job state across snapshot/catch-up when the owning client is absent from quorum. The deterministic local-server stress reproduction is seed 0 on the operation distribution in PR #27880.

It fails assertEqualConsensusOrderedCollections at packages/dds/ordered-collection/src/test/fuzzUtils.ts:159 with Data contents should be equal. For datastore-0/channel-8, client 5 contains ["WWvBB", "WWvBB"] while client 0 contains ["WWvBB"].

A temporary implementation that reconciles job owners absent from quorum both on connect and after message processing makes seed 0 pass, confirming the snapshot/catch-up job-tracking reconciliation defect described in PR #27579.

Seed 180 is a distinct remove-member/requeue ordering defect tracked in a separate issue. Historical seed 54 was also attributed to ConsensusOrderedCollection consistency, but has not been proven to share this root cause.

To Reproduce

Steps to reproduce the behavior:

  1. On PR feat: support single-request create with detached blobs #27880's local-server stress operation distribution, run only seed 0.
  2. Client 3 disconnects.
  3. Client 2 creates datastore-0/channel-8, adds "WWvBB", and acquires it.
  4. Client 5 loads after reconnect/staging operations.
  5. Synchronize at operation 177.
  6. Observe duplicate contents on client 5 and divergent collection state across clients.

Expected behavior

Snapshot/catch-up and quorum changes should reconcile job ownership exactly once. Every client should converge on identical ConsensusOrderedCollection contents without duplicate requeued items.

The fix should include a minimized DDS-level regression test covering the absent-owner quorum transition and a client loading from snapshot/catch-up.

Logs

assertEqualConsensusOrderedCollections
packages/dds/ordered-collection/src/test/fuzzUtils.ts:159
AssertionError: Data contents should be equal

client-5: ["WWvBB", "WWvBB"]
client-0: ["WWvBB"]

Related context:

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't working

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions