ConsensusOrderedCollection requeues removed-member items in inconsistent order

[anthony-murphy]

Describe the bug

ConsensusOrderedCollection can converge on different item ordering across clients after an acquiring client is removed and its job is requeued. This is distinct from the snapshot/catch-up job-tracking reconciliation defect tracked separately.

The deterministic local-server stress reproduction is seed 180 under the operation distribution in PR #27880. It fails assertEqualConsensusOrderedCollections at packages/dds/ordered-collection/src/test/fuzzUtils.ts:159 with Data contents should be equal.

Baseline divergence for datastore-0/channel-4:

• Client 0: ["uxpfF", "9CCyQ", "9CCyQ", "WWvBB", "9CCyQ", "WWvBB", "WWvBB"]
• Client 5: missing the leading "uxpfF"

Applying job-owner/quorum reconciliation fixes the missing value but not convergence: client 5 then contains "uxpfF" after the first two "9CCyQ" entries while client 0 keeps it first. This confirms a separate remove-member/requeue ordering defect.

To Reproduce

Steps to reproduce the behavior:

1. On PR feat: support single-request create with detached blobs #27880's local-server stress operation distribution, run only seed 180.
2. Client 2 adds "uxpfF" to datastore-0/channel-4 at operation 128.
3. Clients 2 and 4 later acquire from channel 4.
4. Remove client 4 at operation 152, synchronize, then load client 5 at operation 171.
5. Perform final synchronization.
6. Observe different ConsensusOrderedCollection contents/order across clients.

Expected behavior

When an acquiring client leaves, requeued items must have one deterministic position on every client, including clients loaded after the removal. All clients should converge on identical ordered contents.

The fix should include a minimized DDS-level regression test covering removal, requeue, snapshot/catch-up, and final ordering.

Logs

Failure:

assertEqualConsensusOrderedCollections
packages/dds/ordered-collection/src/test/fuzzUtils.ts:159
AssertionError: Data contents should be equal

Related context:

• PR #27880
• PR #27579, which documents a separate ConsensusOrderedCollection job-tracking reconciliation defect

[Shivang9983]

Hi Tony Murphy (@anthony-murphy),

I'd like to investigate this requeue ordering divergence.

My plan to reproduce and address this:

1. Run the local-server fuzz test with seed 180 from feat: support single-request create with detached blobs #27880 to isolate the sequence of op applications during member removal at op 152.
2. Trace the requeue insertion order in ConsensusOrderedCollection to ensure requeued entries resolve to a single deterministic sequence across both active clients and late-joining snapshot catch-ups.
3. Add a dedicated DDS-level regression test covering client removal, requeue ordering, and snapshot reconciliation.

Please let me know if I can take this up!