Skip to content

fix(core,sdk): stop chat.agent losing messages during recovery - #4907

Open
ericallam wants to merge 1 commit into
feat/transcript-storage-tri-13667from
feature/tri-13752-chatagent-version-handover-duplicates-messages-and-turns
Open

fix(core,sdk): stop chat.agent losing messages during recovery#4907
ericallam wants to merge 1 commit into
feat/transcript-storage-tri-13667from
feature/tri-13752-chatagent-version-handover-duplicates-messages-and-turns

Conversation

@ericallam

Copy link
Copy Markdown
Member

Summary

When a chat.agent run boots to continue a session (a version handover, or a retry after a crash), it replays the unacknowledged user messages off session.in and dispatches them itself. A previous change stopped the live tail from re-answering those same messages by folding them into the resume cursor in one step. That cursor is what the next boot reads to know where to resume, and folding in every recovered message at once let it advance past a message the run had not answered yet. So if the run answered the first recovered message, wrote its turn boundary, then crashed before dispatching the rest, the next boot resumed past those messages and they were never answered.

Fix

A recovered message is now claimed on the session-stream router instead of folded into the cursor. A claim does two independent things:

  • it drops the message however late the live tail re-delivers it, so a recovered message is never answered twice;
  • it holds the resume cursor behind that message until the boot has dispatched it, so a turn boundary never publishes a cursor past a message still waiting for a turn.

The boot settles each claim as it dispatches the message, or right away for a message it folds into the seed chain or deliberately skips, so the cursor only advances over messages that have actually been handled. A claimed record whose route re-read never arrives over the tail degrades to being answered twice on the next boot, never to being dropped.

Covered by router-level unit tests for the claim/settle floor and a chat.agent boot test asserting the cursor published after the first recovered turn stays behind the still-unanswered ones.

@changeset-bot

changeset-bot Bot commented Sep 8, 2026

Copy link
Copy Markdown

🦋 Changeset detected

Latest commit: 2ece24d

The changes in this PR will be included in the next version bump.

This PR includes changesets to release 27 packages
Name Type
@trigger.dev/sdk Patch
@trigger.dev/core Patch
@trigger.dev/python Patch
@internal/dashboard-agent Patch
@trigger.dev/build Patch
trigger.dev Patch
@trigger.dev/redis-worker Patch
@trigger.dev/schema-to-json Patch
@internal/clickhouse Patch
@internal/llm-model-catalog Patch
@internal/metrics-pipeline Patch
@trigger.dev/rbac Patch
@internal/redis Patch
@internal/replication Patch
@internal/run-engine Patch
@internal/run-store Patch
@internal/schedule-engine Patch
@internal/tracing Patch
@internal/webhook-engine Patch
@internal/webhook-sources Patch
@internal/cache Patch
@trigger.dev/react-hooks Patch
@trigger.dev/rsc Patch
@trigger.dev/database Patch
@trigger.dev/otlp-importer Patch
@trigger.dev/sso Patch
@internal/testcontainers Patch

Not sure what this means? Click here to learn what changesets are.

Click here if you're a maintainer who wants to add another changeset to this PR

@coderabbitai

coderabbitai Bot commented Sep 8, 2026

Copy link
Copy Markdown
Contributor

Review Change StackReview Change Stack

Note

Reviews paused

It looks like this branch is under active development. To avoid overwhelming you with review comments due to an influx of new commits, CodeRabbit has automatically paused this review. You can configure this behavior by changing the reviews.auto_review.auto_pause_after_reviewed_commits setting.

Use the following commands to manage reviews:

  • @coderabbitai resume to resume automatic reviews.
  • @coderabbitai review to trigger a single review.

Use the checkboxes below for quick actions:

  • ▶️ Resume reviews
  • 🔍 Trigger review

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info
⚙️ Run configuration

Configuration used: Repository UI

Review profile: CHILL

Plan: Advanced

Run ID: 43520ea3-e703-4948-847b-d345c6561001

📥 Commits

Reviewing files that changed from the base of the PR and between 52addef and 1268dba.

📒 Files selected for processing (1)
  • packages/trigger-sdk/src/v3/ai.ts

Included review availability: Your plan provides up to 10 included reviews per hour; 9 remain after this review.

📜 Recent review details
⏰ Context from checks skipped due to timeout. (41)
  • GitHub Check: webapp / 🧪 Unit Tests: Webapp (15, 24)
  • GitHub Check: webapp / 🧪 Unit Tests: Webapp (13, 24)
  • GitHub Check: webapp / 🧪 Unit Tests: Webapp (18, 24)
  • GitHub Check: webapp / 🧪 Unit Tests: Webapp (12, 24)
  • GitHub Check: webapp / 🧪 Unit Tests: Webapp (24, 24)
  • GitHub Check: webapp / 🧪 Unit Tests: Webapp (17, 24)
  • GitHub Check: webapp / 🧪 Unit Tests: Webapp (20, 24)
  • GitHub Check: webapp / 🧪 Unit Tests: Webapp (7, 24)
  • GitHub Check: webapp / 🧪 Unit Tests: Webapp (21, 24)
  • GitHub Check: webapp / 🧪 Unit Tests: Webapp (10, 24)
  • GitHub Check: webapp / 🧪 Unit Tests: Webapp (16, 24)
  • GitHub Check: webapp / 🧪 Unit Tests: Webapp (14, 24)
  • GitHub Check: webapp / 🧪 Unit Tests: Webapp (19, 24)
  • GitHub Check: webapp / 🧪 Unit Tests: Webapp (8, 24)
  • GitHub Check: webapp / 🧪 Unit Tests: Webapp (11, 24)
  • GitHub Check: webapp / 🧪 Unit Tests: Webapp (23, 24)
  • GitHub Check: webapp / 🧪 Unit Tests: Webapp (1, 24)
  • GitHub Check: webapp / 🧪 Unit Tests: Webapp (9, 24)
  • GitHub Check: webapp / 🧪 Unit Tests: Webapp (4, 24)
  • GitHub Check: webapp / 🧪 Unit Tests: Webapp (6, 24)
  • GitHub Check: webapp / 🧪 Unit Tests: Webapp (5, 24)
  • GitHub Check: webapp / 🧪 Unit Tests: Webapp (3, 24)
  • GitHub Check: webapp / 🧪 Unit Tests: Webapp (2, 24)
  • GitHub Check: sdk-compat / Node.js 26.4 (warp-ubuntu-latest-x64-4x)
  • GitHub Check: sdk-compat / Node.js 24.18 (warp-ubuntu-latest-x64-4x)
  • GitHub Check: sdk-compat / Deno Runtime
  • GitHub Check: sdk-compat / Bun Runtime
  • GitHub Check: sdk-compat / Cloudflare Workers
  • GitHub Check: packages / 🧪 Unit Tests: Packages (2, 3)
  • GitHub Check: internal / 🧪 Unit Tests: Internal
  • GitHub Check: e2e / 🧪 CLI v3 tests (warp-windows-latest-x64-8x - pnpm)
  • GitHub Check: packages / 🧪 Unit Tests: Packages (1, 3)
  • GitHub Check: e2e / 🧪 CLI v3 tests (warp-windows-latest-x64-8x - npm)
  • GitHub Check: packages / 🧪 Unit Tests: Packages (3, 3)
  • GitHub Check: e2e-webapp / 🧪 E2E Tests: Webapp (1, 2)
  • GitHub Check: e2e-webapp / 🧪 E2E Tests: Webapp (2, 2)
  • GitHub Check: runops-guard / runops-guard
  • GitHub Check: typecheck / typecheck
  • GitHub Check: code-quality / code-quality
  • GitHub Check: Analyze (javascript-typescript)
  • GitHub Check: Build and publish previews
🧰 Additional context used
📓 Path-based instructions (7)
Always import from `@trigger.dev/sdk`.

📄 CodeRabbit inference engine (AGENTS.md)

Files:

  • packages/trigger-sdk/src/v3/ai.ts
**Prefer static imports over dynamic imports.**

📄 CodeRabbit inference engine (AGENTS.md)

Files:

  • packages/trigger-sdk/src/v3/ai.ts
Add crumbs as you write code — not just when debugging.

📄 CodeRabbit inference engine (AGENTS.md)

Files:

  • packages/trigger-sdk/src/v3/ai.ts
In the Trigger.dev SDK (packages/trigger-sdk), prefer isomorphic code like fetch and ReadableStream instead of Node.js-specific code

📄 CodeRabbit inference engine (.github/copilot-instructions.md)

Files:

  • packages/trigger-sdk/src/v3/ai.ts
Use function declarations instead of default exports

📄 CodeRabbit inference engine (.github/copilot-instructions.md)

Files:

  • packages/trigger-sdk/src/v3/ai.ts
Use types over interfaces for TypeScript Avoid using enums; prefer string unions or const objects instead

📄 CodeRabbit inference engine (.github/copilot-instructions.md)

Files:

  • packages/trigger-sdk/src/v3/ai.ts
When creating or editing OTEL metrics (counters, histograms, gauges), ensure metric attributes have low cardinality by using only enums, booleans, bounded error codes, or bounded shard IDs Do not use high-cardinality attributes in OTEL metr...

📄 CodeRabbit inference engine (.cursor/rules/otel-metrics.mdc)

Files:

  • packages/trigger-sdk/src/v3/ai.ts
🔇 Additional comments (12)
packages/trigger-sdk/src/v3/ai.ts (12)

2322-2326: LGTM!

Also applies to: 2357-2363


7172-7173: LGTM!


7448-7452: LGTM!


7541-7542: LGTM!


7551-7564: Traced and correct. The seqNum lookup by message id correctly maps hookRecoveredTurns back to the original replayedInTail records (or leaves seqNum undefined for a hook-synthesized message that never had a record), and the default branches mirror the pre-existing splice/re-dispatch behavior exactly. No claim is left unaccounted for: every excluded entry is picked up by the settle loop at Line 7617-7621.


7595-7601: LGTM!


7610-7620: LGTM!


7804-7804: LGTM!


8055-8055: LGTM!


9636-9636: LGTM!


10012-10012: LGTM!


7270-7282: 🗄️ Data Integrity & Integration

SessionChannelRouter.settleRecovered(seqNum: number) removes only seqNum from #owed. It leaves #claimed unchanged, so late tail delivery remains suppressed. Since #owed is a Set, repeated calls are idempotent. Other owed sequences continue to hold the resume floor.


Walkthrough

The session router now tracks recovered sequence claims and unsettled claims separately. It drops late live-tail records for claimed sequences and holds the resume floor until each claim settles. Chat recovery passes individual sequence numbers to the router and settles dispatched or undispatched recovered records. Tests cover router floor behavior and prevent duplicate recovered chat messages and turns.

Priority: ➖ Normal

Severity of issue fixed: Medium

Merge Risk: 🔵 Low · up to 1268d

Recovery now claims and settles individual messages to prevent message loss, but a gap in recovered sequences can still advance a replay cutoff too far in a narrowed recovery path, potentially suppressing a message.

🚥 Pre-merge checks | ✅ 3 | ❌ 2

❌ Failed checks (2 warnings)

Check name Status Explanation Resolution
Description check ⚠️ Warning The description gives a clear and relevant technical summary of the fix, but it does not follow the repository template. It omits the issue closure line, checklist, Testing section, Changelog section,… Update the description to include Closes #<issue>, complete the checklist, add a Testing section with executed test steps, add a Changelog section, and add screenshots or state that screenshots are not applicable.
Docstring Coverage ⚠️ Warning Docstring coverage is 0.00% which is insufficient. The required threshold is 80.00%. Docstring coverage is scoped to functions touched by this diff. Analyzed 3 functions across 3 files. (1 skipped: 1 … Write docstrings for the functions missing them to satisfy the coverage threshold.
✅ Passed checks (3 passed)
Check name Status Explanation
Title check ✅ Passed The title clearly identifies the core and SDK fix for lost chat.agent messages during recovery. It is concise and specific.
Linked Issues check ✅ Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check ✅ Passed Check skipped because no linked issues were found for this pull request.
Full details: Description check

Explanation

The description gives a clear and relevant technical summary of the fix, but it does not follow the repository template. It omits the issue closure line, checklist, Testing section, Changelog section, and Screenshots section.

Full details: Docstring Coverage

Explanation

Docstring coverage is 0.00% which is insufficient. The required threshold is 80.00%. Docstring coverage is scoped to functions touched by this diff. Analyzed 3 functions across 3 files. (1 skipped: 1 too large.)

✨ Finishing Touches 💡 1
📝 Generate docstrings 💡
  • Create stacked PR
  • Commit on current branch
🧪 Generate unit tests (beta)
  • Create PR with unit tests
  • Commit unit tests in branch feature/tri-13752-chatagent-version-handover-duplicates-messages-and-turns

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

@pkg-pr-new

pkg-pr-new Bot commented Sep 8, 2026

Copy link
Copy Markdown

Open in StackBlitz

@trigger.dev/build

npm i https://pkg.pr.new/@trigger.dev/build@2ece24d

trigger.dev

npm i https://pkg.pr.new/trigger.dev@2ece24d

@trigger.dev/core

npm i https://pkg.pr.new/@trigger.dev/core@2ece24d

@trigger.dev/python

npm i https://pkg.pr.new/@trigger.dev/python@2ece24d

@trigger.dev/react-hooks

npm i https://pkg.pr.new/@trigger.dev/react-hooks@2ece24d

@trigger.dev/redis-worker

npm i https://pkg.pr.new/@trigger.dev/redis-worker@2ece24d

@trigger.dev/rsc

npm i https://pkg.pr.new/@trigger.dev/rsc@2ece24d

@trigger.dev/schema-to-json

npm i https://pkg.pr.new/@trigger.dev/schema-to-json@2ece24d

@trigger.dev/sdk

npm i https://pkg.pr.new/@trigger.dev/sdk@2ece24d

commit: 2ece24d

coderabbitai[bot]

This comment was marked as resolved.

@ericallam
ericallam marked this pull request as ready for review September 8, 2026 11:57
devin-ai-integration[bot]

This comment was marked as resolved.

@ericallam
ericallam force-pushed the feature/tri-13752-chatagent-version-handover-duplicates-messages-and-turns branch from 7d29239 to 9a2cfbb Compare September 8, 2026 12:38
@ericallam
ericallam changed the base branch from main to feat/transcript-storage-tri-13667 September 8, 2026 12:38
@ericallam
ericallam force-pushed the feature/tri-13752-chatagent-version-handover-duplicates-messages-and-turns branch 3 times, most recently from 52addef to 1268dba Compare September 8, 2026 17:43
@ericallam
ericallam force-pushed the feature/tri-13752-chatagent-version-handover-duplicates-messages-and-turns branch from 1268dba to 95d3ab5 Compare September 8, 2026 17:55
Rebased onto the transcript-storage stack. A continuation boot claims recovered session.in seqNums on the router and holds the resume cursor behind each until the boot settles it, so suppressing the tail's re-answer no longer advances the cursor past an un-answered message. Adds a changeset and router/boot tests.
@ericallam
ericallam force-pushed the feature/tri-13752-chatagent-version-handover-duplicates-messages-and-turns branch from 95d3ab5 to 2ece24d Compare September 8, 2026 18:09
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant