Skip to content

Fix the JS test shard dropping results and never awaiting the snapshot tests [CLF-454] - #1571

Merged
zach-klippenstein merged 2 commits into
mainfrom
zachklipp/js-test-shard
Sep 4, 2026
Merged

Fix the JS test shard dropping results and never awaiting the snapshot tests [CLF-454]#1571
zach-klippenstein merged 2 commits into
mainfrom
zachklipp/js-test-shard

Conversation

@zach-klippenstein

Copy link
Copy Markdown
Collaborator

The "JS Unit Tests for KMP Modules" shard has not been testing what it claims to. Two independent problems, both present on main:

1. Gradle throws away most of the Karma results

KGP's Karma reporter (kotlin-web-helpers/dist/karma-kotlin-reporter.js) writes its ##teamcity[...] service messages to stdout back-to-back with no newline between them. KGP's Gradle-side parser (TCServiceMessageOutputStreamHandler) buffers stdout until it sees a newline and drops any "line" longer than 1 MiB, logging:

Cannot process output: too long teamcity service message (more than 1Mb). Event was lost.
java.lang.IllegalStateException: no running test

Since the whole run is a single line, every 1 MiB chunk is discarded and only the final remainder is parsed. On recent main runs Gradle recorded roughly 2,000 of the 27,955 workflow-runtime JS tests that actually run. Any failure inside a dropped chunk is invisible, so the shard is green regardless.

Fix: build-logic/karma.config.d/newline-delimited-kotlin-reporter.js wraps KGP's reporter so every message ends with a newline, and KotlinMultiPlatformConventionPlugin appends that directory to every generated karma.conf.js via KotlinKarma.useConfigDirectory. After this change all 27,955 workflow-runtime and 50 workflow-core tests are recorded and the overflow message is gone.

2. Burst never awaits RenderWorkflowInSnapshotSaveRestoreTest on JS

Burst 2.13.0 only generates specializations that return runTest(...) { … } when the parameterized test function's body calls runTest itself. When runTest is behind a helper (which #1568 introduced to dodge a Kotlin/Native compiler crash), the generated per-value functions call the original as a bare statement and return nothing. On JS that return value is the Promise Mocha has to await, so all 4,096 saves_to_and_restores_from_snapshot_* tests completed instantly and "passed" without their bodies ever finishing. Those thousands of un-awaited bodies then ran concurrently with the rest of the suite and starved the event loop, which is what made unrelated tests hit Mocha's 2 s timeout on #1442.

Fix: the test calls runTest directly again, but with no arguments that reference this (the shape that crashes Kotlin/Native), and launches both runtimes in explicit TestScope(dispatcherUsed) scopes instead. The class KDoc documents both constraints. Burst itself should also be fixed to return the delegate's value; that's a follow-up.

Verification

Locally, with these changes on main:

Task Recorded Failures
:workflow-runtime:jvmTest (this class) 16,384 0
:workflow-runtime:iosSimulatorArm64Test (this class) 16,384 0
:workflow-runtime:jsBrowserTest (full) 27,955 0
:workflow-core:jsBrowserTest (full) 50 0

Ref: CLF-454

🤖 Generated with Claude Code

zach-klippenstein and others added 2 commits September 4, 2026 15:22
KGP's Karma reporter writes its ##teamcity[...] service messages to stdout
back-to-back with no newline between them. KGP's Gradle-side parser
(TCServiceMessageOutputStreamHandler) buffers stdout until it sees a newline
and drops any "line" longer than 1 MiB, logging "too long teamcity service
message (more than 1Mb). Event was lost" and "IllegalStateException: no
running test". Since the whole run is one line, every 1 MiB chunk is thrown
away and only the final remainder is parsed: Gradle was recording roughly
2,000 of the 27,955 workflow-runtime JS tests, and any failure in a dropped
chunk was invisible.

Add a karma.config.d script that wraps the reporter so every message ends
with a newline, and have KotlinMultiPlatformConventionPlugin append that
directory to every generated karma.conf.js. The plugin hooks
KotlinJsTest.onTestFrameworkSet rather than calling useKarma so that it
doesn't replace KGP's default Karma configuration (and its ChromeHeadless
browser) with an empty one.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Burst 2.13.0 only generates specializations that return runTest(...) when
the parameterized test function's body calls runTest itself. Since #1568
moved the body of RenderWorkflowInSnapshotSaveRestoreTest into a helper to
dodge a Kotlin/Native compiler crash, the generated per-value test
functions have been calling the original as a bare statement and returning
nothing. On JS that return value is the Promise Mocha awaits, so all 4,096
saves_to_and_restores_from_snapshot_* tests completed instantly and passed
without their bodies finishing. The un-awaited bodies then ran concurrently
with the rest of the suite and starved the event loop, which is what made
unrelated tests hit Mocha's 2 s timeout on #1442.

Call runTest directly again, but without any argument that refers to
`this` (the shape that crashes the Kotlin/Native backend), and launch both
runtimes in explicit TestScope(dispatcherUsed) scopes instead. The class
KDoc documents both constraints.

Verified locally on main: jvmTest and iosSimulatorArm64Test for this class
each record 16,384 tests with 0 failures, and the full jsBrowserTest run
records 27,955 tests with 0 failures.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
@zach-klippenstein
zach-klippenstein requested a review from a team as a code owner September 4, 2026 22:23
@zach-klippenstein
zach-klippenstein merged commit 068977c into main Sep 4, 2026
143 of 145 checks passed
@zach-klippenstein
zach-klippenstein deleted the zachklipp/js-test-shard branch September 4, 2026 23:00
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants