Skip to content

Commit 35b2cac

Browse files
committed
Merge branch 'features/publish-linux-100-rounds' into develop
2 parents 7418231 + 70ca8cf commit 35b2cac

8 files changed

Lines changed: 263 additions & 176 deletions

File tree

‎AGENTS.md‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -114,7 +114,7 @@ Tags are `vX.Y.Z`; GitHub Releases are cut as normal **Latest** releases
114114
themselves as `-dirty`.
115115

116116
```sh
117-
bun run bench:compare -- --rounds=10 # every arm, machine otherwise idle
117+
bun run bench:compare -- --rounds=100 # every arm, machine otherwise idle, hours
118118
bun run bench:compare:report # regenerates RESULTS.md
119119
```
120120

‎CHANGELOG.md‎

Lines changed: 29 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -69,6 +69,35 @@ breaking. See `ROADMAP.md` for what's coming, and `README.md` →
6969

7070
### Changed
7171

72+
- **The published comparison figures come from a hundred-round run on Linux**
73+
(#1327). All nine arms, one run, one machine, one commit — replacing a
74+
ten-round run on a Windows desktop. Every absolute number moved by a factor
75+
of two to five, so the columns are comparable with each other and with
76+
nothing quoted before.
77+
78+
Three published claims did not survive the move, and the pages carry them as
79+
corrections rather than restating them quietly:
80+
81+
- **The alternating volley is no longer a tie.** At ten rounds those columns
82+
carried spreads from ±27 % to ±82 % and the honest reading was that nobody
83+
was clearly ahead. At a hundred rounds they carry ±2–3 %, and actor-ts
84+
leads the best JVM arm by 1.4×.
85+
- **The round-trip row no longer splits by runtime.** The JVM arms were
86+
reported at 32–38 µs against under 8 µs elsewhere, explained as a non-actor
87+
thread parking on a future. On Linux the same arms at the same versions sit
88+
at 3.5–3.7 µs, beside 3.2 µs for the CLR arm. The mechanism is real and
89+
worth a fraction of a microsecond; the rest belonged to the previous host.
90+
- **"The two lineages agree to within the noise on every row"** was a claim
91+
about the noise, and a hundred rounds leave much less of it. The conclusion
92+
holds — neither lineage has a systematic advantage, and the largest gap at
93+
the Java binding is 6 % — but it is now stated as sizes rather than as
94+
agreement.
95+
96+
The language-binding finding gets stronger and simpler: 26 % and 36 % behind
97+
the Java siblings at a batch of 1 000 (t = 11 and 16), and at a batch of
98+
10 000 the effect is gone rather than merely narrowed (−1.5 % and +0.1 %).
99+
That is much better evidence for the reading the pages already offered.
100+
72101
- **The comparison gains two Scala 3 arms, and all four JVM arms move to one
73102
build tool** (#1229).
74103

‎README.md‎

Lines changed: 29 additions & 34 deletions
Original file line numberDiff line numberDiff line change
@@ -397,22 +397,22 @@ Run either with `bun examples/chat/backend/main.ts --port 2551` (then
397397

398398
## Benchmarks
399399

400-
Measured rather than asserted: one machine, one harness, one workload, ten
401-
interleaved rounds, and every row verified against work the system actually
402-
completed rather than work it was asked for. Each figure is the mean of those
403-
rounds; 🥇 marks the best value in its row.
400+
Measured rather than asserted: one machine, one harness, one workload, a
401+
hundred interleaved rounds, and every row verified against work the system
402+
actually completed rather than work it was asked for. Each figure is the mean
403+
of those rounds; 🥇 marks the best value in its row.
404404

405-
Bun 1.3.1 · AMD Ryzen 9 7940HX.
405+
Bun 1.3.14 · 10 cores of an Intel i9-12900K · Linux.
406406

407407
### Against JavaScript
408408

409-
| Scenario | actor-ts | nact | XState |
410-
| --------------------------- | -------------- | ------------- | --------- |
411-
| tell throughput (batch 1k) | **2.90M/s** 🥇 | 359k/s | 185k/s |
412-
| tell throughput (batch 10k) | **4.55M/s** 🥇 | 389k/s | 183k/s |
413-
| ping-pong (10k exchanges) | **504k/s** 🥇 | 166k/s | 83k/s |
414-
| spawn → started → stopped | 75k/s | **192k/s** 🥇 | 61k/s |
415-
| ask round-trip (p50) | **3.5 µs** 🥇 | 6.7 µs | 13.5 µs * |
409+
| Scenario | actor-ts | nact | XState |
410+
| --------------------------- | --------------- | ------------- | -------- |
411+
| tell throughput (batch 1k) | **13.82M/s** 🥇 | 1.59M/s | 878k/s |
412+
| tell throughput (batch 10k) | **19.05M/s** 🥇 | 1.64M/s | 961k/s |
413+
| ping-pong (10k exchanges) | **2.82M/s** 🥇 | 811k/s | 508k/s |
414+
| spawn → started → stopped | 355k/s | **698k/s** 🥇 | 332k/s |
415+
| ask round-trip (p50) | **0.9 µs** 🥇 | 1.5 µs | 2.2 µs * |
416416

417417
<sub>\* XState has no request/response primitive — that row is `send` plus a
418418
snapshot wait.</sub>
@@ -422,30 +422,25 @@ snapshot wait.</sub>
422422
Each framework appears through both its Java and its Scala API at the same
423423
pinned version, so a gap inside a pair is the language binding.
424424

425-
| Scenario | actor-ts (Bun) | Akka (Java) | Akka (Scala) | Pekko (Java) | Pekko (Scala) |
426-
| --------------------------- | -------------- | ----------- | ------------ | ------------ | ------------- |
427-
| tell throughput (batch 1k) | **2.90M/s** 🥇 | 2.11M/s | 1.31M/s | 1.71M/s | 1.10M/s |
428-
| tell throughput (batch 10k) | **4.55M/s** 🥇 | 3.14M/s | 2.80M/s | 3.04M/s | 2.84M/s |
429-
| ping-pong (10k exchanges) | 504k/s | 404k/s | 369k/s | 343k/s | **603k/s** 🥇 |
430-
| spawn → started → stopped | **75k/s** 🥇 | 28k/s | 26k/s | 27k/s | 28k/s |
431-
| ask round-trip (p50) | **3.5 µs** 🥇 | 34.9 µs † | 37.4 µs † | 32.8 µs † | 32.4 µs † |
432-
| Licence | MIT | BUSL-1.1 | BUSL-1.1 | Apache-2.0 | Apache-2.0 |
433-
434-
<sub>† Asked from a non-actor thread, the JVM parks a thread on a future where
435-
an event loop waits on a microtask.<br/>
436-
The volley columns move by ±27 % to ±82 % between rounds — read that row's
437-
🥇 as "came out highest this time", not as a winner.</sub>
425+
| Scenario | actor-ts (Bun) | Akka (Java) | Akka (Scala) | Pekko (Java) | Pekko (Scala) |
426+
| --------------------------- | --------------- | ----------- | ------------ | ------------ | ------------- |
427+
| tell throughput (batch 1k) | **13.82M/s** 🥇 | 7.23M/s | 5.37M/s | 6.76M/s | 4.34M/s |
428+
| tell throughput (batch 10k) | **19.05M/s** 🥇 | 9.18M/s | 9.03M/s | 9.36M/s | 9.37M/s |
429+
| ping-pong (10k exchanges) | **2.82M/s** 🥇 | 1.99M/s | 1.94M/s | 1.99M/s | 1.99M/s |
430+
| spawn → started → stopped | **355k/s** 🥇 | 105k/s | 97k/s | 102k/s | 90k/s |
431+
| ask round-trip (p50) | **0.9 µs** 🥇 | 3.7 µs | 3.5 µs | 3.6 µs | 3.5 µs |
432+
| Licence | MIT | BUSL-1.1 | BUSL-1.1 | Apache-2.0 | Apache-2.0 |
438433

439434
### Against .NET
440435

441-
| Scenario | actor-ts (Bun) | Akka.NET (C#) | Orleans (C#) |
442-
| --------------------------- | -------------- | ------------- | ------------ |
443-
| tell throughput (batch 1k) | **2.90M/s** 🥇 | 1.38M/s | 288k/s |
444-
| tell throughput (batch 10k) | **4.55M/s** 🥇 | 1.36M/s | 576k/s |
445-
| ping-pong (10k exchanges) | **504k/s** 🥇 | 415k/s | 157k/s |
446-
| spawn → started → stopped | **75k/s** 🥇 | 34k/s | 5k/s ‡ |
447-
| ask round-trip (p50) | **3.5 µs** 🥇 | 7.2 µs | 7.7 µs |
448-
| Licence | MIT | Apache-2.0 | MIT |
436+
| Scenario | actor-ts (Bun) | Akka.NET (C#) | Orleans (C#) |
437+
| --------------------------- | --------------- | ------------- | ------------ |
438+
| tell throughput (batch 1k) | **13.82M/s** 🥇 | 5.96M/s | 695k/s |
439+
| tell throughput (batch 10k) | **19.05M/s** 🥇 | 6.37M/s | 792k/s |
440+
| ping-pong (10k exchanges) | **2.82M/s** 🥇 | 499k/s | 357k/s |
441+
| spawn → started → stopped | **355k/s** 🥇 | 85k/s | 30k/s ‡ |
442+
| ask round-trip (p50) | **0.9 µs** 🥇 | 3.2 µs | 5.5 µs |
443+
| Licence | MIT | Apache-2.0 | MIT |
449444

450445
<sub>‡ Orleans has no caller-visible create or stop; grains activate on first
451446
call, so that row is activation latency rather than a comparable lifecycle.</sub>
@@ -458,7 +453,7 @@ or a real network.
458453
**[what the numbers mean](https://actor-ts.dev/reference/benchmarks/)** — the
459454
spread behind every figure, the pinned version of every arm, and what is
460455
deliberately not measured yet. Reproduce with
461-
`bun run bench:compare -- --rounds=10`.
456+
`bun run bench:compare -- --rounds=100`.
462457

463458
---
464459
## Roadmap & status

‎benchmarks/comparison/README.md‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -448,7 +448,7 @@ bun run typecheck:compare
448448
**Publish from `--rounds`, never from a single run:**
449449

450450
```bash
451-
bun run bench:compare -- --rounds=7
451+
bun run bench:compare -- --rounds=100
452452
```
453453

454454
This runs the arms **interleaved** — round 1 of every arm, then round 2 —

0 commit comments

Comments
 (0)