workload-replay: add console-scalability nightly workflow - #38048
Conversation
Run the console Playwright scalability suite against the console bundled in the materialized image while a captured workload (prod-analytics by default) provides continuous ingestion and query load. The suite only reports timings. Suite failures do not fail the workflow, the nightly runs collect data for determining thresholds. A red step means infrastructure breakage. * executor: test() gains a during_continuous callback. When set, the continuous phase runs until the callback returns instead of for a fixed runtime. * workload-replay: new console-scalability workflow and a Playwright runner service. The image tag must track @playwright/test in console/package.json so the preinstalled browsers stay compatible. * mzcompose: allow the ipc service config key, used to give Chromium a usable /dev/shm. * nightly: new console-scalability step. Uploads Playwright artifacts and the dumped console query set, which doubles as a fresh k6 input. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
After the Playwright suite finishes, replay the query set that its dump-queries step captured against the SQL HTTP endpoint at increasing virtual-user counts, while the workload load keeps running. Like the Playwright suite, k6 failures (including its thresholds) do not fail the workflow, the summary export is uploaded for determining thresholds. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* Tear down the idle Playwright runner container when the workflow ends so it cannot outlive the workflow into CI teardown. * Raise the page-load visibility ceiling to 120s. The suite measures load times rather than asserting them, and the navigation-flow test hit the old 30s ceiling under workload-replay load. * Refresh the k6 label set. clusters.list and replicaUtilizationHistory are SUBSCRIBE-fed now and never appear in queries.json, so their thresholds passed vacuously on empty metrics. Replace them with the newly captured maintainedObjectNames, replicaOfflineEvents, and roles.owners labels, with poll intervals matching the console. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…-nightly # Conflicts: # console/e2e-tests/k6/cluster-detail.ts
def-
left a comment
def-
left a comment
There was a problem hiding this comment.
Nice idea! Let's see how stable this will be.
| parser.add_argument( | ||
| "files", | ||
| nargs="*", | ||
| default=["workload_prod_analytics.yml"], |
There was a problem hiding this comment.
Why are we only running against this one file by default?
There was a problem hiding this comment.
Because the console suite measures fixed pages whose query cost scales with the catalog shape (object counts), not the data content — running every workload would mostly re-measure the same queries against smaller catalogs, and night-over-night timing trends are only comparable against a stable shape. prod-analytics is the largest capture (1,002 tables, 25 MVs, 7 clusters), so it's the single most representative one. Other workloads can still be passed explicitly, e.g. run console-scalability workload_ddnet.yml for a quick local run. Documented the rationale in 80fbbe4 — and if per-shape coverage becomes interesting later, sharding it like the benchmark step is straightforward.
There was a problem hiding this comment.
Changed in 912516f: the workflow now runs every captured workload, sharded across parallel jobs like the benchmark step (--skip-large in nightly), with per-workload Playwright output and k6 dumps/summaries as artifacts.
|
One side note: when you trigger nightly and you only changed one workload, you can only run that one; instead of all of it |
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Shard the workloads across parallel jobs like the benchmark workflow, with --skip-large in nightly. Playwright output and the k6 query dump and summary are kept per workload so artifacts from one workload do not overwrite the next one's. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
|
Addressed the review feedback: as of 912516f the scalability tests now run against all captured workloads instead of just prod-analytics. The workloads are sharded across parallel jobs like the benchmark step ( |
2 participants
Adds a nightly CI step that measures console performance against replayed production workloads.
For each captured workload (sharded across 3 parallel jobs,
--skip-large, 1% initial data), theconsole-scalabilityworkflow:The runs are metrics-only while thresholds are determined from nightly trends: test failures and k6 threshold breaches are logged but do not fail the step. A red step means infrastructure breakage.
Supporting changes:
test()gains an optionalduring_continuouscallback, so the continuous load phase can last exactly as long as the tests.ipcservice config key (Chromium needs more than the 64MB/dev/shmdocker default).clusters.listandreplicaUtilizationHistoryare SUBSCRIBE-fed now and no longer appear as HTTP queries).Validated end-to-end in nightly build 17658 (single-workload version): 13/14 Playwright tests passed under load, k6 completed the full ramp with 0% request failures, and all artifacts uploaded. The two k6 p95 outliers it found (clusterFreshness 3.8s, materializationLag 4.9s at 100 VUs) match the known production bottleneck ranking.
Motivation
Gives us recurring visibility into console performance under load. Part of CNS-122.
🤖 Generated with Claude Code