This directory contains standalone performance scenarios, workload applications, their launcher, and guidance for repeatable profiling and analysis. Keep scenario files, commands, results and interpretation together so a later run can reproduce the same workload.
For micro-level questions about one class or method, use the JMH benchmarks in
microbench. JMH is the benchmark harness; this directory is for documenting the
end-to-end profiling scenario, profile collection, analysis and conclusions. A useful experiment keeps
the workload definition, the revision under test, the profiler options, the raw recording and the
resulting analysis together.
Performance scenarios belong in this directory. Build reusable, mountable workload applications in
tools with ./gradlew :tests:performance:tools:installDist, describe workloads in
scenarios, and run them through the standalone launcher. The launcher owns the
Testcontainers cluster and workload lifecycle directly, consumes recursively merged YAML, persists the resolved
configuration and run artifacts, and does not use a unit-test framework as a process runner. Shared scenario
loading is implemented in common.
The original profiling harness under tests/integration uses TestNG classes as wrappers around a manually run
performance workload. That runner is deprecated: TestNG discovery and test lifecycle add no useful test semantics
to these long-running profiling scenarios and make them harder to invoke and automate as standalone jobs. It is
retained temporarily for its existing v4 and v5 pulsar-perf scenarios while they are migrated. Add new scenarios,
workload applications and profiling support to tests/performance and the standalone launcher instead.
The IoT scenarios exercise keyed telemetry fanout, ordering, client restart and saturation behavior. Run the host-sized scenario with:
./gradlew :tests:performance:launcher:run \
--args='--config tests/performance/scenarios/iot-telemetry-local.yaml'Use the profile task when the selected scenario contains async-profiler options:
./gradlew :tests:performance:launcher:profile \
--args='--config tests/performance/scenarios/iot-telemetry-high-rate-profile.yaml'The Gradle tasks build the Pulsar test image and the workload distribution before launching the scenario. See the IoT scenario reference for topology, correctness checks and output details.
Scenario YAML is a reusable configuration tree rather than a format tied to a test class. The shared loader in
common resolves the tree; launchers and workload applications select the subtree they own. The
standalone launcher uses these top-level sections:
cluster: the Pulsar topology and broker or BookKeeper environment settings;workloads: named workload configurations, currently includingiotTelemetry;profiling: optional async-profiler settings for the broker, producer and consumer processes; andoutput: the run-artifact directory.
Workload-specific fields live below their workload name so another launcher or application can reuse the same
file without interpreting unrelated sections. The launcher writes the fully resolved tree to
resolved-config.yaml in the run directory and mounts that file into workload containers. A workload command can
select its subtree with --config-path.
The iotTelemetry workload can run traffic before measurements begin. Use warmupSeconds with a positive rate,
or use warmupMessages when rate: 0; the two settings are mutually exclusive. warmupRounds repeats that
traffic, and warmupRoundDelaySeconds adds an idle stabilization period after each fully drained round, including
the final round. A round is fully drained only after every backend application has uniquely received its cumulative
warmup message count; producer send completions alone do not release the barrier. The default is one round with no
delay. Warmup traffic remains part of delivery and ordering validation. Producer throughput and the
epoch-millisecond measurement boundaries in
producer-summary.json cover only the configured measurement messages. Every consumer summary records its first
and last measured-message receipt as metadata. The launcher cuts from the producer measurement start through the
latest last receipt across all backend applications.
Use a top-level extends entry to inherit one file or an ordered list of files:
extends: [cluster.yaml, workloads/iot-base.yaml]
workloads:
iotTelemetry:
rate: 1000
clientRestartFraction: 0.1
profiling:
brokerOptions: event=cpu,interval=10ms,jfrsync=profile
producerOptions: ~
retainOriginalRecording: true
createMeasurementRecording: true
output:
directory: build/performance/iot-restart-profileEach inherited path is resolved relative to the file that declares it; absolute paths also work. Parents can
inherit other files recursively. Parents are applied in list order and the current file is applied last. Mappings
merge recursively, while scalar values and lists replace earlier values. An explicit YAML null or ~ removes
an inherited entry. Cycles, missing files, non-mapping roots and invalid extends entries are rejected.
Profiled standalone runs retain the complete JFR and also create a sibling whose name ends in
.measurement.jfr. The measurement recording contains events from the producer's recorded measurement start through
the latest measured-message receipt across all backend applications. This excludes startup, warmup, and shutdown
while retaining the broker and consumer work needed to deliver every measured message. One-time JVM, host, recording
setting and runtime configuration events are copied from the beginning of the complete recording so JDK Mission
Control can describe the source JVM. Set
profiling.retainOriginalRecording: false to remove the complete
recording after a successful cut, or profiling.createMeasurementRecording: false to keep only the complete
recording. Both options default to true and apply to broker, producer and consumer recordings.
Setting both to false intentionally discards all recordings produced by the current run. Retention options do
not remove recordings from earlier runs. Use a fresh output directory for each experiment to keep profiles,
summaries, and histograms together without mixing artifacts from different runs.
These timestamps assume that producer, consumer, and broker clocks agree, as they do for containers on the same Docker host. Multi-host experiments need synchronized clocks; the launcher does not estimate clock skew or correct the cut window. The broker-publish-to-listener latency uses the same clock assumption.
Every IoT run writes producer/produce-latency.hdr with successful measured-message send-completion latency and
one consumer-*/consume-latency.hdr per backend application with measured-message broker-publish-to-listener
latency. Both use microseconds internally and three significant digits. Warmup messages are tagged in the payload
and excluded. Consumer latency uses a timestamp captured on listener entry; the sample is recorded after payload
decoding and key validation, before sequence validation and acknowledgment. Decoding and validation time are
excluded from the latency value.
Render the producer distribution together with the count-weighted merge of all backend-application consumer histograms as PNG and SVG:
./gradlew :tests:performance:launcher:renderHdrHistograms \
--args='--run-directory tests/performance/build/iot-telemetry-high-rate-profile'The default outputs are latency-histograms.png and latency-histograms.svg in the run directory. Pass
--output-prefix /path/to/name or --title 'Comparison label' to change them.
Use the same cutter independently to select a different interval from an existing recording. --from and --to
accept ISO-8601 instants, epoch milliseconds, or offsets from the recording start such as 500ms, 5s, 2m, 1h,
or PT5S. Omit --from to select from the beginning, or omit --to to select through the end. Use --info
without either boundary to display the actual recording start, end and total duration from the JFR chunk headers;
it can also accompany a cut. JFR cutting preserves those source chunk timestamps, so the original recording period
remains available in the cut file and in JDK Mission Control. The
task requires JDK 19 or newer because it uses the public JFR recording writer added in that release:
./gradlew :tests:performance:launcher:runJfrCut \
--args='--input /tmp/full.jfr --from 5s --to 2m --output /tmp/measurement.jfr --info'Java code can call JfrCut.cut(Path input, Instant from, Instant to, Path output) or
JfrCut.cutFrom(Path input, Instant from, Path output) directly without invoking the command-line entry point.
JfrCut.cutUsingTimeExpressions(...) provides the relative and omitted-boundary syntax,
and JfrCut.recordingInfo(...) returns the event range. Events overlapping the half-open interval [from, to)
are retained: duration events ending exactly at from are excluded, instantaneous events at from are included,
and events starting exactly at to are excluded.
For one-off standalone overrides, prefix an existing scalar path with PULSAR_PERFORMANCE_, uppercase it and
separate path elements with underscores. The loader preserves the scalar's YAML type. For example:
PULSAR_PERFORMANCE_WORKLOADS_IOTTELEMETRY_RATE=2000 \
./gradlew :tests:performance:launcher:run \
--args='--config tests/performance/scenarios/iot-telemetry-local.yaml'Environment overrides are applied after inheritance. They only update paths present in the resolved tree, which
keeps misspelled or workload-inapplicable settings from creating new configuration. Store maintained scenarios in
scenarios; use environment overrides for temporary measurements rather than as the only record of
a workload.
The deprecated TestNG runner remains available for the existing scenarios:
./gradlew :tests:integration:profilingIntegrationTest
./gradlew :tests:integration:profilingIntegrationTest --tests "*PulsarProfilingV4Test"The first command profiles the v5 scalable-topic scenario. The second uses the v4 client against a
classic persistent:// topic. Both variants profile a single broker and write recordings and command
output under tests/integration/build/pulsar-profiling. Do not use this runner as the basis for new scenarios.
The harness accepts a YAML scenario file through PULSAR_PROFILING_CONFIG. Start with
pulsar-profiling.yaml; omitted values retain the existing defaults. The
sections correspond to the main components of a run: cluster, load, profiling and output. Individual scalar
values can still be overridden for a one-off run with the PULSAR_PROFILING_ prefix and an upper-case
path, for example:
PULSAR_PROFILING_CONFIG="$PWD/tests/performance/scenarios/pulsar-profiling.yaml" \
PULSAR_PROFILING_LOAD_NUMBER_OF_MESSAGES=1000000 \
./gradlew :tests:integration:profilingIntegrationTest --tests "*PulsarProfilingV4Test"For the v4 scenario, set load.isolatedProducers or load.isolatedConsumers to create that many
independent v4 client instances. The corresponding pulsar-perf command receives
--isolated-clients, a v4-client option; the v5 scenario ignores these fields. The option is mutually exclusive with the regular
producer test-thread option and with consumer listener-thread expansion. Set load.producerCount
and load.consumerCount separately: creating clients does not create producers or consumers.
load.subscriptionType selects the subscription type; producerIoThreads and consumerIoThreads
size the shared client IO pools. maxOutstanding is per producer, not a global limit.
Set load.batchingEnabled: true to use pulsar-perf's default producer batching; the default is
false, preserving unbatched entry-by-entry measurements.
For a single Key_Shared subscription with 500 producers and 20 consumers, use
key-shared-500x20.yaml. The load.messageKeyGenerationMode
option maps to pulsar-perf's --message-key-generation-mode: random uses random integer keys,
autoIncrement uses the sender's message counter, and an empty or null value omits keys.
Override it with PULSAR_PROFILING_LOAD_MESSAGE_KEY_GENERATION_MODE. This scenario disables batching
so every entry has one key, and uses isolated clients with shared resources on both sides.
All consumers use the same subscription; receiverQueueSize is per consumer.
Client profiling is optional and independent of broker profiling. Set profiling.producerOptions
and/or profiling.consumerOptions to async-profiler options, for example
event=cpu,interval=10ms,lock=0,alloc=2m,jfrsync=profile. An empty or null option disables that
client's profiler. Client recordings are named client-producer-*.jfr and client-consumer-*.jfr
in the same output directory. The harness grants native CPU profiling access only to enabled clients.
The broker continues to use -Pinttest.asyncprofiler.opts. To inspect lock contention without
wall-clock sampling overhead, omit wall and use lock=0 to record all supported lock events.
Use the same options for baseline and candidate; extra profiling has a measurement cost.
Broker-side variations from the contention investigations need no dedicated environment switches:
use cluster.brokerEnvs for write-buffer watermarks, dispatcher batch size, broker/BookKeeper IO
thread counts and PULSAR_GC; use cluster.brokerMemory for heap/direct-memory limits and JVM
properties such as allocator or transport selection. These maps let YAML scenarios retain the
complete configuration instead of relying on shell history.
The harness saves resolved-config.yaml with inheritance and environment overrides applied in the output directory.
For v4 production, --num-producers remains the producer count per topic and is distributed across
the isolated clients; when the counts differ, producers are assigned round-robin as evenly as possible.
- IoT telemetry fanout and ordering: keyed telemetry through interchangeable gateways to Key_Shared applications, including isolated shared-resource clients, restart validation, a 500-connection saturation workload, and standalone async-profiler integration.
- Read-completion queue isolation: 500 producers on separate connections to one persistent topic, with one Exclusive consumer. Includes the scenario YAML, an inherited Shared-subscription variant, a Failover variant, explicit 64/32 KiB and 256/128 KiB channel-watermark variations, and baseline/comparison instructions.
Render the CPU, wall-clock, allocation and lock views with:
./gradlew jfrFlamegraphs -Pjfr=tests/integration/build/pulsar-profilingThe .jfr files can also be opened in Eclipse Mission Control or IntelliJ
IDEA. Do not use jfr summary as a measure of profile completeness: recordings made with
jfrsync=profile contain profiler samples that the JDK summary does not show.
On macOS, add the JDK Mission Control application launcher to a directory on PATH:
mkdir -p ~/.local/bin
ln -s /Applications/JDK\ Mission\ Control.app/Contents/MacOS/jmc ~/.local/bin/jmcJDK Mission Control requires an absolute recording path. From the directory containing a recording, open it with:
jmc -open "$PWD/<recording.jfr>"The following shell function accepts a relative or absolute path and resolves it before launching JMC. Add it to
~/.zshrc or the corresponding shell startup file:
jmc-open() {
if [ "$#" -ne 1 ]; then
echo "usage: jmc-open <recording.jfr>" >&2
return 2
fi
local recording directory
recording=$1
directory=$(cd "$(dirname "$recording")" && pwd -P) || return
jmc -open "$directory/$(basename "$recording")"
}With IntelliJ IDEA's command-line launcher installed, open a recording directly with:
idea <recording.jfr>The Jafar MCP server lets an AI coding agent query a recording. Register it once with JBang and JDK 25+:
claude mcp add jafar -- jbang jfr-mcp@btraceio --stdioUse jfr_diagnose and jfr_stackprofile first, then query further with the other Jafar tools when
needed. Save the result beside the recording as <recording>.analysis.md, in addition to showing the
report in the console. A useful starting prompt is:
use Jafar MCP's jfr_diagnose and jfr_stackprofile to analyze @filename.jfr. Besides showing the report on the console, write the analysis in a markdown file with the jfr file as prefix and the suffix as ".analysis.md"
Treat automated analysis as a lead. Confirm a performance claim with a controlled comparison, a JMH benchmark where appropriate, or a second profile.
For an OutOfMemoryError or suspected retention problem, analyze the resulting .hprof with a
headless Eclipse Memory Analyzer (MAT) MCP server such as
mcp-mat. Use the leak suspects report and dominator tree first,
then query paths to GC roots or OQL for the retained objects. Keep the heap dump and the MCP result
outside the source tree when they contain sensitive workload data; record the commands, heap limits and
the resulting conclusions in the experiment notes.