Enabling On-host profiling estimation for Executorch QNN HTP

[quic-boyuc]

User description

Summary

Introduce two officially supported HTP profiling workflows — on-device OpTrace and host-only Hextimate — behind a small public Python API returning QnnHtpProfileArtifacts. Split the demo into three workflow-named example scripts and rewrite the README around the three routes.

Motivation

Profiling QNN-delegated .ptes used to be a single demo with a hidden --online_prepare switch that silently picked between .dlc vs .bin, host-side vs on-device tools, and set (or didn't set) profile_level=3 behind the user's back. This PR replaces that with three named routes, one public API, and a self-contained .pte.

What each topic changes

Public HTP profiling API (backends/qualcomm/debugger/utils.py)

Two workflow-specific entry points instead of one mode-flag function. Each returns a QnnHtpProfileArtifacts per compiled binary in the .pte, carrying binary_path, profile_mode, prepare_mode, qhas_html, qhas_json, chrometrace_json, and HTP graph JSONs.

• generate_htp_profile_result() — on-device OpTrace (both prepare modes).
  • Validates offline-prepare inputs by decoding the .pte's compile spec and rejecting anything without profile_level=3 at the Python boundary, rather than producing an unusable output later.
  • For online-prepare, no profile_level is required — the QNN CLI attaches instrumentation at host-side context-binary-generation time.
• estimate_htp_profile_result() — host-only Hextimate.
  • Enforces online_prepare=True (Hextimate requires a .dlc, not a .bin).
  • Enforces supported-SoC allowlist (SA8540, SA8255, QCS9100, SA8797) before touching QNN tools, so misuse fails fast with a legible error.
  • Enforces minimum QNN SDK version support for Hextimate.
• Design rationale: two functions, not one function with a mode enum. Each route has genuinely different constraints (device vs host, prepare-mode requirements, SoC allowlist) — a shared entry point would push all four validation matrices into the caller. Two functions with clear names is cheaper than one function with four flags.

Compile-spec plumbing (backends/qualcomm/serialization/qc_compiler_spec.fbs, qc_schema.py, utils/utils.py)

• Adds SA8540 = 62 to QcomChipset and the host-side get_soc_to_chipset_map(). The other three Hextimate-supported SoCs (SA8255, QCS9100, SA8797) already existed.
• Wire value 62 matches the QNN SDK's internal SoC ID for SA8540 — the enum wire values are load-bearing, so the addition is at a specific numeric slot, not the tail of the enum.
• Purely additive; no existing entries are renumbered.

Example scripts (examples/qualcomm/util_scripts/)

One script per route, with the route encoded in the filename so a reader picks the right file without reading code:

• htp_profiling_on_device_op_trace_online.py — on-device OpTrace, online_prepare=True, .dlc path.
  • Asserts qnn_config.online_prepare at entry.
  • No profile_level handling — QNN CLI attaches instrumentation host-side.
• htp_profiling_on_device_op_trace_offline.py — on-device OpTrace, online_prepare=False + profile_level=3, .bin path.
  • Asserts qnn_config.profile_level == 3 and not qnn_config.online_prepare at entry.
  • Exercises the new self-contained schematic packaging.
• htp_profiling_on_host_hextimate.py — host-only Hextimate (renamed from the old merged demo, not net-new).
  • Requires --enable_x86_64; no adb involved.

qairt-visualizer is optional in all three scripts:

• If installed: reports open automatically after generation.
• If not installed: qhas_html paths are printed to stdout so users can view them however they want.
• Removes a soft dependency without losing the happy path.

Design rationale: three short files (~90 lines each) beat one file with if args.mode == "..." branches. The old merged demo had a bug where --profile_level 3 was passed under --online_prepare even though online-prepare doesn't need it — the kind of bug you get when one file quietly handles multiple pipelines. Naming a file after its pipeline eliminates the class.

Documentation (backends/qualcomm/debugger/README.md + examples/qualcomm/util_scripts/README.md)

• README rewritten around the five user tasks rather than around the toolchain:
  1. Select a .pte prepare mode (decision table: online-prepare needs Hextimate/graph view; offline-prepare when you already have a profile_level=3 .pte).
  2. Run on-device OpTrace generation.
  3. Run host-only Hextimate estimation.
  4. Inspect the returned QnnHtpProfileArtifacts.
  5. View QHAS reports with QAIRT Visualizer.
• Hextimate limitations and the supported-SoC list are documented in place, next to the code path that enforces them.
• README quick-start snippets link directly to the three renamed example scripts.
• examples/qualcomm/util_scripts/README.md indexes each of the three profiling example scripts by filename.

Tests (backends/qualcomm/tests/test_qnn_delegate.py)

• New public-API test covering unsupported-SoC rejection for Hextimate before reading .pte inputs — this is a defensive-order check (validate first, then touch inputs), not just an error-message check. Catches regressions where validation gets reordered behind expensive .pte parsing.
• test_qnn_backend_generate_optrace refactored in both TestQNNFloatingPointUtils and TestQNNQuantizedUtils:
  • Exercises both prepare modes in one test (online-prepare, then offline-prepare with profile_level=3).
  • Consumes the new QnnHtpProfileArtifacts return shape (.qhas_json, .chrometrace_json).
  • Skip when enable_x86_64 (OpTrace requires on-device execution) or LPAI backend (unsupported).
• test_qnn_backend_generate_hextimate refactored in TestQNNQuantizedUtils:
  • Skip when not enable_x86_64 (Hextimate is host-only; needs --enable_x86_64).
  • Asserts qhas_json is None — Hextimate QHAS JSON is truncated by an upstream SDK bug (division by zero on time_us=0); we surface this by returning None from the API rather than papering over it. The HTML report and chrometrace remain usable.
• Skip logic is complementary: OpTrace ⇒ on-device (enable_x86_64 skips), Hextimate ⇒ host-only (not enable_x86_64 skips). Users run them in two separate test invocations with different --build_folder / --enable_x86_64 combinations.

Verification

Script syntax

python -m py_compile \
  examples/qualcomm/util_scripts/htp_profiling_on_device_op_trace_online.py \
  examples/qualcomm/util_scripts/htp_profiling_on_device_op_trace_offline.py \
  examples/qualcomm/util_scripts/htp_profiling_on_host_hextimate.py

End-to-end example scripts (three routes, matched flags)

python -m examples.qualcomm.util_scripts.htp_profiling_on_device_op_trace_online \
  --host <HOST> --device <DEVICE_SERIAL> --soc_model <SOC_MODEL> \
  --build_folder build-android --online_prepare -a <ARTIFACT_DIR>

python -m examples.qualcomm.util_scripts.htp_profiling_on_device_op_trace_offline \
  --host <HOST> --device <DEVICE_SERIAL> --soc_model <SOC_MODEL> \
  --build_folder build-android --profile_level 3 -a <ARTIFACT_DIR>

python -m examples.qualcomm.util_scripts.htp_profiling_on_host_hextimate \
  --soc_model SA8797 --build_folder build-x86 --enable_x86_64 --online_prepare -a <ARTIFACT_DIR>

Targeted profiling tests (existing framework, no CLI shape change)

python -m backends.qualcomm.tests.test_qnn_delegate \
  TestQNNQuantizedUtils.test_qnn_backend_generate_optrace \
  --host <HOST> --device <DEVICE_SERIAL> --soc_model <SOC_MODEL> --build_folder build-android

python -m backends.qualcomm.tests.test_qnn_delegate \
  TestQNNFloatingPointUtils.test_qnn_backend_generate_optrace \
  --host <HOST> --device <DEVICE_SERIAL> --soc_model <SOC_MODEL> --build_folder build-android

python -m backends.qualcomm.tests.test_qnn_delegate \
  TestQNNQuantizedUtils.test_qnn_backend_generate_hextimate \
  --soc_model SA8797 --build_folder build-x86 --enable_x86_64

[pytorch-bot]

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/pytorch/executorch/22143

• 📄 Preview Python docs built from this PR

Note: Links to docs will display an error until the docs builds have been completed.

This comment was automatically generated by Dr. CI and updates every 15 minutes.

[github-actions]

This PR needs a release notes: label

If your change should be included in the release notes (i.e. would users of this library care about this change?), please use a label starting with release notes:. This helps us keep track and include your important work in the next release notes.

To add a label, you can comment to pytorchbot, for example
@pytorchbot label "release notes: none"

For more information, see
https://github.com/pytorch/pytorch/wiki/PyTorch-AutoLabel-Bot#why-categorize-for-release-notes-and-how-does-it-work.

[quic-boyuc]

@pytorchbot label "release notes: add support for on-host QNN HTP profiling estimation"

[pytorch-bot]

Didn't find following labels among repository labels: release notes: add support for on-host QNN HTP profiling estimation

[psiddh]

cc @Gasoonjia @mohankumarkumar @digantdesai

[psiddh]

@claude review HTP profiling workflows

[claude]

Claude finished @psiddh's task in 6m 31s —— View job

---

Review: HTP profiling workflows

• Read PR context
• Review debugger/utils.py (public API, validation)
• Review compile-spec plumbing (SoC enum)
• Review example scripts (3 routes)
• Review tests

Overall this is a solid refactor — two clearly-named entry points, a frozen QnnHtpProfileArtifacts dataclass, fail-fast validation, and per-route example scripts read well. The qhas_json=None-on-truncation handling is the right call and is nicely documented. I did find a few concrete issues, one of which will break existing tests.

---

🔴 Blocking: three test_qnn_backend_hadamard_transform_* tests were not migrated to the new return shape

generate_htp_profile_result() now returns List[QnnHtpProfileArtifacts], but three callers still consume the old dict return shape and will crash with AttributeError: 'list' object has no attribute 'items':

• test_qnn_backend_hadamard_transform_linear — test_qnn_delegate.py:4471
• test_qnn_backend_hadamard_transform_matmul — test_qnn_delegate.py:4531
• test_qnn_backend_hadamard_transform_conv — test_qnn_delegate.py:4585

All three do:

for _, (_, qhas) in binaries_trace.items():   # old shape; now a List
    with open(qhas, "r") as qhas_file:
        ...

These should iterate the list and read a.qhas_json like the migrated tests (test_qnn_delegate.py:6551). They're device-gated + SDK-gated so CI may skip them, but they're broken for anyone running on device.

Additionally, test_qnn_backend_hadamard_transform_matmul (test_qnn_delegate.py:4523-4529) passes arguments in the old positional order:

generate_htp_profile_result(
    tmp_dir,
    self.chipset_table[TestQNN.soc_model],
    adb,          # <-- lands in pte_path
    pte_path,     # <-- lands in inputs
    [sample_input],  # <-- lands in adb
)

The new signature is (artifact_dir, soc_id, pte_path, inputs, adb). The other two callers (:4463, :4577) already use the new order, so this one is inconsistent and wrong. Fix this →

---

🟠 estimate_htp_profile_result() doesn't enforce online_prepare / SDK version as early as the PR claims

The PR description says Hextimate "Enforces online_prepare=True … before touching QNN tools" and "Enforces minimum QNN SDK version support for Hextimate." In practice, only the SoC allowlist is checked up front (utils.py:693). The flow is:

estimate_htp_profile_result → _validate_hextimate_soc  (early ✓)
                            → _generate_htp_analysis_result
                              → _validate_pte_profile_level   # skips online-prepare delegates
                              → dump_context_from_pte          # dumps the binary
                              → qnn_tool.run                   # ext==".dlc" + SDK check happen HERE

So the online_prepare (.dlc) requirement and the SDK-version gate only fire in QnnTool.run() (utils.py:499-511), after the .pte has been parsed and the binary dumped. Functionally you still get an error, but it contradicts the stated "fail fast before touching inputs" design, and the dedicated regression test (test_estimate_htp_profile_result_rejects_unsupported_soc_before_pte) only covers the SoC path. Consider either validating online_prepare + SDK version in estimate_htp_profile_result() alongside the SoC check, or softening the wording in the docstring/PR.

---

🟡 _qnn_profile_viewer resolves the schematic from os.getcwd()

utils.py:436-443:

# TODO: remove assumption that AOT dumpped schematic file exists in same cwd
schematic = os.path.join(os.getcwd(), f"{schematic_stem}.bin")

Everything else in QnnTool is keyed off self.artifact_dir, but the schematic is looked up relative to the current working directory. For offline-prepare this couples correctness to wherever the process was launched, and the online-prepare qnn-context-binary-generator step writes into artifact_dir (--output_dir self.artifact_dir), not cwd — so an online run whose cwd ≠ artifact_dir would miss it. The .pte-self-contained TODO is acknowledged, but until then it may be worth checking artifact_dir first and falling back to cwd, so the two producers and the consumer agree on a location.

---

🟢 Minor

• Unused build_folder: QnnTool.__init__ stores self.build_folder (utils.py:286) but it's never read. It's threaded through from _generate_htp_analysis_result (utils.py:625) — drop it or use it.
• PEP8 spacing: generate_optrace (utils.py:632) and the block above generate_htp_profile_result (utils.py:631, :641) are single-blank-line separated at module level; lintrunner (E302) will want two. Worth a lintrunner -a pass.
• compile_specs[0] assumption: _validate_pte_profile_level (utils.py:584) reads only the first compile spec per QnnBackend delegate. Fine for current export, but a brief comment on why index 0 is authoritative would help future readers.
• Enum comment nit: SA8540 = 62 # v68 (qc_schema.py:75) and .fbs:68 agree, and _soc_info_table (qc_schema.py:110) + get_soc_to_chipset_map (utils/utils.py:1343) are all wired consistently — 👍. Purely additive, wire value matches the SDK's SoC ID as described.

---

Things that look right

• QnnHtpProfileArtifacts frozen dataclass + visualizer_reports() filtering out None qhas_json is clean and prevents passing a truncated report to the visualizer.
• The two-function split (vs. one function + mode enum) is justified given the divergent validation matrices; naming files after their pipeline is a good call and the assert guards at each script's entry make misuse loud.
• _validated_qhas_json returning None on JSONDecodeError rather than papering over the SDK bug, with the hextimate test asserting qhas_json is None — good defensive symmetry.
• Optional qairt_visualizer import with a stdout fallback in all three scripts removes the soft dependency cleanly.
  · branch clo/dev/boyuc/hextimate

[psiddh]

@quic-boyuc can you fix the lint errors pls ?