Add CodSpeed performance benchmarks for core hot paths - #175143
Conversation
home-assistant
Bot
added
cla-signed
code-quality
small-pr
Check requirementsChecked at commit
📦 pytest-codspeed: 5.0.3
|
There was a problem hiding this comment.
Pull request overview
This PR introduces a CodSpeed performance benchmarking harness to guard Home Assistant Core's busiest runtime hot paths (event bus, state machine, state-change tracking, dispatcher, timers, entity writes, and the template engine) against regressions. Benchmarks live in a new top-level benchmarks/ directory that sits outside testpaths, so the normal test suite never collects them; CodSpeed runs them via a dedicated CI workflow gated on core changes.
Changes:
- Adds a
benchmarks/package with a shared asynchassfixture and three benchmark modules covering events, state machine/entity writes, and templates, usingbenchmark.pedantic(with teardown) and parametrized sizes for size-sensitive paths. - Adds the
pytest-codspeed==5.0.3test dependency and a RuffTID251per-file ignore so benchmarks may importtests.common. - Adds a
.github/workflows/codspeed.ymlworkflow using the CodSpeed v4 action insimulationmode with OIDC auth, gated through.core_files.yaml.
Reviewed changes
Copilot reviewed 8 out of 8 changed files in this pull request and generated 2 comments.
Show a summary per file
| File | Description |
|---|---|
benchmarks/__init__.py |
Marks benchmarks/ as a package. |
benchmarks/conftest.py |
Provides the async hass fixture and a populate_states helper for scaling benchmarks. |
benchmarks/test_core_events.py |
Benchmarks event fire/listen, state-change tracking, dispatcher, and timer scheduling. |
benchmarks/test_core_states.py |
Benchmarks state create/update/report/read paths and the entity write path. |
benchmarks/test_core_template.py |
Benchmarks template compile, render, render-to-info, and state-walking renders. |
pyproject.toml |
Adds TID251 ignore for benchmarks/** so test helpers can be imported. |
requirements_test.txt |
Adds the pytest-codspeed==5.0.3 dependency. |
.github/workflows/codspeed.yml |
New CI workflow running benchmarks via CodSpeed, gated on core changes. |
Congrats! CodSpeed is installed 🎉
You will start to see performance impacts in the reports once the benchmarks are run from your default branch.
|
…eivers bug - Remove redundant explanatory comment for CodSpeed mode input - Add memory mode alongside simulation for allocation measurements - Fix async_dispatcher_connect test to use distinct receiver callables per connection, since receivers are keyed by the callable object - Assert listener/receiver call counts and filtered/cancelled callbacks are (not) invoked, using invariants that hold regardless of how many times the benchmark harness calls the measured function - Assert exact state value in test_state_get instead of just not-None Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
edenhaus
left a comment
edenhaus
left a comment
There was a problem hiding this comment.
CI failing. Please have a look
Maybe because you specified both modes in one job.
|
Please take a look at the requested changes, and use the Ready for review button when you are done, thanks 👍 |
The venv the base job builds with uv points its shebang at uv's managed interpreter, which actions/setup-python never installs, so on a cache hit valgrind found a dangling interpreter and refused to run anything. Use the action every other venv job here uses. Drop the rich override with it. uv pip install ignores it, and that is what builds the venv, so it was doing nothing while reaching every resolution in the repo.
|
@edenhaus Not the two modes, it was the interpreter. Fixed, and dev merged in. The job set Python up with It now uses I also dropped the
../Frenck
Blogging my personal ramblings at frenck.dev |
There was a problem hiding this comment.
Pull request overview
Copilot reviewed 9 out of 9 changed files in this pull request and generated no new comments.
Suppressed comments (2)
benchmarks/conftest.py:25
- Stop Home Assistant during fixture teardown.
async_test_home_assistantdoes not stop the instance itself (the standard fixture does so explicitly), so every benchmark leaves its instance and resources alive and contaminates later timing and memory measurements.
async with async_test_home_assistant() as hass:
yield hass
benchmarks/test_core_states.py:43
- Prepopulate
light.benchmarkbefore invoking the benchmark. The fixture starts with an empty state machine, so the target's first invocation takes the create branch; pytest-codspeed's unmeasured warm-up varies by mode/runtime, making this supposedly update-only measurement mode-dependent.
counter = 0
uv pip install does honour it, and without it the venv build has no solution: surepy pins rich below 11, pytest-codspeed needs 13.8.1 up.
There was a problem hiding this comment.
Pull request overview
Copilot reviewed 9 out of 9 changed files in this pull request and generated no new comments.
Suppressed comments (3)
Previously missed (1) — in code that hasn't changed since the last review.
.core_files.yaml:16
- Handle benchmark-only changes explicitly in the
infojob.test_full_suitedefaults totrueand is only changed for integration updates (ci.yaml:173,186-215), so leaving this filter out ofanydoes not prevent a benchmark-only PR from scheduling the full test suite as the comment claims.
# Performance benchmark suite (CodSpeed). Only gates the benchmark job; kept out
# of the `any` aggregate below so it does not pull in the full test suite.
benchmarks: &benchmarks
benchmarks/conftest.py:25
- Stop each benchmark Home Assistant instance during fixture teardown.
async_test_home_assistantitself only restores test state (tests/common.py:365-372); the normalhassfixture separately callsasync_stop(tests/conftest.py:720-741). Without that step, every benchmark leaves a running instance inINSTANCES, retaining its event loop and resources for the rest of the simulation/memory run.
async with async_test_home_assistant() as hass:
yield hass
benchmarks/test_core_events.py:178
- Give this delayed callback the argument required by
async_call_later. The helper passes the UTC fire time to its action (homeassistant/helpers/event.py:1552-1560), so this zero-argument listener would raiseTypeErrorif the timer ran and could make the cancellation check pass without incrementingcalled.
@callback
def listener() -> None:
nonlocal called
called += 1
There was a problem hiding this comment.
🟡 Changes recommended
The workflow check currently fails, and the benchmark fixture does not configure Home Assistant’s event loop consistently with production.
Once you've addressed the issues Copilot identified, you can request another Copilot review.
Review details
- Files reviewed: 9/9 changed files
- Comments generated: 3
- Review effort level: Balanced
| from collections.abc import AsyncGenerator, Callable | ||
|
|
||
| import pytest | ||
|
|
||
| from homeassistant.core import HomeAssistant | ||
| from tests.common import async_test_home_assistant | ||
|
|
||
|
|
||
| @pytest.fixture | ||
| async def hass() -> AsyncGenerator[HomeAssistant]: | ||
| """Return a running Home Assistant instance for benchmarking. | ||
|
|
||
| Most hot paths under test (``async_fire``, ``async_set``, ``async_render``) | ||
| are ``@callback`` methods, so the benchmark fixture can drive them | ||
| synchronously from within the running loop. | ||
| """ | ||
| async with async_test_home_assistant() as hass: |
| pipdeptree==4.2.2 | ||
| pytest-asyncio==1.4.0 | ||
| pytest-aiohttp==1.1.1 | ||
| pytest-codspeed==5.0.3 |
Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
Proposed change
Adds CodSpeed performance benchmarking to track regressions on the core runtime hot paths, the busiest call sites in the whole process.
This first cut covers the core runtime:
The benchmarks live in a top-level
benchmarks/directory, outsidetestpaths, so the regular test suite never collects them. CodSpeed runs them withpytest benchmarks --codspeed.A few notes on the approach:
10/100/1000), so an algorithmic regression shows up as the curve bending instead of hiding behind a single number.benchmark.pedanticwith setup/teardown so each measured call stays on the branch its name claims (for example, create stays create instead of silently becoming update).Benchmarks run as a job in the existing CI workflow (
ci.yaml), reusing the same change detection and the prepared virtual environment as the test jobs, so no extra dependency install is needed. The job uses the CodSpeed v4 action insimulationmode (instrumented, machine-independent measurement) and authenticates through GitHub OIDC, so noCODSPEED_TOKENsecret is needed. It is gated through the existing.core_files.yamlfilters, so it only runs when core code or the benchmark suite itself changed, and is skipped on forks where OIDC is unavailable.Type of change
Additional information
Checklist
ruff format homeassistant tests)If user exposed functionality or configuration variables are added/changed:
If the code communicates with devices, web services, or third-party tools:
Updated and included derived files by running:
python3 -m script.hassfest.requirements_all.txt.Updated by running
python3 -m script.gen_requirements_all.To help with the load of incoming pull requests: