Reduce Test Time by Scienfitz · Pull Request #922 · emdgroup/baybe

Scienfitz · 2026-09-24T18:50:26Z

aim was to reduce test time via removing redundancies and better parametrizations without loosing much covreage
a large speedup also comes via pytest_xdist running tests in parallel
explanations in each commit

Old

New

Copilot

Copilot review overview

🔵 Needs a closer look

Apply FAST_OPTIMIZATION_SETTINGS to the remaining default optimizer path in the cardinality constraint test.

Review effort: Lite
Findings: None

What changed in this PR

This pull request reduces test and example runtime through parallel execution, fixture caching, lighter workloads, and improved parametrization.

Changes:

Adds and configures pytest-xdist.
Caches reusable fixtures and removes redundant test cases.
Reduces smoke-test workloads and documents testing conventions.

File	Summary
`uv.lock`	Locks xdist dependencies.
`tox.ini`	Enables parallel test execution.
`tests/test_substance_parameter.py`	Reduces test workload.
`tests/test_pending_experiments.py`	Reduces test workload.
`tests/test_partial_measurements.py`	Reduces test workload.
`tests/test_iterations.py`	Deduplicates cases and adds targeted coverage.
`tests/test_input_output.py`	Reduces redundant test configurations.
`tests/test_imports.py`	Consolidates import checks.
`tests/test_campaign.py`	Reduces iterative test batch sizes.
`tests/insights/test_shap.py`	Stabilizes parametrization.
`tests/docs/test_examples.py`	Controls simulation parallelism.
`tests/constraints/test_constraints_continuous.py`	Reduces constraint test workloads.
`tests/constraints/test_cardinality_constraint_continuous.py`	Reduces workloads; one optimizer path still needs the fast settings.
`tests/conftest.py`	Adds fixture caching and fast optimization settings.
`tests/AGENTS.md`	Documents testing conventions.
`pyproject.toml`	Adds the xdist test dependency.
`examples/Multi_Armed_Bandit/bernoulli_multi_armed_bandit.py`	Reduces smoke-test workload.
`examples/Constraints_Discrete/selection_constraints.py`	Reduces smoke-test workload.
`examples/Constraints_Discrete/prodsum_constraints.py`	Reduces smoke-test workload.
`examples/Constraints_Discrete/dependency_constraints.py`	Reduces smoke-test workload.
`examples/Constraints_Discrete/custom_constraints.py`	Reduces smoke-test workload.
`examples/Constraints_Continuous/hybrid_space.py`	Reduces smoke-test workload.
`examples/Backtesting/impute_mode.py`	Reduces smoke-test workload.
`examples/Backtesting/full_initial_data.py`	Reduces smoke-test workload.
`examples/AGENTS.md`	Documents example testing conventions.
`CONTRIBUTING.md`	Documents xdist usage.
`AGENTS.md`	Documents parallel test execution.

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

AVHopp

Mostly minor -only thing to maybe discuss is the import check

AdrianSosic

Awesome 🤩 Absolutely overdue, thanks for taking care of it! The next step after these mechanical steps is to restructure our test suite more cleverly – I bet we can achieve the same speedup once again. But that is a task for the cold winter days ❄️ ☃️

AdrianSosic · 2026-10-08T11:47:26Z

        "PYTHONPATH": os.getcwd(),  # to ensure examples are found
        "MPLBACKEND": "Agg",  # avoid popups resulting from plt.show()
+        "BAYBE_PARALLELIZE_SIMULATION_RUNS": str(
+            example in _PARALLEL_SIMULATION_EXAMPLES


question: what about smoke test settings? would it make sense to disable parallelization there as well, for the same reason?

not entirely sure how you mean this

this check here just picks oen of all the eample so execue the mc simulations in parallel. That adds 1-2s to the (complete total aggregated) testing time due to the parallel threads loading (indepenent of smoke testing)

all other tests are always exectued in serial monte carlo fashion, also independent of smoe testing

can you elaborate which of these branches you think should be affected in what manner by smoke testing?

I meant: should the BAYBE_PARALLELIZE_SIMULATION_RUNS variable perhaps also be set to False whenever SMOKE_TEST is active (because starting the parallel pool is not necessary when there are almost no tasks to be performed in parallel). So basically something like

BAYBE_PARALLELIZE_SIMULATION_RUNS": str( example in _PARALLEL_SIMULATION_EXAMPLES and not "SMOKE_TEST" in os.environ )

or whatever more suitable

The inequality constraint was built with the equality rhs (4.2) while the assertion checks against the inequality rhs (1.3), so the assertion verified a weaker condition than the one imposed. The constraint now uses the intended inequality rhs, aligning construction and assertion.

Two tests run with a batch size of 1 but carried the ID 'b3', which is misleading for '-k' selection and duration reports.

The optimum reference line hard-coded a batch size of 3, which is wrong whenever the example runs with a different batch size (e.g. in smoke mode). Additionally, the plots were saved to the working directory even in smoke mode. The line now uses BATCH_SIZE and saving is skipped under SMOKE_TEST, consistent with other examples.

Test execution in tox (and thus CI) is now distributed across all available cores via pytest-xdist, without changing any test logic. The 'worksteal' distribution handles the very heterogeneous test durations well. A local trial on a representative subset showed a 2.8x wall-time reduction with 4 workers (the size of the GitHub runners) with identical outcomes and coverage. The flag is only set in tox so that plain local pytest runs remain serial and easy to debug.

The SHAP explainer and plot types are stored in sets, whose iteration order depends on the per-process string hash seed. Parametrizing directly over them made the test collection order non-deterministic across processes, which pytest-xdist rejects because all workers must collect identical test lists. Sorting makes the collection deterministic. A comparison of full-suite collections under different hash seeds confirmed no other affected tests.

Pytest rebuilds function-scoped fixtures for every parametrized test case, even if the varied parameters do not affect the fixture. As a consequence, identically configured campaigns were repeatedly iterated, e.g. 25 unique configurations were iterated 260 times in the SHAP tests alone. Iterated campaigns are now cached per session, keyed by the serialized initial campaign together with the number of iterations and the batch size. The key thus captures every aspect that can make campaigns differ, ruling out false cache hits. Each test receives a deep copy, so tests remain isolated even when mutating the campaign. Key creation and copying take a few milliseconds compared to several hundred milliseconds per iteration run. As a trade-off, tests with identical configuration now share the same random measurements instead of each drawing their own. This reduced the serial runtime of the SHAP tests from 115s to 66s.

Two sampling strategies without a hybrid sampler, (None, 0.0) and (None, 1.0), are identical to the default BotorchRecommender because the sampling percentage is ignored in that case. Likewise, the naive hybrid recommender with a BotorchRecommender for the discrete part equals the default naive hybrid recommender. Both configurations were already tested, so the duplicates, which were hidden behind numeric ID suffixes, are removed. This reduces test_recommenders_hybrid from 60 to 48 cases (~21s).

Each of the 48 base kernel/prior combinations was tested both on its own and wrapped in a ScaleKernel with a fixed outputscale prior. Wrapping only varies the inner prior again, which is already covered by the unwrapped base kernels (and the kernel/prior translation is additionally covered by the hypothesis test in test_kernels.py). Now, only one base kernel per kernel class is wrapped, which still exercises the ScaleKernel for every kernel type and reduces test_kernels from 206 to 126 cases (~40s).

The true-positive check for the lazy-import whitelists spawned a separate Python subprocess for every (target, module) pair, paying ~2.5s of interpreter startup and imports each time. Since several modules are whitelisted for multiple targets, the whitelist is now inverted and each module is imported once in a single subprocess that asserts the presence of all its whitelisted targets. The assertions are unchanged, but the number of subprocesses drops from 29 to 15 (~35s).

Most tests verify that the recommendation machinery works end-to-end, not the quality of the found optimum. The default recommender fixtures and the recommender lists in test_iterations.py now use 2 restarts and 16 raw samples instead of the library defaults (10 and 64). The optimization regime itself (multi-start gradient-based optimization, batch size, search space type, sequential vs. joint optimization) is unchanged. This roughly halves the runtime of continuous and hybrid test cases, e.g. from 6.4s to 3.6s for the default hybrid BotorchRecommender. Examples keep the defaults since they serve as documentation.

Most tests using batch sizes above 1 only require a multi-point batch, not a specific size. Since the cost of batch recommendation grows with the batch size, the central batch_size fixture and such tests now use a batch size of 2, which keeps them in the batch regime. Tests where the actual size matters (e.g. batches exceeding a subset, or enough points for a statistical effect) remain unchanged. For continuous cardinality constraints, the random recommender keeps its larger batch since it samples the inactive parameters per point at negligible cost, while the Botorch recommender uses a batch of 2 since the number of optimized subsets does not depend on the batch size. The polytope sampling test draws 5 instead of 20 points, still covering several of its 11 cardinality patterns. To retain coverage of hybrid sampling where the batch exceeds the number of sampled discrete candidates (previously implicitly covered by the batch size of 3), an explicit test case is added.

The test verifies that a task parameter override survives a fit/recommend loop. Since the local ongoing_campaign fixture already provides measurements, a single iteration performs a complete surrogate fit and recommendation with the override in place. The runtime is dominated by the multi-task GP fit, so the second iteration only added cost without additional coverage.

Smoke-test values should only reduce cost while preserving the regime that is exercised in smoke mode. Accordingly: * The continuous hybrid space example, previously the slowest test (~32s) without any smoke-mode handling, now uses a batch size of 2 under SMOKE_TEST (~15s). * The discrete constraint examples use a batch size of 2 under SMOKE_TEST, and the selection and custom constraint examples additionally reduce their iterations to 2. Their assertions validate the search space, not the batch. * The multi-armed bandit example runs 10 instead of 50 iterations under SMOKE_TEST, which still covers the transition from initial to Bayesian recommendations. Examples whose batch size is essential to the narrative or implementation (e.g. the 'plate of four' in the interpoint example) remain unchanged.

In smoke mode, starting the parallel simulation worker pool (one process per core, each importing torch) costs more than the few simulation runs themselves, adding 3-8s per simulation example. Under pytest-xdist, these pools additionally compete with the test workers for cores. Examples are now run with parallel simulation disabled via the environment, without modifying the example scripts. Since no other test exercises the parallel simulation path, one example keeps it enabled.

Reflect the new test conventions in the agent guidelines: batch size 2 as the multi-point case, deterministic and non-redundant parametrization, the ongoing_campaign session cache, reduced optimization effort, parallel-safe tests, regime-preserving smoke values and guarded file outputs in examples.

Checking stdout failed for modules printing on import and dropped the exit-code assertion, contrary to what the commit introducing it stated. The import check helper now accepts several targets, restoring the exit-code mechanism with one subprocess per module.

The assertion was applied to a generator object, which is always truthy, so misspelled whitelist entries went unnoticed.

Scienfitz self-assigned this Sep 24, 2026

Scienfitz force-pushed the repo/reduce_test_time branch from f53a4c1 to bee559e Compare September 25, 2026 11:39

Scienfitz marked this pull request as ready for review September 25, 2026 11:55

Scienfitz requested review from AVHopp and AdrianSosic as code owners September 25, 2026 11:55

Copilot AI lite review requested due to automatic review settings September 25, 2026 11:55

Copilot started reviewing on behalf of Scienfitz September 25, 2026 11:56 View session

Copilot AI reviewed Sep 25, 2026

View reviewed changes

AVHopp approved these changes Oct 5, 2026

View reviewed changes

Scienfitz force-pushed the repo/reduce_test_time branch from 51385cd to e6eb2a1 Compare October 6, 2026 10:48

AdrianSosic approved these changes Oct 8, 2026

View reviewed changes

Scienfitz added 19 commits October 8, 2026 17:18

Fix mislabeled batch size test IDs

5588598

Two tests run with a batch size of 1 but carried the ID 'b3', which is misleading for '-k' selection and duration reports.

Run two iterations in bandit example smoke test

6e2c675

Avoid rebinding module-level names in import tests

7b38172

Simplify sorted non-SHAP explainer parametrization

9d45b00

Scienfitz added 3 commits October 8, 2026 17:18

Use fast optimization settings in cardinality constraint test

ac46cfc

Fix whitelist existence check in lazy loading test

f74f59a

The assertion was applied to a generator object, which is always truthy, so misspelled whitelist entries went unnoticed.

Report offending modules in import test failures

3e10a8d

Scienfitz force-pushed the repo/reduce_test_time branch from e6eb2a1 to 3e10a8d Compare October 8, 2026 15:18

Navigation Menu

Repository navigation

Reduce Test Time - #922

Reduce Test Time#922
Scienfitz wants to merge 22 commits into
mainfrom
repo/reduce_test_time

Scienfitz commented Sep 24, 2026 •

edited

Loading

Uh oh!

Copilot AI left a comment

Uh oh!

AVHopp left a comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

AdrianSosic left a comment

Uh oh!

AdrianSosic Oct 8, 2026

Uh oh!

Scienfitz Oct 8, 2026 •

edited

Loading

Uh oh!

AdrianSosic Oct 9, 2026

Uh oh!

This branch has not been deployed

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

Conversation

Scienfitz commented Sep 24, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Copilot review overview

🔵 Needs a closer look

Uh oh!

AVHopp left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

AdrianSosic left a comment

Choose a reason for hiding this comment

Uh oh!

AdrianSosic Oct 8, 2026

Choose a reason for hiding this comment

Uh oh!

Scienfitz Oct 8, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

AdrianSosic Oct 9, 2026

Choose a reason for hiding this comment

Uh oh!

This branch has not been deployed

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

Scienfitz commented Sep 24, 2026 •

edited

Loading

Scienfitz Oct 8, 2026 •

edited

Loading