Skip to content

TESTING_STANDARDS.md

Latest commit

 

History

History
94 lines (65 loc) · 3.99 KB

TESTING_STANDARDS.md

File metadata and controls

94 lines (65 loc) · 3.99 KB
Raw

Testing Standards

Status: current

CI Timing Tolerances

🕒 Timing tests are inherently flaky in CI

Some tests run on shared CI runners where system load, CPU scheduling, and container overhead can add jitter. In this repository, we accept that timing-based tests can vary by at least ±250ms.

  • The canonical reference for this tolerance is tests/server/timing-delay.test.ts.
  • That test asserts that delay(1000) measurements are within 750ms–1250ms, which is the guardrail we use for all timing-based assertions.

✅ If you adjust timing tests, keep the ±250ms window in mind and ensure CI builds remain stable.

Unit-suite wall-clock budget

The unit budget is 55 seconds. The current baseline contains 2,115 tests; CI runtime depends on shared-runner load. Local HTTP listener tests run in the serial unit-node-http project while the remaining Node tests stay file-parallel. The budget is a wall-clock guard, not a per-test timeout.

Coverage-Hotspots

Folgende Dateien sind aktuell die größten Hotspots und sollten gezielt mit Tests abgedeckt werden:

Datei Aktuelle Coverage Ziel Verantwortung
client/src/hooks/use-compile-and-run.ts 85,5% >80% Compile→Start-Orchestrierung
server/routes/simulation.ws.ts ~55% >70% WebSocket-Lifecycle
server/services/arduino-compiler.ts ~60% >75% Compiler-Logik
server/services/execution-manager.ts ~45% >70% Sandbox-Ausführung
client/src/hooks/useArduinoSimulatorPage.tsx ~50% >75% ViewModel-Komposition

Aktueller Coverage-Hinweis: Die extrahierten Hooks liegen bei mindestens 94,0% Statement-Coverage; der Orchestrator liegt bei 85,5%. Die Größenabweichungen des UI-Adapters und des Orchestrators sind im Phase-2.1-Plan dokumentiert.

Hook-Namenskonvention

UnoSim verwendet camelCase für alle Hooks:

  • ✅ Korrekt: useCompileAndRun, useArduinoSimulatorPage
  • ❌ Falsch: use_compile_and_run, use-arduino-simulator-page

Diese Konvention gilt für alle neuen und bestehenden Hooks im Projekt.

Notes

Heavy stress tests

The multi-instance Arduino CLI stress test is disabled in the regular pipeline because it performs concurrent real compilations and is comparatively slow. Run it explicitly when validating runner isolation and cleanup:

RUN_HEAVY_TESTS=1 ./run-tests.sh

The test is an opt-in production-like check: it starts two real SandboxRunner instances concurrently, verifies that their serial output cannot cross over, and checks that temporary sketch directories are removed. It uses real timers because Docker/Arduino-CLI process events must continue while the test waits. The test has a 120-second per-test budget to accommodate two local compilations; this is a diagnostic gate, not a substitute for the fast deterministic tests.

Without RUN_HEAVY_TESTS=1 (or true), the regular pipeline deliberately skips this resource-intensive check and reports it as skipped. This keeps every local default run fast while still making the stronger isolation check reproducible on developer machines and in a dedicated CI job.

Sandbox contract gate

The current sandbox contract is defined in SECURITY.md. Changes to Docker sandbox flags, runner lifecycle, gateway assumptions or container resource limits must run the documented Phase-3.5 gates, including npm run test:security:inputs, npm run test:docker, RUN_HEAVY_TESTS=1 ./run-tests.sh, ./check-leaks.sh --cleanup, npm audit --omit=dev, npm run check, and npm run check:docs.

Sandbox contract tests must stay additive and must not use real secrets, weaken security defaults, or open public host bindings.

  • Do not use npx playwright test --update-snapshots or similarly destructive flags in CI; snapshot changes must be reviewed explicitly.
  • When modifying timing-sensitive tests, ensure they still pass on low-end CI hosts by running:
    1. npm run check
    2. npm run test:fast
    3. ./run-tests.sh