[API] Add programmable experiment execution and results

[podkidyshev]

Summary

• Add cloudai.api.run_experiment, validate_scenario, list_experiments, and get_experiment. Accept TOML text or configuration file paths and return the existing experiment model.
• Support synchronous execution and a separate background worker with wait=False. Persist results, configuration supplied as text, and background controller logs under a unique experiment directory.
• Keep shared execution in cloudai.handlers; move argument adapters, signal registration, console listing, and bulk config verification into the CLI. Preserve public legacy CLI handler interfaces with deprecation warnings.
• Raise parsing and submission errors to API callers, and select single-sbatch runners per invocation without changing the global registry. Document the API in a dedicated user guide.

Test Plan

• Automated CI.
• Manual macOS standalone test: ran a one-second Sleep scenario through the API synchronously and in the background. Both completed with one run and saved timestamps. Validation, listing, and result retrieval succeeded; the caller's configured logging and signal handlers stayed intact.
• Launched another background experiment from a short-lived Python caller. After that caller exited, a separate process read its completed result.
• Ran the same local Sleep configuration through cloudai run; it exited successfully and reported the case as passed.
• Loaded existing NCCL normal and DSE cluster artifacts locally through get_experiment: both retained their completed status and 36 case metrics; the DSE result retained its recommendation. No new cluster jobs were submitted.

Manual smoke output:

PASS: synchronous and background Sleep, validation, listing, retrieval, caller logging/signals unchanged
Detached worker after caller exit: completed
Read saved EOS artifact: final-nccl-run_2026-09-21_07-55-44 completed tests= 1 metrics= 36 DSE= False
Read saved EOS artifact: final-nccl-dse_2026-09-21_07-55-45 completed tests= 1 metrics= 36 DSE= True

Additional Notes

Second PR in the programmable API series, based on #1046. That PR contains the mechanical handler-module move.

For scenario/system inputs, str means TOML content and Path means a file. For get_experiment, both types mean a result directory or JSON file path. Background workers load installed plugins in a fresh process; in-memory registrations in the caller are not transferred.

[coderabbitai]

Important

Draft PR not reviewed

Draft PRs are not automatically reviewed by default.

• Trigger a manual review

To automatically review draft PRs, update your CodeRabbit configuration:

reviews:
  auto_review:
    drafts: true

---

_{Comment @coderabbitai help to get the list of available commands.}