feat(systems): allow a sub-second monitor_interval

[rutayan-nv]

Summary

Job completion can be checked more often than once a second.

flowchart TB
  subgraph Before["Before: whole seconds only"]
    direction LR
    A["a 2 second job finishes"] --> B["up to 1 second spent<br/>waiting to notice"]
  end
  subgraph After["After: fractions allowed"]
    direction LR
    C["a 2 second job finishes"] --> D["noticed in milliseconds"]
  end
  Before ~~~ After

Loading

How often cloudai checks whether a job has finished could only be set in whole seconds. For trials lasting a couple of seconds, up to a full second of each one was spent waiting to notice it was already done.

monitor_interval is the sleep between job-completion polls, typed int, so the floor was 1 s. Too coarse for a fast backend: a proxy-model trial finishes in ~25 ms, so the poll was ~2.1 s of a 2.15 s per-trial cost. 0 is not the answer — completion is a syscall now, so 0 spins a core. Widened on base System and every subclass that overrides it, plus the slurm_rest_client parameter. time.sleep already takes a float.

Test Plan

0.005 and 0.5 accepted alongside 1 and 60. Revert the field to int → the two fractional cases fail and the integers still pass, which is the evidence existing configs are unaffected.

ruff, pyright clean. Full suite 1979 passed, 5 skipped.

Additional Notes

No polling logic changed, no defaults changed — slurm/lsf stay 60, standalone/kubernetes stay 1.

Widened everywhere rather than just standalone: a float subclass over an int base is legal but leaves the base annotation wrong.

[coderabbitai]

Navigate logical layers of code changes, visualize relationships, and explore their blast radius.

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info

⚙️ Run configuration

Configuration used: Repository: NVIDIA/cloudai/.coderabbit.yaml

Review profile: ASSERTIVE

Plan: Enterprise

Run ID: 3df74273-bc0a-44f0-aefc-64eb77874f45

📥 Commits

Reviewing files that changed from the base of the PR and between 76f1937 and 76f1937.

📒 Files selected for processing (2)

• src/cloudai/systems/slurm/slurm_rest_client.py
• src/cloudai/systems/slurm/slurm_system.py

Included review availability: Your plan provides up to 12 included reviews per hour; 9 remain after this review.

---

📝 Walkthrough

Walkthrough

The PR changes monitor interval declarations from integers to floating-point values across system implementations and Slurm submission. Existing numeric defaults remain unchanged. A standalone system test checks fractional and integer intervals.

Changes

Monitor interval support

Layer / File(s)	Summary
Floating-point interval contracts and test coverage src/cloudai/_core/system.py, src/cloudai/systems/kubernetes/kubernetes_system.py, src/cloudai/systems/lsf/lsf_system.py, src/cloudai/systems/runai/runai_system.py, src/cloudai/systems/slurm/slurm_system.py, src/cloudai/systems/slurm/slurm_rest_client.py, src/cloudai/systems/standalone/standalone_system.py, tests/systems/standalone/test_system.py	Monitor interval fields and the submit_sbatch parameter now use float annotations with numerically unchanged defaults. Parameterized coverage verifies that standalone systems preserve fractional and integer interval values.

Priority: ⬇️ Low

Estimated code review effort: 1 (Trivial) | ~5 minutes

Change: Feature

Merge Risk: ⚪ Minimal · up to 76f19

Fractional intervals pass through the inspected polling paths without truncation, and the defaults remain unchanged. No actionable merge risk was identified.

🚥 Pre-merge checks | ✅ 4

✅ Passed checks (4 passed)

Check name	Status	Explanation
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Title check	✅ Passed	The title clearly describes the main change: allowing sub-second monitor intervals.
Description check	✅ Passed	The description directly explains the float type changes, unchanged defaults, affected systems, and test coverage.

✨ Finishing Touches 💡 1

🛠️ Fix failing CI checks 💡

• Commit to this branch
• Create a new PR

📝 Generate docstrings

• Commit to this branch
• Create a new PR

🧪 Generate unit tests (beta)

• Commit to this branch
• Create a new PR

---

_{Comment @coderabbitai help to get the list of available commands.}