fix(libero): fail loudly on tasks with empty init-state sets

[akushonkamen]

Fixes #246

Description

make_env() in robots/libero/env_server.py:115-117 indexed straight into get_task_init_states(...) with no guard for tasks whose pruned_init file ships zero states:

• seed % len(...) degenerated to seed % 0, surfacing only as a context-free ZeroDivisionError at env_server.py:117;
• aggregation paths that tolerate the crash kept going and silently shrank the evaluation denominator (100 → 90/80), making results incomparable.

Fix:

• Guard make_env(): when the requested task has 0 init states, raise a RuntimeError naming the suite, the requested task id, and every empty task in the suite (best-effort scan via get_num_tasks(); API signature verified against RLinf/LIBERO-PRO@rpent, liberopro/liberopro/benchmark/__init__.py:269-295).
• Document the three currently known empty pruned_init tasks (libero_10_task#2, libero_spatial_task#3/#7) plus the HF-snapshot restore procedure in pro_hybrid_guide.md §2.3.
• Add 3 offline fake-suite unit tests (2 for the guard, 1 healthy-task rid-math regression) in tests/unit_tests/robots/libero/test_env_server_init_states.py.

Data side (BDDL consistency check + pruned_init regeneration) belongs to the upstream LIBERO-PRO simulation pipeline per the issue triage and was not done locally; a timeout 60 probe found no LIBERO entries under ~/.cache/huggingface, so real-dataset reproduction was not executed here (the issue-sanctioned fake-suite path covers the guard).

Testing

• Red (before fix): venvs/rlinf1662/bin/python -m pytest tests/unit_tests/robots/libero/test_env_server_init_states.py -q → 2 failed (test_make_env_fails_loudly_on_empty_init_states, test_make_env_error_lists_every_empty_task_in_suite, both ZeroDivisionError at env_server.py:117), 1 passed.
• Green (after fix): timeout 300 venvs/rlinf1662/bin/python -m pytest tests/unit_tests -k "env_server or init_state" -q --continue-on-collection-errors → 4 passed, 1 skipped (the flag is needed because of a pre-existing, unrelated collection error in tests/unit_tests/rpent/planner/test_codex_contracts.py / openai_codex). Re-confirmed right before submission with the same numbers.
• Regression: timeout 590 venvs/rlinf1662/bin/python -m pytest tests/unit_tests -q --continue-on-collection-errors → 14 failed, 1147 passed, 7 skipped.
• Baseline diff (stashed fix, clean main + new test file): 16 failed = the same 14 pre-existing failures + the 2 guard tests red → zero new failures introduced by the fix.
• Lint: timeout 120 ruff check robots/libero → All checks passed!; ruff format --check robots/libero → 18 files already formatted; tests/unit_tests/robots/libero passes both as well.
• End-to-end: /tmp/issue246-repro/repro_zero_states.py → RuntimeError listing suite, requested task, and the full empty-task inventory for the suite.

Manual verification

Not needed — offline tests fully cover the change. The guard's runtime message was exercised end-to-end via the fake-suite repro script above (no GPU/simulator/model service involved). Real-dataset reproduction was not possible locally (no HF LIBERO cache; see Description).

Checklist

• My code follows the code style of this project.
• I have updated related documentation when needed.
• I have added tests to cover my changes when needed.
• All new and existing tests passed. (14 pre-existing failures on main are unchanged by this PR; see baseline diff above.)

AI disclosure: this change was prepared with AI assistance; the contributor has reviewed the full diff and run all listed checks.

[chatgpt-codex-connector]

Codex Review Summary

This comment shows the latest Codex review activity on this pull request.

Review	Status	Commit	Review trigger
📝 Code Review	⚠️ Failed 2026-10-11T15:19:43.411929Z	8ad9992	PR opened

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

[wilburx813]

Thanks for the contribution! The clearer error message is helpful, but I don’t think this fully fixes #246, since the affected tasks still cannot be evaluated.

Could you investigate why these init-state files are empty and fix the bug in https://github.com/RLinf/LIBERO-PRO/tree/rpent? Ideally, we should restore valid initial states for these tasks and verify that they can be reset successfully.

[akushonkamen]

Investigation of why the init-state files are empty (your question, @wilburx813) — I traced the full chain from the shipped .pruned_init bytes back to the generation script in LIBERO-PRO@rpent. Short version: this is a silent data-release defect in the LIBERO-PRO generation pipeline, not something the RPent harness can repair — and this PR's guard is what turns that defect from silent result-shrinkage into a loud, actionable error.

1. What is actually inside the "empty" files

Mapping task IDs to files (libero_suite_task_map.py: {base}_task suites share the base suite's task list; task_order_index=0 is the identity permutation) and inspecting the pickles:

• liberopro/liberopro/init_files/libero_10_task/KITCHEN_SCENE3_turn_on_the_stove_and_put_the_moka_pot_on_it.pruned_init (= libero_10_task#2)
• liberopro/liberopro/init_files/libero_spatial_task/pick_up_the_black_bowl_on_the_cookie_box_and_place_it_on_the_plate.pruned_init (= libero_spatial_task#3)
• liberopro/liberopro/init_files/libero_spatial_task/pick_up_the_black_bowl_on_the_stove_and_place_it_on_the_plate.pruned_init (= libero_spatial_task#7)

each is a torch-saved archive whose archive/data.pkl is 147 bytes and deserializes to numpy.core.multiarray._reconstruct with shape (0,), dtype float64 — a zero-length ndarray. Every healthy sibling in the same suites is 18952–49352 B with shape (50, 47): 50 states x 47-dim sim state, matching the generator's default num_inits=50. So the file exists, parses fine, and simply contains zero states.

2. The generation-side cause (LIBERO-PRO@rpent)

All .pruned_init files are produced by notebooks/generate_init_states.py:

• it loops num_inits=50 times per BDDL, creating a fresh OffScreenRenderEnv(**env_args) and calling env.get_sim_state() each iteration (lines 49–63);
• every attempt is wrapped in except Exception as e: print(...) (line 64) — failures are swallowed and the loop continues;
• afterwards all_initial_states = np.array(all_initial_states) (line 76) — a totally-failed task yields np.array([]), i.e. exactly the shape-(0,) float64 ndarray found in the shipped files;
• it then writes the archive unconditionally (zipf.writestr("archive/data.pkl", ...), lines 76–79), prints 成功保存 0 个状态 (line 81), and exits 0.

So for these three BDDLs, all 50 env-construction/state-capture attempts raised (scene/object/goal instantiation failing inside robosuite for the perturbed P1 "task" BDDLs — e.g. libero_10_task's KITCHEN_SCENE3 file perturbs the language to "put the pan on it" while the scene carries both a chefmate_8_frypan_1 and a moka_pot_1 and goals reference flat_stove_1_cook_region), the notebook discarded every exception, and the zero-state array was shipped in the asset release with no failure signal. Note these are exactly the three P1 task-perturbation tasks known from #246 — consistent with a perturbed-BDDL/scene incompatibility at generation time, though pinning each BDDL's exact sim error requires running the simulator (not possible in this environment; the same constraint the PR body documents).

3. Why the harness surfaced it as ZeroDivisionError, and why it can't fix it

Downstream, liberopro/liberopro/benchmark/__init__.py:144-153 maps every task to <task>.pruned_init and get_task_init_states() (lines 294–301) torch.loads it without any non-empty check, so the zero-length array flows through cleanly until robots/libero/env_server.py:115-117 computes rid = first_id + (seed % trials) with trials = len(...) = 0 → context-free ZeroDivisionError, and aggregation paths that tolerated the crash silently shrank the evaluation denominator (the 100 → 90/80 incomparability from #246).

A harness fix cannot restore states that were never generated — only the LIBERO-PRO simulation pipeline can regenerate them (make generate_init_states.py fail loudly when a task ends with 0 states instead of writing and exiting 0, fix or re-generate the affected BDDL/scene combinations, re-run the 50-state generation, and re-release the pruned_init assets). What the harness can and should do is exactly what this PR does: refuse to evaluate a task with an empty init-state set, naming the suite, the requested task and every empty task in the suite — that guard is what makes the data defect diagnosable and prevents silently incomparable numbers in the meantime. The PR also documents the HF-snapshot restore procedure for the affected assets (pro_hybrid_guide.md §2.3), which is the immediate operational mitigation until upstream re-releases.

Happy to hand the generation-side findings above to the LIBERO-PRO maintainers, and to verify resets end-to-end once regenerated assets land (needs the sim/GPU stack).

[wilburx813]

Thanks for the investigation! I’m moving this PR to draft for now, pending a fix for the underlying issue.