X9 folds -clean into its arm instead of dropping the earlier wave: an owed rerun of 1-7 kernels erased the other ~40 kernels of the arm in every figure (2026-09-18 rule: pool, latest run per kernel wins)

Author: Yakup Koray Budanaz

docs/DESIGN_data_collection_and_scoring.md

@@ -189,13 +189,11 @@ allowed under commit-single; more than one ACCEPTED submission is not.
   (`experiments.drop_cancelled_task_rows`, with a warning giving the count), the task row included:
   the job ended the agent mid-task (T6), so the rows report part of an episode and the token total
   prices part of one.
-- X9. An arm whose name ends in `-clean` is a re-run of one condition from scratch, launched after
-  something about the earlier wave was found wrong. It carries the SAME identity, so within an
-  identity group (`experiment`, `model`, `language`, `device`, `packet`, `harness`) a `task` row
-  whose arm carries the suffix drops every row of every arm in that group WITHOUT it, at read
-  (`experiments.drop_superseded_arm_rows`, with a warning giving the count and the number of arms).
-  The clean tasks supersede the earlier ones rather than pooling with them; the suffix names no
-  condition, and `arm` and `rep` are deliberately not in the group. The database is not modified (N1).
+- X9. An arm whose name ends in `-clean` (`CLEAN=1` waves and every owed rerun) is a re-run of the
+  arm without the suffix. At read the suffix comes off (`experiments.fold_clean_arms`) and nothing is
+  dropped: both waves' rows pool, and the latest run per kernel (R3) picks between them (2026-09-18
+  user rule). An owed rerun of a few kernels therefore replaces only those kernels. The database is
+  not modified (N1).
 
 ## 3. Per-task answer
 
@@ -444,7 +442,7 @@ family; the table states the pairs it kept.
 | X6 | `experiments.drop_foreign_kernel_rows`, called by `experiments.read_observations` | `test_experiments.py`: foreign-kernel rows dropped with a warning, runs without a task row kept |
 | X7 | `experiments.drop_pre_relaunch_rows`, called by `experiments.read_observations` | `test_experiments.py`: pre-final judge rows dropped with a warning, a task with no stamp untouched, task start over the kept rows |
 | X8 | `experiments.drop_cancelled_task_rows`, called by `experiments.read_observations` | `test_experiments.py`: every row of a cancelled task dropped with a warning, a frame without the column untouched |
-| X9 | `experiments.drop_superseded_arm_rows`, called by `experiments.read_observations`; the `-clean` suffix is written by `CLEAN=1` in `experiments/submit-cpf-llr40.sh` | `test_experiments.py`: superseded arms dropped with a warning, another identity group untouched, a frame with no clean arm untouched |
+| X9 | `experiments.fold_clean_arms`, called by `experiments.read_observations`; the `-clean` suffix is written by `CLEAN=1` and by `submit-owed-wave.sh` | `test_experiments.py`: clean arm folded with every row kept, a 1-kernel rerun keeps the other kernels and wins its own under `latest_runs`, a blank arm stays blank |
 | R1, R2 | `population.graded_episode_rows`, `last_per_episode` | `test_aggregation_population.py`: last submission, non-positive, suspect |
 | R3, R4 | `population.latest_runs`, `arm_kernel_answers`, `kernel_tokens` | rerun supersedes; rerun without answer; undated; start-time tie order |
 | R5 | `population.arm_kernel_answers`, `kernel_tokens(repeats="median")` | median run and carrier; token median |

hpcagent_bench/experiments.py

@@ -308,7 +308,7 @@ def read_observations(path: pathlib.Path) -> "pd.DataFrame":
         drop_foreign_kernel_rows,
         drop_pre_relaunch_rows,
         drop_cancelled_task_rows,
-        drop_superseded_arm_rows,
+        fold_clean_arms,
     ):
         frame = rule(frame)
     return frame
@@ -436,69 +436,21 @@ def fold_renamed_arms(frame: "pd.DataFrame") -> "pd.DataFrame":
     return frame.assign(arm=frame["arm"].astype(str).map(renamed_arm))
 
 
-#: What a launcher appends to re-run an arm from scratch (``CLEAN=1``). It names no condition: the
-#: identity columns are unchanged, and the clean tasks SUPERSEDE the ones before them.
+#: What a launcher appends to re-run an arm (``CLEAN=1``, and every owed rerun). It names no
+#: condition: the identity columns are unchanged.
 CLEAN_SUFFIX: str = "-clean"
 
-#: The identity a clean re-run supersedes within. Not ``arm`` -- the whole point is that the clean arm
-#: and the arm it replaces are two names for one condition -- and not ``rep``, since a designed repeat
-#: is a task of the same condition and is re-run with it.
-CLEAN_GROUP: tuple[str, ...] = ("experiment", "model", "language", "device", "packet", "harness")
 
-
-def clean_identity(frame: "pd.DataFrame") -> "pd.Series":
-    """One identity label per row for :func:`drop_superseded_arm_rows`.
-
-    THE RECORDED COLUMNS ARE NOT ENOUGH ON THEIR OWN. An extracted observations table carries
-    ``language``, ``packet`` and ``harness`` and no ``experiment``, ``model`` or ``device``, so a
-    group built from :data:`CLEAN_GROUP` alone puts every model's C control in one identity, and one
-    model's clean re-run then drops every other model's arm. That is what it did: six finished
-    GPT-OSS-120B arms took 18 arms with them, Qwen3.8-27B's and Kimi-K2.7-Code's included.
-
-    The arm NAME carries what the columns do not, so the label is the recorded columns plus the arm
-    with its ``-clean`` suffix removed. Two arms are then one identity exactly when they are the same
-    name re-run, which is what the suffix means. A name that differs by more than the suffix
-    supersedes nothing, and the failure direction is to keep both waves rather than to delete one.
-    """
-    columns = [column for column in CLEAN_GROUP if column in frame.columns]
-    # A missing cell reads as the empty string: a row with no arm or no recorded packet still needs
-    # one label, and pandas keeps NA through astype(str) and through the string accessors.
-    condition = frame["arm"].astype(str).fillna("").str.removesuffix(CLEAN_SUFFIX).fillna("")
-    labelled = frame.assign(clean_condition=condition)
-    return labelled[[*columns, "clean_condition"]].astype(str).fillna("").agg("\x1f".join, axis=1)
-
-
-def drop_superseded_arm_rows(frame: "pd.DataFrame") -> "pd.DataFrame":
-    """``frame`` without the rows of arms a ``-clean`` re-run superseded (spec X9).
-
-    A clean arm re-runs one condition from an empty workspace after something about the earlier wave
-    was found wrong -- a leaked view, a broken relaunch, a job that requeued onto its own rows. It
-    carries the SAME identity, so without this rule the two waves pool and the defect the re-run
-    exists to escape is averaged back in. Within one identity group (:func:`clean_identity`), a task
-    row whose arm carries the suffix therefore drops every row of every arm in that group without it.
-    The frame changes, never the database (N1), and the count is warned about.
-    """
-    import warnings
-
-    if frame.empty or "arm" not in frame.columns or "record" not in frame.columns:
-        return frame
-    groups = clean_identity(frame)
-    clean = frame["arm"].astype(str).str.endswith(CLEAN_SUFFIX)
-    superseding = set(groups[clean & (frame["record"] == "task")])
-    if not superseding:
+def fold_clean_arms(frame: "pd.DataFrame") -> "pd.DataFrame":
+    """``frame`` with every ``-clean`` arm under the arm it re-ran (spec X9). Nothing is dropped:
+    the waves pool and the latest run per kernel (``population.latest_runs``) picks between them
+    (2026-09-18 user rule). Dropping every earlier row on any clean task row cost whole arms: an
+    owed rerun of 1-7 kernels erased the ~40 kernels of the wave it topped up."""
+    if frame.empty or "arm" not in frame.columns:
         return frame
-    dropped = groups.isin(superseding) & ~clean
-    count = int(dropped.sum())
-    if count:
-        arms = sorted(set(frame.loc[dropped, "arm"].astype(str)))
-        message = f"dropped {count} row(s) of {len(arms)} arm(s) superseded by a clean re-run (spec X9)"
-        warnings.warn(message, stacklevel=2)
-    kept = frame[~dropped]
-    # The suffix names a WAVE, not a condition, so it comes off once the wave it replaces is gone.
-    # Left on, it renames the arm for everything downstream: a pair list, an --arms regex and a
-    # figure's arm pattern all ask for the condition by name and would find nothing.
-    renamed = kept["arm"].astype(str).str.removesuffix(CLEAN_SUFFIX)
-    return kept.assign(arm=renamed.where(renamed.notna(), kept["arm"]))
+    arms = frame["arm"]
+    folded = arms.astype(str).str.removesuffix(CLEAN_SUFFIX)
+    return frame.assign(arm=folded.where(arms.notna(), arms))
 
 
 def main(argv: list[str] | None = None) -> int:

tests/test_experiments.py

@@ -16,6 +16,7 @@
 import pytest
 
 from hpcagent_bench import experiments
+from hpcagent_bench.stats import population
 
 
 def test_a_blank_and_filled_arm_reads_as_one_identity() -> None:
@@ -178,37 +179,39 @@ def clean_frame() -> pd.DataFrame:
     )
 
 
-def test_an_arm_superseded_by_a_clean_rerun_is_dropped_with_a_warning() -> None:
-    """Spec X9: the clean arm re-ran the condition from scratch because the earlier wave was wrong,
-    and the two carry one identity -- pooled, the defect the re-run exists to escape is averaged
-    back in. What survives is reported under the CONDITION's name: the suffix named the wave."""
-    with pytest.warns(UserWarning, match="dropped 2 row"):
-        kept = experiments.drop_superseded_arm_rows(clean_frame())
-    assert set(kept.arm) == {"cpf-llr-focus40-qwen38-c-cpf", "cpf-llr-focus40-qwen38-c-cpfsrc"}
-    assert len(kept[kept.arm == "cpf-llr-focus40-qwen38-c-cpf"]) == 2
+def test_a_campaign_with_no_clean_arm_is_left_alone() -> None:
+    frame = clean_frame()
+    frame = frame[~frame.arm.str.endswith("-clean")]
+    assert experiments.fold_clean_arms(frame).equals(frame)
 
 
-def test_a_clean_rerun_supersedes_only_its_own_identity_group() -> None:
-    """The suffix says which TASKS are live for one condition, not that every other arm of the
-    campaign was re-run; a cpfsrc arm with no clean wave keeps every row."""
-    with pytest.warns(UserWarning, match="spec X9"):
-        kept = experiments.drop_superseded_arm_rows(clean_frame())
-    assert (kept.packet == "cpfsrc").sum() == 2
+def test_a_clean_rerun_folds_into_the_arm_it_re_ran_and_keeps_every_row() -> None:
+    """Spec X9 (2026-09-18 user rule): the suffix names a wave, not a condition, so the clean arm is
+    reported under the arm it re-ran and both waves' rows stay for the latest run to choose from."""
+    kept = experiments.fold_clean_arms(clean_frame())
+    assert len(kept) == len(clean_frame())
+    assert (kept.arm == "cpf-llr-focus40-qwen38-c-cpf").sum() == 4
+    assert (kept.arm == "cpf-llr-focus40-qwen38-c-cpfsrc").sum() == 2
 
 
-def test_a_campaign_with_no_clean_arm_is_left_alone() -> None:
-    """Every wave so far ran without the suffix, and X9 must be invisible to them."""
-    frame = clean_frame()
-    frame = frame[~frame.arm.str.endswith("-clean")]
-    assert len(experiments.drop_superseded_arm_rows(frame)) == len(frame)
+def test_a_one_kernel_owed_rerun_keeps_the_arms_other_kernels_and_wins_its_own() -> None:
+    """The bug: an owed rerun is named -clean and covers a few kernels, and X9 used to drop every row
+    of the wave it topped up -- 40 kernels became the rerun's one."""
+    common = {"record": "task", "run_root": "r", "language": "c", "packet": "", "harness": "claude"}
+    rows = [
+        {**common, "arm": "x-qwen38-c", "job": "1", "run_id": f"a{i}", "benchmark": f"k{i}", "ts_ms": 1}
+        for i in range(3)
+    ] + [{**common, "arm": "x-qwen38-c-clean", "job": "2", "run_id": "b0", "benchmark": "k0", "ts_ms": 2}]
+    latest = population.latest_runs(experiments.fold_clean_arms(pd.DataFrame(rows)))
+    assert sorted(latest.benchmark) == ["k0", "k1", "k2"]
+    assert latest.set_index("benchmark").loc["k0", "job"] == "2"
+    assert set(latest.arm) == {"x-qwen38-c"}
 
 
-def test_a_clean_arm_with_no_task_row_supersedes_nothing() -> None:
-    """A judge row alone does not say a re-run happened: the task row is what records that an agent
-    was launched under the clean arm, the same evidence X6-X8 read."""
-    frame = clean_frame()
-    frame = frame[(frame.record != "task") | (~frame.arm.str.endswith("-clean"))]
-    assert len(experiments.drop_superseded_arm_rows(frame)) == len(frame)
+def test_a_blank_arm_stays_blank_through_the_fold() -> None:
+    frame = pd.DataFrame({"arm": [math.nan, "x-c-clean"], "record": ["call", "task"]})
+    kept = experiments.fold_clean_arms(frame)
+    assert experiments.is_blank(kept.arm.iloc[0]) and kept.arm.iloc[1] == "x-c"
 
 
 @pytest.mark.parametrize(
@@ -287,56 +290,6 @@ def test_nan_is_blank_but_zero_is_not() -> None:
     assert not experiments.is_blank("0")
 
 
-def test_a_clean_rerun_of_one_model_does_not_supersede_another_model(tmp_path: pathlib.Path) -> None:
-    """Spec X9 groups by identity, and an extracted table records language, packet and harness but
-    not the model. Grouping on those alone put every model's C control in one identity, so six
-    finished GPT-OSS-120B re-runs deleted Qwen3.8-27B's and Kimi-K2.7-Code's arms as well."""
-    rows = [
-        {
-            "arm": "cpf-llr-focus40-oss120b-c-clean",
-            "record": "task",
-            "language": "c",
-            "packet": "",
-            "harness": "claude",
-        },
-        {"arm": "cpf-llr-focus40-oss120b-c", "record": "task", "language": "c", "packet": "", "harness": "claude"},
-        {"arm": "cpf-llr-focus40-qwen38-c", "record": "task", "language": "c", "packet": "", "harness": "claude"},
-        {"arm": "cpf-llr-focus40-kimi27sglang-c", "record": "task", "language": "c", "packet": "", "harness": "claude"},
-    ]
-    with pytest.warns(UserWarning, match="superseded by a clean re-run"):
-        kept = experiments.drop_superseded_arm_rows(pd.DataFrame(rows))
-    assert sorted(kept.arm) == [
-        "cpf-llr-focus40-kimi27sglang-c",
-        "cpf-llr-focus40-oss120b-c",
-        "cpf-llr-focus40-qwen38-c",
-    ]
-
-
-def test_a_clean_rerun_supersedes_the_arm_of_its_own_name(tmp_path: pathlib.Path) -> None:
-    """The suffix names no condition, so the clean wave replaces the wave it re-ran and nothing else."""
-    rows = [
-        {"arm": "x-qwen38-c-skills-clean", "record": "task", "language": "c", "packet": "lang-skills", "harness": "h"},
-        {"arm": "x-qwen38-c-skills", "record": "task", "language": "c", "packet": "lang-skills", "harness": "h"},
-        {"arm": "x-qwen38-c", "record": "task", "language": "c", "packet": "", "harness": "h"},
-    ]
-    with pytest.warns(UserWarning, match="superseded by a clean re-run"):
-        kept = experiments.drop_superseded_arm_rows(pd.DataFrame(rows))
-    assert sorted(kept.arm) == ["x-qwen38-c", "x-qwen38-c-skills"]
-
-
-def test_a_surviving_clean_arm_is_reported_under_the_condition_it_re_ran() -> None:
-    """The suffix names a wave, not a condition. Left on the arm, it renames the condition for every
-    consumer downstream: a pair list, an --arms regex and a figure's arm pattern all ask by name."""
-    rows = [
-        {"arm": "x-qwen38-c-clean", "record": "task", "language": "c", "packet": "", "harness": "h"},
-        {"arm": "x-qwen38-c", "record": "task", "language": "c", "packet": "", "harness": "h"},
-        {"arm": "x-oss120b-c", "record": "task", "language": "c", "packet": "", "harness": "h"},
-    ]
-    with pytest.warns(UserWarning, match="superseded by a clean re-run"):
-        kept = experiments.drop_superseded_arm_rows(pd.DataFrame(rows))
-    assert sorted(kept.arm) == ["x-oss120b-c", "x-qwen38-c"]
-
-
 def test_a_column_no_row_in_the_table_ever_recorded_still_fills_from_the_arm_name() -> None:
     """The bug this guards: a column NOTHING recorded reads back from CSV as all-NaN float64, and
     writing an arm's recovered language into that raised ``Invalid value 'c' for dtype 'float64'``