Skip standalone inner-fold evaluation for default-only models

README.md

@@ -574,6 +574,9 @@ budget rationale.
 **What a run records.** A model keeps one result per trial, including its configuration,
 metrics, optional predictions, and phase timings. A system records one final result with its test
 metrics, optional predictions, and total runtime; it does not synthesize model-trial fields.
+Leaderboard training time includes only the refit for runs with one default config;
+otherwise it includes the selected trial's inner fit plus refit, not the full tuning sweep.
+Raw inner-fold timings are retained in both cases.
 
 </details>
 

docs/tuning-regime.md

@@ -140,20 +140,31 @@ hours (`rdblearn`, `tabpfn-rel`, RelGNN, `relgt`):_
 
 | Model | `n_trials` | mean | min | p25 | p50 | p75 | max |
 | --- | --- | --- | --- | --- | --- | --- | --- |
-| `constant-global` | 0 | 0.1 s | 0.0 s | — | 0.0 s | — | 0.3 s |
-| `constant-per-entity` | 0 | 0.2 s | 0.0 s | — | 0.1 s | — | 1.3 s |
-| `lightgbm` | 30 | 14 min | 0.1 min | 0.2 min | 2 min | 28 min | 53 min |
-| `graphsage` | 4 | 47 min | 2 min | 9 min | 35 min | 83 min | 141 min |
-| `rdblearn` | 6 | 11 min | 0.5 min | 2 min | 7 min | 11 min | 50 min |
-| `tabpfn-rel-local` | 3 | 12 min | 0.6 min | 0.8 min | 4 min | 6 min | 85 min |
-| `tabpfn-rel-client` | 3 | 76 min | 18 min | 72 min | 88 min | 94 min | 108 min |
-| `relgnn-es` (RelGNN) | 10 | 73 min | 1 min | 6 min | 28 min | 99 min | 336 min |
-| `relgt` | 9 | 511 min | 40 min | 175 min | 273 min | 466 min | 2442 min |
-| `rt-plurel` | — | 351 min | 109 min | 129 min | 231 min | 516 min | 979 min |
-
-Total runtime is per (dataset, task): all four model phase-time columns summed
-over every trial and refit, or the system's `time_total`, with the distribution
-taken over the 21 RelBench v1 tasks.
+| `constant-global` | 0 | 0.030 s | 0.000 s | 0.001 s | 0.003 s | 0.038 s | 0.193 s |
+| `constant-per-entity` | 0 | 0.114 s | 0.001 s | 0.002 s | 0.008 s | 0.149 s | 0.668 s |
+| `lightgbm` | 30 | 13.5 min | 0.1 min | 0.2 min | 1.5 min | 27.4 min | 52.5 min |
+| `graphsage` | 4 | 44.4 min | 1.9 min | 9.1 min | 34.4 min | 75.5 min | 130.8 min |
+| `rdblearn` | 6 | 14.4 min | 0.4 min | 1.9 min | 10.2 min | 15.5 min | 73.7 min |
+| `tabpfn-rel-local` | 3 | 11.7 min | 0.6 min | 0.8 min | 3.4 min | 5.0 min | 84.9 min |
+| `tabpfn-rel-client` | 3 | 73.4 min | 15.2 min | 63.3 min | 87.5 min | 93.2 min | 99.8 min |
+| `relgnn-es` (RelGNN) | 10 | 68.1 min | 1.4 min | 5.4 min | 26.6 min | 92.1 min | 313.4 min |
+| `relgt` | 9 | 430.9 min | 36.1 min | 124.1 min | 149.8 min | 463.8 min | 2233.0 min |
+| `rt-plurel` | — | 351.0 min | 109.2 min | 129.1 min | 231.4 min | 515.5 min | 979.1 min |
+
+Total runtime is per (model, dataset, task, seed). For tuned model runs, it sums
+inner fit and prediction time across all trials, plus the selected config's outer
+fit and prediction time. A run with exactly one config tagged `default` counts
+only its outer fit and prediction time. Any extra outer evaluation of an
+unselected default config is excluded. Systems use their recorded `time_total`.
+The table reports the distribution across the 21 RelBench v1 tasks at seed 0;
+raw per-trial timings in `results.csv` are unchanged.
+
+The leaderboard's `time_train_s` is a different measure: it includes only the
+refit for a run with exactly one config tagged `default`. For other model runs,
+it includes the selected trial's inner fit plus refit, not the full tuning
+sweep. Raw inner-fold timings remain in the result rows. This rule applies per
+(model, dataset, task, seed), so a task-dependent grid can use different
+accounting across tasks. `time_infer_s` is the final prediction time.
 
 The budget is set in exactly one place — `--n-trials` on the CLI (default 10),
 or `PredictiveContext.fit(n_trials=...)`. Nothing derives it from the search

packages/relarena-core/pyproject.toml

@@ -4,7 +4,7 @@ build-backend = "uv_build"
 
 [project]
 name = "relarena-core"
-version = "0.0.3"
+version = "0.0.4"
 description = "Shared contracts, temporal validation and predictive queries for relational models."
 readme = "README.md"
 license = "Apache-2.0"

packages/relarena-core/src/relarena_core/__init__.py

@@ -16,7 +16,7 @@
 from relarena_core.system import RelArenaSystem
 from relarena_core.tuner import tune
 
-__version__ = "0.0.3"
+__version__ = "0.0.4"
 __all__ = [
     "CacheConfig",
     "CacheMiss",

packages/relarena-core/src/relarena_core/userdb/query.py

@@ -104,6 +104,7 @@ def fit(
         For a sampled space, `n_trials` requests that many random samples in
         addition to the default; for a fixed grid, it caps the ordered grid.
         With `n_trials == 0`, skip tuning and fit the model's default config.
+        A plan containing only the default config also skips tuning.
 
         `cache_dir` is a local directory that caches DFS features across tuning,
         the final fit, and later `predict`, useful for repeated runs on a large
@@ -122,10 +123,23 @@ def fit(
                 "caller-selected timestamp."
             )
         search_space = registry.search_space(model)
+        if callable(search_space):
+            stats = TaskStats(
+                num_train_nodes=len(self._source.inner_split().train_table.df)
+            )
+            search_space = resolve_search_space(search_space, stats)
+        grid = search_space.fixed_grid
+        default_only = (
+            n_trials == 0
+            or not search_space.is_tunable
+            or (
+                grid is not None and grid[:n_trials] == [search_space.default_overrides]
+            )
+        )
 
         # fill: on a custom DB the store starts empty, so build it as we go (the
         # tuning trials + refit then reuse it); a later run reads what this built.
-        if n_trials > 0:
+        if n_trials > 0 and not default_only:
             trials = run_tuning(
                 model_cls,
                 search_space,
@@ -138,15 +152,8 @@ def fit(
             )
             config = select_best(trials, self._source.metric).config
         else:
-            # Resolve a factory search space (e.g. relgt builds its grid from
-            # TaskStats) before reading its defaults; a plain SearchSpace is
-            # returned unchanged.
             trials = None
-            stats = TaskStats(
-                num_train_nodes=len(self._source.inner_split().train_table.df)
-            )
-            space = resolve_search_space(search_space, stats)
-            config = dict(space.default_overrides)
+            config = dict(search_space.default_overrides)
         fitted = model_cls(
             config,
             cache=cache,

packages/relarena-core/tests/test_standalone_runtime.py

@@ -5,6 +5,7 @@
 import numpy as np
 import pandas as pd
 import pytest
+from ConfigSpace import ConfigurationSpace, Integer
 from relbench.base import Database, EntityTask, Table
 
 from relarena_core import RelArenaModel, registry
@@ -14,8 +15,24 @@
 
 
 @pytest.mark.parametrize("refit_full", [False, True])
+@pytest.mark.parametrize(
+    "mode",
+    [
+        "tuned",
+        "default",
+        "single-grid",
+        "zero-budget",
+        "capped",
+        "sampled",
+        "capped-default",
+    ],
+)
 def test_tuning_final_fit_and_original_ids(
-    tmp_path: Path, monkeypatch: pytest.MonkeyPatch, refit_full: bool
+    tmp_path: Path,
+    monkeypatch: pytest.MonkeyPatch,
+    refit_full: bool,
+    mode: str,
+    caplog: pytest.LogCaptureFixture,
 ) -> None:
     customers = pd.DataFrame({"customer_id": ["a", "b", "c", "d"]})
     dates = pd.date_range("2004-01-15", "2005-06-15", freq="30D")
@@ -68,7 +85,7 @@ def fit(
             seed: int,
             time_limit: float | None = None,
         ) -> None:
-            if self.config["fail"]:
+            if self.config.get("fail", False):
                 raise ValueError("Deliberate failed tuning candidate")
             self.mean = float(train_table.df[task.target_col].mean())
             fits.append(
@@ -84,21 +101,59 @@ def predict(self, task: EntityTask, db: Database, table: Table) -> np.ndarray:
 
     monkeypatch.setattr(query_module, "discover_models", lambda **kwargs: None)
     monkeypatch.setattr(registry, "_entries", {})
-    registry.register(
-        SuppliedModel,
-        SearchSpace(
-            default_overrides={"fail": False},
-            fixed_grid=[{"fail": True}, {"fail": False}],
-        ),
+    space = SearchSpace(
+        default_overrides={"fail": False},
+        fixed_grid=[{"fail": True}, {"fail": False}],
     )
+    if mode == "default":
+        space = SearchSpace(default_overrides={"fail": False})
+    elif mode == "single-grid":
+        space = SearchSpace(
+            default_overrides={"fail": False}, fixed_grid=[{"fail": False}]
+        )
+    elif mode == "capped-default":
+        space.fixed_grid = [{"fail": False}, {"fail": True}]
+    elif mode == "capped":
+        space.fixed_grid.append({"fail": False, "x": 1})
+    elif mode == "sampled":
+        space = SearchSpace(
+            default_overrides={"fail": False},
+            space=ConfigurationSpace(space=[Integer("x", (1, 10))]),
+        )
+    plan_calls = []
+    original_configs = SearchSpace.configs
+
+    def configs(self: SearchSpace, n_trials: int, seed: int) -> list[dict]:
+        plan_calls.append((n_trials, seed))
+        return original_configs(self, n_trials, seed)
+
+    monkeypatch.setattr(SearchSpace, "configs", configs)
+    registry.register(SuppliedModel, space)
     spec = PredictiveQuerySpec.from_yaml(tmp_path / "task.yaml", data_dir=tmp_path)
     query = PredictiveContext(spec, data_version="fixture-v1")
     inner, outer = query._source.inner_split(), query._source.outer_split()
-    fitted = query.fit(SuppliedModel.name, n_trials=2)
+    if mode not in {"tuned", "capped", "sampled"}:
+
+        def unexpected_inner() -> None:
+            pytest.fail("Default-only fitting must not construct the inner split")
+
+        monkeypatch.setattr(query._source, "inner_split", unexpected_inner)
+    budget = {"zero-budget": 0, "capped-default": 1}.get(mode, 2)
+    fitted = query.fit(SuppliedModel.name, n_trials=budget)
     assert fitted.config == {"fail": False}
-    assert [trial.ok for trial in fitted.trials] == [False, True]
-    assert fits[0][0] == len(inner.train_table.df)
-    assert fits[0][2] <= inner.cutoff
+    if mode in {"tuned", "capped", "sampled"}:
+        assert len(plan_calls) == 1
+        assert [trial.ok for trial in fitted.trials] == (
+            [True, True, True] if mode == "sampled" else [False, True]
+        )
+        assert fits[0][0] == len(inner.train_table.df)
+        assert fits[0][2] <= inner.cutoff
+    else:
+        assert plan_calls == []
+        assert fitted.trials is None
+        assert len(fits) == 1
+    if mode == "capped":
+        assert sum("dropping 1" in record.message for record in caplog.records) == 1
     expected_rows = len(outer.train_table.df)
     if refit_full:
         expected_rows += len(outer.val_table.df)

packages/relarena/pyproject.toml

@@ -4,7 +4,7 @@ build-backend = "uv_build"
 
 [project]
 name = "relarena"
-version = "0.1.1"
+version = "0.1.2"
 authors = [{ name = "Prior Labs" }]
 license = "Apache-2.0"
 license-files = ["LICENSE", "NOTICE", "src/relarena/models/VENDORED-LICENSES"]
@@ -27,7 +27,7 @@ classifiers = [
     "Topic :: Scientific/Engineering :: Artificial Intelligence",
 ]
 dependencies = [
-    "relarena-core==0.0.3",
+    "relarena-core==0.0.4",
     # Pinned exactly: RelBench ships the dataset/task SHA256 checksums in the package
     # itself (relbench/datasets/hashes.json, relbench/tasks/hashes.json, wired into the
     # pooch registries in relbench/datasets/__init__.py and relbench/tasks/__init__.py),
@@ -76,7 +76,7 @@ kurversc = ["kurversc==0.1.1"]
 # plus the DFS featurization deps.
 # setuptools<82 because featuretools -> woodwork imports pkg_resources at import
 # time, and setuptools 82 removed it (81 still ships it).
-rdblearn = ["tabpfn>=8", "relarena-core[dfs]==0.0.3"]
+rdblearn = ["tabpfn>=8", "relarena-core[dfs]==0.0.4"]
 # rdl: the shared Relational Deep Learning stack for the GNN baselines — PyG + PyTorch
 # Frame + a text embedder, which relbench.modeling.* needs (all lazy-imported inside
 # fit()). GPU recommended. NOTE: PyG neighbor sampling needs `pyg-lib` or `torch-sparse`

packages/relarena/src/relarena/__init__.py

@@ -72,4 +72,4 @@
     "database_checksum",
     "split_checksums",
 ]
-__version__ = "0.1.1"
+__version__ = "0.1.2"

packages/relarena/src/relarena/evaluation/leaderboard.py

@@ -39,15 +39,27 @@ def to_bencheval_frame(results: pd.DataFrame) -> pd.DataFrame:
     - `metric_error` ← `to_metric_error(test_score, <primary metric>)`, using
       the per-row `metric` column (always a registered primary, never the
       auxiliary native-metric columns)
-    - `time_train_s` ← model fit times, or a system's complete `time_total`
+    - `time_train_s` ← refit time for a single default config; otherwise the
+      selected trial's inner fit plus refit, or a system's complete `time_total`
     - `time_infer_s` ← `predict_time_refit`
 
+    Pass every trial, including failed and unselected trials, so a single default
+    config can be identified per (model, dataset, task, seed). Model training time
+    is not the sum over the tuning sweep. Inner timings remain in the raw results.
+
     Only the val-selected config per run contributes (one row per method/task);
     runs without a test score (failed or test-skipped) are dropped. The caller
     must still ensure the method × task matrix is dense before ranking —
     `bencheval` rejects a sparse matrix.
     """
-    rows = results
+    rows = results.reset_index(drop=True)
+    # Identify default-only runs before filtering out unselected trials.
+    default_only = pd.Series(False, index=rows.index)
+    if {"seed", "config_tag"}.issubset(rows.columns):
+        run_size = rows.groupby(["model", "dataset", "task", "seed"], dropna=False)[
+            "config_tag"
+        ].transform("size")
+        default_only = run_size.eq(1) & rows["config_tag"].eq("default")
     if "selected" in rows.columns:
         rows = rows[rows["selected"]]
     rows = rows[rows["test_score"].notna()]
@@ -62,6 +74,8 @@ def seconds(column: str) -> pd.Series:
             return pd.Series(0.0, index=rows.index)
         return rows[column].fillna(0.0)
 
+    tuning_time = seconds("fit_time_tuning").mask(default_only.loc[rows.index], 0.0)
+
     frame = pd.DataFrame(
         {
             "method": rows["model"].to_numpy(),
@@ -70,9 +84,7 @@ def seconds(column: str) -> pd.Series:
             ).to_numpy(),
             "metric_error": metric_error,
             "time_train_s": (
-                seconds("fit_time_tuning")
-                + seconds("fit_time_refit")
-                + seconds("time_total")
+                tuning_time + seconds("fit_time_refit") + seconds("time_total")
             ).to_numpy(),
             "time_infer_s": seconds("predict_time_refit").to_numpy(),
         }

packages/relarena/tests/evaluation/test_leaderboard.py

@@ -327,3 +327,59 @@ def test__compute_leaderboard__kinds_filter__excludes_systems() -> None:
 
     assert "rt-plurel" in set(combined.index)
     assert set(models_only.index) == {"constant-global", "lightgbm"}
+
+
+def test_runtime_infers_default_only_before_filtering_trials() -> None:
+    default = {
+        **_result("model", "dataset", "task", "roc_auc", 0.8),
+        "seed": 0,
+        "config_tag": "default",
+    }
+    legacy = pd.DataFrame([default])
+    other_seed = pd.DataFrame(
+        [
+            {**default, "seed": 1},
+            {
+                **default,
+                "seed": 1,
+                "config_tag": "r1",
+                "selected": False,
+                "test_score": float("nan"),
+                "error": "failed candidate",
+            },
+        ]
+    )
+    system = pd.DataFrame(
+        [
+            {
+                "model": "system",
+                "dataset": "dataset",
+                "task": "task",
+                "metric": "roc_auc",
+                "selected": True,
+                "test_score": 0.8,
+                "time_total": 12.0,
+            }
+        ]
+    )
+    results = pd.concat([legacy, other_seed, system])
+    original = results.copy(deep=True)
+
+    frame = to_bencheval_frame(results)
+
+    assert frame.time_train_s.tolist() == [0.5, 1.5, 12.0]
+    assert frame.time_infer_s.tolist() == [0.2, 0.2, 0.0]
+    pd.testing.assert_frame_equal(results, original)
+
+
+def test_historical_default_only_results_exclude_inner_fit_time() -> None:
+    results = pd.read_csv(_BASELINE_DIR / "results.csv")
+    constants = results[results.model.eq("constant-global")]
+    assert not constants.empty
+    assert constants.config_tag.eq("default").all()
+    assert constants.fit_time_tuning.gt(0).all()
+
+    frame = to_bencheval_frame(results)
+    reported = frame[frame.method.eq("constant-global")]
+
+    assert reported.time_train_s.tolist() == constants.fit_time_refit.tolist()