feat(model): add the GitHub Copilot CLI as a backend

[lufen]

Adds two backends. copilot_chat drives the Copilot CLI as a chat model and
can fill either role, so --backend copilot selects it for BOTH optimizer and
target — the CLI carries its own sign-in, which makes that the only fully local
configuration: a complete train/eval loop with no cloud API key.
copilot_exec is the separate target-only execution harness, alongside the
existing codex/claude/cursor harnesses.

Verified end to end on SearchQA with no credentials configured: baseline eval,
rollout, reflect, aggregate, select, update and gate all execute against the
local CLI.

Safety: chat calls disable built-in MCP servers and custom instructions so the
model sees only the prompt SkillOpt sends, and never pass --allow-all-tools.
Unlike the other exec harnesses, copilot_exec does NOT grant unattended tool
use by default — it requires an explicit copilot_exec_allow_all_tools
opt-in, because a file-edit rollout is the only case that needs it.

Two caveats worth knowing before use: the CLI is an agent rather than a
completions endpoint, so expect roughly 20–40 s per call; and it reports no
token counts, so usage totals are zero for these backends.

[Copilot]

Pull request overview

This PR adds support for running SkillOpt against the local GitHub Copilot CLI in two modes: copilot_chat (chat backend usable as optimizer and target for a fully local run) and copilot_exec (target-only exec harness). It also updates CLI/config wiring, documentation, and tests to cover the new backends and to ensure --backend correctly overrides role backends pinned by the base config.

Changes:

• Introduce copilot_chat backend that drives the copilot CLI as a chat model (optimizer/target), with MCP servers and custom instructions disabled.
• Add copilot_exec execution harness (target-only) with explicit opt-in gating for unattended tool use (--allow-all-tools).
• Extend config/CLI/docs/tests and adjust role-backend resolution so --backend is not silently ignored when base defaults pin roles.

Reviewed changes

Copilot reviewed 17 out of 17 changed files in this pull request and generated 2 comments.

Show a summary per file

File	Description
tests/test_role_backend_resolution.py	New regression tests for role-backend resolution behavior.
tests/test_copilot_exec_backend.py	Adds unit tests for Copilot backend normalization, configuration wiring, and exec/chat safety flags.
skillopt/model/copilot_backend.py	Implements the Copilot CLI chat backend and JSONL parsing.
skillopt/model/common.py	Registers Copilot backends in backend aliases/default-model map.
skillopt/model/codex_harness.py	Adds run_copilot_exec harness + dispatch in run_target_exec.
skillopt/model/backend_config.py	Adds Copilot-related env/config plumbing and backend whitelists.
skillopt/model/init.py	Wires Copilot backends into set_backend / chat routing.
skillopt/engine/trainer.py	Adds _resolve_role_backends and uses it during eval env construction.
skillopt/config.py	Adds config-flattening keys for Copilot settings.
scripts/train.py	Exposes Copilot backends and flags via legacy CLI args.
scripts/eval_only.py	Fixes --backend overriding behavior and adds Copilot CLI/config wiring.
README.md	Updates backend list to include Copilot backends.
docs/reference/config.md	Documents Copilot backend availability and config keys.
docs/reference/api.md	Documents Copilot backends in the public API reference tables.
docs/guide/configuration.md	Adds Copilot CLI backend explanation, env vars, and safety notes.
configs/base/default.yaml	Adds Copilot config fields and comments to base defaults.
CHANGELOG.md	Notes new Copilot backends and their behavior/safety properties.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[Copilot]

Pull request overview

Copilot reviewed 17 out of 17 changed files in this pull request and generated no new comments.

Suppressed comments (3)

skillopt/model/backend_config.py:115

• This return statement is now long enough to be hard to read and is inconsistent with the multi-line style used for other backend sets in this module. Consider formatting it as a multi-line set literal to keep it readable and reduce churn when adding/removing backends.

def is_target_chat_backend() -> bool:
    return TARGET_BACKEND in {"openai_chat", "claude_chat", "qwen_chat", "minimax_chat", "openai_compatible", "copilot_chat"}

skillopt/model/init.py:114

• get_backend_name() has a special-case for copilot_exec but not for the fully-local copilot_chat backend. When both roles are copilot_chat, this currently falls through to the generic "optimizer+target" string ("copilot_chat+copilot_chat"), which is inconsistent with the other unified backends and can confuse logs/telemetry that expect a canonical backend label.

    if optimizer == "openai_chat" and target == "copilot_exec":
        return "copilot_exec"
    if optimizer == "openai_compatible" and target == "openai_compatible":
        return "openai_compatible"
    return f"{optimizer}+{target}"

skillopt/model/backend_config.py:84

• This target-backend whitelist is now a very long single line, unlike the optimizer whitelist above, and is likely to violate line-length/style checks. Wrapping it like the optimizer whitelist keeps formatting consistent and easier to edit when adding more backends.

This issue also appears on line 114 of the same file.

    TARGET_BACKEND = normalize_backend_name(backend or "openai_chat")
    if TARGET_BACKEND not in {"openai_chat", "claude_chat", "qwen_chat", "minimax_chat", "openai_compatible", "copilot_chat", "codex_exec", "claude_code_exec", "cursor_exec", "copilot_exec"}:

[lufen]

Follow-up in 95d5643 addresses the re-review's suppressed suggestion: get_backend_name() now returns the canonical copilot_chat for a both-role fully-local run, consistent with claude_chat/qwen_chat, instead of the generic copilot_chat+copilot_chat. (The long set-literal style note is cosmetic; ruff is clean, so left as-is.) The inline --backend claude override bug was fixed in cb551d9.

[Copilot]

Pull request overview

Copilot reviewed 17 out of 17 changed files in this pull request and generated no new comments.

Suppressed comments (1)

skillopt/model/backend_config.py:35

• COPILOT_EXEC_ALLOW_ALL_TOOLS is read directly from the environment, but get_copilot_exec_config() only accepts values '0' or '1'. If a user sets the env var to a common boolean string like 'true'/'false', SkillOpt will raise ValueError at runtime. Consider normalizing the env var on load the same way other boolean-ish flags are parsed so 'true' becomes '1' and 'false' becomes '0'.

COPILOT_EXEC_ALLOW_ALL_TOOLS = os.environ.get("COPILOT_EXEC_ALLOW_ALL_TOOLS", "0")

[lufen]

Second re-review follow-up in 9ec24fc: COPILOT_EXEC_ALLOW_ALL_TOOLS was read raw at module load, so setting it to true/false in the environment (without calling configure_copilot_exec) made get_copilot_exec_config() raise ValueError. It's now normalized through the existing _parse_bool helper to 0/1 (unknown values fall back to the safe 0), consistent with the other boolean-ish exec flags. Regression added; suites green.

[Copilot]

Pull request overview

Copilot reviewed 17 out of 17 changed files in this pull request and generated no new comments.

Suppressed comments (1)

skillopt/model/codex_harness.py:1407

• When the Copilot CLI emits no assistant messages (empty/invalid JSONL), the loop records each attempt’s stdout/stderr in all_raw, but the final RuntimeError(last_error) drops that context. Unlike other exec harnesses (e.g., cursor/codex) this makes a “no response” failure hard to debug because callers get neither persisted artifacts nor any CLI output.

    combined = "\n\n".join(all_raw)
    raise RuntimeError(last_error)

[lufen]

Third re-review follow-up in bd8e786: the final failure path in run_copilot_exec computed combined from all_raw and then discarded it, raising a bare Copilot CLI returned no response. Unlike the cursor/codex harnesses, copilot_exec persists no artifacts, so an empty or invalid JSONL stream left the caller with no output to debug. The error now carries a bounded (4000-char) tail of the captured stdout/stderr, with a regression covering it.

Suites green (509 passed).

[Copilot]

Pull request overview

Copilot reviewed 17 out of 17 changed files in this pull request and generated no new comments.

Suppressed comments (2)

skillopt/model/common.py:28

• default_model_for_backend() returns an empty string for copilot_chat/copilot_exec. This propagates into the CLI entrypoints (e.g. scripts/eval_only.py uses default_model_for_backend(backend) as a fallback for optimizer/target deployments), and can leave deployments empty (notably for --backend copilot_exec, where the optimizer is still openai_chat). Returning a real default here (or omitting these keys to fall back to the Azure/OpenAI default) avoids accidentally configuring an empty deployment.

    "copilot_exec": "",
    "copilot_chat": "",

skillopt/model/codex_harness.py:1384

• On subprocess.TimeoutExpired, this captures only exc.stdout and drops exc.stderr. The other exec harnesses include stderr in their raw capture, and it’s important for debugging when Copilot emits errors to stderr before timing out.

        except subprocess.TimeoutExpired as exc:
            raw = exc.stdout or ""
            if isinstance(raw, bytes):
                raw = raw.decode("utf-8", "replace")
            all_raw.append(f"===== COPILOT CLI ATTEMPT {attempt + 1} =====\n{raw}")
            raise

[lufen]

Fourth re-review follow-up in 9f43455 — both suggestions were correct, and the first turned out to be a real bug:

Empty optimizer deployment. _BACKEND_DEFAULT_MODELS mapped copilot_chat/copilot_exec to "". That table also feeds the shared deployment fallback in the entry points — cfg.get("optimizer_model", default_model_for_backend(backend)) — and the shipped configs/_base_/default.yaml sets no optimizer_model/target_model keys, so the fallback really is reached. That meant --backend copilot_exec configured an empty optimizer deployment, even though that role is still a real openai_chat model. Dropped the two entries so they fall back to the Azure default; the CLI's own model is unaffected because it comes from copilot_chat_optimizer_model / copilot_chat_target_model.

Timeout stderr. run_copilot_exec captured only exc.stdout on TimeoutExpired. The codex and cursor harnesses both fold stderr into their raw capture, and Copilot can report the cause there before timing out — now captured the same way.

Regressions added for both; suites green (510 passed).

[Copilot]

Pull request overview

Copilot reviewed 17 out of 17 changed files in this pull request and generated no new comments.

Suppressed comments (1)

skillopt/model/codex_harness.py:1391

• In run_copilot_exec, the TimeoutExpired handler spends effort normalizing/combining stdout+stderr and appending to all_raw, but then immediately re-raises. Since all_raw is never returned/persisted on this path, the combined raw is discarded and the comment about “captur(ing) it” is misleading. Either persist/propagate the combined output, or (simpler) remove the dead code and just re-raise like the other exec harnesses.

        except subprocess.TimeoutExpired as exc:
            raw = exc.stdout or ""
            if isinstance(raw, bytes):
                raw = raw.decode("utf-8", "replace")
            # Copilot can report the cause on stderr before timing out; the
            # other exec harnesses capture it, so this must too.
            err = exc.stderr or ""

[lufen]

Fifth re-review follow-up in 494b2d4: correct — the TimeoutExpired handler normalized stdout/stderr into all_raw and then re-raised, but all_raw is local and never returned or persisted on that path. The codex/cursor harnesses do the same normalization because they persist artifacts before raising; copilot_exec has no such step, so the work was dead and my comment about "capturing" it was misleading. Since TimeoutExpired already carries .stdout/.stderr to the caller, the handler now simply re-raises. Suites green (510 passed).

[Copilot]

Pull request overview

Copilot reviewed 17 out of 17 changed files in this pull request and generated no new comments.

Suppressed comments (1)

configs/base/default.yaml:33

• configs/_base_/default.yaml pins copilot_exec_allow_all_tools: false, and both trainer.py and scripts/eval_only.py unconditionally pass that value into configure_copilot_exec(...). This overwrites COPILOT_EXEC_ALLOW_ALL_TOOLS in os.environ on every run, so the documented env-var opt-in (COPILOT_EXEC_ALLOW_ALL_TOOLS=1) cannot take effect unless the config also flips the YAML key. Using null here would preserve the safe default (env default is still false) while allowing an explicit env opt-in or per-run config override.

  copilot_exec_allow_all_tools: false  # copilot_exec only; required for file-edit rollouts

[lufen]

Sixth re-review follow-up in f07022b — good catch, and it silently defeated the safety opt-in documented in this PR.

configs/_base_/default.yaml pinned copilot_exec_allow_all_tools: false, and both trainer.py and scripts/eval_only.py pass cfg.get("copilot_exec_allow_all_tools") straight into configure_copilot_exec(). Because False is not None, it took the assignment branch and overwrote COPILOT_EXEC_ALLOW_ALL_TOOLS in os.environ on every run — so the documented COPILOT_EXEC_ALLOW_ALL_TOOLS=1 opt-in could never take effect.

Changed to null, matching the blank-uses-env convention of the neighbouring copilot_exec_path / copilot_exec_home keys:

• safe default unchanged (the env default is off, and copilot_exec still never passes --allow-all-tools unless both the opt-in and file-edit rollouts are in play);
• an explicit --copilot_exec_allow_all_tools still wins;
• the env opt-in now actually works.

Regression added asserting the base config leaves it null and that None is a no-op. Suites green (511 passed).

[Copilot]

Pull request overview

Copilot reviewed 17 out of 17 changed files in this pull request and generated no new comments.

Suppressed comments (1)

skillopt/model/init.py:256

• The NotImplementedError message is now outdated: copilot_chat is a supported chat backend, but it isn’t listed in the chat_target supported-backends message. This can confuse users when they hit this error with an unsupported backend.

            "chat_target is only supported with target_backend=openai_chat, claude_chat, qwen_chat, minimax_chat, "

[lufen]

Follow-up in a94d115: copilot_chat is a supported chat target but was missing from the NotImplementedError supported-backends list in chat_target — and from the matching message in chat_target_messages, which had the same omission. Both now list it. Suites green (511 passed).

[Copilot]

Pull request overview

Copilot reviewed 17 out of 17 changed files in this pull request and generated no new comments.

Suppressed comments (2)

skillopt/model/copilot_backend.py:216

• This backend records into a TokenTracker but does not expose the standard get_token_summary()/reset_token_tracker() helpers that the rest of the backends provide. That makes copilot_chat usage hard to report/reset consistently and blocks wiring it into skillopt.model.get_token_summary().

def chat_target_messages(
    messages: list[dict[str, Any]],
    max_completion_tokens: int = 16384,
    retries: int = 5,
    stage: str = "target",
    reasoning_effort: str | None = None,
    timeout: float | None = None,
    **_ignored: Any,
) -> tuple[str, dict[str, int]]:
    del max_completion_tokens, reasoning_effort
    config = get_copilot_chat_config()
    return _chat_impl(
        _messages_to_prompt(messages),
        retries,
        stage,
        model=str(config.get("target_model") or ""),
        timeout=timeout,
    )

skillopt/model/init.py:13

• copilot_chat is now a supported backend (and copilot_backend records TokenTracker data), but skillopt.model.get_token_summary() / reset_token_tracker() still aggregate only OpenAI/Claude/Qwen/MiniMax/OpenAI-compatible/Codex. As a result, runs using copilot_chat will omit call counts from the per-step token snapshots in trainer.py and from the overall token summary.

from skillopt.model import azure_openai as _openai
from skillopt.model import claude_backend as _claude
from skillopt.model import codex_backend as _codex
from skillopt.model import copilot_backend as _copilot
from skillopt.model import minimax_backend as _minimax
from skillopt.model import openai_compatible_backend as _openai_compat
from skillopt.model import qwen_backend as _qwen

[lufen]

Follow-up in 9eb7739 — both suggestions were correct and are fixed together, since they were the same gap seen from two sides.

copilot_backend recorded into a TokenTracker but exposed neither get_token_summary() nor reset_token_tracker(), so model.get_token_summary() skipped it entirely and copilot_chat runs omitted their call counts from the per-step token snapshots in trainer.py and from the run summary. Both helpers are now present and wired into the model-level aggregate and reset.

To be clear about the caveat in the PR description: the CLI still reports no token counts, so the token fields remain zero — but the call counts are real and now surface like every other backend.

One thing worth flagging: wiring this in surfaced a latent test-pollution bug. The copilot tests record calls, and once the tracker fed the aggregate, that leftover state leaked into test_combined_token_summary_counts_each_backend_once (it passed in isolation but failed in a full run). Fixed at the source by resetting the tracker in the copilot test fixture rather than loosening the shared assertion.

Suites green (512 passed), and verified order-independent.

[Copilot]

Pull request overview

Copilot reviewed 17 out of 17 changed files in this pull request and generated no new comments.

[Yif-Yang]

Thanks for the careful follow-ups and for the end-to-end SearchQA validation. The Copilot CLI direction is valuable, but the current head still has a few cross-backend contract and safety blockers, so I think we should hold the merge for now:

1. chat_optimizer_messages() / chat_target_messages() expose the shared tools, tool_choice, and return_message contract, but the Copilot implementation silently absorbs them through **_ignored and always returns a string. This breaks callers that legitimately rely on that contract: OfficeQA requests return_message=True and accesses .content, while SpreadsheetBench relies on structured tool_calls. Please implement compatible message/tool-request semantics, or fail fast for combinations that Copilot cannot support rather than silently ignoring them. Contract tests using the same call shapes as those environments would be especially useful.
2. The CLI tool boundary is not yet authoritative. --disable-builtin-mcps disables the GitHub MCP server, and --no-custom-instructions disables instruction loading, but neither disables the built-in read/shell/write tools. The child process also inherits COPILOT_ALLOW_ALL; when it is true, Copilot treats it as --allow-all-tools, bypassing the intended opt-in in this PR. Please sanitize that environment variable unless the SkillOpt opt-in is explicitly enabled, and use a verified no-built-in-tools configuration for copilot_chat. The configured copilot_exec_allow_all_tools value should remain authoritative even when the parent environment has COPILOT_ALLOW_ALL=true.
3. _resolve_role_backends() returns early whenever both role values are truthy and they are not both defaults. That mishandles partial overrides. For example, an explicit optimizer plus the base-config target_backend=openai_chat under backend=copilot_exec leaves the target on OpenAI instead of selecting copilot_exec. Please resolve each role independently and add partial-override tests for the backend mappings.

Two documentation details should also be corrected while updating this:

• The CLI is locally invoked and needs no separate provider API key, but normal GitHub Copilot inference is still a cloud service, so “fully local” is misleading. “CLI-authenticated” or “no separate API key” would be accurate.
• The example uses skillopt-train --cfg; the supported option is --config.

Once these are addressed, we will be happy to re-review. Thank you again for the thorough iteration on this integration.