Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
16 commits
Select commit Hold shift + click to select a range
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
58 changes: 58 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -4,6 +4,64 @@ All notable Phonton CLI release changes should be documented here.

This project follows pre-1.0 SemVer: minor versions may still include breaking changes while the public API and CLI surface settle.

## 0.25.0 - Current local models, local first, comparable proof

### Changed

- [changed] Offline npm preparation accepts lockfiles of up to 2,048 package
entries (was 512) within the existing 2 MiB lockfile bound, enough for a
jest and babel toolchain.

- [changed] `phonton models catalog` lists 2026 local coding models and
marks one by the machine's total memory (the largest GPU, or half of RAM
when that is larger): `qwen3.5:4b` from 4 GB, `qwen3.5:9b` from 6 GB,
`gpt-oss:20b` from 16 GB, `qwen3.6:35b-a3b` from 24 GB, with `gemma4:12b`
and `qwen3.6:27b` as alternatives. Under 4 GB or without a memory reading
nothing is suggested, and `doctor` and `models` point to the catalog
instead of a download. The 2024 `qwen2.5-coder` entries are
gone. Free memory no longer decides the suggestion; the fit column still
reports it.
- [changed] With no provider key anywhere, Phonton defaults to a local model
instead of Anthropic. `phonton doctor` and `phonton models` name the
`phonton models setup MODEL` command for this machine first, then the
provider keys.

### Fixed

- [fixed] `phonton models setup --help` (and `--help` after any other models
command) printed nothing useful and started downloading the 1.5 GB runtime,
taking `--help` as a model name. It now prints the models usage.
- [fixed] Local fit estimates reserved 1 GiB per 4096 context tokens plus 25%
of the weights, so `qwen3.5:4b` could not be calibrated above 4096 tokens on
a 6 GB GPU that serves it at 16384. Estimates now read the KV cache size from
the installed model's attention metadata (32 KiB per token for `qwen3.5`),
add a fixed 1 GiB runtime reserve, and accept a model that fits the largest
GPU plus available RAM as partly offloaded, so the 6 GB tier's `qwen3.5:9b`
is admitted once about 4 GiB of RAM is free. A copy of the same model already
loaded at another context no longer hides its own memory from calibration
or goal admission.
- [fixed] `phonton review` and `phonton why-tokens` listed only the retrieved
snippets (often five unrelated ones, ~500 tokens) as model context. They now
list the files each prompt inlined whole or as excerpts, then the retrieved
snippets, with a per-prompt token estimate next to the provider-reported
input for all calls.

### Added

- [added] `benchmarks/polyglot`: an Aider Polyglot slice runner that scores
Phonton, the bare model and OpenCode on the same exercises, prompt and
local model, reporting pass rate, false-pass rate, tokens and wall time.
- [added] Local goal runs can verify jest projects. Select
`node node_modules/jest/bin/jest.js --ci --json` as the check: each check
copy installs the project's own jest with the reviewed offline `npm ci`
(lockfile integrity, no lifecycle scripts), and a pass needs jest's JSON
report to show at least one passing test in candidate files, no failure, and
V8 coverage of every edited source. Unsupported jest flags, a missing
`--json`, or skipped-only runs stay Not run. Babel configuration is now
protected from model edits like `jest.config.*`. A root `"test": "jest"`
script (literal test paths only, jest declared as a dependency) now
proposes this check and its preparation without `--check`.

## 0.24.1 - Goals that name files get those files

Found by running `phonton goal` on a real JavaScript repository with a cloud
Expand Down
2 changes: 1 addition & 1 deletion Cargo.lock

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

20 changes: 17 additions & 3 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -13,7 +13,7 @@
<p align="center">
<a href="https://github.com/phonton-dev/phonton-cli/actions/workflows/ci.yml"><img alt="CI Status" src="https://github.com/phonton-dev/phonton-cli/actions/workflows/ci.yml/badge.svg"></a>
<a href="https://github.com/phonton-dev/phonton-cli/stargazers"><img alt="GitHub stars" src="https://img.shields.io/github/stars/phonton-dev/phonton-cli?style=flat&label=stars&color=ff69b4"></a>
<img alt="release" src="https://img.shields.io/badge/release-v0.24.1--beta-6c63ff">
<img alt="release" src="https://img.shields.io/badge/release-v0.25.0--beta-6c63ff">
<img alt="license" src="https://img.shields.io/badge/license-MIT%20OR%20Apache--2.0-blue">
</p>

Expand Down Expand Up @@ -90,8 +90,8 @@ asks it only for edits in a format it handled during calibration, runs your
checks on a copy, and lands nothing until you apply the result.

```bash
phonton models catalog # models that fit this machine
phonton models setup qwen2.5-coder:3b # runtime, download,
phonton models catalog # marks the model for this machine
phonton models setup qwen3.5:9b # runtime, download,
# calibrate edit formats, select
phonton # TUI: goals now run on the local model
phonton goal "Add a count() method to TodoStore" --yes --allow-host-checks
Expand All @@ -112,6 +112,20 @@ verify it (an Ollama install can relay cloud models): the TUI plan review says
so and your approval is consent for that goal; headless runs need
`phonton goal --local ... --allow-unverified-runtime`.

`models catalog` recommends one model by total memory (the largest GPU, or
half of RAM when that is larger):

| Memory | Recommended | Alternative |
| --- | --- | --- |
| 4-6 GB | `qwen3.5:4b` | |
| 6-16 GB | `qwen3.5:9b` | `gemma4:12b` (10 GB and up) |
| 16-24 GB | `gpt-oss:20b` | |
| 24 GB and up | `qwen3.6:35b-a3b` | `qwen3.6:27b` |

Under 4 GB, or without a memory reading, nothing is recommended. The
recommendation is a size-based starting point; calibration decides
whether a model is usable. Any other Ollama tag works with `models setup`.

In our smoke runs, qwen2.5-coder:3b on a 6 GB laptop GPU finished small
single-file edits in a Node repo in 20-30 s. Small models work best on one
focused change at a time. Details and limits:
Expand Down
159 changes: 159 additions & 0 deletions benchmarks/polyglot/README.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,159 @@
# Aider Polyglot slice

Runs exercises from the [Aider Polyglot benchmark](https://github.com/Aider-AI/polyglot-benchmark)
through three arms on the same local model and the same prompt, then scores
every run against the exercise's pristine tests:

| Arm | What runs | Its claim of success |
| --- | --- | --- |
| `phonton` | `phonton goal --local "<prompt>"`, then `goal --local apply` | a verified, review-ready candidate |
| `bare` | the model alone: one chat call per try, whole files back | any answer that contains the files |
| `opencode` | `opencode run --pure` with an isolated config, without the user's `~/.claude` instructions or skills | finishing without an error |

The headline numbers are the **pass rate** and the **false-pass rate**: of the
runs where the arm said it succeeded, how many fail the pristine tests.

## Protocol

- **Exercises.** `--per-lang N` takes N exercises per language in a seeded
hash order (`--seed`, default `phonton-polyglot-1`), so a slice is
reproducible and not alphabetical. `--exercises a,b` picks by name. The
full benchmark is 225 exercises in six languages; this runner implements
Python, JavaScript and Rust (113 exercises). Other languages need a test
command in `testCommand()`.
- **Workspace.** Each run gets a fresh temporary git copy of the exercise
without `.meta/` (it holds the reference solution). JavaScript copies share
one `node_modules` junction with jest and babel from the exercise's
`package.json`, plus that install's `package-lock.json`. Phonton leaves the
junction out of its candidate copy and rebuilds `node_modules` from the
lockfile with offline `npm ci` (no lifecycle scripts) before each check;
it runs `node node_modules/jest/bin/jest.js --ci --json`.
- **Prompt.** `.docs/instructions.md` plus `instructions.append.md`, then
Aider's benchmark addendum naming the solution files. Every arm gets the
same text.
- **Scope.** Phonton gets the exercise's solution files as `--files`, except
`Cargo.toml`: it protects files its check depends on and refuses a scope
that names one. The prompt allows standard libraries only, so the manifest
needs no edit. The other arms may still edit it.
- **Tests.** `--tests visible` (default, as in Aider's benchmark) leaves the
test files in the workspace with every skipped JavaScript test enabled.
`--tests hidden` removes them, so no arm can run them.
- **Bare model tries.** With visible tests the bare arm gets Aider's two tries:
the test output goes back once with Aider's "fix the code" message
(`--bare-tries`). With hidden tests it gets one.
- **Scoring.** After the arm finishes, the original test files are written
back (an arm that edits or deletes tests gains nothing; `tests_touched`
flags it) and the language's tests run: `python -m unittest discover
--pattern "*_test.py"`, jest with all tests enabled, `cargo test -- --include-ignored`.
Exit code 0 is a pass.
- **Tokens.** Phonton: input and output tokens of every candidate in the
local run evidence. Bare: the runtime's reported usage for every try.
OpenCode: input, cache-read, output and reasoning tokens from its JSON
step events.
- **Wall time** is the arm's own run, without workspace setup or scoring.
The runner unloads the model from Ollama before every arm run, so each
starts cold and pays its own model load.

## Run

Serve the model with Ollama (or any OpenAI-compatible server), with a context
length every arm can use:

```bash
OLLAMA_CONTEXT_LENGTH=16384 ollama serve
ollama pull qwen3.5:9b
```

Phonton needs that model calibrated and selected in the profile it runs with:

```bash
export PHONTON_HOME=/tmp/phonton-bench
phonton models endpoint http://127.0.0.1:11434
phonton models calibrate qwen3.5:9b && phonton models select qwen3.5:9b
```

Clone the benchmark at a pinned commit and run:

```bash
git clone https://github.com/Aider-AI/polyglot-benchmark ../polyglot-benchmark
git -C ../polyglot-benchmark checkout 7e0611e77b54e2dea774cdc0aa00cf9f7ed6144f
cargo build --release -p phonton-cli
node benchmarks/polyglot/run.mjs --polyglot ../polyglot-benchmark \
--model qwen3.5:9b --per-lang 17 --runtime unverified
```

Before spending GPU time, check the scorer and the language toolchains with
the reference model: every run must pass, and with `--stub` only exercises
whose stub already works (such as the `ledger` refactoring) may pass. The
exception is the 7 Rust exercises with a `.meta/Cargo-example.toml`: their
reference solution needs a crate the exercise does not declare (`pig-latin`
uses `regex`), and arms may only edit the solution files.

```bash
node benchmarks/polyglot/reference-model.mjs ../polyglot-benchmark 11999 &
node benchmarks/polyglot/run.mjs --polyglot ../polyglot-benchmark --model reference \
--endpoint http://127.0.0.1:11999/v1 --arms bare --per-lang 3
```

### On a rented or free GPU

`gpu_job.py` runs the whole slice on a Linux GPU host: it builds Phonton,
installs Ollama 0.34.2, Node and OpenCode at pinned versions, calibrates the
model at 16384 tokens and runs one runner process per language within a time
budget. It writes `out/<lang>/`, `setup.log`, `env.txt` and `ollama.log`.

- **Modal** (24 GB A10G): `modal_job.py` bakes the setup into an image from a
clean, committed checkout and converts a dollar cap into the time budget,
with a hard kill shortly after:

```bash
modal run --detach benchmarks/polyglot/modal_job.py --model qwen3.5:9b --per-lang 17 --max-dollars 9
modal volume get phonton-polyglot runs/<tag> ./polyglot-<tag>
```

- **Kaggle** (GPU T4 x2, 12 h sessions): upload `phonton-src.tar.gz`
(`git archive --format=tar.gz HEAD`) and `commit.txt` as a private dataset,
then push `gpu_job.py` as a private script kernel with that dataset as a
source, `"enable_gpu": "true"`, `"enable_internet": "true"` and
`"machine_shape": "NvidiaTeslaT4"`. Edit `MODEL` and `PER_LANG` defaults in
the script, since kernels take no environment. Attaching a finished run as a
kernel source reuses its `prebuilt/phonton`.

| Flag | Default | Meaning |
| --- | --- | --- |
| `--polyglot DIR` | required | polyglot-benchmark checkout |
| `--model TAG` | required | model for every arm |
| `--endpoint URL` | `http://127.0.0.1:11434/v1` | OpenAI-compatible base for bare and OpenCode |
| `--arms LIST` | `phonton,bare,opencode` | arms to run |
| `--langs LIST` | `python,javascript,rust` | languages |
| `--per-lang N` / `--exercises LIST` | `2` | slice size or explicit exercises |
| `--tests visible\|hidden` | `visible` | whether arms can see the tests |
| `--bare-tries N` | 2 visible, 1 hidden | bare model tries |
| `--runtime unverified` | off | Phonton may use an Ollama it did not start |
| `--bin PATH` | `target/release/phonton` | Phonton build |
| `--opencode PATH` | from `npm root -g` | OpenCode's `bin/opencode` script |
| `--timeout-seconds N` | `900` | per arm run |
| `--out DIR` / `--cache DIR` | `benchmarks/results/...` | artifacts / reusable test deps |

Each run writes `meta.json` (versions, commit, model, hardware, arguments),
`results.jsonl`, `summary.md`, and per exercise and arm: `prompt.md`,
`stdout.txt`, `stderr.txt`, `diff.patch`, `tests.txt`, plus Phonton's
`evidence.json` or OpenCode's config. `benchmarks/results/` is not committed;
copy a run's folder next to its write-up.

## Limits

- Comparisons hold only for one model, one runtime, one context length and one
machine. The runtime's context default caps the bare and OpenCode arms;
Phonton sets its calibrated context per request, which its memory estimate
can hold well below the runtime's (4096 tokens for `qwen3.5:4b` on a 6 GB
GPU). Record both in the write-up.
- Phonton's JavaScript checks need jest's packages in the local npm cache:
its preparation runs `npm ci --offline`. The runner's first `npm install`
fills the cache; a cache miss ends the run unverified, not failed.
- OpenCode reports no verdict, so finishing is its claim; that favours neither
side on pass rate but makes its false-pass rate an upper bound.
- Python is scored with `unittest`, not pytest as in Aider's harness; the
Exercism Python tests are `unittest` classes.
- Single runs vary. Report n and repeat before quoting a difference of a few
exercises.
Loading
Loading