Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
14 changes: 14 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -4,6 +4,20 @@ All notable Phonton CLI release changes should be documented here.

This project follows pre-1.0 SemVer: minor versions may still include breaking changes while the public API and CLI surface settle.

## 0.25.1 - Local runs survive a busy laptop

### Fixed

- [fixed] `phonton goal --local` stopped with `resource_pressure` whenever
available host RAM fell below 1.5 GiB between model calls, even when the
exact calibrated model was still loaded at the calibrated context entirely
in VRAM. That reserve covers starting the runtime, and a loaded copy
already holds its host memory. A chat with a copy entirely in VRAM now
needs 512 MiB free. Cold loads and partly offloaded copies keep their
reserves, and an unknown memory reading still stops the run. In the 0.25.0
Aider Polyglot run on a 16 GB RAM, 6 GB RTX 4050 laptop, this ended 18 of
51 Phonton runs, 13 of 17 in JavaScript, after jest checks.

## 0.25.0 - Current local models, local first, comparable proof

### Changed
Expand Down
2 changes: 1 addition & 1 deletion Cargo.lock

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

2 changes: 1 addition & 1 deletion README.md
Original file line number Diff line number Diff line change
Expand Up @@ -13,7 +13,7 @@
<p align="center">
<a href="https://github.com/phonton-dev/phonton-cli/actions/workflows/ci.yml"><img alt="CI Status" src="https://github.com/phonton-dev/phonton-cli/actions/workflows/ci.yml/badge.svg"></a>
<a href="https://github.com/phonton-dev/phonton-cli/stargazers"><img alt="GitHub stars" src="https://img.shields.io/github/stars/phonton-dev/phonton-cli?style=flat&label=stars&color=ff69b4"></a>
<img alt="release" src="https://img.shields.io/badge/release-v0.25.0--beta-6c63ff">
<img alt="release" src="https://img.shields.io/badge/release-v0.25.1--beta-6c63ff">
<img alt="license" src="https://img.shields.io/badge/license-MIT%20OR%20Apache--2.0-blue">
</p>

Expand Down
3 changes: 2 additions & 1 deletion docs/local-harness-reference.md
Original file line number Diff line number Diff line change
Expand Up @@ -192,7 +192,8 @@ that its own child owns the loopback listener before reporting success.
Before each strategy or edit request, the runner refreshes the runtime version,
installed model digest and size, and cold-load fit from current hardware. If
cold loading no longer fits, the exact calibrated model must be resident at the
required context with at least 1.5 GiB available host RAM. Ollama must report
required context with at least 512 MiB available host RAM when the copy is
entirely in VRAM, or 1.5 GiB when part of it is in RAM. Ollama must report
at least 30 seconds before unload; residency is checked again immediately
before chat. These are point-in-time checks, not a memory reservation.
CLI download output coalesces byte updates while immediately showing stage
Expand Down
5 changes: 3 additions & 2 deletions docs/local-harness.md
Original file line number Diff line number Diff line change
Expand Up @@ -554,8 +554,9 @@ Cold model admission estimates weights, context and a host reserve. Before each
strategy or edit request, the runner refreshes runtime version, installed model
digest and size, observes current host memory, and re-estimates fit at the
calibrated context. If cold loading no longer fits, Ollama must report the exact
calibrated digest already loaded at that context, at least 1.5 GiB available
host RAM, and at least 30 seconds of runtime-reported load time remaining. A
calibrated digest already loaded at that context, at least 512 MiB available
host RAM for a copy entirely in VRAM (1.5 GiB when part of it is in RAM), and
at least 30 seconds of runtime-reported load time remaining. A
resident model admitted at the start must keep its allocation identity.
Residency is checked again immediately before chat. Failed admission stops
before reserving that model call; the receipt records the reason. These
Expand Down
2 changes: 1 addition & 1 deletion package.json
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
{
"name": "phonton-cli",
"version": "0.25.0",
"version": "0.25.1",
"description": "Local-first ADE CLI for planned, verified, reviewable code changes.",
"license": "MIT OR Apache-2.0",
"homepage": "https://github.com/phonton-dev/phonton-cli#readme",
Expand Down
2 changes: 1 addition & 1 deletion phonton-cli/Cargo.toml
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
[package]
name = "phonton-cli"
version = "0.25.0"
version = "0.25.1"
edition.workspace = true
license.workspace = true
repository.workspace = true
Expand Down
105 changes: 66 additions & 39 deletions phonton-worker/src/local_run.rs
Original file line number Diff line number Diff line change
Expand Up @@ -133,11 +133,28 @@ fn creation_system_instruction(path: &Path, protocol: EditProtocol) -> String {
}
}

/// Host RAM a chat with an exact resident model needs free when that copy is
/// entirely in VRAM. Its runtime allocation was made at load time and is
/// already outside the available reading, so the 1.5 GiB cold-load reserve
/// would count it twice; this leaves headroom for the request and the OS. On
/// Ollama 0.34.2 (2026-10-17, RTX 4050) loading qwen3.5:4b at 16384 tokens
/// took about 700 MiB of host RAM; a later 7000-token chat loaded nothing.
// ponytail: fixed floor, measure per-chat host use if a runtime allocates per request.
const VRAM_RESIDENT_HOST_RESERVE_BYTES: u64 = 512 * 1024 * 1024;

/// Whether `resident` may serve the next chat. A copy partly in RAM keeps the
/// full reserve, because paging its weights out would stall generation.
fn resident_headroom(hardware: &HardwareSnapshot, resident: Option<&ResidentModel>) -> bool {
resident.is_some()
&& hardware
resident.is_some_and(|model| {
let reserve = if model.size_vram_bytes >= model.size_bytes {
VRAM_RESIDENT_HOST_RESERVE_BYTES
} else {
phonton_local::GPU_HOST_RESERVE_BYTES
};
hardware
.ram_available_bytes
.is_some_and(|bytes| bytes >= 1536 * 1024 * 1024)
.is_some_and(|bytes| bytes >= reserve)
})
}

async fn confirm_model_identity(
Expand Down Expand Up @@ -180,14 +197,8 @@ async fn admitted_resident_for_generation(
hardware: &HardwareSnapshot,
initial_resident: Option<&ResidentModel>,
) -> std::result::Result<Option<ResidentModel>, GenerationAdmissionError> {
if hardware
.ram_available_bytes
.is_none_or(|bytes| bytes < 1536 * 1024 * 1024)
{
return Err(GenerationAdmissionError::ResourcePressure(
"Available host RAM fell below the 1.5 GiB generation reserve".into(),
));
}
// Host reserves live in the cold-load estimate (unknown memory fails
// there) and in `resident_headroom`; no flat reserve may run first.
confirm_model_identity(runtime, profile, installed.size_bytes).await?;
let fit = runtime
.estimate_cold_load(
Expand Down Expand Up @@ -680,14 +691,8 @@ async fn run_inner(
receipt.state = "budget_exhausted".into();
break;
}
let current = observed_hardware(hardware_override.as_ref()).await;
if current
.ram_available_bytes
.is_none_or(|n| n < 1536 * 1024 * 1024)
{
receipt.state = "resource_pressure".into();
break;
}
// Host memory is admitted immediately before each chat, where the
// exact resident copy can be seen.
if receipt.resident_reuse.is_some()
&& !matches!(
runtime
Expand Down Expand Up @@ -990,15 +995,6 @@ async fn run_inner(
receipt.known_gaps.push("Captured baseline changed during strategy inference; no edit inference started.".into());
break;
}
if observed_hardware(hardware_override.as_ref())
.await
.ram_available_bytes
.is_none_or(|bytes| bytes < 1536 * 1024 * 1024)
{
receipt.state = "resource_pressure".into();
receipt.known_gaps.push("Host RAM fell below the generation reserve after strategy inference; no edit inference started.".into());
break;
}
}
let reserved_left = receipt
.request
Expand Down Expand Up @@ -8719,6 +8715,7 @@ mod tests {
"malicious_repair",
"pressure_after_baseline",
"pressure_with_resident",
"pressure_with_vram_resident",
"model_replaced_after_baseline",
"alias_from_start",
"alias_after_baseline",
Expand All @@ -8745,6 +8742,9 @@ mod tests {
let malicious_repair = scenario == "malicious_repair";
let pressure_after_baseline = scenario == "pressure_after_baseline";
let pressure_with_resident = scenario == "pressure_with_resident";
// After the first load, host RAM falls below the cold-load reserve
// while the exact model stays entirely in VRAM.
let pressure_with_vram_resident = scenario == "pressure_with_vram_resident";
let pressure_after_load = pressure_after_baseline || pressure_with_resident;
let model_replaced_after_baseline = scenario == "model_replaced_after_baseline";
let alias_from_start = scenario == "alias_from_start";
Expand Down Expand Up @@ -8860,6 +8860,8 @@ mod tests {
"/api/show" => serde_json::json!({"model_info":{"general.architecture":"fixture","fixture.context_length":if large_context_budget { 16384 } else if large_creation { 8192 } else { 4096 }},"details":{"format":"gguf"}}),
"/api/ps" => if pressure_with_resident {
serde_json::json!({"models":[{"name":"fixture:small","digest":"fixture-digest","size":4 * 1024 * 1024 * 1024_u64,"size_vram":0,"context_length":4096,"expires_at":"2100-01-01T00:00:00Z"}]})
} else if pressure_with_vram_resident {
serde_json::json!({"models":[{"name":"fixture:small","digest":"fixture-digest","size":2 * 1024 * 1024 * 1024_u64,"size_vram":2 * 1024 * 1024 * 1024_u64,"context_length":4096,"expires_at":"2100-01-01T00:00:00Z"}]})
} else {
serde_json::json!({"models":[]})
},
Expand Down Expand Up @@ -9033,21 +9035,35 @@ mod tests {
let hardware = HardwareSnapshot {
ram_total_bytes: Some(16 * 1024 * 1024 * 1024),
ram_available_bytes: Some(8 * 1024 * 1024 * 1024),
gpus: if pressure_with_vram_resident {
vec![phonton_types::local::GpuSnapshot {
name: "fixture GPU".into(),
total_bytes: 6 * 1024 * 1024 * 1024,
available_bytes: 6 * 1024 * 1024 * 1024,
}]
} else {
Vec::new()
},
..HardwareSnapshot::default()
};
let hardware_reader = if wall_exhausted_before_edit {
let observations = Arc::new(AtomicUsize::new(0));
let snapshot = hardware.clone();
// Observation 0 admits the run; 1 admits the first edit chat.
Arc::new(move || {
if observations.fetch_add(1, Ordering::SeqCst) == 2 {
if observations.fetch_add(1, Ordering::SeqCst) == 1 {
std::thread::sleep(Duration::from_millis(10_100));
}
snapshot.clone()
}) as HardwareOverride
} else if pressure_after_load {
} else if pressure_after_load || pressure_with_vram_resident {
let observations = Arc::new(AtomicUsize::new(0));
let later = HardwareSnapshot {
ram_available_bytes: Some(3 * 1024 * 1024 * 1024),
ram_available_bytes: Some(if pressure_with_vram_resident {
1024 * 1024 * 1024
} else {
3 * 1024 * 1024 * 1024
}),
..hardware.clone()
};
let first = hardware.clone();
Expand Down Expand Up @@ -9301,7 +9317,8 @@ mod tests {
assert_eq!(std::fs::read(&index_path).unwrap(), index_before);
continue;
}
if pressure_with_resident {
if pressure_with_resident || pressure_with_vram_resident {
assert_eq!(receipt.state, "review_ready", "{receipt:?}");
assert!(
paths
.iter()
Expand Down Expand Up @@ -10059,22 +10076,32 @@ mod tests {
}
#[test]
fn resident_reuse_still_requires_host_headroom() {
let resident = ResidentModel {
let mut resident = ResidentModel {
name: "coder:small".into(),
digest: "digest-a".into(),
size_bytes: 1 << 30,
size_vram_bytes: 1 << 30,
context_length: 4096,
expires_at_unix: 4_102_444_800,
};
let mut hardware = HardwareSnapshot {
ram_available_bytes: Some(1536 * 1024 * 1024 - 1),
let ram = |bytes| HardwareSnapshot {
ram_available_bytes: bytes,
..Default::default()
};
assert!(!resident_headroom(&hardware, Some(&resident)));
hardware.ram_available_bytes = Some(1536 * 1024 * 1024);
assert!(resident_headroom(&hardware, Some(&resident)));
assert!(!resident_headroom(&hardware, None));
let admits = |mib: u64, model: &ResidentModel| {
resident_headroom(&ram(Some(mib * 1024 * 1024)), Some(model))
};
// Entirely in VRAM: the runtime already holds its host allocation.
assert!(!admits(511, &resident));
assert!(admits(512, &resident));
assert!(admits(1535, &resident));
// Partly in RAM keeps the full reserve.
resident.size_vram_bytes -= 1;
assert!(!admits(1535, &resident));
assert!(admits(1536, &resident));
// Unknown memory or no exact resident model stays closed.
assert!(!resident_headroom(&ram(None), Some(&resident)));
assert!(!resident_headroom(&ram(Some(8 << 30)), None));
}
#[tokio::test]
async fn python_cache_directories_do_not_become_captured_source() {
Expand Down
44 changes: 44 additions & 0 deletions release-notes/v0.25.1.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,44 @@
# phonton-cli v0.25.1 — Local runs survive a busy laptop

0.25.0 recommends `qwen3.5:4b` for 4-6 GB GPUs, but on exactly that hardware
local goal runs often stopped halfway with `resource_pressure`. 0.25.1 lets a
model that is already loaded in VRAM keep working when the rest of the
machine is busy.

## A loaded model keeps working

`phonton goal --local` refused every model call once available host RAM fell
below 1.5 GiB, even when the exact calibrated model (same digest, same
context) was still loaded entirely in VRAM. That reserve covers starting the
runtime. A model that is already loaded holds its host memory, and the
available-RAM reading already excludes it, so the check counted that memory
twice. Running the project's checks between model calls, such as an offline
`npm ci` and jest, was enough to trip it with a browser and chat apps open.

A call to a copy held entirely in VRAM now needs 512 MiB of free host RAM.
On Ollama 0.34.2 with an RTX 4050, loading `qwen3.5:4b` at 16384 tokens took
about 700 MiB of host RAM; a later 7,000-token call to the loaded copy loaded
nothing. Everything else still fails closed:

- a cold load keeps the 1.5 GiB reserve (2 GiB when the model is partly
offloaded to the CPU);
- a resident copy that is partly in RAM keeps the 1.5 GiB reserve;
- a loaded copy with another digest or a smaller context gets no exception
and is admitted like a cold load;
- an unknown memory reading still stops the run before inference.

Residency is still checked again immediately before each call; it is an
observation, not a reservation.

In the 0.25.0 Aider Polyglot run on a 16 GB RAM, 6 GB RTX 4050 laptop, this
check ended 18 of 51 Phonton runs, including 13 of 17 JavaScript runs, about
100 seconds in, after the jest checks.

## Install

```text
npm install -g phonton-cli
```

Or download a binary from GitHub Releases, or
`cargo install --git https://github.com/phonton-dev/phonton-cli phonton-cli --locked`.
Loading