Issue E: model probe — qwen2.5:7b, granite-4.1:8b, gemma3:4b, gemma 4 vs llama3.1:8b baseline

[augustineuzokwe]

Parent Epic: #353
Depends on: Issue C (proves wire works) and Issue D (CareerChrono baseline established)

Purpose

This is the actual model benchmark. Probe candidate Ollama models against RTIA's 7-sample eval suite + the CareerChrono case. If any candidate beats llama3.1:8b on the regressed metrics (ambiguity_discipline, intent_keyword_overlap, requirement_fidelity per docs/ollama-probe-2026-05-26.md), promote it to DEFAULT_OLLAMA_MODEL.

Candidates

Model	Param	Why
llama3.1:8b	8B	Baseline (already probe-tested)
qwen2.5:7b	7B	Strong community reputation for structured JSON
granite-4.1:8b	8B	IBM, explicitly tuned for "tool calling, instruction following"
gemma3:4b	4B	Verified working on this NAS (forum), faster than llama3
Gemma 4	TBD	LM Studio featured; unverified on Ollama — check availability first

Workload + time

My time: ~2 hours scripting + analysis.

NAS time (unattended): ~6–14 hours wall-clock, dominated by inference:

• Model pulls: ~30–60 min × 4 = 2–4h
• Eval runs: 7 samples × 5–15 min × 4 candidates = 2.5–7h

Realistic: NAS runs overnight, review next morning. ~2–3 wall-clock days total with overnight runs.

Storage: ~20 GB (4 models × ~5 GB) on M.2 SSD — trivial vs 931 GB free.

Cost: if judge stays on Gemini (recommended per docs/ollama-probe-2026-05-26.md §7.3), 4 × "a few cents" per 7-sample eval → roughly a quarter-dollar total. Single cost-approval prompt before this Issue starts.

Steps

1. Verify each candidate exists on Ollama registry (ollama show <model> or https://ollama.com/library/)
2. Write a probe-runner shim around evals/run_evals.py (pulls model, runs suite, captures report, writes comparison row)
3. Run sequentially overnight
4. Read reports; if a winner emerges, propose promoting it to DEFAULT_OLLAMA_MODEL via a new sub-issue

Acceptance criteria

• Each candidate's availability verified
• Probe-runner shim written and reviewed
• Each candidate has a completed evals/reports/*.json
• Comparison written summarising per-metric scores vs llama3.1:8b baseline
• If a winner: separate sub-issue opened to promote it to DEFAULT_OLLAMA_MODEL (with new probe baseline entry for docs/ollama-probe-2026-05-26.md)

Approval gate

Cost approval required before starting (~$0.25 Gemini judge cost).