Codemium treats performance claims as engineering evidence, not marketing copy.
The benchmark is explicitly competitive: Codemium is compared against baseline, caveman, and Ponytail on the same work. Ponytail is a comparison target, not a parent framework or positioning reference for Codemium.
Assumed benchmark:
ASSUMED_NUMBERS.mdpublishes the current synthetic scenario model. It is explicitly not measured product performance.Measured benchmark program:
PUBLIC_BENCHMARK.mddefines the evidence gate for future real runs. Measured datasets belong inresults/.
A public competitive study must include:
baseline— same coding agent, model, reasoning configuration, and environment with no optimization skill;caveman— terse-prose/minimal-control benchmark arm;ponytail— Ponytail on the identical task set and repository starts;codemium— Codemium@cmon those identical conditions.
Internal Codemium variants such as codemium-fast, codemium-deep, different reasoning efforts, or feature-disabled builds are ablation studies. They should not replace the four primary competitive arms.
For every arm:
- use the same task/ticket text;
- start from the same repository commit or clean fixture;
- use the same coding agent host;
- use the same base model and host reasoning configuration unless the study explicitly tests reasoning policy;
- use the same tools, network permissions, environment, timeout, and dependency state;
- isolate runs so one arm cannot inherit another arm's context;
- score the final diff and observable behavior using the same evaluator;
- record raw run data before aggregation;
- use repeated runs (
n >= 4recommended) because agentic results vary.
Where practical, quality/safety scoring should be blind to the arm identity.
A public measured Numbers chart may be rendered only when:
meta.kindis exactlymeasured;- the
baseline,caveman,ponytail, andcodemiumarms all exist; - the four arms cover identical task IDs;
- quality and safety results are recorded for every competitive run;
- measured token values come from host telemetry;
- measured cost comes from billing/run telemetry rather than fabricated or stale pricing estimates.
Synthetic data may be published only when it remains visibly labeled as synthetic/assumed and is never represented as measured product performance.
Lower is better:
- LOC changed;
- total tokens = input + reasoning + output;
- measured cost;
- wall-clock time.
Higher is better:
- quality pass rate;
- safety pass rate.
Diagnostic metrics should also be retained:
- tool calls;
- unique files read;
- duplicate reads;
- unrelated changed lines;
- tests/checks executed;
- regression findings;
- context/cache reuse where measurable.
Efficiency is not a win when correctness or safety falls.
A Codemium result should only be described as better when the quality floor is preserved:
quality >= baseline/control quality
safety >= baseline/control safety
regressions <= controls
Only after that should LOC, tokens, cost, and time be compared.
{
"meta": {
"kind": "measured",
"study_type": "competitive",
"title": "Codemium competitive agent benchmark",
"repository": "owner/repo@commit",
"agent": "Codex",
"model": "same model + reasoning configuration",
"runs_per_arm": 4,
"required_arms": ["baseline", "caveman", "ponytail", "codemium"]
},
"runs": [
{
"task_id": "ticket-01",
"system": "baseline",
"quality_pass": true,
"safety_pass": true,
"input_tokens": 12345,
"reasoning_tokens": 2345,
"output_tokens": 678,
"cost_usd": 0.0123,
"loc_changed": 42,
"seconds": 90
}
]
}Each task_id must appear in every primary arm for publication.
Synthetic/assumed scenario:
python benchmarks/render_numbers.py \
benchmarks/example-runs-v2.json \
--svg benchmarks/demo-numbers.svg \
--markdown benchmarks/demo-NUMBERS.mdPublish measured competitive results:
python benchmarks/render_numbers.py \
benchmarks/results/<study>.json \
--publish \
--svg benchmarks/numbers.svg \
--markdown benchmarks/NUMBERS.mdThe renderer is stdlib-only. System colors are intentionally stable in the primary chart:
- baseline — gray;
- caveman — orange;
- ponytail — green;
- codemium — purple.
As of 2026-08-18, the repository publishes an explicitly synthetic assumption model in ASSUMED_NUMBERS.md: Codemium is modeled at approximately -45% LOC, -31% tokens, -29% cost, and -28% time vs baseline, with the same assumed quality/safety floor.
No complete measured Codemium-vs-baseline-vs-caveman-vs-Ponytail dataset is committed yet, so those percentages are not measured product-performance claims.