Repository navigation
Record each model call's subject and outcome in traces; trace just eval; add scripts/export_training_data.py - #68
Conversation
…add `scripts/export_training_data.py` Steps now note what their model call was about and how it ended (`annotate()`), prompt files and hashes are recorded, eval and replay runs are traced with gold labels and scores, span limits are raised, and an exporter turns traces into TRL-style chat examples per task. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01FomNXensfhyR2ji8FxW2xu
Codex Review SummaryThis comment shows the latest Codex review activity on this pull request.
ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings. |
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 34c152bf5b
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
Edit and fact-list updates scored by `just eval profile-update` called the updaters directly, so their model calls had no step and exported as `(no step)`. Wrap them in `profile.update` (annotated with the entity) and each fact-list stage in its own step (`fact_extract`, `fact_compare`, ...). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01FomNXensfhyR2ji8FxW2xu
Instructor's reask messages nest tool-call name/arguments under `function` and link the tool reply by `tool_call_id`; the flattener read them from the top level and dropped the ids. Read both shapes, record ids, and have the exporter rebuild input messages from the lossless `input.value`. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01FomNXensfhyR2ji8FxW2xu
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 6304b6a1a7
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
… form Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01FomNXensfhyR2ji8FxW2xu
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 1b1cd6882f
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01FomNXensfhyR2ji8FxW2xu
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: f1e7e391f7
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01FomNXensfhyR2ji8FxW2xu
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 46269e9118
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
Instructor validates after the SDK call returns, so a request that exhausts its retries left clean model-call spans. `request_span()` around `_structured_generation()` records `hinbox.request_ok` (and ERROR on raise); the exporter treats calls under a failed request or step as failed. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01FomNXensfhyR2ji8FxW2xu
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 4c8ed5c190
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
…ic retry blocks Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01FomNXensfhyR2ji8FxW2xu
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 14ec27b2f0
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
…y-turns` keeps the rest Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01FomNXensfhyR2ji8FxW2xu
Requested by Alex · project thread
Why
Before the ~150-article Luna pilot, make sure its traces can be turned into training data for small local models later. The traces from #63 already hold every prompt, reply, tool-call argument and token count. What they lacked was a label on each call: which entity or candidate it was about, how it ended, which prompt file produced it, and, for eval runs, the gold answer. Without those, the pilot's thousands of calls would be prompts and replies with no way to tell the good ones from the bad.
What changes
annotate(**attrs)insrc/utils/tracing.pysetshinbox.*attributes on the current step span. Steps record:extract.<type>: entity type, article id, final count, QC flags, whether the repair retry was used. Each attempt also gets its own span with its own count, QC flags andattempt_selected.match_check: both names, entity type,is_match, confidence, which prompt answered, and whether the memo answered.relevance: the verdict and which prompt answered.dispute: both names, similarity, the action taken, confidence.profile.updateandprofile.create: entity, article id, approach, characters before and after, edits applied/appended/skipped, disagreements, QC result.request_span()wraps each structured request, with all of Instructor's retries, and each sectioned call plus its parsing. It ends withhinbox.request_ok, so a request that never produced a valid answer is marked failed even though each model call inside it returned normally.DomainConfig.prompt_path()adds<file>@<hash>to the step'shinbox.prompt_fileslist. Every LLM span also getshinbox.stepand a hash of its system prompt.--local, git commit (with-dirty), and everyHINBOX_*setting except keys, tokens and secrets.functionshape and tool-call ids.just evalandjust eval-replayare traced. Each eval item is its own trace witheval_label,eval_predictionandeval_correct, oreval_passedandeval_scoresfor profile updates. Profile-update eval calls and the fact-list stages get their own steps.source_typewhen the row has one, so book data can be kept apart later.scripts/export_training_data.py(just export-training <traces...> --out <dir>) writes one<task>.jsonlper task of{"messages", "tools", "metadata"}rows, in the shape TRL, Unsloth and Axolotl read:--keep-retry-turnskeeps Instructor's rejected turns too.--keep-failedkeeps them.hinbox.*notes from the call, step, article and eval-item spans are merged intometadata.Nothing changes what the pipeline does; these are extra attributes and spans only.
Tests
tests/test_training_export.pyhas 14 tests:just ciequivalent (ruff check and format, file-length check,pytest -m "not asyncio"): 583 passed.Not measured: trace file size on a real run. Expect tens of MB for 150 articles, because the full article text is in every extraction prompt.
🤖 Generated with Claude Code
https://claude.ai/code/session_01FomNXensfhyR2ji8FxW2xu