Run the fact-list updater in the pipeline (HINBOX_PROFILE_UPDATE_APPROACH=facts), with timelines and saved fact rows

CLAUDE.md

@@ -55,7 +55,7 @@ The CLI entry point `src/process_and_extract.py` coordinates the pipeline end to
    - **Name-variant fallback**: when no candidate clears the similarity threshold, a candidate with a spelling variant of the incoming name still goes to the match check (`names_are_variants()` in `src/utils/name_variants.py`, searched over every entity, not just the capped lexical shortlist). Variants: the same name after `normalize_for_match()` (which drops accents, so "Guantánamo Bay" = "Guantanamo Bay"), a configured equivalence group ("Gitmo"), for locations/organizations the same core name without a leading "the"/"U.S.", a trailing "County"/"City" or a trailing "in/at/near …" description, one word respelled as a plural, ordinal or shortened form ("Commission"/"Commissions", "11"/"11th", "Counterterror"/"Counterterrorism"), or middle words dropped with the same first and last word ("Guantánamo Navy base" / "Guantánamo Bay Navy Base"); for people the same first and last name ("Donald J. Trump"), the same last name with a first name shortened or one letter off ("Clay"/"Clayton", "Shelley"/"Shelly"), or the reverse, or a bare surname ("Trump"); for events the same words in any order ("Firing of Harvey Rishikof as convening authority"). Substring containment and derived acronyms don't count. On the merge gold set this sends 22 of 25 labelled-same pairs to the match check (was 12) and 1 of 166 labelled-different pairs (unchanged). If the match check rejects the closest candidate, the best name-variant candidate gets one more match check.
    - **Merge dispute agent**: When a match result falls in the "gray band" (similarity within ±`MERGE_GRAY_BAND_DELTA` of threshold) with confidence below `MERGE_UNCERTAIN_CONFIDENCE_CUTOFF`, `MergeDisputeAgent` provides a second-stage LLM analysis that can override the initial merge/skip decision. Deferred cases are appended to `{output_dir}/merge_review_queue.jsonl` for human review. When the final verdict is skip or defer, the incoming entity is created as a new entity rather than dropped (unless an entity with the same key already exists, which is never overwritten).
    - **Canonical name selection**: 5-layer deterministic scoring (`score_canonical_name()` in `src/utils/name_variants.py`) picks the best display name, penalizing acronyms, generic phrases, contextual suffixes, and ALL CAPS names (`is_shouting_name()`; a same-name-in-normal-case incoming entity replaces an ALL CAPS one).
-9. **Profile Versioning**: `src/engine/profiles.py` maintains `VersionedProfile` history whenever entity content changes. Creation generates a first draft, then runs up to 3 reflect/regenerate rounds (`iterative_improve()`). Updates apply a list of edits from one model call (`src/engine/profile_edits.py`, no reflection); `HINBOX_PROFILE_UPDATE_APPROACH=rewrite` (or `update_profile(..., approach="rewrite")`) brings back the old whole-profile rewrite with reflection.
+9. **Profile Versioning**: `src/engine/profiles.py` maintains `VersionedProfile` history whenever entity content changes. Creation generates a first draft, then runs up to 3 reflect/regenerate rounds (`iterative_improve()`). Updates apply a list of edits from one model call (`src/engine/profile_edits.py`, no reflection); `HINBOX_PROFILE_UPDATE_APPROACH=rewrite` (or `update_profile(..., approach="rewrite")`) brings back the old whole-profile rewrite with reflection. `HINBOX_PROFILE_UPDATE_APPROACH=facts` uses the fact-list updater (`src/engine/fact_list.py`) instead; its rows (source kinds and quotes, repeat counts, which row a conflict disputes) are saved as a JSON string under `profile["fact_rows_json"]` and reloaded on the next update (`load_rows()`, which drops saved rows whose sentence is no longer in the text). Its compare step has an `update` verdict for a later state of the same thing (moved camps, a decision after "still deciding"), written as a timeline entry next to the old one; a `conflict` must also pass `values_differ()` and `same_subject()` in code.
 10. **Profile Grounding**: Post-processing verification (`verify_profile_grounding()`) extracts citation markers from profile text, looks up the cited source articles, and uses an LLM to check whether each claim is supported. The `GroundingReport` includes a grounding score and per-claim support levels. The pass (`src/engine/grounding_pass.py`) checks up to `grounding_workers` profiles at once after all merging.
 11. **Persistence**: Batched Parquet writes per entity type via `write_entities_table()` avoid write amplification. Every row is given every key before writing, because Arrow takes its columns from the first row only. Article processing status is tracked in a sidecar JSON file (`ProcessingStatus`) rather than rewriting the articles Parquet. An article is marked processed only when every extraction and merge step worked: if an extraction call raised, a type's merge raised, or a profile create/update call failed (`incomplete_reasons()` in `src/process_and_extract.py`; the merger counts these in `MergeStats.failed_profiles`), it is recorded as `incomplete` and retried next run. A failed profile update leaves the article unlinked from that entity so the retry updates the profile. Entity tables are written before the status sidecar (`save_entities_then_status()`), so a failed entity write leaves articles unprocessed. Ctrl+C during article processing or the grounding pass saves the entities merged so far and exits with 130.
 

README.md

@@ -268,7 +268,7 @@ Gold files live in `{output_dir}/eval/`. Each run writes its per-item prediction
 
 The pipeline's `update_profile` uses an edit-list updater (`--approach rewrite` scores the old whole-profile rewrite instead). Rather than writing the whole profile again, the model returns a short list of edits: add these sentences after this existing sentence (or under this heading, or at the end), or replace this one sentence with a sharper version. Code applies the edits to the old text, so a sentence the model never mentions stays exactly as it was, and a replaced sentence keeps its citations. The prompt (`profile_edit.cloud.md` for cloud models, `profile_edit.local.md` for local ones) also says to keep both versions, each with its source, when the article contradicts the profile, and to add little or nothing from an article that only mentions the entity in passing. On the dev and test cases it passes 0.89 on Luna against 0.29 for the rewrite, in about 2s per update instead of 21s. Set `HINBOX_PROFILE_UPDATE_APPROACH=rewrite` to run the pipeline with the old rewrite.
 
-`--approach facts` scores a prototype for very long profiles (`src/engine/fact_list.py`), which no model call ever sees whole. The model lists the article's claims about the entity, each with an exact quote that code checks against the article. Code then finds the few existing sentences most like each claim, and the model says whether the claim is the same, new, or in conflict. A repeated claim is only counted, not cited again, unless it comes from a different kind of source (a court record where the profile rested on a newspaper). A conflict is written by code right after the old sentence, so both versions stay. New facts are placed one section at a time. It isn't used by the pipeline yet.
+`--approach facts` scores a prototype for very long profiles (`src/engine/fact_list.py`), which no model call ever sees whole. The model lists the article's claims about the entity, each with an exact quote that code checks against the article. Code then finds the few existing sentences most like each claim, and the model says whether the claim is the same, new, or in conflict. A repeated claim is only counted, not cited again, unless it comes from a different kind of source (a court record where the profile rested on a newspaper). A conflict is written by code right after the old sentence, so both versions stay. New facts are placed one section at a time. Set `HINBOX_PROFILE_UPDATE_APPROACH=facts` to use it in `just process`; each profile then keeps its fact rows (who said what, which articles only repeated a fact) under `fact_rows_json` for the next update. A later state of the same thing (a transfer, a decision after "still deciding") is kept as a timeline, not called a conflict.
 
 **DSPy experiments** live in [`experiments/dspy_opt/`](experiments/dspy_opt/README.md), in their own environment, so DSPy never enters Hinbox's main dependencies, and LiteLLM is never installed. `just dspy-merge --model <model> --optimizer mipro` optimises the merge check for one model and puts the result on the same scoreboard. `just dspy-profile --model <model> --reflection-model openai/gpt-6-luna` runs GEPA on the edit-list profile updater.
 

configs/guantanamo/prompts/fact_compare.md

@@ -6,9 +6,10 @@ You keep a list of sourced facts about the {entity_type} "{entity_name}". A new
 
 - `same`: one of the known facts already says this, possibly in other words. Give its id in `row`. Set `new_evidence` to true only when the claim comes from a different kind of source than the known fact and that matters to a historian: a court record, official document or first-hand witness where the known fact rested on a secondhand report, or the reverse. A second newspaper repeating the same thing is not new evidence. When the known fact's source is "not recorded", judge from its wording ("a court filing said", "according to his lawyer"); if it gives no source either, set `new_evidence` only for a court record, official document or first-hand witness.
 - `conflict`: one known fact gives a different value for the very same thing, so both can't be true: the same event on a different date, the same count with a different number. Give its id in `row`.
+- `update`: the claim is a later state of the same thing as a known fact, so both were true at their own time: a detainee moved to another camp, a decision made after "still deciding", a new trial date, a new count for a later date, a new role. Give the known fact's id in `row`. Both stay in the profile as a timeline. A completed past event can't be updated: a different date or place for the same bombing, capture, arrival or ruling is a `conflict`, not an `update`.
 - `new`: none of the known facts says this. Also use `new` when the claim adds real detail (a date, an outcome, a new step) to a known fact.
 
-These are not conflicts (most are `new`):
+These are not conflicts (they are `update` or `new`):
 - two different events or steps, such as charges filed in one month and a trial ordered in another;
 - an older report saying something hadn't happened yet;
 - a later development, such as a new role, a transfer, or a rescheduled hearing;