Trio algorithm validator

Replays a corpus of recorded oref algorithm inputs through two versions of the Trio Swift oref implementation and tells you whether they agree.

Underneath is oref-validator, a Swift CLI that replays a corpus against one revision and writes an output tree. compare-branches.py drives it twice and diffs the results, which is what you usually want.

Compare two branches

./compare-branches.py dev perf/oref-swift --limit 3

=== dev
    bb2dc77f5 CI: Bump APP_DEV_VERSION to 0.8.4.40 [skip ci]

=== perf/oref-swift
    50d576c20 Fix build error

========================================================================
A  dev  {'branch': 'dev', 'commit': 'bb2dc77f5d3e...', 'dirty': False}
B  perf-oref-swift  {'branch': 'perf/oref-swift', 'commit': '50d576c20b8e...', 'dirty': False}
ignoring envelope keys ['durationSeconds'], output keys {'determineBasal': ['id']}
------------------------------------------------------------------------
records compared: 138   identical: 138   differing: 0

EQUIVALENT: every compared record matches.
output trees: ./output/dev  ./output/perf-oref-swift

For each revision in turn it checks out Trio/, runs ./build.sh, replays the corpus into output/<revision>/, and then diffs the two trees. The original branch is restored when it finishes — including after a failed build or a Ctrl-C.

--limit is per (function, timezone), so --limit 3 above covers every input type in every timezone in about a minute. Drop it for the real thing:

./compare-branches.py dev perf/oref-swift --jobs 10

That's 239,064 records per side, roughly 12 minutes and 1.5 GB of output each; output/ is gitignored. --resume picks up an interrupted run.

compare-branches.py A B [options]
  A, B                       branch, tag or commit; anything git rev-parse takes
  --limit <n>                records per (function, timezone); omit for everything
  --input <dir>              corpus root (default: ./fixed_bug_inputs)
  --output <dir>             where output trees go (default: ./output)
  --jobs <n>                 parallel workers per run
  --function <name>          repeatable; restrict to one input type
  --resume                   keep existing output trees and fill in the gaps
  --examples <n>             mismatching records to print in full (default: 3)
  --quiet

Exit codes: 0 equivalent, 1 mismatches or per-record failures, 2 setup error.

When they disagree

You get a rollup of which keys differ and in how many records, then the first few records in full. For example:

records compared: 4182   identical: 4179   differing: 3

mismatches by input type: {'determineBasalInput': 3}
status pairs among mismatching records (A,B): {('ok', 'ok'): 3}

differing keys (record count):
        3  output.reason
        2  output.rate

first 3 example(s):

  determineBasalInput/Europe/Berlin/06dafa03-....0.json
    output.rate
      A: 1.65
      B: 1.7

Every mismatching record lands in output/mismatches-<A>-vs-<B>.json with both sides in full.

What it needs, and what it ignores

Trio/ has to be clean — the script switches branches there, and uncommitted work would either block the checkout halfway through or quietly end up in both sides of the comparison. It refuses to start otherwise.

Both runs use --strip-nondeterministic, and the diff additionally ignores whatever keys the two manifest.json files declare nondeterministic (see Keys that don't repeat between runs). Nothing about which keys to skip is hardcoded in the script.

Output trees are emptied at the start of each run unless you pass --resume. Otherwise records left over from a run with a different --function, --limit, or revision would still be sitting in the tree at compare time, and they would read as agreement.

Build

compare-branches.py runs this for you. Build by hand when you want to use the CLI directly.

./build.sh

That's swift build -c release -Xswiftc -enable-testing. The -enable-testing flag is required, not optional — nothing in the oref algorithm is public, so the CLI reaches it with @testable import Trio, and a release build of the dependency won't allow that unless it's asked for on the command line. A plain swift build -c release fails with module 'Trio' was not compiled for testing.

The binary lands at .build/release/oref-validator. There are no external dependencies, so the build works offline.

Using oref-validator directly

One revision at a time, no diffing. Run five records of each kind, just to see it work:

.build/release/oref-validator run --output /tmp/out --limit 5

--limit is per (function, timezone), so that gives you about 228 files spread across every input type and every timezone in the corpus — a good smoke test. It still reads the whole corpus to work out which records those are (see Routing), so budget about 15 seconds rather than the few seconds the 228 records themselves take.

Run one input type:

.build/release/oref-validator run --output /tmp/out --function autosens

Run the whole corpus:

.build/release/oref-validator run --output /tmp/out --jobs 10

That's 239,064 records and takes roughly 12 minutes on Apple silicon (about 14 seconds of that spent reading the corpus to route it), producing about 1.5 GB of output. Add --resume to pick up where an interrupted run left off; resuming skips already-written records without re-reading their inputs.

Looking at a single record

.build/release/oref-validator run-one \
  --input fixed_bug_inputs/autosensInput/America/Chicago/01316eee-4044-4585-b279-f784ef917475.0.json

{"createdAt":1756536157.321045,"durationSeconds":0.07,"function":"autosens",
 "id":"6A46B314-A9CB-4477-A385-2187D0A477B8","output":{"newisf":52,"ratio":1.15},
 "status":"ok","timezone":"America/Chicago"}

Add --log to dump the algorithm's own debug output to stderr. Be aware that the algorithm only logs from five places, all of them in the makeProfile path and all on error conditions, so --log is silent for most records. That's expected.

Comparing two versions by hand

This is what compare-branches.py automates:

git -C Trio checkout <revision-A>
./build.sh && .build/release/oref-validator run --output /tmp/A --jobs 10

git -C Trio checkout <revision-B>
./build.sh && .build/release/oref-validator run --output /tmp/B --jobs 10

Both trees mirror the input tree exactly, so a comparer joins them on relative path. Each tree's manifest.json records which Trio commit produced it and which keys to ignore when diffing — a comparison script shouldn't need to know anything not in that file.

oref-validator command line reference

oref-validator run [options]
  --input <dir>              corpus root (default: ./fixed_bug_inputs)
  --output <dir>             required; mirrors the input tree
  --function <name>          repeatable; default: every type found
  --timezone <id>            repeatable; default: every zone found
  --jobs <n>                 default: number of active cores
  --limit <n>                per (function, timezone); for smoke tests
  --resume                   skip inputs whose output already exists
  --fail-fast                stop on the first per-file failure
  --strip-nondeterministic   null out keys that vary between runs
  --trio <dir>               Trio checkout for provenance (default: ./Trio)
  --quiet / --verbose

oref-validator run-one --input <file> [--output <file>] [--log]
  --force-timezone <id>      replay under a zone other than the record's own

oref-validator --help / --version

--function accepts either form: meal or mealInput.

Exit codes: 0 clean, 1 completed with per-file failures, 2 usage/configuration/IO error.

Why timezone handling drives the design

The oref algorithm resolves basal rates, ISF, carb ratios and target schedules by minutes since local midnight, reading the ambient timezone through Calendar.current and TimeZone.current in a dozen places. Replaying a record under the wrong zone produces wrong numbers with no error and no warning:

autosensInput/America/Chicago/01316eee-....0.json
  under America/Chicago      -> {"newisf":52,"ratio":1.15}
  under America/Los_Angeles  -> {"newisf":53,"ratio":1.14}

This is why the corpus is bucketed by timezone, and why each record has to be replayed under the zone it was recorded in.

The only thing that moves both TimeZone.current and Calendar.current.timeZone is the TZ environment variable, and it only takes effect at process start. (Assigning NSTimeZone.default moves Calendar.current but leaves TimeZone.current pinned to the machine's zone — a half-fix that produces plausible-looking numbers while the determination reason string reports the wrong zone name.)

So run is a supervisor over copies of its own binary: it processes one timezone at a time, spawning workers with TZ set for that zone. Each worker asserts on startup that TZ actually took effect and refuses to run otherwise. Timezones run serially but each is fully parallel, so utilization stays high even though America/Los_Angeles holds about half the corpus.

The upshot for you: results don't depend on your machine's timezone. Running under TZ=Europe/Berlin in your shell produces a byte-identical tree.

Architecture

One binary, three commands

run is a supervisor. It never runs algorithm code itself — it walks the corpus, decides who processes what, spawns workers, and tallies what comes back.

run-worker is where the algorithm actually runs. It's a real subcommand, deliberately undocumented in --help, because it only makes sense when launched by run with a specific TZ and a specific list of files to process.

run-one is the debugging path: one record, envelope printed to stdout.

run-worker and run-one share the same decode-dispatch-encode core, so a record replayed either way produces identical bytes.

What a run does

run (parent)                          no algorithm code runs here
 │
 ├─ walk --input for every *.json, at any depth
 ├─ read each one, take its function and timezone from the record
 ├─ apply --function / --timezone / --limit / --resume filters
 ├─ group by timezone; create every output directory up front
 │
 ├─ for each timezone, one at a time:
 │    │
 │    ├─ write N file lists into a temp directory
 │    └─ spawn N copies of this same binary:
 │
 │         TZ=America/Chicago  oref-validator run-worker --file-list …0.list
 │         TZ=America/Chicago  oref-validator run-worker --file-list …1.list
 │         TZ=America/Chicago  oref-validator run-worker --file-list …2.list
 │
 │       each worker: assert TZ actually took effect, then per file
 │         read → decode → run generator → encode → write output
 │         emit one NDJSON progress line on stdout
 │
 │       parent drains those pipes continuously and tallies results
 │
 └─ write manifest.json

Timezones run serially; the workers within a zone run in parallel. That ordering is forced by TZ being a process-level setting, but it costs little: even though America/Los_Angeles holds about half the corpus, every zone is itself fully parallelized.

Routing: where a record belongs

A record can only be replayed correctly under the timezone it was recorded in, and can only be dispatched if you know which of the five oref entry points it is for. Both facts are in the record, so that is where they are read from — timezone and function, decoded out of every file before anything is spawned.

Nothing reads the directory layout. The corpus happens to be bucketed as <function>Input/<Region>/<City>/, but a flat directory, or a tree organized some other way entirely, routes exactly the same. Relative paths are carried through untouched, so the output tree still mirrors whatever shape the input tree has and a comparer can still join two runs on relative path.

The price is that the parent reads and parses all 17 GB before the first worker starts. That pass is parallel (up to --jobs lanes, striding through the file list so each lane gets the same mix of sizes) and takes about 14 seconds on Apple silicon — a couple of percent of a full run. Two things keep it honest:

• Only timezone and function are decoded, so no payload is ever materialized.
• With --resume, a file whose output already exists is skipped before it is read, so resuming does not pay the routing cost for work already done.

A .json file that does not yield both fields — a stray manifest.json, a scratch file, a record naming a function this tool doesn't know — is skipped, counted, and reported. That's the same "anything not matching is skipped" the old directory walk did, just decided by content rather than by path. Files that aren't .json are ignored outright.

Symlinked directories are followed, since a corpus this size is often assembled out of them, but each resolved target is visited once so a link pointing back up the tree terminates instead of recursing forever.

How work gets divided

Per timezone, the parent spawns N = min(--jobs, ceil(fileCount / 64)) workers — small zones don't need the full job count.

Files are handed out round-robin, not in contiguous blocks, because cost per record varies about threefold by input type. Measured single-threaded on this corpus:

function	files/s
determineBasal	~58
meal	~44
iob	~30
autosens	~19

autosens is slowest because it runs its generator twice (an 8h and a 24h window) and keeps the lower ratio. Round-robin gives every worker the same mix of types, so they finish together instead of one straggling.

Each worker gets an explicit list of files rather than a shard index and a stride. Sharding logic that has to agree on both sides of a process boundary is a good way to silently skip files.

Each line of that list is <function>\t<relative path> — the function the parent read out of the record when it routed the file. Sending it along is what lets the worker check that the file still says the same thing (see Safety rails).

Progress reporting

Workers emit one line of NDJSON per record on stdout ({"path":…,"function":…,"status":…,"ms":…}). function is in there because the parent has no way to infer it from the path, and it's what the manifest's per-function counts are built from; the timezone isn't, since it's constant for a whole worker.

The parent reads those pipes continuously while workers run, rather than after they exit — a worker producing 80,000 progress lines would otherwise fill its pipe buffer and block forever waiting for a parent that's waiting for it.

Counters live in one lock-guarded tally shared across all of a zone's reader queues. Progress goes to stderr, so stdout stays clean.

Safety rails

The tool is built around the fact that its worst failure mode is silent: wrong numbers, no error. So a few things are hard errors that abort the whole run, rather than per-file failures:

• A worker whose TZ didn't take effect. Checked on startup, before any record is touched.
• A record whose timezone names a zone this system doesn't recognize, since it can't be replayed under the zone it was recorded in. Caught during routing, so it fails before any work is done, and the message names the file. --function and --timezone apply first, so a narrowed run isn't held up by a record it was never going to replay.
• A record whose timezone or function disagrees, at process time, with what routing read out of it moments earlier. That means the file changed under the run, and every routing decision made from it — which zone's worker got it, which generator it will be handed to — is now suspect.

Everything else — a record that won't decode, a generator that throws — is recorded in that record's envelope and the run continues. See Output for how those surface. A .json file that can't be routed at all is skipped rather than fatal; it's counted and reported at the start of the run.

Source map

file	role
main.swift	subcommand dispatch, exit codes
Options.swift	hand-rolled flag parsing (no external deps)
RunCommand.swift	the supervisor: grouping, spawning, manifest
WorkerCommand.swift	the per-file loop that runs the algorithm
RunOneCommand.swift	single record, plus the TZ re-exec
Corpus.swift	input tree walk, record routing, output directory creation
RecordProcessor.swift	decode → dispatch → envelope; the TZ guard
AlgorithmDispatch.swift	the five oref entry points and their quirks
InputRecord.swift	record envelope and the five input payloads
OutputEnvelope.swift	output record shape
Coding.swift	encoder/decoder configuration, AnyEncodable
Manifest.swift	manifest shape, git provenance
Tally.swift	counters, progress reporting, NDJSON collection
ValidatorError.swift	fatal errors and exit codes

run-one deserves one note: since TZ only applies at process start, it can't fix the zone in-process either. It reads the record's timezone, and if the current process isn't already running under it, re-executes itself with TZ set. An environment marker on the child stops that from recursing if the zone still won't apply.

Input corpus

Every *.json file under --input, at any depth. Each one is one serialized AlgorithmComparison record from the archived oref-swift-port branch: comparison metadata this tool ignores, the input payload for one algorithm call, and the two fields that place it —

{ "timezone": "America/Chicago", "function": "autosens", … }

Dates are epoch seconds, not ISO 8601.

Layout is not part of the contract. This corpus is bucketed as

fixed_bug_inputs/<functionName>Input/<Region>/<City>/<uuid>.<n>.json

but nothing reads that; see Routing. Types and zones are whatever the records say they are, never hardcoded, so a corpus that starts carrying makeProfile records works without a code change — in a makeProfileInput/ directory or anywhere else.

The current corpus is 239,064 files / 17 GB across 13 timezones:

function	files	zones
meal	133,825	12
iob	82,268	13
determineBasal	20,767	11
autosens	2,204	11

There are no makeProfile records. That's expected, not a corpus bug; the code path exists and works but is unexercised by this corpus.

Output

One output file per input, at the same relative path — the filename stem is not the record id, so relative path is the only stable join key between two runs.

{
  "id": "6A46B314-A9CB-4477-A385-2187D0A477B8",
  "createdAt": 1756536157.321045,
  "timezone": "America/Chicago",
  "function": "autosens",
  "status": "ok",
  "output": { "newisf": 52, "ratio": 1.15 },
  "durationSeconds": 0.0312
}

status is one of:

• ok — output holds the generator's result. Note that output may legitimately be null: MealGenerator returning nil is a real answer, not a failure. determineBasal also reports ok with {"error": "..."} as its output when the algorithm declines to produce a determination, which matches what the JS implementation returned.
• error — the algorithm threw. error holds errorType and message.
• decodeError — the input didn't decode. error.message is the full DecodingError description, which names the offending key and is the fastest way to spot corpus/schema drift.

A per-file failure never stops the run (unless you pass --fail-fast). What does abort it is a record whose timezone or function no longer matches what routing read out of it, because continuing would mean writing silently wrong numbers. See Safety rails.

Note that decodeError is reachable for a file that routed cleanly: routing only needs timezone and function, so a record with those two fields and a payload that has since drifted routes fine and then fails to decode here, which is exactly where you want to find out.

Output is encoded with sorted keys and no pretty-printing, so files are byte-comparable across runs.

manifest.json

Written once per run at the root of the output tree. The important field is trio.commit — it's what tells you which algorithm version produced the tree.

{
  "tool": { "name": "oref-validator", "version": "1" },
  "startedAt": 1785193525.9, "finishedAt": 1785193534.6, "wallSeconds": 8.69,
  "inputRoot": "/abs/path/fixed_bug_inputs",
  "outputRoot": "/abs/path/out",
  "trio": { "commit": "77135c16e...", "branch": "dev", "dirty": false },
  "swiftVersion": "Apple Swift version 6.3.2 (...)",
  "buildConfiguration": "release",
  "options": { "jobs": 14, "resume": false, "limit": 5, "functions": [], "timezones": [] },
  "counts": {
    "total": 228, "ok": 228, "error": 0, "decodeError": 0, "skipped": 0,
    "byFunction": { "meal": { "ok": 60, "error": 0, "decodeError": 0 } },
    "byTimezone": { "America/Chicago": { "ok": 20, "error": 0, "decodeError": 0 } }
  },
  "nondeterministicKeys": { "determineBasal": ["id"] },
  "nondeterministicEnvelopeKeys": ["durationSeconds"]
}

Empty functions / timezones mean "everything found".

Keys that don't repeat between runs

Two things vary run to run, and the manifest declares both so a comparer can skip them without hardcoding anything:

• nondeterministicKeys — keys inside output, per function. DeterminationGenerator mints a fresh UUID() for Determination.id on every call, so determineBasal outputs are never byte-stable. Nothing else in the algorithm reads ambient state.
• nondeterministicEnvelopeKeys — keys of the envelope itself. durationSeconds is timing, not a property of the algorithm.

Ignore exactly those two sets and two runs of the same revision compare equal, including across different --jobs values.

If you'd rather have trees a plain diff -r can handle, pass --strip-nondeterministic, which nulls those keys instead of declaring them. It's off by default because it discards real information.

Development notes

• Don't modify anything under Trio/. Switching it between revisions is the whole point, so it has to stay otherwise pristine — and compare-branches.py won't run at all against a dirty tree.
• run-worker is a real subcommand but is deliberately left out of --help; it's how run applies TZ, not something to invoke by hand.
• --force-timezone on run-one exists to replay a record under a zone that isn't its own. It's a test hook for the timezone regression described above, not something you want in normal use.