feat: record durable run history in a ClickHouse table - #7
Conversation
Review recordVerified: deletion scope untouched — #6's eleven scope mutations re-run against this branch, 11/11 caught. Three constraints checked by test, since this is bookkeeping attached to an irreversible operation:
Sink choice, for the record: not a PVC ( Open question for reviewers: |
|
Open question resolved: Rationale: the table is queryable in ClickHouse for exactly the cases where pod logs are gone, which is the point of the change. The No code change — the branch already defaults to on. |
Pod logs are not a record. The kubelet rotates container output (10Mi over 5
files by default), so `kubectl logs` cannot return the beginning of a long
run, and ttlSecondsAfterFinished deletes the Job and its pods along with
everything they printed. A cleanup that reclaimed terabytes left no evidence
of what it did once that window closed.
Each run now appends structured events to <COLLECTTABLEPREFIX><disk>_log,
beside the auxiliary table and NEVER truncated: phase start, throttled collect
progress, per-sample start, one row per confirmed delete batch, finish with
attempt totals, warnings and errors. Rows carry the scope the run was pointed
at -- bucket, prefix, disk, cluster, dry-run, ClickHouse host, hostname -- so a
row is self-describing evidence rather than a line of text. S3GC_RUNID defaults
to a generated timestamped id and the Job template passes JOB_NAME, so a row
traces back to the Job that wrote it.
ClickHouse is the sink rather than a volume or the bucket: the connection,
credentials and grants already exist; a readOnlyRootFilesystem container cannot
write a file and an emptyDir dies with the pod; and an object written under
S3PATH would be listed by the NEXT collect, found absent from
system.remote_data_paths, and become a deletion candidate -- s3gc would
garbage-collect its own logs.
Three constraints, each with a test, because this is bookkeeping attached to an
irreversible operation:
1. Writes go on ch_writer, never ch_client. do_use() holds ch_client's
session for the whole anti-join stream, and a second query on a held
session is SESSION_IS_LOCKED (373) -- the 0.6.0 defect, which would now
fire mid-delete at the worst possible moment.
2. A logging failure never fails the run. One failure disables the run log
for the remainder of the process rather than retrying every batch, and a
missing CREATE TABLE grant degrades to stdout only with one warning.
3. Messages are redacted through the existing LogFormatter._filter before
insert, so a credential cannot reach a table that outlives the run.
Collect also reports progress at INFO, throttled to every 100k objects. Per-
batch progress was DEBUG-only, so a multi-hour collect emitted about four lines
at --verbose, while --debug emits one line per object and on a large bucket
exceeds the kubelet's rotation limit, destroying the start of its own output.
Opt out with --runlog false / S3GC_RUNLOG_FLAG=false, or RUNLOG=false in the
renderer. Default on: durability is the point. Legacy env files render
unchanged with the run log enabled.
The deletion scope is untouched. All 11 scope mutations from the parent commit
remain caught.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
a7f3ad2 to
b011f37
Compare
1 participant
Summary
Adds a durable ClickHouse run log for collect, dry-run, and delete events. Log writes use a separate client, failures degrade safely to stdout, and messages are redacted.
Validation
Stacked on #6; followed by #9 and #10.