From 71bd2636b99939a00f85ad334abf4e8090867f96 Mon Sep 17 00:00:00 2001 From: Chris Barber Date: Mon, 3 Aug 2026 14:37:46 -0500 Subject: [PATCH 1/7] Document database-wide-only transaction log deletion on RocksDB delete_transaction_logs_before now rejects table-scoped requests on RocksDB (HarperFast/harper#2049), and the deprecated delete_audit_logs_before always errors there since it requires table. Co-Authored-By: Claude Fable 5 --- reference/database/transaction.md | 22 +++++++++++++++++++++- 1 file changed, 21 insertions(+), 1 deletion(-) diff --git a/reference/database/transaction.md b/reference/database/transaction.md index 7af4f925..1d45faad 100644 --- a/reference/database/transaction.md +++ b/reference/database/transaction.md @@ -116,12 +116,14 @@ The `original_records` field contains the record state before the operation was #### `delete_audit_logs_before` -Deletes audit log entries older than the specified timestamp. +Deletes audit log entries older than the specified timestamp. Deprecated in favor of [`delete_transaction_logs_before`](#delete_transaction_logs_before). — Audit log cleanup improved to reduce resource consumption during scheduled cleanups — Storage reclamation: Harper automatically evicts older audit log entries when free storage drops below a configurable threshold + — This operation requires `table`, but on the RocksDB storage engine (the default) history cannot be deleted for a single table because all tables in a database share one transaction log. On RocksDB this operation now always returns an error directing you to `delete_transaction_logs_before`; it remains usable on LMDB. + ```json { "operation": "delete_audit_logs_before", @@ -131,6 +133,24 @@ Deletes audit log entries older than the specified timestamp. } ``` +#### `delete_transaction_logs_before` + +Deletes transaction log entries older than the specified timestamp. + + + +On RocksDB (the default storage engine), deletion is database-wide: all tables in a database share one transaction log, so omit `table` and pass only `database` (or `schema`) and `timestamp`. + + — On RocksDB, a request that includes `table` is now rejected with a `400` error, and a `table` that does not exist returns a `404`. Previously the `table` scope was silently ignored and the entire database's transaction log was purged. On LMDB, `table` scopes the deletion to that table's history, unchanged. + +```json +{ + "operation": "delete_transaction_logs_before", + "database": "dev", + "timestamp": 1598290282817 +} +``` + --- ## Enabling Audit Log Per Table From 220040284a88fd6857abc99c6ef059241c2989c3 Mon Sep 17 00:00:00 2001 From: Chris Barber Date: Mon, 3 Aug 2026 15:35:21 -0500 Subject: [PATCH 2/7] Add Transaction Log Operations section header Co-Authored-By: Claude Fable 5 --- reference/database/transaction.md | 2 ++ 1 file changed, 2 insertions(+) diff --git a/reference/database/transaction.md b/reference/database/transaction.md index 1d45faad..8ecbdb95 100644 --- a/reference/database/transaction.md +++ b/reference/database/transaction.md @@ -133,6 +133,8 @@ Deletes audit log entries older than the specified timestamp. Deprecated in favo } ``` +### Transaction Log Operations + #### `delete_transaction_logs_before` Deletes transaction log entries older than the specified timestamp. From f3d66404c2b9d507527793509ef79c279fd242f3 Mon Sep 17 00:00:00 2001 From: Chris Barber Date: Mon, 3 Aug 2026 16:01:35 -0500 Subject: [PATCH 3/7] Clarify engine scope of the 404 change and document cleanup_deleted_records Co-Authored-By: Claude Fable 5 --- reference/database/transaction.md | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-) diff --git a/reference/database/transaction.md b/reference/database/transaction.md index 8ecbdb95..abf882fc 100644 --- a/reference/database/transaction.md +++ b/reference/database/transaction.md @@ -143,7 +143,9 @@ Deletes transaction log entries older than the specified timestamp. On RocksDB (the default storage engine), deletion is database-wide: all tables in a database share one transaction log, so omit `table` and pass only `database` (or `schema`) and `timestamp`. - — On RocksDB, a request that includes `table` is now rejected with a `400` error, and a `table` that does not exist returns a `404`. Previously the `table` scope was silently ignored and the entire database's transaction log was purged. On LMDB, `table` scopes the deletion to that table's history, unchanged. + — On RocksDB, a request that includes `table` is now rejected with a `400` error; previously the `table` scope was silently ignored and the entire database's transaction log was purged. On either engine, a `table` that does not exist now returns a `404` (previously this was a silent no-op on LMDB). On LMDB, a valid `table` continues to scope the deletion to that table's history, unchanged. + +On LMDB only, the optional `cleanup_deleted_records` (boolean) parameter additionally removes leftover tombstone entries for records deleted before the timestamp — a repair step for tombstones that normal audit log cleanup should already have removed. It is ignored on RocksDB. ```json { From ffe30c4463a26ed38b483b970cc1c9936e75e912 Mon Sep 17 00:00:00 2001 From: Chris Barber Date: Tue, 4 Aug 2026 00:25:22 -0500 Subject: [PATCH 4/7] Address review: job semantics, storage-model consistency, badge placement, type style Co-Authored-By: Claude Fable 5 --- reference/database/transaction.md | 43 +++++++++++++++++++------------ 1 file changed, 27 insertions(+), 16 deletions(-) diff --git a/reference/database/transaction.md b/reference/database/transaction.md index abf882fc..4c4b8d45 100644 --- a/reference/database/transaction.md +++ b/reference/database/transaction.md @@ -11,17 +11,17 @@ title: Transaction Logging # Transaction Logging -Harper provides two complementary mechanisms for recording a history of data changes on a table: the **audit log** and the **transaction log**. Both are available at the table level and serve different use cases. - -| Feature | Audit Log | Transaction Log | -| ----------------------------- | --------------------------------- | ------------------------------ | -| Storage | Standard Harper table (per-table) | Clustering streams (per-table) | -| Requires clustering | No | Yes | -| Available since | v4.1.0 | v4.1.0 | -| Stores original record values | Yes | No | -| Query by username | Yes | No | -| Query by primary key | Yes | No | -| Used for real-time messaging | Yes (required) | No | +Harper provides two complementary mechanisms for recording a history of data changes on a table: the **audit log** and the **transaction log**. Both record and expose history for individual tables, but their underlying storage differs: on RocksDB (the default storage engine) the transaction log is physically a single database-wide log shared by all tables — history is read per table, while deletion operates on the whole database's log. + +| Feature | Audit Log | Transaction Log | +| ----------------------------- | --------------------------------- | ----------------------------------------------------------------------- | +| Storage | Standard Harper table (per-table) | Shared per-database log (RocksDB); clustering streams, per-table (LMDB) | +| Requires clustering | No | Yes | +| Available since | v4.1.0 | v4.1.0 | +| Stores original record values | Yes | No | +| Query by username | Yes | No | +| Query by primary key | Yes | No | +| Used for real-time messaging | Yes (required) | No | ## Audit Log @@ -122,7 +122,7 @@ Deletes audit log entries older than the specified timestamp. Deprecated in favo — Storage reclamation: Harper automatically evicts older audit log entries when free storage drops below a configurable threshold - — This operation requires `table`, but on the RocksDB storage engine (the default) history cannot be deleted for a single table because all tables in a database share one transaction log. On RocksDB this operation now always returns an error directing you to `delete_transaction_logs_before`; it remains usable on LMDB. + — This operation is unsupported on the RocksDB storage engine (the default): it requires `table`, but history cannot be deleted for a single table because all tables in a database share one transaction log. For an existing table the job fails with an error directing you to `delete_transaction_logs_before`; for a nonexistent table the job fails with a not-found error. The operation remains usable on LMDB. ```json { @@ -137,15 +137,17 @@ Deletes audit log entries older than the specified timestamp. Deprecated in favo #### `delete_transaction_logs_before` -Deletes transaction log entries older than the specified timestamp. - + — On RocksDB, a request that includes `table` now fails; previously the `table` scope was silently ignored and the entire database's transaction log was purged. On either engine, a `table` that does not exist now fails with a not-found error (previously this was a silent no-op on LMDB). On LMDB, a valid `table` continues to scope the deletion to that table's history, unchanged. + +Deletes transaction log entries older than the specified timestamp. + On RocksDB (the default storage engine), deletion is database-wide: all tables in a database share one transaction log, so omit `table` and pass only `database` (or `schema`) and `timestamp`. - — On RocksDB, a request that includes `table` is now rejected with a `400` error; previously the `table` scope was silently ignored and the entire database's transaction log was purged. On either engine, a `table` that does not exist now returns a `404` (previously this was a silent no-op on LMDB). On LMDB, a valid `table` continues to scope the deletion to that table's history, unchanged. +`cleanup_deleted_records`: `boolean` (optional) — LMDB only; additionally removes leftover tombstone entries for records deleted before the timestamp, a repair step for tombstones that normal audit log cleanup should already have removed. Ignored on RocksDB. -On LMDB only, the optional `cleanup_deleted_records` (boolean) parameter additionally removes leftover tombstone entries for records deleted before the timestamp — a repair step for tombstones that normal audit log cleanup should already have removed. It is ignored on RocksDB. +The operation runs as a background job: the request itself returns `200` with a job ID, and any rejection surfaces when the job runs. Poll [`get_job`](jobs.md) to observe the outcome — a rejected request ends with job status `ERROR` and a message describing the failure (for example, the RocksDB table-scope rejection). ```json { @@ -155,6 +157,15 @@ On LMDB only, the optional `cleanup_deleted_records` (boolean) parameter additio } ``` +Response: + +```json +{ + "message": "Starting job with id 2fe25039-566e-4670-8bb3-2db3d4e07e69", + "job_id": "2fe25039-566e-4670-8bb3-2db3d4e07e69" +} +``` + --- ## Enabling Audit Log Per Table From 5cc6c517e32ccc7fcd488b7169bb10be70abc0dc Mon Sep 17 00:00:00 2001 From: Chris Barber Date: Tue, 4 Aug 2026 12:28:10 -0500 Subject: [PATCH 5/7] Address review round 2: LMDB example gap, WAL clustering row, data-safety admonition, params/results, get_job error example, 5.2 release note Co-Authored-By: Claude Fable 5 --- reference/database/transaction.md | 45 +++++++++++++++++++++++++++---- release-notes/v5-lincoln/5.2.md | 10 +++++++ 2 files changed, 50 insertions(+), 5 deletions(-) diff --git a/reference/database/transaction.md b/reference/database/transaction.md index 4c4b8d45..c8328366 100644 --- a/reference/database/transaction.md +++ b/reference/database/transaction.md @@ -16,7 +16,7 @@ Harper provides two complementary mechanisms for recording a history of data cha | Feature | Audit Log | Transaction Log | | ----------------------------- | --------------------------------- | ----------------------------------------------------------------------- | | Storage | Standard Harper table (per-table) | Shared per-database log (RocksDB); clustering streams, per-table (LMDB) | -| Requires clustering | No | Yes | +| Requires clustering | No | No (RocksDB, native WAL); Yes (LMDB, clustering streams) | | Available since | v4.1.0 | v4.1.0 | | Stores original record values | Yes | No | | Query by username | Yes | No | @@ -139,15 +139,26 @@ Deletes audit log entries older than the specified timestamp. Deprecated in favo - — On RocksDB, a request that includes `table` now fails; previously the `table` scope was silently ignored and the entire database's transaction log was purged. On either engine, a `table` that does not exist now fails with a not-found error (previously this was a silent no-op on LMDB). On LMDB, a valid `table` continues to scope the deletion to that table's history, unchanged. + — On RocksDB, a request that includes `table` now fails; previously the `table` scope was silently ignored and the entire database's transaction log was purged. On either engine, a `table` that does not exist now fails with a not-found error (previously a typo'd `table` fell through to the database-wide purge on RocksDB, and was a silent no-op on LMDB). On LMDB, a valid `table` continues to scope the deletion to that table's history, unchanged. Deletes transaction log entries older than the specified timestamp. -On RocksDB (the default storage engine), deletion is database-wide: all tables in a database share one transaction log, so omit `table` and pass only `database` (or `schema`) and `timestamp`. +:::warning Database-wide and irreversible +On RocksDB (the default storage engine), deletion is database-wide: all tables in a database share one transaction log, so omit `table` and pass only `database` (or `schema`) and `timestamp`. This deletes whole log files, permanently removing [`read_audit_log`](#read_audit_log) history below the timestamp for **every table in the database** — there is no per-table survivor and no undo. The only recovery route is a [backup](../backups/overview.md), which restores the transaction log alongside the data. Purging below a lagging replica's catch-up position does not lose data on that replica — the sender detects that the requested start predates its retained history and forces a full base copy instead of incremental catch-up — but that full resync is far more expensive than incremental replication, so avoid purging below your slowest replica's position. +::: -`cleanup_deleted_records`: `boolean` (optional) — LMDB only; additionally removes leftover tombstone entries for records deleted before the timestamp, a repair step for tombstones that normal audit log cleanup should already have removed. Ignored on RocksDB. +Parameters: -The operation runs as a background job: the request itself returns `200` with a job ID, and any rejection surfaces when the job runs. Poll [`get_job`](jobs.md) to observe the outcome — a rejected request ends with job status `ERROR` and a message describing the failure (for example, the RocksDB table-scope rejection). +- `database` (or the deprecated `schema` alias): `string` (required) — a request naming neither fails validation before a job starts; a `database` that does not exist fails the job with a not-found error. +- `timestamp`: `number` (required) — epoch milliseconds; entries older than this are deleted. +- `table`: `string` (LMDB only) — scopes deletion to that table's history. On RocksDB the job fails (see the warning above). +- `cleanup_deleted_records`: `boolean` (optional) — LMDB only; additionally removes leftover tombstone entries for records deleted before the timestamp, a repair step for tombstones that normal audit log cleanup should already have removed. Ignored on RocksDB. + +On LMDB, the table-scoped deletion scans the database's full audit history (and `cleanup_deleted_records: true` adds a second full scan of the table's records), so the cost grows with total history depth — schedule accordingly on databases with deep audit history. + +The operation runs as a background job: the request itself returns `200` with a job ID, and any rejection surfaces when the job runs. Poll [`get_job`](jobs.md#get-job) to observe the outcome — a rejected request ends with job status `ERROR` and a message describing the failure (for example, the RocksDB table-scope rejection). + +**RocksDB (database-wide — omit `table`):** ```json { @@ -157,6 +168,17 @@ The operation runs as a background job: the request itself returns `200` with a } ``` +**LMDB (`table` is required — omitting it deletes nothing and reports `entries_deleted: 0` with job status `COMPLETE`):** + +```json +{ + "operation": "delete_transaction_logs_before", + "database": "dev", + "table": "dog", + "timestamp": 1598290282817 +} +``` + Response: ```json @@ -166,6 +188,19 @@ Response: } ``` +`get_job` reports the outcome. A successful job's `result` carries `log_files_deleted` and `entries_deleted` (plus `start_timestamp`/`end_timestamp` on LMDB) — the record of how much was deleted. A rejected request looks like: + +```json +[ + { + "id": "2fe25039-566e-4670-8bb3-2db3d4e07e69", + "type": "delete_transaction_logs_before", + "status": "ERROR", + "message": "There was an error running deleteTransactionLogsBefore job with id 2fe25039-566e-4670-8bb3-2db3d4e07e69 - Table-level transaction log deletion is not supported for RocksDB tables because all tables in a database share one transaction log; to delete the transaction logs for the entire 'dev' database, use delete_transaction_logs_before with only 'database' and 'timestamp'" + } +] +``` + --- ## Enabling Audit Log Per Table diff --git a/release-notes/v5-lincoln/5.2.md b/release-notes/v5-lincoln/5.2.md index 1097504a..c28d4ea1 100644 --- a/release-notes/v5-lincoln/5.2.md +++ b/release-notes/v5-lincoln/5.2.md @@ -14,6 +14,16 @@ New managed-backup operations for RocksDB databases: `create_backup`, `list_back `get_backup` now understands RocksDB: it streams a full-snapshot `tar` of the database — including any file-backed blobs (pass `exclude_blobs: true` to omit them), gzipped by default (pass `gzip: false` for a plain `tar`) — instead of failing. The LMDB behavior (streaming the `.mdb`, with `table`/`tables`/`include_audit`) is unchanged. +## Transaction Log Deletion + +`delete_transaction_logs_before` no longer accepts a `table` scope on RocksDB (the default storage engine): all tables in a database share one transaction log, and the previous behavior silently ignored `table` and purged the **entire database's** log while reporting success ([harper#2049](https://github.com/HarperFast/harper/issues/2049)). Three behavior changes may affect scheduled retention jobs on upgrade: + +- On RocksDB, a request naming a `table` now fails with an error directing you to the database-wide form (`database` + `timestamp` only). +- The deprecated `delete_audit_logs_before` operation requires `table`, so on RocksDB it now always fails with the same guidance; it remains usable on LMDB. +- On either engine, a `table` that does not exist now fails with a not-found error (previously a silent no-op on LMDB, and the database-wide purge on RocksDB). + +See [Transaction Logging](/reference/v5/database/transaction) for the updated operation reference. + ## Querying ### Filtered Vector Search (Predicate-Aware HNSW Traversal) From 98b1a99e71bb72820256b135503e5af8217e5653 Mon Sep 17 00:00:00 2001 From: Chris Barber Date: Fri, 7 Aug 2026 17:21:11 -0500 Subject: [PATCH 6/7] Collapse audit-log/transaction-log distinction; fix conflicting messaging MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Per @kriszyp's review: there is no separate "audit log" — it and the transaction log are one thing (a NATS-era relic). Remove the comparison table and the "two complementary mechanisms" framing; the page now states there is one transaction log per database and adds an :::info callout noting read_audit_log / delete_audit_logs_before / logging.auditLog / @table(audit:) are legacy names for that single log. Merge the Audit/Transaction "Operations" split into one section. Codex threads: - transaction.md: distinguish synchronous request-validation failure (missing database/schema, no job id) from operation-time failures via get_job. - transaction.md: Related-docs Replication line no longer claims clustering is required for the transaction log (RocksDB native WAL). - 5.2.md: delete_audit_logs_before "always fails with the same guidance" qualified to an existing table (nonexistent table 404s first). Co-Authored-By: Claude Opus 4.8 (1M context) --- reference/database/transaction.md | 48 ++++++++++++------------------- release-notes/v5-lincoln/5.2.md | 2 +- 2 files changed, 20 insertions(+), 30 deletions(-) diff --git a/reference/database/transaction.md b/reference/database/transaction.md index c8328366..46cb9721 100644 --- a/reference/database/transaction.md +++ b/reference/database/transaction.md @@ -11,33 +11,25 @@ title: Transaction Logging # Transaction Logging -Harper provides two complementary mechanisms for recording a history of data changes on a table: the **audit log** and the **transaction log**. Both record and expose history for individual tables, but their underlying storage differs: on RocksDB (the default storage engine) the transaction log is physically a single database-wide log shared by all tables — history is read per table, while deletion operates on the whole database's log. +Harper maintains a **transaction log** for every database: a record of every data change, capturing the operation type, the user who made the change, the timestamp, and both the new and original record values. There is one transaction log per database, shared by all tables. -| Feature | Audit Log | Transaction Log | -| ----------------------------- | --------------------------------- | ----------------------------------------------------------------------- | -| Storage | Standard Harper table (per-table) | Shared per-database log (RocksDB); clustering streams, per-table (LMDB) | -| Requires clustering | No | No (RocksDB, native WAL); Yes (LMDB, clustering streams) | -| Available since | v4.1.0 | v4.1.0 | -| Stores original record values | Yes | No | -| Query by username | Yes | No | -| Query by primary key | Yes | No | -| Used for real-time messaging | Yes (required) | No | - -## Audit Log +:::info Audit log and transaction log are the same thing +Some operations and settings still carry the older "audit log" name — `read_audit_log`, `delete_audit_logs_before`, the [`logging.auditLog`](../logging/configuration.md) setting, and the `@table(audit:)` directive — but they all act on this single transaction log. The "audit log" is not a separate mechanism; the distinct-log terminology is a historical artifact. +::: Available since: v4.1.0 -The audit log is a data store that tracks every transaction across all tables in a database. Harper automatically creates and maintains a single audit log per database. The audit log captures the operation type, the user who made the change, the timestamp, and both the new and original record values. +On RocksDB (the default storage engine) the transaction log is physically a single database-wide log shared by all tables: history is read per table, while deletion operates on the whole database's log. -The audit log is **enabled by default**. To disable it, set [`logging.auditLog`](../logging/configuration.md) to `false` in `harper-config.yaml` and restart Harper. +The transaction log is **enabled by default**. To disable it, set [`logging.auditLog`](../logging/configuration.md) to `false` in `harper-config.yaml` and restart Harper. -> The audit log is required for real-time messaging (WebSocket and MQTT subscriptions) and replication. Do not disable it if real-time features or replication are in use. +> The transaction log is required for real-time messaging (WebSocket and MQTT subscriptions) and replication. Do not disable it if real-time features or replication are in use. -### Audit Log Operations +## Operations -#### `read_audit_log` +### `read_audit_log` -Queries the audit log for a specific table. Supports filtering by timestamp, username, or primary key value. +Queries the transaction log for a specific table. Supports filtering by timestamp, username, or primary key value. **By timestamp:** @@ -114,9 +106,9 @@ Timestamp behavior: The `original_records` field contains the record state before the operation was applied. -#### `delete_audit_logs_before` +### `delete_audit_logs_before` -Deletes audit log entries older than the specified timestamp. Deprecated in favor of [`delete_transaction_logs_before`](#delete_transaction_logs_before). +Deletes transaction log entries older than the specified timestamp. Deprecated in favor of [`delete_transaction_logs_before`](#delete_transaction_logs_before). — Audit log cleanup improved to reduce resource consumption during scheduled cleanups @@ -133,9 +125,7 @@ Deletes audit log entries older than the specified timestamp. Deprecated in favo } ``` -### Transaction Log Operations - -#### `delete_transaction_logs_before` +### `delete_transaction_logs_before` @@ -156,7 +146,7 @@ Parameters: On LMDB, the table-scoped deletion scans the database's full audit history (and `cleanup_deleted_records: true` adds a second full scan of the table's records), so the cost grows with total history depth — schedule accordingly on databases with deep audit history. -The operation runs as a background job: the request itself returns `200` with a job ID, and any rejection surfaces when the job runs. Poll [`get_job`](jobs.md#get-job) to observe the outcome — a rejected request ends with job status `ERROR` and a message describing the failure (for example, the RocksDB table-scope rejection). +Request validation runs first and synchronously: a request that omits both `database` and `schema` is rejected immediately with an error and no job ID (there is nothing to poll). Once accepted, the operation runs as a background job that returns `200` with a job ID, and operation-time failures surface through [`get_job`](jobs.md#get-job) — the job ends with status `ERROR` and a message describing the failure (for example, the RocksDB table-scope rejection, or a `table`/`database` that does not exist). **RocksDB (database-wide — omit `table`):** @@ -203,9 +193,9 @@ Response: --- -## Enabling Audit Log Per Table +## Enabling the Transaction Log Per Table -You can enable or disable the audit log for individual tables using the `@table` directive's `audit` argument in your schema: +You can enable or disable the transaction log for individual tables using the `@table` directive's `audit` argument in your schema: ```graphql type Dog @table(audit: true) { @@ -218,7 +208,7 @@ This overrides the [`logging.auditLog`](../logging/configuration.md) global conf ## Related Documentation -- [Logging](../logging/overview.md) — Application and system logging (separate from transaction/audit logging) -- [Replication](../replication/overview.md) — Clustering setup required for transaction logs -- [Logging Configuration](../logging/configuration.md) — Global audit log configuration (`logging.auditLog`) +- [Logging](../logging/overview.md) — Application and system logging (separate from the transaction log) +- [Replication](../replication/overview.md) — Replication and clustering, which consume the transaction log +- [Logging Configuration](../logging/configuration.md) — Global transaction log configuration (`logging.auditLog`) - [Operations API](../operations-api/overview.md) — Sending operations to Harper diff --git a/release-notes/v5-lincoln/5.2.md b/release-notes/v5-lincoln/5.2.md index c28d4ea1..7acb10bd 100644 --- a/release-notes/v5-lincoln/5.2.md +++ b/release-notes/v5-lincoln/5.2.md @@ -19,7 +19,7 @@ New managed-backup operations for RocksDB databases: `create_backup`, `list_back `delete_transaction_logs_before` no longer accepts a `table` scope on RocksDB (the default storage engine): all tables in a database share one transaction log, and the previous behavior silently ignored `table` and purged the **entire database's** log while reporting success ([harper#2049](https://github.com/HarperFast/harper/issues/2049)). Three behavior changes may affect scheduled retention jobs on upgrade: - On RocksDB, a request naming a `table` now fails with an error directing you to the database-wide form (`database` + `timestamp` only). -- The deprecated `delete_audit_logs_before` operation requires `table`, so on RocksDB it now always fails with the same guidance; it remains usable on LMDB. +- The deprecated `delete_audit_logs_before` operation requires `table`, so on RocksDB it now fails for an existing table with the same guidance (a nonexistent table fails with the not-found error described below); it remains usable on LMDB. - On either engine, a `table` that does not exist now fails with a not-found error (previously a silent no-op on LMDB, and the database-wide purge on RocksDB). See [Transaction Logging](/reference/v5/database/transaction) for the updated operation reference. From a7ad4a01b7ba38adc5d66650cb674ecb77f1635b Mon Sep 17 00:00:00 2001 From: Chris Barber Date: Fri, 7 Aug 2026 18:21:09 -0500 Subject: [PATCH 7/7] Address Codex review: fix LMDB result fields, cross-page log terminology, config default MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Codex pass on PR 618 surfaced five findings (no blockers); verified each against harper / rocksdb-js source: - transaction.md: the delete_transaction_logs_before result never carries start_timestamp/end_timestamp (DeleteTransactionLogsBeforeResults is constructed with no args; the LMDB branch only sets entries_deleted). Corrected the field list and noted log_files_deleted is 0 on LMDB. - transaction.md: relabel the get_job example "A rejected request" -> "A failed job" (it's an accepted-then-failed job, not request validation). - transaction.md: soften the RocksDB warning — purgeLogs deletes whole log files (file granularity), so "history below the timestamp" overstated entry-level precision. - Cross-page terminology (kriszyp's directive, extended): drop the audit-log-vs-transaction-log two-log framing in database/overview.md, database/api.md, operations-api/operations.md, and logging/operations.md. - logging/configuration.md: logging.auditLog Default false -> true, matching the shipped default (static/defaultConfig.yaml) and transaction.md. Co-Authored-By: Claude Opus 4.8 (1M context) --- reference/database/api.md | 2 +- reference/database/overview.md | 2 +- reference/database/transaction.md | 4 ++-- reference/logging/configuration.md | 2 +- reference/logging/operations.md | 2 +- reference/operations-api/operations.md | 16 ++++++++-------- 6 files changed, 14 insertions(+), 14 deletions(-) diff --git a/reference/database/api.md b/reference/database/api.md index 1faff146..ef943cfa 100644 --- a/reference/database/api.md +++ b/reference/database/api.md @@ -305,5 +305,5 @@ When a field is typed as `Blob` in the schema, any string or buffer assigned via - [Schema](./schema.md) — Defining tables and blob fields - [Resource API](../resources/resource-api.md) — Full table class method reference -- [Transaction Logging](./transaction.md) — Audit log and transaction log for data change history +- [Transaction Logging](./transaction.md) — The transaction log for data change history - [Configuration](../configuration/options.md) — Blob storage path configuration diff --git a/reference/database/overview.md b/reference/database/overview.md index b7269f85..406b251a 100644 --- a/reference/database/overview.md +++ b/reference/database/overview.md @@ -114,7 +114,7 @@ For deeper coverage of each database feature, see the dedicated pages in this se - **[Jobs](./jobs.md)** — Asynchronous bulk data operations (CSV import/export, S3 import/export) - **[System Tables](./system-tables.md)** — Harper internal tables for analytics, data loader state, and other system features - **[Compaction](./compaction.md)** — Reducing database file size by eliminating fragmentation and free space -- **[Transaction Logging](./transaction.md)** — Recording and querying a history of data changes via audit log and transaction log +- **[Transaction Logging](./transaction.md)** — Recording and querying a history of data changes in the transaction log ## Related Documentation diff --git a/reference/database/transaction.md b/reference/database/transaction.md index 46cb9721..baba4612 100644 --- a/reference/database/transaction.md +++ b/reference/database/transaction.md @@ -134,7 +134,7 @@ Deletes transaction log entries older than the specified timestamp. Deprecated i Deletes transaction log entries older than the specified timestamp. :::warning Database-wide and irreversible -On RocksDB (the default storage engine), deletion is database-wide: all tables in a database share one transaction log, so omit `table` and pass only `database` (or `schema`) and `timestamp`. This deletes whole log files, permanently removing [`read_audit_log`](#read_audit_log) history below the timestamp for **every table in the database** — there is no per-table survivor and no undo. The only recovery route is a [backup](../backups/overview.md), which restores the transaction log alongside the data. Purging below a lagging replica's catch-up position does not lose data on that replica — the sender detects that the requested start predates its retained history and forces a full base copy instead of incremental catch-up — but that full resync is far more expensive than incremental replication, so avoid purging below your slowest replica's position. +On RocksDB (the default storage engine), deletion is database-wide: all tables in a database share one transaction log, so omit `table` and pass only `database` (or `schema`) and `timestamp`. This purges whole log files whose entries predate the timestamp, removing [`read_audit_log`](#read_audit_log) history for **every table in the database** — there is no per-table survivor and no undo. The only recovery route is a [backup](../backups/overview.md), which restores the transaction log alongside the data. Purging below a lagging replica's catch-up position does not lose data on that replica — the sender detects that the requested start predates its retained history and forces a full base copy instead of incremental catch-up — but that full resync is far more expensive than incremental replication, so avoid purging below your slowest replica's position. ::: Parameters: @@ -178,7 +178,7 @@ Response: } ``` -`get_job` reports the outcome. A successful job's `result` carries `log_files_deleted` and `entries_deleted` (plus `start_timestamp`/`end_timestamp` on LMDB) — the record of how much was deleted. A rejected request looks like: +`get_job` reports the outcome. A successful job's `result` carries `entries_deleted` and `log_files_deleted` (the count of purged log files on RocksDB; `0` on LMDB, which has no separate log files) — the record of how much was deleted. A failed job looks like: ```json [ diff --git a/reference/logging/configuration.md b/reference/logging/configuration.md index 4a4b4ac1..14e7d774 100644 --- a/reference/logging/configuration.md +++ b/reference/logging/configuration.md @@ -98,7 +98,7 @@ logging: Type: `boolean` -Default: `false` +Default: `true` Enables audit (table transaction) logging. When enabled, Harper records every insert, update, and delete to a corresponding audit table. Audit log data is accessed via the `read_audit_log` operation. diff --git a/reference/logging/operations.md b/reference/logging/operations.md index de149d62..95d60db5 100644 --- a/reference/logging/operations.md +++ b/reference/logging/operations.md @@ -8,7 +8,7 @@ title: Logging Operations Operations for reading the standard Harper log (`hdb.log`). All operations are restricted to `super_user` roles only. -> Audit log and transaction log operations (`read_audit_log`, `read_transaction_log`, `delete_audit_logs_before`, `delete_transaction_logs_before`) are documented in [Database / Transaction Logging](../database/transaction.md). +> Transaction log operations (`read_audit_log`, `read_transaction_log`, `delete_audit_logs_before`, `delete_transaction_logs_before`) are documented in [Database / Transaction Logging](../database/transaction.md). --- diff --git a/reference/operations-api/operations.md b/reference/operations-api/operations.md index 72adbfd6..db0d3df0 100644 --- a/reference/operations-api/operations.md +++ b/reference/operations-api/operations.md @@ -1163,13 +1163,13 @@ Operations for reading Harper logs. Detailed documentation: [Logging Operations](../logging/operations.md) -| Operation | Description | Role Required | -| -------------------------------- | ---------------------------------------------------------------------- | ------------- | -| `read_log` | Returns entries from the primary `hdb.log` | super_user | -| `read_transaction_log` | Returns transaction history for a table | super_user | -| `delete_transaction_logs_before` | Deletes transaction log entries older than a timestamp | super_user | -| `read_audit_log` | Returns verbose audit history for a table (requires audit log enabled) | super_user | -| `delete_audit_logs_before` | Deletes audit log entries older than a timestamp | super_user | +| Operation | Description | Role Required | +| -------------------------------- | ------------------------------------------------------------------------------------------------------------------------ | ------------- | +| `read_log` | Returns entries from the primary `hdb.log` | super_user | +| `read_transaction_log` | Returns transaction history for a table | super_user | +| `delete_transaction_logs_before` | Deletes transaction log entries older than a timestamp | super_user | +| `read_audit_log` | Returns verbose transaction history for a table, including original record values (requires transaction logging enabled) | super_user | +| `delete_audit_logs_before` | Deletes transaction log entries older than a timestamp (deprecated alias of `delete_transaction_logs_before`) | super_user | ### `read_log` @@ -1199,7 +1199,7 @@ Returns transaction history for a specific table. Optionally filter by `from`/`t ### `read_audit_log` -Returns verbose audit history including original record state. Requires `logging.auditLog: true` in configuration. Filter by `search_type`: `hash_value`, `timestamp`, or `username`. +Returns verbose transaction history including original record state. Requires transaction logging (`logging.auditLog: true`) in configuration. Filter by `search_type`: `hash_value`, `timestamp`, or `username`. ```json {