Transaction log deletion is database-wide on RocksDB; table-scoped requests are rejected

reference/database/transaction.md

@@ -11,17 +11,17 @@ title: Transaction Logging
 
 # Transaction Logging
 
-Harper provides two complementary mechanisms for recording a history of data changes on a table: the **audit log** and the **transaction log**. Both are available at the table level and serve different use cases.
-
-| Feature                       | Audit Log                         | Transaction Log                |
-| ----------------------------- | --------------------------------- | ------------------------------ |
-| Storage                       | Standard Harper table (per-table) | Clustering streams (per-table) |
-| Requires clustering           | No                                | Yes                            |
-| Available since               | v4.1.0                            | v4.1.0                         |
-| Stores original record values | Yes                               | No                             |
-| Query by username             | Yes                               | No                             |
-| Query by primary key          | Yes                               | No                             |
-| Used for real-time messaging  | Yes (required)                    | No                             |
+Harper provides two complementary mechanisms for recording a history of data changes on a table: the **audit log** and the **transaction log**. Both record and expose history for individual tables, but their underlying storage differs: on RocksDB (the default storage engine) the transaction log is physically a single database-wide log shared by all tables — history is read per table, while deletion operates on the whole database's log.
+
+| Feature                       | Audit Log                         | Transaction Log                                                         |
+| ----------------------------- | --------------------------------- | ----------------------------------------------------------------------- |
+| Storage                       | Standard Harper table (per-table) | Shared per-database log (RocksDB); clustering streams, per-table (LMDB) |
+| Requires clustering           | No                                | No (RocksDB, native WAL); Yes (LMDB, clustering streams)                |
+| Available since               | v4.1.0                            | v4.1.0                                                                  |
+| Stores original record values | Yes                               | No                                                                      |
+| Query by username             | Yes                               | No                                                                      |
+| Query by primary key          | Yes                               | No                                                                      |
+| Used for real-time messaging  | Yes (required)                    | No                                                                      |
 
 ## Audit Log
 
@@ -116,12 +116,14 @@ The `original_records` field contains the record state before the operation was
 
 #### `delete_audit_logs_before`
 
-Deletes audit log entries older than the specified timestamp.
+Deletes audit log entries older than the specified timestamp. Deprecated in favor of [`delete_transaction_logs_before`](#delete_transaction_logs_before).
 
 <VersionBadge type="changed" version="v4.3.0" /> — Audit log cleanup improved to reduce resource consumption during scheduled cleanups
 
 <VersionBadge type="changed" version="v4.5.0" /> — Storage reclamation: Harper automatically evicts older audit log entries when free storage drops below a configurable threshold
 
+<VersionBadge type="changed" version="v5.2.0" /> — This operation is unsupported on the RocksDB storage engine (the default): it requires `table`, but history cannot be deleted for a single table because all tables in a database share one transaction log. For an existing table the job fails with an error directing you to `delete_transaction_logs_before`; for a nonexistent table the job fails with a not-found error. The operation remains usable on LMDB.
+
 ```json
 {
 	"operation": "delete_audit_logs_before",
@@ -131,6 +133,74 @@ Deletes audit log entries older than the specified timestamp.
 }
 ```
 
+### Transaction Log Operations
+
+#### `delete_transaction_logs_before`
+
+<EngineBadge engines="RocksDB, LMDB" />
+
+<VersionBadge type="changed" version="v5.2.0" /> — On RocksDB, a request that includes `table` now fails; previously the `table` scope was silently ignored and the entire database's transaction log was purged. On either engine, a `table` that does not exist now fails with a not-found error (previously a typo'd `table` fell through to the database-wide purge on RocksDB, and was a silent no-op on LMDB). On LMDB, a valid `table` continues to scope the deletion to that table's history, unchanged.
+
+Deletes transaction log entries older than the specified timestamp.
+
+:::warning Database-wide and irreversible
+On RocksDB (the default storage engine), deletion is database-wide: all tables in a database share one transaction log, so omit `table` and pass only `database` (or `schema`) and `timestamp`. This deletes whole log files, permanently removing [`read_audit_log`](#read_audit_log) history below the timestamp for **every table in the database** — there is no per-table survivor and no undo. The only recovery route is a [backup](../backups/overview.md), which restores the transaction log alongside the data. Purging below a lagging replica's catch-up position does not lose data on that replica — the sender detects that the requested start predates its retained history and forces a full base copy instead of incremental catch-up — but that full resync is far more expensive than incremental replication, so avoid purging below your slowest replica's position.
+:::
+
+Parameters:
+
+- `database` (or the deprecated `schema` alias): `string` (required) — a request naming neither fails validation before a job starts; a `database` that does not exist fails the job with a not-found error.
+- `timestamp`: `number` (required) — epoch milliseconds; entries older than this are deleted.
+- `table`: `string` (LMDB only) — scopes deletion to that table's history. On RocksDB the job fails (see the warning above).
+- `cleanup_deleted_records`: `boolean` (optional) — LMDB only; additionally removes leftover tombstone entries for records deleted before the timestamp, a repair step for tombstones that normal audit log cleanup should already have removed. Ignored on RocksDB.
+
+On LMDB, the table-scoped deletion scans the database's full audit history (and `cleanup_deleted_records: true` adds a second full scan of the table's records), so the cost grows with total history depth — schedule accordingly on databases with deep audit history.
+
+The operation runs as a background job: the request itself returns `200` with a job ID, and any rejection surfaces when the job runs. Poll [`get_job`](jobs.md#get-job) to observe the outcome — a rejected request ends with job status `ERROR` and a message describing the failure (for example, the RocksDB table-scope rejection).
+
+**RocksDB (database-wide — omit `table`):**
+
+```json
+{
+	"operation": "delete_transaction_logs_before",
+	"database": "dev",
+	"timestamp": 1598290282817
+}
+```
+
+**LMDB (`table` is required — omitting it deletes nothing and reports `entries_deleted: 0` with job status `COMPLETE`):**
+
+```json
+{
+	"operation": "delete_transaction_logs_before",
+	"database": "dev",
+	"table": "dog",
+	"timestamp": 1598290282817
+}
+```
+
+Response:
+
+```json
+{
+	"message": "Starting job with id 2fe25039-566e-4670-8bb3-2db3d4e07e69",
+	"job_id": "2fe25039-566e-4670-8bb3-2db3d4e07e69"
+}
+```
+
+`get_job` reports the outcome. A successful job's `result` carries `log_files_deleted` and `entries_deleted` (plus `start_timestamp`/`end_timestamp` on LMDB) — the record of how much was deleted. A rejected request looks like:
+
+```json
+[
+	{
+		"id": "2fe25039-566e-4670-8bb3-2db3d4e07e69",
+		"type": "delete_transaction_logs_before",
+		"status": "ERROR",
+		"message": "There was an error running deleteTransactionLogsBefore job with id 2fe25039-566e-4670-8bb3-2db3d4e07e69 - Table-level transaction log deletion is not supported for RocksDB tables because all tables in a database share one transaction log; to delete the transaction logs for the entire 'dev' database, use delete_transaction_logs_before with only 'database' and 'timestamp'"
+	}
+]
+```
+
 ---
 
 ## Enabling Audit Log Per Table

release-notes/v5-lincoln/5.2.md

@@ -14,6 +14,16 @@ New managed-backup operations for RocksDB databases: `create_backup`, `list_back
 
 `get_backup` now understands RocksDB: it streams a full-snapshot `tar` of the database — including any file-backed blobs (pass `exclude_blobs: true` to omit them), gzipped by default (pass `gzip: false` for a plain `tar`) — instead of failing. The LMDB behavior (streaming the `.mdb`, with `table`/`tables`/`include_audit`) is unchanged.
 
+## Transaction Log Deletion
+
+`delete_transaction_logs_before` no longer accepts a `table` scope on RocksDB (the default storage engine): all tables in a database share one transaction log, and the previous behavior silently ignored `table` and purged the **entire database's** log while reporting success ([harper#2049](https://github.com/HarperFast/harper/issues/2049)). Three behavior changes may affect scheduled retention jobs on upgrade:
+
+- On RocksDB, a request naming a `table` now fails with an error directing you to the database-wide form (`database` + `timestamp` only).
+- The deprecated `delete_audit_logs_before` operation requires `table`, so on RocksDB it now always fails with the same guidance; it remains usable on LMDB.
+- On either engine, a `table` that does not exist now fails with a not-found error (previously a silent no-op on LMDB, and the database-wide purge on RocksDB).
+
+See [Transaction Logging](/reference/v5/database/transaction) for the updated operation reference.
+
 ## Querying
 
 ### Filtered Vector Search (Predicate-Aware HNSW Traversal)