Skip to content

Latest commit

 

History

History
437 lines (367 loc) · 22.7 KB

database-boundary.md

File metadata and controls

437 lines (367 loc) · 22.7 KB
type Concept
title Database evidence and state-store boundary
okf_status active
tags
public
developer
freshness 2026-09-29
resource https://github.com/alphazede/bran
public_boundary public

Database evidence and state-store boundary

This is the design for issue #27. BRAN touches databases for two unrelated reasons, and they get two unrelated ports:

  1. Evidence source. Read-only access to a database catalog and, when policy allows it, bounded query results that enter BRAN provenance and packets.
  2. State store. Optional durable storage for BRAN-owned derived state: objects, indexes, caches, job metadata, and receipts.

The code is crates/bran-core/src/database/. It is standard-library only, like the rest of bran-core. Default tests make no database-server or network calls.

Decisions

Where the issue offered alternatives, the owner chose the smallest safe option the issue prefers:

Question Decision Reason
One connector or two ports? Two ports: EvidenceEngine and StateStore Different direction (read vs write), credentials, retention, and threat model.
Evidence engine order SQLite first, PostgreSQL second, others deferred SQLite is a bounded local file with no network. PostgreSQL is the first remote engine. MySQL/MariaDB, SQL Server, Oracle, DuckDB, and warehouse APIs wait until the contract is proven.
Default evidence access Introspection only Row access needs an explicit access: bounded-rows plus relation allowlist.
Free SQL Never passthrough SQL text must pass one deterministic gate and match an allowlisted digest. Projections are generated by BRAN and pass the same gate.
State-store backends Local file store first It needs no dependency. SQLite and PostgreSQL state backends are declared but report unavailable (see Not done).

The two ports cannot be configured interchangeably

Both descriptors are small YAML documents parsed by BRAN's existing frontmatter parser. Each parser accepts exactly one kind, an exact key set, and one credential scope:

Evidence source State store
kind evidence-source state-store
Rust type EvidenceSourceDescriptor StateStoreDescriptor
Port EvidenceEngine (read catalog, stream rows) StateStore (put, get, usage)
Keys engine, database, access, relations, queries, snapshot, max_rows, max_bytes, timeout_ms backend, location, namespace, quota_bytes, encryption_key
Shared keys kind, id, credential kind, id, credential
Credential scope credref:evidence/<name> credref:state/<name>
Writes never only inside its own namespace directory
Retention none; evidence lives in the packet that cites it quota-bounded, operator-managed

So:

  • An evidence descriptor given to the state parser fails with WrongKind, and the reverse also fails.
  • A key that belongs to the other port fails with UnknownKey.
  • A state credential in an evidence descriptor fails with CredentialScope, and the reverse also fails. One credential reference can never authorize both ports.
  • The Rust types have no conversion between them, and the two traits share no method or supertrait. FileStateStore::open takes only a StateStoreDescriptor. extract takes only an EvidenceSourceDescriptor.

Test: database::tests::p8_database_boundary_config.

Evidence-source descriptor

kind: evidence-source
id: synthetic-orders
engine: sqlite
database: data/orders.sqlite
access: bounded-rows
relations: [main.customers, main.orders]
queries: [sha256:<64 lowercase hex>]
snapshot: sha256:<64 lowercase hex>
max_rows: 100
max_bytes: 65536
timeout_ms: 5000
  • engine: sqlite or postgresql.
  • database: for sqlite, a relative POSIX path of [A-Za-z0-9._-] segments with no . or .. segment, at most 256 bytes. For postgresql, an opaque host-registered handle name. Connection strings, URLs, hosts, ports, and query parameters are rejected, so they cannot carry credentials.
  • credential: forbidden for sqlite (a local file needs none), required for postgresql.
  • access: introspection (the default when absent) or bounded-rows.
  • relations: required, non-empty, sorted, and unique. Each entry is schema.relation in lowercase [a-z_][a-z0-9_]* parts. It bounds both introspection and row access.
  • queries: optional digests of allowlisted SQL (see the SQL gate below).
  • snapshot: optional pinned snapshot identity, as a sha256: digest. A different or unattested observed snapshot is stale-snapshot.
  • Limits: max_rows 1..=10000 (default 100), max_bytes 1..=4194304 (default 65536), timeout_ms 1..=60000 (default 5000).

State-store descriptor

kind: state-store
id: local-state
backend: file
location: state
namespace: default
quota_bytes: 1048576
  • backend: file, sqlite, or postgresql. Only file is available in this build.
  • location: a safe relative path under the host-provided state root, with the same rules as a SQLite database path. It cannot name an absolute path or climb out of the state root.
  • namespace: a name. A store handle is bound to one namespace when it opens. No API reads across namespaces.
  • quota_bytes: 1..=1073741824.
  • credential: forbidden for file, required for postgresql.
  • encryption_key: an optional credref:state/<name>. The file backend cannot encrypt, so a requested key makes open fail with Unavailable. BRAN never stores plaintext after encryption was requested.

Credentials are references only

The only accepted credential form is credref:<scope>/<name>, with scope evidence or state and name matching [a-z0-9][a-z0-9._-]{0,63}. The host resolves the reference. BRAN never reads, stores, or prints credential material.

Rejected, with a content-free error: inline passwords, user:password@host URLs, env: or file references, a missing or foreign scope, and a name with / or other characters. Descriptor errors carry at most a static field name. They never echo the value, so a secret pasted into the wrong field cannot be reflected into logs, receipts, packets, or error messages.

Test: database::tests::p8_database_boundary_config.

SQL gate

database::sql::gate is the only way to build a GatedQuery. The engine port accepts only a GatedQuery, so model-generated SQL, operator SQL, and BRAN-generated projections all pass through the same deterministic gate. The gate accepts a deliberately small grammar and rejects everything else:

  • exactly one SELECT, first: no ;, no UNION/INTERSECT/EXCEPT, no WITH, no subqueries
  • no backslash anywhere, so escape-string dialects cannot tokenize differently from the gate
  • no comments (--, /*), $$ quoting, @, :, &, or non-ASCII outside string literals
  • no write or DDL keyword (INSERT, UPDATE, DELETE, MERGE, REPLACE, CREATE, DROP, ALTER, TRUNCATE, GRANT, REVOKE, COPY, INTO, RETURNING, and similar)
  • no session or transaction keyword (SET, RESET, PRAGMA, BEGIN, COMMIT, ROLLBACK, SAVEPOINT, LOCK, FOR, and similar)
  • no procedure execution (CALL, EXEC, EXECUTE, DO)
  • no external reference (ATTACH, DETACH, LOAD, IMPORT, OPENROWSET, OPENQUERY, OPENDATASOURCE, dblink, table@link, relation names with more than two parts, or column names with more than three)
  • function calls only from abs, avg, coalesce, count, length, lower, max, min, round, sum, and upper, named directly. Everything else, such as load_extension, readfile, pg_read_file, pg_sleep, or randomblob, is UnsafeFunction. So is a schema-qualified call such as other.count(x), which could resolve to a user-defined function, and a call through a grammar word, literal, or quoted name, such as first(x), like(x, y), 'randomblob'(4), or "abs"(x). Any other ( must follow an operator, ,, (, or one of SELECT, DISTINCT, WHERE, AND, OR, NOT, IN, ON, HAVING, BY, BETWEEN, LIMIT, or OFFSET, where it can only group an expression or list. So x LIKE ('a') must be written x LIKE 'a'.
  • IN must be followed by (. x IN schema.relation would read a relation outside the FROM list, so it is Malformed.
  • every relation in the FROM list is fully qualified schema.relation and present in the descriptor allowlist. Unqualified names are rejected, because a server search path could resolve them to another schema. At parenthesis depth zero, FROM, ,, and JOIN each introduce a relation until WHERE, GROUP, ORDER, HAVING, LIMIT, or OFFSET ends the list, including a , after a join's ON expression. A relation may be followed only by an alias, ,, a join keyword, ON, or one of those clause words. Modifiers such as NOT INDEXED, INDEXED BY, or USING are Malformed, so they cannot hide a later relation from the allowlist check.
  • row queries need ORDER BY, so truncation and result digests are deterministic
  • at most 8192 bytes and 1024 tokens

The first forbidden keyword in token order decides the rejection class. A column whose name is also a listed keyword (for example comment or start) must be double-quoted.

Parameters (? or $1) are accepted for prepared execution. The gate normalizes the token stream (single spaces; unquoted words lowercased; literals and quoted identifiers verbatim) and records sha256 of that text as the query digest. Whitespace and keyword case do not change the digest.

Free SQL must also match one of the descriptor's queries digests. A relation projection (relation plus column list) needs no digest. BRAN generates SELECT "c1", "c2" FROM "schema"."relation" ORDER BY 1, 2 and gates it like any other SQL.

SqlRejection is a content-free enum. It never contains the input.

Adapters must also enforce read-only access in the engine, as defense in depth:

Engine Required engine-level guard
SQLite open mode=ro; PRAGMA query_only=ON; an authorizer that permits only SELECT, READ on allowlisted relations, and allowlisted functions; extension loading off; a progress handler for timeout and cancellation
PostgreSQL (future) SET TRANSACTION READ ONLY in a dedicated transaction; statement_timeout; a role without write, dblink, postgres_fdw, or large-object grants; server-side cancel

tools/ci/database_contract_check.py proves the SQLite column against a real SQLite engine (Python's standard sqlite3), using the synthetic fixture database.

Tests: database::sql::tests::p8_database_sql_gate, plus tools/ci/database_contract_check.py.

Extraction outcomes

database::evidence::extract checks policy before it contacts the engine. A refused request (rows under introspection, a gate rejection, or an SQL digest that is not allowlisted) returns ExtractionRefusal. The engine is never called. An accepted request always yields exactly one outcome.

The effective deadline is the earlier of the caller's deadline and the descriptor's timeout_ms. The adapter receives it with the cancellation flag (Interrupt) on every catalog and row call and must stop in-flight engine work when either fires. BRAN also rechecks both after every engine call: the opening snapshot, the catalog, the row stream (even an empty one), and the closing snapshot. After an interruption it makes no further engine call, and a blocking call that overran or was cancelled never yields admissible evidence.

Outcome When Rows kept Packet admission
schema-only no row request none admitted
bounded-rows every row fit max_rows and max_bytes all admitted
truncated the engine had more rows or bytes than the limits the bounded prefix admitted, with truncated: true
cancelled the cancellation flag was set at any point, or the engine reported cancel none refused
timeout the effective deadline passed at any point, or the engine reported timeout none refused
unavailable no adapter for this engine, or the adapter reported unavailable none refused
stale-snapshot the observed snapshot differs from the pinned one, or changed during extraction none refused
dlp-rejected the DLP check reported findings none refused

Refused outcomes (cancelled, timed out, unavailable, stale, and DLP-rejected) keep no partial rows and no catalog. Their record has "columns":[], so a column name that DLP flagged, or that DLP never checked, cannot reach the sealed record. They produce a receipt and nothing a packet can cite. An engine for which no adapter is compiled in (PostgreSQL today) is reported as unavailable. BRAN never simulates it.

Test: database::evidence::tests::p8_database_evidence_packet.

Provenance

Every extraction produces one canonical JSON record (sorted keys, no whitespace) with:

  • adapter and engine
  • source_id, database (the validated SQLite path or PostgreSQL handle, which cannot contain credentials), and credential (the opaque reference, or null). Changing the configured database changes the record, the evidence digest, and the packet node id.
  • access, relations, and columns (schema.relation.column locators; empty for refused outcomes)
  • derivation: schema-only, projection, or query
  • query_digest: the normalized gate digest, or null
  • snapshot: {"state":"attested","value":…} or {"state":"unavailable","value":null}. Engine snapshot text is untrusted, so anything other than a sha256: digest is sealed as its digest.
  • row_count and truncated
  • extracted_at, supplied by the caller so replays are deterministic
  • limits: max_rows, max_bytes, and timeout_ms
  • source_digest (the canonical catalog) and result_digest (the canonical rows)
  • outcome, dlp (not-evaluated, passed, or findings), and classification and redaction. Both are unavailable in this version. BRAN does not claim a classification or redaction it did not perform.

evidence_digest is sha256 of that record. The same source, request, and snapshot always give the same record and digest. The record never contains credential material, connection strings, or rows.

Packet entry

DatabaseEvidence::packet_evidence turns an admitted extraction into the existing packet inputs:

  • a graph NodeInput with role External, id db:<source_id>:<first 32 hex of evidence_digest>, provenance source database-evidence, locator database:<source_id>#<evidence_digest>, and facts for outcome, truncation, row count, and result digest
  • an EvidenceContent whose content is the provenance record plus one canonical JSON array per row, with preservation anchors for db.outcome, db.truncated, db.row_count, and db.result_digest

PacketAssembler then carries database evidence like any other node. Its provenance and truncation state stay in the packet items and payload. Non-admitted outcomes return NotAdmitted and never become packet items.

Today bran packet and bran query still read repository files only. No CLI command reads database descriptors yet (see Not done).

Test: database::evidence::tests::p8_database_evidence_packet.

State store

StateStore is the port. FileStateStore is the first implementation. The existing process-local MemoryResultStore is unchanged and stays the default. A durable store is opt-in through a state-store descriptor.

Layout under <state root>/<location>/<namespace>/:

  • VERSION: the layout version, 1
  • objects/<sha256 hex>: content-addressed bytes
  • tmp/: staging for writes in progress

Behaviour:

  • Atomic writes. put writes the bytes to tmp/, syncs them, then renames into objects/. A crash leaves a temp file, never a partial object. open deletes leftover temp files and reports how many it removed as recovered_partial_writes.
  • Corruption. get and verify recompute SHA-256. A modified or truncated object is Corrupt. It is never returned as data.
  • Migration. A missing VERSION in an empty namespace is initialised to 1. Any other version is MigrationFailed, and the store refuses to open without writing anything. A malformed VERSION is Corrupt. Version 1 is the first layout, so there is no migration yet. A future migration must keep this refuse-without-writing rule.
  • Quota. A put that would exceed quota_bytes is QuotaExceeded. It leaves no temp file and does not change usage. Repairing a damaged object recounts usage from disk, subtracts the damaged length, adds the true length, and enforces the quota on that total.
  • Confinement. Every directory from the state root down to the namespace, objects/, tmp/, VERSION, and every object and staged entry is inspected without following symlinks before it is read, written, or removed. A symlink or unexpected entry is Unconfined, and open refuses before it creates, writes, or deletes anything. The host-supplied state root itself is trusted.
  • Isolation. A handle can read and write only its own namespace.
  • Canonical data. The store writes only under its own namespace directory. It never reads or writes repository content. The state store is never a source of truth for canonical repository documents; everything in it can be rebuilt or discarded.

Errors are content-free: no paths, bytes, or credentials.

Limits: a namespace allows one writer at a time; two open handles each track their own usage and can together exceed the quota. The confinement checks run before each use, because portable std has no openat or O_NOFOLLOW, so a process that can already write inside the state root could race them. Concurrent server workloads are what a PostgreSQL backend would be for.

Test: database::state::tests::p8_state_store_failures.

Fixture database

fixtures/database/synthetic.sql builds a small synthetic SQLite database (customers, orders, and a view). All names start with Synthetic, and all email addresses use the reserved .invalid domain. No customer data is included. Adversarial data is included too: injection text stored as a value, non-ASCII text, a long value for byte truncation, and a synthetic key-shaped note for DLP.

fixtures/database/queries.tsv is the shared SQL corpus, one class<TAB>effect<TAB>rejection<TAB>sql row per statement. rejection is the exact SqlRejection the Rust gate must return (- for allowed queries). It holds allowed queries plus adversarial statements by class: write, multi-statement, session, procedure, unsafe-function, external-link, compound, secret-reflection, and policy. The effect column says what the engine does without the gate:

  • mutates: the statement changes an unguarded copy of the database. These negative tests are real attacks, not strawmen.
  • gate-only: the statement is a harmless read for SQLite, so the engine guard runs it. Only the BRAN gate refuses it, to keep the grammar small and results deterministic. Examples are compound queries, a trailing ;, a comment, and unqualified or unordered queries.
  • -: the engine guard refuses it too (or cannot parse it).

tools/ci/database_contract_check.py builds the database twice and requires identical dumps. It checks that every row is synthetic. It runs every allowed and gate-only query under the SQLite guard, and requires every other adversarial statement to be refused with the database unchanged. It also shows that a progress-handler interrupt stops a long query, which is how a SQLite adapter implements timeout and cancellation. The Rust gate test reads the same corpus and requires the gate to return the exact listed rejection for all 47 adversarial rows.

Offline default

No part of this design opens a listener or a network connection. The default test suite needs no database server. SQLite evidence is a local file. PostgreSQL stays unavailable until an adapter is explicitly compiled and selected.

Acceptance evidence

Acceptance item Evidence
Separate EvidenceSource and StateStore; not interchangeable This document; database::tests::p8_database_boundary_config
Only validated references to host-managed credentials database::tests::p8_database_boundary_config
Write SQL, multi-statements, unsafe functions, external links, and secret reflection rejected database::sql::tests::p8_database_sql_gate; tools/ci/database_contract_check.py
Eight outcomes typed separately database::evidence::tests::p8_database_evidence_packet
Evidence enters packets with stable provenance and truncation/derivation state Partial. Library level: database::evidence::tests::p8_database_evidence_packet and provenance_binds_database_origin. No real database evidence reaches bran packet yet (see Not done).
Corruption, migration failure, quota exhaustion, and partial writes fail visibly without losing canonical data database::state::tests::p8_state_store_failures, repair_accounts_quota, and state_paths_refuse_symlink_escape
Deterministic synthetic fixture database fixtures/database/; tools/ci/database_contract_check.py
No cloud database or network for the default suite All of the above run offline in ./tools/ci/check.sh --fast

Not done

  • Rust SQLite and PostgreSQL adapters (deferred by owner decision). A Rust SQLite binding needs an engine dependency, and the owner's delegate deferred adding one (2026-09-30). The trade-offs recorded for that decision: rusqlite 0.31 bundles SQLite 3.45 (older than current security fixes) and needs a C compiler, which conflicts with CONTRIBUTING's "stable Rust toolchain, nothing else"; rusqlite 0.33 and later pull foldhash (Zlib), which deny.toml does not allow. Until an adapter lands, the port and gate are proven here and the engine guard is proven through Python's standard sqlite3. PostgreSQL needs a server and a network adapter, and is not attempted. So the packet-entry acceptance item is partial: packet entry is proven through PacketAssembler with a synthetic engine, but no real database evidence can reach a packet yet.
  • SQLite and PostgreSQL state backends. Same dependency decision. The descriptors accept them, and open reports unavailable.
  • CLI wiring. bran packet does not yet load descriptors or run extraction. Packet entry is proven at the library level.
  • State-store retention and garbage collection, backup and restore tooling, encryption at rest, and deterministic state export. The file layout is plain content-addressed files. Restore means copying the files back, then running verify. The rest is follow-up work.
  • Classification and redaction receipts. These are recorded as unavailable until a classifier or redactor exists.