Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
31 changes: 31 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -10,6 +10,37 @@ that restarts at 1 for each Senzing version.

## [Unreleased]

## [4.4.3-4] - 2026-11-02

### Added

- Install script `--license-file <path>`: the license is checked before any step runs (not
empty, base64, a multiple of 4 characters) and stored into the license secret before
setup, never printed.

- `API.SZ_CHECK_LICENSE(<base64>)` checks a license before you store it: an empty or malformed
value is refused at once (`-20111`), then a short setup job starts the engine with it and
reports its expiry and record limit, or why the engine refuses it. Nothing is stored or
restarted; the app never returns or logs the license (it is in the check job's spec while
the job runs, and the CALL is in your query history).

### Fixed

- Docs: CPU autoscaling does not hold resolvers back when Postgres is the bottleneck, as the
4.4.3-3 notes and the Use guide said: at 6 threads a resolver waiting on Postgres stays
above the 40% trigger. The resolver maximum (2 per Postgres core when empty) is what sizes
them to Postgres; past that ratio the gain is small (+9% for a 5th on `STANDARD_L`), and a
`RESOLVER_MAX_INSTANCES` set by hand is reached under any load.

- A Postgres resize reaches the running resolvers at once: `SZ_CONFIGURE` and `SZ_SET_CONFIG`
apply the resolver's instance limits and the pool's `MAX_NODES` to running services when
`PG_COMPUTE_FAMILY`, `RESOLVER_MAX_INSTANCES`, `RESOLVER_INSTANCES`,
`REPLICATOR_INSTANCES` or `POOL_MAX_NODES` changes (they used to wait for `SZ_START`). The
Use guide's resize example now sets only `PG_COMPUTE_FAMILY`; its old 16 resolvers would
exceed `STANDARD_XL`'s sessions at 6 threads.

## [4.4.3-3] - 2026-11-02

### Added

- The resolver service publishes platform metrics (`container.cpu.usage`, memory) to the account's event table, so its autoscaling can be read against its real CPU.
Expand Down
24 changes: 23 additions & 1 deletion docs/install.md
Original file line number Diff line number Diff line change
Expand Up @@ -220,12 +220,34 @@ EULA_ACCEPTED_BY FROM SZ_SNOWFLAKE.API.SZ_HEALTH` shows the acceptance.
## 4. Set the license secret

Run interactively, never committed:
`ALTER SECRET SZ_LICENSE SET SECRET_STRING = '<base64>'`. It stays yours: the app has
`ALTER SECRET SZ_LICENSE SET SECRET_STRING = '<base64>'`, or give the install script
`--license-file <path>`: it checks the file before any step runs (not empty, base64, a
multiple of 4 characters) and stores it without printing it (the `ALTER SECRET` it sends is in
your query history, as when you run it yourself). It stays yours: the app has
`READ`, logs only its expiry date and record limit, and never sends it anywhere. If the
services are already running, `CALL SZ_SNOWFLAKE.API.SZ_RELOAD_SECRETS()` restarts them so
they read it (secrets are read once at start). The same applies to a rotated Postgres
password or a new CA (the infra script's `ROTATE` section).

To renew on a running install, check the new license first, so a bad copy never reaches
the services:

```sql
CALL SZ_SNOWFLAKE.API.SZ_CHECK_LICENSE('<base64>');
-- {"valid": true, "expire_date": "...", "record_limit": ...}: then store it and reload
ALTER SECRET SZ_LICENSE SET SECRET_STRING = '<base64>';
CALL SZ_SNOWFLAKE.API.SZ_RELOAD_SECRETS();
```

`SZ_CHECK_LICENSE` refuses an empty or malformed value at once (`-20111`: one character
pasted too many is a length that is not a multiple of 4), then starts the engine with the
license in a short job and reports its expiry and record limit, or `"valid": false` with
the engine's reason. It stores nothing and restarts nothing. The app never returns or logs
the license; it is in the check job's spec only while that job runs, and, as with
`ALTER SECRET`, the statement you run is in your query history.
`SZ_HEALTH` shows the running license (`LICENSE_EXPIRES`, `LICENSE_DAYS_LEFT`,
`LICENSE_RECORD_LIMIT`, `LICENSE_RECORDS_USED`) and warns 30 days before it expires.

<sub><b>FIRST RUN</b></sub>

## 5. Run setup and start
Expand Down
2 changes: 2 additions & 0 deletions docs/troubleshooting.md
Original file line number Diff line number Diff line change
Expand Up @@ -56,6 +56,8 @@ paused until `SZ_UPGRADE` runs (the install script runs it when a re-run changes
| setup exit 7 `precondition`: `the repository server is PostgreSQL 16..` | the app needs PostgreSQL 17 or later; upgrade your database (Snowflake Postgres: `POSTGRES_VERSION = 17` or later) |
| the infrastructure script stops with `-20914` | `SZ_PG_MODE = 'EXTERNAL'` needs `SZ_PG_HOST` (a host name) and `SZ_PG_PORT` |
| `SZ_SET_CONFIG` / `SZ_START` raise `-20024` naming `PG_EXTERNAL` | `PG_COMPUTE_FAMILY` `EXTERNAL` needs `PG_MEMORY_GIB` and `PG_MAX_CONNECTIONS` (your database's size) in the same `SZ_CONFIGURE` |
| `SZ_CHECK_LICENSE` raises `-20111` | the license is empty (that is the 500-record evaluation limit) or not base64 (the message gives its length; base64 is a multiple of 4 characters): copy it again without extra characters |
| `SZ_CHECK_LICENSE` returns `"valid": false` | the engine refused the license (`error` says why: signature, expired); nothing was stored. Get the license again from Senzing |
| `SZ_PURGE_SOURCE` raises `-20209` | the code is malformed, or the source is registered and enabled: `SZ_UNREGISTER_SOURCE('<CODE>')` first (its records would otherwise come back) |
| `SZ_REGISTER_SOURCE` raises `-20208` | change tracking is off on a source table; nothing was registered. The message carries the statement: as the table owner run `ALTER TABLE <db>.<schema>.<table> SET CHANGE_TRACKING = TRUE;`, then call `SZ_REGISTER_SOURCE` again with the same arguments |
| `RESOLVE_STATE` reads `SOURCES_FAILED` | a source's capture failed (`SOURCE_ERRORS`: usually change tracking turned off, or a stale stream). Fix what it says, then `SZ_REGISTER_SOURCE` again with the same arguments; `RESOLVE_STATE` never reads `CAUGHT_UP` meanwhile |
Expand Down
23 changes: 15 additions & 8 deletions docs/use.md
Original file line number Diff line number Diff line change
Expand Up @@ -187,8 +187,9 @@ and up to 5 GiB each (the node's memory less what Snowflake keeps for itself).
Under load the resolver service adds instances, up to `RESOLVER_MAX_INSTANCES`, when the
average resolver uses more than 40% of its CPU for a minute, and removes one when it uses
less than 25% for three minutes; the pool adds nodes up to `POOL_MAX_NODES` to fit them.
Scaling on CPU also protects Postgres: a resolver waiting on the database uses little CPU,
so a saturated database sheds resolvers instead of gaining them.
CPU scaling does not hold resolvers back when Postgres is the bottleneck: at 6 threads a
resolver waiting on the database still uses 0.4-0.55 of its CPU, so the autoscaler keeps
adding them up to the maximum below. That maximum is what sizes the resolvers to Postgres.
One resolver CPU resolves roughly 35-40 records a second.

Every engine thread holds a Postgres session, and each service 5 more of its own (measured:
Expand All @@ -204,9 +205,15 @@ load on `STANDARD_L` (2 cores), 4 resolvers ran 172 records a second at a Postgr
7; a 5th added 9% and took the load to 18. For a faster first load, size Postgres up (the
infrastructure script's `SZ_PG_FAMILY`, then `SZ_CONFIGURE({'PG_COMPUTE_FAMILY': ..})`):
the maximum and the pool follow it. The install script tells the app the family of the
Postgres instance it created.
`RESOLVER_MAX_INSTANCES` is a cost cap; CPU autoscaling keeps the resolvers to what the load
and the database can take. `STANDARD_L` has 2 cores: in a 200,000-record first load its load
Postgres instance it created. The app cannot see your Postgres instance, so after you
resize it yourself, `SZ_CONFIGURE({'PG_COMPUTE_FAMILY': '<family>'})` is what moves the
limits: without it the resolvers stay sized for the old family (too few after sizing up;
too many for the database after sizing down).
Empty `RESOLVER_MAX_INSTANCES` derives it from the database's cores; a number you set replaces
it and is reached under load whatever Postgres is doing. Past 2 resolver CPUs per core the
gain is small (on `STANDARD_L` a 5th resolver added 9%); where more resolvers start to cost
throughput or Postgres starts to fail has not been measured, so stay near that ratio and
size Postgres up for more. `STANDARD_L` has 2 cores: in a 200,000-record first load its load
average reached 4 with three resolvers, so past three or four resolvers a larger
`PG_COMPUTE_FAMILY` buys more throughput than more resolvers.
`SZ_SET_CONFIG`, `SZ_CONFIGURE` and `SZ_START` refuse (`-20024`, naming the numbers) a
Expand All @@ -219,12 +226,12 @@ Postgres needs at least 8 GB: `STANDARD_L` (2 cores, 8 GB, 40,000 IOPS, 0.0712 c
large first load use `STANDARD_XL` or larger, then size it back down.
Resizing and server settings go in separate statements. The instance restarts for a resize;
the resolvers and the replicator reconnect on their own and resume within about 60 seconds
of Postgres returning. With `STANDARD_XL` (160 sessions) up to 16 resolvers fit
(16 x 9 + 13 = 157):
of Postgres returning. Then tell the app the new family: the resolver maximum and the pool
follow it at once, without a restart (`STANDARD_XL`, 4 cores: 8 resolvers, 10 nodes):

```sql
ALTER POSTGRES INSTANCE SZ_PG SET COMPUTE_FAMILY = 'STANDARD_XL';
CALL SZ_SNOWFLAKE.API.SZ_CONFIGURE({'PG_COMPUTE_FAMILY': 'STANDARD_XL', 'RESOLVER_MAX_INSTANCES': '16', 'POOL_MAX_NODES': '18'});
CALL SZ_SNOWFLAKE.API.SZ_CONFIGURE({'PG_COMPUTE_FAMILY': 'STANDARD_XL'});
-- an instance created before the infrastructure script turned JIT off:
ALTER POSTGRES INSTANCE SZ_PG SET POSTGRES_SETTINGS = '{"postgres:jit": "off"}';
```
Expand Down
Loading