diff --git a/CHANGELOG.md b/CHANGELOG.md index 9ded8c8..8149522 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -10,6 +10,37 @@ that restarts at 1 for each Senzing version. ## [Unreleased] +## [4.4.3-4] - 2026-11-02 + +### Added + +- Install script `--license-file `: the license is checked before any step runs (not + empty, base64, a multiple of 4 characters) and stored into the license secret before + setup, never printed. + +- `API.SZ_CHECK_LICENSE()` checks a license before you store it: an empty or malformed + value is refused at once (`-20111`), then a short setup job starts the engine with it and + reports its expiry and record limit, or why the engine refuses it. Nothing is stored or + restarted; the app never returns or logs the license (it is in the check job's spec while + the job runs, and the CALL is in your query history). + +### Fixed + +- Docs: CPU autoscaling does not hold resolvers back when Postgres is the bottleneck, as the + 4.4.3-3 notes and the Use guide said: at 6 threads a resolver waiting on Postgres stays + above the 40% trigger. The resolver maximum (2 per Postgres core when empty) is what sizes + them to Postgres; past that ratio the gain is small (+9% for a 5th on `STANDARD_L`), and a + `RESOLVER_MAX_INSTANCES` set by hand is reached under any load. + +- A Postgres resize reaches the running resolvers at once: `SZ_CONFIGURE` and `SZ_SET_CONFIG` + apply the resolver's instance limits and the pool's `MAX_NODES` to running services when + `PG_COMPUTE_FAMILY`, `RESOLVER_MAX_INSTANCES`, `RESOLVER_INSTANCES`, + `REPLICATOR_INSTANCES` or `POOL_MAX_NODES` changes (they used to wait for `SZ_START`). The + Use guide's resize example now sets only `PG_COMPUTE_FAMILY`; its old 16 resolvers would + exceed `STANDARD_XL`'s sessions at 6 threads. + +## [4.4.3-3] - 2026-11-02 + ### Added - The resolver service publishes platform metrics (`container.cpu.usage`, memory) to the account's event table, so its autoscaling can be read against its real CPU. diff --git a/docs/install.md b/docs/install.md index e4b6e3b..13a7581 100644 --- a/docs/install.md +++ b/docs/install.md @@ -220,12 +220,34 @@ EULA_ACCEPTED_BY FROM SZ_SNOWFLAKE.API.SZ_HEALTH` shows the acceptance. ## 4. Set the license secret Run interactively, never committed: -`ALTER SECRET SZ_LICENSE SET SECRET_STRING = ''`. It stays yours: the app has +`ALTER SECRET SZ_LICENSE SET SECRET_STRING = ''`, or give the install script +`--license-file `: it checks the file before any step runs (not empty, base64, a +multiple of 4 characters) and stores it without printing it (the `ALTER SECRET` it sends is in +your query history, as when you run it yourself). It stays yours: the app has `READ`, logs only its expiry date and record limit, and never sends it anywhere. If the services are already running, `CALL SZ_SNOWFLAKE.API.SZ_RELOAD_SECRETS()` restarts them so they read it (secrets are read once at start). The same applies to a rotated Postgres password or a new CA (the infra script's `ROTATE` section). +To renew on a running install, check the new license first, so a bad copy never reaches +the services: + +```sql +CALL SZ_SNOWFLAKE.API.SZ_CHECK_LICENSE(''); +-- {"valid": true, "expire_date": "...", "record_limit": ...}: then store it and reload +ALTER SECRET SZ_LICENSE SET SECRET_STRING = ''; +CALL SZ_SNOWFLAKE.API.SZ_RELOAD_SECRETS(); +``` + +`SZ_CHECK_LICENSE` refuses an empty or malformed value at once (`-20111`: one character +pasted too many is a length that is not a multiple of 4), then starts the engine with the +license in a short job and reports its expiry and record limit, or `"valid": false` with +the engine's reason. It stores nothing and restarts nothing. The app never returns or logs +the license; it is in the check job's spec only while that job runs, and, as with +`ALTER SECRET`, the statement you run is in your query history. +`SZ_HEALTH` shows the running license (`LICENSE_EXPIRES`, `LICENSE_DAYS_LEFT`, +`LICENSE_RECORD_LIMIT`, `LICENSE_RECORDS_USED`) and warns 30 days before it expires. + FIRST RUN ## 5. Run setup and start diff --git a/docs/troubleshooting.md b/docs/troubleshooting.md index ee9d4fc..4e01efd 100644 --- a/docs/troubleshooting.md +++ b/docs/troubleshooting.md @@ -56,6 +56,8 @@ paused until `SZ_UPGRADE` runs (the install script runs it when a re-run changes | setup exit 7 `precondition`: `the repository server is PostgreSQL 16..` | the app needs PostgreSQL 17 or later; upgrade your database (Snowflake Postgres: `POSTGRES_VERSION = 17` or later) | | the infrastructure script stops with `-20914` | `SZ_PG_MODE = 'EXTERNAL'` needs `SZ_PG_HOST` (a host name) and `SZ_PG_PORT` | | `SZ_SET_CONFIG` / `SZ_START` raise `-20024` naming `PG_EXTERNAL` | `PG_COMPUTE_FAMILY` `EXTERNAL` needs `PG_MEMORY_GIB` and `PG_MAX_CONNECTIONS` (your database's size) in the same `SZ_CONFIGURE` | +| `SZ_CHECK_LICENSE` raises `-20111` | the license is empty (that is the 500-record evaluation limit) or not base64 (the message gives its length; base64 is a multiple of 4 characters): copy it again without extra characters | +| `SZ_CHECK_LICENSE` returns `"valid": false` | the engine refused the license (`error` says why: signature, expired); nothing was stored. Get the license again from Senzing | | `SZ_PURGE_SOURCE` raises `-20209` | the code is malformed, or the source is registered and enabled: `SZ_UNREGISTER_SOURCE('')` first (its records would otherwise come back) | | `SZ_REGISTER_SOURCE` raises `-20208` | change tracking is off on a source table; nothing was registered. The message carries the statement: as the table owner run `ALTER TABLE .. SET CHANGE_TRACKING = TRUE;`, then call `SZ_REGISTER_SOURCE` again with the same arguments | | `RESOLVE_STATE` reads `SOURCES_FAILED` | a source's capture failed (`SOURCE_ERRORS`: usually change tracking turned off, or a stale stream). Fix what it says, then `SZ_REGISTER_SOURCE` again with the same arguments; `RESOLVE_STATE` never reads `CAUGHT_UP` meanwhile | diff --git a/docs/use.md b/docs/use.md index 61b58a9..4bdc0f6 100644 --- a/docs/use.md +++ b/docs/use.md @@ -187,8 +187,9 @@ and up to 5 GiB each (the node's memory less what Snowflake keeps for itself). Under load the resolver service adds instances, up to `RESOLVER_MAX_INSTANCES`, when the average resolver uses more than 40% of its CPU for a minute, and removes one when it uses less than 25% for three minutes; the pool adds nodes up to `POOL_MAX_NODES` to fit them. -Scaling on CPU also protects Postgres: a resolver waiting on the database uses little CPU, -so a saturated database sheds resolvers instead of gaining them. +CPU scaling does not hold resolvers back when Postgres is the bottleneck: at 6 threads a +resolver waiting on the database still uses 0.4-0.55 of its CPU, so the autoscaler keeps +adding them up to the maximum below. That maximum is what sizes the resolvers to Postgres. One resolver CPU resolves roughly 35-40 records a second. Every engine thread holds a Postgres session, and each service 5 more of its own (measured: @@ -204,9 +205,15 @@ load on `STANDARD_L` (2 cores), 4 resolvers ran 172 records a second at a Postgr 7; a 5th added 9% and took the load to 18. For a faster first load, size Postgres up (the infrastructure script's `SZ_PG_FAMILY`, then `SZ_CONFIGURE({'PG_COMPUTE_FAMILY': ..})`): the maximum and the pool follow it. The install script tells the app the family of the -Postgres instance it created. -`RESOLVER_MAX_INSTANCES` is a cost cap; CPU autoscaling keeps the resolvers to what the load -and the database can take. `STANDARD_L` has 2 cores: in a 200,000-record first load its load +Postgres instance it created. The app cannot see your Postgres instance, so after you +resize it yourself, `SZ_CONFIGURE({'PG_COMPUTE_FAMILY': ''})` is what moves the +limits: without it the resolvers stay sized for the old family (too few after sizing up; +too many for the database after sizing down). +Empty `RESOLVER_MAX_INSTANCES` derives it from the database's cores; a number you set replaces +it and is reached under load whatever Postgres is doing. Past 2 resolver CPUs per core the +gain is small (on `STANDARD_L` a 5th resolver added 9%); where more resolvers start to cost +throughput or Postgres starts to fail has not been measured, so stay near that ratio and +size Postgres up for more. `STANDARD_L` has 2 cores: in a 200,000-record first load its load average reached 4 with three resolvers, so past three or four resolvers a larger `PG_COMPUTE_FAMILY` buys more throughput than more resolvers. `SZ_SET_CONFIG`, `SZ_CONFIGURE` and `SZ_START` refuse (`-20024`, naming the numbers) a @@ -219,12 +226,12 @@ Postgres needs at least 8 GB: `STANDARD_L` (2 cores, 8 GB, 40,000 IOPS, 0.0712 c large first load use `STANDARD_XL` or larger, then size it back down. Resizing and server settings go in separate statements. The instance restarts for a resize; the resolvers and the replicator reconnect on their own and resume within about 60 seconds -of Postgres returning. With `STANDARD_XL` (160 sessions) up to 16 resolvers fit -(16 x 9 + 13 = 157): +of Postgres returning. Then tell the app the new family: the resolver maximum and the pool +follow it at once, without a restart (`STANDARD_XL`, 4 cores: 8 resolvers, 10 nodes): ```sql ALTER POSTGRES INSTANCE SZ_PG SET COMPUTE_FAMILY = 'STANDARD_XL'; -CALL SZ_SNOWFLAKE.API.SZ_CONFIGURE({'PG_COMPUTE_FAMILY': 'STANDARD_XL', 'RESOLVER_MAX_INSTANCES': '16', 'POOL_MAX_NODES': '18'}); +CALL SZ_SNOWFLAKE.API.SZ_CONFIGURE({'PG_COMPUTE_FAMILY': 'STANDARD_XL'}); -- an instance created before the infrastructure script turned JIT off: ALTER POSTGRES INSTANCE SZ_PG SET POSTGRES_SETTINGS = '{"postgres:jit": "off"}'; ```