Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
65 changes: 65 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -12,6 +12,71 @@ that restarts at 1 for each Senzing version.

### Added

- The resolver service publishes platform metrics (`container.cpu.usage`, memory) to the account's event table, so its autoscaling can be read against its real CPU.

- `SZ_HEALTH` shows the running license: `LICENSE_EXPIRES`, `LICENSE_DAYS_LEFT`,
`LICENSE_RECORD_LIMIT`, `LICENSE_RECORDS_USED`, and fails `LICENSE_EXPIRED` and warns
`LICENSE_EXPIRING` (30 days), `LICENSE_EVALUATION` (the 500-record evaluation limit) and
`LICENSE_NEAR_LIMIT` (90%).

- A change to a table a mapping view only looks up re-resolves the records that read it:
a `TABLES` entry with `via` (`{'table': .., 'key_column': .., 'via': {'table': ..,
'key_column': .., 'record_id_column': ..}}`) joins the changed rows to the source table
that points at them; the install script takes it as `--source-via`. `--source` also takes
a record id column per table, `<table>=<column>` (a child table's foreign key).

- The install script checks every prerequisite before any step runs (also with `--dry-run`)
and lists every problem at once, changing nothing: the tools on PATH, the Snowflake CLI's
connector version, the Docker daemon, `--pg-host` resolving, the bundle and `SHA256SUMS`,
the connection and role, the infra secrets and external access integration, and each
`--source` view (`DATA_SOURCE`, `RECORD_ID`, `RECORD_JSON`) and table (visible,
`CHANGE_TRACKING` on, the record id column).

### Changed

- Default engine threads are 6 per CPU (was 4): a 4-thread resolver used about half its CPU,
its threads waiting on Postgres.
- `RESOLVER_MAX_INSTANCES` and `POOL_MAX_NODES` default to empty, derived from Postgres:
2 resolver CPUs per core of `PG_COMPUTE_FAMILY` (4 on `STANDARD_L`, 8 on `STANDARD_XL`;
4 when unknown), and a pool node for each plus the replicator and setup job. On
`STANDARD_L` a 5th resolver took Postgres from load 7 to 18 for 9% more throughput.
An upgrade resets a value still at the old default (7, 9) to empty. The install script
sets `PG_COMPUTE_FAMILY` from the Postgres instance.

- Resolver autoscaling adds an instance above 40% CPU (was 50%) and removes one below 25%
(was 30%). A resolver not waiting on Postgres averages about 48%, so 50% held the second
resolver back 18 minutes in a 200,000-record first load; resolvers waiting on a saturated
Postgres stay below 40%, so the database is still not overloaded.

- **Breaking:** a mapping view's JSON column is now `RECORD_JSON` (was `JSON_DOC`), so a
view is `DATA_SOURCE`, `RECORD_ID`, `RECORD_JSON` — the contract the Senzing MCP server's
Snowflake guidance already uses. No alias: before upgrading, rename the column in each
registered mapping view (`... AS RECORD_JSON`); a view still returning `JSON_DOC` is
refused by `SZ_REGISTER_SOURCE` and cannot be resolved.

### Fixed

- `SZ_REGISTER_SOURCE` upper-cases a plain `record_id_column` (and a `via` entry's columns)
like Snowflake does for unquoted names; a lower-case name used to register but never
match a changed row.

- The install script's wait for `CAUGHT_UP` no longer stops silently: a failed
`SZ_HEALTH` poll is reported and retried, and the wait always ends with one
`resolve wait:` line naming the outcome (`CAUGHT_UP`, timed out, failed source,
error), the elapsed time, records and entities resolved, `SZ_HEALTH.OVERALL` and the
exit code. A timeout says resolution continues in the background.

## [4.4.3-2] - 2026-10-31

### Added

- Auto-pause, on by default: an install idle for 30 minutes suspends its compute pool and
Postgres and wakes on new work; `SZ_PAUSE` / `SZ_UNPAUSE` in `SZ_INFRA_DB.SZ_INFRA`.
`SZ_PG_SHARED` marks a Postgres as shared (never suspended); `SZ_PG_MODE EXTERNAL` uses
your own PostgreSQL 17.
- Two installs in one account (`--app`, `--infra-prefix`); `SZ_UNINSTALL`,
`SZ_PURGE_SOURCE`, `RESOLVE_STATE`, and FAIL/WARN codes in `SZ_HEALTH`.
- Per-architecture bundles (`linux-amd64`, `linux-arm64`, about 0.31 GB each).
- `sz-snowflake-cord-data.tar.gz`, the CORD data for the Senzing Snowflake preview: Senzing's
official [CORD](https://senzing.com/senzing-ready-data-collections-cord/) data from its Las Vegas collection, reshaped into normalized source
tables for six source systems (about 200,000 records), with `load.sql` to load it into
Expand Down
23 changes: 15 additions & 8 deletions docs/ai-assisted.md
Original file line number Diff line number Diff line change
Expand Up @@ -32,7 +32,7 @@ Have ready:
| ----------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------- |
| The [prerequisites](../README.md#prerequisites) | a role such as `ACCOUNTADMIN`, the Snowflake CLI, Docker, `cosign` |
| A release | its tag, e.g. `4.4.3-1`, from this repository's [releases](https://github.com/Senzing/sz-snowflake/releases) |
| Your sources | per data source: a code (`CUSTOMERS`), a mapping view producing Senzing JSON, and its base table(s) with a record id column |
| Your sources | per data source: a code (`CUSTOMERS`), a mapping view (`DATA_SOURCE`, `RECORD_ID`, `RECORD_JSON` = the Senzing JSON), and its base table(s) with a record id column |
| No data of your own? | the assistant can load and map the [CORD data](#no-data-of-your-own) instead |
| A license (optional) | base64; without one the engine stops at 500 records |

Expand Down Expand Up @@ -85,7 +85,11 @@ Fill in the placeholders and paste:
> and one `--source` per data source (every base table needs change tracking: if one
> lacks it, show me `ALTER TABLE <db.schema.table> SET CHANGE_TRACKING = TRUE;` to run
> as the table owner, then re-run this step):
> `--source <CODE>=<db.schema.mapping_view>:<db.schema.table>[:<record id column>]`.
> `--source <CODE>=<db.schema.mapping_view>:<db.schema.table>[=<column>][,...][:<record id column>]`,
> listing every table the view reads that holds the record id (a child table with
> `=<its foreign key>`), plus one `--source-via` for each table the view only looks up,
> so a change to it re-resolves the records that read it:
> `--source-via <CODE>=<db.schema.table>.<key column>:<db.schema.source_table>.<column pointing at it>.<record id column>`.
> 6. While setup runs (2 to 5 minutes, one blocking call), check
> `SELECT SETUP_PHASE, SETUP_RUNNING_SECS FROM SZ_SNOWFLAKE.API.SZ_HEALTH` every 15
> seconds and tell me the phase in one line.
Expand Down Expand Up @@ -116,8 +120,9 @@ values to the assistant:
> my connection after I confirm. Then map each source system in it with the Senzing MCP
> mapping tools (`mapping_workflow`; never hand-write the mapping): create one mapping
> view per source in its schema, show me each view, and pass one `--source` per source
> system to the install script, listing every table the view reads with its record id
> column.
> system to the install script, listing every table the view reads: those holding the
> record id with `--source`, those it only looks up (another system's table, such as the
> contacts view reading `ACCOUNTS`) with `--source-via`.

<sub><b>REPORT</b></sub>

Expand All @@ -135,10 +140,12 @@ When the state reads `CAUGHT_UP`, paste:
>
> Results, schema `SZ_SNOWFLAKE.SZ`: `ENTITY_RECORD` (DATA_SOURCE, RECORD_ID, ENTITY_ID,
> MATCH_KEY, empty for an entity's first record), `ENTITY` (ENTITY_ID, RECORD_COUNT),
> `RELATIONSHIP` (ENTITY_ID_LO, ENTITY_ID_HI, MATCH_LEVEL_CODE, MATCH_KEY). Join each
> record to its source row: for data source `<CODE>`, table `<db.schema.table>` on
> `<record id column>` = RECORD_ID. Show readable fields such as name, date of birth,
> address, phone and email.
> `RELATIONSHIP` (ENTITY_ID_LO, ENTITY_ID_HI, MATCH_LEVEL_CODE, MATCH_KEY). For readable
> fields join each record to its source's mapping view on DATA_SOURCE and RECORD_ID and
> read RECORD_JSON:FEATURES with `JOIN TABLE(FLATTEN(..))`; join name parts with
> `ARRAY_TO_STRING(ARRAY_CONSTRUCT_COMPACT(..))`, not `CONCAT_WS` (see the Use guide).
> Show readable fields such as name, date of birth, address, phone and email; bind search
> text as a parameter.
>
> Sections:
>
Expand Down
4 changes: 3 additions & 1 deletion docs/install.md
Original file line number Diff line number Diff line change
Expand Up @@ -116,7 +116,9 @@ does not let you choose the availability zone, so same-zone placement cannot be
## 2. Install from the bundle

`sz-snowflake-install-<release>.sh` does everything from the downloaded files. Read it first
(`--dry-run` prints every step and runs nothing).
(`--dry-run` prints every step and changes nothing). Before any step it checks every
prerequisite (tools, connection and role, the infra objects, each `--source` view and table,
including `CHANGE_TRACKING`) and lists every problem at once; a dry run runs the same checks.

```bash
./sz-snowflake-install-<release>.sh <release> --pg-host <PG_HOST> --operator-role <role> [--role <role>] \
Expand Down
4 changes: 4 additions & 0 deletions docs/troubleshooting.md
Original file line number Diff line number Diff line change
Expand Up @@ -111,6 +111,7 @@ paused until `SZ_UPGRADE` runs (the install script runs it when a re-run changes
| `PARKED` | a resolver parked (`parked_reason`: the license record limit or expiry), or the replicator parked on `shape_mismatch` | license: set `SZ_LICENSE`, `SZ_RELOAD_SECRETS()`; shape: `CALL SZ_RESYNC()` rebuilds the replica and un-parks the replicator |
| `DEGRADED` | a resolver instance reports `DEGRADED` (engine or repository errors) | `SZ_STATUS()` and the container log |
| `SHAPE_MISMATCH` | the replica's layout is not this version's | `CALL SZ_RESYNC()` |
| `LICENSE_EXPIRED` | the running license expired (`LICENSE_EXPIRES`) | put the new license in the license secret (`ALTER SECRET`), then `CALL SZ_RELOAD_SECRETS()` |
| `SOURCES_NOT_REGISTERED` | a source's code is not yet registered in the engine | wait for the resolver's first batch; then the container log |
| `CLAIMING_PAUSED` | claiming is paused outside an upgrade or suspend (an abandoned `SZ_PREPARE_UPGRADE`) | `SZ_START` or `SZ_RESUME` clears it |
| `WATCH_ERRORS` | a `SZ_WATCH` probe failed (`WATCH_ERRORS` lists them); that check kept its previous value | read the error; it is never reported as the condition it could not check |
Expand All @@ -121,6 +122,9 @@ paused until `SZ_UPGRADE` runs (the install script runs it when a re-run changes
| `REJECTED_KEYS` | records the engine rejected (`SZ.SZ_RESULT` has the reason per key) | fix the records; `SZ_REQUEUE_REJECTED('<CODE>')` |
| `DEAD` | entities or redo records that failed `TIMEOUT_MAX_ATTEMPTS` times | the container log; `SZ_RESYNC()` after the cause is fixed |
| `STARTING_STUCK` | an instance has been `STARTING` longer than `START_WAIT_SECS`: it is still retrying its Postgres connect | Postgres down, full or out of memory; the container log names the error |
| `LICENSE_EXPIRING` | the license expires within 30 days (`LICENSE_DAYS_LEFT`) | renew it, then update the secret and `SZ_RELOAD_SECRETS()` |
| `LICENSE_EVALUATION` | no license: the engine runs Senzing's 500-record evaluation limit (`LICENSE_RECORD_LIMIT`) | put your license in the license secret, then `SZ_RELOAD_SECRETS()` |
| `LICENSE_NEAR_LIMIT` | the crosswalk holds 90% or more of the license's record limit (`LICENSE_RECORDS_USED`) | a license with a larger limit |
| `CONNECTION_BUDGET` | `CONNECTION_DEMAND`, the sessions the services can hold at `RESOLVER_MAX_INSTANCES`, exceeds `CONNECTION_BUDGET` | lower `RESOLVER_MAX_INSTANCES` or the threads, or size Postgres up and set `PG_COMPUTE_FAMILY` |
| `APPLY_LAG` | work queued and no apply for an hour | replicator down (above), or the warehouse: `SZ_WATCH` sizes `<APP>_WH` up when an apply passes `APPLY_SLOW_SECS` |

Expand Down
Loading
Loading