Skip to content

Commit 9062ceb

Browse files
ralyodioclaude
andcommitted
feat(db): move from Turso/libSQL to Postgres via @profullstack/libsql-pg
packages/shared/lib/db.js now builds its client with @profullstack/libsql-pg, which keeps the @libsql/client surface (execute / batch / transaction) the layer was written against over a pg pool and rewrites the SQLite idioms the queries picked up on Turso (strftime('%Y-%m-%dT%H:%M:%fZ','now'), `?` placeholders, INSERT OR IGNORE) per statement. The exported `db` object (one / oneOrNone / manyOrNone / none / tx / batch) is unchanged, so no call site moved. - DATABASE_URL (postgres://) is the only accepted setting; unset, libsql:// or file: throw with the reason, and a leftover TURSO_DATABASE_URL is named with the copy recipe. The orchestrator checks it at start. No file fallback. - migrations-pg/ holds the Postgres schema, generated with `npx libsql-pg convert-schema` and hand-fixed: gen_random_uuid()::text ids (the carried-over randomblob()/random() expression does not run on Postgres), one BEFORE UPDATE trigger function for updated_at (the SQLite AFTER UPDATE triggers were emitted as TODOs), the `default 'x' check (...)` columns the converter mangled, identity bigint for the rowid-alias ids, and every timestamp kept as ISO text via meshhook_now_iso() so the app's `set updated_at = strftime(...)` keeps assigning text to text. - scripts/db-migrate.js applies migrations-pg, one transaction per file, and records checksums in schema_migrations as before; verify-migration.js reads information_schema / pg_indexes instead of sqlite_master and drops the PRAGMA check. setup.js collects a Postgres URL. The SQLite migrations stay in migrations/ until the cutover is proven. - The in-process write lock and SQLITE_BUSY retry (for libSQL's single connection) are gone; a retry for 40001/40P01 replaces them. - Tests: the queue, auth and orchestrator suites need a database and skip unless TEST_DATABASE_URL is set (each test then gets its own schema); db.test.js covers the guard and placeholder rewrite without one. The SvelteKit build externalises @profullstack/libsql-pg and pg. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
1 parent 8acc743 commit 9062ceb

25 files changed

Lines changed: 802 additions & 365 deletions

‎.env.example‎

Lines changed: 7 additions & 9 deletions
Original file line numberDiff line numberDiff line change
@@ -2,17 +2,15 @@
22
# Copy to .env and fill in real values. NEVER commit .env.
33

44
# ---------------------------------------------------------------------------
5-
# Database (Turso / libSQL)
5+
# Database (Postgres)
66
# ---------------------------------------------------------------------------
7-
# Local development — a plain SQLite file, no server required:
8-
# TURSO_DATABASE_URL=file:./meshhook.db
7+
# The only database MeshHook accepts. Anything but postgres:// fails at start
8+
# with the reason; there is no file-database fallback.
9+
DATABASE_URL=postgres://meshhook:password@localhost:5432/meshhook
910
#
10-
# Production — create the database and a token with the Turso CLI:
11-
# turso db create meshhook
12-
# turso db show meshhook --url
13-
# turso db tokens create meshhook
14-
TURSO_DATABASE_URL=libsql://your-database-your-org.turso.io
15-
TURSO_AUTH_TOKEN=your-turso-auth-token
11+
# The Turso settings are retired. A set TURSO_DATABASE_URL is named in the
12+
# error so a deploy that still carries it can be repointed:
13+
# npx libsql-pg copy --from "$TURSO_DATABASE_URL" --token "$TURSO_AUTH_TOKEN" --to "$DATABASE_URL" --verify
1614

1715
# ---------------------------------------------------------------------------
1816
# Secrets vault

‎README.md‎

Lines changed: 8 additions & 12 deletions
Original file line numberDiff line numberDiff line change
@@ -1,10 +1,10 @@
11
# MeshHook — Mesh your webhooks. Orchestrate everything.
22

3-
**MeshHook** is an MIT-licensed, webhook-first workflow engine with a visual builder (SvelteKit/Svelte 5) and Temporal-like durability via **event sourcing on SQLite (Turso)**.
3+
**MeshHook** is an MIT-licensed, webhook-first workflow engine with a visual builder (SvelteKit/Svelte 5) and Temporal-like durability via **event sourcing on Postgres**.
44

55
## Stack
66
- **UI/API**: SvelteKit (Svelte 5)
7-
- **Database**: Turso (libSQL/SQLite) — embeddable locally, replicated in production
7+
- **Database**: Postgres, through [`@profullstack/libsql-pg`](https://github.com/profullstack/libsql-pg) (the queries kept their libSQL shape)
88
- **Workers**: Node.js (or Bun), stateless
99
- **Queue**: SQLite-backed, with visibility timeouts and a dead-letter queue
1010
- **Auth**: self-hosted sessions (scrypt password hashing, opaque session tokens)
@@ -20,7 +20,7 @@
2020
| **Visual Builder** | ✅ Drag-and-drop | ❌ Code-first | ❌ Code-first | **✅ Visual + Code** |
2121
| **Webhook-Native** | ⚠️ Supported | ⚠️ Supported | ❌ Not primary | **✅ Built-in** |
2222
| **Self-Hosted** | ✅ Yes | ✅ Yes | ✅ Yes | **✅ Yes** |
23-
| **Database** | MySQL/Postgres | Postgres | Custom | **SQLite (Turso)** |
23+
| **Database** | MySQL/Postgres | Postgres | Custom | **Postgres** |
2424
| **Transforms** | JavaScript | TypeScript/Python | Any language | **JMESPath** |
2525
| **Realtime Logs** | ❌ Polling | ❌ Polling | ❌ Polling | **✅ Server-Sent Events** |
2626
| **Deployment** | Complex (multiple services) | Complex (workers + API) | Complex (server + workers) | **Simple (single service)** |
@@ -75,13 +75,9 @@
7575
```bash
7676
pnpm run setup
7777
```
78-
Select "Production" or "Staging" and enter your Turso database URL and auth token.
79-
Create them first with the Turso CLI:
80-
```bash
81-
turso db create meshhook
82-
turso db show meshhook --url
83-
turso db tokens create meshhook
84-
```
78+
Select "Production" or "Staging" and enter your Postgres URL
79+
(`postgres://user:pass@host:5432/meshhook`). Anything else is refused.
80+
8581

8682
3. **Run migrations**
8783
```bash
@@ -116,7 +112,7 @@
116112
## Documentation
117113

118114
- [`./docs/Environment-Setup.md`](./docs/Environment-Setup.md) - Detailed environment setup guide
119-
- [`./docs/Turso-Migration.md`](./docs/Turso-Migration.md) - What changed in the move off Supabase
115+
- [`./docs/Turso-Migration.md`](./docs/Turso-Migration.md) - What changed in the move off Supabase (to Turso, since superseded by Postgres: see `migrations-pg/`)
120116
- [`./docs/PRD.md`](./docs/PRD.md) - Product requirements
121117
- [`./docs/Architecture.md`](./docs/Architecture.md) - System architecture
122118
- [`./docs/diagrams/*.puml`](./docs/diagrams/) - PlantUML diagrams
@@ -129,4 +125,4 @@ MeshHook runs as a single service on port 8080 that handles:
129125
- Background job execution
130126
- HTTP request execution with retries
131127

132-
All components communicate through the Turso database - no inter-service HTTP calls needed.
128+
All components communicate through the Postgres database - no inter-service HTTP calls needed.

‎apps/web/package.json‎

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -12,11 +12,11 @@
1212
"format": "prettier --write ."
1313
},
1414
"dependencies": {
15-
"@libsql/client": "^0.15.7",
1615
"@meshhook/shared": "workspace:*",
1716
"@xyflow/svelte": "^1.3.1",
1817
"jmespath": "^0.16.0",
19-
"undici": "^6.19.8"
18+
"undici": "^6.19.8",
19+
"@profullstack/libsql-pg": "^0.1.1"
2020
},
2121
"devDependencies": {
2222
"@eslint/js": "^9.15.0",

‎apps/web/vite.config.js‎

Lines changed: 4 additions & 7 deletions
Original file line numberDiff line numberDiff line change
@@ -1,15 +1,12 @@
11
import { sveltekit } from '@sveltejs/kit/vite';
22

33
/**
4-
* @libsql/client loads a platform-specific native binding (e.g.
5-
* @libsql/linux-x64-gnu) with a dynamic require. Rollup cannot follow that, so
6-
* bundling it fails the build with "Could not dynamically require". Marking it
4+
* @profullstack/libsql-pg sits on `pg`, which tries an optional native binding
5+
* (pg-native) with a dynamic require that Rollup cannot follow. Marking both
76
* external leaves the import in place for Node to resolve from node_modules at
87
* runtime, which is what adapter-node expects anyway.
9-
*
10-
* Keep `libsql` alongside it — that is the package holding the bindings.
118
*/
12-
const NATIVE_DEPS = ['@libsql/client', 'libsql'];
9+
const NATIVE_DEPS = ['@profullstack/libsql-pg', 'pg', 'pg-native'];
1310

1411
export default {
1512
plugins: [sveltekit()],
@@ -24,7 +21,7 @@ export default {
2421
},
2522
build: {
2623
rollupOptions: {
27-
external: [/^\.\.\/\.\.\/workers\//, ...NATIVE_DEPS, /^@libsql\//]
24+
external: [/^\.\.\/\.\.\/workers\//, ...NATIVE_DEPS, /^pg-/]
2825
}
2926
}
3027
};

‎migrations-pg/0001_core_tables.sql‎

Lines changed: 159 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,159 @@
1+
-- MeshHook core tables, Postgres.
2+
--
3+
-- Generated from migrations/0001_core_tables.sql with `npx libsql-pg
4+
-- convert-schema`, then reviewed and hand-fixed. Applied by scripts/db-migrate.js
5+
-- when DATABASE_URL is postgres:// (the only URL the app accepts now).
6+
--
7+
-- Decisions, so the SQLite files and the app's queries keep working unchanged:
8+
--
9+
-- * Every timestamp column stays TEXT holding ISO-8601 UTC, exactly the value
10+
-- the app writes (new Date().toISOString()) and compares. The SQLite
11+
-- default strftime('%Y-%m-%dT%H:%M:%fZ', 'now') becomes meshhook_now_iso(),
12+
-- which produces the same 24-character string, and the libsql-pg rewriter
13+
-- turns the app's inline strftime(...) into the same to_char() expression.
14+
-- Promoting the columns to timestamptz (what the converter suggested) would
15+
-- break every `set updated_at = strftime(...)` the app issues, because
16+
-- Postgres will not assign text to a timestamptz column.
17+
-- * uuid ids default to gen_random_uuid()::text (Postgres 13+). The SQLite
18+
-- randomblob()/random() expression the converter carried over does not run
19+
-- on Postgres (random() is a double there, so `% 4` has no operator).
20+
-- * `integer primary key` (a rowid alias) becomes an identity bigint, so
21+
-- lastInsertRowid keeps working through libsql-pg (it appends RETURNING).
22+
-- * The AFTER UPDATE triggers become one BEFORE UPDATE trigger function.
23+
-- * 0/1 flag columns stay bigint with their CHECK (the app compares = 1).
24+
25+
create extension if not exists pgcrypto;
26+
27+
-- The stored timestamp format: 2026-08-11T12:00:00.000Z.
28+
create or replace function meshhook_now_iso() returns text
29+
language sql stable as $$
30+
select to_char(now() at time zone 'utc', 'YYYY-MM-DD"T"HH24:MI:SS.MS"Z"')
31+
$$;
32+
33+
-- Replaces the per-table AFTER UPDATE triggers: stamp updated_at on the row
34+
-- being written. workflow_runs adds a WHEN guard below so a statement that sets
35+
-- updated_at itself is left alone (the orchestrator sets finished_at and
36+
-- updated_at together).
37+
create or replace function meshhook_set_updated_at() returns trigger
38+
language plpgsql as $$
39+
begin
40+
new.updated_at := meshhook_now_iso();
41+
return new;
42+
end
43+
$$;
44+
45+
create table if not exists projects (
46+
id text primary key default (gen_random_uuid()::text),
47+
owner text not null,
48+
name text not null,
49+
created_at text not null default (meshhook_now_iso()),
50+
updated_at text not null default (meshhook_now_iso())
51+
);
52+
53+
create index if not exists idx_projects_owner on projects(owner);
54+
55+
drop trigger if exists update_projects_updated_at on projects;
56+
create trigger update_projects_updated_at
57+
before update on projects for each row
58+
execute function meshhook_set_updated_at();
59+
60+
-- Encrypted secrets vault. value_encrypted is AES-GCM ciphertext, bytea.
61+
create table if not exists secrets (
62+
id text primary key default (gen_random_uuid()::text),
63+
project_id text not null references projects(id) on delete cascade,
64+
"key" text not null,
65+
value_encrypted bytea not null,
66+
created_at text not null default (meshhook_now_iso()),
67+
updated_at text not null default (meshhook_now_iso()),
68+
unique(project_id, "key")
69+
);
70+
71+
create index if not exists idx_secrets_project_id on secrets(project_id);
72+
73+
drop trigger if exists update_secrets_updated_at on secrets;
74+
create trigger update_secrets_updated_at
75+
before update on secrets for each row
76+
execute function meshhook_set_updated_at();
77+
78+
create table if not exists workflow_definitions (
79+
id text primary key default (gen_random_uuid()::text),
80+
project_id text not null references projects(id) on delete cascade,
81+
slug text not null,
82+
name text not null,
83+
description text,
84+
status text default 'draft' check (status in ('draft', 'published', 'archived')),
85+
user_id text,
86+
version bigint not null default 1,
87+
definition text not null,
88+
created_at text not null default (meshhook_now_iso()),
89+
updated_at text not null default (meshhook_now_iso()),
90+
unique(project_id, slug, version)
91+
);
92+
93+
create index if not exists idx_workflow_definitions_project_id on workflow_definitions(project_id);
94+
create index if not exists idx_workflow_definitions_slug on workflow_definitions(project_id, slug);
95+
create index if not exists idx_workflow_definitions_status on workflow_definitions(status);
96+
create index if not exists idx_workflow_definitions_user_id on workflow_definitions(user_id);
97+
98+
drop trigger if exists update_workflow_definitions_updated_at on workflow_definitions;
99+
create trigger update_workflow_definitions_updated_at
100+
before update on workflow_definitions for each row
101+
execute function meshhook_set_updated_at();
102+
103+
create table if not exists workflow_runs (
104+
id text primary key default (gen_random_uuid()::text),
105+
project_id text not null references projects(id) on delete cascade,
106+
workflow_id text not null references workflow_definitions(id) on delete cascade,
107+
status text not null check (status in ('running', 'succeeded', 'failed', 'paused', 'canceled')),
108+
started_at text not null default (meshhook_now_iso()),
109+
finished_at text,
110+
created_at text not null default (meshhook_now_iso()),
111+
updated_at text not null default (meshhook_now_iso())
112+
);
113+
114+
create index if not exists idx_workflow_runs_project_id on workflow_runs(project_id);
115+
create index if not exists idx_workflow_runs_workflow_id on workflow_runs(workflow_id);
116+
create index if not exists idx_workflow_runs_status on workflow_runs(status);
117+
create index if not exists idx_workflow_runs_project_started on workflow_runs(project_id, started_at desc);
118+
119+
-- Deliberately guarded: the orchestrator sets finished_at and updated_at in one
120+
-- statement, and the trigger must not rewrite the value it just wrote.
121+
drop trigger if exists update_workflow_runs_updated_at on workflow_runs;
122+
create trigger update_workflow_runs_updated_at
123+
before update of status, finished_at on workflow_runs for each row
124+
when (new.updated_at is not distinct from old.updated_at)
125+
execute function meshhook_set_updated_at();
126+
127+
-- Event-sourcing log. The (run_id, ts) index is what every replay query uses;
128+
-- retention is scripts/prune-events.js.
129+
create table if not exists workflow_events (
130+
id bigint generated by default as identity primary key,
131+
run_id text not null references workflow_runs(id) on delete cascade,
132+
ts text not null default (meshhook_now_iso()),
133+
type text not null,
134+
payload text not null,
135+
created_at text not null default (meshhook_now_iso())
136+
);
137+
138+
create index if not exists idx_workflow_events_run_id on workflow_events(run_id);
139+
create index if not exists idx_workflow_events_run_ts on workflow_events(run_id, ts);
140+
create index if not exists idx_workflow_events_type on workflow_events(type);
141+
create index if not exists idx_workflow_events_ts on workflow_events(ts);
142+
143+
create table if not exists audit_log (
144+
id bigint generated by default as identity primary key,
145+
project_id text references projects(id) on delete cascade,
146+
user_id text not null,
147+
action text not null,
148+
resource_type text not null,
149+
resource_id text,
150+
metadata text,
151+
ip_address text,
152+
user_agent text,
153+
created_at text not null default (meshhook_now_iso())
154+
);
155+
156+
create index if not exists idx_audit_log_project_id on audit_log(project_id);
157+
create index if not exists idx_audit_log_user_id on audit_log(user_id);
158+
create index if not exists idx_audit_log_created_at on audit_log(created_at desc);
159+
create index if not exists idx_audit_log_action on audit_log(action);

‎migrations-pg/0002_auth.sql‎

Lines changed: 51 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,51 @@
1+
-- Authentication tables, Postgres. See migrations/0002_auth.sql for the
2+
-- reasoning; the shape is the same, with the conventions of 0001 (text ISO
3+
-- timestamps, gen_random_uuid() ids, a BEFORE UPDATE trigger for updated_at).
4+
5+
create table if not exists users (
6+
id text primary key default (gen_random_uuid()::text),
7+
-- Stored lower-cased; the unique index is therefore already case-insensitive.
8+
email text not null unique,
9+
-- scrypt output, "scrypt$N$r$p$salt$hash". Null for external identities.
10+
password_hash text,
11+
email_verified bigint not null default 0 check (email_verified in (0, 1)),
12+
created_at text not null default (meshhook_now_iso()),
13+
updated_at text not null default (meshhook_now_iso())
14+
);
15+
16+
create index if not exists idx_users_email on users(email);
17+
18+
drop trigger if exists update_users_updated_at on users;
19+
create trigger update_users_updated_at
20+
before update on users for each row
21+
execute function meshhook_set_updated_at();
22+
23+
-- Opaque session tokens; only the SHA-256 of the token is stored.
24+
create table if not exists sessions (
25+
id text primary key,
26+
user_id text not null references users(id) on delete cascade,
27+
expires_at text not null,
28+
created_at text not null default (meshhook_now_iso()),
29+
last_seen_at text not null default (meshhook_now_iso()),
30+
user_agent text,
31+
ip_address text
32+
);
33+
34+
create index if not exists idx_sessions_user_id on sessions(user_id);
35+
create index if not exists idx_sessions_expires_at on sessions(expires_at);
36+
37+
create table if not exists user_settings (
38+
id text primary key default (gen_random_uuid()::text),
39+
user_id text not null unique references users(id) on delete cascade,
40+
theme_preference text default 'light' check (theme_preference in ('light', 'dark')),
41+
created_at text not null default (meshhook_now_iso()),
42+
updated_at text not null default (meshhook_now_iso())
43+
);
44+
45+
create index if not exists idx_user_settings_user_id on user_settings(user_id);
46+
create index if not exists idx_user_settings_theme on user_settings(theme_preference);
47+
48+
drop trigger if exists update_user_settings_updated_at on user_settings;
49+
create trigger update_user_settings_updated_at
50+
before update on user_settings for each row
51+
execute function meshhook_set_updated_at();

0 commit comments

Comments
 (0)