feat: add Langfuse Wasm FDW to the catalog - #622
Conversation
|
hey @distanceqo , thanks for the PR! As this repo is mainly for Supabase maintained FDWs and it is targeting customer on Supabase platform, would you mind move source code into this PR as well? |
Reads trace and observation data from Langfuse, an open source LLM observability platform, so LLM spend can be attributed to rows in a local users table in SQL. Targets the Langfuse Public API rather than the ClickHouse store behind it, which upstream documents as an unstable contract and which Langfuse Cloud users cannot reach at all. Supports select on `traces` and `observations`, import foreign schema, limit and equality pushdown, and time bounds via fromTimestamp/toTimestamp (or fromStartTime/toStartTime on observations — the column and params differ per endpoint). Credentials resolve from a Vault secret name, a secret id, or a plaintext option. Usage and cost arrive as objects keyed by metric name and are flattened into scalar columns so aggregates need no JSON extraction. The fallback chains are load-bearing: `observations` returns calculatedTotalCost with no totalCost key, and promptTokens rather than usageDetails.input. Verified end to end on Supabase against a live Langfuse Cloud project.
6e2438d to
9cb376b
Compare
|
Thanks for taking a look! Source code is now in this PR under Moving into this repo meant a few changes over what I had originally published:
Two things I'd appreciate your call on:
For reference, this was verified end to end on Supabase against a live Langfuse Cloud project — auth out of Vault, both pagination styles, limit/equality/time pushdown, and a join against |
CI runs clippy with RUSTFLAGS="-D warnings", which promotes uninlined_format_args to an error. Nine call sites used the positional form.
|
Fixed the clippy failure — nine Verified with the same commands the workflow uses: All clean. The new run needs your approval to start, since I'm a first-time contributor here. One heads-up in case you hit it: running clippy across the whole |
Also, can you add smoke test case for the Wasm FDW? Below are the related files:
Once the test case is in place, you can run |
Mock responses are shaped so a regression fails rather than silently passing: - Both endpoints report page/totalPages instead of a cursor, and the server is configured with page_size 1 against a two-page traces mock, so the test only sees both rows if pagination is followed. - The traces mock returns nothing for any userId other than user-alice, so a dropped equality pushdown fails the assertion instead of falling back to unfiltered rows. - The two observation rows spell usage and cost differently — one nests them under usageDetails/costDetails, the other only has the flat promptTokens and calculated*Cost forms. Both must map to the same columns, so a missing fallback surfaces as a NULL.
|
Smoke test added — I tried to shape the mock so a regression fails rather than quietly passing:
I verified the mock endpoints directly with curl (pagination, the |
Previously page_size was only checked for being a valid integer; values outside the documented 1-1000 range (e.g. negative or larger than 1000) were silently accepted and passed straight through as the upstream limit= query parameter. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Adds catalog documentation for a community Wasm wrapper for Langfuse, an open source LLM observability platform.
The wrapper exposes traces, observations, token usage, and cost as foreign tables, so LLM spend can be attributed to rows in a local users table in plain SQL:
It targets the Langfuse Public API rather than the ClickHouse store behind it, which upstream documents as not a stable API contract and which Langfuse Cloud users cannot reach at all.
postgres-wasm-fdwtemplate, WIT0.1.0, hence>=0.4.0in the version table.Changes
docs/catalog/langfuse.mdgravatar.mddocs/catalog/index.mddocs/catalog/wasm/index.mdmkdocs.yamlVerification
Installed and queried on a Supabase project against a live Langfuse Cloud project. Everything the docs claim was checked rather than assumed:
limitand equality pushdown; time bounds viafromTimestamp/toTimestamp(andfromStartTime/toStartTimeon observations — the column and params differ per endpoint)sum(total_cost)agrees at0.15584acrosstracesandobservationsfor the same data despite reaching the value through different response fields (totalCostvscalculatedTotalCost), which is why the docs list fallback chainsauth.usersand a local table returns correct per-user attributionThe Limitations section records behaviour found during that testing which may be useful beyond this wrapper:
now()in a time filter is not pushed down — Postgres reportsWrappers: quals = []and filters locally after fetching every page. Literal timestamps are pushed.v2/observationsreads a different store thanobservations. On a fresh cloud project, rows ingested through both/api/public/ingestionand the OTLP endpoint were readable viaobservationswhilev2/observationsreturned an empty result, though it still validated query parameters.