fix(payments): read every IBEX status field; never invent a settled send

[islandbitcoin]

Symptom (found live)

A Cash→BTC conversion on TEST reported "conversion successful" while no funds moved. IBEX recorded no send transaction for the attempt (verified by querying the account directly).

Cause

The IBEX send resolvers read the payment state from exactly one place — transaction.payment.status.id — and silently defaulted to Pending for anything else.

1. The state lives in up to three fields. Per the generated PayInvoiceV2 schema, IBEX returns a top-level status, plus transaction.payment.statusId and transaction.payment.status.id. Unset integers arrive as 0 rather than being omitted, so a single-path read — or lnurl-payment-send's status?.id ?? statusId chain, where a legitimate 0 short-circuits the ?? — can miss a status IBEX actually reported and call a settled or failed payment "pending".

2. The fallback was silent. A response we couldn't parse at all was indistinguishable from a genuine in-flight payment, with nothing recorded. Combined with the client treating PENDING as done, a payment that never executed rendered as a completed conversion.

Change

New shared @services/ibex/payment-status, with two readers for two dialects — the endpoints do not return the same shape:

• paymentSendStatusFromIbex (payInvoiceV2) takes the first recognised id across all three status fields, so an unset 0 can't mask a real status. The top-level status is read only when both payment-level fields are unreadable, and then it can never invent a terminal outcome in either direction:

  • never Success from a lone top-level 2;
  • never Failure from a lone top-level 3 either — accepted only when IBEX also names a failure code (failureReason or transaction.payment.failureId: a number > 0, or a named string reason like "NO_ROUTE"; only an explicit zero or blank means "no failure"). A fabricated "failed" sends the user back to retry with a fresh idempotency key against a send IBEX may in fact have made, which is the same double-pay hazard withPaymentIdempotency (Collapse payInvoice raw-fetch seam now that ibex-client 3.3.0 ApiError carries the body #478) exists to close.
  • Pending needs no gate: it is the outcome the unconfirmed path reports anyway.

• paymentSendStatusFromIbexLnurlPay (payToLnurl 201) — this response carries no top-level status and no transaction.payment.status object. Settlement is read from payment-level evidence only: a recognised transaction.payment.status.id / statusId, or failing those a populated transaction.payment.settleDateUtc (integer epoch or ISO string — that field has no declared type, and every payment-level date this vendor does declare is a string — rejecting 0 and the "0001-01-01T00:00:00Z" zero-date sentinel alike).

  The top-level settleDateUtc is deliberately not read. The vendor's documented 201 example sets it to 1668544241, which is transaction.createdAt ("2022-11-15T20:30:41.960887Z") floored to the second, while every payment-level field in the same body says nothing settled: statusId: 0, payment.settleDateUtc: null, paidMsat: 0, payment.hash: null. On the example's own evidence, then, a 201 is an acceptance rather than a settlement — reading that echo would report every LNURL send as success regardless of outcome, i.e. the exact bug at the top of this PR, on the one rail whose real response has never been observed. The example is now committed as a fixture (test/flash/mocks/ibex/pay-to-lnurl.ts) and both readers are asserted against it.

  Correction — there is no settlement webhook. An earlier revision of this description, and of the reader's doc block, justified the acceptance reading with "Ibex.payToLnurl also registers an async settlement webhook". That was wrong. payToLnurl does pass a webhookUrl (client.ts:348), but the value it passes is ibexWebhookEndpoints.onPay.lnurl, which resolves to <uri>/pay/lnurl/:username — the express route template from webhook-config.ts:13, :username never substituted. The only route mounted at that path is the public GET LNURL-pay callback for inbound payments (webhook-server/routes/on-pay.ts:143); there is no POST handler for it, and the two POST webhook routes that do exist (/pay/invoice, /pay/onchain) are resp.status(200).end() stubs. So an LNURL send this reader reports as pending is never resolved server-side — no webhook, no poller, no reconciliation job — and withPaymentIdempotency replays that pending for 24h under the same key. The conclusion above (do not read the top-level settleDateUtc) is unaffected: it rests on the example's payment-level fields, not on the webhook. The gap is now stated in the reader's doc block and pinned by test/flash/unit/services/ibex/webhook-server/routes/on-pay-lnurl-gap.spec.ts, which fails the moment a real POST handler is mounted, so whoever wires one is forced back to both. See the Known gap section.

• Neither reader ever guesses: with no readable field they return UnconfirmedIbexPayment. paymentSendStatusOrPending / lnurlPaymentSendStatusOrPending — what the resolvers call — record that on the current span at Warn (both readers) with the transaction id, account id and payment hash attached, and still return Pending. (On which span channel, and why level alone was not enough, see "Alerting keys on span status" under Known gap.) Warn rather than Critical because neither endpoint has a captured response: the committed fixtures are the vendor's openapi examples, and the pre-PR intraledger code read the top-level status alone (with a { status: 2 } fixture modelling exactly that), so a rail populating only the field the reader treats as uncorroborated is a live possibility. Paging on every flash-to-flash send until a capture proves otherwise would be an outage, not a severity. Capture one payInvoiceV2 200 and one payToLnurl 201 on TEST, commit them beside the examples, then raise the levels (noted in both readers' doc blocks).

  Why still Pending, deliberately: withPaymentIdempotency (Collapse payInvoice raw-fetch seam now that ibex-client 3.3.0 ApiError carries the body #478) caches only a definitive PaymentSendStatus; an ApplicationError return is left uncached so a same-key retry can re-execute. Returning an error here would therefore make the one case where we don't know whether funds moved the one case a same-key retry could pay twice. Pending keeps it cacheable and replayable. What changes is that the anomaly is recorded rather than invisible, and that a status IBEX did report can no longer be missed.

Applied to four call sites: ln-invoice-payment-send, ln-noamount-usd-invoice-payment-send, send-intraledger (all three on the payInvoiceV2 reader) and lnurl-payment-send (on the LNURL reader), replacing four private copies of the status switch. lnurl-payment-send's local IbexPaymentStatus type and helper go away.

⚠️ Behaviour change on the intraledger rail — read before merging

intraledger: an unreadable IBEX status now returns pending where it previously returned a GraphQL error. send-intraledger used to return UnexpectedIbexResponse for both the 0 case and the default case; intraledger-usd-payment-send.ts maps that to { status: "failed", errors: [...] }. It now reports { errors: [], status: "pending" }, for the double-pay reason above.

A bare top-level status: 3 falls into the same bucket: intraledger's old switch reported it as failed outright, and it now reports pending unless IBEX also names a failure code (see the failure-corroboration rule above).

Since the shipped mobile client renders PENDING as a completed conversion — the symptom this PR opens with — flash-to-flash USD sends move from fail-closed to fail-open on that one case until the client fix lands. This PR should therefore be sequenced behind lnflash/flash-mobile#699, which makes the app render pending honestly. The change is deliberate and pinned by a test (test/flash/unit/app/payments/send-intraledger.spec.ts), documented at the call site and on paymentSendStatusOrPending — it is not an incidental side effect of the refactor.

Behaviour change on the LN rails too — correcting an earlier claim in this description

An earlier revision of this description said "the three LN rails are unaffected: their inline switches already produced Pending for these responses." That was wrong, and the merge-sequencing argument above should not be read as resting on it. The old LN switches read transaction.payment.status.id and nothing else, defaulting to Pending, so two readings genuinely change on ln-invoice-payment-send and ln-noamount-usd-invoice-payment-send:

• A corroborated top-level failure is now failed where it used to be pending. { status: 3, failureReason: 2, transaction: { payment: { statusId: 0 } } } returned pending from the old switch; it now returns failed. Pinned by test/flash/unit/graphql/ln-invoice-payment-send.spec.ts ("reports a corroborated top-level failure").
• A flat payment-level statusId is now read at all. { transaction: { payment: { statusId: 2 } } } (or 3) returned pending from the old switch and now returns success / failed — a status IBEX did report and we were dropping.

Both are fail-closed movements on those rails (pending → a definite outcome), the opposite direction to the intraledger change, and the shipped client already handles them without extra sequencing: send-bitcoin-confirmation-screen.tsx routes SUCCESS/PENDING to the success screen and everything else — FAILURE included — to the error path (errorsMessage || "Something went wrong", error haptic). No client change is required for the LN rails; flash-mobile#699 remains the sequencing constraint for the intraledger pending case only.

Deliberately not touched: the generic "An unexpected error occurred" message for non-insufficient-balance IBEX errors — #476 landed typed INSUFFICIENT_BALANCE and consciously kept the generic text for the rest.

The client half of the bug (the app rendering PENDING as a completed conversion, plus swallowed fee-probe errors) is lnflash/flash-mobile#699.

Known gap

Neither endpoint has a captured response. test/flash/mocks/ibex/pay-invoice.ts and test/flash/mocks/ibex/pay-to-lnurl.ts are the vendor's openapi examples, committed verbatim and labelled as such, not observations — they are in the repo so the readers are asserted against the only evidence that exists rather than against shapes we imagined. Two things follow, both deliberate:

• The LNURL settleDateUtc rule is payment-level only and schema-derived; on the documented 201 example the reader reports pending, which is what shipped before this PR. It cannot report success from a field the vendor's own example populates on an unsettled payment.
• Both unreadable cases record at Warn, not Critical.

Capture one payInvoiceV2 200 and one payToLnurl 201 on TEST, commit them beside the examples, then revisit both the settle-date rule and the severities.

A pending LNURL send is never resolved server-side. Stated plainly because the correction above removed the sentence that implied otherwise: payToLnurl is handed a webhookUrl, but this repo mounts no POST handler behind it (the path it sends is the un-substituted route template /pay/lnurl/:username, and the only route there is the public inbound GET callback). There is no poller and no reconciliation job either. So on the LNURL rail, pending is terminal in practice — withPaymentIdempotency replays it for 24h under the same key, and after flash-mobile#699 makes the client render pending honestly, such a send shows "pending" to the user indefinitely. This is pre-existing (the rail already reported pending for this shape before this PR); what this PR changes is that the gap is documented rather than implied away, and pinned by a test that fails when a real handler lands. Wiring that handler — or a reconciler — is the follow-up.

Alerting keys on span status, not just error.level — checked. recordException (services/tracing.ts:354) calls span.setStatus({ code: SpanStatusCode.ERROR }) unconditionally; ErrorLevel only decides which error.* attributes win in updateErrorForSpan. Recording every unreadable response as an exception would therefore have turned 100% of the LNURL rail's spans red if the 201 acceptance shape is the norm — this PR's own position — and the same on intraledger if the payment-level fields turn out unpopulated. recordUnconfirmed now splits by evidence rather than by severity: a response that reported nothing at all gets span attributes plus an ibex.payment.unconfirmed event (fully queryable, span stays green), while a response carrying a top-level SUCCEEDED/FAILED the corroboration rules refused keeps the recorded exception — there the payload disagrees with itself and the pending may be wrong in a direction that moved money. Carried on UnconfirmedIbexPayment.uncorroboratedOutcome; pinned at the reader and on all three payInvoiceV2 rails.

Tests

• Unit tests on the two readers (test/flash/unit/services/ibex/payment-status.spec.ts): field precedence when fields disagree, the unset-0 mask, failure-code corroboration including named string reasons, the payment-level settleDateUtc in both serialisations plus the zero-date sentinel, the top-level settleDateUtc being ignored in every serialisation, span-attribute identity for both dialects, which of the two span channels each unreadable shape earns (quiet event vs recorded exception, asserted as exactly one and never both), and four invariants — never Success without an explicit success id, never Failure without a corroborated one, an unreadable response is always Pending, and the recorded severity is the error's own.
• The cashout rail now reads the status field at all. ValidOffer.execute checked only resp instanceof IbexError, so a 200 carrying transaction.payment.status.id === 3 counted as paid and Flash submitted a cashout in ERPNext — on the fiat-payout rail, with no balanceVerifier.verifyBalanceMove backstop, this PR's headline defect one call site over. It now refuses submitCashout on Failure (new FailedIbexPayment, mapped in error-map, reported to ops as the payInvoice step) and still submits on Pending, deliberately and documented: the cashout is an async reconciled flow, and holding the submit on a payment that probably settled would strand funds in a draft nobody watches. Pinned in test/flash/unit/app/offers/ops-events-valid-offer.spec.ts.
• The LNURL webhook gap is pinned (test/flash/unit/services/ibex/webhook-server/routes/on-pay-lnurl-gap.spec.ts): the webhook URL handed to IBEX is an un-substituted route template, the only method mounted at that path is GET, and the router's complete POST set is the two no-op stubs. Mounting a real handler fails this test, forcing the reader's doc block back into review.
• Both vendor examples are pinned as fixtures. The payToLnurl 201 example must read as unconfirmed → pending (never success), and the payInvoiceV2 200 example as in-flight — the regression tests for "a creation-time echo is not a settlement".
• Call-site wiring assertions on all four rails. The two readers are structurally interchangeable at the type level (lnurlPaymentSendStatusOrPending accepts a payInvoiceV2 response without complaint), so wrong-reader-on-wrong-endpoint — the bug that had shipped on LNURL — is invisible to the readers' own unit suite and only an assertion at the call site catches it. Each rail asserts a payment-level settle, the unreadable case, and a reading only its own reader makes (the LN rails: a corroborated top-level failure, and the unreadable message in the payInvoiceV2 dialect; LNURL: settlement from a payment-level settle date). ln-noamount-usd-invoice-payment-send gets its first spec file.
• Unit suite: 186 suites / 1710 tests (1707 passing, 3 pre-existing skips). tsc-check, tsc-check-noimplicitany, eslint, build, madge-check and check-yaml all clean.

(Rebased onto current main — the first push was based on a 40-commit-stale main, which is what the earlier schema check was complaining about.)

🤖 Generated with Claude Code

https://claude.ai/code/session_01NEoz7nBtdtsHyuYG5wNPQV