Keep REST endpoints responsive when bitcoind is slow

[EddieHouston]

The REST API stops serving all requests when bitcoind is slow or unresponsive — including endpoints that never touch the daemon.

What happens today

Every REST request is dispatched through an async fn on the Tokio runtime, but nearly all of the handlers are synchronous. They read RocksDB, and POST /tx and POST /txs/package make a blocking JSON-RPC call to bitcoind. A blocking call on an async worker thread ties that thread up for its whole duration.

The runtime has one worker per CPU core, so it only takes a handful of concurrent broadcast requests against a slow daemon to occupy all of them. Once that happens nothing else gets polled, and cheap in-memory endpoints like GET /blocks/tip/height hang too.

It is made worse by all daemon RPC sharing a single connection behind one mutex, with a 10 minute read timeout. A single slow sendrawtransaction blocks every other daemon user — indexing included — and holds a worker thread for up to those 10 minutes.

You can see it locally: pause bitcoind (docker pause, or kill -STOP), fire one POST /tx per CPU core, then ask for /blocks/tip/height. It hangs.

Changes

Run the synchronous handlers on the blocking pool. handle_request now hands the router off to tokio::task::spawn_blocking. GET /block-template is the one genuinely async handler (concurrent callers share a single in-flight fetch) and already offloads its own blocking work, so it stays on the runtime. This keeps the runtime free to answer everything else no matter how long a daemon call takes.

Give client-triggered daemon calls their own bounded path. sendrawtransaction and submitpackage are reachable anonymously over both REST and Electrum, so they no longer use the shared singleton connection. Each call now gets a short-lived connection with a 30 second timeout, and a semaphore caps them at 8 in flight. One slow request can no longer stall indexing or other clients, and the number of threads a wedged daemon can park is explicitly bounded. This reuses the same one-shot connection path getblocktemplate already uses.

Failures are now reported to the caller rather than retried forever, which is what the old path did.

Report daemon problems as gateway errors. These used to surface as 400 Bad Request, which blames the caller for a perfectly valid request and lets caches and load balancers memoize it. Now:

• 503 when the concurrency cap is reached, or bitcoind is still warming up
• 504 when the daemon does not answer in time

Electrum clients get a DaemonError for both.

Tuning

The defaults should be fine as-is:

Variable	Default
DAEMON_PROXY_MAX_CONCURRENCY	8	concurrent client daemon calls
DAEMON_PROXY_RPC_TIMEOUT	30s	I/O timeout for those calls
DAEMON_PROXY_QUEUE_TIMEOUT	5s	how long to wait for a free slot

A new daemon_rpc_proxied Prometheus counter tracks these calls by result (ok / busy / unavailable / error).

Testing

• 8 new unit tests covering the semaphore, the route split, and the status mapping
• Full suite passes on Bitcoin (49 unit, 25 REST, 8 Electrum) and on Liquid (57 unit, 25 REST)

Known gap

In --lightmode, transaction and block reads still go through the shared connection with the long timeout. Those calls are shared with the indexer, so separating them is a larger change. They no longer block the async runtime, so the problem above is fixed either way, but a lightmode deployment could still fill the blocking pool if bitcoind is down. Worth a follow-up.

[FedOken]

light_mode read paths still unbounded

broadcast_raw/submit_package are fixed here, but --lightmode reads (get_block_txids, get_block_meta, get_block_raw, lookup_raw_txn in schema.rs) still call daemon.getblock_raw/gettransaction_raw via the old request() → shared Mutex<Connection> path (600s timeout, unbounded retry), reachable anonymously via GET /tx/:txid, /block/:hash, etc.

Blast radius is smaller now (blocking pool, not async workers), but the shared-connection bottleneck remains for these. Worth a follow-up applying request_proxied there too?
Those methods are also called internally by the indexer in light_mode, so we'd want to route only the REST-triggered calls through request_proxied, not the indexer's own.