Skip to content

Fix deadlock from recursive RLock in Order() - #555

Merged
aarongable merged 1 commit into
letsencrypt:mainfrom
notroj:fix-order-recursive-rlock
Sep 22, 2026
Merged

aarongable merged 1 commit into
letsencrypt:mainfrom
notroj:fix-order-recursive-rlock

Conversation

@notroj

@notroj notroj commented Sep 22, 2026

Copy link
Copy Markdown
Contributor

Proposed fix for #554 from Claude. I've stress tested this locally and have not hit the issue again so far.

Order() held the order read lock across orderForDisplay(), which takes
it again; if a writer queued up in between, the second RLock() blocked
behind it and the writer waited for the first, deadlocking both. Read
the account ID and status under the lock and release it before the
call.

Fixes letsencrypt#554

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
@aarongable
aarongable merged commit 755eb89 into letsencrypt:main Sep 22, 2026
15 checks passed
cursor Bot pushed a commit to CIYAhq/playkeeper that referenced this pull request Sep 26, 2026
…ble has stored its certificate

Pebble v2.10.1 deadlocks on a look at an order that arrives while it
stores the order's certificate. The order handler read-locks the order
twice (wfe.go:2069, then orderForDisplay at 1864), and CompleteOrder's
order.Lock(), just after it logs "Issued certificate serial ... for
order ..." (ca.go:471-474), can land between the two. The look is never
answered, and since GetOrderByID keeps the store's read lock while it
waits for the order, another look at it stops Pebble answering anything.
The fix, letsencrypt/pebble#555, is not in a release yet.

That is the acme-pebble failure on 04da0bd: finalize, one look, "Issued
certificate", then nothing until the 10-second request timeout. The
client now tries such a look again, which would freeze a deadlocked
Pebble, so the test's transport holds a look at a finalized order until
Pebble has logged its certificate, and 100 ms more. It can go once a
Pebble release has the fix.

Locally, 12 clients that each finalize and then look at their order hung
an unguarded Pebble after 5,390, 15,145 and 17,443 orders, each time on
that sequence. Through the guard, 34,880 orders went through.

Co-authored-by: Siya <siyabendoezdemir@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants