Repository navigation
fix(db): require asyncpg>=0.32 so a dropped connection can't wedge the worker - #6107
Open
nicoloboschi wants to merge 1 commit into
Open
nicoloboschi wants to merge 1 commit into
nicoloboschi wants to merge 1 commit into
Conversation
…e worker (#6067) asyncpg <= 0.31 left a timed-out query's cancellation pending until the server answered it. When the connection died first, every later query on it awaited forever, including the ROLLBACK from transaction().__aexit__, so the worker's claim loop stopped silently while /health stayed green. asyncpg 0.32.0 resolves pending cancellations on connection loss and bounds the wait by the command timeout. Adds a regression test that drives PostgresConnection.transaction() through a proxy that swallows replies then drops the connection; it fails on 0.31 and passes on 0.32. 0.32 also makes the pool's min_size a maintained floor (idle connections are no longer closed below it); documented on HINDSIGHT_API_DB_POOL_MIN_SIZE.
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes #6067.
Problem
On asyncpg <= 0.31, when a worker claim query timed out and its connection then dropped, the transaction's ROLLBACK waited forever. The poll loop stopped claiming work with nothing logged,
/healthstayed 200, and operations piled up aspendinguntil a restart. The 1.1.0 image ships asyncpg 0.31.0.Fix
Raise the floor to
asyncpg>=0.32.0(lock: only asyncpg changes). 0.32 resolves pending cancellations when the connection is lost (MagicStack/asyncpg#1361) and bounds that wait by the command timeout (#1370). It's fixed at the source, so every transaction is covered, not just the worker claim.Behaviour changes
InterfaceError: connection is closed; the poll loop logs it and carries on.min_sizeconnections even when idle (before, idle ones could close and the pool could drain to zero). Default is 5; documented onHINDSIGHT_API_DB_POOL_MIN_SIZE, set0to allow draining.Tests
tests/test_db_dropped_connection.py: proxy swallows replies, query times out, connection drops. Fails on 0.31 (stuck in ROLLBACK after 10s), passes on 0.32 (~2s, stable over 5 runs).oracledb, not installed locally; unrelated).