Skip to content

Add lock timeout retry support to apply - #558

Open
tianzhou wants to merge 2 commits into
mainfrom
claude/eager-pasteur-dirv8v
Open

Add lock timeout retry support to apply#558
tianzhou wants to merge 2 commits into
mainfrom
claude/eager-pasteur-dirv8v

Conversation

@tianzhou

Copy link
Copy Markdown
Contributor

Summary

  • Adds --lock-timeout-retries and --lock-timeout-retry-wait flags to pgschema apply. When a statement fails because it could not acquire a lock before --lock-timeout expired (PostgreSQL error 55P03), it is retried with exponential backoff (capped at 30s) instead of failing immediately.
  • Applies to both the concatenated-transaction execution path and the individual-statement execution path (e.g. CREATE INDEX CONCURRENTLY).
  • --lock-timeout-retries requires --lock-timeout to be set, since without a lock timeout a statement blocks indefinitely instead of failing with a retryable error.
  • Non-lock-timeout errors are never retried and are returned immediately.

This addresses #557, which asked for a retry mechanism for gracefully handling lock contention on busy tables (per the referenced zero-downtime migrations technique).

Test plan

  • go build ./...
  • go vet ./...
  • New unit tests for isLockTimeoutError and execWithLockRetry (success-after-retries, retries-exhausted, non-retryable error, zero retries, context cancellation during backoff)
  • New integration tests against embedded Postgres: one where a concurrently held lock is released mid-retry and apply succeeds, one where retries are exhausted while the lock is still held and apply fails with the underlying 55P03 error
  • Full go test ./cmd/apply/... suite passes
  • Updated docs/cli/apply.mdx with the new flags and an example

Generated by Claude Code

Statements that fail to acquire a lock before --lock-timeout expires
(SQLSTATE 55P03) can now be retried with exponential backoff via the
new --lock-timeout-retries and --lock-timeout-retry-wait flags, easing
migrations against busy tables (fixes #557).

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011yLiZRSQzzPgpGz8ymX9SV
Copilot AI lite review requested due to automatic review settings August 26, 2026 13:17
@greptile-apps

greptile-apps Bot commented Aug 26, 2026

Copy link
Copy Markdown
Contributor

Greptile Summary

The PR adds configurable retries with capped exponential backoff for PostgreSQL lock-not-available errors in both grouped and individual migration execution paths.

  • Adds retry flags and propagates their configuration through apply execution.
  • Retries SQLSTATE 55P03 failures while respecting context cancellation.
  • Adds unit and embedded-PostgreSQL integration coverage.
  • Documents the new flags and usage.

Confidence Score: 4/5

The PR appears safe to merge, with non-blocking input and API validation gaps that should be tightened to keep retry behavior predictable.

The retry paths are covered and preserve immediate handling of non-retryable errors, but malformed backoff values can disable the intended waiting behavior and direct ApplyMigration callers can bypass the lock-timeout prerequisite.

Files Needing Attention: cmd/apply/apply.go

Important Files Changed

Filename Overview
cmd/apply/apply.go Adds lock retry configuration and execution logic, but leaves retry inputs and direct ApplyMigration configurations insufficiently validated.
cmd/apply/apply_test.go Covers flag registration and the CLI-level dependency between retry count and lock timeout.
cmd/apply/lock_retry_test.go Exercises retry success, exhaustion, non-retryable errors, disabled retries, and cancellation.
cmd/apply/lock_retry_integration_test.go Verifies successful recovery and retry exhaustion against embedded PostgreSQL.
docs/cli/apply.mdx Documents the retry flags, backoff behavior, prerequisites, and an example invocation.

Flowchart

%%{init: {'theme': 'neutral'}}%%
flowchart TD
  Apply[Apply migration group] --> Exec[Execute SQL]
  Exec -->|Success| Next[Continue migration]
  Exec -->|Non-55P03 error| Fail[Return error]
  Exec -->|55P03 and retries remain| Wait[Wait with capped exponential backoff]
  Wait -->|Context cancelled| Cancel[Return context error]
  Wait -->|Timer expires| Exec
  Exec -->|55P03 and retries exhausted| Fail
Loading

Reviews (1): Last reviewed commit: "Add lock timeout retry support to apply" | Re-trigger Greptile

Comment thread cmd/apply/apply.go
Comment thread cmd/apply/apply.go

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Adds lock-timeout retry behavior to pgschema apply so migrations can tolerate transient lock contention by retrying 55P03 failures with exponential backoff.

Changes:

  • Introduces --lock-timeout-retries and --lock-timeout-retry-wait flags and wires them into both grouped (concatenated) and per-statement execution paths.
  • Implements 55P03 detection and a reusable execWithLockRetry helper with exponential backoff capped at 30s.
  • Adds unit + embedded-Postgres integration tests, and documents the new flags with an example.

Reviewed changes

Copilot reviewed 5 out of 5 changed files in this pull request and generated 3 comments.

Show a summary per file
File Description
docs/cli/apply.mdx Documents the new retry flags and provides an example invocation.
cmd/apply/apply.go Adds retry configuration/flags, validation, and lock-timeout retry execution logic.
cmd/apply/apply_test.go Extends CLI flag/validation tests for the new options.
cmd/apply/lock_retry_test.go Adds unit tests for SQLSTATE detection and retry behavior.
cmd/apply/lock_retry_integration_test.go Adds integration tests validating retries succeed/fail as expected under real lock contention.

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

Comment thread cmd/apply/apply.go
Comment thread cmd/apply/apply.go Outdated
Comment thread cmd/apply/apply.go
Comment on lines +580 to +584
select {
case <-ctx.Done():
return nil, ctx.Err()
case <-time.After(backoff):
}

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Not making this change. time.After in a select/ctx.Done() loop only "leaks" a timer until it fires or becomes unreachable; since Go 1.23 the runtime can garbage-collect an unreferenced, unstopped Timer without waiting for it to fire, so there's no real leak here. This loop also isn't a hot path - it only re-fires on an actual lock-timeout error, bounded by --lock-timeout-retries (a handful of iterations per statement at most), and the existing wait-directive polling loop in cmd/apply/directive.go uses the exact same time.After pattern for consistency. Happy to switch both to time.NewTimer if this ever becomes a tight, high-frequency loop, but for now it'd be defensive coding against a cost that isn't there.


Generated by Claude Code

Reject negative --lock-timeout-retries and a non-positive
--lock-timeout-retry-wait instead of silently treating them as no
retries / a tight retry loop. ApplyMigration re-validates the same
invariants so direct callers can't bypass the CLI's checks. Also
reword the retry progress message from "attempt" to "retry" to match
what the counter actually tracks.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01FA3axbY2LQkGbN7b6d2fY8
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants