fix(deps): pg-orm 2.2.4 to stop the connection pool being poisoned (PPT-2642) - #386
Merged
Conversation
PPT-2642. A burst of concurrent POST /bookings left a connection stuck in an open transaction and back in the pool, after which every write returned 500 with "There is an existing transaction in this connection" until the service restarted. Reads kept working, because they ran inside the orphaned transaction, so the service looked healthy while being unable to book anything. The cause was in pg-orm: crystal-db clears a connection's transaction flag only after the COMMIT it issues, so a COMMIT that fails — the serialization failures the booking controller retries — leaves it set, and the one place connections return to the pool could not see it. spider-gazelle/pg-orm#19 discards such connections instead of releasing them. Verified against this service under the load that triggers it: 12-16 concurrent POST /bookings, images differing only by the pg-orm version. Before, a connection was stranded in 3 of 4 bursts and the service ended unable to write at all; after, 0 of 11 bursts stranded anything and it kept serving writes. Connection reuse under normal traffic is unchanged. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
One line in
shard.lock: pg-orm 2.2.3 → 2.2.4.What it fixes
A burst of concurrent
POST /bookingsleft a connection stuck in an open transaction and back in the pool. Every write after that returned 500 withThere is an existing transaction in this connectionuntil the service restarted — and because reads kept working (they ran inside the orphaned transaction) the service looked healthy while being unable to book anything.That error is a
DB::Error, not aPQ::PQError, sowrap_in_transaction's retry filter never caught it either.Cause
In pg-orm, not here. crystal-db's
TopLevelTransaction#commitissues theCOMMITand only clears the connection's transaction flag afterwards indo_close, so a COMMIT that raises — exactly the serialization failures this controller retries — leaves the flag set for good. pg-orm'sreleaserolled back only transactions it still had a record of, and its ownensurecleared that record first, so the single point where connections return to the pool was blind to it.spider-gazelle/pg-orm#19 discards a connection that is still flagged rather than releasing it.
Verification
Measured against this service, with images differing only by the pg-orm version, under the load that triggers it (12–16 concurrent
POST /bookings):{"500":3}{"201":3}The control was re-run in the same session afterwards to confirm the environment still reproduced rather than having gone quiet.
Connection reuse under normal sequential traffic is unchanged — backend PIDs are stable across bookings, with the same churn pattern as the control, so the new discard path is not firing when it should not.
Reproducer kept at
user-interfaces/e2e/support/repro/reg09-concurrent-bookings.ts.Why it matters now
This is the last thing standing between the new workplace e2e suite and a clean nightly record — its concurrent booking specs are exactly what trips this.
🤖 Generated with Claude Code