Skip to content

fix(deps): pg-orm 2.2.4 to stop the connection pool being poisoned (PPT-2642) - #386

Merged
camreeves merged 1 commit into
masterfrom
fix/PPT-2642-pg-orm-2.2.4
Aug 5, 2026
Merged

fix(deps): pg-orm 2.2.4 to stop the connection pool being poisoned (PPT-2642)#386
camreeves merged 1 commit into
masterfrom
fix/PPT-2642-pg-orm-2.2.4

Conversation

@camreeves

Copy link
Copy Markdown
Contributor

One line in shard.lock: pg-orm 2.2.3 → 2.2.4.

What it fixes

A burst of concurrent POST /bookings left a connection stuck in an open transaction and back in the pool. Every write after that returned 500 with There is an existing transaction in this connection until the service restarted — and because reads kept working (they ran inside the orphaned transaction) the service looked healthy while being unable to book anything.

That error is a DB::Error, not a PQ::PQError, so wrap_in_transaction's retry filter never caught it either.

Cause

In pg-orm, not here. crystal-db's TopLevelTransaction#commit issues the COMMIT and only clears the connection's transaction flag afterwards in do_close, so a COMMIT that raises — exactly the serialization failures this controller retries — leaves the flag set for good. pg-orm's release rolled back only transactions it still had a record of, and its own ensure cleared that record first, so the single point where connections return to the pool was blind to it.

spider-gazelle/pg-orm#19 discards a connection that is still flagged rather than releasing it.

Verification

Measured against this service, with images differing only by the pg-orm version, under the load that triggers it (12–16 concurrent POST /bookings):

build bursts stranding a connection writable afterwards
control (pg-orm 2.2.3) 3 of 4 no — {"500":3}
pg-orm 2.2.4 0 of 11 yes — {"201":3}

The control was re-run in the same session afterwards to confirm the environment still reproduced rather than having gone quiet.

Connection reuse under normal sequential traffic is unchanged — backend PIDs are stable across bookings, with the same churn pattern as the control, so the new discard path is not firing when it should not.

Reproducer kept at user-interfaces/e2e/support/repro/reg09-concurrent-bookings.ts.

Why it matters now

This is the last thing standing between the new workplace e2e suite and a clean nightly record — its concurrent booking specs are exactly what trips this.

🤖 Generated with Claude Code

PPT-2642. A burst of concurrent POST /bookings left a connection stuck in an
open transaction and back in the pool, after which every write returned 500
with "There is an existing transaction in this connection" until the service
restarted. Reads kept working, because they ran inside the orphaned
transaction, so the service looked healthy while being unable to book
anything.

The cause was in pg-orm: crystal-db clears a connection's transaction flag
only after the COMMIT it issues, so a COMMIT that fails — the serialization
failures the booking controller retries — leaves it set, and the one place
connections return to the pool could not see it. spider-gazelle/pg-orm#19
discards such connections instead of releasing them.

Verified against this service under the load that triggers it: 12-16
concurrent POST /bookings, images differing only by the pg-orm version.
Before, a connection was stranded in 3 of 4 bursts and the service ended
unable to write at all; after, 0 of 11 bursts stranded anything and it kept
serving writes. Connection reuse under normal traffic is unchanged.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@github-actions github-actions Bot added the type: bug something isn't working label Aug 5, 2026
@camreeves
camreeves merged commit 25b429c into master Aug 5, 2026
11 checks passed
@camreeves
camreeves deleted the fix/PPT-2642-pg-orm-2.2.4 branch August 5, 2026 13:40
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

type: bug something isn't working

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant