Skip to content

fix(postgres): probe over TCP so dependents aren't released early - #241

Merged
slayerjain merged 2 commits into
mainfrom
fix/pg-healthcheck-tcp
Sep 4, 2026
Merged

fix(postgres): probe over TCP so dependents aren't released early#241
slayerjain merged 2 commits into
mainfrom
fix/pg-healthcheck-tcp

Conversation

@slayerjain

Copy link
Copy Markdown
Member

The bug

postgres' docker-entrypoint runs a temporary server reachable only over the Unix socket (listen_addresses='') once initdb has finished, so it can create the database and run docker-entrypoint-initdb.d.

A pg_isready healthcheck without -h talks to that socket. So it reports "accepting connections" while no TCP listener exists at all — compose marks the service healthy, depends_on: condition: service_healthy releases the dependent, and the dependent connects straight into ECONNREFUSED.

It only bites on a fresh volume, because a populated one skips initdb and the temporary server entirely. That is why it surfaces as a rare flake rather than a consistent failure.

Evidence

From the CI failure this was diagnosed from (enterprise umami-linux, a matrix cell that creates its volume):

event time
temp server "ready to accept connections" (socket only) 08:54:29.422 — false-healthy begins
fast shutdown request 08:54:30.668 — window 1.25s
shutdown checkpoint 08:54:30.78008:54:33.421 (2.7s)
real TCP listener after 08:54:33.583

compose logged Container umami_db WaitingContainer umami_app Started, then the app died with Can't reach database server at 172.78.0.10:5432, while the db showed Up 11 seconds (healthy).

Verification

Reproduced and mutation-tested locally with a compose mirroring the depends_on: service_healthy shape:

healthcheck dependent
pg_isready -U ... -d ... APP_FAILED_TO_REACH_DB, exit 1
pg_isready -h 127.0.0.1 -U ... -d ... APP_OK, exit 0

Directly measured the false-healthy window against the patched file: the old check reports healthy from t=2s while TCP is refused until t=8s; the patched check holds starting and only goes healthy once TCP is genuinely up. Reproduced on postgres 13.3, 15 and 16.

Why -h 127.0.0.1

The stock images ship listen_addresses = '*', so the real server binds 0.0.0.0 — one bind() covers loopback and the container IP, and the static ipv4_address is configured before PID 1 runs, so there is no phase where loopback works and the network IP does not. During startup/recovery pg_isready reports rejecting connections, so the probe correctly keeps failing until the server can actually serve. localhost would be the wrong literal (it may resolve ::1 first).

Note on pg_isready semantics

pg_isready is a liveness probe: it exits 0 even for a nonexistent role/database, and even when pg_hba.conf rejects the connection. The -U/-d flags here are effectively decorative. That is fine for this fix — the real listener only starts after initdb created the role and database — but it does mean the probe is not a substitute for an application-level readiness check.

…arly

postgres' docker-entrypoint runs a temporary server reachable only over the
Unix socket (listen_addresses='') once initdb has finished, so it can create
the database and run docker-entrypoint-initdb.d. A pg_isready probe without
-h talks to that socket, so it answers "accepting connections" while no TCP
listener exists at all. compose then marks the service healthy and
depends_on: service_healthy releases the dependent straight into
ECONNREFUSED.

Measured at ~235ms with no initdb.d scripts, and at 1.2s of false-healthy
followed by a 2.7s shutdown checkpoint in the CI failure this was diagnosed
from. Reproduced on postgres 13.3, 15 and 16. Only bites on a fresh volume,
since a populated one skips the temporary server entirely -- which is why it
surfaced as a rare flake rather than a consistent failure.

Passing -h 127.0.0.1 forces a TCP probe, which tracks the real listener in
every phase: the stock images ship listen_addresses = '*', and during startup
or recovery pg_isready reports "rejecting connections", so the probe
correctly keeps failing until the server can actually serve.

Signed-off-by: slayerjain <shubhamkjain@outlook.com>
…oken

Every one of the 39 lint matrix jobs was failing, on PRs that touch no Go at
all:

  buildir: failed to load package goarch: could not load export data:
  cannot decode "internal/goarch", export data version 4 is greater than
  maximum supported version 2
  golangci-lint exit with code 3

The workflow paired a PINNED linter (v1.63.4) with a FLOATING toolchain
(go-version: 'stable', now go1.27.1). golangci-lint reads the compiler's
export data, and Go 1.27 emits version 4; the x/tools vendored into v1.63.4
reads at most version 2. Nothing in the repo changed -- the last green run
was 2026-08-03, and Go moved underneath it.

Reproduced locally: identical error under go1.27.1, clean under go1.24.4.

The fix is to stop the pair drifting:

  - golangci-lint v1.63.4 -> v2.13.2, whose released binary is built with
    go1.27 and reads export-data v4.
  - golangci-lint-action v3 -> v8. v8 is required for golangci-lint v2 (it
    rejects v1 outright), and its default binary install-mode downloads that
    official release instead of recompiling the linter with whatever Go the
    runner happens to have -- which is what made the old setup fragile.
  - go-version pinned to '1.27' for the lint job only. build.yml deliberately
    stays on 'stable', and both files now say why: building on current Go is
    that job's signal, while a linter must not be outrun by the compiler.
    Every sample's go.mod is <= 1.27, so the lint toolchain builds them all.

.golangci.yml is migrated to the v2 schema. Two v1 settings have no direct v2
spelling and are preserved structurally, so the file now carries comments
explaining both -- they are invisible in the YAML and a future re-run of
`golangci-lint migrate` will quietly undo them:

  - `issues.exclude-use-default: false` survives as the ABSENCE of
    `linters.exclusions.presets`. Adding the presets most v2 templates ship
    would silently switch errcheck back off for ignored Close() errors, which
    is the exact thing the v1 setting existed to prevent.
  - gosimple and typecheck are not missing: S1xxx moved inside staticcheck in
    v2, and compile errors are always surfaced.

Verified 255 -> 0: with the two exclusion rules stripped the suppressed
classes reappear across 13 samples, and with them present every sample is
clean, with no config-load errors in either arm.

v2's staticcheck also runs the ST/QF checks that v1 never did, which surfaces
two real findings, fixed here:

  - go-docker-timefreeze: QF1008, redundant embedded .Time selector.
    jwt.NumericDate embeds time.Time and defines no Before, so the promoted
    method is identical.
  - mux-mysql: ST1005, capitalized error string. It reaches only log.Print;
    the 404 body is a separate literal in controller.go, and no recorded
    keploy artifact contains it.

All 39 matrix samples pass under golangci-lint v2.13.2 + Go 1.27.1, with
GOFLAGS=-mod=readonly as CI runs them.

Signed-off-by: slayerjain <shubhamkjain@outlook.com>
@slayerjain
slayerjain merged commit 2b0a034 into main Sep 4, 2026
41 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant