fix(postgres): probe over TCP so dependents aren't released early - #241
Merged
Conversation
…arly postgres' docker-entrypoint runs a temporary server reachable only over the Unix socket (listen_addresses='') once initdb has finished, so it can create the database and run docker-entrypoint-initdb.d. A pg_isready probe without -h talks to that socket, so it answers "accepting connections" while no TCP listener exists at all. compose then marks the service healthy and depends_on: service_healthy releases the dependent straight into ECONNREFUSED. Measured at ~235ms with no initdb.d scripts, and at 1.2s of false-healthy followed by a 2.7s shutdown checkpoint in the CI failure this was diagnosed from. Reproduced on postgres 13.3, 15 and 16. Only bites on a fresh volume, since a populated one skips the temporary server entirely -- which is why it surfaced as a rare flake rather than a consistent failure. Passing -h 127.0.0.1 forces a TCP probe, which tracks the real listener in every phase: the stock images ship listen_addresses = '*', and during startup or recovery pg_isready reports "rejecting connections", so the probe correctly keeps failing until the server can actually serve. Signed-off-by: slayerjain <shubhamkjain@outlook.com>
…oken
Every one of the 39 lint matrix jobs was failing, on PRs that touch no Go at
all:
buildir: failed to load package goarch: could not load export data:
cannot decode "internal/goarch", export data version 4 is greater than
maximum supported version 2
golangci-lint exit with code 3
The workflow paired a PINNED linter (v1.63.4) with a FLOATING toolchain
(go-version: 'stable', now go1.27.1). golangci-lint reads the compiler's
export data, and Go 1.27 emits version 4; the x/tools vendored into v1.63.4
reads at most version 2. Nothing in the repo changed -- the last green run
was 2026-08-03, and Go moved underneath it.
Reproduced locally: identical error under go1.27.1, clean under go1.24.4.
The fix is to stop the pair drifting:
- golangci-lint v1.63.4 -> v2.13.2, whose released binary is built with
go1.27 and reads export-data v4.
- golangci-lint-action v3 -> v8. v8 is required for golangci-lint v2 (it
rejects v1 outright), and its default binary install-mode downloads that
official release instead of recompiling the linter with whatever Go the
runner happens to have -- which is what made the old setup fragile.
- go-version pinned to '1.27' for the lint job only. build.yml deliberately
stays on 'stable', and both files now say why: building on current Go is
that job's signal, while a linter must not be outrun by the compiler.
Every sample's go.mod is <= 1.27, so the lint toolchain builds them all.
.golangci.yml is migrated to the v2 schema. Two v1 settings have no direct v2
spelling and are preserved structurally, so the file now carries comments
explaining both -- they are invisible in the YAML and a future re-run of
`golangci-lint migrate` will quietly undo them:
- `issues.exclude-use-default: false` survives as the ABSENCE of
`linters.exclusions.presets`. Adding the presets most v2 templates ship
would silently switch errcheck back off for ignored Close() errors, which
is the exact thing the v1 setting existed to prevent.
- gosimple and typecheck are not missing: S1xxx moved inside staticcheck in
v2, and compile errors are always surfaced.
Verified 255 -> 0: with the two exclusion rules stripped the suppressed
classes reappear across 13 samples, and with them present every sample is
clean, with no config-load errors in either arm.
v2's staticcheck also runs the ST/QF checks that v1 never did, which surfaces
two real findings, fixed here:
- go-docker-timefreeze: QF1008, redundant embedded .Time selector.
jwt.NumericDate embeds time.Time and defines no Before, so the promoted
method is identical.
- mux-mysql: ST1005, capitalized error string. It reaches only log.Print;
the 404 body is a separate literal in controller.go, and no recorded
keploy artifact contains it.
All 39 matrix samples pass under golangci-lint v2.13.2 + Go 1.27.1, with
GOFLAGS=-mod=readonly as CI runs them.
Signed-off-by: slayerjain <shubhamkjain@outlook.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The bug
postgres'docker-entrypointruns a temporary server reachable only over the Unix socket (listen_addresses='') onceinitdbhas finished, so it can create the database and rundocker-entrypoint-initdb.d.A
pg_isreadyhealthcheck without-htalks to that socket. So it reports "accepting connections" while no TCP listener exists at all — compose marks the service healthy,depends_on: condition: service_healthyreleases the dependent, and the dependent connects straight intoECONNREFUSED.It only bites on a fresh volume, because a populated one skips
initdband the temporary server entirely. That is why it surfaces as a rare flake rather than a consistent failure.Evidence
From the CI failure this was diagnosed from (enterprise
umami-linux, a matrix cell that creates its volume):08:54:29.422— false-healthy begins08:54:30.668— window 1.25s08:54:30.780→08:54:33.421(2.7s)08:54:33.583compose logged
Container umami_db Waiting→Container umami_app Started, then the app died withCan't reach database server at 172.78.0.10:5432, while the db showedUp 11 seconds (healthy).Verification
Reproduced and mutation-tested locally with a compose mirroring the
depends_on: service_healthyshape:pg_isready -U ... -d ...APP_FAILED_TO_REACH_DB, exit 1pg_isready -h 127.0.0.1 -U ... -d ...APP_OK, exit 0Directly measured the false-healthy window against the patched file: the old check reports healthy from t=2s while TCP is refused until t=8s; the patched check holds
startingand only goeshealthyonce TCP is genuinely up. Reproduced on postgres 13.3, 15 and 16.Why
-h 127.0.0.1The stock images ship
listen_addresses = '*', so the real server binds0.0.0.0— onebind()covers loopback and the container IP, and the staticipv4_addressis configured before PID 1 runs, so there is no phase where loopback works and the network IP does not. During startup/recoverypg_isreadyreports rejecting connections, so the probe correctly keeps failing until the server can actually serve.localhostwould be the wrong literal (it may resolve::1first).Note on
pg_isreadysemanticspg_isreadyis a liveness probe: it exits 0 even for a nonexistent role/database, and even whenpg_hba.confrejects the connection. The-U/-dflags here are effectively decorative. That is fine for this fix — the real listener only starts afterinitdbcreated the role and database — but it does mean the probe is not a substitute for an application-level readiness check.