fix(adbc): set connect timeout to bound stalled dials (DI-5183) - #108
Draft
theyostalservice wants to merge 1 commit into
Draft
fix(adbc): set connect timeout to bound stalled dials (DI-5183)#108theyostalservice wants to merge 1 commit into
theyostalservice wants to merge 1 commit into
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Why
DI-5183 documents dbt Semantic Layer clients that stall against Arrow Flight SQL with no way to distinguish "slow" from "actually dead."
BaseADBCClientcurrently sets no RPC timeout at all when buildingdb_kwargsforadbc_connect(there was a literal# TODO: timeouts are not implemented for ADBC), so a hung connection attempt blocks forever.This PR closes the connect-time half of that gap:
adbc_driver_flightsqlsupports fine-grained, per-RPC-type timeout options, and.connectbounds the dial itself.Handshake-vs-dial verification (read this before reviewing the diff)
Before implementing, I verified — against the actual driver source, not assumption — exactly what
adbc.flight.sql.rpc.timeout_seconds.connectbounds, since the customer-reported symptom in DI-5183 specifically names the FlightHandshakeRPC as the one that hangs.Finding:
.connectbounds only the raw gRPC dial (TCP/TLS connection establishment). It does not, and cannot, attach a deadline toHandshake.Traced through
apache/arrow-adbc(go/adbc/driver/flightsql, pinned driver version 1.12.0, confirmed against thego/adbc/v1.12.0tag):timeouts.goconnectParams()returnsgrpc.ConnectParams{MinConnectTimeout: t.connectTimeout}, wired into the client's dial options inflightsql_database.gogetFlightClient()viagrpc.WithConnectParams(...). This is a transport-level construct governing the dial only..timeout_seconds.*options attach a deadline to an RPC istimeouts.gogetTimeout(), called fromunaryTimeoutInterceptor/streamTimeoutInterceptor. Its method-suffix switch matchesDoGet,GetFlightInfo,DoPut,DoAction—Handshakeis not one of them, for any of the four timeout options, including.connect.DatabaseOptions.AUTHORIZATION_HEADER), the driver never issues aHandshakecall in the first place.flightsql_database.gogetFlightClient()callsflightsql.NewClient(target, nil, middleware, dialOpts...)— passingnilfor theClientAuthHandlerparameter. Inapache/arrow-go'sarrow/flight/client.go,Handshake()is only invoked from two places:AuthenticateBasicToken()(username/password auth, which we don't use —Open()returns early once it sees anAuthorizationheader already set) andAuthenticate()(which requires a non-nilauthHandler, which is never set on our path). SoHandshakeisn't just unbounded here — for this client, it isn't even called.Conclusion: the loanDepot Datadog evidence in DI-5183 (
Call completed for method='HANDSHAKE') was for Power BI's ODBC driver (grpc-c++), a different implementation from this Go-based ADBC driver, which evidently does callHandshakethrough some other code path. It's not established that the ADBC-based cases (AwayDay, INC-7884) hang insideHandshakespecifically — this driver, with our auth configuration, may not exercise that RPC at all. The real hang surface for this client is the dial (grpc.NewClient's lazy connection, triggered on the first RPC issued duringOpen()), which.connectdoes bound.Known limitation: if some other, unverified code path in this driver did ever call
Handshake, this fix would not bound it — there is no ADBC-exposed option that attaches a deadline toHandshakespecifically. Given the auth-handler tracing above, I don't believe that applies to this client, but flagging it since I can't prove a negative over the driver's entire surface.What
BaseADBCClient._extra_db_kwargs()now setsadbc.flight.sql.rpc.timeout_seconds.connectto 15 seconds..query/.fetch/.updateare deliberately left unset — untouched from today — since real customer queries can legitimately run 10+ minutes (seesemantic-layer-gateway's ingress chart, which sets a 670sproxy-read-timeoutfor exactly that reason). A short timeout must never apply to query execution.adbc_driver_flightsql.DatabaseOptionsenum (as pinned here) doesn't expose aTIMEOUT_CONNECTmember, even though the underlying Go/C driver has accepted this string key since before this SDK's declared minimumadbc-driver-flightsql>=0.11.0(confirmed: the option was added toarrow-adbcmain on 2024-02-09, andadbc-driver-flightsql0.11.0 wasn't released until 2024-03-31). So the key is set as a raw string constant with a comment explaining why.Handshakecall completes in ~650-810ms fleet-wide — a reasonable proxy for normal network+TLS overhead on this path — so 15s gives roughly 20x headroom, generous enough to absorb jitter while still failing far faster than the unbounded hang it replaces.# TODO: timeouts are not implemented for ADBCcomment in_handle_errorto reflect that connect timeouts are now implemented while query/fetch/update remain intentionally unset..changes/unreleased/Fixes-*.yaml) per this repo's changelog convention.Testing
tests/api/adbc/test_base_client.py::test_connect_timeout_bounds_hung_dial: spins up a raw TCP server that accepts connections but never speaks any protocol on them (simulating exactly the DI-5183 symptom — a connection that appears established but never progresses), then asserts that connecting throughBaseADBCClient._get_connection_context_manager()fails within a bounded window instead of hanging. Red-green confirmed: with the.connectdb_kwarg commented out, this test genuinely hung for the full 10s bailout window (proving it wasn't already bounded some other way); with the fix restored, it passes in ~1s (using a monkeypatched 1s timeout so the test stays fast regardless of the 15s production default).tests/api/adbc/test_base_client.py::test_extra_db_kwargs_sets_only_connect_timeout: asserts.connectis set and.query/.fetch/.updateare absent fromdb_kwargs, guarding against scope creep.pytest --ignore tests/integration/ --server-schema tests/server_schema.gql— 80 passed (78 pre-existing + 2 new), 0 failed.hatch run dev:ruff check/ruff format --check/basedpyright/python -m mypy dbtslall clean on the changed files (lefthook pre-commit hooks passed on commit).SL_HOST/SL_TOKEN/SL_ENV_IDcredentials against a real Semantic Layer account); out of scope for this change.Refs
Drafted by Claude Sonnet 5 under the direction of @theyostalservice.