-
Notifications
You must be signed in to change notification settings - Fork 64
docs: add connection parameter reference (Thrift vs SEA/Kernel) #445
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Merged
+141
−0
Merged
Changes from all commits
Commits
Show all changes
3 commits
Select commit
Hold shift + click to select a range
File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,136 @@ | ||
| # Connection parameters reference | ||
|
|
||
| This is a reference for every connection parameter accepted by the driver — as a | ||
| **DSN parameter** (`?param=value`) and/or as a **connector option** | ||
| (`dbsql.WithX(...)`) — and whether each is supported on the **Thrift** (default) | ||
| backend and the **SEA / kernel** backend (`useKernel=true` / `WithUseKernel(true)`). | ||
|
|
||
| It is a companion to the [Connection properties](./README.md#connection-properties) | ||
| and [Choosing a backend](./README.md#choosing-a-backend-thrift-vs-seakernel) | ||
| sections of the README, laid out as one comparison matrix per concern. | ||
|
|
||
| ## Legend | ||
|
|
||
| | Symbol | Meaning | | ||
| |:---:|---| | ||
| | ✅ | Supported and honored. | | ||
| | ❌ | Not supported — **rejected** at connect/execute (wraps `dbsqlerr.ErrNotSupportedByKernel` or `dbsqlerr.ErrRequiresKernelBackend`), never silently ignored. | | ||
| | ⚠️ | Accepted but has no effect ("inert" / silently ignored). | | ||
| | — | Not applicable. | | ||
|
|
||
| **Backend selection.** Both backends are selected once per connection via | ||
| `useKernel` / `WithUseKernel(true)`. If the binary was **not** built with the | ||
| `databricks_kernel` tag, selecting the kernel backend fails at connect | ||
| (wraps `dbsqlerr.ErrKernelNotCompiled`) — it never silently falls back to Thrift. | ||
| See [Building](./README.md#building). | ||
|
|
||
| Any parameter not listed below (e.g. `ansi_mode`) is passed through as a | ||
| **session parameter** on both backends. | ||
|
|
||
| ## Endpoint & routing | ||
|
|
||
| | DSN parameter | Connector option | Thrift | Kernel | Default | Notes | | ||
| |---|---|:---:|:---:|---|---| | ||
| | *(host)* | `WithServerHostname` | ✅ | ✅ | *(required)* | Workspace hostname. | | ||
| | *(path)* | `WithHTTPPath` | ✅ | ✅ | *(required)* | Warehouse/endpoint HTTP path. | | ||
| | *(port)* | `WithPort` | ✅ | ❌ | `443` | Kernel connects on **443 only** and rejects any other port. | | ||
| | `warehouseId` | `WithWarehouseID` | ⚠️ | ✅ | | Bare warehouse id; the kernel routes by it (preferred over the HTTP path). The Thrift backend **silently ignores** it. | | ||
| | `catalog` | `WithInitialNamespace` | ✅ | ✅ | | Initial catalog. Kernel applies it post-connect via `USE CATALOG`. | | ||
| | `schema` | `WithInitialNamespace` | ✅ | ✅ | | Initial schema. Kernel applies it post-connect via `USE SCHEMA`. | | ||
| | `useKernel` | `WithUseKernel` | ✅ | ✅ | `false` | Select the SEA/kernel backend. Requires a `databricks_kernel` build. | | ||
|
|
||
| ## Authentication | ||
|
|
||
| | Method | DSN | Connector option | Thrift | Kernel | | ||
| |---|---|---|:---:|:---:| | ||
| | Personal access token (PAT) | `token:<t>@…`, or `accessToken=` / `authType=Pat` | `WithAccessToken` | ✅ | ✅ | | ||
| | OAuth machine-to-machine (M2M) | `clientID=`+`clientSecret=` / `authType=OauthM2M` | `WithClientCredentials` | ✅ | ✅ | | ||
| | OAuth user-to-machine (U2M) | `authType=OauthU2M` | `WithAuthenticator` (u2m) | ✅ | ✅ | | ||
| | Custom token provider / external / static / federated | — | `WithTokenProvider`, `WithExternalToken`, `WithStaticToken`, `WithFederatedTokenProvider*` | ✅ | ❌ | | ||
|
|
||
| Notes for the SEA/kernel backend: | ||
|
|
||
| - Custom OAuth **M2M scopes** are rejected on the kernel path (the kernel applies its | ||
| own default scopes). Default scopes work on both. | ||
| - **U2M** is interactive: on a cache miss, connecting launches the browser and a | ||
| connect-context **deadline is not honored** during the login window. U2M scopes are at | ||
| parity with Thrift. Use PAT or M2M for headless/deadline-bound connects. | ||
| - OAuth token caching/refresh is owned by the kernel on the kernel path (no driver | ||
| config). | ||
|
|
||
| ## Query execution | ||
|
|
||
| | DSN parameter | Connector option | Thrift | Kernel | Default | Notes | | ||
| |---|---|:---:|:---:|---|---| | ||
| | `maxRows` | `WithMaxRows` | ✅ | ⚠️ | `100000` | Max rows per fetch. On the kernel path the kernel manages paging, so this is accepted but has no effect. | | ||
| | `timeout` | `WithTimeout` | ✅ | ❌ | no timeout | Server-side query timeout, in seconds. On the kernel path use the `STATEMENT_TIMEOUT` session parameter instead. | | ||
| | `userAgentEntry` | `WithUserAgentEntry` | ✅ | ✅ | | Identifies your application (partners/ISVs), format `<isv-name+product-name>`. | | ||
| | *(session param)* | `WithSessionParams` | ✅ | ✅ | | Arbitrary session confs (e.g. `ansi_mode`, `STATEMENT_TIMEOUT`, `QUERY_TAGS`). | | ||
| | *(via session param)* | `WithQueryTags` | ✅ | ✅ | | Session-level query tags (serialized into `QUERY_TAGS`). | | ||
| | `timezone` | `WithSessionParams(timezone=…)` | ✅ | ✅ | | Session time zone (e.g. `America/Los_Angeles`). | | ||
| | `enableMetricViewMetadata` | `WithEnableMetricViewMetadata` | ✅ | ✅ | `false` | Enables metric-view metadata (`spark.sql.thriftserver.metadata.metricview.enabled=true`). | | ||
|
|
||
| ## Retry / backoff | ||
|
|
||
| | Connector option | Thrift | Kernel | Default | Notes | | ||
| |---|:---:|:---:|---|---| | ||
| | `WithRetries(retryMax, waitMin, waitMax)` | ✅ | ✅ | `4`, `1s`, `30s` | Retry attempts and exponential-backoff bounds. `retryMax < 0` disables retries. | | ||
| | `WithKernelRetryOverallTimeout(d)` | ❌ | ✅ | kernel default (900s) | Cumulative retry budget across all attempts. No Thrift equivalent. | | ||
|
|
||
| ## Result rendering | ||
|
|
||
| | DSN parameter | Connector option | Thrift | Kernel | Default | Notes | | ||
| |---|---|:---:|:---:|---|---| | ||
| | `useArrowNativeDecimal` | `WithArrowNativeDecimal` | ✅ | ⚠️ | `false` | Thrift: return DECIMAL as native Arrow `decimal128` (lossless string when scanned via `database/sql`). The kernel path already renders DECIMAL as the exact string regardless, so the flag is inert there. | | ||
| | | `WithKernelDecimalAsFloat(b)` | ❌ | ✅ | `false` | Scan top-level DECIMAL as lossy `float64` instead of the exact string. | | ||
|
|
||
| Otherwise results render **byte-for-byte identically** on both backends (scalars, | ||
| DECIMAL as exact string, TIMESTAMP / TIMESTAMP_NTZ shifted into the session time zone, | ||
| INTERVAL, nested ARRAY / MAP / STRUCT and VARIANT as JSON, GEOMETRY / GEOGRAPHY as WKT, | ||
| BINARY as `sql.RawBytes`). | ||
|
|
||
| ## Cloud Fetch | ||
|
|
||
| | DSN parameter | Connector option | Thrift | Kernel | Default | Notes | | ||
| |---|---|:---:|:---:|---|---| | ||
| | `useCloudFetch` | `WithCloudFetch` | ✅ | ⚠️ | `true` | Enable Cloud Fetch. On the kernel path Cloud Fetch is always managed internally, so the flag is inert. | | ||
| | `maxDownloadThreads` | `WithMaxDownloadThreads` | ✅ | ⚠️ | `10` | Concurrent download goroutines (Thrift). Inert on the kernel path. | | ||
| | | `WithKernelMaxChunksInMemory(n)` | ❌ | ✅ | kernel default (16) | Bounds decompressed Cloud Fetch chunks held in memory — trades large-result throughput for peak memory. | | ||
|
|
||
| ## TLS | ||
|
|
||
| | Connector option | Thrift | Kernel | Notes | | ||
| |---|:---:|:---:|---| | ||
| | `WithSkipTLSHostVerify()` | ✅ | ✅ | Disable TLS chain + hostname verification. **Use only for internal private-link hostnames** — susceptible to machine-in-the-middle attacks. | | ||
| | `WithTransport(http.RoundTripper)` | ✅ | ❌ | Supply a custom HTTP transport (custom CA, mTLS, or proxy). Rejected on the kernel path — the kernel uses its own HTTP stack; use `WithKernelTrustedCerts` / `WithKernelProxy` there. | | ||
| | `WithKernelTrustedCerts(pem)` | ❌ | ✅ | Add a PEM CA bundle on top of the system roots (for a re-signing proxy / on-prem CA). Needed because the kernel's TLS stack does not read `SSL_CERT_FILE`. | | ||
| | `WithKernelSkipHostnameVerify()` | ❌ | ✅ | Skip **only** the hostname check while keeping chain validation (finer-grained than `WithSkipTLSHostVerify`). | | ||
|
|
||
| ## Proxy | ||
|
|
||
| Both backends honor the standard `HTTP_PROXY` / `HTTPS_PROXY` / `NO_PROXY` environment | ||
| variables. | ||
|
|
||
| | Connector option | Thrift | Kernel | Notes | | ||
| |---|:---:|:---:|---| | ||
| | *(environment)* | ✅ | ✅ | `HTTP(S)_PROXY` / `NO_PROXY`. | | ||
| | `WithKernelProxy(KernelProxy{URL, Username, Password, BypassHosts})` | ❌ | ✅ | Explicit proxy with out-of-band basic-auth credentials and a structured bypass list — the "advanced" fields the env-var path can't express. Overrides the environment proxy; a malformed URL is rejected at connect. | | ||
|
|
||
| ## Telemetry | ||
|
|
||
| Telemetry applies to **both** backends. When `enableTelemetry` is left unset (the | ||
| default), a server-side feature flag decides whether telemetry is active; setting it | ||
| explicitly overrides the flag. (Exception: on the kernel backend with OAuth **U2M**, | ||
| telemetry is skipped entirely to avoid a second interactive browser flow at connect.) | ||
|
|
||
| | DSN parameter | Thrift | Kernel | Default | Notes | | ||
| |---|:---:|:---:|---|---| | ||
| | `enableTelemetry` | ✅ | ✅ | unset (server flag decides) | Force telemetry on/off, overriding the server feature flag. | | ||
| | `telemetry_batch_size` | ✅ | ✅ | `200` | Events per batch. | | ||
| | `telemetry_flush_interval` | ✅ | ✅ | `30s` | Flush interval. | | ||
| | `telemetry_retry_count` | ⚠️ | ⚠️ | — | **Deprecated and ignored** (retries are owned by the HTTP client + circuit breaker); logs a one-time warning. | | ||
| | `telemetry_retry_delay` | ⚠️ | ⚠️ | — | **Deprecated and ignored** (see above). | | ||
|
|
||
| The kernel path additionally emits a connection-config telemetry event at connect (mode, | ||
| auth mechanism/flow, proxy, arrow, query tags, metric-view); the Thrift path's telemetry | ||
| is unchanged. See [`telemetry/DESIGN.md`](./telemetry/DESIGN.md). | ||
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Uh oh!
There was an error while loading. Please reload this page.