fix(libkapi): raise Kine gRPC keepalive MinTime to stop GOAWAY(ENHANCE_YOUR_CALM) - #459
Draft
Nicklas Frahm (nicklasfrahm) wants to merge 1 commit into
Draft
Nicklas Frahm (nicklasfrahm) wants to merge 1 commit into
Nicklas Frahm (nicklasfrahm) wants to merge 1 commit into
Conversation
…E_YOUR_CALM) [kommodity#458] startKine now hands endpoint.Listen a pre-built Config.GRPCServer with KeepaliveEnforcementPolicy.MinTime raised from etcd's embed default (5s, sized for a multi-tenant server) to 30s. The apiserver opens one etcd3 client per API group against the same local unix socket, and grpc-go's automatic BDP pings from those connections were tripping the 5s window during storage init, producing a GOAWAY(ENHANCE_YOUR_CALM, "too_many_pings") ERROR log and connection churn. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Go test coverage: 21.8% for commit 0d5cb0f
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Fixes #458.
startKine(pkg/libkapi/storage/kine.go) now builds and passes its own*grpc.Serverviaendpoint.Config.GRPCServer, withKeepaliveEnforcementPolicy.MinTimeraised from etcd'sembed.DefaultGRPCKeepAliveMinTime(5s) to 30s.Why: Kine's default gRPC server uses etcd's 5s
MinTime, sized for a multi-tenant etcd server fielding pings from many independent, potentially untrusted clients. The embedded apiserver opens one etcd3clientv3.Clientper API group, all dialing the same local Kine unix socket. Each connection's grpc-go transport sends automatic BDP (bandwidth-delay-product) pings on its own schedule as it receives stream data (e.g. watch events) — these aren't user-configured keepalives and aren't tunable viastoragebackend.Config. During storage initialization, this ping traffic trips the 5s enforcement window on legitimate, trusted (same-process, local-socket-only) connections, and Kine's server sendsGOAWAY(ENHANCE_YOUR_CALM, "too_many_pings"), logged at ERROR bylibkapi/logging.InstallGRPCLogAdapterand causing repeated connection churn (self-healing via reconnect, but noisy).This is candidate fix (4) from the issue — the lowest-blast-radius, fully self-contained mitigation available at the
libkapilayer. Candidates (1) and (2) (sharing/pooling etcd3 client connections, or disabling client-side BDP estimation) would require changes tok8s.io/apiserver's storage factory or its etcd3 client construction, whichstoragebackend.Configdoes not expose today.Test plan
make lint— 0 issuesmake test— all packages pass, including newTestStartKinePassesTunedGRPCServerassertingstartKinewires a non-nilConfig.GRPCServermake build— builds successfullymake build-image— Docker image builds successfully🤖 Generated with Claude Code