Repository navigation
Commit 57d7008
Metrics Server: report NotReady when metrics service gRPC connection is down (#8226)
* Metrics Server: report NotReady when metrics service gRPC connection is down
The keda-operator-metrics-apiserver (the adapter) kept reporting
Kubernetes-Ready even when its gRPC connection to keda-operator:9666
was down for a long time. Because the pod stayed Ready, the external
metrics APIService kept routing HPA external-metric requests to that
replica, where every request failed.
Add a readiness check named 'metricsservice-grpc-connection' that
reports NotReady whenever the gRPC connection to the metrics service is
not in the Ready state. A replica that cannot establish or re-establish
the connection is then removed from the APIService endpoints, so
requests are routed only to replicas that can serve them.
To support the check without blocking, add a non-blocking
GrpcClient.GetConnectionState() that returns the underlying
connection's connectivity.State, and have makeProvider return the
gRPC client so main can register the check via the apiserver's
AddReadyzChecks.
Fixes #8211
Signed-off-by: avneetbansal-aws <284363899+avneetbansal-aws@users.noreply.github.com>
* Metrics Server: reconnect idle gRPC connection from readiness check
The readiness check calls GrpcClient.GetConnectionState(), which only read the connection state. Once a non-Ready state removed the replica from the APIService endpoints, no RPCs flowed through the client, so nothing would wake a connection that gRPC parks in Idle after its idle timeout (30m by default). The replica could then stay NotReady permanently.
GetConnectionState now triggers a non-blocking Connect() when it observes an Idle connection, mirroring WaitForConnectionReady, so the connection leaves Idle and can recover without any RPC. Adds TestGetConnectionStateReconnectsWhenIdle covering this.
Signed-off-by: avneetbansal-aws <284363899+avneetbansal-aws@users.noreply.github.com>
---------
Signed-off-by: avneetbansal-aws <284363899+avneetbansal-aws@users.noreply.github.com>
Co-authored-by: avneetbansal-aws <284363899+avneetbansal-aws@users.noreply.github.com>
Co-authored-by: Jorge Turrado Ferrero <Jorge_turrado@hotmail.es>1 parent cb1ce09 commit 57d7008
3 files changed
Lines changed: 137 additions & 8 deletions
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
27 | 27 | | |
28 | 28 | | |
29 | 29 | | |
| 30 | + | |
30 | 31 | | |
31 | 32 | | |
| 33 | + | |
32 | 34 | | |
33 | 35 | | |
34 | 36 | | |
| |||
70 | 72 | | |
71 | 73 | | |
72 | 74 | | |
73 | | - | |
| 75 | + | |
74 | 76 | | |
75 | 77 | | |
76 | 78 | | |
77 | | - | |
| 79 | + | |
78 | 80 | | |
79 | 81 | | |
80 | 82 | | |
81 | | - | |
| 83 | + | |
82 | 84 | | |
83 | 85 | | |
84 | 86 | | |
85 | 87 | | |
86 | | - | |
| 88 | + | |
87 | 89 | | |
88 | 90 | | |
89 | 91 | | |
| |||
107 | 109 | | |
108 | 110 | | |
109 | 111 | | |
110 | | - | |
| 112 | + | |
111 | 113 | | |
112 | 114 | | |
113 | 115 | | |
114 | 116 | | |
115 | 117 | | |
116 | 118 | | |
117 | 119 | | |
118 | | - | |
| 120 | + | |
119 | 121 | | |
120 | 122 | | |
121 | 123 | | |
122 | 124 | | |
123 | 125 | | |
124 | 126 | | |
125 | 127 | | |
126 | | - | |
| 128 | + | |
127 | 129 | | |
128 | 130 | | |
129 | 131 | | |
| |||
305 | 307 | | |
306 | 308 | | |
307 | 309 | | |
308 | | - | |
| 310 | + | |
309 | 311 | | |
310 | 312 | | |
311 | 313 | | |
312 | 314 | | |
313 | 315 | | |
314 | 316 | | |
| 317 | + | |
| 318 | + | |
| 319 | + | |
| 320 | + | |
| 321 | + | |
| 322 | + | |
| 323 | + | |
| 324 | + | |
| 325 | + | |
| 326 | + | |
315 | 327 | | |
316 | 328 | | |
317 | 329 | | |
| |||
320 | 332 | | |
321 | 333 | | |
322 | 334 | | |
| 335 | + | |
| 336 | + | |
| 337 | + | |
| 338 | + | |
| 339 | + | |
| 340 | + | |
| 341 | + | |
| 342 | + | |
| 343 | + | |
| 344 | + | |
| 345 | + | |
| 346 | + | |
| 347 | + | |
| 348 | + | |
| 349 | + | |
| 350 | + | |
| 351 | + | |
| 352 | + | |
| 353 | + | |
| 354 | + | |
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
106 | 106 | | |
107 | 107 | | |
108 | 108 | | |
| 109 | + | |
| 110 | + | |
| 111 | + | |
| 112 | + | |
| 113 | + | |
| 114 | + | |
| 115 | + | |
| 116 | + | |
| 117 | + | |
| 118 | + | |
| 119 | + | |
| 120 | + | |
| 121 | + | |
| 122 | + | |
| 123 | + | |
| 124 | + | |
| 125 | + | |
| 126 | + | |
| 127 | + | |
| 128 | + | |
109 | 129 | | |
110 | 130 | | |
111 | 131 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
| 1 | + | |
| 2 | + | |
| 3 | + | |
| 4 | + | |
| 5 | + | |
| 6 | + | |
| 7 | + | |
| 8 | + | |
| 9 | + | |
| 10 | + | |
| 11 | + | |
| 12 | + | |
| 13 | + | |
| 14 | + | |
| 15 | + | |
| 16 | + | |
| 17 | + | |
| 18 | + | |
| 19 | + | |
| 20 | + | |
| 21 | + | |
| 22 | + | |
| 23 | + | |
| 24 | + | |
| 25 | + | |
| 26 | + | |
| 27 | + | |
| 28 | + | |
| 29 | + | |
| 30 | + | |
| 31 | + | |
| 32 | + | |
| 33 | + | |
| 34 | + | |
| 35 | + | |
| 36 | + | |
| 37 | + | |
| 38 | + | |
| 39 | + | |
| 40 | + | |
| 41 | + | |
| 42 | + | |
| 43 | + | |
| 44 | + | |
| 45 | + | |
| 46 | + | |
| 47 | + | |
| 48 | + | |
| 49 | + | |
| 50 | + | |
| 51 | + | |
| 52 | + | |
| 53 | + | |
| 54 | + | |
| 55 | + | |
| 56 | + | |
| 57 | + | |
| 58 | + | |
| 59 | + | |
| 60 | + | |
| 61 | + | |
| 62 | + | |
| 63 | + | |
| 64 | + | |
| 65 | + | |
| 66 | + | |
| 67 | + | |
| 68 | + | |
| 69 | + | |
| 70 | + | |
| 71 | + | |
| 72 | + | |
| 73 | + | |
| 74 | + | |
| 75 | + | |
| 76 | + | |
| 77 | + | |
0 commit comments