Skip to content

fix: disable leader election for single replica deployment #2767

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
tginer wants to merge 1 commit into
base: main
Choose a base branch
from
+3 −1
Open

fix: disable leader election for single replica deployment #2767

Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 3 additions & 1 deletion deployments/gpu-operator/templates/operator.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -8,7 +8,7 @@ metadata:
app.kubernetes.io/component: "gpu-operator"
nvidia.com/gpu-driver-upgrade-drain.skip: "true"
spec:
replicas: 1
replicas: {{ .Values.operator.replicas | default 1 }}

@rahulait rahulait Aug 20, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This also needs to be added to values.yaml (maybe commented) so that users can find this.

selector:
matchLabels:
app.kubernetes.io/component: "gpu-operator"
Expand Down Expand Up @@ -39,7 +39,9 @@ spec:
imagePullPolicy: {{ .Values.operator.imagePullPolicy }}
command: ["gpu-operator"]
args:
{{- if gt (int (.Values.operator.replicas | default 1)) 1 }}
- --leader-elect
{{- end }}

@rahulait rahulait Aug 20, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What happens when two gpu-operator controllers come up at same time during upgrade (old controller and new controller)?

@tginer tginer Aug 21, 2026 •
edited
Loading

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

In my opinion leader election is critical under multiple replicas deployed. With a single replica and a very brief overlap period, acting without a leader seems fairly safe.

The tests I performed resulted in:

  • The overlap window was between 10 - 15 seconds
  • Kubernetes reconciliation is idempotent
  • Kubernetes resourceVersion would prevent conflicts as pod B trying to update old version would result in a re-try and read new resourceVersion

Basically the two pods were briefly trying to do the same idempotent work and the old pod was terminated in a 10-15 second-window.

I replicated the idea implemented by NVIDIA in Mellanox/network-operator#2304 but the network operator AI agent bot also raised your concern in my identical PR in the NNO, see Mellanox/network-operator#3056

@rajathagasthya rajathagasthya Aug 21, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Concurrent reconciles by different versions of the operator are almost never a good thing, even if they are idempotent in theory. It can be a source of unexpected bugs and we are sensitive to unnecessary daemonset rollouts (especially triggering unnecessary driver upgrades). So this change does not look safe to me.

Instead of disabling leader election, how about setting a larger --leader-lease-renew-deadline=60s value? That will make the operator tolerate any API outages up to 60s while still holding the lease. That way the operator doesn't enter a crash loop if it can't renew the lease in 10s (controller-runtime default) and doesn't delay ClusterPolicy processing due to exponential backoff. The OLM bundle already does this:

gpu-operator/bundle/manifests/gpu-operator-certified.clusterserviceversion.yaml

Lines 1081 to 1082 in 10ee5b3

- --leader-lease-renew-deadline
- "60s"

{{- if .Values.operator.logging.develMode }}
- --zap-devel
{{- else }}
Expand Down
Loading