Skip to content

feat(server): implement grpc.health.v1.Health/Watch streaming RPC #2569

Description

@bmahabirbu

Problem Statement

Clients that need to observe gateway health changes (e.g., Kaiden, potential Podman Desktop extensions, dashboards, operators) currently have no event-driven mechanism. The only options are:

  • openshell status -o json — polls the unary Health RPC, returns connected/disconnected
  • openshell gateway info -o json — polls GetGatewayInfo, returns healthy/degraded/unhealthy
  • HTTP /readyz — poll-based, and disabled by default for local gateways

All of these require polling. For integrations that need to react to gateway state transitions (healthy → degraded, connected → disconnected), polling adds latency and unnecessary load.

The standard gRPC health checking protocol defines a Watch streaming RPC for exactly this purpose.

Proposed Design

Implement the standard grpc.health.v1.Health/Watch server-streaming RPC using tonic-health.

What's already in place:

  • The /grpc.health. path prefix is already in the unauthenticated bypass list (crates/openshell-server/src/auth/oidc.rs:33), so no auth changes are needed.
  • The ServiceStatus enum (Healthy, Degraded, Unhealthy) already exists in the proto and SDK.
  • The DatabaseHealthMonitor in crates/openshell-server/src/readiness.rs already maintains a tokio::sync::watch channel with health state — this can drive the tonic-health reporter.

Implementation outline:

  1. Add tonic-health dependency to openshell-server.
  2. Register the grpc.health.v1.Health service on the gRPC server alongside the existing OpenShell service.
  3. Wire the DatabaseHealthMonitor's watch channel to the tonic-health reporter so that Watch streams reflect actual readiness state (not just "up = healthy").
  4. Expose the Watch RPC in the SDK client as a streaming health subscription.
  5. Optionally surface it in the CLI as openshell status --watch or openshell status --follow.

Behavior:

  • Watch("") (empty service) returns overall gateway health.
  • Status transitions (ServingNotServing) are pushed to subscribers when the database health monitor detects a change.
  • Existing unary Health and GetGatewayInfo RPCs remain unchanged.

Alternatives Considered

  • Polling openshell status — works but adds latency (seconds) and CPU cost for tight polling intervals. Acceptable as a fallback but not ideal for reactive integrations.
  • WebSocket health endpoint — non-standard, requires a separate HTTP upgrade path, harder for gRPC-native clients to consume.
  • Custom streaming RPC on the OpenShell service — unnecessary when the standard grpc.health.v1 protocol already exists and tooling supports it.

Agent Investigation

  • Confirmed the grpc.health.v1.Health/Watch RPC is not implemented — no tonic-health dependency exists in the project.
  • Confirmed the auth bypass for /grpc.health. is already configured at crates/openshell-server/src/auth/oidc.rs:33 with a test at line 416.
  • Confirmed the DatabaseHealthMonitor (crates/openshell-server/src/readiness.rs) already maintains real-time health state via tokio::sync::watch that could drive the reporter.
  • The current unary Health handler (crates/openshell-server/src/grpc/mod.rs:224-232) unconditionally returns Healthy — this proposal would also improve that by connecting it to actual readiness state.

Metadata

Metadata

Assignees

No one assigned

    Labels

    state:triage-neededOpened without agent diagnostics and needs triage

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions