Skip to content

feat(vm): export driver traces over OTLP #2564

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
krishicks wants to merge 1 commit into
base: main
Choose a base branch
from
Open

feat(vm): export driver traces over OTLP #2564

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
8 changes: 8 additions & 0 deletions Cargo.lock
View file
Open in desktop

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

21 changes: 12 additions & 9 deletions architecture/gateway.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -639,15 +639,18 @@ to export; the SDK's `OTEL_*` variables tune how. Transport is OTLP over gRPC
only. Shared provider, resource, and tracing-layer construction lives in
`openshell-otel`.

Span emission requires no per-handler instrumentation. The `tower_http`
`TraceLayer` in `multiplex.rs` opens a span per inbound request, and that span
continues incoming W3C trace context when present or starts a new trace
otherwise. It is named for the RPC and carries the request ID that also appears
in the gateway's logs — the identifier that lets an operator pivot between a
trace and its log lines. Store and compute-driver spans become children of the
request span. Reconciliation, provider refresh, and driver-watch loops create
their own operation spans because they have no inbound request to provide a
parent. gRPC status is recorded when response trailers arrive.
The `tower_http` `TraceLayer` in `multiplex.rs` opens a span per inbound request,
and that span continues incoming W3C trace context when present or starts a new
trace otherwise. It is named for the RPC and carries the request ID that also
appears in the gateway's logs — the identifier that lets an operator pivot
between a trace and its log lines. Store and compute-driver spans become
children of the request span. Reconciliation, provider refresh, and
driver-watch loops create their own operation spans because they have no
inbound request to provide a parent. gRPC status is recorded when response
trailers arrive.

The gateway forwards OTLP configuration and W3C trace context to managed
external drivers. Each driver exports under its own service name.

Two invariants shape the failure behavior. Telemetry is diagnostic, so no OTLP
failure stops the gateway from serving: a malformed endpoint is logged at
Expand Down
9 changes: 9 additions & 0 deletions crates/openshell-driver-vm/Cargo.toml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -20,12 +20,15 @@ path = "src/main.rs"

[dependencies]
openshell-core = { path = "../openshell-core", default-features = false }
openshell-otel = { path = "../openshell-otel" }
openshell-policy = { path = "../openshell-policy" }
openshell-vfio = { path = "../openshell-vfio" }

bollard = { version = "0.20", features = ["ssh"] }
tokio = { workspace = true }
tonic = { workspace = true, features = ["transport"] }
tower-http = { workspace = true }
http = { workspace = true }
prost = { workspace = true }
prost-types = { workspace = true }
futures = { workspace = true }
Expand All @@ -34,6 +37,9 @@ nix = { workspace = true }
clap = { workspace = true }
tracing = { workspace = true }
tracing-subscriber = { workspace = true }
opentelemetry = { workspace = true }
opentelemetry_sdk = { workspace = true }
tracing-opentelemetry = { workspace = true }
miette = { workspace = true }
url = { workspace = true }
serde = { workspace = true }
Expand All @@ -56,6 +62,9 @@ telemetry = ["openshell-core/telemetry"]

[dev-dependencies]
temp-env = "0.3"
tempfile = "3"
opentelemetry_sdk = { workspace = true, features = ["testing"] }
opentelemetry-proto = { version = "0.32", default-features = false, features = ["gen-tonic", "trace"] }

# smol-rs/polling drives the BSD/macOS parent-death detection in
# procguard via kqueue's EVFILT_PROC / NOTE_EXIT filter. We could use
Expand Down
Loading
Loading