test(godeltaprof): repro for the mutex/block LocsForStack panic (#245)

[korniltsev-grafanista]

Repro only, no fix yet. Both tests in the new package fail on purpose -- see Diagnosis and Proposed fix.

Closes-when-fixed: #245

What #245 actually is

The issue attributes the panic to golang/go#70529 (a rare SIGPROF misclassification window). It reproduces without any of that, and it is our bug: godeltaprof skips the stack normalization step that runtime/pprof performs for block and mutex records.

Since Go 1.23 runtime.saveblockevent records contention stacks in one of two shapes:

capture path	shape	when
fpTracebackPartialExpand	physical: one PC per physical frame, inlined callers not expanded	frame pointer unwinding, the common case
callers	logical: one PC per logical frame, including the virtual PCs the compiler emits for inlined calls	GODEBUG=tracefpunwindoff=1, or m.hasCgoOnStack() -- i.e. the event was sampled while cgo was on the M's stack

pprof_mutexProfileInternal hands out the raw bucket stack, so the consumer sees both shapes. runtime/pprof.printCountCycleProfile normalizes each record with expandInlinedFrames (runtime.CallersFrames over the whole stack) before appendLocsForStack. godeltaprof/internal/pprof.DeltaMutexProfiler.PrintCountCycleProfile passes r.Stack straight to LocsForStack, which assumes logical PCs.

Diagnosis

Two consequences, one end to end test each.

1. Inlined frames are silently dropped from every mutex and block profile. No cgo involved, affects everyone. allFrames expands a physical PC with CallersFrames one PC at a time, and runtime.Frames.Next only inserts the virtual PCs of an inlined call when it can peek at the next PC of the stack (runtime/symtab.go, "So check to see if the implied virtual PC for this PC is the next PC in ci.callers"). Same contention, same process:

godeltaprof:    Unlock;reproLeaf;reproL1;.................................;contendReproMu.func1
runtime/pprof:  Unlock;reproLeaf;reproL1;L2;L3;L4;L5;reproEntry;contendReproMu.func1

2. A dump that mixes both shapes panics, which is #245. A location cached from a logical record records len(pcs) == 6 for a frame that a physical record spells as a single PC, so stk = stk[len(l.pcs):] at proto.go:375 runs off the end:

godeltaprof panicked building the mutex profile: runtime error: slice bounds out of range [6:3]

On a CGO_ENABLED=1 service with Go callbacks (the reporter runs v8go) some contention events are sampled with cgo on the stack and some are not, so every dump mixes the two shapes and every dump is lost -- matching the reported "100% empty server-side, ~24 recovered panics/day". pyroscope-go recovers in Session.dumpMutexProfile, which is why it only shows up as a log line.

The tests

New package godeltaprof/compat/inlineexpansion, kept separate because the runtime never clears mutex buckets: once a poisoned record exists, every later dump in the process panics, which would take the rest of the compat suite down with it. dumpMutexProfile recovers and calls t.Errorf, so each failure is reported without killing the binary.

Both tests are end to end: real contention, real runtime records, no hand-written PC lists.

test	needs cgo	what it does
TestMutexProfileDropsInlinedFrames	no	contends reproMu through a chain the compiler inlines, asserts the inlined frames survive the dump (bug 1)
TestMutexProfileCgoMixedTraceback	yes	contends the same mutex from a Go callback invoked from C and from plain Go goroutines, so the runtime itself produces the mix within one dump, asserts no panic (bug 2)

$ go test -race -run TestMutexProfile ./inlineexpansion/
--- FAIL: TestMutexProfileDropsInlinedFrames (0.20s)
        frame ...inlineexpansion.reproL2 missing from [...contendReproMu.func1 ...reproL1 ...reproLeaf sync.(*Mutex).Unlock]
        frame ...inlineexpansion.reproEntry missing from [...]
--- FAIL: TestMutexProfileCgoMixedTraceback (0.31s)
        godeltaprof panicked building the mutex profile: runtime error: slice bounds out of range [6:3]

Verified on go1.26.3 darwin/arm64, with and without -race, and with CGO_ENABLED=0 (the cgo test drops out, the other one still fails).

Proposed fix

Do what runtime/pprof does: normalize each block/mutex record before it reaches the builder, i.e. runtime.CallersFrames(stk) collecting f.PC+1, into a buffer reused across records. Open questions worth a maintainer decision, which is why this PR is repro only:

• expand before or after d.m.Lookup. Before means the physical and logical spellings of one stack finally dedupe into a single series, and the delta map is keyed by normalized stacks.
• heap profiles are unaffected (mProf_Malloc uses callers, always logical), so the change belongs in DeltaMutexProfiler.PrintCountCycleProfile only.
• worth adding the stdlib's named guard in LocsForStack as well (panic("stack too short to match cached location"), runtime/pprof/proto.go), so a future drift fails loudly instead of as a slice bounds error. The check_golang_profiler_changes note in Mutex/block delta profiles are silently lost: LocsForStack slice-bounds panic (golang/go#70529), recovered and discarded by pyroscope-go #245 stands: the stdlib gained both expandInlinedFrames and that guard, the vendored copy got neither.

Happy to push the fix onto this branch.

🤖 Generated with Claude Code