Qualify compatibility packages on Ubuntu - #36
Conversation
There was a problem hiding this comment.
Review summary
This PR rebalances the LLGo compatibility suite: gonum Limit 45→50 and client-go Limit 45→40, adding 5 gonum test packages (gonum_046–gonum_050) and dropping 5 client-go packages (tools/cache, tools/leaderelection, tools/leaderelection/resourcelock, tools/record, tools/watch) that were excluded for exceeding the runner budget / timing out under LLGo.
The config is well-formed and internally consistent:
- Limits still sum to exactly 200, and the generated manifest holds exactly 200
[[Benchmarks]]entries — matchingEXPECTED_PACKAGESandtest_manifest.py. - The hand edit to the generated
cmd/bent/configs/benchmarks-llgo-compatibility.tomlmatches whatgenerate_manifest.pywould produce: entries are byte-wise sorted within each project,client_gorenumbered contiguously001–040,gonum001–050, no duplicates. - The 5 new client-go excludes exactly correspond to the removed benchmarks; their still-included subpackages (
tools/cache/synctrack,tools/cache/testing,tools/record/util) are correct under the generator's exact-match exclusion. - All 50 gonum entries carry
BuildFlags = ["-tags", "safe"], matchingprojects.toml; no client_go entry does. Versions are consistent (gonumv0.17.0, client-gov0.36.2). test_manifest.py/test_report.pypass;README.md(which only cites the "200" total) needs no update.
One thing worth validating before merge (see inline note): the swap trades I/O/goroutine-heavy client-go packages for CPU/allocation-heavy numerical-linear-algebra packages. lapack/gonum and mat in particular are among gonum's slowest suites and could reintroduce the very LLGo timeout/budget failure the client-go removals were meant to relieve. A pre-merge wall-clock/peak-memory check on the Ubuntu qualification host would de-risk this; a mirrored exclude entry is ready to fall back to if needed.
Also note (non-blocking): the exclusion rationale in the updated comment is empirical (packages timed out / exceeded budget on the qualification host) and can't be verified from the repo alone — worth a sanity check against the CI run logs.
| BuildFlags = ["-tags", "safe"] | ||
|
|
||
| [[Benchmarks]] | ||
| Name = "gonum_046" |
There was a problem hiding this comment.
[P2] Heavy gonum LAPACK/mat suites may reintroduce LLGo timeout risk
The added gonum packages are CPU/allocation-heavy numerical suites. lapack/gonum (gonum_046) dispatches 150+ Test* functions delegating into testlapack helpers that run many randomized matrices per routine (SVD, eigenvalue/Schur, QR/LQ, Dlaqr5, Dbdsqr, Dhseqr, …) — one of gonum's slowest packages under native go test, and materially amplified under LLGo. mat (gonum_048) is likewise a large table-driven conformance suite.
Since these client-go packages were removed for exceeding the runner memory/time budget and timing out under LLGo, this swap risks reintroducing the same failure mode. Recommend measuring gonum_046 (and secondarily gonum_048) wall-clock and peak memory under LLGo on the Ubuntu qualification host before merge, and excluding via projects.toml (as done for the RDF/graph-layout suites) if either exceeds budget.
Summary
client-gopackages with five additional Gonum packagesUbuntu qualification
Validated with Go 1.26.2 and LLGo v1.0.0-pre.3 (
cf91afb98fca979c5195751554b67ec75f8c71d6) on x86_64 Ubuntu.The existing 200-package pass isolated these packages:
k8s.io/client-go/tools/leaderelection: about 30.2 GiB RSSk8s.io/client-go/tools/leaderelection/resourcelock: about 9.8 GiB RSS and over 3 minutesk8s.io/client-go/tools/watch: about 10.0 GiB RSS and over 3 minutesk8s.io/client-go/tools/record: about 5.7 GiB RSS and over 2 minutesk8s.io/client-go/tools/cache: LLGo test reached the 5-minute timeoutThe five replacement Gonum packages then passed the same Bent compatibility flow:
Checks
python3 -m unittest discover -s ci/llgo-compatibility -p 'test_*.py'go test ./cmd/bentgit diff --check