ci: shard race UT critical path - #438
Merged
Merged
Conversation
XuPeng-SH
force-pushed
the
codex/shard-ut-critical-path
branch
from
August 28, 2026 15:11
5c88dac to
97fb41e
Compare
XuPeng-SH
added a commit
to matrixorigin/matrixone
that referenced
this pull request
Aug 28, 2026
## What changed ### Reduce work and contention inside each runner - change light-package parallelism from the measured baseline `p=8` to `p=7`, leaving one runner CPU for race-detector and native work inside active packages - move `pkg/logservice` and the two TAE logstore packages out of the obsolete exclusive group: their `NewTestService` users now allocate independent ports with collision retry - keep `pkg/tests/issues` on an exclusive runner because its shared base cluster lives for most of the test process - move `pkg/tests/issues/isolated` into the embedded group: it has no shared base cluster, while runner-wide lifecycle admission still prevents complete clusters from overlapping - run the HNSW package after the normal light wave with exclusive runner CPU - remove non-orthogonal repetition from the slow HNSW continuous-lifecycle case without removing its distinct test targets: - the adjacent one-shot SmallCap case still executes the complete 400-row randomized mixed batch - the continuous case still executes all ten update cycles - 13 boundary rows cover one update in each existing file, 11 inserts, capacity rollover into two models, and all eight build-worker lanes - insert/update classification, model count, per-model lengths, and every repeated-cycle path are now asserted - restore package-global mocks on every terminal path, isolate shuffle state from the package-global PRNG, log nondeterministic seeds, and keep fallocate output inside `t.TempDir()` ### Split and protect the overall critical path - add an opt-in `UT_SHARD` contract with four runner groups: - `light`: 457 normal race packages followed by the isolated HNSW package - `issues`: the shared-base issues package - `embedded`: 10 dependency-discovered embedded-cluster packages, including `issues/isolated` - `heavy-plan`: four resource-heavy packages, engine shards, and plan shards - make the shard-to-stage map a single explicit source of truth; unknown shard/stage values fail closed - latch routing-helper errors into the final test status, so a future stage typo cannot be mistaken for an intentionally disabled stage - validate at runtime that both the stage map and all 475 packages form complete disjoint partitions; missing, duplicate, unexpected, or duplicated-authority coverage fails before tests start - keep `UT_SHARD=all` as the default for direct developers and existing CI callers - enable CI#438's merged opt-in matrix through `matrixorigin/CI@main` matrixorigin/CI#438 is merged as `20eb9630708696a361b7730b4c14395a8ef9acc2`; #27815 now consumes the merged contract through `matrixorigin/CI@main`. Because MatrixOne uses `pull_request_target`, #27815's workflow-call edit still does not affect its own PR checks; the first real four-shard run starts after this entrypoint change reaches the base branch. ## Baseline and expected impact Exact baseline: run #33175689678, job #98863834673, 58m59s in Unit Testing with light `p=8`: - light: 17m52s; the HNSW continuous case alone was 290.94s - exclusive: 19m43s (`issues`: 9m58s; `issues/isolated`: 3m23s) - embedded: 8m31s - resource-heavy/engine: 8m55s; the 16-GiB cgroup peak occurred here, not in the light stage - plan and setup: remaining critical-path time The balanced four-runner paths are expected to be approximately: - `light`: 14-17 minutes - `issues`: 10-12 minutes - `embedded`: 11-13 minutes - `heavy-plan`: 11-13 minutes Including repeated setup and the summary gate, expected wall time is roughly 16-20 minutes instead of 59 minutes (about 66-73% lower). A fifth runner does not split another material critical path and would add 25% runner capacity for little wall-time benefit. ## Test-quality contract No test package, topology, update cycle, race instrumentation, failure result, timeout, or distinct behavioral target is removed. The HNSW change removes repeated rows whose batch-scale target remains in the adjacent 400-row test; the repeated-lifecycle, mixed update/insert, rollover, randomized-order, and worker-distribution targets remain with stronger oracles. Exact package partition at this head: - total: 475 - light: 457 - isolated HNSW: 1 - exclusive issues: 1 - embedded cluster: 10 - resource heavy: 4 - engine: 1 - plan: 1 - duplicate/missing/extra: 0/0/0 Every shard keeps `-short -race`; CI#438 uses `fail-fast: false` and the established required check succeeds only if all four shard jobs succeed. ## Validation - runtime stage partition and package partition: 475/475, zero duplicate/missing/extra - behavioral head `d67c769a95`: `./optools` passes, including positive/negative routing and expected/actual partition-duplication tests; reused at delivery head `8602561e89` because the final commit only replaces the immutable CI#438 pin with its squash-merged, content-equivalent `@main` target - optimized HNSW continuous regression under `-race -count=10`: pass, 2.015s locally versus 14.316s with the repeated 400-row fixture - exact head full `pkg/vectorindex/hnsw` under `-race`: pass in 7.437s, including the complete randomized 400-row one-shot case - `pkg/tests/ddl` and `pkg/tests/issues/isolated` together under `-race -p 2`: pass (`16.151s` and `119.525s`), proving the isolated package is safe under embedded lifecycle admission - `pkg/logservice` and both TAE logstore packages together under `-race -p 3`: pass; embedded dependency discovery excludes all three - MatrixOne and CI workflows: actionlint pass - `bash -n`, Go formatting, and `git diff --check`: pass --------- Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What changed
ut_shardedinput to the reusable MatrixOne CI workflowlightissuesembeddedheavy-planfail-fast: false, so one failure cannot cancel and hide unexecuted coverage in another shardUT Test on Ubuntu/x86as a fail-closed AND gate over every shardhead.sha, so queued matrix children cannot test different revisions after the branch movesBackward compatibility
ut_shardeddefaults tofalse. In that mode:alljobUT Test on Ubuntu/x86nameUT shard summary (disabled)namevars.UT_PARALLEL || inputs.ut_parallelprecedence remain unchangedCurrent
CI@maincallers were inventoried:matrixorigin/matrixone,matrixorigin/matrixone-inhouse, andmatrixorigin/m4. All remain on the default unsharded path until they explicitly opt in.Coverage and failure contract
The companion MatrixOne runner owns package discovery and validates at runtime that:
all packages = light + HNSW + exclusive issues + embedded cluster + resource heavy + engine + planBoth the stage map and package partition must be complete and disjoint. Every shard keeps
-short -race. The summary runs after failures viaalways()and succeeds only when the aggregate matrix result is exactlysuccess; failure, cancellation, skip, or timeout cannot produce a green required check. The summary performs one bounded shell comparison and no checkout or external operation. Every matrix child checks out the same triggering commit, so the aggregate result cannot combine tests from different PR revisions.Expected impact when enabled
Using the #27812 Ubuntu baseline plus the companion MatrixOne optimizations:
light: approximately 14-17 minutesissues: approximately 10-12 minutesembedded: approximately 11-13 minutesheavy-plan: approximately 11-13 minutesIncluding per-runner setup, expected required-check wall time remains roughly 16-20 minutes instead of 59 minutes (about 66-73% lower). This trades additional parallel runner-minutes for PR latency without increasing concurrency above four runners.
Validation
d303928bb44b24e56ee594f812b582e43be8c854: actionlint and workflow-schema checks passgit diff --check: passalljob, retains the original check name, and skips the summary without allocating a runnermax-parallel: 4needs+always()semantics ensure the summary runs after failed prerequisites; the step accepts only resultsuccessCompanion: matrixorigin/matrixone#27815