Skip to content

perf(partition): reduce consecutive Top-1 candidates before copying - #31001

Open
LeftHandCold wants to merge 2 commits into
matrixorigin:mainfrom
LeftHandCold:codex/partition-topone-run-reduction-20261024
Open

LeftHandCold wants to merge 2 commits into
matrixorigin:mainfrom
LeftHandCold:codex/partition-topone-run-reduction-20261024

Conversation

@LeftHandCold

@LeftHandCold LeftHandCold commented Oct 23, 2026 •

Copy link
Copy Markdown
Collaborator

What type of PR is this?

  • API-change
  • BUG
  • Improvement
  • Documentation
  • Feature
  • Test and CI
  • Code Refactoring

Which issue(s) this PR fixes:

Fixes #31002.

What this PR does / why we need it:

For ROW_NUMBER() ... rn = 1, an ordered group can repeatedly replace the retained row for every incoming row. On the original TSBS CPU-only latest-point query (8.64 million rows, 1,000 hosts), replaceRow accounted for 89% of sampled CPU, mostly copying payload vectors and metadata.

For Top-1 without ties, select the first best row in each consecutive same-group run within the existing hash chunk, then use the existing retain/replace path once. This needs no sorted-input assumption and adds no allocation or retained state. Existing SQL comparators preserve NULL, composite order and first-peer behavior; K > 1 and WITH TIES keep their existing path. Payload ownership, metadata copying, hash admission, cancellation and cleanup remain unchanged.

Validation:

  • Independent stable-sort UTs cover grouped/interleaved/distinct inputs, hash and batch boundaries, NULLs, composite keys, first peers, metadata, reset and two-stage execution. A reversed-comparison mutation is rejected.
  • Public six-row SQL case verifies latest NULL payload, deterministic secondary ties and missing-tag exclusion; normal mo-tester comparison passed twice, 10/10 statements each, with teardown checks.
  • Real-data A/B/B/A under the same 4-CPU, 12-GiB, no-swap limit: 10 measured samples per version/query, complete native outputs checked for all five original CPU-only queries. Q4 median wall time 1.272 s → 0.115 s; whole-unit CPU 4.870 s → 0.409 s. Other query median wall changes were between -1.1% and +1.9%; whole-block memory peaks were 1.49–1.52 GiB with zero OOM/limit events.
  • Paired operator microbenchmarks also cover interleaved groups, all-distinct groups and K=2 controls. Query profiles were collected separately from latency measurements.

Performance A/B uses the same frozen main base b78794175e65f8cf5a4a4ac1fcd2715534f48552 plus this patch. The delivery branch is rebased on 156b7cc6f8; partition, comparator, vector, types and hashmap dependencies are unchanged between those bases. On the final clean delivery head, the full owning partition package, gofmt/diff checks, go vet, configured incremental golangci-lint, and native artifact provenance checks all pass.

This is a draft for review. The measured gain applies to the stated CPU-only query and layout; DevOps/IoT and the broader AgentLogs/JSON comparison remain separate work.

@qodo-code-review

Copy link
Copy Markdown

Qodo reviews are paused for this user.

Troubleshooting steps vary by plan Learn more →

On a Teams plan?
Reviews resume once this user has a paid seat and their Git account is linked in Qodo.
Link Git account →

Using GitHub Enterprise Server, GitLab Self-Managed, or Bitbucket Data Center?
These require an Enterprise plan - Contact us
Contact us →

@LeftHandCold LeftHandCold left a comment

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

代码复核:建议通过,未发现需要 Request Changes 的确定性问题。审查 HEAD:6fa077ce1fa51ab2c1dab4d5773ee75163e80ff4。

设计判断:这是在原 Top-1 执行器内减少无效行复制的局部优化,复用原比较器、hash admission 和 retain/replace 所有权;没有新增缓存或第二套聚合状态。

逐项核对:

  • 仅 limit=1 且非 WITH TIES 生效;连续同组片段局限在原来的 256 行 hash chunk,不要求输入排序,也不跨 batch 保存借用行。
  • 同片段使用现有 ORDER 比较器,只在严格更优时更新,保留第一条相同排序键记录;核对 NULL、升降序、复合键及浮点 NaN/±0 的比较路径。
  • 所有输入仍参与分组 hash;每组最终赢家仍交给原 retain/replace,payload/metadata 的复制与释放责任没有变化。K>1、WITH TIES 保持原分支。
  • 阅读了独立稳定排序 oracle、chunk/batch 边界、交错组、多阶段、latest NULL payload、元数据和复用测试。对全不同组的额外工作是有界相邻 group-id 判断;没有发现新的分配或累计状态。

验证边界:已核对当前 HEAD 的成功 CI 40018738556,并完成 diff --check。本次没有重跑本机全套 UT 或 TSBS。PR 的 1.272s→0.115s 属于作者给定布局的实测,不能外推为任意窗口查询的收益。

当前认证账号也是 PR 作者,故以 COMMENT 记录建议通过;这不是 GitHub 正式 Approval。

This branch was successfully deployed

1 active deployment
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

do-not-merge/wip kind/enhancement size/L Denotes a PR that changes [500,999] lines

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Performance]: partition Top-1 repeatedly copies payload rows within one group run

3 participants