docs(rfc): define analytic optimizer rewrite contracts

[aptend]

What type of PR is this?

• API-change
• BUG
• Improvement
• Documentation
• Feature
• Test and CI
• Code Refactoring

Which issue(s) this PR fixes:

issue #26768

What this PR does / why we need it:

Defines the versioned design contract for the analytic optimizer rewrite series in #27914 and #27915 before implementation review continues.

The RFC fixes rule order and non-fixpoint behavior; NULL, correlation, volatility, lineage, uniqueness, evaluation-domain, and fail-closed invariants; resource, compile-time, and plan-quality budgets; MORPC v42 compatibility; per-rule positive and counterexample validation; and rollback boundaries.

This is a design-only PR. It does not change runtime behavior.

[qodo-code-review]

Code Review by Qodo

🐞 Bugs (0) 📘 Rule violations (0) 📎 Requirement gaps (0)

Great, no issues found!

Qodo reviewed your code and found no material issues that require review

Tip of the day

💡 Did you know, you can describe a rule in plain language on the Rules page and Qodo drafts it for you

More tips ↗ | Customize Qodo ↗ | Qodo docs ↗

[qodo-code-review]

PR Summary by Qodo

Define contracts for analytic optimizer rewrites

📝 Documentation 🕐 40+ Minutes

AI Description

• Defines semantic and fail-closed contracts for stats-independent analytic optimizer rewrites.
• Fixes rewrite ordering, resource budgets, protocol compatibility, validation, and rollback
 boundaries.
• Gates implementation approval on review of the versioned RFC.

Diagram

graph TD
  A["SQL Binding"] --> B["Bind Rewrites"] --> C["Logical Rewrites"] --> D["Costing and Joins"] --> E["Partial Aggregation"] --> F["Physical Distribution"] --> G["Final Plan"]

Loading

High-Level Assessment

The following are alternative approaches to this PR:

1. General memo optimizer

• ➕ Enables global exploration and costing across interacting rewrites.
• ➕ Avoids relying on a permanently fixed sequence of local passes.
• ➖ Requires a substantially larger architectural change before current fixes can land.
• ➖ Still needs the same semantic equivalence and resource contracts defined here.

2. Runtime adaptive optimization

• ➕ Can choose materialization and join strategies from observed execution behavior.
• ➕ May tolerate inaccurate compile-time cardinality estimates better.
• ➖ Requires new observation, topology-switching, ownership, and rollback protocols.
• ➖ Expands runtime complexity and failure modes beyond this rewrite series.

3. Query-specific optimization

• ➕ Could target known TPC-DS regressions with smaller immediate changes.
• ➕ May deliver benchmark improvements quickly.
• ➖ Does not generalize to equivalent analytic SQL shapes.
• ➖ Encodes benchmark identity instead of proving semantic legality.

Recommendation: Adopt the RFC's contract-first, ordered local-rewrite approach for this implementation series. It bounds review and rollback per semantic rule while preserving fail-closed behavior; memo-based or adaptive optimization can be pursued later using these equivalence contracts, while query-specific branching should remain rejected.

Files changed (1) +337 / -0

Documentation (1) +337 / -0

20260901_stats_independent_analytic_plan_rewrites.mdDefine analytic optimizer rewrite contracts +337/-0

Define analytic optimizer rewrite contracts

• Adds a versioned RFC governing legality, deterministic ordering, resources, compatibility, and failure behavior for stats-independent analytic plan rewrites. It specifies per-rule semantic guards, compile-time and plan-quality budgets, MORPC v42 fallback behavior, validation controls, approval gating, and independent rollback boundaries.

docs/rfcs/20260901_stats_independent_analytic_plan_rewrites.md

[XuPeng-SH]

I am requesting changes on design revision v1 at 82f38af for four blocking contract gaps:

1. Protocol v42 is already allocated on current main, and reusing it can produce wrong results during a rolling upgrade. This RFC assigns v42 to vector grouping expansion, while current main assigns v42 to transactional SQL-task child cleanup (MORPCVersion42, latest=42). A cluster already advertising deployment protocol 42 would therefore enable the new grouping plan before all v42 receivers understand ProjectionGroupingFlags, ProjectionGroupingSetCount, and DynamicGrouping; old receivers can ignore the append-only fields and execute different grouping semantics. Rebase on current main, allocate the next unique version, and cover old/new sender-receiver plus rollback boundaries around the actual predecessor.

2. The partial-SUM legality contract does not define a merge-safe numeric domain. “Supported SUM states/types” is not a proof. The linked implementation currently admits every non-DISTINCT function named sum without a type/state guard. Floating SUM is non-associative: if two dimension PKs map to one output group, introducing a per-PK partial aggregate can regroup values such as 1e16, -1e16, and 1 and change the result. Decimal/integer overflow and error timing also need an explicit contract. This contradicts the RFC invariant that aggregate values and the evaluation/error domain are unchanged. Enumerate the exact admitted input/result/state types and NULL/empty/overflow semantics, reject unsafe states, and add positive plus negative black-box counterexamples.

3. The RFC cannot satisfy its own acceptance gate yet. It says the design is accepted only after complete TPCH/TPCDS planner measurements, while #26768 states that M0 frozen artifacts, checksums, collection commands, typed optimizer decisions, and before/after distributions are the acceptance foundation. This PR links no reproducible corpus or report, and green documentation CI cannot establish the p50/p95 and unchanged-control claims. Either attach the exact M0 evidence required by the gate, or clearly make these implementation-approval gates and sequence #27914/#27915 after a reproducible M0 artifact.

4. Failure containment is source-revert-only and has no detection/rollout contract. These two implementation PRs add roughly 6.8k lines across planner, protocol, compiler, and executor paths and enable several wrong-result-capable rules. A targeted code revert still requires diagnosis, rebuild, and deployment, but the RFC defines neither per-rule observability nor an operational disable/rollback boundary. Define how each rule is identified in plan/decision telemetry, how rollout is staged, and how one rule can be disabled or contained without query/table special cases; if an existing optimizer/deployment gate is the intended mechanism, name its owner, default, version behavior, and rollback test.

Also align the RFC lifecycle metadata during the revision: repository statuses are draft/in-progress/completed/rejected/obsolete/postponed; proposed is not a recognized state. Because approval is tied to the exact revision, the accepted revision should land as in-progress, not require an unreviewed semantic/status follow-up.

[XuPeng-SH]

Counterexample-driven follow-up on the same exact head found two additional correctness cases that the revised contract and implementation tests must close:

1. Partial SUM can expand the evaluation domain even for deterministic expressions. Consider a fact row whose dimension key has no matching PK row and whose SUM argument (or fact-side group expression) is deterministic but fallible. The historical plan performs the INNER JOIN first, drops that fact row, and never evaluates the expression. The proposed partial aggregate runs before the join and evaluates it, so the rewritten query can fail while the old query succeeds. #27914 currently checks volatility and side ownership, but not totality/fallibility, and no trusted FK-completeness proof exists. Require bare/structurally-total expressions or a stronger proof, and add an orphan-key + failing-expression negative SQL test.

2. Shared grouping sets need an explicit empty-input contract. For GROUPING SETS ((k), ()) (or ROLLUP) with an actually empty input but stale positive estimates, the empty grouping set must still produce its global aggregate row (COUNT(*) = 0, SUM(...) IS NULL, correct GROUPING() bits). The new expansion projection emits sets only after receiving a non-empty child batch, while the dynamic aggregate has non-empty group keys and therefore no ordinary H0 dummy group. Define how the empty-set row is synthesized and test empty runtime input independently of planner estimates; stats cannot be used as a semantic proof.

The negative matrix should also include: right-side-PK-only ANTI joins as a control for the left-PK lower bound; SQL NULL versus rollup-sentinel collisions; duplicate grouping sets; CTE producer error/cancel with another consumer draining; disk/FD exhaustion during spill; and cross-rule compositions at the declared boundaries (CTE→grouping, outer reassociation→partial SUM, DNF→shuffle, MARK→build-side selection). Each needs an exact result/error/resource oracle, not only a category name in the table.