Support explicit MongoDB filters and aggregation pipelines in external table scans

[iamlinjunhong]

What type of PR is this?

• API-change
• BUG
• Improvement
• Documentation
• Feature
• Test and CI
• Code Refactoring

Which issue(s) this PR fixes:

issue #27536

What this PR does / why we need it:

Support explicit MongoDB filters and aggregation pipelines in external table scans.

Design

• RFC: docs/rfcs/20260826_mongodb_explicit_query.md
• Status: in progress. The RFC decision record contains the independent design PASS for the reviewed revision 76b76496a70d2cca5f408f9ce19a617b5b01f75b.
• Exact-head MongoDB E2E at 8fcd7fc2e076110d6362e70dc7fd2df5f7f8792a: passed 24 scenarios; profiler evidence is raw_scan_documents=5 and pipeline_documents=1.

[qodo-code-review]

Qodo reviews are paused for this user.

Troubleshooting steps vary by plan Learn more →

On a Teams plan?
Reviews resume once this user has a paid seat and their Git account is linked in Qodo.
Link Git account →

Using GitHub Enterprise Server, GitLab Self-Managed, or Bitbucket Data Center?
These require an Enterprise plan - Contact us
Contact us →

[aunjgr]

Reviewed exact head 2abf65ca9eb0823e7137cb61345849f4ecbfe7df against merge base 6a8e7ecc530539948663a35a52259224ee1d128a.

[P1] Use a true minimal row-carrier projection when column pruning removes every mapped column. The new explicit-query path intentionally allows projectedMongoColumns to return no columns for COUNT(*) and query-only scans (pkg/sql/compile/compile.go:2851-2872) and installs NewRowCountConverter. But ProjectionDocument(nil) returns {_id: 0} (pkg/sql/mongodb/predicate.go:148-179). That is an exclusion projection: MongoDB returns every field except _id. The aggregate path appends the same $project, also preserving every non-_id output field.

Consequently a valid query such as SELECT COUNT(*) ... WHERE __mo_query = '{"filter":...}' downloads and validates complete documents even though no value is consumed. A document with an irrelevant field larger than MaxValueBytes/MaxBatchBytes then fails at mongoscan.Call/Converter.AppendDocumentWithBudget with a connector size error; the count would succeed if the scan selected a tiny carrier. This is also an avoidable network and memory amplification for every zero-column count.

For the zero-column path, use an inclusion projection that preserves cardinality while returning a bounded document (for example _id: 1; after an arbitrary pipeline, projecting even a missing _id still emits one small document per pipeline output). Keep the normal mapped-column projection unchanged. Add both Find and Aggregate regressions with a large irrelevant field, proving row count succeeds and the remote projection is genuinely minimal.

[iamlinjunhong]

@Mergifyio queue

[mergify]

Merge Queue Status

• ✅ Entered queue — 2026-09-03 06:34 UTC · Rule: main · triggered by @iamlinjunhong with the @mergifyio queue command
• ✅ Checks passed · in-place
• ✅ Merged — 2026-09-03 07:36 UTC · at 1c1683ca34862de95af3d14302cae100d4de7ef0 · squash

This pull request spent 1 hour 1 minute 41 seconds in the queue, including 1 hour 55 seconds running CI.

Required conditions to merge

• #review-threads-unresolved = 0 [🛡 GitHub branch protection]
  • Support explicit MongoDB filters and aggregation pipelines in external table scans #27553
• github-review-approved [🛡 GitHub branch protection]
  • Support explicit MongoDB filters and aggregation pipelines in external table scans #27553
• any of [🛡 GitHub branch protection]:
  • check-success = Matrixone CI / UT Test on Ubuntu/x86
  • check-neutral = Matrixone CI / UT Test on Ubuntu/x86
  • check-skipped = Matrixone CI / UT Test on Ubuntu/x86
• any of [🛡 GitHub branch protection]:
  • check-success = Matrixone CI / SCA Test on Linux/arm64
  • check-neutral = Matrixone CI / SCA Test on Linux/arm64
  • check-skipped = Matrixone CI / SCA Test on Linux/arm64
• any of [🛡 GitHub branch protection]:
  • check-success = Matrixone Compose CI / multi cn e2e bvt test docker compose(PROXY)
  • check-neutral = Matrixone Compose CI / multi cn e2e bvt test docker compose(PROXY)
  • check-skipped = Matrixone Compose CI / multi cn e2e bvt test docker compose(PROXY)
• any of [🛡 GitHub branch protection]:
  • check-success = Matrixone Utils CI / Coverage
  • check-neutral = Matrixone Utils CI / Coverage
  • check-skipped = Matrixone Utils CI / Coverage
• any of [🛡 GitHub branch protection]:
  • check-success = Matrixone UT Coverage / UT Coverage on Ubuntu/x86
  • check-neutral = Matrixone UT Coverage / UT Coverage on Ubuntu/x86
  • check-skipped = Matrixone UT Coverage / UT Coverage on Ubuntu/x86
• any of [🛡 GitHub branch protection]:
  • check-success = Matrixone Standlone CI / multi CN e2e BVT Test on Linux/x64(COMPOSE, PESSIMISTIC)
  • check-neutral = Matrixone Standlone CI / multi CN e2e BVT Test on Linux/x64(COMPOSE, PESSIMISTIC)
  • check-skipped = Matrixone Standlone CI / multi CN e2e BVT Test on Linux/x64(COMPOSE, PESSIMISTIC)