Generalize IColumnarAggregate routing beyond stratum, and support external-engine/document-path columns

[whilo]

Summary

IColumnarAggregate lets a secondary index compute a group-by/aggregate over its own storage instead of materializing tuples. Today the planner will only route an aggregate to such an index when stratum is on the classpath, and only when the aggregate's columns are datahike attributes the index covers via -indexed-attrs. That makes the protocol effectively stratum-private.

This proposes generalizing the routing so any IColumnarAggregate index can receive an aggregate — including an index whose "columns" are not attributes but fields it exposes through the external-engine mechanism (e.g. nested paths inside an opaque document value). Concretely, this would let a konserve-lmdb document index answer [:find ?city (avg ?age) …] where ?city/?age are nested fields of a :db.secondary/only blob, with no stratum dependency.

Current behavior

Two aggregate fast paths, both gated on stratum:

• try-secondary-index-aggregate (src/datahike/query.cljc ~4012) does call the generic sec/-columnar-aggregate, but every surrounding step is resolved from datahike.index.secondary.stratum via resolve-stratum-fn (query.cljc:4001): stratum-compatible-aggs?, stratum-agg-ops, attr-col-key, columnar-aggregate-from-maps. resolve-stratum-fn returns nil when stratum is absent (requiring-resolve fails), so the whole path bails.
• try-columnar-aggregate (query.cljc:4257) scans PSS into typed columns and calls stratum's columnar-aggregate directly.

The column model is attribute-oriented: candidate columns come from :attr on the scan sub-ops (query.cljc:4031-4036) intersected with sec/-indexed-attrs, and vars map to columns via stratum's attr-col-key.

The two blockers for a non-stratum / document index

1. Stratum coupling. stratum-compatible-aggs?, stratum-agg-ops, attr-col-key, and columnar-aggregate-from-maps are resolved from the stratum namespace. An index that implements IColumnarAggregate but is not stratum gets no routing.
2. Attribute-only columns. A document index covers a single attribute (e.g. :doc/data, :db.secondary/only) and exposes queryable nested paths ([:address :city], [:age]) through external-engine clauses — not as datahike attributes. So all-attrs ∩ -indexed-attrs is empty and the planner never sees :city/:age as aggregatable columns.

Proposal

Split the stratum-specific pieces out of the routing so the index (or a small companion) supplies them, and let external-engine clauses act as column providers.

A. Make the translation pluggable instead of stratum-resolved. Replace the four resolve-stratum-fn calls in try-secondary-index-aggregate with per-index hooks. Options:

• extend IColumnarAggregate (or add a sibling protocol) with -agg-compatible?, -agg-query-spec (build {:group :agg :where} from the plan), and -agg-result->tuples (adapt result maps to find-order tuples); or
• a multimethod keyed on the index's type keyword, defaulting to a generic implementation.

Stratum keeps its current fast, specialized versions by implementing the hooks; a generic default covers simple :count/:sum/:min/:max/:avg + equality/range :where, which is what most external indices need.

B. Treat external-engine binding vars as columns. When an aggregate's group/agg vars are bound by external-engine clauses (the :datahike/external-engine mechanism, query/execute.cljc:5036) that all target the same IColumnarAggregate index, map each var to the column that clause names (for a document index, the projection path), and hand {:group [paths…] :agg [[op path]…] :where …} to -columnar-aggregate. This is the missing link that lets a "column" be a document path rather than an attribute. The external-engine metadata could carry an explicit :provides-column (the path) so the planner has a first-class mapping, rather than inferring it.

C. Minimal interim. Even without B, a generic fallback in try-secondary-index-aggregate — "if stratum is absent but one IColumnarAggregate index covers all find/group/agg columns, build the query-spec generically and call -columnar-aggregate with a default result adapter" — unblocks attribute-covering external indices immediately and is a small, additive change.

Backwards compatibility

Purely additive. Stratum keeps both existing paths (it would implement the pluggable hooks, or the code keeps the stratum branch and adds a generic branch alongside). Indices that don't implement IColumnarAggregate are unaffected and fall through to the relation path as today.

Concrete target

A konserve-lmdb document secondary index (stores an EDN document as one boring blob under a :db.type/bytes :db.secondary/only attribute, indexes selected nested paths) already implements IColumnarAggregate and computes group-by/aggregate over its documents in a single zero-copy pass. It can answer:

[:find ?city (avg ?age) (count ?e)
 :where
 [(konserve-lmdb.datahike/doc-project :idx/docs :ALL [:address :city]) [[?e ?city]]]
 [(konserve-lmdb.datahike/doc-project :idx/docs :ALL [:age])          [[?e ?age]]]]

today only by calling -columnar-aggregate directly. With A+B it would route automatically. (Implementation: replikativ/konserve-lmdb#7 — the aggregate is computed via konserve-lmdb project-reduce; nested-field aggregation is inherently a full scan, since there is no AVET for a field inside a blob.)

What already exists on the index side

• IColumnarAggregate/-columnar-aggregate implemented (both arities, with an entity-filter subset).
• -indexed-attrs, ISecondaryScannable/-sec-value, IVersionedSecondaryIndex.
• A public query-aggregate entry point that does today what the planner would do automatically after this change.

Happy to prototype the generic try-secondary-index-aggregate fallback (option C) against konserve-lmdb's index if that's a useful starting point.