Skip to content

feat(loops): filter and order data-row loops by a cell value - #348

Merged
DavidBabinec merged 10 commits into
CoreBunch:mainfrom
mostafasadeghidev:feat/loop-cell-filter
Aug 18, 2026
Merged

feat(loops): filter and order data-row loops by a cell value#348
DavidBabinec merged 10 commits into
CoreBunch:mainfrom
mostafasadeghidev:feat/loop-cell-filter

Conversation

@mostafasadeghidev

@mostafasadeghidev mostafasadeghidev commented Aug 6, 2026

Copy link
Copy Markdown
Contributor

What

Lets a base.loop on the data.rows source pick which rows it lists, and which cell it sorts by:

  • Filter (Filter by + Condition): one condition on a row's own cell: is, is not, is checked, is unchecked, has any value, has no value.
  • Order by a field: the table's own fields join the Order by list as Field: <Label>, alongside the existing row columns.

Both are configured in the loop's properties panel and take effect in the canvas preview, in published pages, and in the "load more" endpoint.

Why

A loop could pick a table, an order, and a limit, but not a subset. So a page meant to show the four team members flagged for the About page showed the four most recently updated rows instead, and a list meant to follow the source CMS's own publish date followed our updated_at column. The only workarounds were reordering rows by hand or duplicating a table, and both drift the moment someone edits an item.

This is deliberately one condition rather than a query builder: it covers the real cases (a "featured" checkbox, a category, a "show on homepage" flag) without inventing an AND/OR grammar the editor can't express and future maintainers would have to keep sound in two dialects.

How

New pure module src/core/loops/cellFilter.ts owns the whole contract: parsing, the closed operator set, the SQL fragments, and the rule for which fields a single condition can address.

  • The field name is a bound parameter, never SQL text. Postgres reads cells_json #>> array[$n], SQLite json_extract(cells_json, '$.' || ?). A hostile field id cannot reach the statement (covered by a test).
  • The JSON read appears exactly once per fragment. coalesce(…, '') folds the missing-field case into the comparison instead of needing a second is null branch; repeating the expression would repeat its placeholder while the caller binds the field once.
  • SQLite casts the read to text. json_extract returns INTEGER 1/0 for JSON booleans, and SQLite compares by storage class first, so 1 = '1' is false without the cast. The checked/unchecked operators accept both spellings.
  • Ordering rides on orderBy as cell:<fieldId> rather than a new prop, so every caller that already threads orderBy (publisher, canvas preview endpoint, imported data-order-by) gets it for free. Values compare as text: ISO dates sort chronologically, which is the case this exists for.
  • COUNT applies the same filter on both the post-type and data-kind paths, so pagination never advertises rows the page query drops.

Touched: loops/cellFilter.ts (new), loops/sources/dataRows.ts, LoopPropertiesView.tsx, useLoopPreviewItems.ts, server/handlers/cms/data/tables.ts, core/persistence/cmsData.ts.

User impact

Additive. A loop with no cell field configured behaves exactly as before, and a half-configured filter (field picked, operator not yet) keeps listing everything rather than silently emptying the list.

Verification

bun test          # 6661 pass, 0 fail
bun run build     # exit 0
bun run lint      # exit 0

# and the same loop suites against a real Postgres 16:
DB=postgres TEST_POSTGRES_URL=... bun test src/__tests__/loops/   # 38 pass

Two new files: src/__tests__/loops/cellFilter.test.ts (parsing, parameter binding, dialect shapes, which fields a condition may address) and src/__tests__/loops/dataRowsCellFilter.test.ts (behaviour against a real migrated SQLite database, both table kinds, filtered counts, pagination).

Also exercised end-to-end on a real site migrated from another CMS: an About page list that needed 4 of 30 rows, and an index page that had to follow the source CMS's publish date rather than ours.

Review changes

Three defects found in review, fixed on this branch:

  • The filter outlived its table. Switching the loop's table left cellField naming a column the new table does not have. The field picker falls back to its first option when the stored value is not among its own, so the panel read "no filter" while the query still filtered and returned nothing. Changing the table now clears the cell filter and any cell: sort.
  • Unfilterable fields were on offer. A multiSelect, repeater, or multi-value relation stores a collection; the SQL reads one cell as text, so the comparison ran against ["cat_impact"] and no picked value could match. isCellComparableField keeps those, plus media ids and page-tree refs, out of both pickers.
  • The suite never ran on Postgres. It seeded cells_json pre-stringified, which lands as a jsonb string rather than an object, so 12 of 16 assertions failed on Postgres whatever the feature did. Seeding the way production writes fixes that, and the #>> array[$n] half is now covered: 16 green on SQLite, 16 green against a real Postgres 16.

Also: clearer control labels (Filter by / No filter, has any value / has no value, Field: <name> sorts), loop-source-sql-safety.test.ts widened to the whole src/core/loops/ tree so the gate covers cellFilter.ts, the unused cellFilterMatches export removed, and a CHANGELOG.md entry plus docs/features/loops.md brought up to date.

mostafasadeghidev and others added 2 commits August 6, 2026 10:19
A loop could pick a table and an order but not WHICH rows, so a section
that should list the three featured articles listed the three newest
ones. Migrated Webflow sites hit this immediately: their lists are
curated by a boolean field the loop had no way to read.

- New `@core/loops/cellFilter`: parse one condition out of the loop's
  filter bag and render it as SQL. Six operators (is / is not / checked /
  unchecked / has any value / empty), a closed set — never interpolated.
- Both query paths apply it (post-type version join and data-kind direct
  read) and so do their COUNT queries, or pagination advertises rows the
  page query drops.
- The canvas preview endpoint takes the same condition, so the editor
  shows what the published page will emit.
- Properties panel renders a field picker from the selected table, and
  hides the value box for the operators that ignore it.

Dialect notes, both learned the hard way and now pinned by tests:
SQLite binds `?` by position in the TEXT, so the condition's parameters
sit between tableId and limit/offset; and its json_extract returns
INTEGER 1/0 for booleans, which never equals '1' across storage classes
— hence the cast. The JSON read appears exactly once per fragment so the
field name binds once, with coalesce folding in rows that lack the field.

The field NAME binds as a parameter like the value, so no part of a
filter reaches the statement text.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Ordering was limited to the row's own SQL columns, so a list could not be
sorted by a real date, title, or rank that lives in cells_json. A migrated
site could only approximate 'newest first' with import arrival order,
which drifts the moment anything is re-imported.

- `orderBy` now also accepts `cell:<fieldId>`; `parseCellOrder` reads it
  and `cellOrderSql` renders the expression with the field name bound as
  a parameter, so nothing reaches the SQL text.
- Both query paths order by it, values compare as TEXT in both dialects
  (ISO dates sort chronologically; documented that numbers sort
  lexicographically — one predictable rule beats two engine-specific
  ones), and coalesce gives rows lacking the field a defined position.
- The Loop panel lists the selected table's fields as order options.

Riding on `orderBy` rather than a new prop means every caller that already
threads it — publisher, canvas preview, imported data-order-by attributes
— supports this without further plumbing.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@mostafasadeghidev mostafasadeghidev changed the title feat(loops): filter data-row loops by a cell value feat(loops): filter and order data-row loops by a cell value Aug 6, 2026
"has any value" described the same condition in more words. The
operator id was already `isSet`; the label now says so too.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
mostafasadeghidev and others added 7 commits August 17, 2026 16:14
…ution

CoreBunch#300 landed after this branch was cut and rewrote the same function.
It widened loop media resolution from the built-in featured cell to every
schema-declared media field, which meant reading `fields_json` next to
`kind` and threading the field list into both row projections.

The two changes never touch the same concern: this branch adds an
optional cell condition to the count and page queries and a `cell:<field>`
sort, and CoreBunch#300 changes which media ids get resolved afterwards. The
resolution keeps both verbatim — `table.kind` and `collectMediaIds(rows,
fields)` from upstream, the filter and order plumbing from here.

Verified: tsc clean, and both loop suites green with no assertion
failures (the two reported failures are Windows temp-file locking in test
teardown, which reproduce on an untouched upstream checkout).

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Merging upstream put this branch's filter/order plumbing and CoreBunch#300's wider
media resolution into the same file, and the result crossed the 700-line
module ceiling at 711 — CI caught it.

Raising the ceiling would have been the wrong repair. The two halves
answer different questions: the rest of the file decides WHICH rows a
loop returns — the filter, the order, the page window — while this block
decides what a row's media cells CONTAIN once those rows are in hand.
They only ever met because both needed the same page of rows.

So `resolveMediaIdsToPaths`, `collectMediaIds` and `resolvedMediaOverlay`
move to `dataRowsMedia.ts` unchanged, and the source file imports them.
No behaviour changes: 596 lines and 132 rather than 711.

`MediaAssetRow` and the `readMediaCellIds` / `readRepeaterCell` imports
went with them, and the one test that reached for `resolveMediaIdsToPaths`
now imports it from where it lives.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
… an id

A relation cell stores the referenced row's ID. With `cellValue` rendered
as a free-text box, filtering "Category is Impact" meant the author had to
type `z6W88XshljjzArfa4sV0f` — a string the editor never shows anywhere
and gives them no way to look up. The control was present and unusable,
which is worse than absent: it looks like the feature works.

When the chosen field is a relation, the referenced table's rows now load
and `cellValue` becomes a select of their names. Everything else keeps the
text box, and the valueless operators still drop the control entirely.

Rows are labelled by their `name` cell with the slug as fallback — the
same identity the Data workspace shows in its own row list, so the option
an author picks here reads the same as the row they know.

The fetch follows the pattern already in this component: lazy, keyed on
the resolved target table, and a failed load degrades to an empty list
rather than blocking the panel.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Three ways the filter controls disagreed with the query behind them.

Switching the loop's table left `cellField` pointing at the old table.
The field picker falls back to its first option when the stored value
isn't among its own, so the panel read "no filter" while the query still
filtered on a column the new table doesn't have: an empty list with
nothing on screen explaining it. Changing the table now clears the cell
filter, and any `cell:` sort with it.

The field picker offered every field, including ones a single condition
can never match. A `multiSelect`, a `repeater`, or a multi-value
`relation` stores a collection, and the SQL reads one cell as text, so
the comparison runs against `["cat_impact"]` and no value the author
picks can equal it. Media ids and page-tree refs are opaque strings the
editor never shows. `isCellComparableField` keeps all of them out of
both the filter and the order-by list, which is the same reasoning the
relation value picker was added for.

And the labels described the implementation rather than the task: "Only
rows where" reading into "every row", plus "is set" and "is empty",
which name the operator ids. Now "Filter by" / "No filter", "has any
value" / "has no value", and cell sorts read "Field: Role" so they don't
collide with the row's own columns.

`cellFilterMatches` goes with them. It was exported for "the canvas
preview and any future in-memory path", but the canvas fetches through
the loop-preview endpoint and nothing else ever called it.
The suite seeded `cells_json` as `${JSON.stringify(cells)}`, which is
the one place in the repo that pre-stringifies that column; its sibling
`dataRowsFetch.test.ts` passes the object. On SQLite the column is TEXT
so both work, but Postgres stores it as jsonb and a pre-encoded string
lands as a jsonb *string* rather than an object. Every read then returns
null and 12 of the 16 assertions fail, whatever the feature does.

So the half of this change that Postgres actually runs, the
`#>> array[$n]` reads and the parameter numbering around them, had never
been executed. Seeding the way production writes fixes that: 16 green on
SQLite, 16 green against a real Postgres 16 via `DB=postgres
TEST_POSTGRES_URL=... bun test`. `system` binds as a boolean for the
same reason.

`loop-source-sql-safety.test.ts` widens to the whole `src/core/loops/`
tree. It exists to catch Postgres-isms in loop SQL and stopped at
`sources/`, which left `cellFilter.ts` one level up, the file carrying
the most engine-specific SQL in the subsystem, outside the gate.
The `data.rows` section still claimed filters "narrow by status, author,
category-like fields, date", which was never true and is now wrong in a
new way. It gets the real contract instead: the three `filters` keys,
the closed operator set, the `cell:<fieldId>` order form, which fields
either picker offers and why the rest are excluded, and the rule that
changing the table clears both.

The canvas-path table gains the loop-preview endpoint's new query
params, and the walkthrough now names the controls an author sees.
@DavidBabinec
DavidBabinec merged commit 949ddb2 into CoreBunch:main Aug 18, 2026
3 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants