feat: replace Elasticsearch with PostgreSQL full-text search (PPT-2644, 2/2) - #443
Draft
camreeves wants to merge 3 commits into
Draft
feat: replace Elasticsearch with PostgreSQL full-text search (PPT-2644, 2/2)#443camreeves wants to merge 3 commits into
camreeves wants to merge 3 commits into
Conversation
…on (PPT-2644) - Utils::TextSearch: never-500 q -> tsquery sanitizer (AND of prefix tokens) - application.cr: search_tsquery/search_limit/search_offset + paginate_search; fix the Link header off-by-one that skipped one row per page - modules: parent-driver search via EXISTS join (has_parent parity); scoped listing now uses IN() (the ES version returned nothing for >1 visible module) - zones: tree/tag filters in SQL, children counts via GROUP BY - drivers: straight conversion NOTE: pg-orm 2.2.x Collections cannot compile a single-argument raw where() (NamedTuple-in-union overload) — always pass bind args. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…PT-2644) - docker-compose: remove elastic + search-ingest services/anchors - spec/helper: remove refresh_elastic/ensure_keyword_mapping/clear_elastic - test_base_index macro: synchronous save->query->assert (no index lag) - root.cr: /reindex + /backfill deprecated no-ops; search_ingest out of the cluster version fan-out; require removed - shard.yml/lock: search-ingest dependency removed (neuroplastic remains transitively via placeos-models until the models major drops it) - spec/migration: TEMPORARY models branch pin (revert before merge) Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…2644) All 24 Elasticsearch-backed controllers now query PostgreSQL directly: filters become SQL WHERE clauses, q matches the generated search_vector columns, and ES parent-child becomes EXISTS joins (modules<->driver, system-triggers<->trigger, systems<->trigger instances, assets<->asset_type). paginate_results and the ES spec helpers are gone. Deliberate behavior fixes (each spec-pinned): - systems?trigger_id= worked never (unsatisfiable ES query) — real join - systems?email= was a no-op alone and WIDENED the group/subsystem zone scope when combined — now a strict AND filter - scoped (non-admin) module listing returned [] when the user could access more than one module — now IN() - alerts multi-dashboard scoping matched nothing for authorities with more than one dashboard — now IN() - alerts?enabled=false and asset_categories?hidden=false were ignored (falsy-Bool) — now filter both values - system-triggers q was silently ignored — now matches the parent trigger's text - assets can now be searched by asset-type name (previously a TODO) Known ES-era quirks preserved: zones/assets/pending_mails multi-zone AND-containment, public/important/triggered only-when-true filters, undecrypted settings search results. pg-orm gotchas encoded: never single-arg raw where (does not compile); arrays cannot bind as one parameter — sql_array() splices per-element placeholders; empty arrays guard with an empty-page short-circuit. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Part 2 of PPT-2644. Depends on PlaceOS/models#322 (the
search_vectormigration) being deployed; builds against models 9.x as-is.What
Every index route (24 controllers) now queries PostgreSQL directly instead of Elasticsearch:
qis translated byUtils::TextSearchinto a bound-parameterto_tsquery('simple', …)of AND-ed prefix tokens against the generatedsearch_vectorcolumns. The sanitizer never raises and degrades ES-era syntax (field prefixes,+|-"()*~, boolean words) into plain terms — pinned by spec vectors including Backoffice'stags:(+level AND +building)string.WHEREclauses with the same semantics they effectively had (multi-zone AND-containment, only-when-true booleans, etc.).EXISTSjoins: modules↔driver (search modules by driver name — instantly consistent on driver rename, unlike the stale-copy ES join docs), system-triggers↔trigger, systems↔trigger-instances, and assets↔asset-type (previously a commented-out TODO, now implemented).name, iddefault) — ES "relevance" order was already discarded by the id-rehydration path, and intra-page order was PG-arbitrary./reindex+/backfillare deprecated 200 no-ops (Backoffice's admin card calls them);search_ingestleft the/cluster/versionsfan-out; thesearch-ingest+ directneuroplasticdeps are gone (neuroplastic remains transitively via models until the models major drops it).X-Total-Count, and theLinkheader off-by-one that silently skipped one row per page (also affecting the 7 pre-existing SQL routes) is fixed.refis accepted-and-ignored;fieldslikewise (it never was a projection).refresh_elastic/sleep/poll sites now synchronous asserts.Deliberate behavior fixes (each with a pinning spec)
GET /systems?trigger_id=— never returned anything (ES query was unsatisfiable) → real join.GET /systems?email=— no-op alone; widened the group/subsystem authorization scope when combined → strict AND filter.GET /modulesreturned[]whenever the user could access >1 module →IN().IN().alerts?enabled=false,asset_categories?hidden=falsewere silently ignored → filter both values.GET /systems/:id/triggers?q=was silently ignored → matches parent trigger text.New coverage for previously untested surface: authentications (×3, no spec file existed), domains index, schema, webrtc
/rooms, short_url index, and the systems filter matrix (capacity/bookable/features/public/signage/zone_id/email/trigger_id).Deliberate divergences
password_digest, api-key secrets, unencryptedsettings_string, …).qis AND-of-prefixes (ES OR-ed terms; autocomplete UIs already intersect client-side).fields=name^2boost syntax is ignored.⚠ Before merge
spec/migration/Dockerfileis TEMPORARILY pinned to the models PR branch — revert to master once feat: PG full-text search columns (PPT-2644, 1/2) models#322 merges.--docs) pending.Rollout: merge models#322 → this → one release with ES still deployed-but-unread (rollback = previous image) → then infra removal (branches prepared: local, k8s-helm, PlaceOS meta, init, user-interfaces e2e, backoffice, ts-client, docs).
Full plan + research: https://gist.github.com/camreeves/4b3cb2cffd2544e894ec34dcdae799a6
🤖 Generated with Claude Code