C source architecture

This document explains how the C modules cooperate to turn a session-local hypothetical IVFFlat definition into a planner candidate visible only to safe plain EXPLAIN planning.

Static module relationships

flowchart LR
    PG[(PostgreSQL 16)]
    SQL[SQL installation objects]

    Entry["pgvector_hypo.c<br/>PostgreSQL adapter"]
    Safety["planning_safety.c<br/>candidate admission"]
    Registry["registration.c<br/>session registry + definition parser"]
    Model["ivfflat_model.c<br/>registered definition +<br/>candidate derivation"]
    Type["ivfflat_type.c<br/>catalog-backed storage policy"]

    SQL -->|binds SQL functions| Entry
    PG -->|utility, planner, relation-info,<br/>and EXPLAIN-name hooks| Entry
    Entry -->|enter, leave, admit| Safety
    Entry -->|create, drop, reset, list,<br/>name, inject candidates| Registry
    Registry -->|create, derive,<br/>cost, inspect| Model
    Model -->|resolve official identity,<br/>validate, estimate rows,<br/>size physical items| Type

    Registry -. "PostgreSQL cost callback:<br/>lookup fake OID, then delegate" .-> Model

    classDef adapter fill:#e0e7ff,stroke:#4338ca,stroke-width:2px;
    classDef deep fill:#0f172a,color:#fff,stroke:#0f172a,stroke-width:3px;
    class Entry adapter;
    class Safety,Registry,Model,Type deep;

The dependency direction is intentionally one-way:

pgvector_hypo.c
  ├── planning_safety.h
  └── registration.h
        └── ivfflat_model.h
              └── ivfflat_type.h

ivfflat_model.c does not reach back into Registration. PostgreSQL requires the cost estimator to be a function pointer on IndexOptInfo, so Registration provides that adapter: it resolves the fake OID to the owned model and delegates the calculation to ivfflat_model_costestimate().

End-to-end runtime flow

Registering a hypothetical index

sequenceDiagram
    autonumber
    actor User
    participant SQL as pgvector_hypo_create_index()
    participant Reg as Registration
    participant Parser as PostgreSQL parser/catalogs
    participant Model as IvfflatModel
    participant Type as IvfflatType

    User->>SQL: CREATE INDEX text
    SQL->>Reg: registration_from_sql(sql)
    Reg->>Parser: parse and transform IndexStmt
    Parser-->>Reg: relation, key tree, predicate,<br/>opclass name list, options
    Reg->>Reg: create per-attempt child memory context
    Reg->>Model: ivfflat_model_create(...)
    Model->>Type: resolve key result type + IVFFlat + opclass
    Type-->>Model: captured official catalog policy
    Model->>Type: validate fixed dimensions,<br/>estimate indexable rows, size items
    Model-->>Reg: registration-time model + page estimate
    Reg->>Reg: allocate checked fake OID
    Reg->>Model: ivfflat_model_assign_identity(model, oid)
    Reg->>Reg: commit ordered registration metadata
    Reg-->>SQL: RegistrationResult
    Note over Reg,Model: Any error after child creation deletes only the attempt context
    SQL-->>User: indexrelid, indexname

No catalog row or relation file is created. Each retained model lives in its own child of the current backend's registration context, so it is session-local rather than transaction-local. Failed attempts delete their child atomically; drop deletes one child; reset clears the parent and every child.

Planning a safe plain `EXPLAIN`

sequenceDiagram
    autonumber
    actor User
    participant Hook as pgvector_hypo.c hooks
    participant Safe as PlanningSafety
    participant PG as PostgreSQL planner
    participant Reg as Registration
    participant Model as IvfflatModel
    participant Type as IvfflatType

    User->>Hook: EXPLAIN SELECT ... ORDER BY embedding operator query LIMIT n
    Hook->>Safe: planning_safety_enter_utility(statement)
    Safe-->>Hook: previous safety snapshot
    Hook->>PG: continue ProcessUtility chain
    PG->>Hook: planner hook
    Hook->>Safe: planning_safety_enter_planner()
    Hook->>PG: continue planner chain
    PG->>Hook: get_relation_info hook
    Hook->>Safe: admits_candidate(enabled)?
    Safe-->>Hook: true only for top-level plain EXPLAIN
    Hook->>Hook: temporarily publish in-progress RelOptInfo
    Hook->>Reg: registration_add_relation_indexes(relation, root, rel, oid)
    Reg->>Reg: derive partition ancestry once for this relation

    loop each matching registration in stable order
        Reg->>Model: ivfflat_model_derive_index(...)
        Model->>Type: revalidate captured catalog identity
        Model->>Model: map key + predicate to leaf,<br/>prepare trees, estimate predicate
        Model->>Type: validate result typmod +<br/>estimate indexable rows
        Model-->>Reg: synthetic IndexOptInfo or NULL
        Reg->>Reg: attach registration_costestimate callback
        Reg->>PG: append candidate to RelOptInfo.indexlist
    end
    Hook->>Hook: restore planner relation array

    PG->>Reg: cost callback(IndexPath)
    Reg->>Reg: lookup model by fake index OID
    Reg->>Model: ivfflat_model_costestimate(model, path, ...)
    Model-->>PG: startup cost, total cost, selectivity, pages
    PG->>PG: compare all paths and choose a plan
    PG->>Hook: EXPLAIN index-name hook(fake OID)
    Hook->>Reg: registration_name(fake OID)
    Reg-->>Hook: readable hypothetical name
    Hook-->>User: plan text or JSON

    Hook->>Safe: planning_safety_leave_planner()
    Hook->>Safe: planning_safety_leave_utility(snapshot)

The physical and hypothetical candidates use PostgreSQL's normal path generation and comparison. The extension contributes metadata and cost, but it does not force an index scan.

Why execution cannot use a fake index

stateDiagram-v2
    [*] --> UtilitySeen
    UtilitySeen --> PlainExplain: EXPLAIN without ANALYZE or EXECUTE
    UtilitySeen --> Ineligible: ordinary execution / ANALYZE / EXECUTE
    PlainExplain --> TopPlanner: planner depth = 1
    PlainExplain --> NestedPlanner: planner depth > 1
    TopPlanner --> CandidateAdmitted: enabled
    TopPlanner --> CandidateWithheld: disabled
    NestedPlanner --> CandidateWithheld
    Ineligible --> CandidateWithheld
    CandidateAdmitted --> ExplainPlanOnly
    CandidateWithheld --> NormalPlanning

IndexOptInfo.hypothetical is descriptive metadata, not the safety mechanism. Safety comes from withholding the candidate unless PlanningSafety proves the current work is enabled, top-level, plain EXPLAIN planning.

`src/pgvector_hypo.c`

Responsibility

This is the PostgreSQL adapter and composition root. It:

installs and restores PostgreSQL hooks;
chains any previously installed hook;
exposes C functions declared by the extension SQL file;
converts C results into PostgreSQL tuples and tuplestores;
delegates safety decisions and registration behavior to deeper modules.

It intentionally does not know IvfflatModel internals.

PostgreSQL extension interfaces

Interface	Called by	Current use
`_PG_init()`	PostgreSQL when loading the shared library	Defines `pgvector_hypo.enabled`, installs four hooks, and initializes Registration.
`_PG_fini()`	PostgreSQL when unloading	Restores the previous hook pointers.
`pgvector_hypo_create_index()`	Installed SQL function	Converts SQL text and delegates to `registration_from_sql()`.
`pgvector_hypo_drop_index()`	Installed SQL function	Removes one session registration by fake OID.
`pgvector_hypo_reset()`	Installed SQL function	Clears all session registrations.
`pgvector_hypo_list()`	Installed SQL function	Materializes `RegistrationInfo` rows supplied by `registration_visit()`.

Hook adapters

Internal hook	PostgreSQL seam	Current use
`pgvector_hypo_process_utility()`	`ProcessUtility_hook`	Detects eligible plain `EXPLAIN` context and restores state with `PG_FINALLY`.
`pgvector_hypo_planner()`	`planner_hook`	Tracks planner nesting depth and restores it even when planning errors.
`pgvector_hypo_get_relation_info()`	`get_relation_info_hook`	Requests synthetic indexes only when candidate admission succeeds and lends the in-progress `RelOptInfo` to candidate selectivity estimation.
`pgvector_hypo_get_index_name()`	`explain_get_index_name_hook`	Maps an admitted fake OID to its readable hypothetical name.

The EXPLAIN-name adapter checks planning_safety_admits_name() before requesting a registry lookup. Ineligible contexts therefore preserve previous-hook fallback without touching Registration.

PostgreSQL calls get_relation_info_hook before build_simple_rel() places the current RelOptInfo in PlannerInfo.simple_rel_array. Predicate selectivity functions expect that lookup to work. The adapter therefore places the supplied relation in its own empty slot only around Registration's derivation call and restores the prior value with PG_FINALLY before chaining the previous hook. This narrow temporal adapter keeps PostgreSQL's planner timing out of the model interface and cannot leak state across an error.

return_registration_result() and put_registration_info() are local presentation helpers. They are not interfaces consumed by other source files.

`src/planning_safety.c` and `.h`

Responsibility

This module owns the candidate-admission invariant. Its implementation keeps a backend-local snapshot containing:

whether the active utility statement is a plain EXPLAIN;
current planner nesting depth.

Interface

Interface	Caller	Current use
`planning_safety_enter_utility(statement)`	ProcessUtility adapter	Classifies the statement and returns the previous snapshot.
`planning_safety_leave_utility(snapshot)`	ProcessUtility adapter	Restores state after normal return or PostgreSQL error unwinding.
`planning_safety_enter_planner()`	Planner adapter	Increments planner depth.
`planning_safety_leave_planner()`	Planner adapter	Decrements planner depth and asserts balanced use.
`planning_safety_admits_candidate(enabled)`	Relation-info adapter	Requires enabled + plain EXPLAIN + planner depth exactly one.
`planning_safety_admits_name(enabled)`	EXPLAIN-name adapter	Allows fake-name rendering only in enabled plain EXPLAIN context.

A statement is not plain when ANALYZE is true or the explained statement is EXECUTE. Nested planning is deliberately excluded from candidate admission.

`src/registration.c` and `.h`

Responsibility

Registration is the owner of session-local hypothetical definitions. It hides:

the backend parent context, per-registration child contexts, and ordered list;
SQL parsing and transformed IndexStmt validation;
physical CREATE INDEX ownership parity before model/statistics access;
ordinary-column or immutable-expression key extraction, predicate extraction, and lists option parsing;
fake-OID allocation and catalog collision checks;
model lifecycle;
relation candidate injection;
fake-OID lookup needed by the PostgreSQL cost callback.

Public value types

`RegistrationResult`

Returned after creation. It contains only the fake OID and display name needed by the SQL adapter.

`RegistrationInfo`

A read-only presentation snapshot used by pgvector_hypo_list(). Its pointer is valid only during the visitor callback. Callers do not receive the mutable model or registry list. The snapshot retains PostgreSQL's BlockNumber internally; the SQL adapter widens it to int64 and exposes pages as bigint, so every valid relation block count remains lossless.

Interface

Interface	Caller	Current use
`registration_initialize()`	`_PG_init()` and registration path	Creates the session parent context lazily.
`registration_from_sql(sql)`	SQL create adapter	Parses one supported `CREATE INDEX`, requires relation ownership through PostgreSQL's DDL callback, builds a model in an atomic child context, allocates identity, and commits ordered metadata.
`registration_name(indexoid)`	EXPLAIN-name adapter	Returns the registered display name for a fake OID, or `NULL`.
`registration_drop(indexoid)`	SQL drop adapter	Unlinks one registration and deletes its child context; returns whether it existed.
`registration_reset()`	SQL reset adapter and evidence workflows	Clears the parent context, ordered list, and fake-OID cursor.
`registration_visit(visitor, context)`	SQL list adapter	Presents each registration without exposing registry representation.
`registration_add_relation_indexes(relation, root, rel, relid)`	Relation-info hook	Derives and appends matching synthetic planner candidates.

Important internal flow

registration_costestimate() is the PostgreSQL cost adapter attached to every synthetic IndexOptInfo. PostgreSQL supplies only the IndexPath; Registration uses path->indexinfo->indexoid to recover the owned model from the ordered registry, then delegates the algorithm to ivfflat_model_costestimate(). All relation-specific definition work has already happened during candidate derivation; the callback owns only pgvector-compatible path costing.

The ordered list remains authoritative for public newest-first visitation, fake-OID lookup, and candidate traversal. A measured keyed-lookup experiment did not earn its extra representation; see the architecture deepening log. Partition ancestry is transient planner work, computed once per relation-info invocation and borrowed by model matching.

`src/ivfflat_model.c` and `.h`

Responsibility

IvfflatModel is opaque outside its implementation. It represents a registered IVFFlat definition and knows how to derive relation-specific planner metadata. Its implementation owns:

the registered relation OID, ordinary-column attribute number when applicable, and serialized transformed key and predicate trees;
the index tablespace resolved from the registration-time default_tablespace policy;
registration-time dimensions and page estimate;
pre-build list-occupancy and integer-page-phase estimation;
direct/ancestor matching and parent-to-leaf mapping of both key and predicate;
catalog-tree checks that make stale functions, operators, attributes, types, or typmods fail closed;
PostgreSQL preparation of expression and predicate trees;
relation-specific predicate selectivity and page estimation;
IndexOptInfo construction;
pgvector-compatible IVFFlat costing.

Opclass/type identity, dimensional validity, indexability rules, and physical item representation belong to IvfflatType; the model asks that policy instead of switching on opclass names.

Lifecycle interface

Interface	Caller	Current use
`ivfflat_model_create(...)`	Registration create path	Validates the relation and transformed key, captures PostgreSQL's omitted-`TABLESPACE` result, resolves storage policy, serializes key/predicate trees, and computes conservative registration metadata.
`ivfflat_model_assign_identity(model, oid)`	Registration create path	Stores the fake OID and builds the readable index name.

Model allocation has no field-wise destroy interface. Registration owns the containing child memory context and releases the model atomically by deleting that context.

Relation and planner interface

Interface	Caller	Current use
`ivfflat_model_derive_index(...)`	Registration injection	Performs matching, catalog revalidation, tree remapping/preparation, row/page estimation, and `IndexOptInfo` construction as one fail-closed operation.
`ivfflat_model_costestimate(...)`	Registration cost adapter	Runs generic index costing, probe-ratio adjustments, and the TOAST startup branch.

Returning NULL from derivation means the registration is not applicable or has become stale. Registration does not need to reproduce a match/can-model/make-index protocol, so the seam has one success value: a complete planner candidate.

For expression candidates, indexkeys[0] is zero, indexprs contains the prepared expression, and indextlist exposes that same result type and typmod. For ordinary keys, indexkeys[0] is the mapped attribute number and indexprs is empty. Predicates are kept in PostgreSQL's implicit-AND form, leaving normal check_index_predicates() to decide implication and populate predOK and indrestrictinfo.

Candidate tuple metadata follows PostgreSQL rather than physical AM exclusions: a full index advertises the base relation's rel->tuples, while a partial index advertises predicate-selective indexed rows. NULL and normalization exclusions still determine physical IVFFlat pages. Keeping those meanings separate matches PostgreSQL 16's IndexOptInfo contract.

An omitted TABLESPACE is resolved once at registration, including PostgreSQL's ACL checks, and the captured OID becomes candidate reltablespace. A later default_tablespace change does not retroactively move the modeled definition. Dropping a captured tablespace makes derivation fail closed. Explicit TABLESPACE syntax remains outside the supported surface.

For a direct leaf EXPLAIN, the planner already holds the leaf lock. Opening a registered partition parent with a blocking lock would invert PostgreSQL DDL's parent-to-child lock order. Parent-to-leaf tree mapping therefore takes the parent lock conditionally and returns NULL when it is unavailable; the next planning pass can derive the candidate after contention ends.

Read-only inspection interface

The ivfflat_model_indexoid(), relid(), attnum(), lists(), dimensions(), pages(), and name() functions are used only inside Registration to construct RegistrationInfo or perform ownership work. They do not expose the model to the PostgreSQL SQL adapter.

Two meanings of pages

The model's stored pages value is registration-time definition metadata and is shown by the list interface. It deliberately ignores a predicate and uses a conservative no-expression-statistics policy.
ivfflat_model_derive_index() computes candidate pages from the current relation's planner row estimate, supported column statistics, and mapped predicate selectivity. This matters for partitions and changed statistics.

`src/ivfflat_type.c` and `.h`

Responsibility

This deep module is the authority for supported pgvector catalog identity, build-validity rules, indexable-row policy, and physical item representation. It accepts exactly seven official definitions:

Storage type	IVFFlat maximum	Accepted official opclasses
`vector(n)`	2,000	`vector_l2_ops`, `vector_ip_ops`, `vector_cosine_ops`
`halfvec(n)`	4,000	`halfvec_l2_ops`, `halfvec_ip_ops`, `halfvec_cosine_ops`
`bit(n)`	64,000	`bit_hamming_ops`

Each registration receives an opaque captured policy containing the extension, schema, access-method, type, opclass, opfamily, and support-procedure OIDs. Static specifications describe the expected official combinations; custom opclasses are rejected even when their names resemble an official one.

Interface

Interface	Caller	Current use
`ivfflat_type_resolve(attribute_type, access_method, opclass_name)`	Model creation	Applies PostgreSQL explicit/default opclass resolution, then requires exact official extension membership and support procedures.
`ivfflat_type_dimensions(type, typmod, key_name)`	Model creation	Enforces a fixed typmod, the storage maximum, and support-procedure-driven spherical-k-means dimension validity.
`ivfflat_type_accepts(type, result_type, typmod)`	Candidate derivation	Performs a pure compatibility check against an already revalidated policy.
`ivfflat_type_is_valid(type)`	Candidate derivation	Rechecks every captured catalog identity and support procedure so stale registrations are withheld.
`ivfflat_type_estimate_rows(type, relation, attnum, rows)`	Model creation and derivation	Returns `IvfflatRowEstimate`: indexable rows plus a dominant indexed-value fraction from column NULL/MCV statistics and optional support-procedure-2 norm calls; expression keys remain conservative.
`ivfflat_type_page_layout(type, dimensions)`	Model page estimator	Returns aligned list-item and index-tuple sizes, including item identifiers.
`ivfflat_type_*` identity accessors	Candidate construction	Lend validated access-method, input-type, opclass, opfamily, and readable names without exposing representation.

IvfflatPageLayout is an internal C interface between the type and model modules. Vector and halfvec use their dense datum headers and component widths; bit uses PostgreSQL's VARBITTOTALLEN() representation. It is not visible to SQL users.

Page layout is deterministic storage policy; pre-build occupancy uncertainty is not. The model applies one storage-agnostic prior: split a known dominant MCV, use Poisson occupancy for sparse lists, and use a discrete-uniform integer-page phase plus one aggregate standard deviation for dense lists. A measured fixed-bit-specific phase/headroom policy did not generalize and was rejected, so IvfflatType exposes no stochastic tuning constants.

Object ownership and lifetimes

flowchart TD
    Backend["PostgreSQL backend process"]
    RegistryContext["Registration parent context<br/>session lifetime"]
    Lookup["Ordered registration list<br/>registry metadata"]
    Records["Registration owner records<br/>context + model pointers"]
    ModelContexts["Per-registration child contexts<br/>until drop/reset/session end"]
    Models["Opaque IvfflatModel + strings"]
    Policies["Captured IvfflatType<br/>catalog policy"]
    Specs["Static supported-definition specs<br/>process lifetime"]
    PlannerContext["Planner memory context<br/>one planning pass"]
    Indexes["Synthetic IndexOptInfo nodes"]
    Info["RegistrationInfo snapshot<br/>one visitor callback"]

    Backend --> RegistryContext
    RegistryContext --> Lookup
    RegistryContext --> ModelContexts
    ModelContexts --> Records
    ModelContexts --> Models
    ModelContexts --> Policies
    Lookup -.points to.-> Records
    Records -.points to.-> Models
    Models -.points to.-> Policies
    Policies -.borrow.-> Specs
    Backend --> PlannerContext
    PlannerContext --> Indexes
    Models -->|derive| Indexes
    Models -->|project| Info

    classDef owned fill:#0f172a,color:#fff,stroke:#0f172a;
    classDef borrowed fill:#eef2ff,stroke:#4f46e5,stroke-dasharray:5 5;
    class RegistryContext,Lookup,Records,ModelContexts,Models,Policies,PlannerContext,Indexes owned;
    class Specs,Info borrowed;

Important lifetime rules:

RegistrationResult.indexname and registration_name() point into the registration memory context; callers must not free them.
RegistrationInfo is stack-backed during registration_visit() and must not be retained by a visitor.
Each model's IvfflatType policy lives in the same registration child context; it borrows a static immutable supported-definition specification.
Synthetic IndexOptInfo nodes belong to the active planner memory context.
Creation errors delete only the attempt child context; earlier registrations remain valid.
Drop invalidates pointers for exactly one registration and deletes its child context.
Reset invalidates all model/name pointers and restarts fake-OID allocation.

Quick navigation by change type

Change	Start here	Then inspect
SQL function behavior	`pgvector_hypo.c`	`registration.h`, installation SQL
Safe EXPLAIN admission	`planning_safety.c`	hook adapters in `pgvector_hypo.c`
Supported CREATE INDEX syntax	`registration.c`	`ivfflat_type.c`, regression SQL
New pgvector storage type/opclass	`ivfflat_type.c`	model statistics/page logic, evidence matrix
Page estimate discrepancy	`ivfflat_model.c`	`ivfflat_type_page_layout()`, physical evidence
Cost discrepancy	`ivfflat_model_costestimate()`	pgvector baseline and planner settings
Partition or expression behavior	`ivfflat_model_derive_index()`	Registration candidate injection and PostgreSQL tree mapping
Fake OID or session lifecycle	`registration.c`	list/name SQL adapters

FilesExpand file tree

c-source-architecture.md

Latest commit

History

c-source-architecture.md

File metadata and controls

C source architecture

Static module relationships

End-to-end runtime flow

Registering a hypothetical index

Planning a safe plain EXPLAIN

Why execution cannot use a fake index

src/pgvector_hypo.c

Responsibility

PostgreSQL extension interfaces

Hook adapters

src/planning_safety.c and .h

Responsibility

Interface

src/registration.c and .h

Responsibility

Public value types

RegistrationResult

RegistrationInfo

Interface

Important internal flow

src/ivfflat_model.c and .h

Responsibility

Lifecycle interface

Relation and planner interface

Read-only inspection interface

Two meanings of pages

src/ivfflat_type.c and .h

Responsibility

Interface

Object ownership and lifetimes

Quick navigation by change type

Planning a safe plain `EXPLAIN`

`src/pgvector_hypo.c`

`src/planning_safety.c` and `.h`

`src/registration.c` and `.h`

`RegistrationResult`

`RegistrationInfo`

`src/ivfflat_model.c` and `.h`

`src/ivfflat_type.c` and `.h`