Skip to content

Harden filter validation and prevent delimiter key collisions #239

Description

@shaleenji

Problem Statement

The filter indexes encode compound keys with : as a delimiter. If user-provided
field names or category values also contain :, distinct logical (field, value) pairs can collide at the byte-key level. That can produce incorrect
filter matches.

Users are also affected by inconsistent validation behavior: malformed JSON,
unsupported field values, schema type mismatches, invalid operators, and invalid
range shapes should be rejected clearly rather than being skipped or producing
undefined filter state.

Proposed Solution

Reject unsafe or invalid filter inputs before reading or writing the filter
indexes:

  • malformed filter JSON;
  • unsupported filter value types;
  • empty field names;
  • schema type mismatches;
  • invalid operators;
  • invalid numeric range shape or order;
  • : in filter field names;
  • : in category values.

These rejections should be caller-fixable validation errors and should propagate
to API callers as HTTP 400 responses.

Alternatives Considered

Switch immediately to a length-prefixed or binary tuple key format. That is more
robust long term, but it is a breaking on-disk format change and needs migration
or rebuild tooling.

Escape : inside user fields and values. That avoids rejecting input, but still
requires a careful compatibility story for existing keys and makes key handling
more complex.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't working

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions