Behavioral Fixes for limit and skip

[thehabes]

Summary

Resolves #301.

Resolves #300. POST /query and HEAD /query now page over .sort({ _id: 1 }), so page boundaries no longer rest on natural order.

Partially addresses #307. mergeSearchResults() now ranks on __rerum.score, the field the branch pipelines actually write. The embedded-object _id dedup bug is not fixed here.

What changed

Request	Before	After
?limit=1e3	1 record	400
?limit=10abc	10 records	400
?limit=250.7	250 records	400
?limit=abc, ?limit=0x10, ?limit=	100 (the default)	400
?limit=0, ?limit=-5	100 (the default)	400
?limit=100&limit=200	100	400
?limit=200&limit=100	200	400
?skip=1e3	offset 1	400
?skip=2.9	offset 2	400
?skip=abc, ?skip=-5, ?skip=	offset 0	400
?limit=1000	500, silently	500, reported in headers
?skip=150000	the page at 100000, forever	400 naming the maximum

Values are now validated as decimal integer strings before they are parsed, because Number.parseInt guesses. A parameter supplied more than once arrives from Express as an Array, which has no single correct reading, so it is refused rather than resolved by position.

The two maximums are deliberately not alike

An over-maximum limit is clamped. A page size the server can honour in part is conventional to reduce, and rejecting it would be the more breaking choice.

An over-maximum skip is rejected. Clamping it served the page at the maximum on every request past it — skip=100000, skip=100100 and skip=250000 returned byte-identical bodies — so a client advancing skip and stopping on an empty page never terminated, and accumulated the same records on every pass. There is no honest reading of it, so it is refused with a message naming the configured maximum.

Every paged response now reports what was applied

Pagination-Limit: 500
Pagination-Skip: 0
Pagination-Limit-Max: 500
Pagination-Skip-Max: 100000

The reported limit and skip are the values actually applied, which is what lets a client tell a truncated page from a genuine final one. Names are unprefixed to match the existing Annotations-Gathered and Current-Overwritten-Version headers rather than introduce a new prefix. Access-Control-Expose-Headers is already *, so browser clients can read them with no CORS change.

Breaking

Requests that return 200 today return 400. Land on dev first and give known client maintainers notice.

The skip rejection is the part most likely to be hit by something real, which is the point of it. {"@type":"oa:Annotation"} still returns a document at skip=100000 on production, so the boundary is reachable by live traffic.

The published pagedQuery example in public/API.html was the non-terminating shape, so it is rewritten: it advances by the page size the server reports applying rather than by results.length, stops on a short page, and lets the boundary 400 end a walk that runs too deep.

Out of scope

• The OpenAPI contract does not document limit or skip at all today and no drift guard covers them. Left to The pagination contract is undocumented, misdocumented, and absent from the OpenAPI contract #305.
• rest.messenger appends "The body of your request was invalid… make sure it is valid JSON" to every 400, which reads oddly on a query-string error. Narrowing it means touching the shared error renderer for all 400s. The pagination messages are written to stand on their own; worth its own issue.
• HEAD /query answering an empty page with 404 (HEAD /query answers an empty page with 404 where POST /query answers 200 [] #304) is untouched. It is blocked on HEAD history - since - query #96, and the reading there is that the .head() handler should be removed from /query rather than repaired — note that /since/:_id and /history/:_id are .get() routes where HEAD is valid and should stay.

[cubap]

Review: pagination fixes look good, two asks before merge

The behavioral change is what #300 and #301 need: limit/skip stop being guessed, skip past the ceiling returns 400, /query pages over a stable _id order, and the applied/maximum values reach the client in headers. I tested the diff locally and the logic holds.

Two things I'd like addressed before merging:

1. getPagination(query, res, defaultLimit) signature. Passing res as a positional argument so it can set headers mixes pagination parsing with HTTP plumbing. In controllers/gog.js the call becomes getPagination(req.query, null, 50), which is readable only if you already know what the null means. Could we make this clearer? Options:

   • Move header reporting out of getPagination entirely, e.g. reportPaginationHeaders(res, limit, skip, limitMax, skipMax) called by each controller.
   • Or pass an options object: getPagination(req.query, { res, defaultLimit: 50 }).
     This keeps the utility testable without a response double and removes the null dance.

2. Shared decimal-integer validation helper. resolveQueryCap and readWholeNumberParam both use /^\d+$/ to reject non-decimal input. Extracting one isDecimalWholeNumber(value) guard would remove the duplication and make the policy — "only plain decimal digits are valid" — live in one place. That also makes it easier to add tests for every vector in limit and skip silently guess at invalid input, and an over-maximum skip returns the same page forever #301 (1e3, 10abc, 250.7, 0x10, etc.) without duplicating the table.

Related: the test file should cover the exact vector list from #301. The implementation rejects them correctly, but __tests__/utils.test.js only asserts a subset, so the contract isn't fully guarded yet.

Cross-link: CenterForDigitalHumanities/TinyNode#132 consumes these Pagination-* headers and should land after this one.