Guard author lists and publisher URLs against Scholar regressions - #58
Merged
Conversation
The first unattended run under the new merge logic (9a7b0bf) walked two fields backwards, both in places the existing guards didn't reach. URL: the container-title guard stops a published paper being relabelled as a preprint, but the link is a separate field, and Scholar replaced schreiner2026synaptic's Nature URL with a bioRxiv abstract while the venue correctly stayed "Nature". Reject a preprint-host URL when it would replace a publisher URL — but only for entries we believe are published. An entry whose venue is still bioRxiv genuinely is a preprint, and its bioRxiv link is the canonical one; guarding those would freeze the link at whichever mirror was seen first. wei2026dynamic is exactly that case, so its bioRxiv URL is left alone. Matching is on the URL host, not the whole string, so a publisher URL with "arxiv" somewhere in the path isn't caught. author: Scholar returned draelos2020online with Eva Naumann dropped, and an unconditional overwrite took it. Losing a co-author is worse than being slow to add one, so only accept a list at least as long as the stored one. Also repair the two entries the run damaged: restore the Nature URL (verified 200) and re-add Naumann in her original position. Left the escholarship URL change alone — publisher to publisher, not a regression — and wei2026dynamic's bioRxiv URL, which is correct for a preprint. Verified by replaying the exact payload from 9a7b0bf against the repaired data: neither field is damaged now, and both guards log why. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_013CB1BSCz8YKpK4YyZiN81C
jmxpearson
marked this pull request as ready for review
August 1, 2026 22:51
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The first unattended scraper run under the new merge logic (
9a7b0bf) walked two fields backwards. Data integrity held — 92 entries, no duplicates, manual overrides intact — but two guards were missing.What regressed
schreiner2026synapticnature.com/articles/s41586-026-10510-xbiorxiv.org/…699515.abstractdraelos2020onlineThe venue guard held in both cases —
schreiner2026synapticstill correctly reads "Nature". The link and the author list were simply never guarded.Fixes
URL — reject a preprint-host URL that would replace a publisher URL, but only for entries we believe are published. An entry whose venue is still bioRxiv genuinely is a preprint and its bioRxiv link is canonical; guarding those would freeze the link at whichever mirror was seen first.
wei2026dynamicis exactly that case and is deliberately left alone. Matching is on the URL host, so a publisher URL with "arxiv" in the path isn't caught.author — only accept a list at least as long as the stored one. Losing a co-author is worse than being slow to add one.
Data repair
Restored the Nature URL (verified HTTP 200) and re-added Naumann in her original position. Left alone: the escholarship URL change (publisher → publisher, not a regression) and
wei2026dynamic's bioRxiv URL.Verification
Replayed the exact payload from
9a7b0bfagainst the repaired data — neither field is damaged now, and both guards log their reasoning. Full suite passes: merge, selection, badge overrides, manual CCN metadata, and the new guard tests, plus a no-op reconcile that rewrites 0 ids and leaves 0 duplicates.🤖 Generated with Claude Code