test(waterdata): assert documented columns match the collection schema - #424
Draft
thodson-usgs wants to merge 3 commits into
Draft
thodson-usgs wants to merge 3 commits into
thodson-usgs wants to merge 3 commits into
Conversation
thodson-usgs
force-pushed
the
test/documented-properties-monitor
branch
from
September 23, 2026 02:32
d72f7f4 to
55e13d4
Compare
thodson-usgs
added a commit
to thodson-usgs/dataretrieval-python
that referenced
this pull request
Sep 23, 2026
Comparing each collection's /schema with its getter's signature found 20 returned columns that were reachable only through **queryables, so the getter documented neither the column nor the filter: - get_field_measurements: control_condition, day, field_measurements_series_id, measurement_rated, month, reading_type, time_of_day, year - get_peaks: qualifier, time_of_day, value - get_monitoring_locations: revision_created, revision_modified, revision_note - get_combined_metadata: data_gap_interval, reading_type - get_time_series_metadata: data_gap_interval, parameter_description - get_field_measurements_metadata: reading_type - get_channel: channel_location_direction Each is now a named parameter, described in the service's own words. day, month and year take the integer annotation get_peaks already uses. The monitoring-location attributes every collection accepts as filters but does not return stay in **queryables. Existing calls send the same request as before. Stacked: this commit also carries DOI-USGS#422 (Water Data API v1), DOI-USGS#423 (continuous method_category and the API-version monitor) and DOI-USGS#424 (the documented-columns monitor), which merge first. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
thodson-usgs
force-pushed
the
test/documented-properties-monitor
branch
from
September 23, 2026 02:36
55e13d4 to
4f81930
Compare
…n setting The Water Data OGC collections moved to v1 in September 2026; v0 stays online until June 2027. Every OGC request now goes to /ogcapi/v1. Statistics and STAC stay on v0, because neither has a v1. The version is a setting, like base_url, not a getter parameter. WaterdataConfiguration gains api_version, which can be set in a configure() block or in the [waterdata] table of the configuration file, but not from the environment. The file now refuses the settings in BLOCK_ONLY_SETTINGS rather than all of ADAPTER_ONLY_SETTINGS: base_url can send requests to another host, and a version cannot. v1 drops four time-series-metadata queryables (begin_utc, end_utc, state_name, hydrologic_unit_code) and answers 400 or 500 to them. get_time_series_metadata sends a call that names any of them (state resolves to state_name) to v0 and emits a DeprecationWarning; the removal date is 2027-06-01, when the service retires v0. The v0 URL is set per request, through an api_version argument to ogc_api_url() and get_ogc_data(), rather than by entering configure(), so it never overrides a version the caller set. Two behavior changes, both in NEWS: time-series-metadata begin/end are UTC with a time zone, and get_field_measurements returns time as a date. v1 also adds statistics_begin to time-series-metadata. The queryables snapshot and the time-series-metadata fixture are regenerated from v1. Closes DOI-USGS#421. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
USGS added method_category to the continuous collection in September 2026, which failed the nightly queryables snapshot test. It is the RLMS method category code (STNRD, LMTUS, EXPER, UNKWN). It is returned on every record and is null for time series that have not been categorized. latest-continuous does not have it. - Regenerate waterdata_queryables.json and the continuous fixture. - Add method_category to get_continuous's signature, its docstring, and its list of returned columns. - Add live tests for the API version each Water Data family serves. The OGC and STAC versions are read from the self link in each root document, because OGC answers 200 for any version segment. Statistics has no root document, so the next version is probed. A second test checks that the version the package requests still returns data. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
get_daily, get_continuous and get_time_series_metadata list their returned columns in the properties docstring. The lists are written by hand and went out of date twice without any test failing: continuous was missing method_category (DOI-USGS#423), and time-series-metadata was missing data_gap_interval (DOI-USGS#422). Add a live test that compares each list with the collection's /schema and names the docstring to edit when they differ. id is excluded from both sides: only time-series-metadata lists it in its schema, though all three collections accept it. Stacked on DOI-USGS#422 and DOI-USGS#423, which merge first; review this commit alone. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
thodson-usgs
added a commit
to thodson-usgs/dataretrieval-python
that referenced
this pull request
Sep 24, 2026
Comparing each collection's /schema with its getter's signature found 20 returned columns that could be passed only through **queryables, so the getter documented neither the column nor the filter: - get_field_measurements: control_condition, day, field_measurements_series_id, measurement_rated, month, reading_type, time_of_day, year - get_peaks: qualifier, time_of_day, value - get_monitoring_locations: revision_created, revision_modified, revision_note - get_combined_metadata: data_gap_interval, reading_type - get_time_series_metadata: data_gap_interval, parameter_description - get_field_measurements_metadata: reading_type - get_channel: channel_location_direction Each is now a named parameter, described in the service's own words. day, month and year are typed as integers, as get_peaks already types them. The monitoring-location attributes that every collection accepts as filters but does not return stay in **queryables. Existing calls send the same request as before. Stacked on DOI-USGS#422, DOI-USGS#423 and DOI-USGS#424, which merge first; review this commit alone. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
thodson-usgs
force-pushed
the
test/documented-properties-monitor
branch
from
September 24, 2026 14:29
4f81930 to
016025a
Compare
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Stacked on #422 and #423. This PR's single commit also contains their changes. Review
tests/waterdata_properties_test.py, theNEWS.mdentry and thetests/contracts/README.mdline; the rest belongs to #422 and #423 and will drop out of the diff once they merge.Why
get_daily,get_continuousandget_time_series_metadatalist their returned columns in thepropertiesdocstring. The list is hand-written, and it had gone stale twice without anything noticing:continuousmethod_categorytime-series-metadatadata_gap_intervalThe queryables monitor catches new filters, but nothing checks the documented columns.
Change
tests/waterdata_properties_test.pyis a live test that compares each getter's list with the collection's/schema. When they differ, it names the docstring to edit. The getter-to-collection table is written out rather than discovered, so a getter that loses its list fails the test instead of being skipped.idis excluded on both sides, because onlytime-series-metadatalists it in its schema even though all three accept it.This test detects drift; it does not generate the list. Generating it would make the docs build depend on the live service and would not update
help()or IDE tooltips, where these lists are read.Verification
continuousandtime-series-metadataand passed ondaily.🤖 Generated with Claude Code