Skip to content

Commit 55b19f8

Browse files
thodson-usgsclaude
andcommitted
feat(waterdata): name every column the OGC collections return
Comparing each collection's /schema with its getter's signature found 20 returned columns that were reachable only through **queryables, so the getter documented neither the column nor the filter: - get_field_measurements: control_condition, day, field_measurements_series_id, measurement_rated, month, reading_type, time_of_day, year - get_peaks: qualifier, time_of_day, value - get_monitoring_locations: revision_created, revision_modified, revision_note - get_combined_metadata: data_gap_interval, reading_type - get_time_series_metadata: data_gap_interval, parameter_description - get_field_measurements_metadata: reading_type - get_channel: channel_location_direction Each is now a named parameter, described in the service's own words. day, month and year take the integer annotation get_peaks already uses. The monitoring-location attributes every collection accepts as filters but does not return stay in **queryables. Existing calls send the same request as before. Stacked: this commit also carries DOI-USGS#422 (Water Data API v1), DOI-USGS#423 (continuous method_category and the API-version monitor) and DOI-USGS#424 (the documented-columns monitor), which merge first. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
1 parent 75e56ab commit 55b19f8

26 files changed

Lines changed: 1060 additions & 185 deletions

‎NEWS.md‎

Lines changed: 8 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1,3 +1,11 @@
1+
**09/22/2026:** Twenty columns the Water Data OGC collections return are now named, documented parameters on the getter that returns them; before, they were reachable only through `**queryables`. `get_field_measurements()`: `control_condition`, `day`, `field_measurements_series_id`, `measurement_rated`, `month`, `reading_type`, `time_of_day`, `year`. `get_peaks()`: `qualifier`, `time_of_day`, `value`. `get_monitoring_locations()`: `revision_created`, `revision_modified`, `revision_note`. `get_combined_metadata()`: `data_gap_interval`, `reading_type`. `get_time_series_metadata()`: `data_gap_interval`, `parameter_description`. `get_field_measurements_metadata()`: `reading_type`. `get_channel()`: `channel_location_direction`. Existing calls are unaffected: these names are sent to the service exactly as they were through `**queryables`.
2+
3+
**09/22/2026:** The Water Data OGC getters now request **v1** of the Water Data APIs (`api.waterdata.usgs.gov/ogcapi/v1`), [released September 2026](https://waterdata.usgs.gov/blog/api-v1-release/); v0 stays online until June 2027. **New:** `WaterdataConfiguration(api_version="v0")`, or `api_version = "v0"` in the `[waterdata]` table of the configuration file, pins the version for the transition. It replaces only the version segment of the OGC path; the Samples, Statistics and STAC families are versioned separately and have no v1. **Behavior change:** `waterdata.get_time_series_metadata()` returns `begin` and `end` in UTC with a time zone, and no longer returns `begin_utc`, `end_utc`, `state_name` or `hydrologic_unit_code`. **Deprecation:** passing `begin_utc`, `end_utc`, `state`, `state_name` or `hydrologic_unit_code` to that getter, as a filter or in `properties`, emits a `DeprecationWarning` and sends the call to v0, which serves them until June 2027; the shim may be removed on or after 2027-06-01. Use `begin`, `end`, and `get_combined_metadata()` instead. **Behavior change:** `waterdata.get_field_measurements()` returns `time` as a date rather than a datetime, parsed to a tz-naive midnight timestamp as `get_daily()` already does; the time of day is in `time_of_day`. The `field-measurements-metadata` collection has no `time` field and is unaffected.
4+
5+
**09/22/2026:** New live test: each getter that lists its returned columns in its `properties` docstring (`get_daily`, `get_continuous`, `get_time_series_metadata`) is checked against its collection's schema, so a field USGS adds or removes fails the nightly run instead of leaving the docstring wrong.
6+
7+
**09/22/2026:** `waterdata.get_continuous()` accepts `method_category`, which the `continuous` collection began publishing in September 2026: the RLMS method category code (`STNRD`, `LMTUS`, `EXPER` or `UNKWN`) for the method in effect over an observation's interval. It is returned on every record and is null for time series that have not been categorized. It was already reachable through `**queryables`; it is now a documented parameter. `get_latest_continuous()` is unchanged, because `latest-continuous` does not publish the field.
8+
19
**09/09/2026:** **Bug fix:** code and identifier columns keep their leading zeros. A bare `pandas.read_csv` infers a zero-padded code as a number, so `waterdata.get_samples()` returned parameter code `00060` as `60` and HUC12 `070700050502` as `70700050502`, and `nwis.get_info()` returned `huc_cd` `02060005` as `2060005`. One rule now decides what a code column is — a name ending in `code`, the RDB abbreviation `_cd`, or a name containing `identifier`, `huc`, or `fips` — and every delimited response is parsed through it: the Samples and WQP CSV readers, `rdb.read_rdb` (which reads the names from the RDB header rather than the caller listing them), and the Water Use CSV pages. **Behavior change:** these columns now hold strings. `waterdata.get_samples()`: `USGSpcode`, `Location_HUCEightDigitCode`, `Location_HUCTwelveDigitCode`, `SampleCollectionMethod_Identifier` (`get_samples_summary()` shares the parse; no column in its current profile was affected). `nwis.get_info()`, `nwis.what_sites()`, and `nwis.get_record(service="site")`: `huc_cd`, `state_cd`, `county_cd`, `district_cd`. A comparison against a number — `df["USGSpcode"] == 60` — or a merge onto a numeric key now matches nothing instead of raising, so compare against the padded string (`== "00060"`) or call `.astype(int)` where the number is what you want. **Behavior change:** a count whose name reads as an identifier is numeric again. WQP's `AlternateLocation_IdentifierCount` has been read as text since 05/31/2026 because "Identifier" appears in its name; a name ending in `count` is now excluded from the rule, so the same column has one dtype in every service that reports it. Measurement columns are unchanged, and the `waterdata` OGC getters and `ngwmn` were never affected: their JSON responses deliver codes as strings and numeric coercion there is limited to a fixed list of measurement columns. **Correction to the 1.2.0 notes:** the same fix was applied to the nine `wqp` getters on 05/31/2026 and never recorded here — `wqp.get_results()` and the `what_*` getters have returned HUCs, parameter codes, and FIPS codes as strings since that release.
210

311
**09/01/2026:** **Announcement:** We at USGS Water Data for the Nation want your feedback! Tell us how we're doing by taking our quick [survey](https://usgswaterresources.gov1.qualtrics.com/jfe/form/SV_07gX8G1DeOtVrH8), available through September 2026.

‎dataretrieval/_configuration_core.py‎

Lines changed: 76 additions & 26 deletions
Original file line numberDiff line numberDiff line change
@@ -11,6 +11,7 @@
1111

1212
import math
1313
import os
14+
import re
1415
import stat
1516
import sys
1617
import warnings
@@ -26,11 +27,17 @@
2627
from dataretrieval.exceptions import ConfigurationError
2728

2829
#: Settings only an adapter can hold, because they name one service. No
29-
#: package-wide value could mean anything for them: there is no one base URL.
30+
#: package-wide value could mean anything for them: there is no one base URL,
31+
#: and a version is a segment of one service's paths.
3032
#:
3133
#: The package-wide roster is :data:`SETTINGS`, declared below the class it is
3234
#: derived from.
33-
ADAPTER_ONLY_SETTINGS: tuple[str, ...] = ("base_url",)
35+
ADAPTER_ONLY_SETTINGS: tuple[str, ...] = ("base_url", "api_version")
36+
37+
#: The adapter-only settings the file refuses too, leaving a ``configure()``
38+
#: block as the only source that can supply one (ADR 0011). Named for the
39+
#: *source*, the term the chain spells ``_BLOCK`` everywhere else.
40+
BLOCK_ONLY_SETTINGS: tuple[str, ...] = ("base_url",)
3441

3542
#: Environment variable backing a setting (precedence step 2).
3643
#:
@@ -52,12 +59,9 @@
5259

5360
#: Variables the environment is *refused* for, by setting. Named rather than left out of
5461
#: :data:`ENV_VARS`, so a caller who exports ``API_USGS_BASE_URL`` gets an error instead
55-
#: of an ignored variable. The file refuses the same key in the same words
56-
#: (:func:`_accepted_keys`): a base URL set outside the code could redirect the library
57-
#: to another host without a reader of the script seeing it (ADR 0011).
58-
#:
59-
#: Derived from :data:`ADAPTER_ONLY_SETTINGS` so the file and the environment
60-
#: cannot drift apart on which settings are code-only.
62+
#: of an ignored variable. Derived from :data:`ADAPTER_ONLY_SETTINGS`, because a
63+
#: variable applies to every adapter and those settings name one. The file
64+
#: refuses only :data:`BLOCK_ONLY_SETTINGS` (ADR 0011).
6165
_REFUSED_ENV_VARS: dict[str, str] = {
6266
name: f"API_USGS_{name.upper()}" for name in ADAPTER_ONLY_SETTINGS
6367
}
@@ -393,9 +397,7 @@ def _provenance(self) -> str:
393397
# Plain mixins rather than ``BaseConfiguration`` subclasses: a group has no
394398
# adapter and cannot be passed to :func:`configure`, so keeping it off that
395399
# branch leaves one linear base for the behavior. Frozen because a dataclass
396-
# may not mix frozen and non-frozen bases; fields collect in reverse MRO order,
397-
# so an adapter composing all four reads ``retries, stall_timeout, base_url,
398-
# concurrency, parallel_chunks``.
400+
# may not mix frozen and non-frozen bases. Fields collect in reverse MRO order.
399401

400402

401403
@dataclass(frozen=True)
@@ -413,6 +415,13 @@ class _Redirectable:
413415
base_url: str | None = _UNSET
414416

415417

418+
@dataclass(frozen=True)
419+
class _Versioned:
420+
"""An adapter whose service publishes its API under a version path segment."""
421+
422+
api_version: str | None = _UNSET
423+
424+
416425
@dataclass(frozen=True)
417426
class _Concurrent:
418427
"""An adapter that issues more than one request per call."""
@@ -771,6 +780,7 @@ def _coerce_count(value: object, label: str, optional: str) -> str:
771780
#: so the wider check cannot change a TOML outcome.)
772781
_TYPES: dict[str, Callable[[object, str, str], str]] = {
773782
"api_key": _coerce_string,
783+
"api_version": _coerce_string,
774784
"base_url": _coerce_string,
775785
"progress": _coerce_progress,
776786
"concurrency": _coerce_concurrency,
@@ -899,6 +909,26 @@ def _parse_base_url(raw: str, label: str) -> str:
899909
return value
900910

901911

912+
#: The one shape a version takes in every Water Data path: ``v`` and digits.
913+
_API_VERSION_RE = re.compile(r"^v\d+$")
914+
915+
916+
def _parse_api_version(raw: str, label: str) -> str:
917+
"""Parse an API version: the segment the service publishes it under, ``v1``.
918+
919+
Checked as a shape, not against a list. This module cannot know which
920+
versions a service has published, and a closed list here would refuse a
921+
version the service already serves until a release of this package named it.
922+
"""
923+
value = raw.strip()
924+
if not _API_VERSION_RE.match(value):
925+
raise ConfigurationError(
926+
f"{label} must be the version segment of the service's path, "
927+
f"such as 'v1' (got {raw!r})."
928+
)
929+
return value
930+
931+
902932
def _parse_progress(raw: str, label: str, *, strict: bool) -> bool:
903933
"""Parse a progress toggle, optionally preserving legacy env truthiness."""
904934
value = raw.strip().lower()
@@ -931,6 +961,7 @@ def _parse_progress(raw: str, label: str, *, strict: bool) -> bool:
931961
"parallel_chunks": _parse_parallel_chunks,
932962
"stall_timeout": _parse_seconds,
933963
"base_url": _parse_base_url,
964+
"api_version": _parse_api_version,
934965
}
935966

936967

@@ -1048,11 +1079,13 @@ def _adapter_file_settings(
10481079

10491080
where = f"[{adapter}]"
10501081
# An adapter this process has not imported declares no vocabulary, so its
1051-
# table is checked against the package-wide settings alone: refusing a key
1052-
# for want of a schema would make the file's validity depend on which
1053-
# optional extras happened to be installed.
1082+
# table is checked against every setting this release has a grammar for:
1083+
# refusing a key for want of a schema would make the file's validity depend
1084+
# on which optional extras happened to be installed.
10541085
accepted = settings_for(adapter)
1055-
validated = _scalars(table, path, where, SETTINGS if accepted is None else accepted)
1086+
validated = _scalars(
1087+
table, path, where, _ALL_SETTINGS if accepted is None else accepted
1088+
)
10561089
label = f"{path} {where}"
10571090
result: Mapping[str, tuple[str, str]] = MappingProxyType(
10581091
{name: (value, label) for name, value in validated.items()}
@@ -1230,7 +1263,7 @@ def _accepted_keys(
12301263
# and :func:`_named_profile` refuses a table inside a profile, so a
12311264
# sub-table here is always a profile rather than deeper nesting.
12321265
continue
1233-
if key in ADAPTER_ONLY_SETTINGS:
1266+
if key in BLOCK_ONLY_SETTINGS:
12341267
# Rejected from the file wherever it appears. A file that
12351268
# redirects a data-retrieval library to another host is a
12361269
# supply-chain hazard; an in-code block keeps the redirect
@@ -1240,16 +1273,7 @@ def _accepted_keys(
12401273
"configure() block, never from a file."
12411274
)
12421275
if key not in allowed:
1243-
if key in SETTINGS:
1244-
# A real setting, in a table that does not read it. Unlike an
1245-
# unrecognized name -- which may belong to a newer release --
1246-
# this cannot become meaningful later, and ignoring it without an error
1247-
# would leave a caller believing they had tuned something. See
1248-
# ADR 0010.
1249-
raise ConfigurationError(
1250-
f"{path}: {key!r} at {where} is not a setting that table "
1251-
f"accepts. It accepts: {', '.join(sorted(allowed))}."
1252-
)
1276+
_reject_known_setting(key, path, where, allowed)
12531277
warnings.warn(
12541278
f"{path}: unknown setting {key!r} at {where} (ignored). "
12551279
f"Known settings: {', '.join(SETTINGS)}.",
@@ -1261,6 +1285,32 @@ def _accepted_keys(
12611285
return out
12621286

12631287

1288+
def _reject_known_setting(
1289+
key: str, path: Path, where: str, allowed: frozenset[str] | tuple[str, ...]
1290+
) -> None:
1291+
"""Raise if *key* is a setting this release knows but that table cannot use.
1292+
1293+
Returns for a name this release does not know, which the caller warns about
1294+
instead: only one of the two can be a typo for a newer release, and a real
1295+
setting in a table that does not read it cannot become meaningful later --
1296+
ignoring it without an error would leave a caller believing they had tuned
1297+
something (ADR 0010).
1298+
"""
1299+
if where == _TOP_LEVEL and key in ADAPTER_ONLY_SETTINGS:
1300+
# The generic message below would list only top-level settings, none of
1301+
# which is the one to write.
1302+
raise ConfigurationError(
1303+
f"{path}: {key!r} at {where} names one service and has no "
1304+
"package-wide value; set it in the table of the adapter it belongs "
1305+
"to, such as [waterdata]."
1306+
)
1307+
if key in _ALL_SETTINGS:
1308+
raise ConfigurationError(
1309+
f"{path}: {key!r} at {where} is not a setting that table "
1310+
f"accepts. It accepts: {', '.join(sorted(allowed))}."
1311+
)
1312+
1313+
12641314
def _checked_table(
12651315
table: dict[str, Any],
12661316
path: Path,

‎dataretrieval/_deprecation.py‎

Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -19,6 +19,10 @@
1919
"waterdata.get_cql(service=)": "2027-08-09",
2020
"wateruse": "2027-08-11",
2121
"ogc.interruptions": "2027-08-25",
22+
# Set by the service, not by this package: v0 of the collection serves these
23+
# filters until June 2027, and the shim cannot outlive the endpoint it sends
24+
# to (https://waterdata.usgs.gov/blog/api-v1-release/).
25+
"waterdata.get_time_series_metadata(v0 filters)": "2027-06-01",
2226
}
2327

2428

‎dataretrieval/configuration.py‎

Lines changed: 62 additions & 8 deletions
Original file line numberDiff line numberDiff line change
@@ -72,6 +72,7 @@
7272
# isort: off
7373
from dataretrieval._configuration_core import (
7474
ADAPTERS as ADAPTERS,
75+
BLOCK_ONLY_SETTINGS as BLOCK_ONLY_SETTINGS,
7576
CONFIG_PATH_ENV as CONFIG_PATH_ENV,
7677
CONCURRENCY_UNBOUNDED as CONCURRENCY_UNBOUNDED,
7778
DEFAULT_CONCURRENCY as DEFAULT_CONCURRENCY,
@@ -94,6 +95,7 @@
9495
_Frame as _Frame,
9596
_named_profiles as _named_profiles,
9697
_NO_FILE as _NO_FILE,
98+
_parse_api_version as _parse_api_version,
9799
_parse_base_url as _parse_base_url,
98100
_parse_concurrency as _parse_concurrency,
99101
_parse_parallel_chunks as _parse_parallel_chunks,
@@ -112,6 +114,7 @@
112114
_SettingValue as _SettingValue,
113115
_UNSET as _UNSET,
114116
_validated_raw as _validated_raw,
117+
_Versioned as _Versioned,
115118
config_path as config_path,
116119
settings_for as settings_for,
117120
)
@@ -655,6 +658,45 @@ def base_url(*, adapter: str | None = None, default: str | None = None) -> str |
655658
return _parse_base_url(raw, label)
656659

657660

661+
@overload
662+
def api_version(*, adapter: str | None = ...) -> str | None: ...
663+
664+
665+
@overload
666+
def api_version(*, adapter: str | None = ..., default: str) -> str: ...
667+
668+
669+
def api_version(
670+
*, adapter: str | None = None, default: str | None = None
671+
) -> str | None:
672+
"""An adapter's configured API version, falling back to *default*.
673+
674+
Settable from code and from the file: an adapter configuration may include
675+
it, and so may that adapter's table. The environment refuses it
676+
(:data:`_REFUSED_ENV_VARS`), as it refuses every adapter-only setting -- a
677+
variable is package-wide, and a version names one service's paths.
678+
679+
Like :func:`base_url`, there is no package-wide default: the version an
680+
adapter's requests use is the adapter's own, so the service passes its own
681+
-- ``api_version(adapter="waterdata", default=OGC_API_VERSION)`` -- and the
682+
version stays declared in the module that builds the path.
683+
684+
Parameters
685+
----------
686+
adapter : str, optional
687+
Whose version to resolve.
688+
default : str, optional
689+
The version the service is written against, returned when nothing
690+
configured one. Omitted, ``None`` is returned -- which is what
691+
:func:`show_configuration` passes, since it has no service default to
692+
supply.
693+
"""
694+
raw, label, _source = _resolve("api_version", adapter)
695+
if raw is None:
696+
return default
697+
return _parse_api_version(raw, label)
698+
699+
658700
# --- resolution ----------------------------------------------------------
659701

660702
#: Which source of the chain supplied a resolution. Machine-readable so a
@@ -729,16 +771,27 @@ def _check_env_not_refused(name: str) -> None:
729771
Refused before anything is consulted, not when the chain reaches the environment
730772
source. The file and the environment refuse ``base_url`` as one rule (ADR 0011), so
731773
a variable that cannot work is not outranked, with no error, by a block that happens
732-
to work.
774+
to work. ``api_version`` is refused from the environment alone: the file may
775+
set it, in an adapter's table, and the message says so.
733776
"""
734777
refused = _REFUSED_ENV_VARS.get(name)
735-
if refused is not None and refused in os.environ:
736-
raise ConfigurationError(
737-
f"{_env_label(refused)} is set, but {name!r} may only be set "
738-
"in code, in a configure() block, never from the environment. Unset "
739-
f"it and pass the value on the adapter's configuration, e.g. "
740-
f"WaterdataConfiguration({name}=...)."
741-
)
778+
if refused is None or refused not in os.environ:
779+
return
780+
block_only = name in BLOCK_ONLY_SETTINGS
781+
fault = (
782+
"may only be set in code, in a configure() block, never from the environment"
783+
if block_only
784+
else "names one service and has no package-wide value, so the environment "
785+
"cannot set it"
786+
)
787+
# The file is a source for everything except the block-only settings, so it
788+
# is offered as an alternative to exactly the settings it can supply.
789+
or_the_file = "" if block_only else ", or in that adapter's table of the file"
790+
raise ConfigurationError(
791+
f"{_env_label(refused)} is set, but {name!r} {fault}. Unset it and pass "
792+
f"the value on the adapter's configuration, e.g. "
793+
f"WaterdataConfiguration({name}=...){or_the_file}."
794+
)
742795

743796

744797
def _resolve_from_block(
@@ -837,6 +890,7 @@ def _display_progress(_adapter: str | None = None) -> str:
837890
"parallel_chunks": lambda adapter: str(parallel_chunks(adapter=adapter)),
838891
"stall_timeout": lambda adapter: f"{stall_timeout(adapter=adapter):g}s",
839892
"base_url": lambda adapter: base_url(adapter=adapter) or "<service default>",
893+
"api_version": lambda adapter: api_version(adapter=adapter) or "<service default>",
840894
}
841895

842896
if set(_DISPLAYS) != set(_ALL_SETTINGS): # pragma: no cover - guards a coding error

0 commit comments

Comments
 (0)