Skip to content

Skip tests relying on quay.io, the Tool Sheds, Dockstore and usegalaxy.eu when they are down - #1726

Open
jmchilton wants to merge 1 commit into
galaxyproject:masterfrom
jmchilton:skip-tests-when-external-services-down
Open

jmchilton wants to merge 1 commit into
galaxyproject:masterfrom
jmchilton:skip-tests-when-external-services-down

Conversation

@jmchilton

Copy link
Copy Markdown
Member

Planemo already had one of these — skip_if_zenodo_down, added in 423898f. This applies the
same skip_if_site_down mechanism to the rest of the tests that need a live external service,
mirroring galaxyproject/galaxy#23685 on the Galaxy side.

Motivating failure: a quay.io read timeout failed test_data_manager_docker_mount in CI on an
unrelated branch. Container resolution timed out, so the job ran without the biocontainer,
bowtie2 wasn't there, and the data manager JSON didn't match:

urllib3.exceptions.ReadTimeoutError: HTTPSConnectionPool(host='quay.io', port=443):
    Read timed out. (read timeout=12)
galaxy.tool_util.deps.containers ERROR Could not get container description for tool
    'bowtie2_index_builder_data_manager'
...
galaxy.tool_util.verify INFO: ## files diff ... found diff = 2

New probes

In tests/test_utils.py, beside the existing skip_if_zenodo_down:

skip_if_dockstore_down      = skip_if_site_down("https://dockstore.org/")
skip_if_quay_down           = skip_if_site_down("https://quay.io/")
skip_if_toolshed_down       = skip_if_site_down("https://toolshed.g2.bx.psu.edu/")
skip_if_testtoolshed_down   = skip_if_site_down("https://testtoolshed.g2.bx.psu.edu/")
skip_if_usegalaxy_eu_down   = skip_if_site_down("https://usegalaxy.eu/")

galaxy-util 26.0.1 exports only skip_if_github_down and skip_if_workflowhub_down —
skip_if_quay_down, skip_if_dockstore_down and skip_if_galaxy_depot_down landed in Galaxy
dev with #23685 but aren't in the pinned release yet. So these are defined locally against the
shared skip_if_site_down primitive and can collapse into upstream imports when the pin moves.
The two that do exist upstream are imported from galaxy.util.unittest_utils rather than
re-exported.

Tests decorated

service tests
quay.io test_data_manager_docker_mount, test_workflow_test_repository_installation_gxformat2, test_serve_interactivetool — all --biocontainers
Tool Shed 7 tool id/version linting tests in test_cmd_workflow_lint.py; 5 workflow autoupdate tests (test_autoupdate_workflow_from_multiple_tool_sheds also gets skip_if_testtoolshed_down)
Dockstore test_run_trs_id
github.com test_ok_http (lints a tool over https from raw.githubusercontent.com), test_git_ls_remote
usegalaxy.eu test_training_generate_from_wf_command_remote_wf, test_training_init_command_tutorial_remote_wf

Each new decorator sits innermost, below the existing @skip_if_environ(...) marks, so the env
gate is still a collection-time pytest mark and the network probe only runs if the test would
actually execute.

tests/test_trs_id.py is deliberately untouched — it mocks Dockstore with responses, so it
needs no probe.

Verification

  • All seven probe URLs return HTTP 200 (is_site_up requires exactly 200, so a URL that never
    returns 200 would skip its tests permanently — worth checking, and they're clean).
  • Confirmed the decorator actually skips: pointed one at an unresolvable host and it raised
    SkipTest: Test depends on [...] being up and it appears to be down.
  • Ran the decorated tests that don't need a managed Galaxy — 7 tool shed linting tests plus
    test_git_ls_remote and test_ok_http: 9 passed.
  • Whole suite still collects: 593 tests, no import errors.
  • black / isort / flake8 clean.

Caveat

The probes hit root pages, so they catch a service that is unreachable, not a slow API behind a
working front page. The quay.io failure above was a 12s read timeout on the API while the front
page was presumably fine — so this would not have caught that specific one. It is the same
tradeoff #23685 accepted upstream; a tighter probe would mean per-service API URLs and more
maintenance.

🤖 Generated with Claude Code

…y.eu when down

Planemo already uses galaxy-util's skip_if_site_down for Zenodo. Apply it to the
rest of the tests that need a live external service - a quay.io read timeout just
failed test_data_manager_docker_mount in CI with an unrelated diff.

New probes in tests/test_utils.py next to skip_if_zenodo_down: dockstore, quay,
toolshed, testtoolshed, usegalaxy.eu. galaxy-util 26.0.1 exports only
skip_if_github_down and skip_if_workflowhub_down, so the rest are defined locally
until the pin picks up galaxyproject/galaxy#23685.

Decorated:
- quay.io: --biocontainers tests in test_cmd_test.py and test_cmd_serve.py
- Tool Shed: tool id/version linting in test_cmd_workflow_lint.py, workflow
  autoupdate tests (one also needs the Test Tool Shed)
- Dockstore: test_run_trs_id
- github.com: test_ok_http, test_git_ls_remote
- usegalaxy.eu: the two remote-workflow training tests

Probes hit root pages, so they catch an unreachable service, not a slow API
behind a working front page.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
@jmchilton
jmchilton marked this pull request as ready for review September 24, 2026 20:00
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant