Skip to content

Make --test_data resolve tool test inputs - #1725

Open
jmchilton wants to merge 1 commit into
galaxyproject:masterfrom
jmchilton:test-data-option-fix
Open

jmchilton wants to merge 1 commit into
galaxyproject:masterfrom
jmchilton:test-data-option-fix

Conversation

@jmchilton

Copy link
Copy Markdown
Member

Fixes #1536 — --test_data was inert for test input resolution.

What was wrong

Two independent gaps, either one alone enough to break the option.

1. The Galaxy config key does not exist. planemo/galaxy/config.py wrote

test_data_dir=test_data_dir,  # TODO: make gx respect this

Galaxy has no test_data_dir option — grep -rn 'test_data_dir\b' lib/galaxy/config/ finds nothing. The real key is tool_test_data_directories, read at lib/galaxy/config/__init__.py:1020 and wired to TestDataResolver at lib/galaxy/app/__init__.py:988. The TODO was accurate; Galaxy was never going to respect a key it doesn't have.

2. The client-side fallback was never given the directory. Test inputs resolve in two stages (galaxy/tool_util/verify/interactor.py:528-579): first GET /api/tools/{id}/test_data_download, then a local fallback over self.test_data_directories, populated from a test_data kwarg at :256. galaxy-tool-util's own runner passes it (verify/script.py:426); planemo's galaxy_interactor_kwds did not.

Both are planemo-side. Galaxy and galaxy-tool-util already provide the hooks.

Reproduction

Before, against a tool whose input lives outside its directory:

AssertionError: Test input file (planemo_external_test_data.txt) cannot be found.
  .../galaxy/tool_util/verify/interactor.py:579, via remote_to_input -> _get_path_or_location

which is simonbray's report verbatim.

Test

tests/test_cmd_test.py::CmdTestTestCase::test_test_data_option — an integration test, not a unit test: it stands up Galaxy, uploads, and runs the tool.

The fixture matters. The input is named planemo_external_test_data.txt and kept in tests/data/external_test_data/, deliberately not a test-data/ directory. A stock filename would be served by TestDataResolver's default galaxy-test-data.git resolver and the test would pass without the fix.

Verified red-to-green, and each fix isolated:

result
neither fix fails, at interactor.py:579
config key only passes
interactor kwarg only passes
both (this PR) passes

Also re-ran test_tool_in_directory and test_test_index, which exercise the no---test_data path — both still pass, since an unset option leaves the key None and Galaxy falls back to its defaults.

Open question for review

Each fix is independently sufficient, so this is belt-and-braces. Worth deciding whether both belong.

The one behavioural tradeoff is in the config key. TestDataResolver splits its value on commas and defaults to test-data,https://github.com/galaxyproject/galaxy-test-data.git, so setting tool_test_data_directories replaces those defaults — including the remote resolver. I read an explicit --test_data as "use this", so overriding seems right, and the tool's own directory is still searched first server-side. But it would change behaviour for someone passing --test_data while relying on stock Galaxy test data. Alternatives:

  • keep as-is (override)
  • append instead: f"{test_data_dir},test-data", though that hardcodes part of an upstream default
  • drop the config change and ship only the interactor kwarg, which is purely additive

Happy to go whichever way. Draft until that's settled.

🤖 Generated with Claude Code

Two independent gaps, either alone enough to break the option:

- galaxy config wrote `test_data_dir`, which is not a Galaxy option at
  all; the real key is `tool_test_data_directories`, feeding
  TestDataResolver. The existing TODO admitted as much.
- interactor kwds omitted `test_data`, so the client-side fallback in
  `_find_in_test_data_directories` had nothing to search.
  galaxy-tool-util's own runner passes it.

Integration test covers a tool whose input lives outside its directory,
under a name no stock resolver can supply.

Fixes galaxyproject#1536.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
@jmchilton
jmchilton marked this pull request as ready for review September 23, 2026 17:42
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

--test_data option of planemo t does not work

1 participant