Skip to content

Fix silently skipped collection output assertions in workflow tests - #1723

Open
jmchilton wants to merge 2 commits into
galaxyproject:masterfrom
jmchilton:collection-output-assertions
Open

jmchilton wants to merge 2 commits into
galaxyproject:masterfrom
jmchilton:collection-output-assertions

Conversation

@jmchilton

@jmchilton jmchilton commented Sep 22, 2026 •

Copy link
Copy Markdown
Member

Summary

Fixes #1705.

Two bugs in planemo/test/_check_output.py, both of which silently skip a collection output assertion:

  • for_collections() matched only element_tests. Now also matches class: Collection, which is the only gate under which a collection assertion carrying no per-element tests — collection_type or element_count alone — is expressible. On master such an output matches nothing, falls through to the file checker, and fails with an unrelated "No path specified for expected output file".
  • _check_output_collection() used TestCollectionOutputDef.from_dict(), which discards top-level collection_type, so the collection type was never asserted. Switched to from_yaml_test_format(), which maps it onto attributes["type"] and resolves the elements alias.

The same predicate gates planemo/runnable.py:402, so this fixes both call sites.

Deliberately not matching a bare elements: key, though Galaxy's framework-test gate does. It is documented in docs/test_format.rst only for collection inputs, no IWC workflow uses it for an output, and on its own it does not declare a collection — it stays an alias resolved once the output is known to be one. That leaves bare elements: as the single remaining spelling the two tools disagree on.

No dependency bump needed — from_yaml_test_format has been in galaxy-tool-util since 25.1, the floor of the current pin.

Testing

New tests/test_check_output.py, 9 pure unit tests, no Galaxy server. 5 fail against current master, including a list assertion against a list:paired collection reporting zero problems, and a class: Collection + element_count output that nothing detected at all. All 9 pass after. Black/isort/flake8 clean.

🤖 Generated with Claude Code

jmchilton and others added 2 commits September 22, 2026 00:19
for_collections() only matched element_tests, so an output spelled
class: Collection + elements: fell through to the file checker. And
from_dict() drops top-level collection_type, so the type was never
asserted - from_yaml_test_format() maps it onto attributes["type"].

Fixes galaxyproject#1705

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Only `element_tests` and `class: Collection` declare a collection output.
`elements` stays an alias resolved by from_yaml_test_format once the output
is known to be a collection. Add coverage for the class-only spelling
(element_count with no element_tests), which nothing detected before.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
@jmchilton
jmchilton marked this pull request as ready for review September 22, 2026 17:23
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Workflow test collection assertions are silently skipped unless spelled element_tests

1 participant