Skip to content

CI verifier's 300 s poll cannot fit inside task_manager's 300 s subprocess timeout #270

Description

@TagoreZhao

tasks/github/standard/mcpmark-cicd/linting_ci_workflow/verify.py:288 waits for
GitHub Actions runs to complete:

start_time = time.time()
timeout = 300
while time.time() - start_time < timeout:
    ...
    time.sleep(10)

while src/base/task_manager.py:261 runs that verifier as

subprocess.run(self._get_verification_command(task), capture_output=True,
               text=True, timeout=300)

The two budgets are equal, so once the poll loop actually needs its full 300 s the
subprocess is killed before the loop exits — the checks after the loop, which are
the ones that produce the verdict, never run. The result is a TimeoutExpired
recorded as a verification error instead of a pass or a fail.

This looks like an unintended consequence of #255, which raised this verifier's
workflow-completion wait from 90 s to 300 s ("for cold GH runners + npm ci + glob").
The change reached main in the #264 release, and the diff on this file is exactly
- timeout = 90 / + timeout = 300. At 90 s the loop always finished well
inside the subprocess budget; at 300 s it exactly meets it, so the headroom is gone.

Related but not a fix: open PR #223 adds an explicit except subprocess.TimeoutExpired
around this call, reporting "Verification script timed out after 300s". That makes the
failure legible, but it keeps timeout=300, so the run still ends without a verdict.

It only bites when the awaited CI state does not settle quickly: in our runs 22 of
25 verified in 5–43 s, and 3 stalled and were killed at ~305 s, losing their grade.

Neither number is configurable — the 300 in run_verification is a literal,
repeated across six task managers, and github uses the base class's.

Two ways to close it, either is fine:

  • lower the verifier's own poll budget so it always fits (e.g. 240 s), or
  • raise run_verification's timeout above the longest verifier poll, ideally
    deriving one from the other rather than repeating a constant.

Happy to send a PR for whichever you prefer.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions