Skip to content

Flaky: spawn-child tests time out when import genvarloader is slower than their hard-coded budget #332

Description

@d-laub

Two tests hard-code short timeouts around a spawned child process that must re-import genvarloader before it can satisfy the assertion. On a slow or loaded filesystem the import alone exceeds the timeout, so the tests fail (or hang) for reasons unrelated to the code under test.

Affected

  • tests/unit/test_shm_layout.py::test_dense_cross_processq.get(timeout=10), child target _child_read is module-level, so spawn re-imports the test module and transitively all of genvarloader.
  • tests/unit/test_producer.py::test_producer_exception_pushed_to_queuep.join(timeout=15) then assert not p.is_alive().

Measurement (shared HPC node, package on a network filesystem):

PARENT import genvarloader: 105.38s   # cold page cache
PARENT import genvarloader:  15.28s   # warm

Even warm, import genvarloader is ~15s — above test_dense_cross_process's 10s budget and level with test_producer_exception_pushed_to_queue's 15s. The child pays that cost again because spawn does not inherit the parent's imported modules.

Observed behavior

Same commit, same dedicated 32-CPU allocation, differing only in node load:

Node load Result
quiet full suite 1110 passed, 56 skipped, 4 xfailed, 0 failed in 6m05s
loadavg ~30 these 2 fail (_queue.Empty, assert not True), 3/3 reproducible
loadavg ~64 these 2 never complete — killed at a 38-minute limit

The failure modes are always transport-shaped (_queue.Empty, child still alive) and never a data or assertion mismatch, which is what makes this diagnosable as latency rather than logic.

Why it matters

It costs real debugging time: these look exactly like a regression in the shared-memory layout or the producer's error path, and were initially misattributed twice — once to "pre-existing node flakiness" and once to an unrelated dependency bump — before being traced to import latency. Anyone running the suite outside CI on a busy or network-backed machine will hit it.

Possible fixes (not prescribing one)

  • Scale the timeouts to a measured import cost, or take them from an env var with a generous default.
  • Have the child signal readiness after import and start the clock there, so the budget covers only the work under test.
  • Poll for a condition with a long ceiling instead of a fixed sleep/timeout.
  • Move the child target to a tiny module that does not import genvarloader where the test does not need it.

Not urgent — CI is green and this is environmental — but it is a real fragility in the tests, not in the code they cover.

Metadata

Metadata

Assignees

No one assigned

    Labels

    type: bugSomething isn't working

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions