Two tests hard-code short timeouts around a spawned child process that must re-import genvarloader before it can satisfy the assertion. On a slow or loaded filesystem the import alone exceeds the timeout, so the tests fail (or hang) for reasons unrelated to the code under test.
Affected
tests/unit/test_shm_layout.py::test_dense_cross_process — q.get(timeout=10), child target _child_read is module-level, so spawn re-imports the test module and transitively all of genvarloader.
tests/unit/test_producer.py::test_producer_exception_pushed_to_queue — p.join(timeout=15) then assert not p.is_alive().
Measurement (shared HPC node, package on a network filesystem):
PARENT import genvarloader: 105.38s # cold page cache
PARENT import genvarloader: 15.28s # warm
Even warm, import genvarloader is ~15s — above test_dense_cross_process's 10s budget and level with test_producer_exception_pushed_to_queue's 15s. The child pays that cost again because spawn does not inherit the parent's imported modules.
Observed behavior
Same commit, same dedicated 32-CPU allocation, differing only in node load:
| Node load |
Result |
| quiet |
full suite 1110 passed, 56 skipped, 4 xfailed, 0 failed in 6m05s |
| loadavg ~30 |
these 2 fail (_queue.Empty, assert not True), 3/3 reproducible |
| loadavg ~64 |
these 2 never complete — killed at a 38-minute limit |
The failure modes are always transport-shaped (_queue.Empty, child still alive) and never a data or assertion mismatch, which is what makes this diagnosable as latency rather than logic.
Why it matters
It costs real debugging time: these look exactly like a regression in the shared-memory layout or the producer's error path, and were initially misattributed twice — once to "pre-existing node flakiness" and once to an unrelated dependency bump — before being traced to import latency. Anyone running the suite outside CI on a busy or network-backed machine will hit it.
Possible fixes (not prescribing one)
- Scale the timeouts to a measured import cost, or take them from an env var with a generous default.
- Have the child signal readiness after import and start the clock there, so the budget covers only the work under test.
- Poll for a condition with a long ceiling instead of a fixed sleep/timeout.
- Move the child target to a tiny module that does not import
genvarloader where the test does not need it.
Not urgent — CI is green and this is environmental — but it is a real fragility in the tests, not in the code they cover.
Two tests hard-code short timeouts around a
spawned child process that must re-importgenvarloaderbefore it can satisfy the assertion. On a slow or loaded filesystem the import alone exceeds the timeout, so the tests fail (or hang) for reasons unrelated to the code under test.Affected
tests/unit/test_shm_layout.py::test_dense_cross_process—q.get(timeout=10), child target_child_readis module-level, sospawnre-imports the test module and transitively all ofgenvarloader.tests/unit/test_producer.py::test_producer_exception_pushed_to_queue—p.join(timeout=15)thenassert not p.is_alive().Measurement (shared HPC node, package on a network filesystem):
Even warm,
import genvarloaderis ~15s — abovetest_dense_cross_process's 10s budget and level withtest_producer_exception_pushed_to_queue's 15s. The child pays that cost again becausespawndoes not inherit the parent's imported modules.Observed behavior
Same commit, same dedicated 32-CPU allocation, differing only in node load:
1110 passed, 56 skipped, 4 xfailed, 0 failed in 6m05s_queue.Empty,assert not True), 3/3 reproducibleThe failure modes are always transport-shaped (
_queue.Empty, child still alive) and never a data or assertion mismatch, which is what makes this diagnosable as latency rather than logic.Why it matters
It costs real debugging time: these look exactly like a regression in the shared-memory layout or the producer's error path, and were initially misattributed twice — once to "pre-existing node flakiness" and once to an unrelated dependency bump — before being traced to import latency. Anyone running the suite outside CI on a busy or network-backed machine will hit it.
Possible fixes (not prescribing one)
genvarloaderwhere the test does not need it.Not urgent — CI is green and this is environmental — but it is a real fragility in the tests, not in the code they cover.