🐛 CI failure: JET launcher killed with exit 137 after GPT-583M inference tests pass

[balasaajay]

Describe the bug

Six GPT-583M dynamic-inference CI jobs are marked failed even though their final child GPU workloads succeeded and the downloaded test artifacts show successful inference/validation. The parent launch_jet_workload.py process is killed with exit code 137 after downloading the workload assets, before replaying the test logs.

This is a post-test launcher failure, not an observed CUDA runtime error or golden-value mismatch in these six jobs. Updating their goldens will not address the launcher termination.

@NVIDIA/mcore-oncall

Failing run

• PR: #6991 — chore: Update base image to 26.08
• Tested commit: a4d622074ccb990aaa4598923cec4c998774ea11
• Observed: September 10, 2026; platform dev_dgx_h100.
• Internal CI pipeline ID: 67130833. Internal CI links are intentionally omitted.

Parent job ID	Affected test recipe
433441346	gpt_dynamic_inference_tp1_pp1_583m_chunked_prefill_cuda_graphs
433441347	gpt_dynamic_inference_tp1_pp1_583m_cuda_graphs_fp8_logitsmatch
433441353	gpt_dynamic_inference_tp1_pp1_583m_prefix_caching_chunked_prefill_cuda_graphs
433441355	gpt_dynamic_inference_tp1_pp1_583m_prefix_caching_cuda_graphs
433441368	gpt_dynamic_inference_tp2_pp2_583m_cuda_graphs
433441370	gpt_dynamic_inference_tp2_pp2_583m_prefix_caching_cuda_graphs

Error

The parent job reports the Python launcher process as Killed, followed by:

ERROR: Job failed: command terminated with exit code 137

Evidence distinguishing this from a test failure:

• All six final child GPU workloads succeeded; downloaded rank-0 exit artifacts are 0.
• The prefix-caching and TP2/PP2 jobs (433441353, 433441355, 433441368, 433441370) each contain five successful inference executions and five passing test_inference_pipeline checks (1 passed).
• The chunked-prefill job (433441346) also has explicit passing inference validation in its successful replacement child workload; an earlier child launch failed before producing test logs.
• Parent termination occurs after asset downloads, before the normal main-log replay. There is no Python traceback establishing why the launcher was killed.

Steps/Code to reproduce bug

Observed in the normal JET-backed functional CI workflow at the tested commit:

1. Run one of the affected recipes on dev_dgx_h100 through tests/test_utils/python_scripts/launch_jet_workload.py.
2. Allow the child GPU workload and its inference validation to finish.
3. Observe the parent launcher while it downloads and processes the workload logs/artifacts.
4. Compare the successful child status and exit artifacts with the parent's exit code 137.

A minimal deterministic reproduction has not yet been established. Running the pytest validation alone does not exercise the failing parent log-handling path.

Expected behavior

The parent launcher should finish processing the logs and report the successful workload result without being killed.

Additional context

• Host-memory pressure during log aggregation is a hypothesis, not a confirmed OOM. Exit code 137 alone does not establish the source of SIGKILL; runner/container memory counters or OOM events are needed.
• At the tested commit, extract_torchrunlogs_to_string loads per-rank logs using readlines(). The post-download path also loads the main log and builds concatenated log strings. This is a relevant area to measure, not a proven fault location.
• Suggested investigation: capture launcher peak RSS and runner/container termination details, correlate them with downloaded log sizes, and assess whether log processing can be streamed or bounded.
• The decode-only throughput failure and optimized-SwiGLU log-probability mismatch are separate failures and are not included in this issue.
• No assignee was selected: the latest change to the first affected recipe on main was authored by the svcnemo-autobot automation account.
• Triaged using the mcore-create-issue skill.