PyTorch Red Hat CI

Red Hat's downstream CI for PyTorch, building and testing on RHEL (Red Hat Enterprise Linux). Integrated with PyTorch's upstream CI via Cross-Repository CI Relay (CRCR).

How It Works

pytorch/pytorch
  │
  ├─ PR events ──▶ repository_dispatch (via CRCR) ──▶ rhel96-build-test.yml [disabled]
  │
  ├─ nightly branch ──▶ cron schedule ──▶ crcr-nightly.yml [active, CUDA]
  │                                           │
  │                                           ├─ Extracts source main SHA from nightly commit
  │                                           ├─ Builds PyTorch in RHEL 9.6 CUDA container
  │                                           ├─ Runs delta-based test determination
  │                                           └─ Executes categorized tests (cpu, inductor, sgpu, mgpu)
  │
  └─ nightly SHA ──▶ workflow_dispatch ──▶ crcr-nightly-rocm.yml [manual, ROCm]
                                              │
                                              ├─ Builds PyTorch in RHEL 9.6 ROCm container
                                              ├─ Runs sanity or critical ROCm tests
                                              └─ HUD/CRCR callbacks currently disabled (PUSH_TO_HUD=false)

Platforms

Runner	OS	Accelerator	Status
`linux.rhel96`	RHEL 9.6	CUDA	Active
`linux.rhel96-rocm`	RHEL 9.6	ROCm	Active (manual validation)

Workflows

`crcr-nightly.yml` — Nightly RHEL 9.6 Build & Test (Active)

Runs daily at 08:30 UTC via cron, or manually via workflow_dispatch.

Pipeline: build → determine-tests → cpu-tests → inductor-tests → sgpu-tests → mgpu-tests (sequential)

Manual Dispatch

The workflow can be triggered manually from the Actions tab with two optional inputs:

Input	Description	Default
`sha`	pytorch/pytorch SHA to build against (leave empty for latest nightly)	(empty = latest nightly)
`test_categories`	Which test stages to run after build	`all`

Test category options:

Selection	What runs
`all`	Build + all test stages (same as cron)
`cpu`	Build + determine-tests + CPU tests only
`inductor`	Build + determine-tests + inductor tests only
`sgpu`	Build + determine-tests + single-GPU tests only
`mgpu`	Build + determine-tests + multi-GPU tests only
`build-only`	Build only, skip all tests

Cron-triggered runs always execute all stages regardless of these inputs. The run title displays the selected category (e.g., [Nightly] RHEL 9.6 @ manual [sgpu]).

Build (`linux.rhel96`, 10h timeout)

Fetches the two most recent commits from pytorch/pytorch's nightly branch
Extracts the source main SHA from the nightly release commit message (nightly commits embed the original main SHA in parentheses)
Builds PyTorch from source inside a RHEL 9.6 UBI container using podman build
RHEL subscription credentials are passed via BuildKit --secret mounts (never appear in image layers or docker history)
Produces a tagged container image (rhel9-pytorch-nightly:<sha>) for downstream test jobs
Pushes to Quay.io with a reproducible tag:
```
quay.io/aipcc/pytorch:rhel9_6_pytorch_nightly_main_git<7char_sha>_cuda13_0
```
Tag components: rhel9_6 (OS), nightly (pipeline), main (PyTorch branch), git<sha> (commit), cuda13_0 (CUDA version)

Determine-tests (`linux.rhel96`, 10h timeout)

Computes the diff between the current and previous source SHAs
Runs merge_test_results.py inside the built container (heuristic + structural call graph analysis)
Validates discovered test names against run_test.py's accepted test list
Excludes CUDA-only tests (e.g., test_overrides) from cpu and inductor categories
Outputs base64-encoded, categorized test lists (cpu, inductor, sgpu, mgpu)
Falls back to full test suite (from test_config.py) if delta produces no results

Test Jobs (`linux.rhel96`, 24h job timeout)

Job	Category	GPU Requirement	Per-command timeout	Runs After
`cpu-tests`	CPU-only PyTorch tests	None	2 hours	`determine-tests`
`inductor-tests`	TorchInductor + Dynamo + Export	None	2 hours	`cpu-tests`
`sgpu-tests`	Single-GPU tests	≥ 1 GPU	2 hours	`inductor-tests`
`mgpu-tests`	Multi-GPU + distributed tests	≥ 2 GPUs	12 hours	`sgpu-tests`

Each test job:

Mounts the command list as a file into the container (-v /tmp/<job>_test_commands.txt:/tmp/test_commands.txt:ro) — avoids shell quoting issues with bash -c argument passing
Writes each command to /tmp/_run.sh and executes via bash /tmp/_run.sh — preserves -k filter quoting (e.g., -k "TestA or TestB") that would otherwise be mangled by nested eval
Uses single-quoted bash -c '...' for the outer podman shell — eliminates escape gymnastics
Wraps each command with timeout to prevent individual hangs from blocking the pipeline (2 hours for cpu/inductor/sgpu, 12 hours for mgpu)
Runs with CONTINUE_THROUGH_ERROR=True, collecting pass/fail counts and printing a :::SUMMARY::: block
Streams output in real-time via tee (no buffering)
Reports accurate job status: a final "Fail job if tests failed" step checks the test step's outcome and exits with code 1 if there were failures, ensuring the job conclusion is failure despite continue-on-error: true on the test step

`crcr-nightly-rocm.yml` — RHEL 9.6 ROCm Build & Test (Manual)

Triggered only via workflow_dispatch while the linux.rhel96-rocm (MI355X / gfx950) runner and image are being validated. Cron and HUD reporting will be enabled after manual soak.

Pipeline: rocm-build → determine-tests → inductor-tests → sgpu-tests → mgpu-tests

Same category split as CUDA (scripts/test_config.py), sized for a multi-GPU MI355X host.

Manual Dispatch

Input	Description	Default
`sha`	pytorch/pytorch SHA to build against (leave empty for latest nightly)	(empty = latest nightly)
`test_tier`	`sanity` / `critical` lists from `test_config.py`, or `build-only`	`critical`
`test_categories`	`all` / `inductor` / `sgpu` / `mgpu`	`all`
`no_cache`	Force `podman --no-cache` full rebuild (keep on until the ROCm image is validated)	`true`

Selection	What runs
`sanity` + `all`	Build + short sanity lists for inductor, sgpu, mgpu
`critical` + `all`	Build + critical inductor / sgpu / mgpu suites (default)
`critical` + `sgpu`	Build + sgpu only (skips inductor/mgpu jobs)
`build-only`	Build only, skip tests

ROCm Build (`linux.rhel96-rocm`, 10h timeout)

Resolves the source main SHA from pytorch/pytorch nightly (or uses the manual sha input)
Builds PyTorch from source with USE_ROCM=1 / USE_CUDA=0 via docker/Dockerfile.rhel9-rocm
Pins ROCm 7.14.0 via amdgpu-install / rocmradeon/el9/26.13 (classic rocm/el9/7.14* 404s)
Defaults to --no-cache so a green build is a real compile (set no_cache=false later for faster rebuilds)
Verifies the image can import torch with a non-empty torch.version.hip before push

Pushes to Quay with tag:

quay.io/aipcc/pytorch:rhel9_6_pytorch_nightly_main_git<7char_sha>_rocm7_14_0

Determine-tests (`linux.rhel96-rocm`)

Resolves inductor / sgpu / mgpu command lists from scripts/test_config.py (--sanity or --critical)
No delta/heuristic path yet (full tier lists every run)

ROCm Test Jobs (`linux.rhel96-rocm`, 24h timeout each)

Job	Category	GPU requirement	Per-command timeout
`inductor-tests`	TorchInductor	optional (≥1 for GPU paths)	2h
`sgpu-tests`	Single-GPU (`HIP_VISIBLE_DEVICES=0`)	≥ 1	2h (+ quick sanity gate)
`mgpu-tests`	Multi-GPU / distributed (RCCL via `test_c10d_nccl`)	≥ 2	12h

Shared behavior:

Podman: --ipc=host + /dev/kfd + /dev/dri (no --shm-size)
CONTINUE_THROUGH_ERROR=True; summaries report completed with failures (no hard job fail on suite failures)
HUD/CRCR callbacks are disabled (PUSH_TO_HUD=false) until the pipeline is manually validated

`rhel96-build-test.yml` — PR Build & Sanity Tests (Disabled)

Triggered by CRCR repository_dispatch (pull_request type). Currently disabled (.disabled suffix) while the nightly workflow is being stabilized. Will be re-enabled once nightly results are consistently stable.

Build job:

Checks out pytorch/pytorch at the dispatched SHA
Builds PyTorch from source on RHEL

Sanity test job:

import torch verification
Tensor ops and dtype checks
Autograd backward pass
Model serialization round-trip
Core test_torch.py smoke tests

CRCR Integration Level

Currently at L2 — nightly builds and tests run daily, with results reported back to the PyTorch HUD via the CRCR callback action. Each pipeline stage (build, cpu, inductor, sgpu, mgpu) reports its conclusion individually, giving per-job visibility on the HUD dashboard.

HUD Reporting

Each job in the nightly pipeline sends a completed callback to the PyTorch CRCR relay with event-type: nightly and delivery-id set to the resolved pytorch/pytorch source SHA. The following job names appear on HUD:

Job	HUD `job-name`
build	`linux-rhel9.6-cuda13.0-py3.12-gcc11-x86_64 / build`
cpu-tests	`linux-rhel9.6-cuda13.0-py3.12-gcc11-x86_64 / test (cpu, linux.rhel96)`
inductor-tests	`linux-rhel9.6-cuda13.0-py3.12-gcc11-x86_64 / test (inductor, linux.rhel96)`
sgpu-tests	`linux-rhel9.6-cuda13.0-py3.12-gcc11-x86_64 / test (sgpu, linux.rhel96)`
mgpu-tests	`linux-rhel9.6-cuda13.0-py3.12-gcc11-x86_64 / test (mgpu, linux.rhel96)`

GPU test jobs only report to CRCR when GPUs are actually available on the runner — skipped tests are not reported, avoiding misleading success entries on HUD.

ROCm jobs (rocm-build, rocm-tests) intentionally do not report to HUD yet: PUSH_TO_HUD=false in crcr-nightly-rocm.yml. Flip that flag once the ROCm pipeline has been manually validated.

External Results Relay

The results relay accepts submissions from GitHub Actions repositories whose OIDC tokens are listed in config/rhel_allowlist.yml. Submission authorization and HUD forwarding are separate: the Lambda determines forward_to_hud from this file, so a sender cannot opt itself into HUD.

Use forward_to_hud: false while onboarding or validating a new partner. Its results are received and shown in the receiver workflow, but are not forwarded to CRCR/HUD. Enable forwarding only after the partner is ready:

allowed_repos:
  - repo: partner-org/partner-repo
    forward_to_hud: false

L2 Promotion Criteria

This repo was promoted to L2 after meeting the following criteria from RFC-0050:

Criterion	Status
Nightly callback success rate ≥ 80%	Met
Results visible on PyTorch HUD	Met
Timeout rate < 1%	Met
Active for ≥ 1 month	Met

Test Determination

The nightly workflow uses a dual-strategy approach for delta-based test selection:

Tool	Strategy	Best For
`targeted_tests.py`	File-path heuristic mapping	Python file changes, test moves
`structural_tests.py`	C++ call graph + binding analysis	C++ kernel/op changes
`merge_test_results.py`	Union of both + dedup	Combined coverage
`test_config.py`	Static full suite (fallback)	When delta produces nothing

The unified merger (merge_test_results.py) runs both tools and deduplicates results. If the structural analyzer is not installed or its index is unavailable, the system gracefully falls back to heuristic-only mode.

Test commands are validated against run_test.py's accepted test list before execution to filter out invalid entries. Tests that unconditionally require a CUDA driver (e.g., test_overrides) are excluded from CPU and Inductor categories.

Critical Tests

A fixed set of critical tests always runs regardless of what the delta determines. These cover core subsystem health and are prepended to the delta results (deduplicated):

Category	Critical Tests
cpu	`test_torch`, `test_autograd`, `test_linalg`, `test_sparse`, `test_unary_ufuncs`, `test_binary_ufuncs`
inductor	`inductor/test_torchinductor`, `inductor/test_cpu_repro`
sgpu	`test_nn`, `test_torch`, `test_cuda`, `test_ops`, `test_unary_ufuncs`, `test_binary_ufuncs`, `test_autograd`
mgpu	`distributed/test_c10d_common`, `distributed/test_c10d_nccl`, `distributed/test_distributed_spawn`

To list critical tests for a category: python scripts/test_config.py cpu --critical --commands-only

Environment variables:

STRUCTURAL_ANALYSIS_DEPTH — Override call graph walk depth (default: 3)
CONTINUE_THROUGH_ERROR — Set to True inside containers; allows all tests to run even if some fail

Directory Structure

.github/workflows/
  crcr-nightly.yml              # Active CUDA nightly pipeline
  crcr-nightly-rocm.yml         # Manual ROCm build/test (HUD disabled)
  rhel96-build-test.yml.disabled # PR workflow (disabled)

docker/
  Dockerfile.rhel9              # RHEL 9.6 UBI build image (conda, CUDA, PyTorch from source)
  Dockerfile.rhel9-rocm         # RHEL 9.6 UBI build image (conda, ROCm, PyTorch from source)

scripts/
  merge_test_results.py         # Unified test merger (heuristic + structural)
  targeted_tests.py             # File-path heuristic test selector
  torchtalk_tests.py            # Structural call graph analyzer wrapper
  test_config.py                # Static categorized test suites (full-suite fallback)
  requirements-structural.txt   # Dependencies for structural analysis

Prerequisites

The linux.rhel96 (CUDA) and/or linux.rhel96-rocm (ROCm) self-hosted runners must be registered and online
podman must be available on the runner for container-based builds
This repo must be on the CRCR allowlist to receive dispatches:
```
L2:
  - TorchedHat/pytorch-redhat-ci
```
For GPU test jobs, the runner must have NVIDIA GPUs with drivers installed

Secrets

Secret	Used By	Purpose
`RHEL_SUBSCRIPTION_ACTIVATION_KEY`	Build (Dockerfile)	RHEL subscription for `dnf` access
`RHEL_SUBSCRIPTION_ORG_ID`	Build (Dockerfile)	RHEL org ID for subscription-manager
`QUAY_USERNAME`	Build (push step)	Quay.io robot account or username
`QUAY_PASSWORD`	Build (push step)	Quay.io password or token

Navigation Menu

Repository files navigation

PyTorch Red Hat CI

How It Works

Platforms

Workflows

`crcr-nightly.yml` — Nightly RHEL 9.6 Build & Test (Active)

Manual Dispatch

Build (`linux.rhel96`, 10h timeout)

Determine-tests (`linux.rhel96`, 10h timeout)

Test Jobs (`linux.rhel96`, 24h job timeout)

`crcr-nightly-rocm.yml` — RHEL 9.6 ROCm Build & Test (Manual)

Manual Dispatch

ROCm Build (`linux.rhel96-rocm`, 10h timeout)

Determine-tests (`linux.rhel96-rocm`)

ROCm Test Jobs (`linux.rhel96-rocm`, 24h timeout each)

`rhel96-build-test.yml` — PR Build & Sanity Tests (Disabled)

CRCR Integration Level

HUD Reporting

External Results Relay

L2 Promotion Criteria

Test Determination

Critical Tests

Directory Structure

Prerequisites

Secrets

Related Resources

About

Used by

Contributors

Languages

Name		Name	Last commit message	Last commit date
.github/workflows		.github/workflows
aws/lambda/results_relay		aws/lambda/results_relay
config		config
docker		docker
scripts		scripts
README.md		README.md

Latest commit

History

Folders and files

Repository files navigation

PyTorch Red Hat CI

How It Works

Platforms

Workflows

crcr-nightly.yml — Nightly RHEL 9.6 Build & Test (Active)

Manual Dispatch

Build (linux.rhel96, 10h timeout)

Determine-tests (linux.rhel96, 10h timeout)

Test Jobs (linux.rhel96, 24h job timeout)

crcr-nightly-rocm.yml — RHEL 9.6 ROCm Build & Test (Manual)

Manual Dispatch

ROCm Build (linux.rhel96-rocm, 10h timeout)

Determine-tests (linux.rhel96-rocm)

ROCm Test Jobs (linux.rhel96-rocm, 24h timeout each)

rhel96-build-test.yml — PR Build & Sanity Tests (Disabled)

CRCR Integration Level

HUD Reporting

External Results Relay

L2 Promotion Criteria

Test Determination

Critical Tests

Directory Structure

Prerequisites

Secrets

Related Resources

About

Resources

Stars

Watchers

Forks

Used by

Contributors

Languages

`crcr-nightly.yml` — Nightly RHEL 9.6 Build & Test (Active)

Build (`linux.rhel96`, 10h timeout)

Determine-tests (`linux.rhel96`, 10h timeout)

Test Jobs (`linux.rhel96`, 24h job timeout)

`crcr-nightly-rocm.yml` — RHEL 9.6 ROCm Build & Test (Manual)

ROCm Build (`linux.rhel96-rocm`, 10h timeout)

Determine-tests (`linux.rhel96-rocm`)

ROCm Test Jobs (`linux.rhel96-rocm`, 24h timeout each)

`rhel96-build-test.yml` — PR Build & Sanity Tests (Disabled)