[agent-job-health] Agent Job Health: 2026-10-02 fleet failure rate 11.8% (25/212), novel Avenger + redact-secrets clusters

[github-actions[bot]]

Summary

• Window evaluated: data available spans roughly 2026-10-02 08:24 UTC → 23:34 UTC (~15h). The log-fetch pipeline targets a full trailing 24h (--start-date -1d --count 3000), but ran into its own count/time ceiling before reaching back to 2026-10-01 23:34 UTC — so this report covers the most recent ~15h of the 24h window, not the full 24h. Treat the rates below as representative of the busy tail of the day, not a strict 24h figure.
• total_runs: 213 (212 completed + 1 still in_progress at analysis time — this very Agent Job Health Monitor run — excluded from rate math)
• distinct_workflows: 85
• agent_failures: 25 (agent job specifically; activation/detection/safe-output/other-job failures and skipped agent jobs are excluded per scope)
• Run-weighted fleet failure rate: 25 / 212 = 11.8%
• Workflow-weighted: median 0.0%, mean 16.3% (68 of 85 workflows had zero agent-job failures)
• Tracked vs novel: 8 tracked-by-precedent (32%) / 17 novel (68%) of the 25 failures
• Trend: previous recorded point (2026-09-23) was 17.97% run-weighted — today's 11.8% is a ~6.2pt decrease, not a regression.

Warning

The median-vs-mean gap (0.0% vs 16.3%) confirms the fleet rate is driven by a handful of chronic/novel offenders, not a broad regression. One workflow (Avenger) failed its agent job on 6/6 observed runs (100%), and a distinct "Redact secrets in logs" failure recurs across 6 unrelated workflows — novel and not matched to any open issue found in this session's search (filed as a new issue, see below).

Failure Rate by Step

Failing step (agent job)	Runs	Share of agent_failures
Execute GitHub Copilot CLI	8	32%
Execute Codex CLI	6	24%
Redact secrets in logs (standalone)	5	20%
Redact secrets in logs + Handle agent failure (cascade)	1	4%
Checkout actions folder (cascade → many downstream steps)	1	4%
Parse agent logs for step summary (cascade)	2	8%
Start MCP Gateway	1	4%
Warm up model (Ollama)	1	4%

Tracked Failures

Workflow	Failing runs	Notes
PR Triage Agent	2 (§37047749014, §37007549310)	"Execute GitHub Copilot CLI" step failure
Code Scanning Fixer	2 (§37050855897, §37010266650)	"Execute GitHub Copilot CLI" step failure
Delight	1 (§37022510540)	"Execute GitHub Copilot CLI" step failure
Daily Reliability Review	1 (§37074826241)	"Execute GitHub Copilot CLI" step failure
Matt Pocock Skills Reviewer	1 (§37065011612)	"Execute GitHub Copilot CLI" step failure
Daily SPDD Spec Planner	1 (§37030207112)	"Execute GitHub Copilot CLI" step failure

Caveat on "tracked": every prior recorded window (2026-08-23, 2026-08-28, 2026-09-23) shows this exact "Execute GitHub Copilot CLI step failure" signature as a recurring top cluster, previously linked to issues #54186 / #55413 in earlier windows. search_issues in this session returned 0 directly visible matches for "Copilot CLI segfault"-style queries, with several candidate results filtered out by integrity policy before I could read them to confirm open/closed status. I'm classifying this cluster as tracked by historical precedent, not by a freshly confirmed open issue — please verify #54186/#55413 (or their successors) are still open and still match before assuming this is fully covered.

Novel Failure Clusters

1. Avenger — "Execute Codex CLI" step failure (6 runs, 1 workflow, 100% of Avenger's observed runs)

• Runs: §37061250616, §37067359802, §37048842301, §37073003325, §37027896092, §37013841183
• Engine: Codex. Every single Avenger run in the sample failed at the "Execute Codex CLI" step — a clean, 100%-reproducible single-workflow failure.
• search_issues found nothing visibly open for "avenger codex" (candidates were integrity-filtered).
• Blast radius: single workflow, but total chronicity (100% failure) makes it the single biggest contributor to the fleet rate (24% of all agent failures).
• Does not meet the issue-autocreation bar (needs ≥2 distinct workflows or ≥10% of runs; this is 1 workflow / 2.8% of runs) — flagged here for manual triage instead of auto-filed.

2. "Redact secrets in logs" step failure across 6 unrelated workflows (6 runs, 6 distinct workflows) — issue filed

• Runs: §37009644234 (Workflow Normalizer), §37048684261 (Daily Cache Strategy Analyzer), §37055781123 (Daily Ambient Context Optimizer), §37076914188 (Detection Analysis Report), §37047829177 (Deep Report), §37019914795 (Daily Agent of the Day Blog Writer, which also cascades into "Handle agent failure")
• In each of these runs, the step immediately before it ("Execute <engine> CLI") is recorded as success — only the shared "Redact secrets in logs" post-step fails. This points at a shared pipeline component (the log-redaction step itself), not at any individual workflow's prompt/logic.
• Suspected root cause: a bug or resource limit in the redaction step triggered by something specific to certain log shapes/sizes, independent of engine or workflow — needs a representative log pull from one of the 6 runs to confirm.
• Blast radius: fleet-wide infra component, not workflow-specific — meets the issue-autocreation bar (≥2 distinct workflows). A new issue has been filed for this cluster.

Schedule Heartbeat

The fleet has ~230 active schedule-triggered workflows (from gh-aw status). A full per-workflow list_workflow_runs sweep across all 230 was out of scope for this run's budget. Instead, sub-daily (hourly-or-more-frequent) workflows were cross-checked against the observed-run sample; the one apparent gap (Daily Trajectory Grader Implementer, "every 30 minutes") was individually verified via the GitHub Actions API and found to have run at 2026-10-02 23:31:02 UTC — just outside the edge of this session's observed sample window, not an actual blind spot.

Workflow	Last observed run (verified)	Expected cadence	Gap
Daily Trajectory Grader Implementer	2026-10-02 23:31 UTC	every 30 min	none — false positive from partial sampling window

No blind spots detected among the sub-daily-cadence workflows checked. A full sweep of all ~230 schedule-triggered workflows (including daily/weekly ones) was not completed this run — recommend a follow-up pass with a wider log-fetch budget if deeper coverage is wanted.

View per-workflow breakdown (workflows with ≥1 agent-job failure)

Workflow	Runs observed	Agent failures	Failure rate
Avenger	6	6	100%
PR Triage Agent	2	2	100%
Code Scanning Fixer	2	2	100%
Deep Report	2	1	50%
Matt Pocock Skills Reviewer	7	1	14.3%
PR Code Quality Reviewer	7	1	14.3%
PR Sous Chef	41	2	4.9%
Workflow Normalizer	1	1	100%
Delight	1	1	100%
Daily BYOK Ollama Test	1	1	100%
Daily Reliability Review	1	1	100%
Daily Cache Strategy Analyzer	1	1	100%
Daily Ambient Context Optimizer	1	1	100%
Daily SPDD Spec Planner	1	1	100%
Daily Fact	1	1	100%
Detection Analysis Report	1	1	100%
Daily Agent of the Day Blog Writer	1	1	100%
(remaining 68 of 85 workflows)	—	0	0%

Recommendations

1. Follow up on the new "Redact secrets in logs" issue — a shared-pipeline-step failure across 6 engines/workflows is higher leverage to fix than any single workflow, since fixing it likely clears ~24% of this window's novel failures at once.
2. Triage Avenger's 100% Codex CLI failure. Six-for-six is not noise — pull one full run's Codex CLI step log (e.g. §37013841183) to find the actual error before it becomes invisible background noise in the fleet average. Below the auto-file threshold, so needs a manual issue if it keeps recurring.
3. Reconfirm whether [aw-failures] [P0] Cross-engine agent-CLI segfault (exit 139) — confirmed on Auto-Triage/Pi CLI, now spread to 8 workflows acros #54186/[aw-failures] [aw] Copilot-BYOK auth rejected by api-proxy — Codex 404 (Ponytail Reviewer, Issue Monster) + Claude 401 (Smoke Claude on Copilo #55413 (Copilot CLI step failures) are still open and still match — this signature has now appeared as a top cluster in 4 consecutive measurement windows (08-23, 08-28, 09-23, 10-02) without resolution. If those issues are stale/closed, a fresh issue is warranted.
4. Budget a wider log-fetch window for the next run so Phase 1 and Phase 6 can cover the full trailing 24h instead of the ~15h this run captured, and so all ~230 schedule-triggered workflows can be swept for blind spots rather than just the sub-daily ones.

References:

• §37013841183 — Avenger Codex CLI failure
• §37076914188 — Redact-secrets cluster, Detection Analysis Report
• §37007549310 — tracked Copilot CLI cluster, PR Triage Agent

Generated by 🩺 Agent Job Health Monitor · claude · agent · 807.2 AIC · ⌖ 6.81 AIC · ⊞ 6.8K · ◷

• expires on Oct 5, 2026, 3:53 PM UTC-08:00

[github-actions[bot]]

This discussion was automatically closed because it expired on 2026-10-05T23:53:59.883Z.

Closed by Workflow