hack/metrics: add ci-failure-analysis script

[Aditya Shantanu (aditya-shantanu)]

Summary

Adds hack/metrics/ci-failure-analysis.py, a script that samples recent failed pr-workflow CI runs, fetches their job logs, and classifies each failure by root cause rather than treating all CI failures as equivalent.

Failure categories:

• named_test_fail — a specific Go test emitted --- FAIL: TestName
• no_free_workers — envtest/integration tests hit worker resource contention
• e2e_timeout — e2e tests timed out waiting for actor/service responses (503s, connection resets)
• gcs_access — GCS/S3 bucket access denied when fetching sandbox assets
• license_check — hack/verify/licenses.sh detected uncommitted LICENSES changes
• verify_fail — other hack/verify-all.sh step failed
• unknown — could not classify from log output

Usage:

python hack/metrics/ci-failure-analysis.py \
  --repo agent-substrate/substrate \
  --fetch-limit 500 --sample 40

# Save for later comparison
python hack/metrics/ci-failure-analysis.py \
  --repo agent-substrate/substrate \
  --fetch-limit 500 --sample 40 \
  --save ci-breakdown.json

Sample output (40 sampled runs, Aug 2026):

Named Go test failure  50%  ████████████████
No free workers        30%  ██████████
E2e timeout / 503       5%  █
Unclassified           15%  █████

Named test failures:
  6x  TestDurableDirLifecycle
  4x  TestActorLifecycle
  2x  TestMultipleDurableDirLifecycle
  2x  TestSyncer_UpdateWorker_RetryOnVersionConflict
  2x  TestLoaderConcurrentHandshakes

This informed the timeout fixes in #800 and is intended to feed the planned scheduled CI remediation agent.

Checklist

• Issue is linked above (see Fix flaky tests: e2e demo suite timeouts and syncer poll timeout #799)
• Tests pass locally (go test ./...)
• Documentation updated if behavior changed (script is new, no existing docs affected)