Skip to content

test_harvest.sh §7 flakes intermittently with git plumbing errors ("could not parse HEAD", "could not read <sha>") #94

Description

@moodmosaic

Summary

tests/test_harvest.sh §7 ("Behavioural: harvest.sh --dry survives >20 commits") fails intermittently in ./tests/test.sh --unit with git plumbing errors that don't reproduce on a single-test re-run:

=== 7. Behavioural: harvest.sh --dry survives >20 commits ===
fatal: could not parse HEAD

or:

=== 7. Behavioural: harvest.sh --dry survives >20 commits ===
error: Could not read c8329c769513da7943f88d476e57ffbc63a02ebc
error: could not parse commit c8329c76...
fatal: the remote end hung up unexpectedly

Re-running ./tests/test.sh --unit (or ./tests/test_harvest.sh in isolation) passes 20/20. Five consecutive runs after a failure have all been clean.

Impact

Pre-merge gating: a CI run that hits this looks like a real regression when it isn't. Currently mitigated by re-running, but that erodes trust in the suite.

Likely root cause

§7 is by far the most I/O-heavy unit test: 100 --allow-empty commits with ~1 KiB subject padding, in a scratch repo under $TMPDIR, then git push to a scratch bare under /tmp.

Both error patterns are git's "I expected this ref/object to be readable and it wasn't" signature, which lines up with one of:

  1. Auto-GC race. 100 loose objects approaches git's gc.auto threshold (default 6700 loose objects, but the trigger also depends on pack-refs timing). An auto-GC kicked off mid-§7 by an earlier test in the same suite could pack-then-prune under the test's feet.
  2. Host config interference. test_harvest.sh does not export GIT_CONFIG_GLOBAL=/dev/null / GIT_CONFIG_SYSTEM=/dev/null (the isolation we added to test_launch.sh and test_session_end_push.sh in Pin Codex CLI, fix signing, isolate test gitconfig #91), so any host-side core.hooksPath, gc.*, maintenance.*, or pack.* config leaks in.
  3. Filesystem hiccup on /tmp. Less actionable, but plausible on tmpfs/btrfs under load.

Proposed mitigations (not a fix yet — just lower flake rate)

Two cheap, additive changes worth trying first:

--- a/tests/test_harvest.sh
+++ b/tests/test_harvest.sh
@@ -1,5 +1,9 @@
 #!/bin/bash
 set -euo pipefail
+
+# Isolate from host gitconfig (gc.*, maintenance.*, hooks, etc.).
+export GIT_CONFIG_GLOBAL=/dev/null
+export GIT_CONFIG_SYSTEM=/dev/null

and inside §7's scratch repos:

@@ -198,6 +198,7 @@
 git clone -q "$HARVEST_BARE" "$HARVEST_AGENT"
 cd "$HARVEST_AGENT"
+git config gc.auto 0
 git checkout -q agent-work

If the flake survives both, the next step is GIT_TRACE=1 ./tests/test_harvest.sh on a failing run to see what git was actually doing when it lost the HEAD ref.

Repro

Not consistently reproducible. Observed during runs of ./tests/test.sh --unit on the master branch and on fix/codex-at-capacity-retriable (PR for #85), with two distinct error wordings on consecutive failures. Five clean re-runs in a row immediately after both failures.

Version

0.20.12 (and 0.20.13-pending on the codex-at-capacity branch).

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions