Skip to content

Commit 9669a8f

Browse files
tryakshclaude
tryaksh
and
claude
committed
Twelve sleeping shells killed the overnight campaign; two supervisors replace them
At 06:25 the machine ran out of whatever Git-Bash's fork emulation needs. `queue_noised_skill_cert.sh: fork: retry: Resource temporarily unavailable`, and with it went capture seed 71 at 2,805 of 3,100 epochs, extraction seed 71 at 12,541 of 12,600, and the force-feedback seating policy at 1,400 of 3,000. Not one evaluation stage had started, because the submission gate was queued behind a certification that is not on the critical path. The cause was the structure, not the load. Twelve queue scripts, each parked in an `until ... sleep` loop, each with a child. Stages that must run in order belong in one script as consecutive lines. supervise_evaluation.sh and supervise_training.sh replace all twelve. Eighteen shells became four. The evaluation supervisor runs the RGB-D gate first -- seeds 5070 and 6070 for each arm, with the 4070 archives that survived on disk -- then the datum-pair certificate, then the noised skill certification. The training supervisor resumes the three interrupted runs cheapest-first, sixty epochs then three hundred, and suppresses the PhysX joint warning that had grown the force run's log to 21 MB. Nothing was lost but the hours: every interrupted run resumes from its own last checkpoint, the noised extract checkpoint the gate needs is intact, and all three cohort archives from seed 4070 are still on disk. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
1 parent 6199179 commit 9669a8f

3 files changed

Lines changed: 217 additions & 0 deletions

File tree

‎AGENTS.md‎

Lines changed: 17 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -102,6 +102,23 @@ specification.
102102
than GPU utilization or a single line: a saturated GPU time-slicing four jobs
103103
looks identical to a busy one.
104104

105+
**Twelve sleeping queue shells killed the machine, and that matters more
106+
than the throughput number.** On 2026-09-04 at 06:25 the overnight campaign
107+
died with `fork: retry: Resource temporarily unavailable`, taking three
108+
trainings with it at 95%, 99.5% and 47% of their epochs and starting not one
109+
evaluation stage. The cause was the *structure*, not the load: twelve queue
110+
scripts each parked in an `until ... sleep` loop, each with a child, on
111+
Git-Bash's emulated fork. Stages that must run in order belong in one script
112+
as consecutive lines, not in separate processes waiting on each other's log
113+
files. `supervise_evaluation.sh` and `supervise_training.sh` replace all
114+
twelve with two, and the footprint went from eighteen shells to four.
115+
116+
A second lesson from the same crash: **suppress known-harmless simulator
117+
warnings in long runs.** The force-feedback training's log reached 21 MB,
118+
almost entirely `PhysicsUSD: CreateJoint - found a joint with disjointed body
119+
transforms` written on every reset of every environment, and that run made the
120+
least progress of the three.
121+
105122
Not yet measured: whether a fourth concurrent training still adds. The cheap
106123
way to find out is to median the period when the campaign naturally drops to
107124
one run, rather than stopping work to stage it.

‎artifacts/campaign/supervise_evaluation.sh‎

Lines changed: 139 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,139 @@
1+
#!/usr/bin/env bash
2+
# One sequential evaluation supervisor. Replaces nine waiting queues.
3+
#
4+
# **Why this exists.** The overnight campaign ran twelve queue scripts at once,
5+
# each parked in an `until ... sleep` loop, each with a child shell. At 06:25 the
6+
# machine ran out of whatever Git-Bash's fork emulation needs and everything died
7+
# together: `queue_noised_skill_cert.sh: fork: retry: Resource temporarily
8+
# unavailable`, three trainings gone at 95%, 99.5% and 47% of their epochs, and
9+
# not one evaluation stage started. Eighteen sleeping shells was the design
10+
# defect. Stages that must run in order should be lines in one script, not
11+
# separate processes waiting on each other's log files.
12+
#
13+
# Order is by what the paper needs, not by what was queued first. The RGB-D
14+
# cohorts are the submission gate and spent the night blocked behind a
15+
# certification that is not on the critical path; they go first now.
16+
set -u
17+
cd /d/6axis-space-robotics || exit 1
18+
PY="C:/isaac-sim/python.bat"
19+
ROOT="logs/rl_games/zero_g_blade_insertion_contact"
20+
G="$ROOT/grapple_grasp_l0_seed70_v7m130/nn/last_zero_g_blade_insertion_contact_ep_3100_rew_30.262873.pth"
21+
E="$ROOT/grapple_extract_l0_seed70_v18pin/nn/last_zero_g_blade_insertion_contact_ep_12600_rew_172.70488.pth"
22+
I="$ROOT/grapple_insert_l0_seed70_v13m130/nn/last_zero_g_blade_insertion_contact_ep_8000_rew_-42.01845.pth"
23+
NOISED="$ROOT/grapple_extract_l0_seed70_v19noised/nn/last_zero_g_blade_insertion_contact_ep_14600_rew_166.19054.pth"
24+
25+
SCOPE_1="Simulation only. No result here was produced on real hardware."
26+
SCOPE_2="Every module-state channel the policies and the guard consume is camera-derived unless this report's scope says otherwise."
27+
SCOPE_3="Success requires 0.70 s supported settling, release of both robot-side supports, then a separate 0.70 s rack-only recheck on the disclosed break-rated Rack-to-module load path."
28+
29+
say () { echo "[$(date +%H:%M:%S)] $*"; }
30+
31+
# ---------------------------------------------------------------- the gate
32+
# Seeds 5070 and 6070 for each arm; 4070 survived the crash on disk.
33+
finish_arm () {
34+
tag="$1"; title="$2"; extract="$3"; note="$4"; shift 4
35+
for seed in 5070 6070; do
36+
out="artifacts/campaign/rgbdcohorts/${tag}_seed${seed}"
37+
if [ -f "${out}.npz" ]; then say " $tag seed $seed already on disk; skipping"; continue; fi
38+
"$PY" scripts/run_workflow_demo.py --headless \
39+
--workflow relocate --curriculum_stage 0 \
40+
--task Isaac-ZeroG-Blade-GrappleVisionTwoSlot-Workflow-v0 \
41+
--grasp_checkpoint "$G" --extract_checkpoint "$extract" --insert_checkpoint "$I" \
42+
--num_envs 8 --seed "$seed" --steps 1900 \
43+
--robot_rail_on_relocation --latch_on_release --latch_joint_mode fixed \
44+
--latch_rated_force_n 20000 --latch_rated_torque_nm 1000 \
45+
--latch_position_stiffness_n_per_m 40000 --latch_rotation_stiffness_nm_per_rad 20000 \
46+
--destination_channel_relief_m 0.0046125 --mating_mode compliant --mating_force_cap_n 1000 \
47+
--release_sequence simultaneous --perception_backend fiducial_pnp --rack_retention \
48+
"$@" \
49+
--report "${out}_report.json" --episode_metrics "${out}.npz" \
50+
> "${out}.log" 2>&1
51+
rc=$?
52+
say " $tag seed $seed exit=$rc"
53+
ERR=$(./.venv/Scripts/python.exe -c "import json,sys; print(json.load(open(sys.argv[1])).get('error') or '')" "${out}_report.json" 2>/dev/null)
54+
if [ -n "$ERR" ] || [ ! -f "${out}.npz" ]; then
55+
say " $tag seed $seed FAILED: ${ERR:-no episode metrics}"
56+
return 1
57+
fi
58+
done
59+
./.venv/Scripts/python.exe scripts/aggregate_evaluation.py \
60+
--episodes "artifacts/campaign/rgbdcohorts/${tag}_seed4070.npz" \
61+
"artifacts/campaign/rgbdcohorts/${tag}_seed5070.npz" \
62+
"artifacts/campaign/rgbdcohorts/${tag}_seed6070.npz" \
63+
--output "evidence/workflow_robot_carried_vision_${tag}_certification.json" \
64+
--title "$title" --scope "$SCOPE_1" "$SCOPE_2" "$note" "$SCOPE_3" \
65+
> "artifacts/campaign/rgbdcohorts/aggregate_${tag}.log" 2>&1
66+
rc=$?
67+
say "$tag aggregate exit=$rc"
68+
tail -6 "artifacts/campaign/rgbdcohorts/aggregate_${tag}.log"
69+
}
70+
71+
say "STAGE 1/3 the submission gate: two seeds for each RGB-D arm"
72+
73+
finish_arm noised_extract \
74+
"RGB-D chain with extraction trained on the estimator's error" "$NOISED" \
75+
"One change from the published camera-driven cohort: the extraction policy resumed the certified v18pin checkpoint on a task whose module-derived observations carry the deployed estimator's certified residual, sample-and-hold and miss rate. Capture and the seating controller are unchanged."
76+
77+
finish_arm kinematic_velocity \
78+
"RGB-D chain with the module velocity taken from the robot rather than the cameras" "$E" \
79+
"One change from the published camera-driven cohort, and no retrain: every policy is the published checkpoint, and the module-velocity channel reports zero before capture and the wrist's own velocity after it. That is encoder and forward-kinematics information; no module state is read." \
80+
--module_velocity_source kinematics
81+
82+
finish_arm noised_extract_kinematic_leadin \
83+
"RGB-D chain, best available configuration: trained on the estimator's error, velocity from the robot, guard on the flare's catch" "$NOISED" \
84+
"Three changes from the published camera-driven cohort, each of which is measured alone in its own certification: the retrained extraction policy, the kinematic velocity channel, and the guarded advance admitting on the entry flare's catch rather than on the estimator's own noise bound." \
85+
--module_velocity_source kinematics --fiducial_guard_bounds lead_in
86+
87+
say "STAGE 2/3 the datum-pair perception certificate"
88+
mkdir -p artifacts/campaign/datumpair datasets
89+
"$PY" scripts/collect_grapple_vision.py \
90+
--task Isaac-ZeroG-Blade-GrappleVisionTwoSlot-Collect-v0 \
91+
--output datasets/fiducial_rgbd_datum_pair_seed286.npz \
92+
--samples 1024 --num_envs 16 --seed 286 \
93+
--rgb_source raw --pose_distribution workflow_envelope \
94+
> artifacts/campaign/datumpair/collect.log 2>&1
95+
rc=$?
96+
say "collect exit=$rc"
97+
if [ -f datasets/fiducial_rgbd_datum_pair_seed286.npz ]; then
98+
"$PY" scripts/certify_fiducial_perception.py \
99+
--dataset datasets/fiducial_rgbd_datum_pair_seed286.npz \
100+
--report evidence/fiducial_rgbd_datum_pair_seed286.json \
101+
> artifacts/campaign/datumpair/certify.log 2>&1
102+
rc=$?
103+
say "certify exit=$rc"
104+
tail -12 artifacts/campaign/datumpair/certify.log
105+
fi
106+
107+
say "STAGE 3/3 the noised extraction skill certification"
108+
mkdir -p artifacts/campaign/noisedcert
109+
for arm in noised clean; do
110+
case "$arm" in
111+
noised) TASK="Isaac-ZeroG-Blade-GrapplePin-ExtractNoised-Play-v0" ;;
112+
clean) TASK="Isaac-ZeroG-Blade-GrapplePin-Extract-Play-v0" ;;
113+
esac
114+
rows=()
115+
for seed in 1070 2070 3070; do
116+
for stage in 0 1 2; do
117+
out="artifacts/campaign/noisedcert/${arm}_s${stage}_seed${seed}"
118+
"$PY" scripts/play.py --headless --task "$TASK" --checkpoint "$NOISED" \
119+
--num_envs 128 --episodes 512 --curriculum_stage "$stage" --seed "$seed" \
120+
--episode_metrics "${out}.npz" > "${out}.log" 2>&1
121+
rc=$?
122+
say " $arm stage=$stage seed=$seed exit=$rc"
123+
[ -f "${out}.npz" ] && rows+=("${out}.npz")
124+
done
125+
done
126+
if [ "${#rows[@]}" -gt 0 ]; then
127+
./.venv/Scripts/python.exe scripts/aggregate_evaluation.py --episodes "${rows[@]}" \
128+
--output "evidence/grapple_extract_v19noised_${arm}_certification.json" \
129+
--title "Extraction skill retrained on the estimator's error, scored on the ${arm} task" \
130+
--scope "$SCOPE_1" \
131+
"The checkpoint resumed the certified v18pin weights on a task whose module-derived observations carry the deployed estimator's certified residual, sample-and-hold and miss rate." \
132+
> "artifacts/campaign/noisedcert/aggregate_${arm}.log" 2>&1
133+
rc=$?
134+
say "$arm aggregate exit=$rc"
135+
tail -5 "artifacts/campaign/noisedcert/aggregate_${arm}.log"
136+
fi
137+
done
138+
139+
say "evaluation supervisor done"

‎artifacts/campaign/supervise_training.sh‎

Lines changed: 61 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,61 @@
1+
#!/usr/bin/env bash
2+
# One sequential training supervisor. Replaces three slot queues and their waiters.
3+
#
4+
# The overnight run died at 06:25 with `fork: retry: Resource temporarily
5+
# unavailable` and took three trainings with it -- capture seed 71 at 2,805 of
6+
# 3,100 epochs, extraction seed 71 at 12,541 of 12,600, and the force-feedback
7+
# seating policy at 1,400 of 3,000. All three resume from their own last
8+
# checkpoint, so nothing is lost except the hours.
9+
#
10+
# Two things changed besides the structure.
11+
#
12+
# **One training at a time, beside one evaluation.** Three concurrent trainings
13+
# aggregate more frames per second than one, which is why rule 7 says run three,
14+
# but that measurement was taken on a machine that had not yet fallen over. Until
15+
# the fork exhaustion is understood, throughput is worth less than finishing.
16+
#
17+
# **The force run's log is suppressed.** It reached 21 MB, almost entirely
18+
# `PhysicsUSD: CreateJoint - found a joint with disjointed body transforms`,
19+
# emitted every reset of every environment. That is a known and harmless warning
20+
# about the release latch, and writing it a few million times is not free.
21+
set -u
22+
cd /d/6axis-space-robotics || exit 1
23+
PY="C:/isaac-sim/python.bat"
24+
ROOT="logs/rl_games/zero_g_blade_insertion_contact"
25+
26+
say () { echo "[$(date +%H:%M:%S)] $*"; }
27+
28+
resume () {
29+
run="$1"; task="$2"; seed="$3"; iters="$4"; extra="${5:-}"
30+
ckpt=$(ls -t "$ROOT/$run/nn/"last_*.pth 2>/dev/null | grep -vE '_rew__' | head -1)
31+
if [ -z "$ckpt" ]; then say "$run has no checkpoint to resume; skipping"; return 1; fi
32+
say "resuming $run from $(basename "$ckpt" | grep -oE 'ep_[0-9]+')"
33+
# shellcheck disable=SC2086
34+
"$PY" scripts/train.py --headless --task "$task" \
35+
--num_envs 512 --seed "$seed" --robustness_level 0 --max_iterations "$iters" \
36+
--checkpoint "$ckpt" --run_name "$run" $extra \
37+
> "artifacts/campaign/train_${run}_resumed.log" 2>&1
38+
rc=$?
39+
say "$run exit=$rc, highest epoch now $(ls "$ROOT/$run/nn/"*.pth 2>/dev/null | grep -oE 'ep_[0-9]+' | grep -oE '[0-9]+' | sort -n | tail -1)"
40+
}
41+
42+
# Nearly finished, so cheapest first: sixty epochs and three hundred.
43+
say "STAGE 1/4 extraction seed 71, 59 epochs from the end"
44+
resume grapple_extract_l0_seed71_v18stage "Isaac-ZeroG-Blade-GrapplePin-Extract-v0" 71 12600
45+
46+
say "STAGE 2/4 capture seed 71, 295 epochs from the end"
47+
resume grapple_grasp_l0_seed71_v8scratch "Isaac-ZeroG-Blade-GrapplePin-Grasp-v0" 71 3100
48+
49+
say "STAGE 3/4 the force-feedback seating policy, from 1,400"
50+
resume grapple_insert_l0_seed70_v33force "Isaac-ZeroG-Blade-GrapplePin-InsertForce-v0" 70 3000
51+
52+
say "STAGE 4/4 capture seed 72 from scratch, for the seed spread"
53+
"$PY" scripts/train.py --headless \
54+
--task Isaac-ZeroG-Blade-GrapplePin-Grasp-v0 \
55+
--num_envs 512 --seed 72 --robustness_level 0 --max_iterations 3100 \
56+
--run_name grapple_grasp_l0_seed72_v8scratch \
57+
> artifacts/campaign/train_grasp_seed72_v8scratch.log 2>&1
58+
rc=$?
59+
say "capture seed 72 exit=$rc"
60+
61+
say "training supervisor done"

0 commit comments

Comments
 (0)