Measure the capability envelope and correct two claims it disproved

Certification only showed the policy works inside its training distribution.
Two sweeps push the promoted Level-2 policy past that range, one axis at a
time, 500 episodes per point at full reset distance.

Initial pose error is the binding axis. Success falls 100% at 1-2x trained
reset noise, 97.0% at 3x, 87.8% at 4x, 62.4% at 6x, 42.6% at 8x, 21.2% at 12x,
so roughly double the trained noise is absorbed for free and the half-success
point is near 7x. Every point, including 12x, had zero instability and zero
non-finite terminations: the policy stops completing insertions rather than
diverging.

Two claims did not survive contact with the data, and both are corrected in
place rather than dropped:

The predicted failure mode was wrong. Margin analysis said orientation would
break first because it consumed 97.8% of its tolerance. The measured failure is
lateral divergence, with terminal lateral error p95 jumping from 0.0 mm at 3x to
60.6 mm at 4x and tripping the 60 mm predicate. Orientation rises in failing
episodes but is a symptom, not the termination.

The Level-2 blade-mass claim is close to vacuous. Sweeping mass to 1-50 kg, more
than triple the trained range, is still 100% with nearly identical terminal
metrics. In zero gravity with a fixed-joint grasp and millimetre-scale
quasi-static motion, inertial forces are tiny, so mass barely enters the
dynamics. The mass-bucket gate passed because the task is mass-insensitive, not
because the policy learned mass robustness. Read Level 2 as tight-clearance
robustness only.

Also records that contact forces are solved but never observed or limited, so
there is no damage-safety claim, and promotes that to the next roadmap item.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

Author: tryaksh

CLAUDE.md

@@ -26,12 +26,15 @@ Sim2Real plan.
 Levels 0, 1, and 2 of the secured-grasp insertion curriculum are promoted, each
 on three held-out evaluation seeds, at 100% success over roughly 9,000 episodes
 per level (27,121 total) with terminal metrics captured before Isaac Lab's
-automatic reset. Level 2 covers 1.5 mm side clearance and 5–15 kg payload mass.
-Level 3 stiction is physically blocked; terminal orientation error already
-consumes 97.8% of its tolerance, so that is the axis to instrument first. The
+automatic reset. Level 2 covers 1.5 mm side clearance and 5–15 kg payload mass. Envelope sweeps
+past the trained range show initial pose error is the binding axis (half-success
+near 7× trained noise, failing by lateral divergence, never by numerical
+instability), while blade mass is not a meaningful axis in this regime, which
+weakens the Level-2 mass claim. Level 3 stiction is physically blocked. The
 blade is held by a PhysX fixed joint standing in for an already-secured grasp;
-that is not learned grasping. Full numbers, limitations, and the pre-existing
-`train.py --smoke` probe defect live in `docs/status.md`.
+that is not learned grasping. Contact forces are solved but never observed or
+limited, so there is no damage-safety claim. Full numbers, limitations, and the
+pre-existing `train.py --smoke` probe defect live in `docs/status.md`.
 
 ## Operating rules
 

README.md

@@ -219,7 +219,8 @@ snapshot is:
 | Phase-2 robust task | Integration passed | CUDA smoke checkpoints saved at level 0 and level 4; six actions, zero gravity, no sensors, mass/friction/stiction ranges and compliant mount constructed; this is not convergence evidence |
 | Secured-grasp Level 0 (collision-free) | Promoted on three held-out seeds | Epoch 700 achieved 9,086/9,086 deterministic successes across near/medium/full starts on seeds 1060/2060/3060 (Wilson 95% lower bound 0.9996), with zero timeout, failure, mount-instability, or non-finite termination. Terminal pose, velocity, and cycle-time metrics are captured before Isaac Lab's automatic reset. |
 | Secured-grasp Level 1 (physical side rails) | Promoted on three held-out seeds | Fine-tuning the epoch-700 policy for 500 more PPO epochs achieved 9,014/9,014 deterministic successes on seeds 1061/2061/3061 with real wide side-rail collision and doubled reset joint noise (Wilson 95% lower bound 0.9996), zero instability. Stage-0 terminal axial error improved from 4.15 mm to 1.65 mm. |
-| Secured-grasp Level 2 (tight rails + 5–15 kg mass) | Promoted on three held-out seeds | Fine-tuning the Level-1 policy for 600 more PPO epochs achieved 9,021/9,021 deterministic successes on seeds 1062/2062/3062 with 1.5 mm side clearance and blade mass randomized over an observed 5.00–14.97 kg, including 100% in each of the low/mid/high mass bands. Full-distance cycle time improved to 7.20 s median. Terminal orientation error reaches 97.8% of its tolerance, the thinnest remaining margin. Level 3 stiction settling and Level 4 remain blocked. |
+| Secured-grasp Level 2 (tight rails + 5–15 kg mass) | Promoted on three held-out seeds | Fine-tuning the Level-1 policy for 600 more PPO epochs achieved 9,021/9,021 deterministic successes on seeds 1062/2062/3062 with 1.5 mm side clearance and blade mass randomized over an observed 5.00–14.97 kg. Full-distance cycle time improved to 7.20 s median. Level 3 stiction settling and Level 4 remain blocked. |
+| Capability envelope | Measured, not certified | Pushing the Level-2 policy past its training range: success degrades gracefully with initial pose error (100% at 1–2×, 97.0% at 3×, 62.4% at 6×, 21.2% at 12×), failing by lateral divergence with **zero** instability at every point. Blade mass is flat at 100% out to 1–50 kg, which shows the task is nearly mass-insensitive in this regime and that the Level-2 mass claim is weak. |
 | Nominal insertion baseline | Diagnosed, not promoted | Superseded 300-iteration curriculum: 56.35% full-distance and 22.57% near-distance deterministic success on unseen seed 1042; all failures were timeouts and the 90% gate was not met |
 | Mixed-curriculum axial baseline | Diagnosed, not promoted | Fresh 300-iteration run stayed correctly at Level 0 and achieved 1,292/2,000 (64.6%) near-distance deterministic success; lateral error remained above tolerance, motivating three-axis translation control |
 
@@ -294,6 +295,12 @@ validated.
 - Because every certification episode succeeded, the reported terminal error
   distribution is bounded by the success criterion itself. It shows where
   inside the tolerance box the policy lands, not accuracy independent of it.
+  The capability-envelope sweeps exist to cover that gap.
+- Blade-mass robustness is close to vacuous for this task: in zero gravity with
+  a fixed-joint grasp and millimetre-scale quasi-static motion, mass barely
+  enters the dynamics, and a sweep to 1–50 kg is still 100%.
+- Contact forces are solved by PhysX but never observed, limited, or reported,
+  so nothing here demonstrates the insertion would not damage a connector.
 - Level-0 margin is thin on two axes: terminal axial error reaches 11.96 mm of
   the 12 mm limit and orientation error reaches 0.0484 of the 0.0524 rad limit.
   Lateral error and blade velocities keep comfortable margin.

docs/claim_vs_evidence.md

@@ -26,13 +26,15 @@ insertion of an already-secured replacement blade into a rack in microgravity**.
 | --- | --- | --- |
 | A PPO policy inserts a secured blade with no rack contact, from three reset distances | 9,086 / 9,086 deterministic episodes, seeds 1060/2060/3060, Wilson 95% lower bound 0.9996 | `evidence/rigid_grasp_l0_ep700_certification.json`. Simulation only. Held-out *evaluation* seeds, one training seed |
 | The policy still succeeds when the rack side rails are physically collidable and initial pose error is doubled | 9,014 / 9,014 deterministic episodes, seeds 1061/2061/3061, Wilson 95% lower bound 0.9996 | `evidence/rigid_grasp_l1_ep1200_certification.json`. Fine-tuned 500 PPO epochs from the Level-0 checkpoint |
-| The policy still succeeds at 1.5 mm side clearance with payload mass randomized over 5-15 kg | 9,021 / 9,021 deterministic episodes, seeds 1062/2062/3062, and 100% in each of the low, mid, and high mass bands over an observed 5.00-14.97 kg | `evidence/rigid_grasp_l2_ep1800_certification.json`. Fine-tuned 600 PPO epochs from the Level-1 checkpoint |
+| The policy still succeeds at 1.5 mm side clearance with payload mass randomized over 5-15 kg | 9,021 / 9,021 deterministic episodes, seeds 1062/2062/3062, and 100% in each of the low, mid, and high mass bands over an observed 5.00-14.97 kg | `evidence/rigid_grasp_l2_ep1800_certification.json`. **The mass half of this is close to vacuous**: a sweep to 1-50 kg is also 100%, because zero-g quasi-static motion makes the task nearly mass-insensitive. Read this as tight-clearance robustness, not payload robustness |
 | Progressive fine-tuning improved precision and speed, it did not merely preserve success | Stage-0 terminal axial error fell from 4.15 mm mean (L0) to 1.65 mm (L1); full-distance median cycle time fell from 7.90 s (L0) to 7.20 s (L2) despite tighter clearance and a threefold mass range | Same reports. Distribution is bounded by the success criterion, see below |
 | Terminal-state evidence cannot be corrupted by the simulator's automatic reset | `TerminalMetricsMixin` snapshots each episode inside `_reset_idx`; unit tests assert the captured value differs from the post-reset value | `tests/test_terminal_metrics.py` |
 | No episode ended in numerical or physics instability | Zero non-finite and zero mount-instability terminations across 27,121 episodes | Categorization prefers instability over success when both fire in one control step |
 | Simulated cycle time is measured, not estimated | Median 7.2 s at full reset distance, 1.2 s at near distance, at 30 Hz control | Simulated time. Does not include perception, approach, grasp, or extraction |
 | Training is reproducible on consumer hardware | 512 environments on a 12 GB laptop GPU; 500 PPO epochs in about 22 minutes at 5,000-8,500 environment-steps/s | Isaac Sim 5.1 publishes a 16 GB VRAM minimum; this runs under it via benchmark-driven environment counts |
-| The remaining margin is known and stated, not hidden | Terminal orientation error at Level 2 reaches 0.0512 rad of a 0.0524 rad limit, 97.8% of budget | Orientation is the least-headroom axis and the predicted first failure mode at Level 3 |
+| The operating envelope is characterized, not just the operating point | Success degrades monotonically from 100% at the trained pose error to 97.0% at 3×, 62.4% at 6×, and 21.2% at 12×, with the half-success point near 7× | `evidence/rigid_grasp_l2_envelope_pose_error.json`. 500 episodes per point, one axis varied at a time |
+| The policy degrades safely rather than diverging | Zero instability and zero non-finite terminations at every sweep point, including 12× pose error where it fails four episodes in five | It stops completing insertions; it never blows up numerically |
+| The dominant failure mode is identified | Lateral divergence: terminal lateral error p95 goes from 0.0 mm at 3× to 60.6 mm at 4×, tripping the 60 mm failure predicate. Timeouts stay a small minority | This contradicted the prediction from margin analysis, which expected orientation to fail first. Orientation rises in failing episodes but is a symptom |
 
 ## What is not established
 
@@ -42,6 +44,8 @@ insertion of an already-secured replacement blade into a rack in microgravity**.
 | Sim2Real transfer | No real UR10e, hardware-in-the-loop rig, wrist force/torque sensor, calibrated camera, orbital acceleration data, or radiation dataset has been used |
 | Accuracy independent of the success criterion | Because every certification episode succeeded, the terminal error distribution is bounded by the success box. It shows where inside tolerance the policy lands, not error it was free to exceed |
 | Robustness to rail stiction or mount compliance | Level 3 stiction reaches valid geometry but cannot settle below velocity limits and is documented as blocked, not hidden. Level 4 floating-mount wobble is blocked behind it |
+| Payload-mass robustness in any meaningful sense | The task is nearly mass-insensitive in this regime, so the mass sweep is flat. A real mass axis needs faster motion, real grasp friction where weight sets slip margin, or gravity |
+| Damage safety | Contact forces are solved by PhysX but are not observed, limited, or reported. There is no peak-force or impulse budget, so nothing here shows the insertion would not bend a connector |
 | Cross-seed training repeatability | Each promoted policy comes from one training seed. The three certification seeds vary *evaluation* initial conditions only |
 | Perception | The policy consumes ground-truth blade pose. The vision student is scaffolding and has not been trained from the promoted policy |
 | Industrial fidelity | Rack, blade, and rail are primitive proxies with no connector, latch, cable, chamfer, measured tolerance, or force-displacement curve |
@@ -65,6 +69,13 @@ insertion of an already-secured replacement blade into a rack in microgravity**.
 - **Gate is in code.** `scripts/aggregate_evaluation.py` exits non-zero when any
   stage, the pooled rate, any randomized-parameter bucket, or the instability
   count fails. Thresholds are arguments, recorded in the report.
+- **Stress runs cannot pose as certification.** Evaluating outside the trained
+  distribution flips the report to `simulation_capability_envelope` and marks
+  the gate non-applicable, so a deliberately degraded result can never be
+  mistaken for a promotion.
+- **A failed prediction is recorded, not quietly dropped.** Margin analysis said
+  orientation would break first. The sweep showed lateral divergence. Both are
+  in `docs/status.md`.
 
 ## Honest one-line summary
 

docs/roadmap.md

@@ -8,29 +8,40 @@
    `scripts/aggregate_evaluation.py` pools runs and applies the gate.
 2. ~~**Promote L1.**~~ Done 2026-08-08. See `docs/status.md`.
 3. ~~**Promote L2.**~~ Done 2026-08-08. See `docs/status.md`.
-4. **Resolve L3 physically before training.** Plot rail force, blade velocity,
+4. ~~**Characterize the capability envelope.**~~ Done 2026-08-09. Pose error is
+   the binding axis (half-success near 7× trained noise, failing by lateral
+   divergence); blade mass is not a meaningful axis in this regime. See
+   `docs/status.md`.
+5. **Make contact force a measured quantity, then a constraint.** PhysX solves
+   contact forces but nothing observes, limits, or reports them, so there is no
+   damage-safety claim at all. Add a contact sensor on the blade, record peak
+   force and accumulated impulse into the terminal metric row, then train a
+   force-limited variant with abort-and-retry. This is the difference between an
+   academic insertion demo and an industrial one.
+6. **Resolve L3 physically before training.** Plot rail force, blade velocity,
    contact impulse, and action versus time. Check whether the stiction
    implementation injects energy or chatters at zero velocity. Prefer a
    continuous, measured friction/connector force curve and force/admittance-
-   limited insertion over arbitrary reward changes. The Level-2 certification
-   predicts where this breaks: terminal orientation error already consumes 97.8%
-   of its tolerance, so orientation is the axis to instrument first.
-5. **Add industrially meaningful hardware proxies.** Replace primitive
+   limited insertion over arbitrary reward changes. Instrument the lateral axis
+   first: the envelope sweep showed lateral divergence, not orientation, is what
+   actually terminates failing episodes, even though orientation consumes 97.8%
+   of its tolerance at Level 2.
+7. **Add industrially meaningful hardware proxies.** Replace primitive
    rail/handle geometry with measured, non-proprietary CAD; add chamfers,
    latch/connector engagement, wrench sensing, peak force/impulse limits, and
    abort/recovery behavior.
-6. **Learn grasp/extraction as a separate skill.** Validate force closure and
+8. **Learn grasp/extraction as a separate skill.** Validate force closure and
    axial pull with real pad collision before PPO. Do not remove the fixed joint
    from insertion until that gate passes.
-7. **Compose skills under a task manager.** Prefer separately validated
+9. **Compose skills under a task manager.** Prefer separately validated
    reach/grasp/extract/stow/insert policies over one monolithic sparse-reward
    policy.
-8. **Perception and adaptation.** Collect RGB/proprio/action data from the
+10. **Perception and adaptation.** Collect RGB/proprio/action data from the
    robust state policy, behavior-clone a vision policy, then fine-tune with
    asymmetric PPO. Keep ground-truth blade pose out of the deployable actor.
    Consider a real-world residual/adaptation layer only after safe hardware
    instrumentation exists.
-9. **Portfolio release.** Publish a short side-by-side video, learning curve,
+11. **Portfolio release.** Publish a short side-by-side video, learning curve,
    held-out failure montage, benchmark JSON, model card, and the one-page claim
    versus evidence table. Put checkpoints and videos in GitHub Releases, not Git
    history.

docs/status.md

@@ -120,6 +120,63 @@ evaluator and for a two-iteration checkpoint-resume training run through the
 passed for Levels 0, 1, and 2. Sustained environment-only benchmarks passed at
 1024 state and 256 vision environments; full PPO memory differs.
 
+## Capability envelope
+
+Certification says the policy works inside the distribution it trained on. It
+says nothing about where that ends. These sweeps push the promoted Level-2
+policy past its training range, one axis at a time, at full reset distance with
+500 episodes per point on evaluation seed 7060. They are **measurements, not
+certification**: the reports carry `evidence_type:
+simulation_capability_envelope` and their promotion gate is marked
+non-applicable.
+
+### Initial pose error is the binding axis
+
+Scale multiplies the trained reset joint-noise envelope of 0.001/0.002/0.004 rad
+by stage. Blade mass is held at the trained range.
+
+| Pose noise | 1× | 2× | 3× | 4× | 6× | 8× | 12× |
+| --- | ---: | ---: | ---: | ---: | ---: | ---: | ---: |
+| Success | 100% | 100% | 97.0% | 87.8% | 62.4% | 42.6% | 21.2% |
+| Failures | 0 | 0 | 15 | 58 | 166 | 268 | 394 |
+| Timeouts | 0 | 0 | 0 | 3 | 22 | 19 | 0 |
+
+Report: `evidence/rigid_grasp_l2_envelope_pose_error.json`.
+
+Degradation is monotonic and graceful, with roughly double the trained noise
+absorbed for free and the half-success point near 7×. **Every point, including
+12×, had zero instability and zero non-finite terminations**: the policy fails
+by not completing the insertion, never by diverging numerically.
+
+The failure mechanism is **lateral divergence, not orientation**. Terminal
+lateral error p95 jumps from 0.0 mm at 3× to 60.6 mm at 4× and 92.9 mm at 12×,
+tripping the 60 mm lateral failure predicate; timeouts stay a small minority.
+This *contradicts* the earlier prediction from the Level-2 margin analysis, which
+expected orientation to break first because it consumed 97.8% of its tolerance.
+Orientation error does rise in failing episodes, but it is a symptom; the
+termination is lateral.
+
+### Blade mass is not a meaningful axis in this regime
+
+Pose error held at the trained level; only the mass range varies.
+
+| Mass range (kg) | 5–15 (trained) | 3–20 | 2–25 | 1–35 | 1–50 |
+| --- | ---: | ---: | ---: | ---: | ---: |
+| Success | 100% | 100% | 100% | 100% | 100% |
+
+Report: `evidence/rigid_grasp_l2_envelope_blade_mass.json`. Terminal metrics are
+nearly identical at every range (axial p95 about 9.7 mm, orientation p95 about
+0.050 rad).
+
+**This weakens the Level-2 claim and should be stated plainly.** "Robust to
+5–15 kg payload mass" sounds substantial but is close to vacuous here: in zero
+gravity, with the blade held by a fixed joint and the tool moving in
+millimetre-scale quasi-static increments at 30 Hz, inertial forces are tiny, so
+mass barely enters the dynamics. The Level-2 mass-bucket gate passed because the
+task is insensitive to mass, not because the policy learned mass robustness. A
+mass axis only becomes meaningful with faster motion, real grasp friction where
+weight sets slip margin, or gravity.
+
 ## Demonstration assets
 
 Recorded from the promoted Level-2 checkpoint at full reset distance, 300