[Roadmap] MindAct 0.2: reproducible embodied-policy evaluation

[lvyufeng]

Summary

This issue tracks the proposed product and engineering roadmap for MindAct after the repository pivot from MindNLP/MindTorch to a PyTorch-native toolkit for embodied policies.

The near-term goal is not to build another general-purpose training framework. MindAct should first become a reproducibility harness for embodied-policy experiments:

Train or load a policy, evaluate it on a simulator, and explain exactly what produced the result.

LeRobot and LIBERO will be the first reference integrations, while the core contracts remain framework- and simulator-neutral.

PR #2772 established the foundation and is now merged:

• deterministic, dependency-free evaluation runner;
• frozen configuration and result records;
• write-once manifest and evaluation artifacts;
• fake policy/environment smoke path;
• CLI and Python API contracts.

Product direction

Primary positioning

Reproducible evaluation and diagnostics for embodied policies.

MindAct should own experiment orchestration, provenance, deterministic evaluation, result comparison, and failure diagnostics. It should reuse native training/inference implementations from LeRobot and other ecosystems rather than reimplementing every optimizer, scheduler, and distributed-training feature.

Reference workflow

pinned dataset/checkpoint
        -> policy adapter
        -> observation/action bridge
        -> simulator environment adapter
        -> deterministic rollouts
        -> manifest + episode records + metrics + optional video
        -> comparison and diagnostics

Milestones

M0 — Foundation (complete)

Delivered by PR #2772.

• Core protocols and optional-dependency boundaries
• Strict, frozen experiment configuration
• Deterministic EvaluationRunner
• Write-once manifest.json
• evaluation/results.json and evaluation/episodes.jsonl
• Dependency-free fake evaluation CLI
• Unit/integration smoke coverage without torch, LeRobot, or LIBERO

M1 — Integration readiness and developer experience

Goal: make environment and dependency failures diagnosable before a real rollout starts.

Candidate work:

• Add mindact doctor for Python, PyTorch, device, MuJoCo, LeRobot, LIBERO, renderer, and checkpoint-access diagnostics.
• Add mindact manifest show for human-readable provenance inspection.
• Define a stable checkpoint reference model for local paths and Hub revisions.
• Define observation/action schema and normalization contracts.
• Add integration-test fixtures and headless/offscreen configuration guidance.

Exit criteria: a contributor can determine whether their machine is ready for a real evaluation without reading adapter internals, and all checks remain clean when optional dependencies are absent.

M2 — Minimal LIBERO environment adapter

Goal: create and close one real LIBERO environment through the MindAct protocol.

Initial scope should stay intentionally narrow:

• One supported task suite and one task as the reference path.
• Serial, single-environment execution.
• Seeded reset, normalized step, and guaranteed close.
• Explicit LIBERO and simulator version reporting.
• Headless/offscreen operation in CI or a documented local smoke path.
• No vectorized environments or broad task-suite abstraction in the first cut.

Exit criteria: a real LIBERO adapter passes lifecycle and contract tests, including missing-dependency skips and cleanup on failure.

M3 — Minimal LeRobot policy adapter

Goal: load one pinned policy checkpoint and produce actions through the MindAct policy contract.

The first reference implementation should target ACT or another well-supported LeRobot policy, not every policy family.

• Lazy-load LeRobot and PyTorch only when the adapter is requested.
• Support a local checkpoint and a pinned Hugging Face revision.
• Capture checkpoint identity, repository revision, policy revision, and runtime device.
• Normalize inference mode, device placement, and no-gradient execution.
• Keep checkpoint-format assumptions inside the adapter.

Exit criteria: a pinned checkpoint can be loaded and invoked on a documented synthetic observation fixture without importing optional dependencies from import mindact.

M4 — First real end-to-end evaluation

Goal: run one real policy on one real LIBERO task and produce auditable artifacts.

Target interface:

mindact eval configs/experiments/libero-act.yaml \
  --runner libero \
  --checkpoint <pinned-checkpoint>

• Implement the narrow observation/action bridge required by the reference ACT + LIBERO combination.
• Record preprocessing, normalization, task selection, checkpoint, runtime device, simulator version, and code revision.
• Produce manifest, config copy, aggregate metrics, per-episode JSONL, and optional video.
• Add a fixed-seed rerun check and a documented sanity comparison against a public baseline where feasible.
• Clearly distinguish smoke/test-double output from real benchmark output.

Exit criteria: a new user can follow the installation guide and run a small, fixed-episode real evaluation on a supported machine.

M5 — Comparability and diagnostics

Goal: turn provenance into a user-visible advantage rather than only an internal artifact format.

• Add mindact compare <run-a> <run-b>.
• Compute a canonical comparison fingerprint from dataset, checkpoint, environment, task set, preprocessing, seed schedule, and evaluation settings.
• Warn when runs are not directly comparable.
• Add mindact manifest diff for identity/config differences.
• Add failure artifacts with episode, step, phase, exception type, and safe message.
• Add model-free trajectory diagnostics: episode length, action magnitude/smoothness, gripper transitions, latency, and failure-step distributions.

Exit criteria: two runs can be compared from the command line with explicit warnings for incompatible provenance.

M6 — Training orchestration (after real evaluation)

Goal: provide a thin, provenance-first training entry point without replacing native framework trainers.

• Add a pinned LeRobot dataset adapter with dataset revision and normalization metadata.
• Invoke native LeRobot training for the first supported policy.
• Persist checkpoint lineage and training metrics.
• Optionally chain training completion into the real evaluation runner.
• Defer distributed training, sweeps, remote execution, and broad trainer abstractions until a concrete use case requires them.

Proposed PR sequence

1. PR2 — mindact doctor + integration readiness contracts
2. PR3 — minimal LIBERO environment adapter
3. PR4 — minimal LeRobot/ACT policy adapter
4. PR5 — observation/action bridge + first real evaluation
5. PR6 — manifest inspection and run comparison
6. PR7 — trajectory diagnostics and failure artifacts
7. PR8 — LeRobot dataset and thin training orchestration

The exact split can change if the first real integration reveals a better boundary, but every PR should leave the dependency-free core usable and tested.

MindAct 0.2 target

A realistic 0.2 preview should let a user:

1. install the required optional extras;
2. run mindact doctor;
3. load a pinned ACT checkpoint;
4. evaluate one supported LIBERO task for a small number of episodes;
5. obtain immutable provenance, episode records, metrics, and optional video;
6. rerun with a fixed seed and understand any differences;
7. compare two runs while seeing warnings for incompatible inputs;
8. continue using configuration checks and the fake runner without any optional ML dependencies.

Explicit non-goals for this roadmap

• Rebuilding PyTorch, LeRobot, or a general-purpose trainer.
• Supporting every simulator or policy family in the first release.
• Distributed execution, cloud orchestration, experiment sweeps, or a hosted dashboard.
• Treating a fake runner as a benchmark result.
• Adding eager imports of torch, LeRobot, LIBERO, or other optional dependencies.
• Sacrificing write-once provenance or deterministic behavior for convenience.

Design constraints

• Core dependencies remain limited to NumPy and PyYAML.
• Integration boundaries use runtime-checkable protocols and duck typing.
• Configuration and result records remain frozen dataclasses.
• Every real run writes a manifest before rollout and never mutates it afterward.
• Optional integrations stay behind lazy-loading boundaries.
• New behavior must include dependency-free tests and clean optional-dependency skips.

This is a proposed roadmap for discussion. The first implementation after this issue is expected to start with M1/M2, not with a broad training-loop rewrite.

[lvyufeng]

M1 progress: the developer-experience half is up for review in #2774.

Delivered:

• mindact doctor — reports MindAct/Python/platform plus importability of torch, Hugging Face Hub, LeRobot, and LIBERO, with install hints. Package availability is reported separately from runtime validation, and a missing optional package does not make the core installation unhealthy.
• mindact manifest show <run-dir-or-manifest> — read-only provenance inspection through the existing manifest parser, accepting either a run directory or a manifest.json path.
• Frozen, JSON-safe DiagnosticCheck / DoctorReport records.
• Docs updated: architecture command list, installation verification steps, evaluation guide, and a new developer-tools guide.

Still open in M1, deliberately deferred until the upstream APIs can be validated against a real installation rather than guessed:

• checkpoint reference model for local paths and pinned Hub revisions;
• observation/action schema and normalization contracts;
• headless/offscreen integration-test fixtures.

M2 (minimal LIBERO environment adapter) has not started. Its constructor and step/reset normalization need to come from the installed LIBERO API, so that work should land as its own PR.