JxMinecraftAgent

JxMinecraftAgent is an experimental, local-first Minecraft Java Edition survival agent. A shared TypeScript agent brain can drive either Mineflayer or an authenticated Fabric/Baritone client, use a locally served fine-tuned model for planning, record trajectories, update macro-level policies, and expose its state through a React dashboard.

Important

This project is a work in progress. The reported live-training runs currently reach approximately the iron stage. A complete fresh-world survival playthrough has not yet been demonstrated. The deterministic mock environment covers the planned progression through victory, but it is not evidence of a live Minecraft completion.

Dashboard preview in the deterministic mock environment.

How it works

flowchart LR
    MC["Minecraft client or private server"] <--> EX["Mineflayer or Fabric/Baritone<br/>one active executor"]
    EX <--> CORE["TypeScript agent core"]
    CORE --> SKILLS["Reflexes, skills, progression, and rewards"]
    CORE <--> LLM["Local fine-tuned planner<br/>OpenAI-compatible endpoint"]
    CORE <--> MEMORY["SQLite memory, replays, and trajectories"]
    CORE <--> RL["Python macro-RL and dataset tools"]
    CORE <--> TEACHER["Optional teacher and training governor"]
    CORE --> UI["React dashboard and replay viewer"]

Loading

The control loops are asynchronous: movement and skill execution continue while the planner or teacher model is working. The language model selects validated goals and macro actions; it does not directly emit keyboard, mouse, or network-packet commands.

The project uses a single-account switchable hybrid design. Mineflayer and Baritone share the same durable agent state, but they never connect to the account at the same time. Executor switching is locked while an episode or Minecraft connection is active.

Current status

Area	Status
Shared agent brain and executor interface	Implemented and covered by automated tests
Single-account Mineflayer/Baritone switching	Implemented and covered by automated tests
Reflexes, macro skills, progression, memory, rewards, and replay export	Implemented
Deterministic progression from spawn to victory	Covered in the mock environment
Live survival progression	In active development; reported to reach approximately iron
Local planner and trajectory collection	Implemented; local model endpoint and weights are external
Macro-RL, evaluation, and champion/challenger gating	Implemented as research infrastructure; training is ongoing
Full live fresh-world victory	Not yet demonstrated

The machine-readable acceptance status is maintained in docs/acceptance-audit.json. Run pnpm acceptance-audit for the current summary.

Highlights

• Deterministic safety reflexes that do not depend on model latency.
• Reusable, executor-independent Minecraft skills and a bounded action queue.
• Mineflayer and Fabric/Baritone execution behind one typed interface.
• Legitimate progression logic without operator commands, seed access, world-file inspection, teleportation, packet manipulation, or precomputed structure coordinates.
• SQLite-backed memory, replay recording, trajectory export, benchmark history, and policy artifact decisions.
• Local Qwen-compatible planning with schema validation and deterministic fallbacks.
• Optional teacher review with a deterministic training governor and strict resource limits.
• Responsive dashboard, emergency stop, executor controls, learning indicators, and replay viewer.

Repository layout

Path	Purpose
apps/agent-core	Main runtime, API, scheduler, persistence, and orchestration
apps/baritone-bridge	Authenticated Fabric/Baritone bridge for Minecraft 1.21.8
apps/dashboard	Live React dashboard
apps/replay-viewer	Recorded episode viewer
apps/mock-server	Deterministic local observation server
packages/	Shared executors, skills, progression, planning, memory, rewards, evaluation, and telemetry
python/	Macro-RL, dataset export, and evaluation tools
config/	Versioned default, development, and test configuration
docs/	Architecture, setup, progression, bridge, RL, and acceptance notes
scripts/	Audits, local launcher helpers, and live-playthrough verification

Requirements

• Node.js 24 LTS recommended; Node.js 22.18 or newer is supported by the current codebase.
• pnpm 11.7.0 or newer.
• Python 3.12 for RL and dataset tooling.
• Java 21 for the Fabric/Baritone bridge.
• Minecraft Java Edition 1.21.8 for live Baritone-compatible runs.
• A private world or server where automation is allowed.
• A locally served, OpenAI-compatible planner endpoint for non-mock operation.

The pinned compatibility baseline is Mineflayer 4.37.1, Mineflayer Pathfinder 2.4.5, Baritone 1.15.0, Fabric Loader 0.19.3, and Fabric API 0.133.4+1.21.8.

Quick start: mock mode

git clone https://github.com/JxAgentReal/JxMinecraftAgent.git
Set-Location JxMinecraftAgent
pnpm install --frozen-lockfile
Copy-Item .env.example .env
pnpm typecheck
pnpm exec vitest run
pnpm mock

In another terminal, start the dashboard:

pnpm dashboard

On Windows, ./start-all.ps1 starts the mock agent API, dashboard, replay viewer, and mock observation server together. Use ./start-all.ps1 -Stop to stop the processes it launched.

Live mode

1. Copy .env.example to .env and set the Minecraft account, executor, bridge token, and local model endpoint.
2. Prepare a private Minecraft 1.21.8 client or server where automation is permitted.
3. For Mineflayer, set ACTIVE_EXECUTOR=mineflayer. For Fabric/Baritone, build and install the bridge and set ACTIVE_EXECUTOR=baritone.
4. Start the local planner endpoint, then run pnpm development and pnpm dashboard.
5. Keep only one executor connected. The runtime enforces this rule and locks switching during active sessions.

See docs/setup-windows.md, docs/setup-linux.md, and docs/baritone-bridge.md for details.

Configuration and local data

.env.example documents the supported environment variables. Never commit .env, bridge tokens, Minecraft login data, teacher API keys, model weights, SQLite databases, world files, replays, trajectories, or training checkpoints.

Those files are intentionally excluded from Git. Runtime directories under data/ retain only empty placeholders in the repository and are populated locally as the agent runs. The fine-tuned model and its weights are also external to this repository; the agent connects to them through LOCAL_LLM_BASE_URL.

Verification

pnpm typecheck
pnpm exec vitest run
python -m pytest -q
pnpm security-audit
pnpm acceptance-audit
pnpm --filter @jx/baritone-bridge test

The live verification helper is intentionally separate because it requires a running Minecraft environment and credentials:

pnpm verify-live-playthrough -- --executor mineflayer

Do not interpret mock victory, unit tests, training loss, or teacher approval as proof of a live completion. Promotion is gated by held-out benchmark evidence, and the final goal remains a verified fresh-world survival playthrough.

Near-term roadmap

• Make live wood, stone, food, iron, and recovery behavior consistently reproducible across random seeds.
• Stabilize tool selection, crafting, smelting, navigation, and inventory recovery beyond iron.
• Validate Nether portal construction and live dimension transitions.
• Collect more high-quality trajectories and train/evaluate challenger adapters.
• Validate fortress, blaze, pearl, stronghold, End, dragon, and exit-portal behavior in real gameplay.
• Produce repeatable fresh-world victory evidence before claiming completion.

Responsible use

Use this project only in local worlds or private servers where automation is explicitly allowed. It is not designed for public-server evasion, competitive play, anticheat bypasses, or hidden-information access. JxMinecraftAgent is an independent research project and is not affiliated with Mojang Studios or Microsoft.