🇺🇸 English | 🇨🇳 简体中文
🎮 Results & Demos | ⚡ Quickstart | 💻 Run locally | 🤗 Models | 📊 Benchmarks | 📚 Docs

JevAny is open infra for System 1 decision model training and deployment, covering data preparation, model adaptation and evaluation. Use a released model or train on your own data to route support tickets, select tools, or choose a robot's next action. One API takes the state, question and candidate options, then directly returns a choice and its probabilities.

🎮 Results and Demos

Full benchmark results and evaluation details.

The following 30 examples are archived replays from an earlier compatible JevAny checkpoint. The current default release is JevAny-Qwen3.8-27B. Explore the cases, or run a model locally to try your own inputs and see its choices and probabilities.

⚡ Jev inside LLM agent loops

Jev chooses among bounded candidate actions supplied by the environment or proposed by the LLM, which handles planning, recovery, and completion. The animations compare LLM only (left) with LLM + Jev (right) at equal reward. Steps are illustrated; accelerated playback preserves each pair's measured completion-time ratio. Click an animation to enlarge it.

Golden rules

Delegate the selection bottleneck, not the task. Jev is useful in the narrow gap where the LLM already knows the valid alternatives, but repeated frontier selection is costly or measurably unreliable.

Jev has two valid jobs: remove frontier work, or correct a measured local ranking weakness. If it does neither, it is overhead.

• Judge the phase, not the benchmark. WebShop and Terminal-Bench both mix open-ended reasoning with bounded choices. Delegate only the bounded phase; trivial choices are cheaper to execute directly, while missing strategies stay with the LLM.
• The menu is the capability ceiling. Give Jev 2–4 currently valid counterfactual branches that include the correct action and lead to different outcomes. Jev can rank expressed options; it cannot invent a missing plan.
• For efficiency, replace reasoning; do not append a judge. One LLM plan should fund several Jev choices. In the GPT-5.6-sol + 27B FrozenLake cell, LLM calls fell 64.4%; WebArena, where delegation did not remove frontier work, added 5.6% calls and 28.2% tokens. Quality-only use should first show a ranking gain in paired or D1 shadow evaluation.
• Stay inside the feedback horizon. Continue only while each action is reversible and its semantic effect is immediately observable. Return to the LLM on novelty, stale candidates, delayed feedback, or recovery.
• Promote autonomy with evidence. Move from D0 LLM-only → D1 shadow → D2 one-step → D3 routine-default → D4 bounded subgoal. Keep only non-dominated points: equal or higher verifier reward at lower cost, or higher reward with the added cost explicitly reported. Delegation coverage itself is not a win.

	1. WebShop Jev selects the requested color and size from LLM-generated menus before the LLM buys the product. Actions fall from 9 to 5, LLM calls from 9 to 4, tokens from 38,852 to 14,256, and time from 18.54 to 7.83 seconds.
	2. FrozenLake After one LLM plan, Jev checks each new state and chooses among four directions. Both runs reach the goal in 4 moves; LLM decision calls fall from 4 to 1, tokens from 2,338 to 663, and time from 19.7 to 16.7 seconds.
	3. Terminal-Bench For sqlite-db-truncate, Jev selects raw-page inspection from three commands, then the LLM recovers and verifies ten rows. Tool commands fall from 13 to 7, LLM calls from 15 to 8, tokens from 202,050 to 121,293, and time from 187.9 to 144.7 seconds.

Broader paired evaluations show that gains vary by task. The full results, delegation protocol, and technical report describe where Jev helps and when to return control to the LLM.

Task	Success	Efficiency
FrozenLake (GPT-5.6-sol, 10 pairs)	100% → 100%	LLM calls −64.4%, tokens −63.1%, time −37.6%
WebShop (LLM-generated menus, 3 pairs)	67% → 100%	LLM calls −21.4%, tokens −14.3%, time −15.0%
WebArena (6 pairs)	50% → 50%	LLM calls +5.6%, tokens +28.2%, time −0.4%
Terminal-Bench (6 pairs)	1/6 → 3/6	LLM calls −9.0%

📑 Table of Contents

• 🎮 Results and Demos
• ⚡ 1. Quickstart
  • 💻 1.1 Run locally
  • 🛠️ 1.2 JevAny Training
  • 🚀 1.3 JevAny Deployment
• 🤗 2. Pretrained Models
• 📊 3. Benchmark Results
  • ⏱️ 3.1 Inference efficiency
• 🕹️ 4. Examples & Test Environments
• 🧩 5. Supported Model Families
• 📚 6. Documentation and Contributing

⚡ 1. Quickstart

Use Python 3.12 or newer. Clone the repository and install the lightweight package:

git clone https://github.com/SimpleJev/JevAny.git
cd JevAny
python3.12 -m venv .venv
source .venv/bin/activate
python -m pip install -e .

Keep this environment active and work from the repository root. Start with a local demo, then train on your own data or use the API.

💻 1.1 Run locally

Choose a model that fits your computer:

Model	Hardware	Start here
Qwen 0.8B starter	CPU · 16 GB RAM recommended	Train the small adapter on the bundled tickets
JevAny-Qwen 4B	CUDA · ~8 GB for BF16 base weights, plus runtime memory	Load the released model
JevAny-Qwen 27B	CUDA · ~54 GB for BF16 base weights, plus runtime memory	Choose the larger checkpoint

The local model guide covers preparation and loading. Released models download on first use and reuse the local cache. With the model server running, open a second terminal in the same checkout:

source .venv/bin/activate
jevany demo --base-url http://127.0.0.1:8008 --text-only

Open http://127.0.0.1:8090, choose Test and connect, then edit Try your own decision and press Ask the model. Change the state or options to see how its decision changes. Games, robotics and replays are available in the same playground.

🛠️ 1.2 JevAny Training

Train your own System 1 model on the same state and questions you send at inference, with a label for each question. Start with the bundled synthetic support tickets, then train on your own labelled data. The starter recipe uses Qwen3.5-0.8B on CUDA with BF16 and writes runs/my-jev:

python -m pip install -e '.[train]'
jevany data init --out data/starter
jevany data validate data/starter/train.jsonl
jevany train --config recipes/sft.toml --dry-run
jevany train --config recipes/sft.toml

After training, try the checkpoint on the included ticket request:

jevany decide examples/request.json --checkpoint runs/my-jev

Pass --data to train on your own JSONL data, or use recipes/finetune.toml to adapt the released 27B model. See the training guide for CPU settings, multimodal data and standard torchrun launches. For image/video training or fine-tuning the released 27B model, install .[train,multimodal].

After SFT, you can continue with experimental RLCR, which rewards correctness and probability calibration:

jevany train --config recipes/rlcr.toml

🚀 1.3 JevAny Deployment

Install the serving dependencies and start the released Qwen 4B model on a CUDA GPU. See the hardware and loading guide for memory requirements.

python -m pip install -e '.[serve,multimodal]'
jevany serve --checkpoint SimpleJev/JevAny-Qwen3.5-4B-LoRA \
  --device cuda --dtype bf16 --port 8008

The default path favors reproducibility. CUDA deployments can opt into BF16 LoRA merging, SDPA and torch.compile; the useful settings differ between 4B and 27B. See the inference acceleration guide for commands, H200 measurements and accuracy caveats.

To serve your training output, replace the checkpoint ID with runs/my-jev. Keep the server running. In a Python session using the same environment, send a ticket and the departments that can handle it:

from jevany import Choice, JevClient

jev = JevClient("http://127.0.0.1:8008")
result = jev.system_one(
    state={"ticket": "I was charged twice. Please help."},
    questions={
        "department": Choice(
            instructions="Which team should handle this?",
            criteria={"billing": "Payment problems", "shipping": "Delivery problems"},
        ),
    },
)
answer = result["answers"]["department"]
print("Selected team:", answer["choice"])
print("Probabilities:", answer["probabilities"])

choice is one of the department names; probabilities maps each name to its probability. Your application can use these fields to route the ticket or ask for review when the decision is uncertain. Use Noul for yes/no questions, such as whether a ticket needs urgent review, and Score for ordered levels, such as low, normal and high priority. See the API reference for all three question types.

For in-process inference, load a model in Python and use the same interface. For image and video inputs, follow the media setup.

🤗 2. Pretrained Models

For a first local run, choose a model and hardware in Run locally.

Model	Readout	Intended use
JevAny-Gemma-4B	Pointer	Compact Gemma release
JevAny-Qwen3.5-4B	Pointer	Compact, flexible choice count
JevAny-Qwen3.5-4B-Direct-Token	Direct-token	Best released 4B JevBench accuracy
JevAny-Qwen3.8-27B	Pointer	Default; highest released accuracy
JevAny-Muse-Glimmer-30B	Pointer	Muse Glimmer alternative

These LoRA adapters were trained with SFT on 1,772,725 text records containing 2,180,242 labelled decisions; see training compute and experiments for the setup. Full-parameter SFT and further post-training improvements are planned.

The corresponding base model is loaded separately and its license and access terms apply. Allow roughly twice the base parameter count in bytes for BF16 weights, plus runtime memory. See the hardware and loading guide.

Pointer and direct-token models share the same API. Pointer supports up to 4,096 options within the context limit; direct-token supports up to 255. See readout choices for training and accuracy tradeoffs.

📊 3. Benchmark Results

JevAny-Qwen3.8-27B leads both benchmarks and has the lowest NLL and Brier. Among 4B releases, direct-token leads on JevBench; pointer leads on Transfer.

Model	Transfer ↑	JevBench ↑	NLL ↓	Brier ↓	ECE ↓
Kev-4B	74.19%	75.32%	0.858	0.380	0.125
Kev-27B	82.31%	85.28%	0.533	0.265	0.050
Jev 1.13.0	85.37%	86.58%	0.644	0.212	0.033
Laya	52.29%	58.01%	1.264	0.615	0.127
JevAny releases
JevAny-Gemma-4B	70.84%	77.49%	0.706	0.369	0.056
JevAny-Qwen3.5-4B	78.68%	80.09%	0.587	0.297	0.035
JevAny-Qwen3.5-4B-Direct-Token	78.20%	80.95%	0.564	0.291	0.029
JevAny-Muse-Glimmer-30B	83.46%	87.45%	0.464	0.229	0.032
JevAny-Qwen3.8-27B	86.04%	90.04%	0.388	0.195	0.026

NLL, Brier and ECE are measured on Transfer.

Full results and protocols · Machine-readable results · Method and ablation report

The external comparison uses the complete Typed Decisions test split, the full 3,220-record JevJudge multimodal suite, and its 724-record text slice. The same 13 models stay in the same order; — means unsupported native input or no matching result.

• JevAny-Qwen3.8-27B: 72.8% Typed accuracy, 62.3% JevJudge full accuracy, and 66.4% JevJudge text-only accuracy.
• All five JevAny releases completed 3,220/3,220 native text, image, and video records. Their full-suite accuracy ranges from 51.5% to 62.3%; the strongest complete open baseline scores 48.2%.
• On the JevJudge text-only subset, Qwen3.8-27B scores 66.4% and Kev-27B scores 64.2%. Kev has no native image/video path, so its full result is —.
• Jev 1.13 (OpenRouter): 72.7% Typed accuracy (published) and 65.1% JevJudge text-only accuracy; its full result is — because the endpoint is text-only. Published Decider 1 (76.8%) and Liquid d1 (74.2%) remain higher on Typed Decisions.

All three panels use accuracy. JevJudge full covers all 3,220 multimodal records; text-only is its 724-record text subset. The benchmark's official skill_role metric remains in the detailed evaluation notes.

Full external tables and reproducibility notes · Machine-readable chart results

⏱️ 3.1 Inference efficiency

On one H200, CUDA Graphs cut Qwen3.8-27B median latency from 113.54 to 30.53 ms (3.72×) on 231 JevBench questions; fused SDPA plus CUDA Graphs cut Muse-Glimmer-30B from 100.71 to 43.25 ms (2.33×) on a balanced 44-request Transfer panel. Accuracy stayed at 207/231 and 38/44, with no argmax changes. The table uses the H200 headline measurements for 27B and 30B, while retaining the original apples-to-apples A100-40GB comparison for the 4B models. Latency is comparable within each row; the fixed evaluation panels are listed explicitly. In the plot, diamonds show the 27B/30B H200 arrows and circles show the A100 cohort; the H200 points are not mixed into the A100 frontier.

Model	Hardware	Before	After	Speed-up	Accuracy check	Fixed panel
JevAny-Qwen3.5-4B	A100	104.6 ms	25.3 ms	4.1×	78.68% → 78.87%	Transfer, 1,046
JevAny-Qwen3.5-4B-Direct-Token	A100	106.4 ms	25.9 ms	4.1×	78.11% → 78.39%	Transfer, 1,046
JevAny-Gemma-4B	A100	106.3 ms	31.9 ms	3.3×	70.84% → 70.84%	Transfer, 1,046
JevAny-Muse-Glimmer-30B	H200	100.71 ms	43.25 ms	2.33×	38/44 → 38/44	Transfer sample, 44
JevAny-Qwen3.8-27B	H200	113.54 ms	30.53 ms	3.72×	207/231 → 207/231	JevBench public, 231

Median model-call latency, serial batch size 1. See the full report for the apples-to-apples A100 comparison and panel limitations.

_{4B releases on one A100-40GB, Transfer-v9. Each bar fills at 1/20 of real time and stops at that stage's median latency.}

Full tables, setup and other models · How to enable · H200 results · A100 results

🕹️ 4. Examples & Test Environments

The playground includes the three environments below. These GIFs preserve historical model actions and option probabilities; run the current JevAny-Qwen3.8-27B checkpoint with the commands in the playground guide.

🤖 4.1 Robot peg insertion

Use a Franka gripper to grasp, align and insert a peg, checked by PyBullet contact physics.

🔫 4.2 Doom corridor · 3D

Clear the final room by defeating the enemies on the left and right, then move forward. The environment uses ViZDoom and the included Freedoom assets.

⛏️ 4.3 Crafter survival · 2D

Gather wood, craft tools and mine stone while managing health and supplies.

🎮 4.4 Try the playground

With a model running from Run locally, open the playground:

jevany demo --base-url http://127.0.0.1:8008 --text-only

Open http://127.0.0.1:8090, choose Test and connect, and try your own decision. To let the model control a game, install the optional engines and restart the playground:

python -m pip install -e '.[demo]'
jevany demo --base-url http://127.0.0.1:8008 --text-only

Choose Run model, then One decision or Run automatically. Play yourself lets you control the game. Live control sends text state to the model; robot control uses the .[robotics] extra.

For the bundled recordings, run jevany demo and choose Replay. Playback works on CPU without model weights. See the playground guide for platform requirements and environment APIs, or integrations to combine JevAny decisions with an LLM planner.

🧩 5. Supported Model Families

Model IDs, supported inputs and setup requirements.

📚 6. Documentation and Contributing

Training · Deployment · API · Data · Evaluation · Agent harness protocol · Contributing

To contribute a model adapter, evaluation or application example, start with the contribution guide. The technical report describes model design, multimodal support, the agent-harness study and appendix, negative results, and open questions.

Code and starter data are Apache-2.0. Some components are adapted from Kev; see NOTICE and ACKNOWLEDGEMENTS.md. Base models and upstream datasets retain their own terms.