Can reinforcement learning improve a competent heuristic in a partially observed, six-player team game? This project implements the game, builds stronger deduction baselines, trains small neural agents, and makes their decisions inspectable.
The result is useful even where RL failed: imitation produced a competitive agent, but PPO did not consistently improve it across training seeds. The repository includes the evidence, a runnable baseline, and a guided full-game replay.
Requires Python 3.11+. The game, heuristic agents and viewer use only the standard library. No GPU, model download, account, Node.js or frontend build is needed. From the cloned repository, create and activate a virtual environment:
python -m venv .venvActivate with source .venv/bin/activate on macOS/Linux or
.venv\Scripts\Activate.ps1 in Windows PowerShell. Then:
python -m pip install -e ".[dev]"
python -m pytest
python -m literature.evaluate --team-a set_aware --team-b constraint --games 20 --paired --seed 42 --output outputs/quickstart
python -m literature.probe serve --root outputs --port 8765Open http://127.0.0.1:8765/. The evaluation writes a summary, per-game metrics,
configuration and sampled replays. --paired plays both team assignments, so the
command produces 40 games. Choose a fresh output directory when rerunning.
PyTorch tests skip unless the optional RL dependencies are installed.
To inspect the included 141-action guided game instead, stop the server and run:
python -m literature.probe serve --root examples --port 8765Open the guided replay at action 11. Compare asking rules, follow the actor's deductions, and jump to the zero-score fault at action 113. This losing game was selected to explain behavior; it is not a representative performance sample. Viewer guide.
- Complete v1 game: six seats, two fixed teams, 54 cards, nine six-card sets, asks, atomic declarations, fault penalties, showdown and separate training cutoffs.
- Information boundaries: own hand plus public history for each actor; omniscient replay and player-only exports are separate representations.
- Baselines: random, original heuristic, constraint-based deduction, and completion-aware asking. Public hand capacities strengthen ownership inference.
- Small neural experiments: imitation and hybrid PPO learn asks while sharing a fixed declaration controller. These are non-LLM agents.
- Inspection: deterministic replay, exact policy-input records, legal actions, rewards, heuristic explanations and optional local LLM call telemetry.
SetAware won 86.4% against the original heuristic and 77.0% against ConstraintGreedy, each over 250 paired deals / 500 games.
The final replication trained three no-entropy PPO seeds for 20,480 games each (61,440 total). Every table cell below uses 100 new paired deals / 200 games:
| Policy | Wins vs ConstraintGreedy | Mean utility | Wins vs SetAware | Mean utility |
|---|---|---|---|---|
| Imitation reference | 85.0% | +0.319 | 51.5% | +0.021 |
| PPO seed 41 | 84.0% | +0.312 | 35.5% | −0.074 |
| PPO seed 42 | 84.5% | +0.330 | 55.0% | +0.036 |
| PPO seed 43 | 59.5% | +0.101 | 43.5% | −0.036 |
Utility is terminal team score difference divided by nine. All 1,600 final evaluation games completed without truncation. The small initial no-entropy pilot looked promising; the longer replication did not establish a consistent gain.
Read the research results for separate experiment cohorts, paired uncertainty, validation curves, and limitations. Published evidence includes per-game outcomes and the imitation checkpoint; bulk local logs and PPO weights are not required for the quickstart.
python -m pip install -e ".[rl]"
python -m literature.evaluate --team-a hybrid_ppo --checkpoint artifacts/checkpoints/imitation.pt --team-b set_aware --games 20 --paired --seed 4200 --output outputs/imitation-checkThis works on CPU. GPU training and local LLM serving have separate setup needs; see training and model instructions. Local LLM runs in the historical results are bounded smoke tests, not evidence of strong LLM gameplay.
The neural encoder is a lossy history summary; its ownership weights are heuristic estimates, not calibrated posteriors. Declarations remain scripted for neural agents. Frozen-opponent win rates do not measure exploitability, equilibrium play, unseen-partner coordination or cross-game transfer. Legal bad moves are preserved: the featured heuristic can fault when every candidate ties at zero.
An omniscient replay contains every hand even when the UI hides some cards. Use player-only exports for actor datasets; perspective filtering is not access control.
- Rules and explicit house-rule choices
- Information model and export boundaries
- Evaluation protocol · Initial engine/LLM smoke results
- PPO diagnosis and falsifiable hypotheses
- Future experiment ideas and historical compute estimates
- GitHub publication checklist and résumé material
- Local packaging validation and CI boundaries
Core code is in literature/; packaged viewer assets are in literature/viewer/;
reproduction and analysis commands are in scripts/. The historical manual client
is retained for reference but is not the supported quickstart. MIT license.