Skip to content

ARCHITECTURE.md

Latest commit

 

History

History
144 lines (131 loc) · 13.5 KB

ARCHITECTURE.md

File metadata and controls

144 lines (131 loc) · 13.5 KB
Raw

Architecture

A map of the repo for someone who intends to change it. The CHANGELOG is the memory (every bug, measured, with the gate that now holds it); TESTING.md is the testing philosophy; SECURITY.md is the trust model. This file is the layout: what each part is for, where state lives, and which invariants the CI keeps true.

The box, end to end

  agent client (opencode, Claude Code, any OpenAI/Anthropic client)
      |  :30001
      v
  keepalive-proxy.py ── feeds ──> cockpit request feed / zombie guard
      |  127.0.0.1 :30000
      v
  text engine (docker, exactly one lane enabled at a time)
      ├── 27B lane: qwen38-sglang(.service)      SGLang, NVFP4/FP8 + DFlash2 draft
      ├── flash lane: qwen38-flash(.service)      SGLang, 176B NVFP4 + NEXTN, PLE table on NVMe
      ├── Flash FP4-Local lane: qwen38-flash-local(.service)  vLLM + NVIDIA's b12x, 176B FP4-Local + MTP, its table read in place
      ├── Kolibri lane: qwen38-kolibri(.service)  vLLM 0.29 + Aleph Alpha's plugin, Kolibri-1 FP8 or NVFP4
      ├── Nemotron lane: qwen38-nemotron(.service)  vLLM nightly + NVIDIA's b12x, Bolt 74B + LiLiCorr draft
      ├── Nemotron Super lane: qwen38-nemotron-super(.service)  the same image, Super VL 120B NVFP4 + LiLiCorr draft, 1M window
      └── Ornith lane: qwen38-ornith-122b(.service)  the Flash FP4-Local lane's image, Ornith 1.5 122B NVFP4 + MTP
  HF cache (weights, pinned revisions)    ~/flashnext-ple (176B N-gram table)
                                          ~/flash-local-ple (FP4-Local table's copies, cache on another disk)

  cockpit :30090 ── sudo exact argv ──> systemd (start/stop/switch/flush)
      └── relay :30091 ──> opencode serve 127.0.0.1:4096 (Agent view)

Everything the stack writes to the machine lives in one config dir, ~/.config/qwen38 (called CONFIG_DIR in every script): the API key (0600, also the cockpit login), the patched chat templates, the flash launcher, the reduced draft vocabulary, the deployed copy of the proxy, the opencode config artifact, the opencode.off and cockpit.off markers, the cockpit state files (cockpit-secret 0600, cockpit-events.jsonl, cockpit-history.json, wedge-*.txt dumps), and unit backups (*.bak-preupdate). Outside it: systemd units in /etc/systemd/system, one sudoers file in /etc/sudoers.d, the pyspy forensics wrapper in /usr/local/bin, the oc launcher in ~/.local/bin, and the two weight trees (HF cache, PLE dir). ./uninstall.sh --list is the authoritative inventory of all of it.

The parts, and the one job of each

file the job notes worth reading
install.sh converging installer: refuses root before it writes, reads the installed unit so a re-run keeps your target, port, cache and context mode, renders every artifact from the pins, installs the cockpit as step 10/10, installs every other lane (since v1.20 the other text lane is this script started again for it, QWEN38_SECONDARY; the side lanes are their own installers), since v1.33 on demand (their units; prepare-lane.sh builds each at its first Load), proves each one it builds whole with the served lane stopped once, and ends on its URL the convergence block and resolve_flash_*_args (a flag computed before convergence caused the worst bug in v1.8.0); pins at the top are a contract, not decoration
keepalive-proxy.py the only component that reads bytes it does not control keepalive at SSE event boundaries, zombie abort ordering (x-override-rid, abort before close, drain otherwise), oversize guard that counts via /tokenize and prices images from their headers, corruption tripwire, and since v6.20 a refusal for the request fields SGLang leaves unbounded and dies on rather than rejecting (top_logprobs/logprobs/top_logprobs_num past a ceiling, sglang#40076; stop_token_ids and input_ids past the vocabulary, learned from the one refusal message that names it; n past 128, sglang#31597, whose fixes were closed unmerged); the v6.x header is its real version, read live by the cockpit; since v6.19 it answers POST /v1/systemone itself (typed decisions read off top_logprobs, one single-token branch per question, never token_ids_logprob: that path kills this build's scheduler on a mixed batch, sglang#34719; the state is fenced with a start-up token so it cannot write the framing of its own question, the caller's socket is watched so an abandoned fan-out is stopped and aborted, and the request path is decoded once before any check reads it, the way the engine routes it)
run.sh / switch-model.sh / uninstall.sh / get.sh foreground serving, surgical live switch, read-only-first removal, one-liner bootstrap the switch never starts, stops or restarts an engine (it restarts opencode-web when the limits change) and never touches images; it stages unit files at fixed paths because the cockpit's sudoers pins that argv exactly
install-kolibri.sh, install-nemotron.sh, install-nemotron-super.sh, install-flash-local.sh, install-ornith-122b.sh, vllm-lane.sh the vLLM lanes' installers, each with its pins, its image and its unit, and what the five share in vllm-lane.sh: the engine's cache and port, the preflight and the room, the fetch, the service, the uninstall, the proof each refuses before it downloads what does not fit; a build of an image stays while an installed unit names it (the two Nemotron lanes run one image, lanes.sh); a Nemotron lane counts as begun once its own first Load began, not once the other built the image
oc-limits.sh the one table of opencode limits, three callers the comment block next to each number is the measurement record; the ceiling/threshold/two-thousand-step invariant is CI-held
dashboard/ the cockpit: state from a real generation canary, not /health (a wedged SGLang answers /health fine); the vLLM lanes' (Kolibri, the two Nemotron lanes, Flash FP4-Local, Ornith) from vLLM's own /health, which generates nothing, and its /metrics lifecycle.py is the pure state machine (100% branch coverage, mutation-scored), recipes.py derives "what the repo says this lane runs" from the repo's own text, registry.py knows the checkpoints, agent_relay.py is the browser boundary; every privileged call is exact argv through sudo, every action audited
flash-sglang/ vendored engine overlay, sha256-manifested, provenance in its ATTRIBUTION.md the record of what upstream replaced (its content is in the served image); nothing builds it since v1.18.7 retired OVERLAY_FLASH=1, whose image the launcher of v1.8 did not fit. The 27B lane had one too until v1.14, retired once measurement showed upstream carried all of it
bench.sh, bench-matrix.sh, needle.sh, bench-agent.py, conc-check.py, tools-check.py the instruments: single-lane probe, frozen comparable battery, long-context retrieval, the agent-loop shape, the concurrency question, and whether a client can execute what the model emits the battery is versioned and frozen so numbers stay comparable across years; bench.sh asks the engine which model it serves before printing a reference
patch-yarn.py, patch-template.py, build-token-map.py, oc-*.py, oc-*.sh the config surgery, all idempotent, all reversible patch-yarn has --restore/--check because a native install coming home from 1m would otherwise crash at load
check-pins.sh supply-chain availability, over the network, on demand deliberately not in CI: a green build must not depend on Hugging Face being up
evals/ the two quality drivers, MMLU and HumanEval, because the image's own invocations do not run and fail by printing an empty score the failure modes and the exact docker commands are in its README; gated against a stub sglang so the contracts that make them work cannot rot silently
docs/ one subject per file, everything the README is an entry point to: a page per lane (the 27B's 1M mode, flash, Flash FP4-Local, image, video, Kolibri, Bolt, Super VL, speech), models on an external disk, the cockpit tour, operations, engine restarts, opencode, pi and omp, clients, typed decisions, the memory trap, upstream standing the README's index table is the map; a section that outgrows the README moves here and leaves a summary and a link behind
extras/ opt-ins that earned their keep on the reference box CAKE anti-bufferbloat for downloads, opencode auto-continue plugin

Invariants the CI holds (each one exists because it once broke)

  1. The five independent spellings of the target list (selector options, action enum, install.sh case, switch-model.sh guard, cockpit BUILTIN_IDS) must agree; a failing run names the odd one out.
  2. The pin blocks: each consumer's own grep must select, by name, every pin it uses; adding a pin to install.sh without naming it breaks the build on purpose.
  3. The sudoers allowlist carries no wildcard, matches every privileged call switch-model.sh makes, and stages paths cross-checked character for character against their callers.
  4. The opencode limits chain: compaction threshold + one worst step <= the proxy ceiling, table and ceiling from oc-limits.sh only, no caller may own a copy of a number.
  5. Every serving lane carries --sleep-on-idle; speculative tiers run 3/1/4 steps; the FP8 path keeps its KV flag and the NVFP4 path does not gain one; flash recipes keep the radix cache and the api-key.
  6. The proxy's outcome vocabulary is a closed set: adding an outcome fails CI until it names what it means.
  7. Python is stdlib only, everywhere, forever; what an engine needs lives in its lane's venv (SGLang for images and video, Kokoro for speech, which speech-server.py imports only where it speaks); node --check holds the one JS plugin; no em or en dashes anywhere tracked.
  8. Tests are counted: what unittest reports versus what the files declare, per file and in aggregate, because a suite that ran nothing once passed.
  9. The offline suite cannot touch the box: witness HOME hashed either side, listening sockets compared, argv recorded by every stub.
  10. Coverage floors only rise, set from the most constrained runner; the pure-logic modules are at 100% branch coverage and the mutation score is the real quality gate.
  11. No test file in tests/ is left unrun: the unittest modules are discovered, the script-style ones must each be named by a step, and declared must equal ran both in aggregate and per file. Four tracked files had fallen off the hand-kept list, and one suite declared two tests and ran zero on every runner (a SkipTest out of setUpClass reports one skip with testsRun 0; skip with a class decorator).
  12. Neither entry point installs as root. A sudo bash run succeeds into /root with a new API key and leaves every client on 401 against a healthy engine, which is the one failure shape this stack has that produces no error anywhere.
  13. No engine starts on memory the last one has not given back: every engine unit runs engine-preflight.sh before its engine, every installer that writes such a unit puts the script in place first (run on its own, the image installer would otherwise leave a lane that never starts), one attempt fits in the unit's start timeout, and the script is no input of the engine, so a new one never restarts a healthy engine.
  14. The flash lane does not start on a GPU nvidia-smi reports outside its ATS addressing mode (flash-ats-check.sh, issue #54; a mode it does not name leaves the engine to decide): its launcher asks before the table or the image and ends a start it cannot make with an exit code its unit does not restart on, nothing downloads for it there (an update gives it its units only and starts nothing for it, prepare-lane.sh and switch-model.sh refuse it), and the script is no input of the engine.

How a change moves through the repo

install.sh renders (never edits) the units, the launcher, the configs and the templates from the pins; switch-model.sh rewrites exactly the lines it owns and defers the rest to the next install.sh; the cockpit derives what the repo says a lane should run from install.sh and the unit templates, so adding a flag means updating the recipe builders and their tests in the same commit (a recipe built from an unrendered placeholder is refused by builtin() on purpose). The three files that must agree on any target change (oc-limits.sh, the install.sh case, the cockpit selector) are checked against each other by CI, so the honest workflow is: change them, run ./ci-local.sh, read what it names.

Known open edges (stated, not hidden)

  • No image this repo can serve ships an sm_121 build of sgl_kernel: both the official release and the locally built image this lane used until v1.14 carry sm90 and sm100 only, and GB10 loads the sm100 cubin. It measures fine, it is simply not a tuned path.
  • The 176B boot rewrites its 47.7 GiB N-gram table every start (12 to 15 min to /health, docs/flash-lane.md); the cost, the reason, and the upstream shape of a fix are in the flash launcher's comments.
  • FP8 combined with the 1M context is not memory-measured; the README says to run at native until the curve exists.
  • Multi-tenant use does not exist. The proxy has optional TLS and optional per-client identity since v6.16, both off by default and naming clients rather than limiting them; the cockpit and the relay speak plain HTTP. SECURITY.md names these as the design's edges, and anything past them starts as an issue with a measured motivation.