Add vlmbench-cli-skill with SKILL.md and MODELS.md

skills/vlmbench-cli-skill/MODELS.md

@@ -0,0 +1,43 @@
+# Supported Models
+
+Default vLLM serving arguments for OCR/VLM models. Use `--serve-args` to pass these when running benchmarks:
+
+```bash
+uvx vlmbench run --serve --backend vllm --model <model-id> --serve-args '<args>' -i <input>
+```
+
+| Model | Params | vLLM `--serve-args` | Notes |
+|---|---|---|---|
+| `lightonai/LightOnOCR-2-1B` | 1B | `--limit-mm-per-prompt '{"image": 1}' --mm-processor-cache-gb 0 --no-enable-prefix-caching` | |
+| `zai-org/GLM-OCR` | 0.9B | `--allowed-local-media-path /` | **Profile: `glm-ocr`** — requires transformers >= 5.0.0, vLLM nightly, MTP speculative decoding. Use `--profile glm-ocr` or `PROFILE=glm-ocr`. |
+| `rednote-hilab/dots.ocr` | 3B | `--trust-remote-code --gpu-memory-utilization 0.95` | |
+| `allenai/olmOCR-2-7B-1025-FP8` | 8B (FP8) | `--max-model-len 16384` | Based on Qwen2.5-VL-7B |
+| `Qwen/Qwen3-VL-8B-Instruct` | 9B | `--mm-encoder-tp-mode "data"` | |
+| `Qwen/Qwen3-VL-8B-Instruct-FP8` | 9B (FP8) | `--mm-encoder-tp-mode "data"` | |
+| `deepseek-ai/DeepSeek-OCR-2` | 3B | N/A | Not supported in upstream vLLM; requires [custom wheel](https://github.com/deepseek-ai/DeepSeek-OCR-2) |
+
+## Model Profiles
+
+Some models need custom environments (non-standard vLLM images, extra pip installs, special serve args). These are defined as profile directories in `vlmbench/profiles/<name>/` containing `config.toml` + `setup.sh`, shipped via PyPI.
+
+```bash
+# List available profiles
+uvx vlmbench profiles
+
+# Local: build once, serve, benchmark
+make build PROFILE=glm-ocr
+make serve PROFILE=glm-ocr
+make benchmark PROFILE=glm-ocr BENCHMARK_ARGS="--no-serve --base-url http://localhost:8000/v1"
+
+# HF Jobs (setup.sh runs at container start)
+make hf-benchmark PROFILE=glm-ocr FLAVOR=a100-large
+```
+
+Available profiles:
+
+| Profile | Model | Task | Custom Image | Custom Setup |
+|---|---|---|---|---|
+| `glm-ocr` | `zai-org/GLM-OCR` | completion | `vllm/vllm-openai:nightly` | transformers from source |
+| `qwen3-vl-2b-embed` | `Qwen/Qwen3-VL-Embedding-2B` | embedding | — | — |
+| `dse-qwen2-vl` | `MrLight/dse-qwen2-2b-mrl-v1` | embedding | — | — |
+| `qwen3-vl-2b-reranker` | `Qwen/Qwen3-VL-Reranker-2B` | score | — | — |

skills/vlmbench-cli-skill/SKILL.md

@@ -0,0 +1,60 @@
+---
+name: vlmbench
+description: >
+  VLM benchmark CLI — run, compare, and reproduce VLM inference benchmarks.
+  Use when the user wants to: (1) benchmark a VLM model's throughput, latency,
+  or VRAM usage, (2) compare results across models or configs, (3) choose or
+  configure a backend (vLLM Docker, vLLM native, Ollama, SGLang, cloud API),
+  (4) look up model-specific --serve-args, or (5) debug server startup or
+  GPU detection issues. Triggers on: vlmbench, VLM benchmarking, OCR model
+  evaluation, inference throughput, tokens/sec, TTFT, TPOT, VRAM.
+---
+
+# vlmbench
+
+All CLI logic is in `vlmbench/cli.py` (single file). Entry point: `main()`.
+
+## Running benchmarks
+
+Prefer `uvx` — no install needed:
+
+```bash
+# macOS (Ollama auto-starts)
+uvx vlmbench run -m qwen3-vl:2b -i ./images/
+
+# Linux (vLLM Docker auto-starts with --gpus all)
+uvx vlmbench run -m Qwen/Qwen3-VL-8B-Instruct -i ./images/
+
+# Linux (native vLLM)
+uvx vlmbench run -m Qwen/Qwen3-VL-8B-Instruct -i ./images/ --backend vllm
+
+# Compare results
+uvx vlmbench compare results/*.json
+```
+
+`run` is the default subcommand — flags like `--model` implicitly invoke it.
+
+## Backend resolution
+
+| Platform | `auto` resolves to | Model ID style |
+|---|---|---|
+| macOS | `ollama` | `qwen3-vl:2b` |
+| Linux | `vllm-openai:latest` (Docker) | `Qwen/Qwen3-VL-2B-Instruct` |
+
+Docker backends use `--gpus all --ipc=host`. Servers run in tmux sessions (`vlmbench-vllm`, `vlmbench-ollama`, `vlmbench-sglang`).
+
+## Model-specific serve-args
+
+See [MODELS.md](MODELS.md) for the full table of tested models and their required `--serve-args`. Reference this file when the user asks which models are supported or how to serve a specific model.
+
+## Concurrency sweep
+
+Use `--concurrency 4,8,16,32,64` to run at multiple concurrency levels in a single invocation. Produces one JSON per level (tagged `c4`, `c8`, etc.) and a consolidated comparison table. A single value (e.g. `--concurrency 8`) runs at that level only.
+
+## Key metrics
+
+TTFT (ms), TPOT (ms), tok/s, img/s, latency s/img, VRAM peak (MiB), prompt/completion token counts. Results saved as JSON to `./results/{backend}-v{version}-{model}-{gpu}-{tag}.json`.
+
+## GPU detection
+
+`CUDA_VISIBLE_DEVICES` is respected via `_nvidia_gpu_index()` -> `nvidia-smi --id={idx}`. When VRAM or GPU name looks wrong, check this env var first.