vLLM Blackwell (SM120) with NVFP4 & FP8 KV Cache Quantization

This repository provides an automated build and serving stack for vLLM optimized for NVIDIA Blackwell GPUs (RTX 5090, B200 / SM120) with support for NVFP4 / FP8 KV cache quantization, automatic maximum context length, and fast uv-based container builds.

---

Directory Structure

vllm-nvfp4-docker/
├── docker/
│   ├── Dockerfile.prebuilt      # Fast uv-based build using pinned pre-compiled wheels (No build.sh required)
│   ├── Dockerfile.builder-base  # CUDA 13.3 devel + NCCL intermediate builder image (for source compilation)
│   └── patches/
│       └── flashinfer-cu130.patch # FlashInfer CUDA 13 wheel index fix
├── compose.yaml                 # Docker Compose service definition (targets Dockerfile.prebuilt)
├── .env / .env.example          # Environment & serving configuration
├── build.sh                     # Unified build script (for compiling vLLM from source PR)
├── .gitignore                   # Ignores build checkouts, local caches, and secrets
└── README.md                    # Architecture and documentation

---

Quick Start

1. Prerequisites

• GPU: NVIDIA RTX 5090 or other Blackwell architecture GPU (Compute Capability 12.0).
• Driver: NVIDIA Linux driver supporting CUDA 13.x (e.g. >= 570.xx).
• Docker: Docker Engine with NVIDIA Container Toolkit installed.

2. Configuration

Copy the configuration template (if not already created):

cp .env.example .env

Adjust parameters in .env if desired (e.g. MODEL_NAME, TOKENIZER_NAME, KV_CACHE_DTYPE, PORT, MAX_NUM_SEQS).

3. Build & Run Container

Option A: Fast uv-based Build (Recommended)

Build and run directly using Docker Compose without compiling from source:

docker compose build
docker compose up -d

Option B: Compile from Source (Custom PRs)

If you need to compile a specific vLLM PR or custom source modifications:

./build.sh
docker compose up -d

---

Operations

Follow container logs:

docker compose logs -f

Check status and healthcheck:

docker compose ps

Stop the server:

docker compose down

---

Serving Optimization (RTX 5090 / SM120)

• MAX_MODEL_LEN=auto: Automatically discovers and maximizes the context length based on available GPU VRAM.
• MAX_NUM_SEQS=2: KV cache memory scales directly with concurrent streams. For single-user / local desktop inference, setting this to 2 maximizes available context window length while fitting within 32 GB VRAM.
• SPECULATIVE_METHOD=mtp: Enables Multi-Token Prediction with 2 speculative tokens for higher generation throughput.
• CUDAGRAPH_MODE=PIECEWISE: Optimized piecewise CUDA graph capture sizes [2, 4].
• PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True: Mitigates VRAM fragmentation during large context prefill.

---

API Usage Example

Once the container is healthy, send requests using any OpenAI-compatible client or curl:

curl http://localhost:8000/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "unsloth/Qwen3.8-27B-GGUF:Q4_K_M",
    "messages": [
      {"role": "user", "content": "Explain KV cache quantization in two sentences."}
    ],
    "temperature": 0.7
  }'