🗺️ Roadmap: vLLM Full Compatibility (NPU-first)

[lvyufeng]

Goal

Make candle a backend capable of running vLLM workloads end-to-end, starting with an NPU-first target (Ascend / HCCL / ACLNN).

Near-term MVP target:

• single-node
• text generation
• TP=1 first
• no quantization initially

Longer-term target:

• tensor parallel
• graph capture
• quantization
• serving-grade throughput

---

Current State

Capability	Status	Notes
NPU backend (ACLNN)	✅ Strong	Mature op coverage, allocator, streams
NPU graph capture (NPUGraph)	✅ Exists	Needs vLLM-style validation
HCCL distributed collectives	✅ Strong	all_reduce / all_gather / reduce_scatter / all_to_all
CUDA backend	❌ Skeletal	Not viable for vLLM today
NCCL backend	❌ Missing	CUDA TP path blocked
torch.compile	⚠️ No-op stub	No real graph/compile backend
torch.library / custom_op	✅ Exists	Useful for custom kernel registration
torch.ops namespace	❌ Stub only	load_library() no-op
SDPA / fused attention	⚠️ Composite only	No paged attention / flash attention
Meta/Fake tensors	❌ Missing	Needed for shape-only init/tracing
Quantization	❌ Missing	No GPTQ/AWQ/FP8/INT8 kernels
Tensor-level record_stream()	❌ Missing	Allocator-level only

---

Key Blockers

1. No PagedAttention / fused decode attention kernel
2. torch.ops namespace is stubbed
3. No meta/fake tensor support
4. No real torch.compile / graph lowering story
5. No quantized kernel path
6. CUDA backend is not a practical target yet

Given current candle strengths, vLLM-on-NPU is realistic; vLLM-on-CUDA is not a near-term goal.

---

Sub-Issues

Phase 1: API Compatibility Foundation (Target: July 2026)

• [Task][vLLM] 1.1: Fix Library() 3-arg constructor #235 — 1.1 Fix Library() 3-arg constructor ⤴️ also HF [Task][HF] 1.4: Fix Library() 3-arg constructor #226
• [Task][vLLM] 1.2: Implement functional torch.ops namespace #236 — 1.2 Implement functional torch.ops namespace
• [Task][vLLM] 1.3: Add meta/fake tensor support #237 — 1.3 Add meta/fake tensor support ⤴️ also HF [Task][HF] 1.1: Meta device and Module.to_empty() implementation #223
• [Task][vLLM] 1.4: Expose Tensor.record_stream(stream) #238 — 1.4 Expose Tensor.record_stream(stream)
• [Task][vLLM] 1.5: Make torch.compile(..., backend="eager") a valid no-op #239 — 1.5 Make torch.compile(..., backend="eager") a valid no-op ⤴️ partial overlap with HF [Task][HF] 1.3: Promote torch submodule stubs from conftest to candle source #225

Phase 2: Core Serving Kernels (Target: September 2026)

• [Task][vLLM] 2.1: Implement PagedAttention for NPU #240 — 2.1 Implement PagedAttention for NPU
• [Task][vLLM] 2.2: Implement npu_fusion_attention compatibility wrapper #241 — 2.2 Implement npu_fusion_attention compatibility wrapper
• [Task][vLLM] 2.3: Add fused RoPE / rotary embedding kernel for NPU #242 — 2.3 Add fused RoPE / rotary embedding kernel for NPU
• [Task][vLLM] 2.4: Validate and harden NPUGraph for decode replay #243 — 2.4 Validate and harden NPUGraph for decode replay

Phase 3: Runtime Integration (Target: November 2026)

• [Task][vLLM] 3.1: Validate tensor-parallel execution with HCCL #244 — 3.1 Validate tensor-parallel execution with HCCL
• [Task][vLLM] 3.2: Add end-to-end vLLM integration tests #245 — 3.2 Add end-to-end vLLM integration tests
• [Task][vLLM] 3.3: Add quantization entry-point stubs #246 — 3.3 Add quantization entry-point stubs
• [Task][vLLM] 3.4: Benchmark serving throughput / latency #247 — 3.4 Benchmark serving throughput / latency

Phase 4: Full Serving Feature Set (Stretch — Q1 2027)

• [Task][vLLM] 4.1: Tensor parallel > 1 end-to-end #248 — 4.1 Tensor parallel > 1 end-to-end
• [Task][vLLM] 4.2: Quantized serving (GPTQ/AWQ/FP8) #249 — 4.2 Quantized serving (GPTQ/AWQ/FP8)
• [Task][vLLM] 4.3: Scheduler / memory manager validation under continuous batching #250 — 4.3 Scheduler / memory manager validation under continuous batching
• [Task][vLLM] 4.4: Multi-model benchmark suite #251 — 4.4 Multi-model benchmark suite

---

Cross-Roadmap Duplicates

Capability	vLLM	HF	Megatron	Notes
Meta device	#237	#223	—	Resolving one closes both
Library() 3-arg	#235	#226	—	Resolving one closes both
_dynamo / compiler stubs	#239	#225	—	Partial overlap; coordinate work

---

Milestones / DDL

• API compatibility base: 2026-07-31
• NPU serving MVP (single model, TP=1, no quant): 2026-09-30
• NPU production-oriented integration (graphs + TP validation + E2E tests): 2026-11-30
• Full-feature serving stretch goal: 2027-03-31

---

Success Criteria

• import vllm-style initialization paths no longer fail on missing PyTorch APIs
• Candle can execute prefill + decode for at least one small causal LM on NPU
• NPUGraph replay is validated on decode workloads
• HCCL-backed TP primitives are validated for serving use cases
• Throughput gap vs torch_npu baseline is understood and benchmarked