🗺️ Roadmap: Hugging Face Ecosystem Full Compatibility (transformers / accelerate / peft / trl)

[lvyufeng]

Goal

Make candle a drop-in PyTorch replacement for the entire HuggingFace ecosystem. Target:

USE_CANDLE=1 python -c "
from transformers import AutoModelForCausalLM, AutoTokenizer
tokenizer = AutoTokenizer.from_pretrained('gpt2')
model = AutoModelForCausalLM.from_pretrained('gpt2')
print(tokenizer.decode(model.generate(tokenizer('Hello', return_tensors='pt').input_ids, max_new_tokens=20)[0]))
"

Priority follows the HF dependency chain: transformers (core) → accelerate → peft → trl.

---

Current State

Capability	Status	Notes
nn.Module (state_dict, load_state_dict, hooks)	✅ Ready	Full implementation
nn modules (Linear, Conv, Norm, RNN, Attention, etc.)	✅ Ready	Comprehensive coverage
Optimizers & LR Schedulers	✅ Ready	Adam, AdamW, SGD, etc. with state_dict
AMP (autocast + GradScaler)	✅ Ready	Device-type dispatch
Serialization (save/load)	✅ Ready	PyTorch zip checkpoint format
scaled_dot_product_attention	⚠️ Naive composite	No Flash/fused attention
DataLoader + Samplers	✅ Ready	Multi-worker, pin_memory
Distributed (DDP, FSDP, collectives)	✅ Ready	Gloo + HCCL backends
torch.utils.checkpoint	✅ Exists	Needs HF integration validation
torch.library / custom_op	✅ Ready	2-arg Library ctor only
compat/transformers test framework	✅ Ready	Tier 1-3 model test runner
Meta device / to_empty()	❌ Missing	Blocks low_cpu_mem_usage loading
Native safetensors loading	❌ In compat layer only	Not in src/candle/
Text generation (GenerationMixin)	❌ Missing	No generate(), KV-cache, sampling
torch.compile / _dynamo	⚠️ No-op stub	Blocks compiled model paths
Quantization (GPTQ/AWQ/bitsandbytes)	❌ Stub only	quantize_per_tensor raises RuntimeError
PEFT (LoRA adapter injection)	❌ Untested	Module internals may work
Accelerate integration	❌ Untested	Low-level pieces exist

---

Sub-Issues

Phase 1: Foundation (Target: v0.1.1 — June 2026)

• [Task][HF] 1.1: Meta device and Module.to_empty() implementation #223 — 1.1 Meta device and Module.to_empty() ⤴️ also vLLM [Task][vLLM] 1.3: Add meta/fake tensor support #237
• [Task][HF] 1.2: Native safetensors loading in candle source #224 — 1.2 Native safetensors loading in candle source
• [Task][HF] 1.3: Promote torch submodule stubs from conftest to candle source #225 — 1.3 Torch submodule stubs promoted to candle source ⤴️ partial overlap with vLLM [Task][vLLM] 1.5: Make torch.compile(..., backend="eager") a valid no-op #239
• [Task][HF] 1.4: Fix Library() 3-arg constructor #226 — 1.4 Fix Library() 3-arg constructor ⤴️ also vLLM [Task][vLLM] 1.1: Fix Library() 3-arg constructor #235

Phase 2: HF-Compatible Inference (Target: v0.2.0 — September 2026)

• [Task][HF] 2.1: Text generation / GenerationMixin support #227 — 2.1 Text generation / GenerationMixin support
• [Task][HF] 2.2: SDPA optimization paths and attention mask handling #228 — 2.2 SDPA optimization paths and attention mask handling
• [Task][HF] 2.3: Gradient checkpointing integration with HF transformers #229 — 2.3 Gradient checkpointing integration ⤴️ also Megatron [Task][Megatron] 2.4: Validate activation checkpointing compatibility #258
• [Task][HF] 2.4: Run and baseline compat/transformers tier-1 report #230 — 2.4 Run and baseline compat/transformers tier-1 report

Phase 3: Training & Fine-Tuning (Target: v0.3.0 — December 2026)

• [Task][HF] 3.1: Accelerate Accelerator.prepare() compatibility #231 — 3.1 Accelerate Accelerator.prepare() compatibility
• [Task][HF] 3.2: Quantization foundation (bitsandbytes 4-bit/8-bit) #232 — 3.2 Quantization foundation (bitsandbytes 4-bit/8-bit)
• [Task][HF] 3.3: PEFT adapter injection compatibility (LoRA) #233 — 3.3 PEFT adapter injection compatibility
• [Task][HF] 3.4: TRL (Trainer/SFT/DPO) compatibility #234 — 3.4 TRL (Trainer/SFT/DPO) compatibility

---

Cross-Roadmap Duplicates

Capability	HF	vLLM	Megatron	Notes
Meta device	#223	#237	—	Resolving one closes both
Library() 3-arg	#226	#235	—	Resolving one closes both
_dynamo / compiler stubs	#225	#239	—	Partial overlap; coordinate work
Activation checkpointing	#229	—	#258	HF = basic paths; Megatron adds selective recompute + RNG

---

Success Criteria

• v0.1.1: from transformers import AutoModel; model = AutoModel.from_pretrained("bert-base-uncased") works without compat conftest
• v0.2.0: model.generate() works for GPT-2/LLaMA-class models; tier-1 compat test pass rate ≥ 80%
• v0.3.0: Fine-tuning with accelerate + PEFT LoRA works end-to-end