qlora-rs

Candle NF4 QLoRA layers on peft-rs; GGUF export is custom (not llama.cpp Q4_0).

Honest product class: NF4 pack/unpack + layer-level QLoRA (QuantizedLinear / peft-rs LoRA) on candle 0.11. This is not bitsandbytes, not a HuggingFace PEFT/QLoRA stack, and not a llama.cpp-compatible GGUF writer.

Version: 1.3.0 in Cargo.toml / GitHub Release. crates.io 1.3.0 (published 2026-08-24). Tag v1.1.0 was GitHub-only. Dense F32 merge helper is in this tree; custom GGUF is still not llama.cpp. This docs commit does not cut 1.3.1 or publish 1.3.0.

Docs: CONTRIBUTING.md · SECURITY.md · CODE_OF_CONDUCT.md · CHANGELOG.md · roadmap.md · docs/VERSIONING.md · docs/DEPENDENCIES.md · fixtures/README.md

Honest capability matrix

Capability	Status
NF4 pack/unpack + block absmax scales (CPU)	Yes
Double quant (simplified global u8 scales; not bitsandbytes-identical)	Yes — drops resident FP32 scales
QuantizedLinear / QLoraLayer (single layer)	Yes
Honor compute_dtype (BF16/F16/F32) in forward dequant	Yes
Multi-layer train (QLoraTrainer MSE/CE, AdamW + PagedAdamW writeback)	Yes (layer stack; not full HF LM / 7B)
Grad accumulation + global L2 clip	Yes (summed micro-batches; max_grad_norm)
NF4 golden fixtures (CPU CI)	Yes (fixtures/nf4_goldens.json)
QNAT export + load round-trip	Yes (export_native / load_native, v2)
HF model load / Linear inject / datasets	No (later work)
Dense merge helper merge_lora_into_dequantized	Yes — F32 NF4 dequant (not compute_dtype / cached BF16) + LoraLayer::scaling (rsLoRA); ship via axolotl HF merge
Emit PEFT adapter_model.safetensors / adapter_config.json	No — after train, save adapters via peft-rs save_pretrained_hf
GGUF export usable by llama.cpp / Ollama / LM Studio	No — custom type GGUF_TYPE_QLORA_NF4 (0x4E4634 ≠ ggml 2); metadata qlora.llama_cpp_compatible=false. Convert a dense HF dir with convert_hf_to_gguf.py
CUDA CubeCL NF4 kernels	Not built — cuda only enables candle-core/peft CUDA backends
Fused unsloth NF4-GEMM dispatch	No — should_dispatch_unsloth_nf4_gemm() is always false (CPU dequant + Candle matmul)
FP4 public CPU API	No — NF4-only public quant API (kernel stubs only)

Version: 1.3.0 (see Cargo.toml; Keep a Changelog + docs/VERSIONING.md). GitHub tag v1.3.0; crates.io newest is 1.2.0. This is a primitives + layer scaffolding library, not a drop-in bitsandbytes + PEFT + transformers QLoRA stack.

Features

• 🦀 Pure Rust (default features = CPU)
• 📉 ~4× weight storage reduction vs FP32 for NF4-packed tensors (scales overhead remaining)
• 📦 Dual export: GGUF-flavored custom NF4 layout + Candle native (QNAT)
• 🔗 Integrates with peft-rs for LoRA adapters
• ✅ Default-feature cargo test suite (unit + integration)

Installation

[dependencies]
qlora-rs = "1"

Optional:

qlora-rs = { version = "1", features = ["cuda"] }

Requires candle-core / candle-nn 0.11 and peft-rs 1.2.1. MSRV: Rust 1.96.

cuda enables candle-core/cuda and peft-rs/cuda only. It does not compile the CubeCL kernel sources under src/kernels/ (cubecl deps are disabled).

peft-rs pin

Build context	peft-rs dependency
crates.io / CI (committed)	peft-rs = "1.2.1"
Local sister tree	gitignored [patch.crates-io] peft-rs = { path = "../peft-rs" }

Quick Start

Quantize Weights

use qlora_rs::{quantize_nf4, dequantize_nf4};
use candle_core::{Device, Tensor};

fn main() -> anyhow::Result<()> {
    let device = Device::Cpu;

    let weights = Tensor::randn(0.0, 1.0, (4096, 4096), &device)?;
    let quantized = quantize_nf4(&weights, 64)?;  // block_size = 64

    println!("Original size: {} bytes", 4096 * 4096 * 4);
    println!("Quantized size: {} bytes", quantized.size_bytes());

    let restored = dequantize_nf4(&quantized, &device)?;
    let _ = restored;
    Ok(())
}

QLoRA Layer

use qlora_rs::{QLoraConfig, QuantizedLinear};
use candle_core::{Device, Tensor, DType};

fn main() -> anyhow::Result<()> {
    let device = Device::Cpu;
    let config = QLoraConfig::default();

    let weights = Tensor::randn(0.0, 1.0, (768, 768), &device)?;
    let layer = QuantizedLinear::from_weight(&weights, None, &config, &device)?;

    let input = Tensor::zeros(&[1, 10, 768], DType::F32, &device)?;
    let output = layer.forward(&input)?;

    println!("Trainable parameters: {}", layer.num_trainable_parameters());
    let _ = output;
    Ok(())
}

Deploy (not this crate's GGUF)

Train in NF4 if you want. Ship dense HF (PEFT adapters or merged weights) via axolotl-rs merge / export. Let llama.cpp convert_hf_to_gguf.py + llama-quantize produce standard GGUF.

export_gguf / merge_and_export_gguf write a custom NF4-in-GGUF-container (GGUF_TYPE_QLORA_NF4). That is internal / Candle-only. It is not llama.cpp Q4_0.

Prefer export_native / load_native (QNAT v2) for Candle-side round-trips.

use qlora_rs::{merge_lora_into_dequantized, QuantizedLinear};
// W' is dense F32 (NF4 unpacked at F32, LoRA scale from LoraLayer::scaling).
// Write it into an HF model.safetensors under the original Linear name.
let w_merged = merge_lora_into_dequantized(&layer)?;

Training (layer stack — PR-051)

Honest scope: train LoRA adapters on a stack of QuantizedLinear layers, not a full HuggingFace language model.

What works today:

1. Create QLoraTrainer + var_builder()-backed layers
2. Stack multiple QuantizedLinear layers
3. training_step (MSE) or training_step_lm (CE)
4. Standard AdamW or PagedAdamW with Var writeback — adapters do update (see integration tests)

What is not claimed:

• HF AutoModel load / Linear inject / datasets
• Production 7B end-to-end fine-tunes
• Bitsandbytes-identical double quant or optimizer paging semantics
• Writing PEFT adapter directories — this crate has no save_pretrained. After train, pass LoRA A/B through peft-rs save_pretrained_hf (not save_pretrained, which uses native keys under PEFT filenames).

QLoraTrainingConfig::{save_every, warmup_steps, page_size} are kept as 1.x public fields and are unused by training_step: there is no checkpoint writer, LR warmup is adapter_config.lr_schedule (not warmup_steps), and page_size is not a paging granule. Do not assume autosave or bitsandbytes paging.

cargo run --example qlora_training

// Multi-layer MSE sketch (see examples/qlora_training.rs)
let mut trainer = QLoraTrainer::new(training_config, device);
let layer0 = QuantizedLinear::from_weight_with_varbuilder(&w0, None, &cfg, trainer.var_builder().pp("l0"))?;
let layer1 = QuantizedLinear::from_weight_with_varbuilder(&w1, None, &cfg, trainer.var_builder().pp("l1"))?;
trainer.init_optimizer(&[&layer0, &layer1])?;
let loss = trainer.training_step(&[&layer0, &layer1], &input, &targets)?;

NF4 goldens (PR-052)

Fixed-input reference vectors live under fixtures/nf4_goldens.json. CPU CI tests in tests/nf4_goldens.rs assert packed codes, scales, and dequant. Offline regeneration notes (optional bnb cross-check, not required in CI): fixtures/README.md.

Scope: NF4-only public API (QLO-P1-02)

The public quantization API is NF4 only (quantize_nf4, dequantize_nf4, QuantizedTensor).

• There is no public quantize_fp4 / FP4 CPU path.
• FP4 CubeCL kernels under src/kernels/fp4.rs are source stubs and are not compiled (same as other CubeCL kernels).
• If you need uniform 4-bit, use NF4 or an external crate — do not assume FP4 is available via qlora-rs.

QNAT native format (QLO-P1-03)

Candle-native QNAT files support export and load:

use qlora_rs::{export_native, load_native, quantize_nf4, NativeMetadata};

let q = quantize_nf4(&weights, 64)?;
export_native(&[("w", &q)], Some(NativeMetadata::default()), "model.qnat")?;
let model = load_native("model.qnat")?;
let restored = model.get("w").unwrap();
assert_eq!(restored.data, q.data);

Format is VERSION 2 (per-tensor flags for double-quant / zero-points). Not a GGUF / llama.cpp format.

NF4 Quantization

NF4 (4-bit NormalFloat) uses 16 quantization levels optimized for normally-distributed data (QLoRA paper):

-1.0, -0.696, -0.525, -0.395, -0.284, -0.185, -0.091, 0.0,
 0.080, 0.161, 0.246, 0.338, 0.441, 0.563, 0.723, 1.0

Double quantization (simplified)

When double_quant: true, block scales are encoded to u8 with a single scale-of-scales and resident FP32 scales are dropped. This is not bitsandbytes nested-block double quant; see QuantizationConfig docs and size_bytes().

Storage (not a VRAM bench)

NF4 stores ~4 bits per weight plus per-block scales. That is arithmetic on the packed tensor, not a measured end-to-end VRAM or speed number. This repo has no dated METRICS row for training-time memory.

Known Issues / Dependency notes

Unmaintained paste (RUSTSEC-2024-0436)

Transitive via gemm → candle-core. Mitigation lives in sibling crates (not inside this package):

Path	Role
/root/work/qlora-paste (or tzervas/qlora-paste)	Maintained paste fork
/root/work/qlora-gemm (or tzervas/qlora-gemm)	gemm fork using qlora-paste

To patch in a workspace root Cargo.toml:

[patch.crates-io]
gemm = { path = "../qlora-gemm/qlora-gemm" }
# (and the gemm-* / qlora-gemm-* members as published by that workspace)

There is no qlora-rs/gemm-fork/ directory.

CUDA

• Feature cuda → candle + peft CUDA only.
• src/kernels/* CubeCL sources are not compiled.

Contributing

See roadmap.md for next work. Historical PHASE/STATUS/ANALYSIS files live in docs/archive/ and are not source of truth.

License

MIT only. See LICENSE. Third-party crates and inspirations: NOTICE.

Earlier docs claimed dual MIT OR Apache-2.0; this tree ships LICENSE only, matching Cargo.toml license = "MIT".