feat(ltx2): add LTX-2.5 distilled generation support

[kiarina]

Summary

Add LTX-2.5 distilled generation on Apple Silicon while preserving the existing mlx_video.models.ltx_2 design and LTX-2/LTX-2.3 compatibility.

Review guide by commit

1. a956b3a — split checkpoints, Gemma 4, 22B transformer, convolutional VAE, T2V, and I2V
2. b73e6a8 — selective component download, documentation, and model-path tests
3. 73d919b — extracted ancestral Euler implementation and sampler tests
4. 3ca6fef — audio generation, A2V, A2V+I2V, audio VAE/vocoder, and mux fix
5. c8000e1 — DurationHead, automatic frame-count prediction, and CLI controls
6. 7d8b2f0 — T2V/I2V prompt enhancement with a separate generative Gemma 4 model
7. fae541a — prompt-driven multishot example and limitations
8. 5f008b8 — initial DFR pipeline with generated keyframes, reference conditioning, and detailing IC-LoRA
9. 1e071c7 — expose the LTX generator alias used throughout the README
10. dc33a96 — score-free 3D neighborhood-attention Metal kernel and eager reference
11. aa18e5a — SIMD-group specialization for the model's head dimension of 64
12. ef28627 — absolute RoPE, NA blocks, SwiGLU, and causal pixel shuffle
13. e634684 — five-stage DiffVAE decoder, strict checkpoint loader, and CLI integration
14. 4775630 — receptive-field-halo spatial tiling
15. 5001b17 — DFR keyframe dual-stream joint attention
16. f9a458a — keyframe-aware spatial tiling
17. d29c248 — plain and keyframe-aware temporal tiling

Supported scope

• T2V
• I2V, including first/last-frame conditioning
• Synchronized audio-video generation
• A2V
• Combined audio and image conditioning
• Distilled two-stage pipeline
• Convolutional video VAE
• Automatic duration prediction from the prompt
• T2V prompt enhancement
• Reference-image-aware I2V prompt enhancement
• Prompt-driven multishot generation
• LTX-2.5 DFR with the convolutional video VAE
• LTX-2.5 diffusion video VAE, including DFR generated-keyframe decoding

Implementation

• Detect the official LTX-2.5 split checkpoint layout.
• Download only the five components required by the supported pipeline.
• Reuse Gemma 4 Unified from mlx-vlm>=0.7.1.
• Load the tokenizer, config, and text projections embedded in the text-encoder checkpoint.
• Build the 22B transformer from checkpoint metadata.
• Support independent FFN bias, gated-attention, and cross-attention AdaLN configuration.
• Apply the keyframe absolute-position embedding used by LTX-2.5.
• Use the rectified-flow ancestral Euler sampler for LTX-2.5 stage 1.
• Load the split video VAE, audio VAE, and BigVGAN + BWE vocoder.
• Strict-load the 3.8 MB DurationHead and predict seconds from video/audio connector outputs.
• Clamp predicted duration and snap it to the causal VAE's 8k+1 frame grid.
• Expose --auto-duration MIN_SECONDS MAX_SECONDS.
• Use a separate generative Gemma 4 E2B-it model for LTX-2.5 prompt enhancement.
• Expose --prompt-enhancer-repo and use the official LTX-2.5 T2V/I2V system prompts.
• Pass the reference image to the I2V prompt enhancer.
• Preserve the legacy layouts and deterministic sampler for LTX-2/LTX-2.3.

Multishot does not require a separate inference path. The README now includes a regular T2V example that describes shots in order, repeats subject and wardrobe details, and allocates enough frames for the sequence. Shot lengths and cut positions are model-controlled, and an explicit hard cut may be rendered as a smooth transition.

DFR adds generated keyframe slots on the official 24/32-frame segment grid and jointly denoises the video, audio, and slots during stage 1. It spatially upscales the video and generated keyframes, then runs stage 2 with the half-resolution video appended as clean reference tokens. The loader reads the reference downscale factor from the official detailing adapter metadata and merges its 480 LoRA pairs at a default strength of 0.5.

The CLI exposes --pipeline dfr, --detailing-lora, and --detailing-lora-strength. When the internal DFR canvas is padded to a segment boundary, the decoded video and generated audio are trimmed back to the requested duration.

The documented mlx_video.generate command is now exposed as a project-script alias while preserving the existing mlx_video.ltx_2.generate entry point.

The diffusion video decoder implements the official five-stage 2048/1024/512/512/256-channel architecture. Its final stage runs eight 11x11x11 neighborhood-attention blocks. The fused Metal kernel uses BF16 inputs, float32 accumulation, and online softmax without materializing an attention-score tensor; a 32-lane SIMD-group path specializes the model's head dimension of 64.

The loader splits fused checkpoint QKV tensors into independent projections and strict-loads the 396-tensor checkpoint. Select it with --video-decoder diffusion; the convolutional decoder remains the default lower-memory path.

With DFR, generated keyframe latents pass through all five decoder stages. Each video query attends to its local 3D window and the spatial windows of its two nearest keyframe planes. Each keyframe query attends to its own plane and two nearest video frames in the same online softmax.

--diffusion-vae-spatial-tiles and --diffusion-vae-temporal-tiles split stage 4/5 with receptive-field halos. Spatial tiling uses a 24-cell stage-4-input halo. Temporal tiling uses a 22-cell halo and preserves the causal leading-frame phase.

Mux fix

The existing FFmpeg -shortest path could truncate video frames when audio was slightly shorter than the video. In the reproduced case, a 25-frame video became 21 frames.

The audio stream is now padded with apad, preserving every video frame while still ending the mux at the video duration.

Not yet supported

• Dev pipeline
• DFR image conditioning, A2V, streaming, and temporal upscaling
• Combining spatial and temporal DiffVAE tiling in the same decode

Verification

Measured on a Mac Studio M4 Max with 128 GB unified memory at 768x512, 121 frames, and 24 fps:

Mode	Time	Peak memory
T2V	108.7 s	37.81 GB
I2V	120.8 s	39.54 GB
Generated audio	118.1 s	37.81 GB
A2V	106.4 s	37.81 GB

• Every video is H.264 with 121 frames.
• Generated audio is AAC 48 kHz stereo.
• A2V audio is AAC 16 kHz stereo.
• Automated image and audio checks found no NaNs or infinities.
• 55 related tests pass.
• Fresh Python 3.12 install and CLI imports pass.
• LTX-2 regression generation passes.
• A short prompt predicted 4.72 seconds / 113 frames; a longer prompt predicted 5.28 seconds / 121 frames.
• End-to-end generation with omitted num_frames produced a 113-frame MP4 in 29.0 seconds at a 37.03 GB peak.
• Omitting num_frames with legacy LTX-2 still produces the existing 33-frame default.
• T2V enhancement produced a caption containing framing, camera motion, lighting, and soundscape details.
• I2V enhancement correctly grounded the caption in the reference image's magenta pixel-art cat, black background, white ears, and black eyes.
• Enhancement + duration prediction + generation produced 57 frames in 26.8 seconds at a 36.97 GB peak.
• I2V enhancement + generation completed in 32.3 seconds at a 37.08 GB peak.
• At 768x512 / 241 frames / 24 fps with the same prompt and seed, LTX-2.5 generated a medium shot, close-up, and warm-lit wide shot while preserving the subject and wardrobe in 219.1 seconds at a 39.55 GB peak.
• The legacy LTX-2 comparison also preserved the subject and wardrobe, but followed the requested three-shot structure less closely; it completed in 196.3 seconds at a 39.07 GB peak.
• At 768x512 / 121 frames / 24 fps, convolutional-VAE DFR completed in 179.6 seconds at a 41.25 GB peak. The same prompt and seed on the regular distilled path took 103.4 seconds at a 37.81 GB peak.
• The DFR output showed finer fur, edge, and grass detail and more stable subject shape over time.
• The initial DFR path supports T2V and optional generated audio. I2V, A2V, streaming, temporal upscaling, and diffusion-VAE decoding are not yet supported.
• DFR + generated audio + automatic duration completed end to end. The DurationHead predicted 4.88 seconds / 113 frames; DFR padded its internal canvas to 121 frames and produced an exact 113-frame / 4.708-second MP4 plus a 48 kHz stereo WAV trimmed to 4.708 seconds. The muxed AAC stream was 4.693 seconds and did not truncate the video.
• Automated checks found no NaNs, infinities, static/gray video, or silent audio in that output. Generation took 187.5 seconds at a 41.25 GB peak.
• Legacy LTX-2 distilled regression generation passes after the DFR changes.
• 22 selected LTX-2.5 tests, including the new DFR tests, pass.
• A fresh Python 3.12 install, DFR imports, and the documented CLI alias/options pass.

DiffVAE measurements on the same Mac Studio M4 Max with 128 GB unified memory at 768x512 / 121 frames / 24 fps:

Decoder path	Time	Peak memory
Convolutional VAE	103.4 s	37.81 GB
DiffVAE, untiled	149.2 s	51.33 GB
DiffVAE, 2x2 spatial tiles	216.9 s	37.81 GB
DiffVAE, 2 temporal tiles	185.4 s	46.21 GB
DFR + keyframe DiffVAE	241.1 s	49.71 GB
DFR + keyframe DiffVAE, 2x2 spatial tiles	331.0 s	41.25 GB
DFR + keyframe DiffVAE, 2 temporal tiles	292.4 s	46.24 GB

• Small full/tiled fixtures match numerically. Real-video comparisons do not show an error concentration at spatial or temporal tile seams.
• Representative DiffVAE outputs contain exactly 121 finite, non-static frames.
• Final combined regression covered DFR generated keyframes + DiffVAE + 2x2 spatial tiling + generated audio + automatic duration. It predicted 4.88 seconds / 113 frames, padded the internal canvas to 121 frames, and produced an exact 113-frame / 4.708-second MP4 and a 48 kHz stereo WAV of the same duration in 328.1 seconds at a 41.25 GB peak.
• DiffVAE I2V and A2V generation pass at 256x256 / 25 frames.
• Legacy LTX-2 convolutional-VAE regression generation passes.
• All 40 test_ltx25_* tests pass.
• A fresh Python 3.12 install, DiffVAE imports, the documented CLI alias, and all decoder/tiling options pass.

The repository-wide test suite currently stops during collection because the existing tests/test_generate_dev.py imports the removed mlx_video.generate_dev module. This PR does not change that unrelated test.

Model and license

Weights are not redistributed. The loader downloads the required components from Lightricks/LTX-2.5; users must accept the LTX-2.x Community License on Hugging Face first.

The DFR detailing adapter is downloaded separately from the gated Lightricks/LTX-2.5-22b-IC-LoRA-Pixel-Spatial-Upscaler repository and is not redistributed.

The diffusion decoder checkpoint is downloaded from the gated LTX-2.5 repository only when --video-decoder diffusion is selected and is not redistributed.

Prompt enhancement separately uses mlx-community/gemma-4-e2b-it-bf16. The LTX-2.5 Gemma 4 Unified text-encoder checkpoint is encode-only, so this follows the official implementation by keeping the generative instruct checkpoint separate.