v0.19.7 Patch Release

What's Changed

• Add configurable dtype for ZeRO checkpoint export by @gaoxiaomo in #8318
• Raise the per-tensor norms to norm_type when combining them by @vineethsaivs in #8313
• Fix the seq-first Ulysses all2all output layout by @vineethsaivs in #8317
• Count each module object once when aggregating flops profiler totals by @ebarkhordar in #8320
• Fallback for unsupported Hybrid Engine policies by @LiRunGuo in #8265
• [tiled mlp] reshape instead of view by @stas00 in #8348
• Add pin_empty helper for empty scratch destinations by @sfc-gh-truwase in #8332
• Fix Triton NFS detection crash when df wraps long device names by @atirna in #8259
• [AutoTP] Replace tp_shard process-wide globals with per-model AutoTPMeta by @delock in #8241
• feat(cpu-adam): add ARM SVE update kernel by @xylian86 in #8365
• Add macOS (MPS) CI workflow and a torch floor check for the MPS accelerator by @PKUWZP in #8335
• Add an opt-in DeepEP transport for the AutoEP expert all-to-all by @yh0903 in #8213
• remove the dead non-Triton attention path from TritonSelfAttention by @Anai-Guo in #8349
• perf(rollout): profile prefill and decode forwards by @nathon-lee in #8350
• DeepCompile: stabilize ZeRO-3 parameter guards by @tohtana in #8328
• fix: Only bind device id when needed, Fixes #8248 by @pengdurice in #8269
• Make the WarmupCosineLR ratio flags reach the config by @vineethsaivs in #8268
• Forward free_data through partition() instead of hardcoding True by @vineethsaivs in #8305
• Recognize Qwen3.5's RMSNorm variants in AutoTP module loading by @promptsmith1990 in #8306
• Forward barrier device_ids to communication backends by @tohtana in #8312
• Fix flops profiler counts for transposed convolutions by @vineethsaivs in #8323
• Broadcast elementwise flops from the trailing dimension by @vineethsaivs in #8324
• Replace cuda_graph assertion with explicit ValueError; fix custom_op type annotations by @chakshu-dhannawat in #8336
• Keep the elasticity batch overrides out of the caller's config dict by @alanhuangyoo in #8329
• Stop the curriculum schedule starting below min_difficulty by @vineethsaivs in #8334
• Add an opt-in fused weighted restore for AutoEP by @yh0903 in #8326
• Add nccl_version to the source-checkout torch_info fallback by @alanhuangyoo in #8383
• docs: add test discipline rules to agent guidelines by @delock in #8372
• Preallocate the static KV cache with config.head_dim by @ebarkhordar in #8389
• Skip fp16-config tests on accelerators without fp16 support by @delock in #8398
• Do not pin DDP device_ids for CPU reference models by @delock in #8399
• Use device names, not rank ids, for device placement in test helpers by @delock in #8397
• [Ulysses] Carry the KV head count per DistributedAttention by @alanhuangyoo in #8316
• [Workflow] Raise modal CI timeouts to absorb slower sandbox provisioning by @delock in #8403
• Preserve AutoEP score correction bias buffers by @poorpaper in #8369
• op_builder: use C++20 for nvcc on CUDA 13+ by @bheu in #8422
• Reshape instead of view in TiledFusedLogitsLoss by @alanhuangyoo in #8362
• Fix AutoTP + deep compile collectives silently drop when AC is on by @pengdurice in #8355
• fix(lr_schedules): make --lr_range_test_staircase an opt-in flag by @Anai-Guo in #8337
• Split the modal CI budget into acquisition and test phases by @delock in #8404
• [Workflow] run modal GPU workflows only from the merge queue by @delock in #8412
• Read rope_theta from rope_parameters in the Llama injection policy by @alanhuangyoo in #8341
• Stop the debug name maps from pinning the model they snapshot by @alanhuangyoo in #8356
• Muon runs no Newton-Schulz at ZeRO stage 0, the default: run it by @alanhuangyoo in #8442
• Filter --include against the real slots, not against itself by @vineethsaivs in #8239
• Add an opt-in ZeRO-1/2 gradient norm fast path by @yh0903 in #8331
• [AutoEP]Fix optimizer and replaced MOE parameter mismatch by @pengdurice in #8377
• Muon is silently disabled under ZeRO-3 when the model is built with zero.Init by @alanhuangyoo in #8438
• Avoid collective token preparation for AutoEP DeepEP by @yh0903 in #8423
• [MPS] Update C++ Standard in CPUAdamBuilder by @jinyouzhi in #8466
• Split DeepCompile ZeRO-3 memory scheduler by @tohtana in #8233
• Describe universal checkpoint shards as affine maps by @Achyuthan-S in #8385
• Fix comms logger KeyError when log_name is omitted by @jinyouzhi in #8267
• Fix universal checkpoint resume across AutoTP sizes by @0z5a in #8474
• feat(rollout): add continuous batching generation prototype by @nathon-lee in #8368
• Fix ZeRO parameter alignment for grouped_mm by @fwerkor in #8277
• Carry the affine scale on the replicated map, not the split by @Achyuthan-S in #8477
• [muon] Reconcile the momentum dtype when a checkpoint is restored by @alanhuangyoo in #8433
• Deprecate unused DeepSpeed features by @sfc-gh-truwase in #8490
• [muon] Keep the momentum out of steps the loss scaler discards by @alanhuangyoo in #8435
• Remove triton compatibility check for fp_quantizer by @hwchen2017 in #8492
• Muon silently discards the param groups it is given by @alanhuangyoo in #8440
• [Phase 2] Add NEON SIMD path for CPU Adam on AArch64 by @PKUWZP in #8453
• Fix sequence overlap backward gradient permutation by @taking-lying-flat in #8342
• Validate positive inference and HybridEngine max output tokens by @taking-lying-flat in #8343
• Offload a saved view when it is the last value holding its storage by @pengdurice in #8388
• Honor --include/--exclude in the SLURM launcher by @vineethsaivs in #8304
• Make wait() idempotent on AllGatherHandle and NoGatherHandle by @pengdurice in #8487
• Deprecate sparse attention by @hwchen2017 in #8493
• feat(rollout): add continuous batching profiling by @nathon-lee in #8494
• Probe the device module for train_cifar's fork_rng device entries by @delock in #8407
• Gate the offload-state memory deltas on allocator-backed stats by @delock in #8409
• Read rope_theta from rope_parameters across Inference V2 by @alanhuangyoo in #8345
• Partition AutoEP expert parameters per layer under ZeRO-3 by @tohtana in #8424
• Give Muon's momentum the dtype of the gradient it is combined with by @alanhuangyoo in #8483
• Update version.txt after 0.19.6 release by @loadams in #8333
• Add DCO sign-off to release commits by @loadams in #8527
• Emit affine maps from AutoTP layers by @Achyuthan-S in #8519
• Cache the Modal sandbox...

v0.19.6 Patch Release

What's Changed

• Fix checkpoint rank selection for Ulysses sequence parallelism by @sfc-gh-thonguyen in #8226
• Return a copy from OnebitLamb.get_lamb_coeffs by @vineethsaivs in #8227
• Update version.txt after 0.19.5 release by @loadams in #8243
• Implement the documented per-param-group lists in OneCycle by @vineethsaivs in #8201
• Keep the elastic batch size within max_train_batch_size by @vineethsaivs in #8237
• [DeepCompile] Fix KeyError on frozen parameters in ZeRO-3 by @roycho96 in #8214
• [DeepCompile] Fix staticmethod handling on Python 3.9 by @baremetaldevx86 in #8240
• Upgrade pip in Python install-smoke jobs by @tohtana in #8247
• (1/2) Implementing Compiler Pass for AutoTP by @therealnaveenkamal in #8204
• Fix DeepCompile last use for unconsumed waits by @tohtana in #8254
• Add configurable sum gradient reduction by @zupengwang in #8232
• Add native (DeepNVMe) host-memory pinning backend for accelerators by @sfc-gh-truwase in #8211
• Add compile.offload_activation_pin_memory for DeepCompile activation offload by @sfc-gh-truwase in #8258
• Fix int32 overflow in Triton grouped-GEMM expert offset by @PKUWZP in #8261
• NCCL and other backend PG timeout 30min -> 10min by @stas00 in #8253
• Fix ZeRO-3 crash in AutoTP universal-checkpoint metadata by @PKUWZP in #8270
• Add CUDA graph support for HybridEngine generation by @PKUWZP in #8271
• Fix docstring Args entries that name a parameter the function does not take by @darkdi in #8223
• Keep parameter dtype through ZeRO-3 weight quantization by @adityasingh2400 in #8215
• [AutoTP] Complete uneven sharding and universal checkpoint support by @jinyouzhi in #8185
• Route FPDT and checkpoint writer pins through accelerator pin_memory by @sfc-gh-truwase in #8257
• Add fused triton kernel for swiglu by @hwchen2017 in #8244
• Make pass contracts cover the passes DeepCompile actually schedules; Renaming files by @pengdurice in #8251
• Non-reentrant activation checkpoint CPU offload by @sfc-gh-truwase in #8282
• Enable activation offloading by @pengdurice in #8255
• [typo] Add raise for RuntimeError by @jinyouzhi in #8281
• Fix ZeRO-1/2 with zero-sized parameters by @fwerkor in #8280
• docs: note async cpu_checkpointing perf and expandable_segments by @sfc-gh-truwase in #8287
• Fix ZeroDivisionError in compute_elastic_config return_microbatch on non-0.2 elasticity by @ebarkhordar in #8286
• Fix repeated FlopsProfiler metric accumulation by @baremetaldevx86 in #8246
• fix: stop DeepSpeedConfig writing max_grad_norm back into the caller's config dict by @ebarkhordar in #8289
• Enable DeepSpeed support on Apple Silicon (MPS) with ZeRO Stage 1-3 by @PKUWZP in #8293
• fix(engine): resolve inference workspace attribute lookup by @nathon-lee in #8288
• [AutoTP] Convert HF embedding_rowwise tp_plan entries to SKIP specs by @ebarkhordar in #8294
• Fix zero reduce bucket size validation by @fwerkor in #8266
• Register native pinned host memory with CUDA for GPU DMA by @sfc-gh-truwase in #8283
• Drop the documented grad_hooks ZeRO option, which does not exist by @vineethsaivs in #8242
• Skip zero-sized parameters in HP fragment mapping by @fwerkor in #8298
• perf(rollout): add HybridEngine rollout profiling by @nathon-lee in #8295
• Route isend/irecv to nonblocking backend methods and stage them as async on MPS by @PKUWZP in #8303
• Fix device mismatch in test_gate_up_partition_covers_the_whole_weight by @alanhuangyoo in #8308
• Import print_dist in auto_tp by @alanhuangyoo in #8311
• Reject invalid ZenFlow ratio and update interval boundaries by @tandede in #8274
• Avoid redundant copies in MPS P2P staging by @FU-max-boop in #8314
• Fix AutoTP metadata updates for unsharded modules by @zupengwang in #8299
• [Apple Silicon Support Phase 1] Add Metal FusedAdam kernel and CPU Adam build for Apple Silicon by @PKUWZP in #8300
• Enable optimized Adam backend for MuonWithAuxAdam optimizer by @jinyouzhi in #8278
• feat(opsd): share prompt prefill across rollout samples by @nathon-lee in #8296
• stop allocating per-element temporaries in the overflow check by @stas00 in #8325
• Fix shared loss gradient accumulation by @tohtana in #8245
• Isolate DeepCompile list-schedule test ops by @tohtana in #8319
• Fix ZeRO-3 synchronization during OPSD rollout by @LiRunGuo in #8264
• [AutoTP] Fix training lm_head routing by @gaoxiaomo in #8302
• Route ZeRO/SuperOffload pin sites through accelerator pin_memory by @sfc-gh-truwase in #8256
• Honor adam_w_mode in the CPU multi_tensor_adam binding by @PKUWZP in #8307

New Contributors

• @sfc-gh-thonguyen made their first contribution in #8226
• @baremetaldevx86 made their first contribution in #8240
• @darkdi made their first contribution in #8223
• @fwerkor made their first contribution in #8280
• @alanhuangyoo made their first contribution in #8308
• @tandede made their first contribution in #8274
• @FU-max-boop made their first contribution in #8314
• @LiRunGuo made their first contribution in #8264
• @gaoxiaomo made their first contribution in #8302

Full Changelog: v0.19.5...v0.19.6

v0.19.5 Patch Release

What's Changed

• Update version.txt after 0.19.4 release by @loadams in #8219
• Restore the custom DCO workflow by @tohtana in #8220
• Update actions versions for workflows. by @loadams in #8228
• Do not let one op builder's compatibility probe break importing deepspeed by @adityasingh2400 in #8216
• Fix ZeRO++ secondary shard copy for small params by @ZyEng-Art in #8210
• fix(zero3): async grad offload + pinned offload buffers by default by @delock in #8207
• Fix AutoEP ZeRO-1/2 universal conversion by @tohtana in #8198
• Fix compilation error for torch 2.12+ by @hwchen2017 in #8238
• Split native host pinning into standalone pin_memory op by @sfc-gh-truwase in #8236
• Unmanaged gradient accumulation: ZeRO offload support by @sfc-gh-truwase in #8225

New Contributors

• @ZyEng-Art made their first contribution in #8210

Full Changelog: v0.19.4...v0.19.5

v0.19.4 Patch Release

What's Changed

• Validate warmup_type in WarmupCosineLR like WarmupLR by @sohumt123 in #8151
• Run pull request code only inside a Modal Sandbox by @tohtana in #8170
• [AutoTP] Allow ZeRO stage 3 inference with tensor parallelism by @delock in #8167
• Fix autotuning get_val_by_key to search all nested subdicts by @ebarkhordar in #8177
• Enable support for Tutel when k != 1 for shared moe by @jinyouzhi in #8174
• NVMe write warning by @sfc-gh-truwase in #8164
• Guard LRRangeTest and OneCycle schedulers against zero step sizes by @ebarkhordar in #8166
• Fix WarmupLR collapsing multi-group base LRs to group 0's by @ebarkhordar in #8171
• Stabilize fork-sensitive nightly tests and AutoSP coverage by @tohtana in #8172
• Make the OneCycle stair counts actually change the schedule by @vineethsaivs in #8179
• [AutoTP] Enable HF colwise_gather_output to support lm_head replace by @jinyouzhi in #8146
• Scope DeepCompile compiler state to graph and engine lifecycles by @tohtana in #8159
• Clarify merge commit exemption for signoff check in AGENTS.md and CLAUDE.md by @jinyouzhi in #8194
• [AutoTP] Preserve universal checkpoint metadata for HuggingFace tp_plan by @jinyouzhi in #8187
• Derive AutoEP rank splits from the per-expert count exchange by @yh0903 in #8190
• Warn when zero.Init silently falls back to a single rank (#8084) by @akshansh47 in #8089
• Release GIL in blocking AIO bindings by @tohtana in #8196
• Enable ZeRO-3 linear wrapper for existing models by @tohtana in #8189
• Skip CUDA transformer forward tests when TransformerBuilder is incompatible by @sfc-gh-truwase in #8200
• Clamp capacity to num_tokens in MoE gating drop branches (complete #5353) by @ebarkhordar in #8155
• Preserve tensor learning rates across scheduler updates by @n33levo in #8202
• [AutoTP] Fix ZeRO-3 checkpoint consolidation to gather across TP and DP by @delock in #8168
• Add managed_gradient_accumulation for ZeRO stage 0/1 by @sfc-gh-truwase in #8184
• MoE dispatch: fix silent Tutel + TP corruption, speed up native and Tutel paths by @jinyouzhi in #8195
• Add keyword argument support to activation checkpointing by @zupengwang in #8182
• Remove redundant DCO workflow by @tohtana in #8188
• Preserve tensor learning rates in OneCycle by @tohtana in #8205
• Add Triton grouped-GEMM for MoE experts on Ampere/Ada by @hwchen2017 in #8180
• Wiring up offload_opt_states by @pengdurice in #8186
• Fix DeepCompile ZeRO-3 gathered parameter ownership by @tohtana in #8157
• Consolidate LR scheduler tensor-LR tests into a contract suite by @delock in #8206
• Fix MLU_Accelerator conformance to the DeepSpeedAccelerator ABC by @ebarkhordar in #8208
• Reduce redundant work in AutoEP token routing by @hwchen2017 in #8209
• Make PipelineModule.set_checkpoint_interval actually change the interval by @vineethsaivs in #8178
• Share DeepNVMe pinned-tensor manager and route swap buffers through I/O handles by @sfc-gh-truwase in #8212
• Stop trim_mean sorting the caller's list in place by @vineethsaivs in #8199
• Unmanaged gradient accumulation: ZeRO stage 2 support by @sfc-gh-truwase in #8203
• Fix ZeRO-3 all_reduce param fetch stride for padded parameters by @ebarkhordar in #8158
• Unmanaged gradient accumulation: ZeRO stage 3 support by @sfc-gh-truwase in #8217
• Update version.txt pre-0.19.4 release by @loadams in #8218

New Contributors

• @sohumt123 made their first contribution in #8151
• @ebarkhordar made their first contribution in #8177
• @yh0903 made their first contribution in #8190
• @akshansh47 made their first contribution in #8089
• @n33levo made their first contribution in #8202
• @zupengwang made their first contribution in #8182
• @pengdurice made their first contribution in #8186

Full Changelog: v0.19.3...v0.19.4

v0.19.3 Patch Release

What's Changed

• Update version post release by @loadams in #8070
• Validate fp16 dynamic loss scaling parameters are positive by @aryanputta in #8050
• Add merge queue DCO workflow by @tohtana in #8086
• Stop obsolete CI jobs on workflow cancellation by @tohtana in #8034
• Fix ZeRO-3: Use per-param dtype for output buffers in _allgather_params_coalesced by @albertvillanova in #8073
• fix: close file descriptor in deepspeed_io_handle_t::wait() to prevent fd leak by @MarkCLChang in #8075
• Add type hints to top-level public API functions by @arunshar in #8079
• Default gradient_clipping to 1.0 by @sfc-gh-truwase in #8068
• activation_checkpointing: default num_layers to None so configure() assert fires by @Kymi808 in #8041
• Add configurable engine log level by @sfc-gh-truwase in #8067
• [CI] diff driven test selection by @stas00 in #8077
• Enable bf16 check_grad_overflow by default (matching fp16) by @yongzhe-wang in #8035
• Support AutoEP with ZeRO-3 zero.Init source modules by @tohtana in #8060
• Update publication list in README.md by @PKUWZP in #8095
• fix: AutoTP partition_config uses full hierarchical module path by @delock in #8088
• Reject Muon optimizer with reduce_scatter in ZeRO-1/2 by @whycoming in #8090
• Fix DeepCompile profile metadata backfill by @tohtana in #8094
• Add numerical-correctness test for Muon under ZeRO-1/2 by @whycoming in #8091
• Don't swallow KeyboardInterrupt/SystemExit in NPUOpBuilder by @ajinkyajawale14499 in #8096
• Avoid CUDA context initialization during op compatibility checks at import by @Achyuthan-S in #8078
• [DeepCompile] fix gather params in dynamo skipped frames for ZeRO3 by @XAheli in #8059
• Feat: zero3 deprecate elastic checkpoint by @nathon-lee in #8099
• Fix eigenvalue parsing for compression-only quantize configs by @sowndappan5 in #8057
• fix: log eigenvalue monitor values by @he-yufeng in #8049
• Fix DeepCompile profiling memory cleanup by @tohtana in #8106
• Add optional torchembed RoPE backend to apply_rotary_pos_emb by @py-ai-dev in #8052
• Fix minor comment/docstring typos in runtime and inference modules by @nathon-lee in #8046
• feat(zenflow): run the overlapped CPU optimizer in a native process by @Antlera in #8058
• fix: add buffer-length check in shm.cpp by @orbisai0security in #8082
• Fix FlopsProfiler crash when dp_world_size is None under sequence parallelism by @vineethsaivs in #8122
• Fix Python 3.9 import-time TypeError in AutoEP ep_router by @vineethsaivs in #8119
• Add input validation for save_checkpoint and warmup LR schedulers by @jahnavi-yelamanchi in #8126
• Fix ZeRO-3: use per-param dtype for quantize-scale buffer in _allgather_params_coalesced by @roycho96 in #8128
• Fix ZeRO-3 autocast gather with mixed parameter dtypes by @tohtana in #8113
• doc: Remove suggestion to build extensions in parallel by @Flamefire in #7899
• feat(types): add type hints to public comm API functions by @NovusEdge in #8118
• Add Hybrid Engine rollout in DeepSpeed to support On-Policy Distillation (OPSD) Trainer by @PKUWZP in #8027
• Add AutoEP + AutoTP parallel folding by @tohtana in #8064
• Make DCO workflow Probot compatible by @tohtana in #8110
• ZeRO 1/2: wait on all IPG-bucket producer streams in average_tensor (#8061) by @arunshar in #8080
• Fix CommsLogger.stop_profiling_comms not disabling global profiling by @vineethsaivs in #8137
• Guard WarmupCosineLR against total_num_steps == warmup_num_steps (ZeroDivisionError) by @vineethsaivs in #8142
• Add EXAONE 4.5 model support for Inference V2 by @Bias92 in #8121
• fix(autotp): when using autotp, ignore the consistency of certain data within the tp_group. by @1787648106 in #8125
• Reduce blockDim in fake_quantize_kernel for improved SM occupancy by @flutist in #8115
• Fix DeepSpeedInferenceConfig crash on bool moe backward-compat value by @winklemad in #8145
• fix(exceptions): narrow broad Exception clauses to specific types by @sowndappan5 in #8136
• fix(muon): support ZeRO-1/2 reduce scatter by @Micuks in #8141
• [DeepCompile] Add lightweight pass contracts for optimization passes by @jahnavi-yelamanchi in #8139
• Fix DeepCompile fallback handling for scalar outputs by @tohtana in #8149
• Use synthetic CIFAR data in AlexNet tests by @tohtana in #8143
• Update CI badge by @tohtana in #8150
• Update Zhipeng Wang affiliation to Google in COMMITTERS.md by @PKUWZP in #8153
• Fix ZeRO-3 hooks for attribute-delegating modules by @Functionhx in #8144
• Fix repeated gradient scaling across pipeline stages by @Functionhx in #8154
• Z3: Support for activation ckpt with frozen params by @sfc-gh-truwase in #8148
• Adopt PTF Code of Conduct by @sfc-gh-truwase in #8165

New Contributors

• @aryanputta made their first contribution in #8050
• @albertvillanova made their first contribution in #8073
• @MarkCLChang made their first contribution in #8075
• @arunshar made their first contribution in #8079
• @Kymi808 made their first contribution in #8041
• @yongzhe-wang made their first contribution in #8035
• @whycoming made their first contribution in #8090
• @ajinkyajawale14499 made their first contribution in #8096
• @Achyuthan-S made their first contribution in #8078
• @XAheli made their first contribution in #8059
• @he-yufeng made their first contribution in #8049
• @py-ai-dev made their first contribution in #8052
• @vineethsaivs made their first contribution in #8122
• @jahnavi-yelamanchi made their first contribution in #8126
• @NovusEdge made their first contribution in #8118
• @1787648106 made their first contribution in #8125
• @winklemad made their first contribution in #8145
• @Micuks made their first contribution in #8141
• @Functionhx made their first contribution in #8144

Full Changelog: v0.19.2...v0.19.3

v0.19.2 Patch Release

What's Changed

• fix(fp16): filter requires_grad in FP16 optimizer flat buffer init by @avicooper1 in #8029
• Run AutoSP compile tests sequentially by @tohtana in #8020
• Fix PR-target workflow concurrency groups by @tohtana in #8017
• Fix full CI test isolation for ZeRO chmod and NVMe quantization tests by @tohtana in #8008
• Keep required CI checks visible for ignored paths by @tohtana in #8019
• Bump version by @sfc-gh-truwase in #8030
• Add engine.coalesce_grad_reduction() for ZeRO 1/2/3 multi-backward by @roycho96 in #7992
• feat(zero): enable torch.func transforms on engine for ZeRO 0/1/2 by @roycho96 in #8026
• Simplify module_inject.transpose by @xbcReal in #8028
• Fix DeepCompile all-gather scheduler candidate selection by @tohtana in #8033
• Version fix to unblock pypi by @sfc-gh-truwase in #8039
• Bump version after 0.19.1 release by @tohtana in #8040
• Fix DeepCompile ZeRO-3 release parameter lifetime by @tohtana in #8032
• Fix ZenFlow ZeRO-3 selective optimizer crash with parameter offload on nvme by @Antlera in #8042
• Add test coverage for Muon muon_lr/adam_lr overrides by @sowndappan5 in #8047
• Avoid HF Hub access in CPU unit test setup by @tohtana in #8053
• Fix DeepCompile ZeRO-1 grad target lifetime by @tohtana in #8036
• Normalize ZeRO-3 DeepCompile grad dtype before reduction by @tohtana in #8038
• Remove AutoSP assertion against Transformers version by @tohtana in #8044
• fix(transformer): use correct stride in Transpose_Kernel shared memory indexing to eliminate bank conflicts by @flutist in #8055
• zero3: invalidate coordinator trace on hook re-registration by @roycho96 in #8043
• Consistent fp32 grads flow by @sfc-gh-truwase in #8056
• Add AutoEP by @tohtana in #7938
• Fix: ZenFlow Adam integration for updated PyTorch backward flow (#7759) by @Antlera in #7771
• Pass expected grad dtype to register_z3_param in ZeRO-3 release test by @tohtana in #8063
• Add Biren SUPA accelerator support by @frozenleaves in #8054
• Mixed-precision: per-policy param/buffer dtype cast (preserve fp32 buffers) by @sfc-gh-truwase in #8066

New Contributors

• @avicooper1 made their first contribution in #8029
• @sowndappan5 made their first contribution in #8047
• @flutist made their first contribution in #8055
• @frozenleaves made their first contribution in #8054

Full Changelog: v0.19.1...v0.19.2

v0.19.1

What's Changed

• Update version post-v0.19.0 release by @loadams in #7996
• Add office hours times/link on the README by @loadams in #8004
• Update topkgating probability-mask test expectation by @tohtana in #8007
• Optimize singleton MoE collectives by @Tianyi-Franklin-Wang in #7997
• zero3: SDMA allgather via mori (sdma_allgather) by @inkcherry in #7999
• fix(io): close aio_fd in FastFileWriter._fini to prevent fd leak by @jg-heo in #8005
• Auto-detect CUTLASS for EvoformerAttention by @MaxTretikov in #8000
• fix: use subprocess instead of os.system in data_analyzer.py by @orbisai0security in #7994
• Fix ZeRO-3 forward crash on modules with plain dict _parameters by @roycho96 in #8009
• Remove stale step() docstring from DeepSpeedCPUAdam by @lucaspirola in #8011
• Add configurable torch-latest dependency versions by @tohtana in #8016
• Run FastFileWriter fd-close test outside pytest-forked by @tohtana in #8015
• Make GitHub Actions job names unique by @tohtana in #8014
• Support bf16 optimizer states with CPU offload by @lucaspirola in #8010
• [fix] fix test_zf.py hang bug by @xbcReal in #8012
• [Blog] Muon Optimizer Support in DeepSpeed by @delock in #7962
• fix gemma4 num attention head bugs (from #7975) by @delock in #7990
• fix: add setup_context for torch.func compatibility by @roycho96 in #7916
• Sort and dedupe -gencode flags emitted by op_builder.builder by @adityasingh2400 in #8021
• fix(zero): enable vmap on LinearFunctionForZeroStage3 by @roycho96 in #8023
• Support flash-attn 2.7.0 in FPDT attention by @xbcReal in #8022
• Fix DeepCompile AOT kwargs patching for PyTorch >= v2.11 by @tohtana in #8024

New Contributors

• @Tianyi-Franklin-Wang made their first contribution in #7997
• @jg-heo made their first contribution in #8005
• @MaxTretikov made their first contribution in #8000
• @orbisai0security made their first contribution in #7994
• @lucaspirola made their first contribution in #8011
• @xbcReal made their first contribution in #8012
• @adityasingh2400 made their first contribution in #8021

Full Changelog: v0.19.0...v0.19.1

v0.19.0

What's Changed

• Update version after latest release (v0.18.9) by @loadams in #7936
• Refactor consolidate transpose by @nathon-lee in #7934
• Fix/fix autotp universal checkpoint ci by @tohtana in #7937
• Fix process hang in process-group shutdown by @Flamefire in #7941
• Zero3 defragment utility by @nathon-lee in #7940
• [SP] add SP deny list instead of allow by @kashif in #7887
• fix(zero): detach flat buffer to prevent autograd inplace error on CP… by @delock in #7948
• Fix FPQuantizer build by @Flamefire in #7963
• Fix zero 1 and 2 CPU-offloaded gradient norm by @alek6kun in #7967
• Fix overlap-comm buffer lifetimes by @tohtana in #7965
• Fix DeepCompile+Z3 on PyTorch v2.9/2.10 by @tohtana in #7951
• Fix WarmupCosineLR multi-group initialization by @tohtana in #7969
• Enable PyTorch version selection for full test by @tohtana in #7968
• fix(fp_quantizer): fix UB and negative shift warnings in fp_quantize_impl.cu by @Cursx in #7973
• fix(op_builder): avoid duplicate/wrong -gencode flags by @Cursx in #7974
• Rename dequantization template parameters by @Flamefire in #7976
• Avoid CUDA reinit error in CI tests by @tohtana in #7977
• Fix ZeRO-1/2 CPU-offloaded gradient loss with multiple backward() per step by @roycho96 in #7981
• deepcompile: Fix backward graph recompilation due to unbalanced forward/backward visits by @eternalNight in #7980
• Fix Adam subgroup inconsistency by @st-bang97 in #7982
• Dynamic offload compatible with static optimizer offload by @sfc-gh-truwase in #7979
• Fix modal ci timeout by @sfc-gh-truwase in #7989
• Fix BF16_Optimizer last-microbatch grad leak under ZeRO-1 by @maxyu1115 in #7985
• fix: topkgating major bug by @excepshenal in #7986
• Add DeepSpeed NVTX domain support by @heurry in #7988
• Add Gram Newton-Schulz orthogonalization for Muon optimizer by @delock in #7953
• [AutoSP] (Sequence Parallelism) support for Multimodal Models (ViT + LLM) by @nathon-lee in #7984
• Update version.txt before 0.19.0 release by @loadams in #7995

New Contributors

• @alek6kun made their first contribution in #7967
• @Cursx made their first contribution in #7973
• @roycho96 made their first contribution in #7981
• @st-bang97 made their first contribution in #7982
• @maxyu1115 made their first contribution in #7985
• @excepshenal made their first contribution in #7986
• @heurry made their first contribution in #7988

Full Changelog: v0.18.9...v0.19.0

v0.18.9 Patch Release

What's Changed

• Respect $TRITON_HOME by @Flamefire in #7907
• Add Feature Universal Checkpoint for AutoTP by @nathon-lee in #7908
• fix: remove unnecessary shell=True in ROCm GPU architecture detection by @instantraaamen in #7915
• Don't detect local GPU if $DS_IGNORE_CUDA_DETECTION is set by @Flamefire in #7896
• Add HuggingFace tp_plan support for AutoTP by @delock in #7901
• fix: handle non-existent path in is_nfs_path for Triton autotune cache by @Krishnachaitanyakc in #7921
• Fix backward compatibility of torch.amp.custom_fwd for PyTorch < 2.4 by @tohtana in #7920
• Extending Muon Optimizer Support for ZeRO Stage 3 by @PKUWZP in #7919
• Add news item for ASPLOS 2026 Best Paper Award by @PKUWZP in #7923
• fix(superoffload) preserve multi-group updates with shared cpu buffers (#7905) by @xylian86 in #7906
• AGENTS.md: Add pre-commit command to existing CI requirements line by @delock in #7930
• Update README with latest news from DeepSpeed by @PKUWZP in #7931
• Merging AutoSP into DeepSpeed by @neeldani in #7860
• Add fallback to full test by @tohtana in #7933
• Remove Microsoft Corporation copyright from AGENTS.md and CLAUDE.md by @PKUWZP in #7932
• Update version.txt for latest incoming release 0.18.9 by @loadams in #7935

New Contributors

• @instantraaamen made their first contribution in #7915
• @Krishnachaitanyakc made their first contribution in #7921
• @neeldani made their first contribution in #7860

Full Changelog: v0.18.8...v0.18.9

v0.18.8 Patch Release

What's Changed

• Suppress see_memory_usage logs by @sfc-gh-truwase in #7891
• [Bloom] Fix hangs of bloom test by @k-artem in #7890
• double reduction user-friendly error by @stas00 in #7895
• Fix async_io ops building error on Huawei Ascend NPU by @huangyifan0610 in #7894
• Fix Evoformer's multi-arch dispatch root cause by @tohtana in #7881
• fix: Validate fp16.loss_scale is finite and non-negative by @nathon-lee in #7889
• Add AGENTS.md and CLAUDE.md with project rules for AI coding agents by @delock in #7902
• fix(zero3): use current_stream() instead of default_stream() for grad… by @michaelroyzen in #7898
• Update version by @loadams in #7903

New Contributors

• @huangyifan0610 made their first contribution in #7894
• @michaelroyzen made their first contribution in #7898

Full Changelog: v0.18.7...v0.18.8