[ROADMAP][2026 Q3] Megatron Core Roadmap

[sbhavani]

This roadmap outlines the key Megatron Core features, enhancements, and improvements planned for Q3 2026. The 26.08 release has been delivered; remaining items track ongoing Q3 work. This is a tentative roadmap and subject to change.

Last updated: September 8, 2026.

For detailed information on past releases, see the Megatron Core release notes. For the previous quarterly roadmap, see Q2 2026 Roadmap #4997. For component-specific plans and status, see the Q3 MoE Roadmap #6757 and Q3 Megatron FSDP v2 Roadmap #6812.

---

Q3 Roadmap (26.08 delivered; ongoing Q3 work)

MoE

• DeepSeek-V4 architecture support - Integrate CSA/HCA hybrid attention, DSA, and mHC; support Hash MoE and ClampedSwiGLU; add MTP with mHC, fused kernels, Muon/AdamW training recipes, packed sequence handling, and long-context validation (tracker #4468, native hybrid orchestration #6402, Hash MoE #6403, fused mHC #6401; foundational dev PRs #4458, #4481, and #4518).
• Hash MoE and ClampedSwiGLU - Implement Megatron Core support for DeepSeek-V4-specific routing and activation functions (#4481, #5130, #6403).
• MoE communication overlap - THD dispatcher support is delivered (#5008); latent shared-expert overlap and DeepEP V2 ElasticBuffer work continue (#5135, #4632).
• Optimized MoE kernels - Optimize Blackwell performance by integrating GEMM + SwiGLU fused MLP, mHC, and batchedGEMM fusion techniques.
• MegaMoE / ultra-sparse MoE - Research expanded expert counts, sparsity-driven activations, predictive routing mechanisms, and adaptable recomputation strategies (Megatron-LM #6810, Transformer Engine #3350).
  • MOK implementation - Integrate Mixture-of-Kittens as a pluggable MoE megakernel backend (#6572).

Model Architecture and Support

• DeepSeek-V4 recipes - Complete recipes with end-to-end convergence validation across Megatron-LM and Megatron Bridge.
• Long-context validation - Verify convergence for 4K/16K dense and 64K/1M sparse training setups.
• HybridModel - Introduce a heterogeneous-layer model definition for DeepSeek-V4 and future hybrid configurations, replacing GPTModel (#4537, #5030, #5031, #4538, #4539).
• Nemotron 3 on HybridModel - Make Nemotron 3 native to HybridModel.
• * DeepSeek-V4 transition to HybridModel - Move DeepSeek-V4 development from GPTModel to the new HybridModel (#5042).
• Qwen3-Next support — delivered - Support the GDN-based hybrid architecture, attention and shared-expert output gates, QK LayerNorm weight-decay handling, context parallelism, and packed sequences (#1989, #2751, #2752, #2753, #2642, #2645).
  • GDN architecture and parallelism — delivered - Support Gated Delta Net for Qwen3-Next, including context parallelism, packed sequences, and unified per-sequence AlltoAll (#1989, #2642, #2645, #4913).
  • Fused GDN attention — in progress - Integrate the Transformer Engine fused GDN kernels into Megatron Core (Megatron-LM #6645, Transformer Engine #3351).
• Qwen3.5 / Qwen3.5-VL support — in progress - Add model architecture logic, packed sequence handling, Qwen3.5-VL validation, and bidirectional checkpoint conversion between Hugging Face and Megatron Core. Foundational dev work has landed (#4748, #4749, #4750, #4751); HybridModel migration and pipeline-parallel integration continue (#6315, #6081).
• SFTDataset class - Add a new pre-tokenized, packed implementation (#5017).

Parallelism

• Megatron FSDP - Provide an FSDP2-compatible API, communication-compute overlap, activation recompute support, and pooled memory allocation.
• NCCL EP support — delivered - Add an expert-parallel transport path and dispatch-manager support (#5129, #4644).
• Next-generation dynamic DP - Support multimodal data and improve performance.
• Communication enhancements / DeepEPv2 - Integrate the DeepEPv2 token dispatcher and enable NCCL EP support.

Performance and Memory

• Deterministic training - Track fixed-environment repeatability, baseline equivalence, restart replay, scale validation, and performance overhead (#5785).
• HybridModel 1F1B overlap — in progress - The common schedule-plan foundation is delivered (#4941); grouped HybridStack support, EP-overlap scheduling, FSDP, and training integration continue (#4942, #4943, #4944).
• Checkpoint load performance - Reduce cross-rank reads for torch_dist load (#4628).
• Reduce MoE peak memory - Avoid full intermediate list/cat buffers and chunk the return all-to-all/unpermute path.
• Full-model CUDA Graph with paged stashing - Advance paged stashing to enable full-iteration CUDA Graph capture and optimize memory for dropless MoE configurations.
• Enhanced attention mechanisms - Advance kernel development for GDN, NSA, and associated Transformer Engine or cuDNN Frontend integrations.
• GDN memory optimization - Implement fine-grained activation offloading and selective recompute strategies for in-projection, conv1d, and gated delta rule logic.

Precision and Optimizers

• MXFP8 and NVFP4 training paths - Continue improving MXFP8/NVFP4 parameter gathering and add support for low-precision GroupedGEMM.
• Low-precision guide - Document practical low-precision training recipes and optimizer configuration recipes.
• Advanced Muon optimizer features - Implement Newton-Schulz orthogonalization coefficients, FP8 primary weights, and precision-aware behavior for DeepSeek-V4.
• MLA support for Muon - Extend the existing metadata-tagging scheme for splitting MLA up-projections (#5015).
• Model-Optimizer PTQ and QAD - Surface Model-Optimizer-based FP8/NVFP4 post-training quantization (PTQ) and quantization-aware distillation (QAD).
• PTQ and QAD documentation - Surface PTQ and QAD documentation using Nemotron 3 as examples.

Inference

• Sliding-window attention in dynamic batching - Bring dynamic inference to parity with static Transformer Engine attention for SWA models (#5138).
• ETP - Add support for expert tensor parallelism (#4743).
• Async scheduling — delivered - Introduce asynchronous dynamic-inference scheduling and overlap, then enable it by default (#5453, #5549, #5939, #6851).

Multimodal

• MIMO (Multimodal In, Multimodal Out) extensions - Expand the TP/DP heterogeneous primitive and enable autonomous nD parallelism for submodules.
• Pipeline parallel support for the language model - Allow colocated layouts where the destination language grid has PP > 1 while the source encoder grid remains PP = 1 (#4784).
• VLM CUDA Graph support - Capture the language model submodule (#4519).

* : Feature is being developed on the dev branch first and may not be part of the 26.08 main release.

---

How to Provide Feedback

We welcome community input on prioritization. Please:

1. React to items you would like prioritized.
2. Comment on this issue with use cases, constraints, and hardware / model configurations.
3. Open focused feature requests with the enhancement label.
4. Contribute pull requests for roadmap items where possible.

---

Credits

This roadmap reflects the collective efforts of NVIDIA, external contributors, and the Megatron Core community.

[satwiksps]

Hello @sbhavani
I would like to work on moving quantize.py to the root folder.

[sbhavani]

@satwiksps thanks for volunteering! that's not the best first issue. I'll take a look through our roadmap/issues and try to label some good first issues this week for you to take on.

[wujingyue]

@sbhavani FYI, I made a project tracker for MFSDP: https://github.com/orgs/NVIDIA/projects/276. Feel free to add a link into the roadmap.

[z1ying]

thanks for volunteering! that's not the best first issue. I'll take a look through our roadmap/issues and try to label some good first issues this week for you to take on.

Hi @sbhavani, I‘d also love to contribute. Looking forward to the good first issues!

[lk137095576]

@sbhavani Are we still doing Echo? #2368 .
I see it's gone from the Q3 2026 roadmap. Moonep might be a better option?

[sbhavani]

@lk137095576 please see #6134 for initial MoonEP support

[adityaghai07]

@sbhavani — I've opened #7274 (draft) for the "PTQ and QAD documentation" item. Happy to close it if this is already owned internally.

Two gaps it addresses:

• PTQ material exists under examples/post_training/modelopt/ but isn't surfaced in the docs site. ModelOpt is only referenced in get-started/overview.md and get-started/install.md, with no page in the Advanced Features toctree.
• QAD has no user-facing documentation. --qad-train-target {base,mtp,both} and _freeze_for_qad are implemented, but QAD only appears in source comments. This includes the detail that freezing one side of an MTP model also freezes that side's router bias terms, which are updated from token counts rather than gradients.

The new page covers quantize → evaluate → distil → export, plus multi-node Slurm, using nvidia/NVIDIA-Nemotron-3-Nano-30B-A3B-BF16 as the worked example. Flags and defaults are taken from the source rather than the existing prose.

One question I've flagged on the PR: the _freeze_for_qad docstring calls the MTP target the "production two-phase recipe" but doesn't specify the ordering. Confirmation that distil base → train heads is correct would be helpful.