Skip to content

Initialize multimodal models from language checkpoints - #7251

Draft
tylerpoon wants to merge 3 commits into
NVIDIA:pull-request/7250from
tylerpoon:tpoon/github-vlm-sft-04-llm-only-checkpoint-init
Draft

Initialize multimodal models from language checkpoints#7251
tylerpoon wants to merge 3 commits into
NVIDIA:pull-request/7250from
tylerpoon:tpoon/github-vlm-sft-04-llm-only-checkpoint-init

Conversation

@tylerpoon

@tylerpoon tylerpoon commented Sep 11, 2026

Copy link
Copy Markdown
Contributor
  • I, the PR author, have personally reviewed every line of this PR.

What does this PR do?

Support initializing the language component of a multimodal model from a language-only checkpoint while leaving vision parameters locally initialized. Preserve legacy checkpoint enum deserialization compatibility and add regression tests.

This is part of a review-split multimodal SFT series. Production run scripts, data YAMLs, checkpoint conversion tooling, and separate CI compatibility fixes are outside this series.

Dependencies and review scope

Stacked on #7250 (which builds on #7249). Review only the checkpoint-initialization and legacy-deserialization commits above that parent. The review base uses its bot-created mirror; retarget to main before merging.

Validation

  • Replayed onto public main at 46fb90ca942a2950c58c5654b3e56877e88a3e35, preserving atomic signed and signed-off commits.
  • The original combined feature stack had multi-node Super SFT validation before this migration; that is not an isolated-PR or rebased-head test result.
  • GitHub CI is pending. No GPU tests have been rerun locally for these rebased heads.
  • Local Black, isort, pylint, and Ruff checks passed for paths covered by the repository autoformat script (megatron/core and tests). Mypy is unavailable locally. Changed Python files pass syntax parsing; no CI configuration changes are included.

Kept as a draft pending CI and dependency integration.

@copy-pr-bot

copy-pr-bot Bot commented Sep 11, 2026

Copy link
Copy Markdown

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

@tylerpoon

Copy link
Copy Markdown
Contributor Author

/ok to test 479d923

Signed-off-by: Tyler Poon <tpoon@nvidia.com>
Signed-off-by: Tyler Poon <tpoon@nvidia.com>
Signed-off-by: Tyler Poon <tpoon@nvidia.com>
@tylerpoon
tylerpoon force-pushed the tpoon/github-vlm-sft-04-llm-only-checkpoint-init branch from 479d923 to b487695 Compare September 11, 2026 19:21
@tylerpoon

Copy link
Copy Markdown
Contributor Author

/ok to test b487695

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant