Parent initiative: #7267
Work
Add a pre-wrap PEFT hook to the Megatron-LM training path. The hook must apply configured adapters and freeze the base model before distributed model wrapping.
Done when
megatron.training can apply the migrated PEFT adapters during model setup.
- Adapter application happens before DDP or FSDP wraps the model.
- Base-model parameters are frozen while adapter parameters remain trainable.
- The path supports performant LoRA, canonical LoRA, and DoRA.
- Tests verify adapter placement, parameter freezing, and ordering relative to model wrapping.
Parent initiative: #7267
Work
Add a pre-wrap PEFT hook to the Megatron-LM training path. The hook must apply configured adapters and freeze the base model before distributed model wrapping.
Done when
megatron.trainingcan apply the migrated PEFT adapters during model setup.