HDP: Implement official RL post-training objective

[Max-Bin]

Background

The official Hyper Diffusion Planner (HDP) paper proposes a reward-weighted diffusion post-training method for improving planning safety and robustness after supervised pretraining.

This issue tracks the implementation of the official HDP-RL formulation in the HDP integration branch.

Scope

The goal is to implement the official HDP-RL method as faithfully as possible while keeping the existing Tier IV Diffusion Planner inputs and evaluation infrastructure.

The implementation should cover:

• Reward-weighted diffusion imitation objective described in the HDP paper.
• Velocity-trajectory action representation used by HDP-RL.
• Hybrid waypoint / velocity loss compatibility.
• Group-normalized rewards when sampling multiple candidate trajectories per scene.
• Filtering of samples where all candidate actions receive identical rewards, if applicable.
• EMA / old-policy handling consistent with the official method.
• A clean training entry point for RL post-training after supervised HDP pretraining and SFT.

Questions to resolve

• How closely should the Tier IV implementation follow the official HDP-RL objective versus adapting it to the existing DP training stack?
• Which reward terms should be enabled by default?
• How should pseudo-closed-loop rewards be computed with the available offline data and evaluator?
• How should reward normalization and invalid-sample filtering be handled?
• What metrics should be used to decide whether HDP-RL improves over supervised HDP?

Development workflow

This is HDP-specific work. Future code changes should be reviewed through PRs targeting the HDP integration branch rather than the main branch.

Proposed target branch:

feature/hyper-diffusion-planner

Expected outcome

• A faithful official HDP-RL training path.
• Clear documentation of the reward objective and training procedure.
• Validation showing whether RL post-training improves planning quality over the supervised HDP checkpoint.