How to use HGNN+ for RL

[FernandezR]

Hello,

Is there a way to use HGNN+ for Reinforcement Learning using batch training? The only examples I see are doing classification tasks with labels and do not show how to use your library for RL.

Thanks

[yifanfeng97]

HGNN+ can be used as a feature extractor or policy/value network component in RL. The key insight is that DHG models are standard PyTorch nn.Modules — you can integrate them into any RL framework.

Example: HGNN+ as a feature extractor in a policy network

import torch
import torch.nn as nn
import dhg

class HGNNPolicy(nn.Module):
    def __init__(self, state_dim, hid_dim, action_dim):
        super().__init__()
        # HGNN+ layer as feature extractor
        self.hgnn = dhg.nn.HGNNPConv(state_dim, hid_dim, use_bn=False)
        # MLP head for action logits
        self.actor = nn.Linear(hid_dim, action_dim)
        self.critic = nn.Linear(hid_dim, 1)

    def forward(self, X, hg):
        # X: vertex features (state representation)
        # hg: hypergraph structure (can be dynamic per step)
        features = self.hgnn(X, hg)
        action_logits = self.actor(features)
        value = self.critic(features)
        return action_logits, value

Key considerations for RL:

1. Dynamic hypergraph: If the hypergraph changes at each step, rebuild it with dhg.Hypergraph(num_v, new_e_list) and call hg.to(device). This has some overhead — see 关于稠密图数据用于训练 #12 for performance discussion.

2. Static hypergraph: If the structure is fixed (e.g., multi-agent coordination on a fixed topology), build it once and reuse — much faster.

3. On-policy training (PPO, A2C): Standard usage — collect rollouts with the current policy, then update. The HGNN forward pass is differentiable as normal.

4. Off-policy training (DQN, SAC): Store (X, hg, action, reward, next_X, next_hg) in replay buffer. The hypergraph structure can be stored as edge lists and rebuilt on sampling.

The existing classification examples demonstrate the forward/backward API — the same model(X, hg) call works in an RL loop.

[yifanfeng97]

Answered above with code examples for integrating HGNN+ into RL policy networks. Feel free to reopen if you need more details.