DPSS-RL: Strategy-Level Reinforcement Learning for Portfolio Management

Figure 1: Strategy-level GRPO framework

Official PyTorch implementation of DPSS-RL, a strategy-level reinforcement learning framework for dynamic portfolio management.

Overview

DPSS-RL formulates portfolio management as a strategy-selection problem. Rather than directly predicting individual asset weights, the agent selects among four predefined portfolio allocation strategies at each rebalancing point.

The method adapts Group Relative Policy Optimization (GRPO) to financial decision-making. It compares relative trajectory performance within sampled groups and trains the policy with a modified Sharpe reward and clipped objective.

Key Features

• Strategy-level reinforcement learning over established allocation policies
• GRPO-based optimization using relative trajectory performance
• Modified Sharpe reward for sign-aware risk-adjusted evaluation
• Asset-exclusion evaluation without retraining
• Financial evaluation across market regimes, costs, and sampling settings

Framework

Strategy-Level State Representation

The state concatenates 13 portfolio-level features for each of four strategies: Risk Parity (RP), Maximum Sharpe (MS), Minimum Variance (MV), and Protective Asset Allocation (PAA). The resulting state dimension is 52.

Portfolio Strategy Action Space

The discrete action space contains four actions. At each 20-trading-day rebalancing point, one action selects the corresponding portfolio strategy.

Modified Sharpe Reward

The selected strategy is evaluated using the annualized Israelsen modified Sharpe ratio over the subsequent 20-trading-day horizon, with a zero risk-free rate.

GRPO-based Policy Optimization

Multiple strategy trajectories are sampled for each group. Trajectory-level returns are compared within the group, min-max normalized into relative advantages, and used to optimize the policy with a clipped GRPO objective.

Project Structure

DPSS-RL/
├── data_pipeline/           # Data preprocessing and RL state construction
├── modeling/
│   ├── grpo_sharpe/         # Proposed GRPO model
│   ├── ppo_benchmark/       # PPO baseline
│   └── sac_benchmark/       # SAC baseline
├── backtest/                # Portfolio construction and backtesting
├── evaluation/              # Evaluation and statistical tests
├── result/                  # Paper figures
├── scripts/                 # Training and evaluation entry points
├── CODE_MAP.md
├── DATA_CONTRACT.md
└── requirements.txt

See CODE_MAP.md for the detailed paper-to-code mapping.

Installation

Python 3.10 or later is recommended. A CUDA-compatible GPU is recommended for training.

git clone https://github.com/finxlab/DPSS-RL.git
cd DPSS-RL
pip install -r requirements.txt

Install a PyTorch build compatible with the target CPU or CUDA environment.

Data Setup

Large raw datasets, processed tensors, and trained checkpoints are not included. Place local inputs under data/ according to the paths in the model YAML files.

The study uses 30 assets - 18 global equity indices and 12 commodity futures - from February 2002 through December 2024. A 252-trading-day lookback produces the following effective chronological splits:

Split	Period	Role
Train	2003.02.19-2013.12.31	Policy training
Validation	2014.01.01-2018.12.31	Early stopping and model selection
Test	2019.01.01-2024.12.31	Final out-of-sample evaluation

See DATA_CONTRACT.md for feature and preprocessing details.

Usage

Run all commands from the repository root. Local data is required for training; evaluation additionally requires a checkpoint configured in modeling/grpo_sharpe/config/test_config.yaml.

Train GRPO

bash scripts/train_grpo.sh

Train PPO

bash scripts/train_ppo.sh

Train SAC

bash scripts/train_sac.sh

Evaluate GRPO

bash scripts/eval_grpo.sh

Configuration

Setting	Main implementation value
State dimension	52
Strategies / actions	4
Lookback	252 trading days
Rebalancing interval	20 trading days
Strategy samples per group, M	16
Sampled trading groups, N	2048
Policy updates per rollout, mu	50
Clipping epsilon	0.2
Actor learning rate	1e-6
Risk-free rate	0
Default transaction cost	30 bps

Results

Reported results for the complete out-of-sample period, January 2019-December 2024, are taken from Table 4.1 of the paper. RL values are averages across five random seeds.

Model	Annualized Return	Sharpe	Sortino	Calmar	MDD
GRPO	18.7%	1.154	2.363	1.037	0.180
PPO	11.5%	0.769	1.174	0.531	0.216
SAC	7.9%	0.523	0.797	0.266	0.298
MV	9.6%	0.912	1.501	0.561	0.172

The paper additionally reports market-regime performance, asset-exclusion robustness without retraining, transaction-cost sensitivity, action-sampling sensitivity, two-sided Newey-West HAC mean-return tests, and Ledoit-Wolf-style Sharpe-ratio comparisons.

Citation

Citation information will be updated upon publication.