LLM Collaboration – Writing

This repo provides the extended environments for CoMLRL.

This repository contains the writing-task experiments in [AAAI26] LLM Collaboration with Multi‑Agent Reinforcement Learning.

Installation

Install CoMLRL:

pip install comlrl
# Install PyTorch compatible with your device

Or via conda-forge:

conda install -c conda-forge comlrl
# Install PyTorch compatible with your device

Benchmarks

• ArXiv Abstract Expansion: OpenMLRL/arXiv_abstract (train[:1000], val[:1000])
• TLDR Summarization: trl-lib/tldr (train[:1000], test[:1000])

Training Scripts

python LLM_Collab_Writing/train_grpo.py \
  --config LLM_Collab_Writing/configs/grpo_arxiv_config.yaml

python LLM_Collab_Writing/train_magrpo.py \
  --config LLM_Collab_Writing/configs/magrpo_tldr_config.yaml

python LLM_Collab_Writing/train_madpo.py \
  --config LLM_Collab_Writing/configs/madpo_tldr_config.yaml

python LLM_Collab_Writing/train_marlhf_iter.py \
  --config LLM_Collab_Writing/configs/marlhf_iter_tldr_config.yaml

Override any configuration value inline with --override:

python LLM_Collab_Writing/train_magrpo.py \
  --config LLM_Collab_Writing/configs/magrpo_arxiv_config.yaml \
  --override agent_model.name='Qwen/Qwen3-7B' magrpo.agent_learning_rate=3e-6

Settings

Single Turn

Writing runs are strictly single-turn. All training entrypoints enforce num_turns=1; configs that specify other values raise an error.

Preference Training

MADPO and MARLHF are available through train_madpo.py and train_marlhf.py. Their iterative variants use train_madpo_iter.py and train_marlhf_iter.py. The iterative configs support current, history-checkpoint, local-model, and API comparators independently from the current, nearest-k, all-history, and lambda-decay replay modes.

Formatters

• ArXiv: Agent 1 writes background/motivation; Agent 2 writes methodology/implications.
• TLDR: Agent 1 produces a concise summary; Agent 2 expands with additional details and vocabulary diversity.
• GRPO mode: A single agent emits both paragraphs separated by [PARAGRAPH_SPLIT], which the reward splits internally.

Reward Structure

Rewards reuse the level-based metrics from the paper:

1. Structural token limits.
2. Relative length coordination.
3. Vocabulary diversity (unique word ratios).
4. Style mix (transition-word coverage + Jaccard overlap).

The same functions back evaluation loggers for the baselines.

Logging

Evaluation wrappers adapt the original logging utilities to the unified MAGRPOTrainer API, yielding aggregated metrics such as token ratios, transition coverage, and gated vs. ungated rewards. Weights & Biases configs mirror the code-generation project; set wandb.project, wandb.entity, and wandb.name in YAML or via overrides.