PREFINFER: Preference Inference from User-LLM Interaction History

We introduce PREFMINE, an automated data construction method for synthesizing high-quality preference inference data. Each example contains a synthetic interaction history, a current user request, and the user's preference relevant to the current request. Using this pipeline, we construct PREFMINE-15K, a supervised fine-tuning dataset for training models to infer request-relevant preferences from user interaction histories. We further use PREFMINE-15K to train PREFINFER, a preference inference model whose predictions can be used to guide downstream LLMs toward responses that better align with individual user preferences.

• 🤗 Dataset: PREFMINE-15K
• 🤗 Model: PREFINFER
• 🤗 Paper: paper

Data Construction

This repository contains the code for constructing preference inference data and ablation data.

Dependencies can be installed with:

pip install -r requirements.txt

The repository was tested with Python 3.10. The provided requirements list covers the main data construction scripts.

Directory Structure

.
├── assets/                 # Prompts used for data construction
├── data/                   # Source data directory
├── run_pipeline.sh         # Main data construction pipeline
├── run_ablation.sh         # Scripts for constructing ablation data
├── *.py                    # Top-level scripts for data construction and processing
└── README.md

Data Construction

The top-level Python files implement the main data construction pipeline.

The full source data under data/ is included in PREFMINE-15K/construct_source. The data format is described in data/README.md.

Before running the pipeline, we locally deployed gpt-oss-120b with 4 A800 80G GPUs, a maximum context length of 32,000 tokens, and a maximum concurrency of 64. The code assumes an OpenAI-compatible API endpoint at http://localhost:28001/v1 by default. To use a different port or model service, please update the endpoint configuration in call_api.py.

To run the main construction pipeline:

bash run_pipeline.sh

Please refer to the comments in this script for detailed input/output files and running steps.

Ablation Data Construction

Scripts for constructing ablation data are provided in this repository.

To construct data for ablation experiments:

bash run_ablation.sh

Please refer to the comments in this script for detailed input/output files and running steps.

License

The code in this repository is released under the MIT License. See the LICENSE file for details.