Ad Incrementality and Targeting Policy

Question

When advertising capacity is limited, should impressions go to people with the highest predicted conversion probability or to people with the highest predicted incremental response?

Data

The analysis uses the corrected Criteo Uplift Prediction Dataset v2.1, a randomized advertising holdout with 13,979,592 rows. The reported results use a two-million-row working sample and evaluate two outcomes: conversion and site visit.

The features are anonymized. The analysis therefore stays at the level the data supports: treatment policy, incremental outcomes, and budget allocation. It does not invent customer personas or behavioral explanations.

Approach

• Estimate intention-to-treat lift from randomized assignment.
• Keep assignment effects separate from exposure-aware estimates, since actual exposure is selected rather than randomized.
• Compare conversion propensity, treated-response propensity, no-ad propensity, and T-, S-, X-, and transformed-outcome uplift learners.
• Evaluate each ranking at 5%, 10%, 20%, 30%, 50%, and 100% targeting shares.
• Measure cumulative incremental gain, AUUC, uplift calibration, bootstrap uncertainty, and cost-sensitive thresholds.

Findings

Outcome	Randomized ITT lift	Best 5% policy	Estimated gain at 5%	Random 5%	Best 20% policy
Conversion	+0.124 percentage points	X-learner uplift	1,575 conversions	124	No-ad propensity
Visit	+1.056 percentage points	S-learner uplift	9,162 visits	1,056	S-learner uplift

For visits, the S-learner leads the 5% and 20% budget tiers and overall AUUC. For conversions, the X-learner is strongest at the narrowest budget, while a no-ad propensity ranking leads at 20% and overall AUUC. Uplift modeling changes the decision, but it does not dominate every outcome or budget.

Decision

Model choice should be made jointly with the outcome and targeting capacity. Under a narrow conversion budget, the X-learner is the strongest evaluated policy. For visit growth, the S-learner is consistently stronger. A single propensity or uplift ranking should not be treated as universally optimal.

Limitations

• Conversion is rare in the working sample: 0.298%.
• The policy comparison is an offline evaluation on a two-million-row sample, not a live campaign result.
• Actual exposure is not randomized, so exposure-aware contrasts are not given the same causal interpretation as assignment effects.
• Anonymized features prevent substantive segment interpretation.
• Policy value depends on campaign cost, outcome value, and future population stability.

The analysis report contains the full conversion and visit results, uncertainty estimates, and diagnostic figures.

Reproduce the analysis

The synthetic-data path exercises the complete modeling workflow without requiring the Criteo download:

python -m venv .venv
source .venv/bin/activate
make install
make synthetic
make analyze-synthetic

The commands for the corrected Criteo dataset and the two-million-row analyses are documented in the Makefile. Raw and processed data remain excluded from version control.

Data use

Raw data is not included in this repository. The dataset source page provides the download and citation for the accompanying benchmark paper.