Adversarial Frontiers: Minimum-Norm Attack Ensembles for Robustness Evaluation

Robust accuracy reported at a single perturbation budget hides most of what describes a model, and rankings read off one budget do not survive a change of budget. We evaluate robustness over the whole perturbation range instead, with ensembles of minimum-norm attacks that run under a query budget you fix in advance.

Method Overview

A minimum-norm attack returns the smallest perturbation that breaks each sample, so one run gives robust accuracy at every budget. An ensemble is a set of such attacks over ℓ0, ℓ1, ℓ2 and ℓ∞, sharing a query budget you fix in advance.

Keeping the smallest perturbation per sample gives the attack frontier, the tightest curve the ensemble produces for a model. The hardest perturbation per sample across the 20 CIFAR-10 models we tested gives the defense frontier. DOI is the area under a model's robustness curve divided by the area under the frontier's, so it ranks models with no choice of ε.

Repository Layout

• ensemble_tutorial.ipynb — the tutorial, opens in Colab
• src/ens_runner.py — ensemble execution, robustness curve, AUREC and DOI
• src/attacks/ — the attack implementations and their registry
• src/models/ — BenchModel, the wrapper that enforces the per-sample query budget
• src/utils/ — perturbation initialization, norms, memory tracking
• ensembles/ — the ensembles we report, 4 norms by 3 query budgets
• data/cifar10_eval_1000.npz — the 1000 CIFAR-10 test images every campaign evaluated
• data/reference_cifar10.npz — precomputed distances, defense frontier and DOI for the 20 CIFAR-10 models we tested
• data/model_id_mapping.json — campaign directory to paper ID (C1..C20)
• assets/make_figures.py — regenerates the static figure used in the tutorial

Model checkpoints are not redistributed. RobustBench downloads them on first use into models/.

Using Our Code

Setup

python -m venv .venv && source .venv/bin/activate
pip install -r requirements.txt

Torch with CUDA is recommended. Apple MPS and CPU also work, and the device is chosen automatically.

Run an Ensemble

import sys; sys.path.insert(0, "src")
import ens_runner as er

x, y = er.load_samples(1000)          # the 1000 CIFAR-10 test images we evaluated
spec = er.load_ensemble("L2", 4000)   # an ensemble with a 4000-query budget

dist, per_attack = er.run_ensemble(
    net, x, y, "L2", spec["attacks"], er.pick_device()
)

net is any torch.nn.Module. dist[i] is the attack frontier: the smallest perturbation any attack in the ensemble found for sample i. per_attack keeps each attack's own result.

Score the Result

import numpy as np
ref = np.load("data/reference_cifar10.npz", allow_pickle=True)
eps_max, defense = float(ref["L2/eps_max"]), ref["L2/defense"]

xs, rho = er.robustness_curve(dist, eps_max)   # the curve
area    = er.aurec(dist, eps_max)              # AUC under it
score   = er.doi(dist, defense, eps_max)       # DOI against the defense frontier

Evaluate Your Own Model

run_ensemble takes any torch.nn.Module, so a model of your own needs no registration anywhere:

net = MyDefense().eval().to(device)      # logits from (N, 3, 32, 32) inputs in [0, 1]
dist, _ = er.run_ensemble(net, x, y, "L2", spec["attacks"], device)

Two requirements. The model takes inputs in [0, 1] and returns logits, so any input normalization belongs inside the module. And to compare its DOI against the numbers we report, evaluate it on er.load_samples(n), the same CIFAR-10 images every run in the paper used.

Evaluate a RobustBench Model

from robustbench.utils import load_model
net = load_model(model_name="Addepalli2021Towards_RN18", dataset="cifar10",
                 threat_model="Linf", model_dir=er.ensure_model_dir())

er.model_choices("L2") lists the models we tested in the paper, each with the DOI we report, so a run can be checked against a known result.

Additional Options

Norms. L0, L1, L2, Linf. The threat model of a checkpoint is set by how it was trained, independently of the norm you attack it in.

Query budgets. 4000, 8000 and 12000 queries per sample, the three tiers we report in the paper, giving the 12 ensembles in ensembles/. Higher budgets are supersets of lower ones.

Samples. er.load_samples(n) returns the first n of the 1000 images used in every campaign, and is bit-identical to robustbench.data.load_cifar10. Pass your own tensors to run_ensemble to evaluate on anything else.

Paths. Everything resolves relative to src/ens_runner.py. Override with AF_ENSEMBLES, AF_SAMPLES, AF_DATA, AF_MODELS, AF_MAPPING, AF_REFERENCE.

Dig Into the Tutorial

The tutorial walks through the whole method interactively, from what a robustness curve is, through picking an ensemble and running it, to reading DOI against the models we tested in the paper.

Or run it locally with Jupyter:

jupyter notebook ensemble_tutorial.ipynb

Citation

@article{scionis2026adversarial,
  title={Adversarial Frontiers: Minimum-Norm Attack Ensembles for Robustness Evaluation},
  author={Scionis, Luca and Melis, Luca and Pintor, Maura and Brau, Fabio and Demontis, Ambra and Fumera, Giorgio and Roli, Fabio and Biggio, Battista},
  journal={arXiv preprint arXiv:2607.19855},
  year={2026}
}

CIFAR-10 is redistributed here as the 1000-image evaluation subset, for reproducibility.

Acknowledgments

This work has been carried out while L. Scionis and L. Melis were enrolled in the Italian National Doctorate on AI run by the Sapienza University of Rome in collaboration with the University of Cagliari. This research has been partially supported by the Horizon Europe projects Sec4AI4Sec (GA no. 101120393), and CoEvolution (GA no. 101168560), and by project FISA-2023-00128 funded by the MUR program Fondo italiano per le scienze applicate.