Decision curve analysis when the outcome is recorded with error.
Decision curves show whether a prediction model does more good than harm at a
given risk threshold. They are usually drawn with outcomes taken from diagnosis
codes, self-report or text mining, and these outcomes are not perfect. A model
can look useful on recorded outcomes and be harmful on true ones, or the other
way round. truedca shows how the curve is distorted, how robust a conclusion
is, and how to correct the curve with a small chart review.
This repository holds both packages and everything needed to reproduce the paper.
| Folder | Contents |
|---|---|
r/ |
R package truedca |
python/ |
Python package truedca (same functions, tested against the R results) |
paper/ |
Simulation study, worked examples, tables, figures and numerical checks of the theorems |
data/ |
Open data used in the paper (eICU demo tables, NHANES download script, derived analysis files) |
R (4.1 or later):
# install.packages("remotes")
remotes::install_github("Marwan-Alqumbaey/truedca", subdir = "r")Python (3.9 or later):
pip install "git+https://github.com/Marwan-Alqumbaey/truedca#subdirectory=python"| R / Python | Purpose |
|---|---|
nb_curve() |
Decision curve from the outcomes as recorded |
dca_window() |
Thresholds where a recorded-outcome curve cannot show benefit |
warp_threshold() |
Threshold at which recorded outcomes actually judge a decision |
nb_adjust() |
Corrected curve when error rates are known (they may differ between flagged and unflagged patients) |
nb_bounds() |
Sharp bounds, tipping points and their confidence limits when error rates are only partly known |
nb_bias_analysis() |
Probabilistic bias analysis when no chart review is possible |
pilot_v() |
Guess outcome uncertainty from a small pilot review |
twophase_design() |
Chart review probabilities that make the corrected curve most precise |
design_gain() |
How much that design can gain over simple random sampling, before any chart is reviewed |
twophase_dca() |
Corrected curve from a chart review subsample, with split score intervals and simultaneous bands |
make_strata() |
Strata that cross risk groups with the recorded outcome |
plot_dca() |
Simple plot |
library(truedca)
set.seed(1)
n <- 20000
risk <- plogis(rnorm(n, -2.5, 1.2))
truth <- rbinom(n, 1, risk)
y <- ifelse(truth == 1, rbinom(n, 1, 0.7), rbinom(n, 1, 0.01)) # coded outcome
dca_window(se = 0.7, sp = 0.99) # floor 0.01, ceiling 0.70
# How robust is the gain over treat all if sensitivity is only known to be >= 0.6?
nb_bounds(risk, y, c(0.05, 0.1, 0.2), se0 = c(0.6, 1), sp0 = c(0.99, 1))
# Review 400 charts: a pilot of 100, then 300 placed where they matter most
pilot <- ifelse(seq_len(n) %in% sample.int(n, 100), truth, NA)
v <- pilot_v(risk, y, pilot)
design_gain(risk, y, 300, range = c(0.02, 0.3), comparator = "both", v = v)
prob <- twophase_design(risk, y, 300, range = c(0.02, 0.3), comparator = "both", v = v)
prob[!is.na(pilot)] <- 1 # pilot charts are already reviewed
seen <- ifelse(!is.na(pilot) | rbinom(n, 1, prob) == 1, truth, NA)
fit <- twophase_dca(risk, y, seen, prob, thresholds = seq(0.02, 0.3, 0.02))
plot_dca(fit)import numpy as np
import truedca as td
rng = np.random.default_rng(1)
risk = 1 / (1 + np.exp(-rng.normal(-2.5, 1.2, 20000)))
truth = rng.binomial(1, risk)
y = np.where(truth == 1, rng.binomial(1, 0.7, 20000), rng.binomial(1, 0.01, 20000))
print(td.nb_bounds(risk, y, [0.05, 0.1, 0.2], se0=(0.6, 1), sp0=(0.99, 1)))See paper/README.md. The Reproduce results workflow re-runs the
numerical checks of the theorems, the first part of the simulation study, both examples
from the raw open data, and all tables and figures, and compares every number with the
stored results.
Alqumbaey M, Nasher YM. Decision curve analysis with misclassified outcomes: bias,
sensitivity analysis and optimal two-phase validation. Software: Alqumbaey M, Nasher YM.
truedca 0.1.0. Zenodo. https://doi.org/10.5281/zenodo.23194144. See CITATION.cff.
- Marwan Alqumbaey (ORCID 0009-0004-8688-9556), Faculty of Medicine and Health Sciences, Sana'a University, Sana'a, Yemen
- Yousef M. Nasher (ORCID 0009-0001-3704-3149), Faculty of Medicine and Health Sciences, Sana'a University, Sana'a, Yemen
Code: MIT. Data: see data/README.md.