diff --git a/research/README.md b/research/README.md index b7c475d..ff9f9fd 100644 --- a/research/README.md +++ b/research/README.md @@ -22,5 +22,7 @@ shipped or stale analysis stops being useful and starts being noise. | File | Status | |---|---| | `log_retransformation_bias.{md,py,json}` | **Open** — log-target models are flagged "unreliable" for a mean-vs-median gap; proposes fixing the verdict logic + an opt-in mean correction. | +| `bias_gate_materiality.{md,py,json}` | **Open** — the `unreliable` verdict tracks holdout size, not bias magnitude: same model and same bias, flagged 10% of the time at n=50 and 100% at n=16,000. Proposes an effect-size floor, a log-space test, and `unknown` instead of a small-n free pass. | +| `methodology_review.md` + `benchmark_methodology_gaps.py` + `methodology_gaps_benchmark_results.txt` | **Open (partly superseded)** — July methodology audit. Findings 3 & 4 still open; 1 superseded by `log_retransformation_bias`, 2 sharpened by `bias_gate_materiality`, the OOD bullet shipped as #36. See its Status header. | | `out_of_domain_flag.{md,py,json}` | **Open** — nothing in the predict path notices a part outside the data the model was built on; on a well-fit model an impossible part earns a `trust` verdict. Proposes an `in_domain` block that caps the verdict. | | `whatif_reliability_flag.md` | **Shipped (#35).** Retained only because the CHANGELOG cites it as rationale. | diff --git a/research/benchmark_methodology_gaps.py b/research/benchmark_methodology_gaps.py new file mode 100644 index 0000000..d149e79 --- /dev/null +++ b/research/benchmark_methodology_gaps.py @@ -0,0 +1,302 @@ +"""Quantify three methodology gaps on the case-study datasets (findings PR). + +This script produces the numbers behind ``research/methodology_review.md``. +It changes nothing in core — it measures how much each gap actually costs on +the aerospace-fasteners and used-cars case studies, so the follow-up core +fixes can be sized by evidence rather than by principle. + +Experiment 1 — retransformation (smearing) bias + Log-target models back-transform with plain ``exp()``. Because + ``E[exp(yhat + eps)] != exp(yhat)``, the price-space point prediction is + systematically low by roughly ``E[exp(eps)]``. Duan's smearing estimator + corrects it: multiply the naive prediction by ``S = mean(exp(eps_i))`` + over *out-of-sample* residuals (a boosted tree's training residuals are + near zero, so S must come from data the model never fit — here a + calibration split; in the product it could come from the holdout). + Protocol: 80/20 pool/eval, then 75/25 train/cal inside the pool. Train + exactly as the case study does (XGBoost, tuned). Report eval-set mean + residual, MAE, R² and the bias t-test before and after smearing. + +Experiment 2 — pre-split leakage + The product computes outlier bounds and RF-importance feature selection + on the FULL dataset before the train/test split, so the holdout has + leaked into those decisions — and, for drop policies, the holdout itself + is cleaned of the extremes production will see. Two mechanisms, measured + separately with fixed (untuned) XGBoost so the only difference is where + the decision was fitted: + * selection — auto-mode top-6 features chosen on pool+eval (leaky, + mirrors the product) vs pool only (clean); same eval set for both. + * outliers — drop-policy IQR bounds fitted on pool+eval (leaky) vs + pool only (clean). Reported three ways: clean fit on raw eval + (honest), leaky fit on raw eval (bounds leakage alone), leaky fit on + bounds-cleaned eval (what the product reports today). + Run at three catalog sizes (400 / 2,000 / full rows) because leakage is + a small-data problem — the case studies are large, a typical procurement + CSV is not. + +Experiment 3 — residual-bias test space + The trustworthy/unreliable verdict hangs on a one-sample t-test of RAW + PRICE residuals (quality.residual_bias_p) even for log-target models, + where residuals are heavily skewed. Reuse Experiment 1's model: report + the p-value in price space (product today), in log space, and a Wilcoxon + signed-rank test, plus the verdict each one implies. + +Run: python research/benchmark_methodology_gaps.py [fasteners|cars] + (no argument = both sequentially) +""" +from __future__ import annotations + +import sys +import time +import warnings +from pathlib import Path + +import numpy as np +import pandas as pd +from scipy.stats import ttest_1samp, wilcoxon +from sklearn.metrics import mean_absolute_error, r2_score +from sklearn.model_selection import train_test_split + +# sklearn 1.9 deprecation chatter from TargetEncoder inside the product +# pipeline; irrelevant to the measurements and it drowns the results. +warnings.filterwarnings("ignore", category=FutureWarning) + +HERE = Path(__file__).resolve().parent +REPO = HERE.parent +sys.path.insert(0, str(REPO / "src")) + +from p2predict.feature_selection import get_most_predictable_features +from p2predict.outliers import detect_outliers +from p2predict.prepare_data import Get_Column_Types +from p2predict.quality import assess_model, residual_bias_p +from p2predict.training import resolve_log_target, start_training + +AUTO_MODE_MAX_FEATURES = 6 # cli/train.py auto-mode default + +DATASETS = { + "fasteners": dict( + csv=HERE / "aerospace-fasteners/data/bolts_clean.csv", + target="unit_price_each_usd", + features=["material", "head_style", "thread_diameter_in", "length_in", + "thread_class", "thread_series", "finish", + "tensile_strength_psi", "threads_per_inch", + "width_across_flats_in"], + log_target="on", budget="fast", + ), + "cars": dict( + csv=HERE / "used-cars/data/vehicles_training.csv", + target="price", + features=["year", "odometer", "manufacturer", "condition", "fuel", + "transmission", "drive", "type", "state", "paint_color"], + log_target="auto", budget="fast", + ), +} + +# (rows, seeds) — leakage shrinks with n, so measure at catalog sizes too. +LEAKAGE_SCALES = ((400, 10), (2000, 5), (None, 2)) + + +def load(cfg) -> pd.DataFrame: + df = pd.read_csv(cfg["csv"]) + return df[df[cfg["target"]].notna()].reset_index(drop=True) + + +def fit_xgb(X_train, y_train, cfg, tune): + num, cat = Get_Column_Types(X_train) + log_target, _ = resolve_log_target(y_train, mode=cfg["log_target"]) + model, _, log_target = start_training( + X_train, y_train, num, cat, algorithm="xgboost", + budget=cfg["budget"], tune=tune, log_target=log_target, + ) + return model, log_target + + +# --------------------------------------------------------------------------- +# Experiments 1 & 3 — smearing bias and the bias-test space +# --------------------------------------------------------------------------- + +def run_smearing_and_bias_test(name: str, cfg) -> None: + print("-" * 76) + print(f"[{name}] EXPERIMENT 1 — retransformation (smearing) bias") + print("-" * 76) + t0 = time.time() + df = load(cfg) + X, y = df[cfg["features"]], df[cfg["target"]] + X_pool, X_eval, y_pool, y_eval = train_test_split( + X, y, test_size=0.2, random_state=0) + X_train, X_cal, y_train, y_cal = train_test_split( + X_pool, y_pool, test_size=0.25, random_state=0) + + model, log_target = fit_xgb(X_train, y_train, cfg, tune=True) + if not log_target: + print(f"log-target resolved OFF for {name}; smearing does not apply.") + return + print(f"rows: train {len(X_train):,} / cal {len(X_cal):,} / " + f"eval {len(X_eval):,} trained in {time.time() - t0:.0f}s") + + # Duan smearing factor from out-of-sample (calibration) log residuals. + pred_cal = np.asarray(model.predict(X_cal), dtype=float) + resid_cal = np.log(y_cal.to_numpy()) - np.log(pred_cal) + smear = float(np.mean(np.exp(resid_cal))) + lognormal = float(np.exp(np.var(resid_cal) / 2.0)) + + y_true = y_eval.to_numpy(dtype=float) + naive = np.asarray(model.predict(X_eval), dtype=float) + smeared = naive * smear + eval_factor = float(np.mean(np.exp(np.log(y_true) - np.log(naive)))) + + print(f"smearing factor S — Duan (calibration): {smear:.4f} " + f"lognormal exp(sigma^2/2): {lognormal:.4f} " + f"same factor measured on eval: {eval_factor:.4f} " + f"(1.0 = no retransformation bias)") + print(f"{'':<28} {'naive exp()':>14} {'x S (smeared)':>14}") + rows = [ + ("mean residual $ (pred-act)", np.mean(naive - y_true), np.mean(smeared - y_true)), + ("median residual $", np.median(naive - y_true), np.median(smeared - y_true)), + ("MAE $", mean_absolute_error(y_true, naive), mean_absolute_error(y_true, smeared)), + ("median APE %", + np.median(np.abs(naive - y_true) / y_true) * 100, + np.median(np.abs(smeared - y_true) / y_true) * 100), + ("R^2 (price space)", r2_score(y_true, naive), r2_score(y_true, smeared)), + ("bias t-test p (price)", residual_bias_p(y_true, naive), residual_bias_p(y_true, smeared)), + ] + for label, a, b in rows: + print(f"{label:<28} {a:>14,.4f} {b:>14,.4f}") + + print() + print("-" * 76) + print(f"[{name}] EXPERIMENT 3 — residual-bias test space (naive predictions)") + print("-" * 76) + resid_price = y_true - naive + resid_log = np.log(y_true) - np.log(naive) + p_price_t = float(ttest_1samp(resid_price, 0.0).pvalue) + p_log_t = float(ttest_1samp(resid_log, 0.0).pvalue) + p_wilcoxon = float(wilcoxon(resid_price).pvalue) + r2 = r2_score(y_true, naive) + n = len(y_true) + v_price = assess_model(r2, p_price_t, n)["verdict"] + v_log = assess_model(r2, p_log_t, n)["verdict"] + v_wilcoxon = assess_model(r2, p_wilcoxon, n)["verdict"] + print(f"{'test':<38} {'p-value':>10} verdict (gate at p<=0.05)") + print(f"{'t-test, price space (product today)':<38} {p_price_t:>10.4f} {v_price}") + print(f"{'t-test, log space':<38} {p_log_t:>10.4f} {v_log}") + print(f"{'Wilcoxon signed-rank, price space':<38} {p_wilcoxon:>10.4f} {v_wilcoxon}") + print() + + +# --------------------------------------------------------------------------- +# Experiment 2 — pre-split leakage (selection + outlier bounds) +# --------------------------------------------------------------------------- + +def iqr_bounds(frame: pd.DataFrame, columns) -> dict: + """Tukey bounds per column, fitted on ``frame`` (mirrors detect_outliers).""" + bounds = {} + for col in columns: + if not pd.api.types.is_numeric_dtype(frame[col]): + continue + _, lower, upper = detect_outliers(frame[col]) + bounds[col] = (lower, upper) + return bounds + + +def within_bounds(frame: pd.DataFrame, bounds: dict) -> pd.Series: + mask = pd.Series(True, index=frame.index) + for col, (lower, upper) in bounds.items(): + if np.isnan(lower): + continue + mask &= frame[col].between(lower, upper) | frame[col].isna() + return mask + + +def select_top_features(frame: pd.DataFrame, target: str) -> list[str]: + ranked = get_most_predictable_features(frame, target, output_only_headers=True) + return ranked.head(AUTO_MODE_MAX_FEATURES).tolist() + + +def eval_r2(model, X_eval, y_eval) -> float: + return float(r2_score(y_eval.to_numpy(dtype=float), + np.asarray(model.predict(X_eval), dtype=float))) + + +def run_leakage(name: str, cfg) -> None: + df_all = load(cfg) + target = cfg["target"] + features = cfg["features"] + + print("-" * 76) + print(f"[{name}] EXPERIMENT 2 — pre-split leakage (fixed untuned XGBoost)") + print("-" * 76) + + for n_rows, n_seeds in LEAKAGE_SCALES: + n = len(df_all) if n_rows is None else min(n_rows, len(df_all)) + sel_deltas, out_honest, out_leaky_raw, out_leaky_clean = [], [], [], [] + sel_changed = 0 + t0 = time.time() + for seed in range(n_seeds): + df = df_all.sample(n=n, random_state=seed).reset_index(drop=True) + pool_idx, eval_idx = train_test_split( + df.index, test_size=0.2, random_state=seed) + pool, eval_ = df.loc[pool_idx], df.loc[eval_idx] + + # -- mechanism A: feature selection fitted pre- vs post-split + frame_cols = features + [target] + top_leaky = select_top_features(df[frame_cols], target) + top_clean = select_top_features(pool[frame_cols], target) + sel_changed += int(set(top_leaky) != set(top_clean)) + r2_leaky = eval_r2( + fit_xgb(pool[top_leaky], pool[target], cfg, tune=False)[0], + eval_[top_leaky], eval_[target]) + r2_clean = eval_r2( + fit_xgb(pool[top_clean], pool[target], cfg, tune=False)[0], + eval_[top_clean], eval_[target]) + sel_deltas.append(r2_leaky - r2_clean) + + # -- mechanism B: drop-policy outlier bounds fitted pre- vs post-split + numeric_feats = [c for c in features + if pd.api.types.is_numeric_dtype(df[c])] + cols = numeric_feats + [target] + b_leaky = iqr_bounds(df, cols) # product: bounds see eval rows + b_clean = iqr_bounds(pool, cols) + pool_leaky = pool[within_bounds(pool, b_leaky)] + pool_clean = pool[within_bounds(pool, b_clean)] + eval_cleaned = eval_[within_bounds(eval_, b_leaky)] + + m_leaky = fit_xgb(pool_leaky[features], pool_leaky[target], cfg, tune=False)[0] + m_clean = fit_xgb(pool_clean[features], pool_clean[target], cfg, tune=False)[0] + out_honest.append(eval_r2(m_clean, eval_[features], eval_[target])) + out_leaky_raw.append(eval_r2(m_leaky, eval_[features], eval_[target])) + if len(eval_cleaned) >= 2: + out_leaky_clean.append( + eval_r2(m_leaky, eval_cleaned[features], eval_cleaned[target])) + + def ms(vals): + return f"{np.mean(vals):+.4f} ± {np.std(vals):.4f}" + + print(f"\nn={n:,} rows, {n_seeds} seed(s) ({time.time() - t0:.0f}s)") + print(f" selection: eval R^2 (leaky top-6) − (clean top-6): {ms(sel_deltas)}" + f" [top-6 set differed in {sel_changed}/{n_seeds} seeds]") + print(f" outliers (drop): eval R^2 — honest (clean bounds, raw eval): " + f"{np.mean(out_honest):.4f}") + print(f" leaky bounds, raw eval: " + f"{np.mean(out_leaky_raw):.4f} (bounds leakage alone: " + f"{np.mean(out_leaky_raw) - np.mean(out_honest):+.4f})") + if out_leaky_clean: + print(f" leaky bounds, CLEANED eval (product today): " + f"{np.mean(out_leaky_clean):.4f} (reported-vs-honest gap: " + f"{np.mean(out_leaky_clean) - np.mean(out_honest):+.4f})") + print() + + +def main() -> None: + names = sys.argv[1:] or list(DATASETS) + for name in names: + cfg = DATASETS[name] + print("=" * 76) + print(f"{name.upper()} — {cfg['csv'].name}") + print("=" * 76) + run_smearing_and_bias_test(name, cfg) + run_leakage(name, cfg) + + +if __name__ == "__main__": + main() diff --git a/research/bias_gate_materiality.md b/research/bias_gate_materiality.md new file mode 100644 index 0000000..56e08dd --- /dev/null +++ b/research/bias_gate_materiality.md @@ -0,0 +1,112 @@ +# Finding: the "unreliable" gate measures holdout size, not bias + +**Status: open.** Sharpens finding #2 of [`methodology_review.md`](methodology_review.md) +and shares a root cause with [`log_retransformation_bias.md`](log_retransformation_bias.md). +All numbers from `bias_gate_materiality.py` (results in +`bias_gate_materiality_results.json`). **No core code changed.** + +## TL;DR + +`assess_model()` calls a model biased when `residual_bias_p <= 0.05`, where +`residual_bias_p` (`quality.py:333`) is a one-sample t-test of price-space +residuals against zero. A p-value answers *"could this bias be noise?"* — a +question whose answer depends on sample size. It does not answer *"is this +bias big enough to matter?"*, which is what the verdict claims to report. + +Consequence, measured: **hold the model and its bias exactly fixed, vary only +the holdout size, and the verdict flips.** Same predictions, same error, from +`trustworthy` to `unreliable` — because n grew. + +| n holdout | median mean-residual | median bias p | % of draws flagged `unreliable` | +|---|---|---|---| +| 50 | −5.4% | 3.2e-01 | **10%** | +| 100 | −5.4% | 1.9e-01 | 23% | +| 250 | −6.1% | 6.4e-02 | 47% | +| 500 | −5.7% | 6.6e-03 | 80% | +| 1,000 | −5.7% | 9.2e-05 | 96% | +| 2,500 | −5.8% | 1.1e-09 | 100% | +| 16,000 | −5.8% | 9.8e-55 | **100%** | + +Heavy-equipment model, holdout R² 0.737; 200 random draws per row. The bias +column is flat. The verdict column goes from 10% to 100%. + +## The flaw in one line + +`unbiased = residual_bias_p > UNBIASED_P` (`quality.py:200`) treats **failure +to reject** as **proof of no bias**. Those are not the same claim, and the gap +between them is exactly sample size. So the gate fails in *both* directions: + +- **Large holdout → guaranteed `unreliable`.** Any systematic offset, however + commercially irrelevant, reaches p < 0.05 once n is big enough. The user with + the best price history is the most likely to be told their model can't be + trusted. +- **Small holdout → free pass.** The battery-management-IC model carries a + **−7.5% mean residual** on 30 holdout parts and is stamped `trustworthy` + (p = 0.59), because 30 points can't reject anything. This is the dangerous + direction, and it is the one the July review explicitly ruled out when it + said "failure direction is under-trust, so live users see over-caution, not + wrong prices." Over-caution is the large-n direction. Small-n is over-trust. + +## What the verdict costs + +`unreliable` is not a footnote. The MCP server instructions tell the agent to +lead with the verdict, and its headline says the model's "single-number +estimates aren't trustworthy. Use it only to compare options, not to set an +absolute target" (`quality.py:218`) — it withdraws the single-part appraisal, +which is the product's main use. Both flagship case studies currently earn it: + +| Dataset | R² | mean resid | median resid | p (price space) | verdict today | +|---|---|---|---|---|---| +| used cars (16,000 holdout) | 0.735 | −27.2% | **+1.7%** | 4.4e-71 | `unreliable` | +| heavy equipment (16,000 holdout) | 0.737 | −5.8% | **+1.7%** | 9.8e-55 | `unreliable` | +| battery mgmt ICs (30 holdout) | 0.606 | −7.5% | −0.8% | 5.9e-01 | `trustworthy` | + +Note the median column: both large models are within ±2% on the *typical* +part. What fails them is the mean, and for a log-target model the mean is the +wrong statistic — `exp(E[log Y])` targets the median by construction +(mechanism in [`log_retransformation_bias.md`](log_retransformation_bias.md)). + +## Correction to the July review + +The review reported that testing in log space flips both case-study models to +`trustworthy`/`usable`. Independently refit here (xgboost, seed 11, 80/20): + +- **heavy equipment reproduces** — p goes 9.8e-55 (price) → **0.585** (log), + verdict `unreliable` → `trustworthy`. +- **used cars does not** — p goes 4.4e-71 → **0.034**, still under 0.05, still + `unreliable`. Its mean residual is −27.2%, far larger than a retransformation + artifact; the heavy right tail of the vehicle catalog leaves real mean bias + that a log-space test should not excuse. + +Different fit, different numbers — this doesn't make the review wrong about +the mechanism. But it does mean **testing in log space is not on its own a +sufficient fix**, and a p-value at 0.034 vs 0.585 deciding a headline verdict +is itself the argument for a materiality floor. + +## Proposed fix + +Two changes in `quality.py`, no model-format change, no retrain: + +1. **Add an effect-size floor.** Flag bias only when it is *both* statistically + detectable *and* commercially material — e.g. `|median relative residual| > + 5%`. Judge the number a buyer would feel, not the one n inflates. +2. **Test where the model is unbiased.** For log-target models run the test on + log residuals, so a median-unbiased model isn't failed for a mean it never + claimed to estimate. +3. **Stop reading small-n silence as a pass.** Below the size where a material + bias would be detectable, the honest verdict is `unknown`, which already + exists — not `trustworthy`. + +Item 3 matters most for safety and is the one nobody has proposed yet. + +## Limits + +- Models here are refit (xgboost, no tuning, seed 11) rather than the on-disk + case-study models, which don't store signed holdout residuals. R² lands + within ~0.04 of the published figures; the effect is a property of the test, + not of a particular fit. +- A 5% floor is a starting point, not a derived constant. Picking it properly + wants a look across more models than three. +- Fixing the gate does not make a mean-biased model mean-unbiased. It stops + *mislabelling*; whether to also correct the back-transform is the separate + open question in `log_retransformation_bias.md`. diff --git a/research/bias_gate_materiality.py b/research/bias_gate_materiality.py new file mode 100644 index 0000000..021abb1 --- /dev/null +++ b/research/bias_gate_materiality.py @@ -0,0 +1,153 @@ +"""Is the 'unreliable' gate measuring bias, or measuring sample size? + +Reproduction script for `research/bias_gate_materiality.md`. Read-only: it +trains throwaway models in memory and never writes a model or touches core. + +Three experiments per dataset: + 1. price-space vs log-space bias test, and the verdict each produces + 2. effect size (mean/median residual %) alongside the p-value + 3. the same model judged on holdouts of growing size -- does the verdict + track the bias, or track n? + +Run from the repo root: python research/bias_gate_materiality.py +""" +from __future__ import annotations + +import json +import sys +from pathlib import Path + +import numpy as np +import pandas as pd +from scipy.stats import ttest_1samp +from sklearn.model_selection import train_test_split + +REPO = Path(__file__).resolve().parents[1] +sys.path.insert(0, str(REPO / "src")) + +from p2predict.quality import assess_model, residual_bias_p # noqa: E402 +from p2predict.training import start_training # noqa: E402 + +SEED = 11 + +# (label, csv (full), csv (committed sample fallback), target, features) +DATASETS = [ + ("used cars", + "case-studies/used-cars/data/vehicles_training.csv", + "case-studies/used-cars/data-sample/vehicles_sample.csv", + "price", + ["year", "odometer", "manufacturer", "condition", "fuel", + "transmission", "drive", "type", "state", "paint_color"]), + ("heavy equipment", + "case-studies/heavy-equipment-sales/data/bulldozers_training.csv", + "case-studies/heavy-equipment-sales/data-sample/bulldozers_sample.csv", + "sale_price_usd", + ["age_at_sale", "sale_year", "product_group", "product_size", + "enclosure", "state"]), + ("battery mgmt ICs", + "case-studies/battery-management-ics/data/bmics_clean.csv", + "case-studies/battery-management-ics/data-sample/bmics_sample.csv", + "unit_price_at_1_usd", + ["manufacturer", "Battery Chemistry", "Interface", "max_cells_supported", + "op_temp_min_C", "op_temp_max_C", "package_pins", "is_multi_cell"]), +] + + +def log_space_bias_p(y_true, y_pred) -> float: + """The same t-test, run where a log-target model is actually unbiased.""" + yt = np.asarray(y_true, float) + yp = np.asarray(y_pred, float) + ok = (yt > 0) & (yp > 0) + resid = np.log(yt[ok]) - np.log(yp[ok]) + return float(ttest_1samp(resid, 0.0).pvalue) if resid.size >= 2 else float("nan") + + +def fit(df: pd.DataFrame, target: str, features: list[str]): + df = df.dropna(subset=[target]) + df = df[df[target] > 0] + X, y = df[features], df[target] + numeric = [f for f in features if pd.api.types.is_numeric_dtype(X[f])] + categorical = [f for f in features if f not in numeric] + X_train, X_test, y_train, y_test = train_test_split( + X, y, test_size=0.2, random_state=SEED + ) + model, _, log_target = start_training( + X_train, y_train, numeric, categorical, "xgboost", log_target=None + ) + return model, X_test, y_test, log_target + + +def run(label: str, csv: str, sample: str, target: str, features: list[str]) -> dict: + path = REPO / csv + used = csv + if not path.exists(): + path, used = REPO / sample, sample + if not path.exists(): + print(f"skip {label}: no data", file=sys.stderr) + return {} + + df = pd.read_csv(path, low_memory=False) + model, X_test, y_test, log_target = fit(df, target, features) + y_pred = np.asarray(model.predict(X_test), float) + y_true = np.asarray(y_test, float) + + from sklearn.metrics import r2_score + r2 = float(r2_score(y_true, y_pred)) + n = len(y_true) + + resid_pct = (y_true - y_pred) / y_true * 100 + mean_pct, median_pct = float(np.mean(resid_pct)), float(np.median(resid_pct)) + p_price = residual_bias_p(y_true, y_pred) + p_log = log_space_bias_p(y_true, y_pred) + + now = assess_model(r2, p_price, n)["verdict"] + in_log = assess_model(r2, p_log, n)["verdict"] + + print(f"\n=== {label} ({used.split('/')[-1]}, log_target={log_target}, " + f"n_holdout={n:,}, R2 {r2:.3f})") + print(f" mean residual {mean_pct:+.1f}% median residual {median_pct:+.1f}%") + print(f" bias p, price space {p_price:.2e} -> verdict '{now}'") + print(f" bias p, log space {p_log:.2e} -> verdict '{in_log}'") + + # Experiment 3: hold the model and its bias FIXED, vary only the holdout + # size. 200 random draws per size, so the rate is not one lucky sample. + print(f" same model, same bias -- only n varies (200 draws each)") + print(f" {'n holdout':>10} {'median mean-resid':>18} {'median bias p':>14} " + f"{'% unreliable':>13}") + ladder = [] + rng = np.random.default_rng(SEED) + for k in [50, 100, 250, 500, 1000, 2500, 5000, 10000, n]: + if k > n: + continue + draws = 1 if k == n else 200 + ps, ms, unreliable = [], [], 0 + for _ in range(draws): + idx = rng.choice(n, size=k, replace=False) if k < n else np.arange(n) + p_k = residual_bias_p(y_true[idx], y_pred[idx]) + ps.append(p_k) + ms.append(float(np.mean((y_true[idx] - y_pred[idx]) / y_true[idx] * 100))) + unreliable += assess_model(r2, p_k, k)["verdict"] == "unreliable" + pct = unreliable / draws * 100 + ladder.append({"n": int(k), "draws": draws, + "median_mean_resid_pct": float(np.median(ms)), + "median_bias_p": float(np.median(ps)), + "pct_flagged_unreliable": pct}) + print(f" {k:>10,} {np.median(ms):>17.1f}% {np.median(ps):>14.2e} " + f"{pct:>12.0f}%") + + return {"dataset": label, "data_file": used, "log_target": bool(log_target), + "n_holdout": n, "r2": r2, "mean_residual_pct": mean_pct, + "median_residual_pct": median_pct, "bias_p_price_space": p_price, + "bias_p_log_space": p_log, "verdict_today": now, + "verdict_log_space_test": in_log, "holdout_ladder": ladder} + + +def main() -> None: + out = [r for r in (run(*d) for d in DATASETS) if r] + dest = REPO / "research" / "bias_gate_materiality_results.json" + dest.write_text(json.dumps(out, indent=2, default=str)) + print(f"\nwrote {dest.relative_to(REPO)}") + + +if __name__ == "__main__": + main() diff --git a/research/bias_gate_materiality_results.json b/research/bias_gate_materiality_results.json new file mode 100644 index 0000000..c3dd1f6 --- /dev/null +++ b/research/bias_gate_materiality_results.json @@ -0,0 +1,180 @@ +[ + { + "dataset": "used cars", + "data_file": "case-studies/used-cars/data/vehicles_training.csv", + "log_target": true, + "n_holdout": 16000, + "r2": 0.7353941576631716, + "mean_residual_pct": -27.198471760493604, + "median_residual_pct": 1.7033311883896907, + "bias_p_price_space": 4.357957258363544e-71, + "bias_p_log_space": 0.034380963485015265, + "verdict_today": "unreliable", + "verdict_log_space_test": "unreliable", + "holdout_ladder": [ + { + "n": 50, + "draws": 200, + "median_mean_resid_pct": -12.13203226108943, + "median_bias_p": 0.30522821520488375, + "pct_flagged_unreliable": 12.5 + }, + { + "n": 100, + "draws": 200, + "median_mean_resid_pct": -14.866099944313994, + "median_bias_p": 0.11797094096280161, + "pct_flagged_unreliable": 33.0 + }, + { + "n": 250, + "draws": 200, + "median_mean_resid_pct": -24.480070662698587, + "median_bias_p": 0.021131114285942405, + "pct_flagged_unreliable": 64.0 + }, + { + "n": 500, + "draws": 200, + "median_mean_resid_pct": -23.41629003387107, + "median_bias_p": 0.0012603221086395387, + "pct_flagged_unreliable": 92.5 + }, + { + "n": 1000, + "draws": 200, + "median_mean_resid_pct": -27.209608982124074, + "median_bias_p": 1.4456126754135706e-05, + "pct_flagged_unreliable": 99.5 + }, + { + "n": 2500, + "draws": 200, + "median_mean_resid_pct": -27.402488299308864, + "median_bias_p": 1.6290560213405546e-12, + "pct_flagged_unreliable": 100.0 + }, + { + "n": 5000, + "draws": 200, + "median_mean_resid_pct": -27.036266004790317, + "median_bias_p": 2.5077476624600483e-23, + "pct_flagged_unreliable": 100.0 + }, + { + "n": 10000, + "draws": 200, + "median_mean_resid_pct": -27.367186414618114, + "median_bias_p": 1.3567403697448546e-45, + "pct_flagged_unreliable": 100.0 + }, + { + "n": 16000, + "draws": 1, + "median_mean_resid_pct": -27.198471760493604, + "median_bias_p": 4.357957258363544e-71, + "pct_flagged_unreliable": 100.0 + } + ] + }, + { + "dataset": "heavy equipment", + "data_file": "case-studies/heavy-equipment-sales/data/bulldozers_training.csv", + "log_target": true, + "n_holdout": 16000, + "r2": 0.7372389079537782, + "mean_residual_pct": -5.797412863436343, + "median_residual_pct": 1.706896998355263, + "bias_p_price_space": 9.794172013683398e-55, + "bias_p_log_space": 0.5849488735995615, + "verdict_today": "unreliable", + "verdict_log_space_test": "trustworthy", + "holdout_ladder": [ + { + "n": 50, + "draws": 200, + "median_mean_resid_pct": -5.444721698790989, + "median_bias_p": 0.31793557277289963, + "pct_flagged_unreliable": 9.5 + }, + { + "n": 100, + "draws": 200, + "median_mean_resid_pct": -5.35476375369146, + "median_bias_p": 0.19219692346950218, + "pct_flagged_unreliable": 23.0 + }, + { + "n": 250, + "draws": 200, + "median_mean_resid_pct": -6.14803377632524, + "median_bias_p": 0.06417769739873808, + "pct_flagged_unreliable": 47.0 + }, + { + "n": 500, + "draws": 200, + "median_mean_resid_pct": -5.718994965023414, + "median_bias_p": 0.006557710973585649, + "pct_flagged_unreliable": 80.0 + }, + { + "n": 1000, + "draws": 200, + "median_mean_resid_pct": -5.73801408146366, + "median_bias_p": 9.164335419107919e-05, + "pct_flagged_unreliable": 96.5 + }, + { + "n": 2500, + "draws": 200, + "median_mean_resid_pct": -5.827380261763855, + "median_bias_p": 1.1256645330614853e-09, + "pct_flagged_unreliable": 100.0 + }, + { + "n": 5000, + "draws": 200, + "median_mean_resid_pct": -5.822081225840856, + "median_bias_p": 5.7550065116760525e-18, + "pct_flagged_unreliable": 100.0 + }, + { + "n": 10000, + "draws": 200, + "median_mean_resid_pct": -5.800075819153561, + "median_bias_p": 1.6034903656914787e-34, + "pct_flagged_unreliable": 100.0 + }, + { + "n": 16000, + "draws": 1, + "median_mean_resid_pct": -5.797412863436343, + "median_bias_p": 9.794172013683398e-55, + "pct_flagged_unreliable": 100.0 + } + ] + }, + { + "dataset": "battery mgmt ICs", + "data_file": "case-studies/battery-management-ics/data/bmics_clean.csv", + "log_target": false, + "n_holdout": 30, + "r2": 0.6063022242956453, + "mean_residual_pct": -7.527171528525162, + "median_residual_pct": -0.8401482015296718, + "bias_p_price_space": 0.585188042426547, + "bias_p_log_space": 0.7720011510827383, + "verdict_today": "trustworthy", + "verdict_log_space_test": "trustworthy", + "holdout_ladder": [ + { + "n": 30, + "draws": 1, + "median_mean_resid_pct": -7.527171528525162, + "median_bias_p": 0.585188042426547, + "pct_flagged_unreliable": 0.0 + } + ] + } +] \ No newline at end of file diff --git a/research/methodology_gaps_benchmark_results.txt b/research/methodology_gaps_benchmark_results.txt new file mode 100644 index 0000000..d77edcb --- /dev/null +++ b/research/methodology_gaps_benchmark_results.txt @@ -0,0 +1,92 @@ +============================================================================ +FASTENERS — bolts_clean.csv +============================================================================ +---------------------------------------------------------------------------- +[fasteners] EXPERIMENT 1 — retransformation (smearing) bias +---------------------------------------------------------------------------- +rows: train 22,000 / cal 7,334 / eval 7,334 trained in 12s +smearing factor S — Duan (calibration): 4.0528 lognormal exp(sigma^2/2): 3.7627 same factor measured on eval: 3.8996 (1.0 = no retransformation bias) + naive exp() x S (smeared) +mean residual $ (pred-act) -49.3807 15.3724 +median residual $ 0.2427 29.6599 +MAE $ 59.8763 84.2201 +median APE % 81.1849 314.8019 +R^2 (price space) -0.0074 0.0711 +bias t-test p (price) 0.0000 0.0000 + +---------------------------------------------------------------------------- +[fasteners] EXPERIMENT 3 — residual-bias test space (naive predictions) +---------------------------------------------------------------------------- +test p-value verdict (gate at p<=0.05) +t-test, price space (product today) 0.0000 unreliable +t-test, log space 0.9553 usable +Wilcoxon signed-rank, price space 0.0000 unreliable + +---------------------------------------------------------------------------- +[fasteners] EXPERIMENT 2 — pre-split leakage (fixed untuned XGBoost) +---------------------------------------------------------------------------- + +n=400 rows, 10 seed(s) (11s) + selection: eval R^2 (leaky top-6) − (clean top-6): -0.0405 ± 0.3444 [top-6 set differed in 8/10 seeds] + outliers (drop): eval R^2 — honest (clean bounds, raw eval): -0.0862 + leaky bounds, raw eval: -0.0984 (bounds leakage alone: -0.0122) + leaky bounds, CLEANED eval (product today): -0.3864 (reported-vs-honest gap: -0.3002) + +n=2,000 rows, 5 seed(s) (7s) + selection: eval R^2 (leaky top-6) − (clean top-6): -0.0257 ± 0.0311 [top-6 set differed in 3/5 seeds] + outliers (drop): eval R^2 — honest (clean bounds, raw eval): -0.0869 + leaky bounds, raw eval: -0.0860 (bounds leakage alone: +0.0009) + leaky bounds, CLEANED eval (product today): -0.1160 (reported-vs-honest gap: -0.0291) + +n=36,668 rows, 2 seed(s) (21s) + selection: eval R^2 (leaky top-6) − (clean top-6): +0.0109 ± 0.0109 [top-6 set differed in 0/2 seeds] + outliers (drop): eval R^2 — honest (clean bounds, raw eval): -0.0692 + leaky bounds, raw eval: -0.0759 (bounds leakage alone: -0.0066) + leaky bounds, CLEANED eval (product today): 0.0303 (reported-vs-honest gap: +0.0996) + +============================================================================ +CARS — vehicles_training.csv +============================================================================ +---------------------------------------------------------------------------- +[cars] EXPERIMENT 1 — retransformation (smearing) bias +---------------------------------------------------------------------------- +rows: train 48,000 / cal 16,000 / eval 16,000 trained in 11s +smearing factor S — Duan (calibration): 1.0786 lognormal exp(sigma^2/2): 1.1164 same factor measured on eval: 1.0831 (1.0 = no retransformation bias) + naive exp() x S (smeared) +mean residual $ (pred-act) -1,419.3064 2.1688 +median residual $ -348.8113 631.1590 +MAE $ 4,387.7833 4,410.7872 +median APE % 18.1418 18.7224 +R^2 (price space) 0.7408 0.7536 +bias t-test p (price) 0.0000 0.9700 + +---------------------------------------------------------------------------- +[cars] EXPERIMENT 3 — residual-bias test space (naive predictions) +---------------------------------------------------------------------------- +test p-value verdict (gate at p<=0.05) +t-test, price space (product today) 0.0000 unreliable +t-test, log space 0.5023 trustworthy +Wilcoxon signed-rank, price space 0.0000 unreliable + +---------------------------------------------------------------------------- +[cars] EXPERIMENT 2 — pre-split leakage (fixed untuned XGBoost) +---------------------------------------------------------------------------- + +n=400 rows, 10 seed(s) (12s) + selection: eval R^2 (leaky top-6) − (clean top-6): +0.0125 ± 0.0383 [top-6 set differed in 4/10 seeds] + outliers (drop): eval R^2 — honest (clean bounds, raw eval): 0.5227 + leaky bounds, raw eval: 0.5129 (bounds leakage alone: -0.0098) + leaky bounds, CLEANED eval (product today): 0.5145 (reported-vs-honest gap: -0.0082) + +n=2,000 rows, 5 seed(s) (8s) + selection: eval R^2 (leaky top-6) − (clean top-6): +0.0001 ± 0.0047 [top-6 set differed in 0/5 seeds] + outliers (drop): eval R^2 — honest (clean bounds, raw eval): 0.6505 + leaky bounds, raw eval: 0.6521 (bounds leakage alone: +0.0016) + leaky bounds, CLEANED eval (product today): 0.6768 (reported-vs-honest gap: +0.0263) + +n=80,000 rows, 2 seed(s) (44s) + selection: eval R^2 (leaky top-6) − (clean top-6): +0.0000 ± 0.0000 [top-6 set differed in 0/2 seeds] + outliers (drop): eval R^2 — honest (clean bounds, raw eval): 0.7325 + leaky bounds, raw eval: 0.7293 (bounds leakage alone: -0.0032) + leaky bounds, CLEANED eval (product today): 0.7922 (reported-vs-honest gap: +0.0597) + diff --git a/research/methodology_review.md b/research/methodology_review.md new file mode 100644 index 0000000..98d59cf --- /dev/null +++ b/research/methodology_review.md @@ -0,0 +1,220 @@ +# Methodology review — what's missing, scientifically + +A statistics/ML audit of the modeling core: training and model selection +(`training.py`), preprocessing (`preprocessing.py`, `outliers.py`, +`feature_selection.py`), conformal intervals (`intervals.py`), explanations +(`explain.py`), and quality verdicts (`quality.py`). Findings are ranked by +how much they can mislead the numbers we hand a category manager. The Tier-1 +items are **quantified on the case-study datasets** by +`research/benchmark_methodology_gaps.py`; measured results live in +`research/methodology_gaps_benchmark_results.txt` and are summarized +inline below. + +Per the project's working agreement, this PR changes nothing in core — it +proposes; the follow-up PRs dispose. + +## Status (2026-09-10) — read this first + +This review was written 2026-07-08 and sat open for two months. Two of its +findings have since been re-derived independently and now have their own, +deeper write-ups; two are still open and unaddressed in core. Current state: + +| Finding | Status | +|---|---| +| **1. Retransformation bias** (`exp()` back-transform undershoots the mean) | **Superseded** by [`log_retransformation_bias.md`](log_retransformation_bias.md), which re-derived it from the heavy-equipment study with a reproducible script and an explicit mean-vs-median options analysis. Read that instead. | +| **2. Bias gate flunks good models** | **Open, and sharpened** — see [`bias_gate_materiality.md`](bias_gate_materiality.md), which measures the gate across three datasets and shows the failure is *worse and two-sided*: the verdict tracks holdout size, not bias. One correction to this review: it reported the used-cars model flipping to `trustworthy` in log space; an independent fit does **not** reproduce that flip. | +| **3. Split hygiene / raw-holdout reporting** | **Open**, but narrower than ranked here — the inflation only reaches users who set `--feature-outliers drop`; the default `warn` is unaffected. The sub-item "wire `find_leaky_features` into the CLI" partly landed: it is used by the MCP server (`propose_training_plan`, `train`), not the CLI. | +| **4. Group-aware splits** | **Open.** No `GroupShuffleSplit`/`GroupKFold` anywhere in `src/`. | +| Tier-2/3: *silent out-of-distribution predictions* | **Worked up** as [`out_of_domain_flag.md`](out_of_domain_flag.md) (merged in #36), which confirmed the bullet and showed the reliability verdict is structurally incapable of catching it. | + +The Tier-1 numbers below are from the original July run and have not been +re-measured except where a linked document says otherwise. + +## What's already solid + +Worth stating first, because these are the traps most tools in this space +fall into and P2Predict doesn't: + +- **Target encoding is cross-fitted.** The `TargetEncoder` sits inside the + pipeline handed to CV, uses internal 5-fold cross-fitting with + empirical-Bayes smoothing (`smooth="auto"`), and refits per fold — the + classic target-encoding leakage bug is absent (`preprocessing.py`). +- **Conformal intervals are done right.** Split conformal with the exact + finite-sample quantile `ceil((n+1)(1−α))/n`, conservatively rounded; + Mondrian tercile banding when calibration n ≥ 150; multiplicative bounds + for log-target models so ranges never go negative (`intervals.py`). +- **Explanations are exact.** `LinearExplainer`/`TreeExplainer` only — no + KernelExplainer approximations (`explain.py`). +- **Verdicts are honesty-gated** on holdout size and residual bias + (`quality.py`). + +## Tier 1 — the math is biased or leaks (fix) + +### 1. Retransformation bias: log-target models back-transform with plain `exp()` + +`build_pipeline` wraps the estimator in +`TransformedTargetRegressor(func=np.log, inverse_func=np.exp)` +(`training.py:152-154`). Because `E[exp(ŷ+ε)] ≠ exp(ŷ)`, the naive +back-transform is biased low in price space by roughly `E[exp(ε)]` — and it +fires precisely on the heavily-skewed datasets where the log wrap +auto-triggers. The textbook fix is Duan's smearing estimator: multiply the +naive prediction by `S = mean(exp(ε̂ᵢ))` over **out-of-sample** residuals +(a boosted tree's training residuals are near zero, so S must come from the +holdout or CV out-of-fold residuals, not the training fit). + +**Measured (Experiment 1):** + +- **Used cars** (48k train / 16k eval, R² 0.74): naive predictions run + **7.9% low in mean terms** (S = 1.079). Mean holdout residual −$1,419 → + **+$2** after smearing; price-space R² 0.741 → 0.754; the bias t-test goes + from p < 10⁻⁴ to p = 0.97. Median APE essentially unchanged + (18.1% → 18.7%). Textbook case: real bias, cleanly corrected. +- **Fasteners** (intrinsically noisy catalog, log residuals heavy-tailed): + S = **4.05**. Smearing fixes the mean (−$49 → +$15) but multiplies every + quote by ~4×, so median APE explodes 81% → 315%. The correction is + mathematically right for the mean and commercially wrong for a per-part + quote on this data. + +**Nuance the numbers confirm:** naive `exp()` is approximately +*median*-unbiased (fasteners median residual $0.24), while smearing targets +*mean*-unbiasedness — and on heavy-tailed log residuals S is dominated by a +few extreme ratios. Recommendation for the core fix: apply smearing where the +mean matters (aggregate-spend estimates, batch totals) and keep the per-part +point estimate median-anchored, or expose both — but the choice should be +explicit rather than an accident of `inverse_func=np.exp`. + +### 2. Pre-split leakage: outlier bounds and feature selection see the holdout + +The CLI computes Tukey outlier bounds (`cli/train.py`, target and feature +policies) and RF-importance feature selection +(`feature_selection.get_most_predictable_features`, auto mode keeps top 6) +on the **full dataset before the train/test split**. Two consequences: + +- the selection/bounds decisions have seen the holdout rows (information + leakage into the reported R², band errors, and conformal calibration); +- with `drop` policies, the holdout itself is cleaned of the extremes + production quotes will contain, so the reported error understates + deployment error. + +**Measured (Experiment 2) — smaller than principle suggests:** + +- The *information leakage itself* is *negligible* at every scale tested: + bounds fitted pre- vs post-split move eval R² by ≤ ~0.01; leaky feature + selection is noisy at n=400 (±0.34 across seeds on fasteners) but not + systematically optimistic. +- The *material* piece is **reporting on a cleaned holdout** under `drop` + policies: at full scale the reported R² beats the honest raw-holdout R² by + **+0.06 on cars (0.792 vs 0.733)** and **+0.10 on fasteners**; at small n + the direction is erratic (fasteners n=400: −0.30) because dropping extremes + also shrinks target variance. The default `warn` policy is unaffected. + +**Fix (hygiene, demoted below #3 in urgency):** split first; fit outlier +policy and feature selection on the training rows only; report holdout +metrics on raw holdout rows so the quoted error reflects the quotes +production will actually see. + +### 3. The trustworthy/unreliable gate runs a t-test in the wrong space + +`quality.residual_bias_p` runs `ttest_1samp` on **raw price residuals** +(`quality.py:333-340`) even for log-target models, and `assess_model` gates +`trustworthy` vs `unreliable` on that single p-value at 0.05. Price residuals +on skewed targets are heavily skewed and dominated by the expensive tail — +the t-test's assumptions are weakest exactly where it decides the verdict. +Compounding: the retransformation bias of finding #1 shows up as a price-space +mean shift, so a model that is well-behaved in log space gets branded +"unreliable" for a defect the back-transform introduced. + +**Measured (Experiment 3) — both case studies flip verdict:** + +| dataset | t-test, price space (today) | t-test, log space | +|---|---|---| +| cars (R² 0.74) | p < 10⁻⁴ → **unreliable** | p = 0.50 → **trustworthy** | +| fasteners | p < 10⁻⁴ → **unreliable** | p = 0.96 → **usable** | + +Both models are well-behaved in log space; the price-space test is flunking +them for the retransformation bias of finding #1. A price-space Wilcoxon +also trips (p < 10⁻³) — which exposes a second defect: the gate is **pure +statistical significance with no materiality threshold**. On a 16k-row +holdout, a −$349 median residual on ~$18k cars (−2%) is "significant" and +costs the model its verdict. + +**Fix:** for log-target models test residuals in log space; add an +effect-size floor (e.g. |mean or median residual| above some % of the mean +price) alongside the p-value; re-run after the smearing correction of +finding #1. + +### 4. No group-aware validation + +Random KFold inside `HalvingRandomSearchCV` and a random 80/20 +`train_test_split` (`prepare_data.py`) mean near-duplicate parts, part-family +variants, and same-supplier rows straddle the train/test line. Holdout scores +are optimistic on real catalogs, which are full of such near-duplicates. Not +benchmarked here (the case-study CSVs carry no part-family key), but the +mechanism is well-established. + +**Fix:** optional `--group-column` → `GroupShuffleSplit` for the holdout and +`GroupKFold` for CV; recommend it in `propose_training_plan` when a column +looks like a part/family/supplier identifier. + +## Tier 2 — missing methodology that undermines the product's claims + +5. **No out-of-distribution guard at predict time.** An unseen supplier + silently gets the global target mean (`TargetEncoder`) or an all-zero + one-hot vector; numeric inputs outside the training range are not flagged + anywhere in `predict`/`what_if`/`predict_batch`. The tool hands a + category manager a confident benchmark for a part unlike anything it + trained on. Fix: flag unseen category levels and out-of-range numerics in + the predict payload with a `say_to_user` caveat. +6. **Intervals are not input-adaptive.** Width is constant (or + piecewise-constant across 3 predicted-price terciles); it never widens + for genuinely unusual parts. Later: normalized conformal scores or CQR. +7. **Model selection rides a single CV point estimate** — chosen on + log-space R² but graded on raw-space R², no 1-SE rule, no selection-aware + inference. The family choice is noisy on small data and the winning CV + score is optimistically biased. +8. **Log-model dollar attribution divides by the sum of log contributions** + (`explain.py`, `dollar_attribution`): when drivers nearly cancel + (`Σφ ≈ 0`) the rescale blows up. Needs a guard. Related: the user-facing + "feature signal" uses native importances (coefficients / tree gain) while + explanations use SHAP — two different importance notions surfaced to the + same user. +9. **One holdout does everything** — R², the bias test, per-band APE, and + conformal calibration all ride the same ≤20% sample. Valid individually, + but there is no independent check on any of them. + +## Tier 3 — worth noting + +10. No minimum-sample guard to train — only post-hoc verdict caveats; no + learning-curve signal to tell a user "50 more rows would help". +11. XGBoost regularization (`reg_alpha`/`reg_lambda`/`min_child_weight`) and + early stopping are untuned; no ElasticNet/LightGBM in the zoo. +12. The leakage screen `find_leaky_features` is wired into the MCP path only; + the CLI `train` never calls it. +13. Impurity-based importances (biased toward high-cardinality + target-encoded columns) drive both auto-selection and the user-facing + "price drivers", while an unbiased permutation-importance helper + (`model_evals.calculate_feature_importance`) sits unused. +14. No model-staleness signal: `training_date` is persisted but never + checked, and there is no drift monitoring story. + +## Proposed follow-up PRs (re-ordered by the evidence) + +1. **Bias test in the right space, with a materiality floor** (`quality.py`): + log-space residuals for log-target models plus an effect-size threshold. + Highest urgency — today it mislabels both flagship case-study models as + "unreliable"-grade, and the failure direction (under-trust) is quietly + costing good models their verdict. +2. **Smearing made explicit** for the log-target back-transform + (`training.py`): correct the mean where the mean matters (batch totals, + aggregate spend), keep per-part quotes median-anchored; document the + choice. On cars this is a real 8% mean underestimate. +3. **Split-first hygiene** (`cli/train.py`, `mcp/server.py`): fit outlier + policy and feature selection on training rows only and report holdout + metrics on raw holdout rows (closes the +0.06–0.10 reported-vs-honest R² + gap under drop policies); wire `find_leaky_features` into the CLI on the + way through. +4. **Optional group column** (`prepare_data.py`, `training.py`): + `GroupShuffleSplit`/`GroupKFold`, surfaced in `propose_training_plan`. + +Tier-2 items are follow-on candidates once the above land; #5 (OOD flags) is +the highest-value of them for the agentic surface.