[ahmed-khalil-hafsi]

Rebases the July methodology audit onto current main, moves it into research/, and adds a measured follow-on finding. No core changes.

Why this sat for two months

It bundles four findings, so merging it meant accepting a four-item agenda. Two have since been re-derived independently, which is evidence the bundle was too big to act on — not that the findings were wrong. This PR splits the bundle apart instead.

What changed in this update

• Rebased onto main and moved all three files into research/ — the PR still targeted analysis/, which 429ee7c renamed away. Merging as-is would have resurrected the dead folder.
• Added a Status header to methodology_review.md so nothing dangles:

Finding	Status
1. Retransformation bias	Superseded by log_retransformation_bias.md
2. Bias gate flunks good models	Open, sharpened by the new doc below
3. Split hygiene / raw-holdout reporting	Open, narrower than ranked — only bites --feature-outliers drop; default warn unaffected. Sub-item "wire find_leaky_features into the CLI" partly landed (it's in the MCP server, not the CLI).
4. Group-aware splits	Open — no GroupShuffleSplit/GroupKFold in src/
Tier-2/3: silent OOD predictions	Worked up as out_of_domain_flag.md, merged in #36

• Added bias_gate_materiality.{md,py,json} — finding v0.4: SHAP per-prediction explanations #2, measured.

The new finding: the gate measures n, not bias

assess_model() calls a model biased when residual_bias_p <= 0.05 — a t-test on price-space residuals (quality.py:333). A p-value answers "could this be noise?", which depends on sample size. It does not answer "is this big enough to matter?", which is what the verdict claims to report.

Measured by holding the model and its bias fixed and varying only the holdout size — 200 random draws per row, heavy-equipment model, R² 0.737:

n holdout	median mean-residual	median bias p	% flagged unreliable
50	−5.4%	3.2e-01	10%
250	−6.1%	6.4e-02	47%
1,000	−5.7%	9.2e-05	96%
16,000	−5.8%	9.8e-55	100%

The bias column is flat. The verdict column goes 10% → 100%.

It fails in both directions, because unbiased = p > 0.05 reads failure to reject as proof of no bias:

• Large holdout → guaranteed unreliable. Both flagship case studies earn it today despite median residuals of +1.7% — the user with the best price history is likeliest to be told their model can't be trusted.
• Small holdout → free pass. The battery-management-IC model carries a −7.5% mean residual on 30 parts and is stamped trustworthy. This over-trust direction is the one the July review explicitly ruled out ("failure direction is under-trust").

Dataset	R²	mean resid	median resid	p	verdict today
used cars (n=16,000)	0.735	−27.2%	+1.7%	4.4e-71	unreliable
heavy equipment (n=16,000)	0.737	−5.8%	+1.7%	9.8e-55	unreliable
battery mgmt ICs (n=30)	0.606	−7.5%	−0.8%	5.9e-01	trustworthy

One correction to the July review

It reported both case-study models flipping to trustworthy/usable when tested in log space. An independent refit reproduces this for heavy equipment (p 9.8e-55 → 0.585) but not for used cars (4.4e-71 → 0.034, still failing) — its −27.2% mean residual is larger than a retransformation artifact can explain. So testing in log space is not on its own a sufficient fix, and a verdict hinging on 0.034 vs 0.585 is itself the argument for a materiality floor.

Reproduce

python research/bias_gate_materiality.py
python research/benchmark_methodology_gaps.py [fasteners|cars]