Repository navigation
MCP-friendly model quality: get_model_quality + centralized judgment layer - #23
Merged
Merged
Conversation
The only computed quality verdict was an R2-only label, duplicated in the trainer and unavailable to agents in structured form; the judgment that matters for procurement (unbiased? which bands to trust? which features are quotable?) lived only as prose. - New src/p2predict/quality.py: single source for all quality thresholds and verdicts (r2 label, band reliability, feature signal, multi-factor assess_model, build_quality_report). Pure stats (summary_metrics, error_by_price_band) now live here; plotting.py imports them so the PDF and JSON compute identical numbers. - New MCP tool get_model_quality(model_id): structured, agent-readable report with computed verdicts (assessment.unbiased/headline, per-band reliability trust|caution|quote, per-feature signal strong|moderate|weak). - generate_report now echoes the same quality block alongside the PDF path, so the agent can reason over the report instead of only handing back a file. - De-dup: server.py and cli/train.py both use quality.r2_quality_label. - Tests: tests/test_quality.py (8) + MCP get_model_quality / report-echo tests.
- assess_model now returns a headline 'verdict' that folds in bias and sample
size: trustworthy | usable | unreliable (biased) | unknown (bias not
measurable) | insufficient_data (too few holdout points). A modest-R2 but
unbiased model reads as 'usable', not just 'Needs Improvement'.
- confidence ('high'|'limited'|'insufficient') from holdout size; small holdouts
get an explicit caveat and tiny ones short-circuit to insufficient_data.
- Per-band low_confidence flag + note when a band is too thin to act on; a
calibration_note when there are too few points to band at all.
- get_model_quality gains include_holdout=true -> returns raw actual/predicted
arrays so an agent with a plotting/code tool can draw its own charts.
- Tests cover every verdict/confidence path, thin bands, and include_holdout.
ahmed-khalil-hafsi
added a commit
that referenced
this pull request
Jun 28, 2026
…layer (#23) * Add get_model_quality MCP tool + centralized quality judgment layer The only computed quality verdict was an R2-only label, duplicated in the trainer and unavailable to agents in structured form; the judgment that matters for procurement (unbiased? which bands to trust? which features are quotable?) lived only as prose. - New src/p2predict/quality.py: single source for all quality thresholds and verdicts (r2 label, band reliability, feature signal, multi-factor assess_model, build_quality_report). Pure stats (summary_metrics, error_by_price_band) now live here; plotting.py imports them so the PDF and JSON compute identical numbers. - New MCP tool get_model_quality(model_id): structured, agent-readable report with computed verdicts (assessment.unbiased/headline, per-band reliability trust|caution|quote, per-feature signal strong|moderate|weak). - generate_report now echoes the same quality block alongside the PDF path, so the agent can reason over the report instead of only handing back a file. - De-dup: server.py and cli/train.py both use quality.r2_quality_label. - Tests: tests/test_quality.py (8) + MCP get_model_quality / report-echo tests. * Make quality verdict bias- and confidence-aware; expose raw holdout - assess_model now returns a headline 'verdict' that folds in bias and sample size: trustworthy | usable | unreliable (biased) | unknown (bias not measurable) | insufficient_data (too few holdout points). A modest-R2 but unbiased model reads as 'usable', not just 'Needs Improvement'. - confidence ('high'|'limited'|'insufficient') from holdout size; small holdouts get an explicit caveat and tiny ones short-circuit to insufficient_data. - Per-band low_confidence flag + note when a band is too thin to act on; a calibration_note when there are too few points to band at all. - get_model_quality gains include_holdout=true -> returns raw actual/predicted arrays so an agent with a plotting/code tool can draw its own charts. - Tests cover every verdict/confidence path, thin bands, and include_holdout. --------- Co-authored-by: Ahmed K. Hafsi <25119333+ahmed-hafsi@users.noreply.github.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Adds an agent-readable quality report to the MCP layer and centralizes model-quality judgment in one module.
The gap
generate_reportreturned only a PDF path — an agent can't read numbers out of a PDF to reason or relay. And the only computed verdict was an R²-only label, duplicated in the trainer and CLI, which contradicts the toolkit's own guidance ("judge by bias, not R² alone"). The richer judgment (bias, calibration-by-band, feature signal) existed only as prose.Changes
src/p2predict/quality.py(new) — single source of truth for quality thresholds + verdicts:r2_quality_label,band_reliability(trust/caution/quote),feature_signal(strong/moderate/weak),assess_model(multi-factor: combines accuracy and bias),build_quality_report.summary_metrics,error_by_price_band) now live here;plotting.pyimports them so the PDF and JSON compute identical numbers.get_model_quality(model_id)(new MCP tool) — structured report: provenance, metrics,assessment, per-bandreliability, per-featuresignal.generate_reportnow echoes the samequalityblock alongside the PDF path.server.pyandcli/train.pyboth callquality.r2_quality_label.Verification
On real BMIC data,
get_model_qualityreproduces the case study's hand-written trust map automatically: "modest but unbiased", cheap bands →quote, mid bands →trust, supplier/chemistry/interface →strong.Tests:
tests/test_quality.py(8) + MCPget_model_quality/ report-echo tests. Full touched-area suite green.