Skip to content

Commit 9fe66cd

Browse files
matthew-kuanclaude
andcommitted
feat: wire the Gemini client and land steps 2-7
1. CLIENT. build_model_callable() returns a real RouterClient reading GEMINI_API_KEY from the environment via a dependency-free .env loader. Temperature 0, structured response schema, prompt-hash response cache for byte-identical re-runs. The first attempt failed 89 of 110 rows: on a free-tier key gemini-2.5-flash allows 20 requests PER DAY, and the retry loop burned 3x quota per failure. Daily-quota 429s now fail fast, per-minute ones honour the server's retry delay, and requests are paced. Default model is gemini-3.5-flash-lite, which has its own quota and measured ~10x faster. Result: 110/110 valid on first try, 0 re-asks, 0 fallbacks. 2. BUG interviewstreet#1. Removed "unknown" from LOW_VALUE_TYPES. It was turning the documented "cap at digest" into a force to mute on every fallback row, because the safe default is typed unknown -- which is what muted msg_056. 3. SALIENCE. Fixed the weights rather than the docs. The scale is now symmetric by construction: reported/replied 1.0, muted-after/fast-open 0.9, dismissed/opened 0.65. reaction_time_minutes was carried through the whole pipeline and never used; it now drives the fast-open tier. 4. BASE RATES. The context now states per-user open share, group volume and notification load relative to their norms, so engagement is read against a denominator instead of absolutely. The open share spans 0.17 to 0.91 across users, so "opened 8 of 10" previously meant nothing on its own. 5. (a) Media-only history is scored on its interpretation, and similarity takes max(text-only, text+media) -- comparing text+OCR against a text-only row inflates the union and had silently dropped two sim=1.00 muted duplicates from msg_066's evidence. (b) Evidence F1 always emits a float. (c) Calibration is reported separately for router-decided and guard-overridden rows. 6. README known-limitations: no holdout, 30 in-sample rows, per-type accuracy is noise below ~3 examples, the guard cannot separate msg_056 from msg_040, and what personalization does and does not mean here. 7. validate_submission.py checks the full contract independently and prints the distribution; package.py builds code.zip and verifies .env is absent. Eval with the live router: action 86.7%, type 70.0%, joint 66.7%, ECE 0.081 (router-decided 0.079 at 85.7% accuracy), weighted cost 0.133/row, zero severe errors. Golden set 6/7. Tests: 478 assertions across five suites. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
1 parent a37f902 commit 9fe66cd

27 files changed

Lines changed: 1862 additions & 225 deletions

‎.env.example‎

Lines changed: 0 additions & 9 deletions
This file was deleted.

‎cache/media_interpretations.json‎

Lines changed: 485 additions & 0 deletions
Large diffs are not rendered by default.

‎cache/override_audit.csv‎

Lines changed: 20 additions & 36 deletions
Original file line numberDiff line numberDiff line change
@@ -1,37 +1,21 @@
11
message_id,rule,from_action,to_action,from_type,to_type,from_confidence,to_confidence,disagreement,model_fell_back,model_reason,rule_reason
2-
msg_091,reported_sender,digest,mute,unknown,spam,0.50,0.90,1,1,"Routing could not be validated for this message, so it was held for later review rather than interrupting the user.","The user has reported every past message from this sender and opened none, so this sender is suppressed."
3-
msg_040,muted_by_user,digest,mute,unknown,unknown,0.50,0.85,1,1,"Routing could not be validated for this message, so it was held for later review rather than interrupting the user.",The user has muted this conversation and the message carries no direct request.
4-
msg_069,muted_by_user,digest,mute,unknown,unknown,0.50,0.85,1,1,"Routing could not be validated for this message, so it was held for later review rather than interrupting the user.",The user has muted this conversation and the message carries no direct request.
5-
msg_030,muted_by_user,digest,mute,unknown,unknown,0.50,0.85,1,1,"Routing could not be validated for this message, so it was held for later review rather than interrupting the user.",The user has muted this conversation and the message carries no direct request.
6-
msg_052,scam_signature,digest,mute,unknown,scam,0.50,0.93,1,1,"Routing could not be validated for this message, so it was held for later review rather than interrupting the user.","Sender uses talabat-refund.com instead of the brand's talabat.com, is unverified, and has 61 reports."
7-
msg_009,muted_by_user,digest,mute,unknown,unknown,0.50,0.85,1,1,"Routing could not be validated for this message, so it was held for later review rather than interrupting the user.",The user has muted this conversation and the message carries no direct request.
8-
msg_016,reported_sender,digest,mute,unknown,spam,0.50,0.90,1,1,"Routing could not be validated for this message, so it was held for later review rather than interrupting the user.","The user has reported every past message from this sender and opened none, so this sender is suppressed."
9-
msg_014,opted_out_promotions,digest,mute,unknown,unknown,0.50,0.88,1,1,"Routing could not be validated for this message, so it was held for later review rather than interrupting the user.",The user opted out of promotions from this brand on 2026-07-17 23:55.
10-
msg_078,reported_sender,digest,mute,unknown,spam,0.50,0.90,1,1,"Routing could not be validated for this message, so it was held for later review rather than interrupting the user.","The user has reported every past message from this sender and opened none, so this sender is suppressed."
11-
msg_018,reported_sender,digest,mute,unknown,spam,0.50,0.90,1,1,"Routing could not be validated for this message, so it was held for later review rather than interrupting the user.","The user has reported every past message from this sender and opened none, so this sender is suppressed."
12-
msg_108,scam_signature,digest,mute,unknown,scam,0.50,0.93,1,1,"Routing could not be validated for this message, so it was held for later review rather than interrupting the user.","Sender uses chase-secure-alert.com instead of the brand's chase.com, is unverified, and has 55 reports."
13-
msg_019,scam_signature,digest,mute,unknown,scam,0.50,0.93,1,1,"Routing could not be validated for this message, so it was held for later review rather than interrupting the user.","Sender uses chase-secure-alert.com instead of the brand's chase.com, is unverified, and has 55 reports."
14-
msg_088,muted_by_user,digest,mute,unknown,unknown,0.50,0.85,1,1,"Routing could not be validated for this message, so it was held for later review rather than interrupting the user.",The user has muted this conversation and the message carries no direct request.
15-
msg_056,muted_by_user,digest,mute,unknown,unknown,0.50,0.85,1,1,"Routing could not be validated for this message, so it was held for later review rather than interrupting the user.",The user has muted this conversation and the message carries no direct request.
16-
msg_063,reported_sender,digest,mute,unknown,spam,0.50,0.90,1,1,"Routing could not be validated for this message, so it was held for later review rather than interrupting the user.","The user has reported every past message from this sender and opened none, so this sender is suppressed."
17-
msg_061,muted_by_user,digest,mute,unknown,unknown,0.50,0.85,1,1,"Routing could not be validated for this message, so it was held for later review rather than interrupting the user.",The user has muted this conversation and the message carries no direct request.
18-
msg_017,reported_sender,digest,mute,unknown,spam,0.50,0.90,1,1,"Routing could not be validated for this message, so it was held for later review rather than interrupting the user.","The user has reported every past message from this sender and opened none, so this sender is suppressed."
19-
msg_058,muted_by_user,digest,mute,unknown,unknown,0.50,0.85,1,1,"Routing could not be validated for this message, so it was held for later review rather than interrupting the user.",The user has muted this conversation and the message carries no direct request.
20-
msg_026,scam_signature,digest,mute,unknown,scam,0.50,0.93,1,1,"Routing could not be validated for this message, so it was held for later review rather than interrupting the user.","Sender uses amazonpay-delivery.in instead of the brand's amazon.in, is unverified, and has 47 reports."
21-
msg_046,reported_sender,digest,mute,unknown,spam,0.50,0.90,1,1,"Routing could not be validated for this message, so it was held for later review rather than interrupting the user.","The user has reported every past message from this sender and opened none, so this sender is suppressed."
22-
msg_085,scam_signature,digest,mute,unknown,scam,0.50,0.93,1,1,"Routing could not be validated for this message, so it was held for later review rather than interrupting the user.","Sender uses hdfcbank-kyc.in instead of the brand's hdfc.bank.in, is unverified, and has 38 reports."
23-
msg_028,opted_out_promotions,digest,mute,unknown,unknown,0.50,0.88,1,1,"Routing could not be validated for this message, so it was held for later review rather than interrupting the user.",The user opted out of promotions from this brand on 2026-07-17 23:55.
24-
msg_064,reported_sender,digest,mute,unknown,spam,0.50,0.90,1,1,"Routing could not be validated for this message, so it was held for later review rather than interrupting the user.","The user has reported every past message from this sender and opened none, so this sender is suppressed."
25-
msg_087,muted_by_user,digest,mute,unknown,unknown,0.50,0.85,1,1,"Routing could not be validated for this message, so it was held for later review rather than interrupting the user.",The user has muted this conversation and the message carries no direct request.
26-
msg_066,opted_out_promotions,digest,mute,unknown,unknown,0.50,0.88,1,1,"Routing could not be validated for this message, so it was held for later review rather than interrupting the user.",The user opted out of promotions from this brand on 2026-07-17 23:55.
27-
msg_051,opted_out_promotions,digest,mute,unknown,unknown,0.50,0.88,1,1,"Routing could not be validated for this message, so it was held for later review rather than interrupting the user.",The user opted out of promotions from this brand on 2026-07-17 23:55.
28-
msg_024,opted_out_promotions,digest,mute,unknown,unknown,0.50,0.88,1,1,"Routing could not be validated for this message, so it was held for later review rather than interrupting the user.",The user opted out of promotions from this brand on 2026-07-17 23:55.
29-
msg_036,scam_signature,digest,mute,unknown,scam,0.50,0.93,1,1,"Routing could not be validated for this message, so it was held for later review rather than interrupting the user.","Sender uses razorpayx-payouts.com instead of the brand's razorpayx.com, is unverified, and has 20 reports."
30-
msg_076,scam_signature,digest,mute,unknown,scam,0.50,0.93,1,1,"Routing could not be validated for this message, so it was held for later review rather than interrupting the user.","Sender uses talabat-refund.com instead of the brand's talabat.com, is unverified, and has 61 reports."
31-
msg_034,muted_by_user,digest,mute,unknown,unknown,0.50,0.85,1,1,"Routing could not be validated for this message, so it was held for later review rather than interrupting the user.",The user has muted this conversation and the message carries no direct request.
32-
msg_011,muted_by_user,digest,mute,unknown,unknown,0.50,0.85,1,1,"Routing could not be validated for this message, so it was held for later review rather than interrupting the user.",The user has muted this conversation and the message carries no direct request.
33-
msg_110,reported_sender,digest,mute,unknown,spam,0.50,0.90,1,1,"Routing could not be validated for this message, so it was held for later review rather than interrupting the user.","The user has reported every past message from this sender and opened none, so this sender is suppressed."
34-
msg_074,muted_by_user,digest,mute,unknown,unknown,0.50,0.85,1,1,"Routing could not be validated for this message, so it was held for later review rather than interrupting the user.",The user has muted this conversation and the message carries no direct request.
35-
msg_073,reported_sender,digest,mute,unknown,spam,0.50,0.90,1,1,"Routing could not be validated for this message, so it was held for later review rather than interrupting the user.","The user has reported every past message from this sender and opened none, so this sender is suppressed."
36-
msg_054,reported_sender,digest,mute,unknown,spam,0.50,0.90,1,1,"Routing could not be validated for this message, so it was held for later review rather than interrupting the user.","The user has reported every past message from this sender and opened none, so this sender is suppressed."
37-
msg_084,opted_out_promotions,digest,mute,unknown,unknown,0.50,0.88,1,1,"Routing could not be validated for this message, so it was held for later review rather than interrupting the user.",The user opted out of promotions from this brand on 2026-07-17 23:55.
2+
msg_091,reported_sender,mute,mute,scam,spam,0.99,0.90,1,0,The counterpart has been unanimously reported by other users and the message is a phishing attempt.,"The user has reported every past message from this sender and opened none, so this sender is suppressed."
3+
msg_052,scam_signature,mute,mute,scam,scam,0.99,0.93,0,0,"The scam signature is true with a domain mismatch and a young account, triggering stage 1 safety.","Sender uses talabat-refund.com instead of the brand's talabat.com, is unverified, and has 61 reports."
4+
msg_016,reported_sender,mute,mute,scam,spam,0.95,0.90,1,0,The counterpart has been unanimously reported by members for suspicious phishing activity.,"The user has reported every past message from this sender and opened none, so this sender is suppressed."
5+
msg_078,reported_sender,mute,mute,scam,spam,0.98,0.90,1,0,"An unverified group member asks for bank details with a high forward count, triggering stage one safety.","The user has reported every past message from this sender and opened none, so this sender is suppressed."
6+
msg_018,reported_sender,mute,mute,spam,spam,0.95,0.90,0,0,The counterpart has been unanimously reported by members for sending prize scam forwards.,"The user has reported every past message from this sender and opened none, so this sender is suppressed."
7+
msg_108,scam_signature,mute,mute,scam,scam,0.99,0.93,0,0,The sender's scam signature and young domain triggered a Stage 1 safety stop.,"Sender uses chase-secure-alert.com instead of the brand's chase.com, is unverified, and has 55 reports."
8+
msg_019,scam_signature,mute,mute,scam,scam,0.95,0.93,0,0,"The sender's account exhibits a scam signature with a domain mismatch, making the message unsafe in stage one.","Sender uses chase-secure-alert.com instead of the brand's chase.com, is unverified, and has 55 reports."
9+
msg_063,reported_sender,mute,mute,scam,spam,0.95,0.90,1,0,An unverified group member is demanding payment and requesting credential confirmation in stage 1.,"The user has reported every past message from this sender and opened none, so this sender is suppressed."
10+
msg_017,reported_sender,mute,mute,spam,spam,0.95,0.90,0,0,The counterpart sender has been unanimously reported by members for suspicious payment demands.,"The user has reported every past message from this sender and opened none, so this sender is suppressed."
11+
msg_026,scam_signature,mute,mute,scam,scam,0.99,0.93,0,0,The sender's domain does not match the brand's and the scam signature is true in stage one.,"Sender uses amazonpay-delivery.in instead of the brand's amazon.in, is unverified, and has 47 reports."
12+
msg_046,reported_sender,mute,mute,scam,spam,0.99,0.90,1,0,"The sender is asking for an OTP and account credentials, making it a safety violation in Stage 1.","The user has reported every past message from this sender and opened none, so this sender is suppressed."
13+
msg_085,scam_signature,mute,mute,scam,scam,0.95,0.93,0,0,The scam signature is true with an unverified account requesting an OTP.,"Sender uses hdfcbank-kyc.in instead of the brand's hdfc.bank.in, is unverified, and has 38 reports."
14+
msg_064,reported_sender,mute,mute,scam,spam,0.95,0.90,1,0,The counterpart has been unanimously reported by members for suspicious refund and KYC demands.,"The user has reported every past message from this sender and opened none, so this sender is suppressed."
15+
msg_036,scam_signature,mute,mute,scam,scam,0.99,0.93,0,0,The sender's domain does not match the brand and the scam signature is true in stage 1.,"Sender uses razorpayx-payouts.com instead of the brand's razorpayx.com, is unverified, and has 20 reports."
16+
msg_076,scam_signature,mute,mute,scam,scam,0.99,0.93,0,0,The sender's account exhibits a confirmed scam signature and domain mismatch in stage 1.,"Sender uses talabat-refund.com instead of the brand's talabat.com, is unverified, and has 61 reports."
17+
msg_110,reported_sender,mute,mute,scam,spam,0.99,0.90,1,0,The counterpart has been unanimously reported and the message demands a login code.,"The user has reported every past message from this sender and opened none, so this sender is suppressed."
18+
msg_073,reported_sender,mute,mute,scam,spam,0.95,0.90,1,0,The sender has been unanimously reported and the message is an unverified security threat.,"The user has reported every past message from this sender and opened none, so this sender is suppressed."
19+
msg_054,reported_sender,mute,mute,scam,spam,0.95,0.90,1,0,"An unverified account is demanding a payment fee for a loan inside a group, triggering stage 1 safety.","The user has reported every past message from this sender and opened none, so this sender is suppressed."
20+
msg_062,quiet_hours,notify,digest,event,event,0.88,0.82,1,0,A trusted group admin sent a time-sensitive update that should interrupt the user.,"Delivered during the user's quiet hours (21:30-06:30), so it waits for the digest."
21+
msg_077,quiet_hours,notify,digest,event,event,0.90,0.82,1,0,A trusted group admin sent a time-sensitive update that should interrupt the user.,"Delivered during the user's quiet hours (21:00-06:00), so it waits for the digest."

0 commit comments

Comments
 (0)