Skip to content

Commit 5368996

Browse files
committed
purpose: fix vacuous matching, name the sections, guard the doc hand-copies
Audit of this round's purpose-registry work, on the three axes asked for. EFFECTIVENESS — three checks were VACUOUS. Naive substring matching satisfied `ct` inside "a-CT-ion", `fit` inside "bene-FIT", `is a` inside "this IS A summary". On an adversarial deck carrying only those three phrases, product_pitch reported NOTHING missing at all. `_names()` now anchors at a word START for Latin (a start anchor, not a full boundary, because `feasib` and `demo` are deliberate prefixes), plain substring for CJK (no word boundaries), substring for a term with no alphanumerics (`%`). Non-indicative terms removed. before tumour board missing=['presentation','outcome'] job talk missing=['track record','research plan'] product pitch missing=[] after tumour board missing=['presentation','investigations','outcome'] job talk missing=['track record','research plan','fit'] product pitch missing=['positioning'] …and five sections did not list their OWN NAME as a term, so `job_talk/track record` reported MISSING on a slide reading "My track record". GENERALISATION — the two reference files are hand-copies of the registry, and a hand-copy drifts. Writing one out got 1 of 12 wrong within minutes: the runbook said "lab meeting" (a trigger word) where every gate message prints `research_meeting`, so an agent grepping the doc for the name in its own error found nothing. `check_purpose --selftest` now checks both files for every genre name, every section name, the delivery-mode note, and the stated count. Each arm is mutation-tested in the suite — including a duplicate of an already-documented genre, which only the count can catch, proving that arm is not dead code. NON-CLAUDE AGENTS — `codex-runtime.md` still said "Four genres" and never mentioned `purposes.py`; a capability absent from that runbook is one the next Codex run rediscovers by failing a gate, or never uses. It now states 12 of 13 with every section, why the thirteenth (webinar) has no list, and how matching works. The medium-vs-genre message was also duplicating a clause on that path. Verified: test_purpose_sections 176 passed · test_schema_reach 27 passed · gate_parity, codex_delivery_gate, codex_scaffold_complete, step1_artifacts, smoke_deckkit ok · inventory/tests-wired/parity/reference-code/repo-integrity/ secrets clean · lossless 2719/2719. Synced to ~/.agents/skills/slide-maker and re-run there.
1 parent db06dd0 commit 5368996

4 files changed

Lines changed: 192 additions & 16 deletions

File tree

skills/slide-maker/references/codex-runtime.md

Lines changed: 13 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -212,11 +212,19 @@ be reconstructed post-hoc at the delivery gate.
212212
device that advisory is the expected result — say so in the plan rather than adding a key you
213213
do not want.
214214

215-
**The deck's GENRE may declare required content — `check_purpose.py`, on both gate paths.** Four
216-
genres carry sections their audience is asked to judge against, so a deck missing one is not a lean
217-
deck: **grant** (aims · feasibility · risk) · **progress/guidance committee** (progress · plan ·
218-
**ask**) · **journal club** (attribution · critique) · **clinical case** (presentation ·
219-
investigations · management · outcome). 🔴 **This binds from the RECORD, and the Codex schema has no
215+
**The deck's GENRE may declare required content — `check_purpose.py`, on both gate paths.** TWELVE of the thirteen
216+
genres in `design-by-purpose.md` carry sections their audience is asked to judge against, so a deck
217+
missing one is not a lean deck: **grant** (aims · feasibility · risk) · **committee** (progress ·
218+
plan · **ask**) · **journal club** (attribution · critique) · **clinical case** (presentation ·
219+
investigations · management · outcome) · **defense** (contributions · limitations · future work) ·
220+
**conference talk** (contribution · evidence · limitations) · **job talk** (track record · research
221+
plan · fit) · **exec readout** (the ask · the number · the risk) · **research meeting** (lab/group meeting: what
222+
changed · open questions) · **work status** (outcome · the ask) · **product pitch** (positioning · benefits) ·
223+
**teaching** (objectives · worked example · recap). 🔴 The thirteenth, **webinar**, has no list on
224+
purpose — it is a delivery MODE, not a genre, and `check_purpose` says so and asks you for the
225+
GENRE rather than silently checking nothing. Matching is word-START boundary for Latin (naive
226+
substring made three of these VACUOUS: `ct` inside a-CT-ion, `fit` inside bene-FIT) and plain
227+
substring for CJK. 🔴 **This binds from the RECORD, and the Codex schema has no
220228
`purpose` key** — so it reads `interview.record` (the user's own answers), the `content.
221229
audience_brief` `who` + `decisions`, and any `design.purpose` you add. On a SUPERVISED Codex run
222230
`delegated_picks` records no purpose axis at all, so `interview.record` is frequently the only place

skills/slide-maker/scripts/check_purpose.py

Lines changed: 73 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -36,6 +36,7 @@
3636
import argparse
3737
import json
3838
import os
39+
import re
3940
import sys
4041

4142
sys.path.insert(0, os.path.dirname(os.path.abspath(__file__)))
@@ -114,6 +115,31 @@ def recorded_purpose(gates):
114115
return " ".join(b for b in bits if b)
115116

116117

118+
def _names(term, blob):
119+
"""Does `blob` NAME `term`? Word-START boundary for Latin, plain substring for CJK.
120+
121+
🔴 Naive substring matching made three of these checks VACUOUS, which is worse than having no
122+
check at all — it passes every deck and teaches the reader that the gate is noise. Measured on
123+
a deck containing only "Our action plan for the project", "The benefit of this approach" and
124+
"This is a summary":
125+
126+
clinical_case/investigations satisfied by "ct" inside a-CT-ion / proje-CT
127+
job_talk/fit satisfied by "fit" inside bene-FIT
128+
product_pitch reported NOTHING missing at all
129+
130+
The boundary is on the START only, never the end, because several terms are deliberate
131+
PREFIXES — "feasib" must match feasibility and feasible, "demo" must match demonstration. A
132+
trailing boundary would silently break those, which is the same class of quiet failure.
133+
A term with no letters or digits at all (`%`) has no word boundary to anchor to and falls back
134+
to substring.
135+
"""
136+
if any(ord(c) > 0x2E80 for c in term):
137+
return term in blob # CJK has no word boundaries
138+
if not any(c.isalnum() for c in term):
139+
return term in blob # e.g. "%" — nothing to anchor
140+
return re.search(r"(?<![a-z0-9])" + re.escape(term), blob) is not None
141+
142+
117143
def check(pptx, purpose_text, *, extra_terms=None, waive=None):
118144
"""(problems, facts). problems = [(code, message), ...]. Raises when nothing binds."""
119145
import purposes
@@ -147,7 +173,7 @@ def check(pptx, purpose_text, *, extra_terms=None, waive=None):
147173
return [], facts
148174
for label, keys in p.required_sections:
149175
keys = tuple(keys) + tuple(more.get(label.lower(), ()))
150-
if not any(str(k).lower() in blob for k in keys):
176+
if not any(_names(str(k).lower(), blob) for k in keys):
151177
facts["missing"].append(label)
152178
problems.append((
153179
"MISSING SECTION",
@@ -196,6 +222,52 @@ def _selftest():
196222
bad.append("the shared .deck-gates.json shape was not read")
197223
if purposes.match(recorded_purpose(codex)) is None:
198224
bad.append("the Codex evidence shape was not read")
225+
# 🔴 THESE DOCS ARE HAND-COPIES OF THIS REGISTRY, AND HAND-COPIES DRIFT.
226+
# Both files list the genres and their sections in prose, because that is where each runtime's
227+
# agent LEARNS the genres exist — `design-by-purpose.md` on the shared path, `codex-runtime.md`
228+
# on the Codex one. A genre absent from the doc its runtime reads is one that run never uses.
229+
# Measured: writing that list out by hand got 1 of 12 wrong within minutes (it said "lab
230+
# meeting", the colloquial trigger, where every gate message says `research_meeting`), so an
231+
# agent grepping the doc for the name in its own error found nothing. Not a style check: it is
232+
# the doc-vs-code drift that makes a capability invisible to whichever runtime reads that file.
233+
COUNTS = {11: "eleven", 12: "twelve", 13: "thirteen", 14: "fourteen", 15: "fifteen"}
234+
REF = os.path.join(os.path.dirname(os.path.abspath(__file__)), "..", "references")
235+
# (file, anchor, span) — an anchor bounds the check to the paragraph that makes the claim;
236+
# None means the whole file is the genre reference.
237+
DOCS = (("codex-runtime.md", "The deck's GENRE may declare required content", 2400),
238+
("design-by-purpose.md", None, None))
239+
total = len(purposes.PURPOSES) + len(purposes.NOT_A_GENRE)
240+
for fname, anchor, span in DOCS:
241+
try:
242+
doc = open(os.path.join(REF, fname), encoding="utf-8").read()
243+
chunk = doc if anchor is None else doc[doc.index(anchor):doc.index(anchor) + span]
244+
except (OSError, ValueError) as exc:
245+
bad.append("could not read the genre reference in references/%s (%s) — that file is "
246+
"where one runtime learns these genres exist" % (fname, exc))
247+
continue
248+
para = " ".join(chunk.lower().split())
249+
for p_ in purposes.PURPOSES:
250+
if p_.name.replace("_", " ") not in para:
251+
bad.append("references/%s never names %r — every gate message prints that exact "
252+
"name, so an agent grepping its own error finds nothing" % (fname, p_.name))
253+
for lbl, _terms in p_.required_sections:
254+
if lbl.lower() not in para:
255+
bad.append("references/%s omits section %r of %r — a doc that understates a "
256+
"genre's content reads as complete, which is worse than omitting it"
257+
% (fname, lbl, p_.name))
258+
for _key in purposes.NOT_A_GENRE: # the key is a TUPLE of trigger words
259+
medium = _key[0] if isinstance(_key, tuple) else _key
260+
if medium not in para:
261+
bad.append("references/%s never mentions %r as a DELIVERY MODE — that path would "
262+
"look like it silently checks nothing there" % (fname, medium))
263+
word = COUNTS.get(total)
264+
if word is None:
265+
bad.append("the registry now holds %d entries, outside COUNTS in this selftest — extend "
266+
"the map so each doc's stated count keeps being checked" % total)
267+
elif word not in para:
268+
bad.append("references/%s does not say %r where it states its coverage, but the registry "
269+
"now holds %d entries — the stated count has gone stale" % (fname, word, total))
270+
199271
for b in bad:
200272
print(" ✗", b)
201273
print("[purpose] selftest %s" % ("FAILED" if bad else "ok"))

skills/slide-maker/scripts/purposes.py

Lines changed: 19 additions & 10 deletions
Original file line numberDiff line numberDiff line change
@@ -85,7 +85,8 @@ class Purpose:
8585
"published", "achieved", "进展", "已完成", "去年")),
8686
("plan", ("plan", "next", "remaining", "timeline", "roadmap", "schedule", "ahead",
8787
"计划", "下一步", "时间线", "剩余")),
88-
("ask", ("advice", "guidance", "decision", "question", "input", "feedback",
88+
# no bare "input": "input data" is not an ask, and the rest already cover it
89+
("ask", ("the ask", "advice", "guidance", "decision", "question", "feedback",
8990
"recommend", "建议", "决策", "请教", "问题")),
9091
),
9192
fidelity="What is NOT done yet must read as not done. A committee's value is advice on open "
@@ -101,8 +102,8 @@ class Purpose:
101102
binds_on=("journal club", "paper presentation", "reading group", "present a paper",
102103
"discuss the paper", "文献汇报", "文献阅读", "组会读论文", "论文分享", "读书会"),
103104
required_sections=(
104-
("attribution", ("et al", "authors", "published in", "doi", "arxiv", "reference",
105-
"citation", "source", "作者", "发表于", "文献")),
105+
("attribution", ("attribution", "et al", "authors", "published in", "doi", "arxiv",
106+
"reference", "citation", "source", "作者", "发表于", "文献")),
106107
("critique", ("limitation", "limitations", "weakness", "critique", "caveat",
107108
"my take", "assessment", "concerns", "局限", "不足", "评价", "批评")),
108109
),
@@ -161,7 +162,7 @@ class Purpose:
161162
required_sections=(
162163
("contribution", ("contribution", "we propose", "our method", "this work", "novelty",
163164
"我们提出", "本文", "贡献")),
164-
("evidence", ("result", "results", "evaluation", "experiment", "we evaluate",
165+
("evidence", ("evidence", "result", "results", "evaluation", "experiment", "we evaluate",
165166
"benchmark", "结果", "实验", "评估")),
166167
("limitations", ("limitation", "limitations", "caveat", "does not", "fails when",
167168
"局限", "不足")),
@@ -177,7 +178,7 @@ class Purpose:
177178
binds_on=("job talk", "faculty interview", "faculty position", "tenure track",
178179
"interview seminar", "求职报告", "教职面试"),
179180
required_sections=(
180-
("track record", ("published", "my work", "i have", "prior work", "to date",
181+
("track record", ("track record", "published", "my work", "i have", "prior work", "to date",
181182
"已发表", "我的工作", "过往")),
182183
("research plan", ("research plan", "next five years", "future programme", "my lab",
183184
"will pursue", "agenda", "研究计划", "未来五年")),
@@ -197,7 +198,7 @@ class Purpose:
197198
required_sections=(
198199
("the ask", ("we recommend", "the ask", "decision", "approve", "we propose", "request",
199200
"建议", "决策", "请批准")),
200-
("the number", ("revenue", "cost", "roi", "budget", "headcount", "%", "growth",
201+
("the number", ("the number", "revenue", "cost", "roi", "budget", "headcount", "%", "growth",
201202
"impact", "收入", "成本", "预算", "指标")),
202203
("the risk", ("risk", "risks", "mitigation", "downside", "what could go wrong",
203204
"assumption", "风险", "假设", "下行")),
@@ -247,8 +248,10 @@ class Purpose:
247248
binds_on=("product pitch", "sales deck", "product demo", "customer presentation",
248249
"product launch", "go-to-market", "产品介绍", "产品发布", "销售材料", "客户演示"),
249250
required_sections=(
251+
# 🔴 no "is a": it is not a positioning phrase, it is English. It matched "This is a
252+
# summary" and made this whole section vacuous.
250253
("positioning", ("for teams who", "for people who", "positioning", "we help",
251-
"is a", "designed for", "定位", "面向", "帮助"))
254+
"designed for", "built for", "helps you", "定位", "面向", "帮助"))
252255
,
253256
("benefits", ("benefit", "benefits", "you can", "saves", "so that", "value",
254257
"outcome for you", "收益", "价值", "你可以")),
@@ -268,7 +271,9 @@ class Purpose:
268271
required_sections=(
269272
("objectives", ("objective", "objectives", "learning goal", "by the end", "you will "
270273
"be able", "what you will learn", "目标", "学习目标", "本节")),
271-
("worked example", ("example", "worked example", "let us", "walk through", "case",
274+
# bare "case" is not a teaching signal ("in case", "use case", "showcase"); the
275+
# distinctive words already carry it
276+
("worked example", ("example", "worked example", "let us", "walk through", "case study",
272277
"demo", "例子", "示例", "演练")),
273278
("recap", ("recap", "summary", "to summarise", "to summarize", "key points",
274279
"takeaway", "小结", "总结", "要点")),
@@ -279,6 +284,10 @@ class Purpose:
279284
),
280285
)
281286

287+
# 🔴 EVERY section's own NAME is one of its terms. Sounds obvious; five sections shipped without
288+
# it, and `job_talk/track record` reported MISSING on a deck whose slide said "My track record"
289+
# — the section's own words. A term list assembled from synonyms forgets the word it is a synonym
290+
# OF. `tests/test_purpose_sections.py` asserts this for every section so a new one cannot repeat it.
282291
BY_NAME = {p.name: p for p in PURPOSES}
283292

284293
# 🔴 DELIVERY MODES, NOT GENRES — deliberately absent from PURPOSES, with the reason.
@@ -292,8 +301,8 @@ class Purpose:
292301
NOT_A_GENRE = {
293302
("webinar", "online presentation", "zoom talk", "teams talk", "virtual talk", "livestream",
294303
"网络研讨会", "线上分享", "直播"):
295-
"a webinar is a delivery MODE, not a genre — the same session can be a lecture, a product "
296-
"pitch or an exec readout, and those need different content. Record the GENRE as the "
304+
"the same session can be a lecture, a product pitch or an exec readout, and those need "
305+
"different content. Record the GENRE as the "
297306
"purpose (the medium is already handled by the type floors and the safe-area rules), e.g. "
298307
"'a teaching webinar' or 'an investor webinar'.",
299308
}

0 commit comments

Comments
 (0)