Preference Harness

Turn “it feels off” into a preference an agent can test.

Preference Harness is a small suite of Agent Skills plus a deterministic CLI. It helps an agent compare controlled alternatives, record a human decision without rewriting it, and check whether the resulting rule transfers to a new artifact.

It is not a universal design critic. It does not produce a taste score, choose for the decision-maker, or let preference override correctness.

Why this is a suite

The same visual treatment can work on a portfolio and fail in a dashboard. Preference Harness keeps a common evidence contract while routing work to a scenario-specific module.

Skill	Job
preference-calibrate	Turn a vague rejection into a controlled comparison and a scoped rule
preference-marketing-web	Apply a matching rule to public and editorial web pages
preference-product-ui	Apply a matching rule to task-oriented product interfaces
preference-verify	Test an exact revision without merging preference, task fitness, and correctness
prose-verdict	Return PUBLISH or HOLD for complete public prose without rewriting it

The bundled router refuses unsupported and ambiguous scenario/axis pairs. There is no generic fallback.

Install

List the available skills:

bunx skills add Arakiss/preference-harness --list

Install the complete suite for Codex and Claude Code:

bunx skills add Arakiss/preference-harness \
  --skill '*' \
  --agent codex \
  --agent claude-code

Version 0.1 is installed as a suite. The scenario modules call the calibration and verification modules, and the portable CLI ships with preference-calibrate.

Review every skill before installing it. Skills are instructions that an agent will follow.

The smallest useful workflow

1. Name one decision and one unresolved axis.
2. Keep the content, task, environment, and objective constraints fixed.
3. Present two to six neutrally labeled alternatives.
4. Record choose, reject_all, or abstain literally.
5. Turn the signal into a rule with an explicit scope and exclusions.
6. Predict the result on a held-out artifact before showing it.
7. Assess correctness, task fitness, and preference in separate lanes.

The CLI stores that sequence in a versioned JSON dossier. Version 0.1 deliberately keeps one active scoped rule per dossier. A corrected rule appends a replacement with supersedes; unrelated preferences compose as separate dossiers instead of a mutable universal profile.

node skills/preference-calibrate/scripts/preference-harness.mjs route \
  --scenario marketing-web \
  --axis composition \
  --json

{
  "axis": "composition",
  "module": "preference-marketing-web",
  "registry_version": 1,
  "scenario": "marketing-web"
}

An unsupported route stops instead of borrowing a rule from the wrong kind of work:

node skills/preference-calibrate/scripts/preference-harness.mjs route \
  --scenario native-mobile \
  --axis motion \
  --json

{
  "error": "NO_EXACT_ROUTE",
  "message": "No exact route for scenario \"native-mobile\" and axis \"motion\". Add a tested module; do not fall back to generic taste."
}

What a dossier preserves

• the exact artifact revision and decision context;
• evidence kind, origin, intended claim use, relative path, and SHA-256;
• the factors changed in each alternative;
• controlled versus confounded comparisons;
• declared factor changes, presentation order, and literal responses;
• append-only response and assessment corrections;
• append-only rule-state transitions;
• a hash-linked audit receipt for every dossier mutation;
• scoped preference rules and exclusions;
• a held-out prediction recorded before the response;
• separate correctness, task-fit, and preference assessments.

The schema rejects unknown fields, including an overallScore.

“Controlled” refers to the declared factor manifest. The CLI does not compare pixels or DOM trees to prove that no undeclared change slipped into a candidate. Likewise, it records presentation order but does not certify the randomization method.

See the dossier schema and the architecture note.

Supported routes in 0.1

Scenario	Axes
marketing-web	composition, density, typography, color, texture, interaction
product-ui	information-density, hierarchy, navigation, control-prominence, status-visibility, error-recovery
technical-writing	opening, structure, density, claim-scope, voice

Native mobile, developer tools, data visualization, and motion are deliberately unsupported in 0.1. They need different tasks and evidence. Refusal is safer than pretending one design checklist fits everything.

Evidence and limits

The repository includes deterministic contract tests and synthetic fixtures. They verify routing, dossier semantics, evidence isolation, correction chains, held-out ordering, portable bundles, and refusal behavior. Synthetic oracles do not count as human preference evidence.

Live pilots and their artifact origins are listed in the evaluation protocol. A result is reported as a raw observation, not a percentage or a universal rule.

Run the complete synthetic example:

bun run build
bash examples/minimal/run.sh

It exercises the contract without pretending a scripted oracle is human preference evidence, so its final public verdict is intentionally NO VERDICT.

The Open Source redesign case study does the same with a real website decision reconstructed after the event. The agent selected one treatment after the human delegated the choice. The dossier records that response as model review, creates no preference rule, and returns NO VERDICT. Its run, validation result, and public reports are committed and reproducible.

Development

Requirements:

• Bun
• Node.js 22 or newer for the bundled CLI; CI covers the maintained 22 and 24 release lines
• Python 3 for the dependency-free skill validator

bun install
bun run build
bun run check

bun run build refreshes the two portable CLI bundles. bun run check type-checks the source, runs the test suite, verifies that both committed bundles match the source, validates every skill, and executes the bundled CLI and scanner adapter directly with Node.js.

Exit codes

Code	Meaning
0	Success
1	I/O or internal failure
2	Invalid command usage
3	Unsupported or ambiguous route, or invalid registry
4	Invalid dossier
5	Valid dossier that is not release-ready
6	Unsupported schema version

Contributing

New scenario modules need their own task model, objective gates, native evidence format, refusal conditions, and at least one held-out evaluation. A list of generic design principles is not a scenario module.

See CONTRIBUTING.md.

License

MIT