Skip to content

Transferability study PR edits - #1

Open
plstcharles-saifh wants to merge 5 commits into
pierreolivierbonin:transferability-studyfrom
lawzero-org:transferability-edits
Open

Transferability study PR edits#1
plstcharles-saifh wants to merge 5 commits into
pierreolivierbonin:transferability-studyfrom
lawzero-org:transferability-edits

Conversation

@plstcharles-saifh

Copy link
Copy Markdown

Summary of improvements I did while doing my own review of lawzero-org#2:

  • Isolated transferability smoke/limited outputs so validation runs cannot contaminate full-sweep resume state;
  • Fixed CueFlip correctness issues around GSM8K operation-flip cache validation, HumanEval cue-uptake scoring, and analysis denominators;
  • Added CueFlip rendered_chat prompt mode, which builds neutral system/user messages, renders them locally with the HF chat template, and still calls /v1/completions (should be more aligned with Qwen3 usage expectations);
  • Added regression tests for the cache validator, HumanEval scoring, CueFlip analysis, prompt rendering, and prompt-format resume filtering.

Next step on my end: test the monster, and take a look at the eval results...

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant