Skip to content

Add prefill method support without generation - #183

Open
craftingmod wants to merge 9 commits into
JamePeng:mainfrom
craftingmod:prefill-feature
Open

craftingmod wants to merge 9 commits into
JamePeng:mainfrom
craftingmod:prefill-feature

Conversation

@craftingmod

@craftingmod craftingmod commented Sep 26, 2026 •

Copy link
Copy Markdown

Summary

  • Add create_chat_prefill method which could be used to prompt evaluation.
    • Expose a prefill-only chat API that evaluates and returns the final next-token without generation.
  • Add chat_template_kwargs support in create_chat_completion for overriding chat_template_kwargs each completion.
    • can override extra_template_arguments value of each completion.

Maybe useful when implementing probability classification.

AI Disclose

I've used GPT 6 Luna to implementing features, but I manually reviewed and understood how to works.

@craftingmod

craftingmod commented Sep 26, 2026 •

Copy link
Copy Markdown
Author
  • Why i made this PR: I want trying something like system-one in comfyui 😕
  • Code quality might be bad so feel free to edit or close PR 😉

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant