Skip to content

Add Whisper Chat Template Support - #981

Merged
Sayan Shaw (sayanshaw24) merged 1 commit into
mainfrom
sayanshaw/whisper-chat
Jul 25, 2025
Merged

Add Whisper Chat Template Support#981
Sayan Shaw (sayanshaw24) merged 1 commit into
mainfrom
sayanshaw/whisper-chat

Conversation

@sayanshaw24

@sayanshaw24 Sayan Shaw (sayanshaw24) commented Jul 24, 2025

Copy link
Copy Markdown
Collaborator

Updates

Neither OpenAI nor HuggingFace has explicit Whisper chat template functionality.

  1. OpenAI doesn't expose Whisper via a chat interface like ChatCompletion. Instead, their Whisper API uses raw audio uploads. For finetuning or embedding Whisper in pipelines, they rely on pre-tokenized sequences. They don’t use Jinja2 or chat templates for Whisper at all — it is purely sequence input with prepended tokens like <|startoftranscript|> manually inserted.
  2. In HuggingFace transformers, for Whisper, you are expected to pass in the template manually or have it defined in tokenizer_config.json.

However, the expected logic (same for HF and OAI) should emulate concatenating message['content'], with no roles, separators, etc. In this PR, we thereby automatically handle this in ORT Extensions as well.

Validation

  • Native C++ testing against baseline output from HuggingFace Whisper model.
  • Python unit testing against HuggingFace transformers

@sayanshaw24
Sayan Shaw (sayanshaw24) marked this pull request as ready for review July 24, 2025 22:34
@sayanshaw24
Sayan Shaw (sayanshaw24) requested a review from a team as a code owner July 24, 2025 22:34
@sayanshaw24
Sayan Shaw (sayanshaw24) merged commit 41b89b1 into main Jul 25, 2025
46 checks passed
@sayanshaw24
Sayan Shaw (sayanshaw24) deleted the sayanshaw/whisper-chat branch July 25, 2025 02:00
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants