feat(stt): add Nemotron live-input streaming sessions

[TheAngryPit]

Context

Addresses #944. Nemotron has reusable incremental frontend and encoder state, but the generic realtime STT server cannot select it because the model lacks create_streaming_session().

This contribution reuses the work in #774 and #878 rather than adding another encoder or protocol.

Description

Adds an independent live-input session with copied mono PCM input, persistent frontend/Conformer/RNNT state, text deltas, cooperative joint-evaluation budgeting, idempotent close, tail flushing, cancellation and reset. Pending PCM is limited to 30 seconds with explicit overflow. The session retains no cumulative transcript. Existing whole-file APIs are unchanged.

Changes in the codebase

• Model factory exposing the established session protocol.
• session.py implementing session-local state and bounded ingestion.
• Regression tests for arbitrary partitions, short tails, partials before close, blank predictor behavior, silence, lifecycle and independent sessions.
• Usage and readiness limitations in docs/guides/nemotron-live-input.md.

Changes outside the codebase

None. No Nativ installation, local façade, OpenClaw configuration, credentials or running service changed. Nativ adoption is separate (#499 in Blaizzy/nativ).

Additional information

This is a draft, not a claim of production dictation readiness. The actual ASGI WebSocket handler now passes partial-before-commit, completion and second-turn tests with a tiny Nemotron model. A separate local multilingual checkpoint proof ran five paced Portuguese and five English synthesized utterances (20 ms / 16 kHz chunks), matching the existing streaming decoder after outer-whitespace normalization. First-partial p50/p95: 1.912/3.433 s; close-to-final p50/p95: 0.267/0.676 s, maximum 0.921 s. This is session latency, not WebSocket/Nativ end-to-end latency. The Portuguese fixture has the same recognition error in both decoding paths. Long-session memory residency and broader speech evidence remain open.

Tests run on Apple Silicon using an isolated Python and read-only existing MLX dependencies, without installing into Nativ. Missing WebRTC VAD and the repo-prescribed formatters were installed only in a disposable environment. Pytest plugin autoload and cache writes were disabled. The existing weights pytest was skipped because no model path was supplied to that suite; real-checkpoint evidence came from the separate paced harness. Black 26.3.1 and isort 5.13.2 passed on all changed Python files. Full upstream CI remains to be checked.

Focused command: pytest -q -p no:cacheprovider mlx_audio/stt/tests/test_nemotron_session.py mlx_audio/stt/tests/test_nemotron_asr.py (launched with the existing embedded Python and read-only pytest path). Final focused run: 21 passed, 1 weights test skipped. Includes fixed-seed real tiny-decoder parity, not only scripted logits, and special-token/symbol-cap transitions. git diff --check passed.

Implementation and tests were AI-assisted; a separate read-only review checked session transitions and identified additional parity coverage, which was added. No maintainer approval is implied.

Checklist

• Tests added/updated
• Documentation updated
• Issue referenced
• Actual ASGI WebSocket route test (tiny model)
• Local multilingual model and session latency evidence (limited samples)
• Full CI and formatting

Latest focused result: 22 passed, 1 skipped, including test_nemotron_realtime.py. The skipped test is the separately environment-gated weights pytest. No Nativ end-to-end claim.

[github-actions]

⚠️ These MLX-Audio test jobs fail only on this PR while the current main baseline passes them:

• core

Review the failing CI output and run the corresponding command from .github/workflows/tests.yml locally. If a failure is unrelated, leave a note for the maintainers.

[TheAngryPit]

The STT collection failure was introduced by the new actual-server test: CI installs .[all,dev], but the all extra omitted python-multipart while the server extra already declares it. Commit 2540344 adds that same existing dependency to all; the test remains enabled. Waiting for the fresh CI result. Commit verification remains a separate unresolved item.

[TheAngryPit]

Replaced the three unsigned commits with GitHub-signed commit c706c4b. Verified with git diff --exit-code that its tree is identical to 2540344; the original history remains preserved on the fork branch codex/nemotron-live-input-before-signing. The new CI run may still need maintainer approval. Downstream server integration is now proposed in Blaizzy/mlx-vlm#2167, with 36 local server tests passing and real Nemotron partial-before-commit proof over a loopback WebSocket through the unmodified Nativ Python overlay.

[TheAngryPit]

Follow-up on CI run 33971837501: STT and every other job passed except core. All 12 core failures report ffmpeg not found, from existing MP3/server tests. Signed commit 85688af adds brew install ffmpeg plus a version check to the core job only; no tests are disabled. The new Tests and Checks run 33973485596 and Docs run 33973485415 currently require maintainer approval. Please approve the pending runs when convenient.

[Lazarus-931]

hi @TheAngryPit, thanks for the pr!

I think this is solid, but maybe we can define a streaming.py for stt/ which holds a StreamingSession protocol of some sort, such that models can have this modular approach:

stt/models/nemotron_asr/session.py with NemotronStreamingSession

and

stt/models/voxtral_realtime/streaming.py with VoxtralStreamingSession

such that server.py can create and drive either session through the shared interface

what do u think?

[TheAngryPit]

Thanks @Lazarus-931 — agreed. Pushed 0bb3167 implementing your suggestion: mlx_audio/stt/streaming.py now defines a shared StreamingSession Protocol, and both model factories plus the server helper return that interface.

Kept it structural and minimal: input_sample_rate, feed, step, close and done. No mandatory inheritance or registration, no model-name branches, and no decoder behavior changes. Model-specific factory options remain where they already live; Nemotron reset/cancel are optional extensions rather than requirements imposed on Voxtral.

Added one parametrized contract test that creates both real session implementations through the server helper using tiny random models and deterministic token selection, then checks incremental output before close and completion after draining. Added docs/guides/streaming-stt.md for future model adoption, including threading, PCM ownership, completion and budget semantics.

Local validation: 66 passed, 3 checkpoint-gated skipped, and 18 subtests passed across the shared contract, Nemotron, Voxtral streaming and server tests. Commit is GitHub-signed. Happy to adjust the interface naming/details to your preferred convention.

[Lazarus-931]

see comment, looks good otherwise, thanks for contribution!

[Lazarus-931]

hi @TheAngryPit, CI failed

[TheAngryPit]

I've added the ffmpeg because of that in reality

[Lazarus-931]

I've added the ffmpeg because of that in reality

oh, can you try adding to see if CI is solved?

[TheAngryPit]

Restored FFmpeg in the core job in 4192224. My earlier check was too narrow: the new streaming tests use raw PCM, but adding python-multipart to the all extra also enables the existing server tests, which otherwise skip at collection. Those tests do need FFmpeg; run 34147461145 has 12 failures reporting it missing, while STT passes.

There is a separate remaining issue: the earlier run with FFmpeg installed failed test_audio_io.py::TestAudioIOEdgeCases::test_clipping because FFmpeg rejected the four-sample FLAC block (invalid block size: 4). Restoring the dependency addresses the missing-executable failures, not that codec failure. No tests have been disabled or weakened; the new CI result still needs verification.

[TheAngryPit]

We could maybe try and see if we can evolve the tests