Benchmark of token streaming policies for LLM serving, measuring user-perceived interactivity, chunk readability, and infrastructure overhead across workload types.
Author: Joao Felipe De Souza Year: 2026
Standard serving benchmarks measure TTFT and TPOT from the server perspective. This benchmark measures what the user actually perceives during streaming:
- TTI: time to first interactive chunk
- TTRC: time to first readable (word-boundary) chunk
- SPIS: streaming perceived interactivity score (cadence-focused)
- SPIS-R: readable-aware SPIS (adds midword chunk penalty)
The full pipeline is modeled: token generation, SSE framing, network transport with jitter, and client-side render frame coalescing at 60 FPS.
timer_50ms achieves SPIS of 99.5-99.9 and SPIS-R of 97-99 across natural and chat workloads, outperforming both semantic and token-count policies.
When midword chunk penalty is included, timer_100ms outperforms timer_50ms in code workloads because longer intervals increase word-boundary alignment. In subword_heavy workloads, all policies degrade to SPIS-R of 73-77.
word_boundary_50ms achieves midword chunk fraction of 0.19-0.21 versus 0.38-0.40 for timer policies. It is optimal when chunk readability matters more than first-byte latency.
Increasing batch size from 1 to 32 reduces SPIS by less than 1 point. Flush policy differences of 20-40 SPIS points dominate completely. Streaming UX is controlled by client-side policy, not server batching.
In code and subword_heavy workloads, midword chunk fraction rises to 0.60-0.80 regardless of flush policy. The readability bottleneck is at the tokenization layer, not the streaming policy layer.
| Policy | Mode | Description |
|---|---|---|
| every_token | token | Flush on every token |
| tokens_4/8/16 | token | Flush every N tokens |
| timer_50ms | timer | Flush every 50ms |
| timer_100ms | timer | Flush every 100ms |
| hybrid_4_or_100 | hybrid | 4 tokens or 100ms, whichever first |
| word_boundary_50ms | semantic | Flush at word boundaries, 50ms fallback |
| punctuation_100ms | semantic | Flush at punctuation, 100ms fallback |
| render_frame_aware | render | Flush at 60 FPS boundaries |
| end_of_response | end | Wait for complete response |
- chat_short / chat_long: conversational text
- natural_text: well-formed prose, high word-boundary probability
- subword_heavy: text with many subword tokens (word-boundary prob 0.20)
- code_heavy / codegen: code generation (punctuation-dense, mid-word-prone)
- mixed_outputs: varied output lengths
- sharegpt_realistic: realistic chat distribution
- Universal default: timer_50ms
- Code workloads: timer_100ms (better word alignment)
- Readable-first applications: word_boundary_50ms
- Lowest TTI: render_frame_aware
- Never use: tokens_16 or end_of_response for interactive applications
From the project directory:
cd ~/dev/token-streaming-latency-bench
source venv/bin/activate
python -u run.py
results/summary_v11.csv
plots/
token-streaming-latency-bench/
|-- src/
| |-- __init__.py
| |-- config.py
| |-- workload.py
| |-- streaming.py
| |-- simulator.py
| |-- bench.py
| |-- analysis.py
|-- results/
|-- plots/
|-- run.py
|-- SUMMARY.txt
|-- DESIGN.md
|-- LICENSE
|-- README.md
|-- requirements.txt
|-- .gitignore
Streaming policy is a three-way trade-off between time-to-first-visible-text, chunk cadence fluidity, and chunk readability. Timer_50ms is the recommended universal default. Timer_100ms is preferred for code workloads. Word_boundary_50ms is optimal when readability matters more than latency. The SPIS-R metric introduced here provides a more complete picture of perceived streaming quality than cadence-only metrics.
MIT License. See LICENSE.