Tokens per second explained — throughput vs latency
Tokens per second is generation throughput after the model starts talking. Time-to-first-token (TTFT) is how long you stare at a spinner. Users feel TTFT; batch jobs feel tokens/sec × output length.
A model at 80 tok/s and 1.5s TTFT feels sluggish in chat even if it would win a throughput benchmark. A model at 40 tok/s and 150ms TTFT feels snappy for short answers. Measure both. The arena records them on replays so you can compare relative speed among roster models.
Popular frontier models by quality
Live OpenRouter pricing and Arena quality, cached about an hour. Not a static blog table.
| Model | Provider | Blend $ / 1M | Quality | Context |
|---|---|---|---|---|
| Claude Opus 5 | Anthropic | 15.00 | 93.4 | 1,000,000 |
| Claude Fable 5 | Anthropic | 30.00 | 91.1 | 1,000,000 |
| Qwen3.8 27B | Qwen | 1.71 | 89.0 | 1,000,000 |
| Gemini 3.7 Flash | 2.25 | 88.4 | 1,048,576 | |
| DeepSeek V4 Pro 0423 | DeepSeek | 1.14 | 88.2 | 1,024,000 |
Why output length dominates
Time ≈ TTFT + (output tokens / tok/s). A 20-token reply is all TTFT. A 2,000-token essay is all throughput. Coding agents that emit files care about tok/s; support macros care about TTFT.
Prefill vs decode
Long prompts make TTFT worse because the model must read the prompt (prefill) before decoding. That is why stuffing context hurts UX even when the window allows it.
Batching
Providers batch requests. Your tok/s at 2am may not match a load test at noon. Publish p95, not a single demo.
Speed vs quality sliders
Our leaderboard has a speed weight, but we do not have measured tok/s for every catalog row. For rows in the arena roster, believe the replay metrics more than the slider. For others, assume flash/mini is faster until proven otherwise.
Cost of waiting
Human time is on the bill. If a 20s reasoning model saves a 10-minute debugging session, it is “fast.” If it writes a three-sentence email, it is slow. Speed is relative to the task, like quality.
FAQ
What is a good tokens/sec?
- For chat, 40+ with low TTFT feels fine. For batch, higher is better until the GPU bill says stop.
Why is first token slow?
- Cold starts, queueing, and long prompts (prefill). Shorten the prompt; pick a faster host.
Does streaming change tok/s?
- It changes perception, not the underlying decode rate.
Where can I see numbers?
- The arena UI on this site, for models in the replay roster.
Related guides
Also try: LLM leaderboard, compare tool, live arena.