Tokens per Second Is Not the Only Metric That Matters

The short answer:
Tokens per second (TPS) measures how fast a model generates text once it has started. It is useful, but incomplete. Time to first token, end-to-end time to a useful answer, memory fit, output quality, and failure rate often matter more for real work. A setup that looks fast on a cold, short-prompt benchmark can feel slow or unreliable in daily use.

Who This Is For

This article is for developers who see large tokens-per-second numbers in local model discussions and want a clearer way to judge whether a Qwen setup will actually feel good. It builds on the evaluation standards and size-trade-off posts earlier in this series.

What Tokens per Second Actually Measures

When people say “tokens per second,” they usually mean decode speed: how quickly new tokens are produced after the model has processed the prompt.

That number is dominated by memory bandwidth for single-user local inference. Larger models and higher-precision weights generally mean lower TPS on the same hardware. Short, shallow-context benchmarks tend to produce the most flattering numbers.

TPS does not fully capture:

• How long you wait before the first token appears

• How long context affects both prefill and generation

• Whether the answer is correct or usable

• How often you have to retry

• Whether the model even fits without painful offloading

The Metrics That Sit Beside TPS

Time to first token (TTFT)

TTFT is the delay between sending a request and seeing the first output token. It is largely driven by prefill (processing the prompt and building the KV cache), plus model load time if the model was not already resident.

Long system prompts, retrieved documents, and chat history all increase TTFT. A model with excellent decode TPS can still feel sluggish if every request starts with a multi-second pause.

Time to useful answer

This is the practical metric many workflows care about:

How long until I have something I can use?

It combines:

• TTFT

• Generation time for the needed output length

• Any “thinking” or intermediate tokens that never appear in the final answer

• Time spent fixing or regenerating bad outputs

A faster model that frequently needs a second try can lose to a slightly slower model that gets the job done on the first pass.

Memory fit and stability

A configuration that only runs with heavy CPU offload, constant swapping, or aggressive context limits is not “fast” in any useful sense. Clean fit at the context length you actually use is a prerequisite metric.

Quality and reliability on your tasks

Correctness, structural validity (valid JSON, working code), instruction following, and consistency across similar prompts are not optional extras. They determine whether speed is spent producing value or producing retries.

Behavior under realistic context

Many published TPS figures use short prompts and nearly empty caches. Real sessions accumulate history. Prefill cost grows with prompt length; decode can slow as the KV cache grows. Measure at the context lengths you actually use.

Why Single-Number Rankings Mislead

Consider two setups on the same machine:

• Setup A: Higher TPS on a short synthetic prompt, but weaker multi-step reasoning and frequent need to re-prompt.

• Setup B: Lower TPS, but higher first-pass success on your real coding or extraction tasks.

For interactive work, Setup B often wins on time-to-useful-answer and on frustration. For overnight batch jobs, raw throughput may matter more. The right metric depends on the job.

Other common distortions:

• Cold vs warm — Including model load time makes one run look worse than steady-state use.

• Batch size 1 vs saturated throughput — Multi-user serving numbers are not the same as single-user interactive speed.

• Peak vs sustained — Thermal throttling and memory pressure can erase flattering first-run numbers.

• Thinking models — Models that generate long internal reasoning before the visible answer can post decent TPS while feeling slow to the user.

A Practical Measurement Habit

When you evaluate a local Qwen configuration, record at least:

1. Configuration — model tag, quantization, runtime, hardware, context limit

2. TTFT — on a warm model, with a prompt length you actually use

3. Decode TPS — steady generation after the first token

4. End-to-end time — for a representative successful answer

5. Success rate — on a small fixed set of your own prompts

6. Memory notes — whether it fit cleanly, any offload, any instability

You do not need a research harness. A spreadsheet and a fixed prompt pack are enough to stop yourself from optimizing the wrong number.

What “Fast Enough” Usually Means

Rough orientation for single-user interactive use:

• Below ~10 TPS — often feels slow for chat-length answers

• ~15–30 TPS — comfortable for many people reading along

• Above ~40–50 TPS — faster than most people read; further gains matter less for pure reading, more for long generations and agent loops

These are not hard rules. A 12 TPS model that rarely needs retries can beat a 60 TPS model that frequently produces unusable output.

For batch and offline jobs, lower TPS is often acceptable if the machine can run unattended and the quality is sufficient.

How This Changes Decision-Making

Instead of asking only “Which Qwen size has the highest tokens per second on my GPU?” ask:

• Does it fit at the context I need?

• How long until the first token on a realistic prompt?

• How often is the first answer good enough?

• What is the time to a usable result, including fixes?

• Is the experience stable across a normal work session?

TPS remains a useful signal for hardware and quantization choices. It should not be the sole ranking criterion.

Connecting to the Series

The evaluation standards article explained what we refuse to claim. The 7B vs 14B vs 32B comparison applied size trade-offs without treating speed as the whole story. This post makes the metric problem explicit.

Later reality-check posts will keep separating configuration, task success, and resource behavior. The goal is not the flashiest decode number. The goal is a local setup that is fast enough, correct enough, and stable enough for the work you actually do.

Make it run. Make it useful.