What to Measure When Testing Qwen Locally

The short answer:
When you test a local Qwen setup, measure more than tokens per second. Record the exact configuration, whether the model loads cleanly, time to first token, steady generation speed, memory use, answer quality on your real tasks, failure modes, and the time it takes to get a usable result. A fast model that frequently needs retries is not a good local setup.

Who This Is For

This article is for developers who already run Qwen locally and want a practical measurement checklist before comparing models, quantizations, or hardware. It is also the standard this site uses when discussing local tests: configuration first, then behavior, then speed.

Why a Single Number Is Not Enough

Tokens per second is easy to quote and easy to misuse. It usually reflects decode speed after the prompt has been processed. It does not tell you:

• How long you wait before the first token appears

• Whether the model fits without painful offloading

• Whether the answer is correct for your task

• How often you must regenerate

• How the system behaves with longer context

Local testing should answer a practical question: Will this setup work on my machine, for my budget, on the work I actually do?

The Measurement Checklist

1. Configuration (always record this first)

Without configuration, a result is not comparable.

Record at least:

• Model family, size, and exact tag or filename

• Quantization (for example Q4_K_M, Q5_K_M, Q8_0)

• Runtime and version (Ollama, llama.cpp, vLLM, etc.)

• Hardware: GPU model and VRAM, or CPU and system RAM

• Context length used in the test

• Any special flags (offloading, thread count, flash attention, etc.)

If two runs differ in any of these, do not treat them as the same test.

2. Fitness: does it run cleanly?

Before quality or speed:

• Does the model load without error?

• Does it stay within available memory at the chosen context?

• Is generation stable for the response length you need?

A model that only runs with heavy swapping or constant crashes is not a successful local setup, regardless of benchmark screenshots.

3. Time to first token (TTFT)

TTFT is the delay between sending a request and seeing the first output token. It is largely driven by prompt processing (prefill) and model load state.

Measure it on:

• A warm model (already loaded)

• A prompt length similar to your real usage

A high tokens-per-second figure behind a multi-second pause still feels slow in interactive work.

4. Decode speed (tokens per second)

Measure steady generation after the first token. Note:

• Single-user / batch-size 1 unless you are testing a server under load

• Context length at measurement time

• Whether the number is peak, average, or sustained

Use TPS as one signal among several, not as the ranking definition.

5. Time to useful answer

This is often the most honest interactive metric:

How long until I have something I can actually use?

It includes:

• TTFT

• Generation time

• Any hidden reasoning tokens

• Time spent fixing or regenerating bad outputs

A slightly slower model with higher first-pass success can win on this metric.

6. Memory behavior

Track:

• Peak VRAM or RAM during load and generation

• Whether layers were offloaded to CPU

• What happens when context grows

Clean fit at the context you need is more valuable than a marginal larger model that constantly spills.

7. Task quality on your prompts

Generic leaderboard scores are orientation, not a substitute for your workload.

Build a small fixed set of prompts from real work, for example:

• Short coding or refactor tasks

• Extraction or summarization on realistic text

• Structured output (JSON or simple schemas)

• Multi-turn follow-ups if your workflow is conversational

Score outcomes that matter: correctness, validity, need for retries, and whether the answer is usable without heavy editing.

8. Failure modes

Document:

• Refusals or derailments on ordinary prompts

• Broken structured output

• Context overflow or quality drop at longer inputs

• Inconsistency across similar questions

Failures are part of the result. Hiding them produces rankings that do not survive contact with real use.

9. Stability across a session

One good answer is not enough. Check:

• Behavior after many turns

• Performance as the KV cache grows

• Thermal throttling or slowdown over time

Interactive tools live in sessions, not in single-shot benchmarks.

A Simple Test Template

For each configuration you care about, fill in:

1. Config — model, quant, runtime, hardware, context

2. Load — success / failure, memory notes

3. TTFT — warm run, stated prompt length

4. TPS — decode rate, stated conditions

5. Task pack — N prompts, pass / fail / notes

6. Time to useful answer — median or typical

7. Failures — what broke

8. Verdict — keep, drop, or retest with one changed variable

Change one variable at a time when comparing setups.

What Not to Treat as Proof

• Vendor slides without your configuration

• Tokens per second from a short cold prompt presented as daily experience

• Tiny sample sizes turned into precise rankings

• Mixed hardware or mixed quantizations in one “winner” claim

• First-person performance stories that cannot be reproduced from published settings

This site’s standard is transparency over theater.

How Much Testing Is Enough?

You do not need a research lab.

• Smoke test: load + one short prompt + one longer prompt

• Working comparison: 10–30 real prompts, fixed config, recorded outcomes

• Deeper evaluation: more prompts, multiple context lengths, repeated runs

Fifty well-chosen prompts from your own work often beat a thousand generic benchmark items for deciding whether to keep a local setup.

Practical Conclusion

Good local Qwen testing is configuration-aware, task-aware, and honest about failure. Measure fitness, latency, speed, memory, quality, and time to a usable result. Rank setups by whether they work cleanly for your machine and your work—not by the most flattering single number.

When the measurements are clear, the choice usually becomes clear too.

Make it run. Make it useful.