TTFT vs TPS vs ITL: How to Read Local LLM Benchmark Results

Quick Answer

TTFT, TPS and ITL answer three different questions. TTFT tells you how long the user waits before the first output appears. ITL tells you how smoothly tokens arrive after generation starts. TPS tells you how much aggregate token throughput the server produces. A useful local LLM benchmark needs all three, because optimizing one can make another worse.

Key Takeaways

· Low TTFT matters most for interactive chat and coding.

· Low ITL produces smoother streamed output after the first token.

· High aggregate TPS is important for multi-user serving and batch generation.

· Higher concurrency can raise TPS while also increasing TTFT and ITL.

· Always compare metrics at the same input length, output length, model and concurrency.

TTFT: Time to First Token

Time to first token measures the delay between sending a request and receiving the first streamed output. It includes prompt processing, scheduling and the work required before decoding can begin. Long prompts usually increase TTFT because the model must process more input before producing the first token.

For interactive local AI, TTFT is often the first metric users notice. A system can generate tokens quickly after it starts yet still feel slow if the first token takes tens of seconds.

ITL: Inter-Token Latency

Inter-token latency measures the gaps between streamed outputs. The official vLLM benchmark documentation records ITL samples between consecutive streamed outputs and reports aggregated statistics. Lower ITL generally means the answer appears more fluid once generation has started.

ITL is especially useful when comparing models or concurrency levels that have similar aggregate throughput. A server may achieve a high TPS number by batching many requests while each user experiences wider gaps between tokens.

TPS: Throughput

Tokens per second can refer to single-stream generation or aggregate server throughput, so the test methodology must be stated. In the CWWK vLLM test, TPS is aggregate throughput across the benchmark load. At Qwen3.5-35B-A3B-AWQ input length 128 and concurrency 32, the platform reached 100.85 TPS. That is not a claim that one chat session outputs 100.85 tokens every second.

A Real Example From CW-A9 Testing

The following rows use the same Qwen3.5-35B-A3B-AWQ model, concurrency 32 and output length 32. Only the input length changes. The result makes the relationship between prompt size, response delay and serving throughput clear.

Nothing about the CPU or GPU changed between these rows. The prompt became longer. That single variable was enough to cut aggregate throughput dramatically and increase first-token delay. This is why “supports 128K context” is not a useful performance claim by itself.

Why Higher Concurrency Can Look Better and Feel Worse

Continuous batching allows a serving engine to keep the accelerator busy with several requests. Aggregate TPS can rise because the hardware is doing more useful work overall. At the same time, each request competes for scheduling time and cache capacity, so TTFT and ITL can increase.

The correct setting depends on the product goal. A private assistant for one person should prioritize response latency. A shared internal API may accept a few seconds more TTFT if it can serve many users efficiently. Batch summarization may prioritize total throughput over interactivity.

Which Metric Should You Optimize?

A strong benchmark should also report percentiles, errors, prompt lengths and concurrency. Averages can hide a few very slow requests. The CW-A9 local AI article explains the tested Ryzen AI Max+ 395 platform, while the dedicated benchmark should be used for model-by-model results.

For repeatable testing, keep model revision, quantization, serving engine, input and output lengths, concurrency, warm-up procedure and sampling settings fixed. Record the software stack as well. If one variable changes between runs, the benchmark may still be useful, but it is no longer a clean hardware comparison. Use representative prompts rather than a single synthetic case, and report failed or unstable runs instead of silently discarding them.

FAQ

Is TPS the same as tokens per second in a chat window?

Not necessarily. Check whether the benchmark reports one request or aggregate throughput across many concurrent requests.

Which metric matters most for RAG?

Usually TTFT is critical because retrieval already adds work before generation. Long retrieved context can increase TTFT further.

Why does ITL increase at long context?

Longer sequences consume more cache and attention work. The effect depends on model architecture, backend and serving strategy.

Should I benchmark with tiny prompts?

Only if that matches the workload. Use several prompt sizes that reflect real chat, code, document and RAG requests.

Related CWWK Resources

· Ryzen AI Max+ 395 Local LLM Benchmark

· Local LLM Memory Requirements

· CWWK CW-A9 AI Mini PC

External Reference

· vLLM Benchmark CLI: TTFT, ITL and TPOT definitions