Tokens Per Second: The LLM Speed Benchmark That Actually Matters for Coding Agents

What Tokens Per Second Actually Measures

A token is the atomic unit of LLM input and output. One token maps to roughly 3-4 characters of English text or 2-3 characters of code. The word “function” is one token. A line of Python like def calculate_total(items): is about 7 tokens.

Tokens per second measures the rate at which a model generates output tokens during the decode phase of inference. After the model processes your input prompt (the prefill phase), it begins generating output tokens one at a time, each conditioned on every token that came before it. The speed of this autoregressive generation is what tok/s captures.

A model running at 100 tok/s produces about 75-100 words per second. That is roughly 15x faster than the average human typing speed and 20x faster than comfortable reading speed. At 50 tok/s, the model already outruns any human consumer of its output. So why does speed beyond 50 tok/s matter? Because not every consumer is human.

Three Metrics, One Name

When someone says “tokens per second,” they could mean any of three distinct measurements. Confusing them leads to bad provider decisions.

A provider quoting “800 tok/s” might mean 800 tok/s throughput across 10 concurrent users, each experiencing 80 tok/s. Or it might mean 800 tok/s on a single request with a tiny 0.8B model. Or it might mean 800 tok/s on a batch of pre-cached prompts with artificially low TTFT. The number without the measurement conditions is meaningless.

The Latency Equation

Total response latency for a single request breaks down to:

Latency = TTFT + (output_tokens / output_speed)

A request generating 500 tokens with 0.5s TTFT and 100 tok/s output speed takes 5.5 seconds total. The same request with 0.2s TTFT and 200 tok/s takes 2.7 seconds. Both TTFT and output speed contribute, but their relative importance depends on your use case. Streaming chatbots prioritize TTFT. Batch processing prioritizes throughput. Coding agents prioritize output speed on specific subtasks, as we will see.

Why Coding Agents Care About a Different Speed

Human users perceive speed through TTFT and reading speed. Once tokens arrive faster than ~250 words per minute, a faster model feels the same. The bottleneck shifts to human cognition.

Coding agents have no cognition bottleneck. They consume output at the speed it arrives and immediately take the next action. Every millisecond of generation latency delays the next tool call, the next file edit, the next test run. In a 30-step agentic task, a 5-second delay per step means 2.5 minutes of pure waiting.

The Three Speed Bottlenecks

Agent task latency decomposes into three bottlenecks, and most optimization ignores the biggest one:

Reasoning is the step that needs a frontier model. Apply is the step that needs speed. When a coding agent asks Claude or GPT to rewrite an entire file to apply a 5-line change, it is paying frontier-model prices for a mechanical merge operation. The fix is to route the apply step to a model built for speed on that specific task.

Provider Speed Comparison (April 2026)

The table below shows output speed (single-request tok/s) for notable providers and models. Numbers are from Artificial Analysis, BenchLM, and provider-reported benchmarks. All measurements use standardized tokenization.

Two patterns stand out. First, specialized inference hardware (Cerebras, Groq, SambaNova) runs 5-20x faster than GPU-based API providers on the same open-source models. The tradeoff is model selection: these providers only support a handful of models. Second, task-specific models demolish general-purpose speed records. Morph Fast Apply at 10,500 tok/s is not competing with Claude at 40 tok/s on the same task. It is doing a narrower task (code merging) fast enough that the step becomes negligible in the overall pipeline.

What Affects Your Tokens Per Second

Six factors determine the tok/s you actually experience. Understanding them explains why the same model at the same provider can feel fast one day and slow the next.

1. Model Size

The dominant factor. Decode speed is memory-bandwidth-bound: each token requires reading the full model weights from memory. Larger models have more weights to read. On the same GPU, a 7B model runs 3-5x faster than a 70B. The relationship is not perfectly linear because larger models also have more layers of KV cache to read, and cache access patterns interact with memory bandwidth differently.

2. Quantization

Reducing weight precision from FP16 (16-bit) to INT4 (4-bit) halves memory reads per token, roughly doubling speed. FP8 gives a 33% improvement with near-zero accuracy loss. INT4 methods like AWQ and GPTQ push further but introduce 1-3% accuracy degradation. The kernel implementation matters more than the quantization method: AWQ with Marlin kernels runs at 741 tok/s versus 68 tok/s without, a 10.9x difference from the same quantized weights.

3. Context Length

Longer input prompts increase TTFT linearly because prefill processes all input tokens. They also grow the KV cache, which the model reads at every decode step. On a 70B model, a 2K-token prompt might yield 80 tok/s output speed while a 100K-token prompt drops to 50 tok/s as the KV cache exceeds on-chip memory and spills to slower storage.

4. Batch Size

Batching multiple requests together increases GPU utilization and system throughput. Continuous batching (used by vLLM, SGLang, and TensorRT-LLM) can push total throughput to 2,400-2,780 tok/s on an H100 with 100 concurrent requests. But batching trades individual speed for collective throughput. Each user gets a fraction of the total.

5. Hardware Architecture

Cerebras wafer-scale engines and Groq LPUs are designed for sequential token generation. They keep model weights in on-chip SRAM instead of HBM, eliminating the memory bandwidth bottleneck that limits GPU decode speed. The result: 5-20x faster per-request speeds on supported models. NVIDIA GPUs compensate with software optimizations (speculative decoding, continuous batching, Flash Attention) that narrow but do not close the gap.

6. Speculative Decoding

A small draft model (1-7B parameters) generates 3-12 candidate tokens per step. The target model verifies them all in one parallel forward pass. At 70-90% acceptance rates on domain-specific tasks, this yields 2-3x speedup with zero quality loss. Now standard in vLLM, SGLang, and TensorRT-LLM. Morph Fast Apply uses speculative decoding on its 7B model to reach 10,500 tok/s, because code merge patterns are highly predictable for a well-tuned draft model.

How to Measure Tokens Per Second Correctly

Most published benchmarks are useless for predicting your production experience. A model that runs at 200 tok/s in a controlled test can drop to 60 tok/s under moderate concurrency, or spike its TTFT from 0.5s to 5s during peak load.

Benchmarking Tools

The Apply Speed Gap

The speed tables above compare models on general text generation. For coding agents, a different benchmark matters: how fast can the model merge an edit into an existing file?

Most coding agents use their frontier model (Claude, GPT, Gemini) for every step, including the mechanical step of applying edits. The frontier model rewrites the entire file. For a 1,000-line file, that means generating 3,500-4,500 tokens at 40-130 tok/s. That is 27-112 seconds per edit, and each edit costs frontier-model prices ($3-15 per million output tokens).

Morph Fast Apply is a 7B model trained specifically on code merging. It takes the original file, a code edit snippet with “existing code” markers for unchanged sections, and an instruction. It returns the complete merged file. At 10,500 tok/s, a 500-line file merges in 0.8 seconds. A 1,000-line file takes 1.3 seconds. Compare that to 12+ seconds with a frontier model.

The speed comes from three factors: a small model (7B parameters), speculative decoding tuned for highly predictable code merge patterns, and custom CUDA kernels. Code merging is a well-defined enough task that a specialized model beats general-purpose models on both speed and accuracy. Benchmarks show 100% merge success rate when pairing Fast Apply with GPT-5, Claude Sonnet 4, and DeepSeek. Search-and-replace with the same models drops to 84-96%.

The cost reduction compounds with the speed gain. Fast Apply uses 50-60% fewer tokens per edit (because it does not rewrite unchanged code) at a per-token price that is 10-18x cheaper than frontier models. A 20-edit coding task drops from $1.06-1.36 in apply costs to $0.02-0.03.

Frequently Asked Questions

What does tokens per second mean for LLMs?

Tokens per second measures the rate at which a model generates output during inference. One token is about 3-4 characters. The metric has three variants: TTFT (time to first token), output speed (generation rate after the first token), and throughput (total tokens across all concurrent requests). A model at 100 tok/s generates about 75-100 words per second.

What is a good tokens per second speed?

Depends on the use case. For chatbots, 50+ tok/s is sufficient since it outruns human reading speed. For coding agents, faster is meaningfully better because the agent executes on output immediately. Frontier API models run at 40-130 tok/s. Specialized inference providers like Cerebras reach 2,100 tok/s on open-source models. For code apply operations, Morph Fast Apply runs at 10,500 tok/s.

Which LLM inference provider is fastest?

As of April 2026: Cerebras leads on open-source model speed at 2,100 tok/s for Llama 3.1 70B. SambaNova follows at 580 tok/s with the best TTFT (0.2s). Groq delivers 276 tok/s with the most consistent speed across context lengths. Among frontier API models, Gemini 2.5 Flash leads at 238 tok/s. See the full comparison tables above.

What is the difference between TTFT and output tokens per second?

TTFT measures the delay before the first token appears, determined by the prefill phase. Output tok/s measures generation speed after the first token, determined by the decode phase. A model can have fast TTFT and slow generation, or vice versa. Reasoning models (o3, GPT-5.2, Gemini 2.5 Pro) have especially high TTFT (20-130 seconds) because they do extensive thinking before generating.

What affects tokens per second speed?

Model size (7B runs 3-5x faster than 70B), quantization (INT4 roughly doubles speed vs FP16), context length (longer contexts slow both TTFT and output speed), batch size (increases system throughput, can slow individual requests), hardware (Cerebras and Groq chips vs NVIDIA GPUs), and software optimizations (speculative decoding gives 2-3x speedup).

Why do coding agents need faster tok/s than chatbots?

Chatbot users read at 4-5 words per second. A model at 50 tok/s already outruns them. Coding agents do not read. They execute. An agent waiting for a 1,000-line file rewrite at 80 tok/s stalls for 12 seconds. At 10,500 tok/s, the same edit completes in 1.3 seconds. That 10-second gap compounds across every edit in a multi-step task.

How do I benchmark LLM speed correctly?

Standardize tokenization (use tiktoken o200k_base for cross-model comparison). Separate TTFT from output speed. Test at your actual concurrency, not single-request. Run for 72+ hours to capture load variance. Report p50, p95, and p99 latencies. Tools: NVIDIA GenAI-Perf for self-hosted engines, Artificial Analysis and BenchLM for API providers.