Time to first token
How long a user waits before the answer starts.
InferenceX, explained
Understand LLM inference performance
Raw throughput does not tell you how an agent feels. Read latency, speed, and capacity together.
Generation speed
How quickly the answer continues after it starts.
Concurrency
How many requests share the system at once.
What selected runs mean
These are independent InferenceX measurements, not Morph production benchmarks. Each row links to its source run. Similar model names are not interchangeable.
DeepSeek V4 Pro on B300
8K input, 1K output, concurrency 1
- First token
- 0.35 sec
- Generation
- 216 tok/sec/user
Fast for one interactive agent.
View source runDeepSeek V4 Pro on B200
8K input, 1K output, concurrency 64
- First token
- 4.25 sec
- Generation
- 40 tok/sec/user
More total work, but users wait longer.
View source runKimi K3 on B300
Agent trace, concurrency 1
- First token
- 1.69 sec
- Generation
- 220 tok/sec/user
Responsive generation on a long agent trace.
View source runGLM 5.2 on B300
Agent trace, concurrency 1
- First token
- 0.80 sec
- Generation
- 334 tok/sec/user
Very fast generation on this trace.
View source runInteractive explorer
Compare speed and inference cost
Filter attributed measurements by model, GPU, and minimum generation speed. A missing cost means the source run did not report enough information to calculate it.
Kimi K3 on B300
AgentX coding agent trace
- User speed
- 50 tok/sec
- GPU throughput
- 6,083 tok/sec
- Estimated compute cost
- $0.10 per million tokens
- Serving setup
- vLLM, FP4
Kimi K3 on B300
AgentX coding agent trace
- User speed
- 100 tok/sec
- GPU throughput
- 4,051 tok/sec
- Estimated compute cost
- $0.15 per million tokens
- Serving setup
- vLLM, FP4
Kimi K3 on B300
AgentX coding agent trace
- User speed
- 150 tok/sec
- GPU throughput
- 2,075 tok/sec
- Estimated compute cost
- $0.30 per million tokens
- Serving setup
- vLLM, FP4
Kimi K3 on B300
AgentX coding agent trace
- User speed
- 200 tok/sec
- GPU throughput
- 2,027 tok/sec
- Estimated compute cost
- $0.31 per million tokens
- Serving setup
- vLLM, FP4
DeepSeek V4 Pro on B300
8K input, 1K output
- User speed
- 216 tok/sec
- GPU throughput
- 216 tok/sec
- Estimated compute cost
- Not reported
- Serving setup
- SGLang, FP4
DeepSeek V4 Pro on B200
8K input, 1K output at concurrency 64
- User speed
- 40 tok/sec
- GPU throughput
- 2,560 tok/sec
- Estimated compute cost
- Not reported
- Serving setup
- SGLang, FP4
Reference data is directional. Model version, workload, context length, concurrency, cache state, precision, framework, and topology must match before a result can size a production endpoint.
The common mistake
Do not maximize throughput first
A highly batched run can produce impressive total throughput while making every interactive user wait. Choose an acceptable first token delay and generation speed first. Then compare the capacity that remains inside those limits.
Context length, cache reuse, framework, precision, topology, and software version can change the answer. Always compare the same workload.
Turn benchmarks into a capacity plan
Use your team size and traffic shape to estimate monthly demand, then verify with your traces.