Decode throughput by categoryMedian per-pass decode t/s at batch = 1, 5 passes per category. The dashed line is the weighted combined score.
Hover a bar for its IQR and coefficient of variation — a high CV means the category's passes disagree, so read small differences there with care.
Stream-update gap by categoryThis server packs several tokens into one stream update, so there is no per-token latency to plot. These are the measured wall-clock gaps between updates: p50 is the typical rhythm, p99 the stutter. Lower is better.
update p50update p99
Hover a bar for the tokens landing per update. p99 is an upper bound on any pause a reader feels; it is not comparable across servers that pack different numbers of tokens into an update.
Aggregate throughput under concurrencyTotal tokens/sec across all in-flight requests as load rises — where this flattens is the throughput knee.Time-to-first-token under concurrencyQueueing shows up here first: p50 is the typical wait, p99 the tail.
TTFT p50TTFT p99
With a modest request count per level, p99 rests on very few observations — treat a spike as a prompt to re-run, not a conclusion.
Prompt processing throughput by depthMedian prefill t/s (prompt tokens ÷ TTFT) at increasing input depth, cold prefix cache. The band spans the 1% low to the 99% high.Full numbers
Single-stream, batch = 1 · several tokens per stream update
category
passes
TTFT p50
TTFT p99
update p50 (ms)
update p99 (ms)
tok/update
decode med
±IQR
CV
chat
5
2,289.3
2,649.2†
1.2
83.3
1.04
46.2
6.0
11.0%
code
5
2,310.5
2,347.4†
1.0
78.7
1.04
52.9
0.6
1.6%
file edit
5
2,559.1
2,877.6†
0.8
96.5
1.06
55.5
2.9
4.5%
json
5
2,382.1
2,436.1†
0.9
86.6
1.04
51.4
5.2
9.0%
math
5
2,294.9
2,512.3†
0.9
74.2
1.02
59.9
3.4
6.3%
prose
5
1,996.5
2,280.9†
22.2
69.1
1.01
48.7
0.9
1.2%
reasoning
5
2,224.1
2,398.6†
1.0
81.5
1.02
44.8
8.5
17.5%
summarization
5
2,132.5
2,552.0†
0.9
76.2
1.03
51.8
5.0
11.9%
Concurrency sweep
level
ok/req
aggregate t/s
TTFT p50
TTFT p99
per-req decode med
1
48/48
39.2
2,322.3
2,833.0†
50.9
2
48/48
39.4
11,165.1
24,721.7†
51.1
4
48/48
38.7
30,729.9
73,108.2†
49.2
8
48/48
38.5
50,831.5
268,746.9†
50.2
Reasoning / answer split · TTFA is time to the first answer token