For ninety days we benchmarked twelve inference providers — every major cloud and five specialists — from 32 vantage points across four continents. One number tells the story of 2025: median time-to-first-token fell from 480ms to 92ms in eighteen months. Latency, not raw capability, is now the axis providers compete on.
Why? Model quality converged. On our internal evals, the top eight models land within four points of each other on general tasks. When everyone is smart, the product experience is decided by how fast the first token hits the screen — and whether the stream ever stutters.
Latency is the new accuracy
Users perceive sub-100ms first tokens as instant. Between 100ms and 300ms, an interface feels responsive but mechanical. Past 500ms, engagement measurably decays: in our product analytics, chat sessions with p95 first-token latency above 500ms are 31% shorterthan fast ones with the same model. People don't consciously notice the difference — they just stop asking follow-up questions.
What the numbers show
- Median TTFT, top quartile: 92ms — sub-100ms first tokens are now table stakes, not a differentiator.
- Inter-token stability matters more: providers with the best p50 often had the worst p99 jitter. A stream that hitches every 40 tokens reads worse than one that starts 50ms later.
- Geography still dominates: the same provider varied by up to 210ms between regions. Single-region inference is a latency tax on half your users.
- Cost per token fell 74% year over year at equivalent quality — routing mid-tier prompts to smaller models is where the real savings hide.
How sub-100ms actually happens
Three techniques account for most of the gains. Speculative decoding drafts tokens with a small model and verifies them in batches with the large one — free 2-3× throughput when the draft model agrees. Paged KV-cache with prefix reuse means a returning conversation skips re-encoding its history. And edge routing terminates TLS near the user, streaming tokens over an already-warm connection to the nearest cluster.
Measure it yourself — the methodology is one loop and a high-resolution timer:
const started = performance.now();
let first: number | null = null;
let tokens = 0;
const stream = await client.chat.completions.create({
model: "tapotik-2-flash",
messages: [{ role: "user", content: PROMPT }],
stream: true,
});
for await (const chunk of stream) {
if (chunk.choices[0]?.delta?.content) {
first ??= performance.now() - started; // time to first token
tokens++;
}
}
console.log({ ttft_ms: first, tok_per_s: tokens / ((performance.now() - started) / 1000) });Nobody churns because your model scored two points lower on MMLU. They churn because the answer took a second to start.
What this means if you're building
Treat latency as a product metric, not an infra metric. Put p95 TTFT on the same dashboard as retention. Stream everything — non-streamed endpoints are a design decision to feel slow. And route: reserve your largest model for requests that need it, because the fastest token is the one a smaller model produced correctly.
Full methodology, per-provider data and the harness we used are in the appendix repo. Benchmarks rot fast — we'll rerun this quarterly and keep the charts live.