We measure latency
and do not add to it.

How to read it. Added p50 is over the harness’s own no-gateway baseline. That baseline is one bridge traversal and every gateway is two, so each figure carries one hop that belongs to the topology. The socat byte pipe parses nothing and measures exactly that hop, so read every other row against it. llmbridge’s streaming row measured under the pipe by 0.09 ms, which is noise around zero and reads as at the floor, never as faster than copying bytes. The cores column is not optional: GoModel and Bifrost take every core through the Go runtime and LiteLLM runs one worker per core, so throughput quoted without cores flatters them and understates a single-threaded gateway by about an order of magnitude. The last column is throughput per percent of CPU. Coverage: llmbridge serves four of the six variants, refusing the Anthropic messages pair because that translation is not built, and the translating arm refuses responses. A refusal is not a failure. LiteLLM completed 57,690 of 100,000 streamed chat requests inside the window, and Bifrost’s messages/stream failed outright. TensorZero and OmniRoute are configuration problems, not measurements of the software, and belong in no headline. Two divergences from the published harness, stated because they matter: Docker’s default seccomp filter was off for every gateway, since it blocks io_uring and would have put llmbridge on epoll, and the host is a 12-CPU laptop against the 2-vCPU c7i.large behind ENTERPILOT’s tables, so these rows compare within this table and never with theirs. One llmbridge worker throughout. bench/run_enterpilot.sh in the repository reproduces it, and every variant with the resolved image digests is committed as data next to it.

llmbridge's ceiling here (~84k RPS) is set by the worker's own CPU on its single thread. llmbridge is single-threaded, so one core is its hard ceiling, and at saturation that thread is 87–92% busy, i.e. out of headroom. (The machine reports ~95% idle only because one busy core is ~8% of 12 logical CPUs; reaching the rest needs --workers N.) Splitting that thread's own CPU time: ~89% kernel, ~7% llmbridge, the biggest kernel slice being the TCP stack at 32.7%, the price of being a TCP proxy. Thermally dependent: 87k cold, 82k at 85 °C. Everything runs over 127.0.0.1. Scaling past this ceiling is --workers N (shared-nothing, SO_REUSEPORT), which nearly doubles it per worker; a separate-host run is on the roadmap.

Honest methodology: single Linux host (i7-9750H), mock + proxy + load generator co-located, single worker/thread each, plain HTTP, mock backend (the model's latency is excluded on purpose, to isolate gateway overhead). llmbridge is proxy-self-measured, and LiteLLM is client-measured (e2e − backend). The throughput ceiling on this box is the worker's own CPU, at saturation it burns 87–92% of one core, and perf puts ~89% of that in the kernel (32.7% in the TCP stack) against 6.7% in llmbridge's own code. So the ceiling is a single-thread limit, and the headroom is in kernel work per request, not in the translator. The figure is also thermally dependent (87k cold, 82k at 85 °C on this laptop). Fully reproducible. ./bench/run_headtohead.sh and ./bench/saturate.sh. A clean separate-host run is on the roadmap.

Every column here is what the gateway adds, not what the request took. The no-gateway control's time to first token is 30.7–30.8 ms at every concurrency (that floor is the mock provider's own think time); llmbridge measures 30.6–30.9 ms, i.e. the same number within ±0.2 ms of run-to-run noise, so it adds nothing measurable and absolute TTFT through it stays ~31 ms. LiteLLM holds at 16 streams, then queues. Delivery: llmbridge 99.93–100% of the achievable token stream, LiteLLM 4% at 512 streams. Its ceiling is roughly 1,100 tokens/s per worker regardless of offered load.
TTFT here is time to first token, not first byte: the harness stamps the arrival of the first SSE chunk that actually carries a token, skipping the response headers, so a provider (or gateway) cannot look fast by sending headers early.

Honest methodology: the provider stamps its own monotonic emission time inside every token, so added latency is arrival − emission, no clock sync, and no assumption about when a token should have arrived, which is what makes coordinated omission impossible here. LiteLLM gets a full unmeasured warm-up round (its cold start is severe), one worker each, order alternated per level. LiteLLM's per-token latency above 2 s exceeds our histogram's range, so we print "> 2 s" instead of a number that would look like a percentile but is really just the maximum. llmbridge never overflowed it. Single co-located host, plain HTTP; absolute tails are a dev-box upper bound. Reproduce: ./bench/run_stream_headtohead.sh, full method, harness and host configuration in BENCHMARKS.md.