The inference tape

Where is your
inference wasting
time and money?

Kottos AI records the cost, cache usage, response timing and errors of requests. By understanding your usage, we improve your cost by offering intelligent routing.

We call that record the inference tape.

Tips to reduce your bill See your Claude Code usage → View the trial dashboard ↗

Built on llmbridge, our open-source C++ gateway with measured microsecond overhead.

The trial dashboard is shared and public. For a private account, join the beta.

How it fits together

Observe, analyze, decide, route, verify.

Observe, analyze, decide, route, verify, and the result is the next observation
01ObserveEvery request recorded: tokens, cache, timing, status.
02AnalyzeCost and delivery per model and venue, from the usage each provider reports.
03DecideWhat matters: quotes, cache state or venue health.
04RouteThe request to the chosen venue, with the estimate.
05VerifyThe estimate sits beside the result, per request, on the tape.
 the result becomes the next observation Observe, analyze, decide, route, verify, and the result is the next observation
01ObserveEvery request recorded: tokens, cache, timing, status.
02UnderstandCost and delivery per model and venue, from the usage each provider reports.
03DecideQuotes, cache state and venue health price each available option.
04RouteThe request goes to the chosen venue, with the estimate riding along.
05VerifyThe estimate sits beside the result, per request, on the tape.
 the result is the next observation
01 / Start with your own usage

What are your session metrics?

With only two-line changes in your settings, you can see your usage in the trial dashboard. These changes put Kottos AI between your Claude Code and Anthropic and forwards your requests to store the results. We never store prompt or response text, only metadata.

Eight requests from our own Claude Code traffic · 09/16/2026 · times in UTC as recorded on the tape
TimeModelVenueTTFTTotalInCachedOutExpectedCostStatus
20:42:11claude-fable-5-1anthropic-direct1,262 ms1,953 ms255,805255,07019$0.11$0.07200
20:42:06claude-fable-5-1anthropic-direct2,873 ms4,355 ms255,074255,058227$0.09$0.08200
20:24:22claude-fable-5-1anthropic-direct1,056 ms1,916 ms255,564254,8738$0.10$0.07200
20:24:17claude-fable-5-1anthropic-direct3,956 ms4,101 ms254,905253,632185$0.12$0.10200
20:24:14claude-sonnet-5anthropic-directn/a1,765 ms142,451142,0639$0.03$0.03200
20:24:04claude-fable-5-1anthropic-direct6,302 ms9,362 ms253,664252,709664$0.11$0.12200
20:23:59claude-sonnet-5anthropic-directn/a2,019 ms142,153141,6849$0.03$0.03200
20:23:49claude-fable-5-1anthropic-direct6,013 ms9,933 ms252,713252,689655$0.09$0.10200

Take a look at the CACHED column: nearly the whole prompt is read from cache on every turn, which is why a quarter of a million input tokens costs cents. Expected is what the router priced the request at before sending it. Cost is calculated from the usage the provider reported, at published rates. The two Sonnet requests were not streamed, so they carry no first-token time and read n/a.

Open the dashboard to see:

Usage and cost by model

Input, output, cache reads and cache writes, priced at the applicable published rates.

Time to first token

How long requests took, with gateway and upstream timing shown separately.

Failures

Request status and recorded retries, alongside the requests that completed successfully.

You can start with one provider. The record is useful before you change where any request goes.

Cost is based on published rates so it is not a provider invoice and does not include subscription terms or negotiated discounts.

Set up the trial
02 / Understand the difference

Token prices are just part of a bigger picture.

Nowadays, many models offer prompt caching. This means that long prompts can cost from cents to dollars, depending on how much cache is read. If a long prompt has been cached, switching to a different provider would mean losing this advantage. So choosing whether to keep the current venue or switch to a different one is important.

Public per million tokens prices for Fable 5.1 are: $10 for input, $50 for output, $0.25 for read, and $12.5 for write. However, as of 09/16/2026, our net effective cost for this model is $0.54 (99% cached). Comparing cost based on input price only would miss on this effect.

The tape highlights these differences through the usage each provider reports. Timing and errors are also provided, so you can compare how prices relate to other quality factors.

// an example from our own data

Using the tape to optimize a cache config for us.

Anthropic allows users to write cache tokens for 5 minutes at 1.25x the price, or write them for 1 hour at 2.00x the price. We simulated our own 6,489 Claude Opus 5 requests with a five-minute cache config instead of the one-hour we have by default. The five-minute simulation was 36% more expensive.

03 / Choose based on the evidence

With the evidence in hand, Kottos AI delivers intelligent routing.

Once a client defines its constraints, Kottos AI does the rest. Kottos AI intelligent routing is derived from predictions generated from served requests. It accounts for cache behavior, price, timing, and venue health. The specific routing behavior is defined by the customer's preference.

When routing is based on expected cost, Kottos AI stores the estimate next to the actual cost. These values are shown as EXPECTED and COST in the tape samples above. Routing decisions need to be recorded for posterior verification.

Can Kottos AI make your inference cheaper and faster?

In recent models, OpenAI added different serving tiers: flex, standard, and fast. By default, OpenAI serves the standard tier. The flex tier is cheaper but may be slower than standard. Similarly, the fast tier is more expensive but can be faster than standard. Kottos AI can optimize cost and time by routing to flex when it predicts it will be faster than standard. So we ran an experiment: we had a 200 questions conversation with GPT 5.6 Terra. We sent the 200 questions to the standard tier, and simultaneously through Kottos AI to decide which tier to send it to. Cumulative cost and time are shown below,

Cumulative cost over the same 200 turns. The routed line finishes at $0.1735 against $0.3501 pinned, 50% less.
Cumulative time served over 200 turns. The routed line stays below the pinned line throughout and finishes 82 seconds sooner, 199 seconds against 281.

This experiment ran on 09/21/2026 between 16:25 and 17:20 CT. We sent the exact same 200 questions to each case: the standard tier, and the automatic Kottos AI selection. Every request succeeded, and both arms were sent identical prompts: 1,376,742 input tokens each.

Routing in more detail →
04 / The engine behind the record

Our measurements should not add latency to the request path.

llmbridge is our open-source C++ gateway, available under Apache 2.0. It is the core piece that handles the requests and responses under the Kottos AI routing layer. This gateway needs to be extremely fast and robust under heavy loads.

We compared llmbridge's performance under different benchmarks. In one published measurement, llmbridge added 0.045 ms to an 815 ms time to first token. More information about benchmarks can be found below.

See the benchmarks Run llmbridge

Drop it in

# Add a provider and translate OpenAI ↔ provider on the fly:
llmbridge --listen 8088 --upstream https://api.anthropic.com --upstream-dialect anthropic --workers 4

# Client speaks the OpenAI API. Just point at :8088.
Lluís Antoni Jiménez Rugama
Lluís Antoni Jiménez Rugama
Founded by Lluís Antoni Jiménez Rugama, a PhD in applied mathematics with nine years in high-frequency and electronic trading. That experience taught him not to trust a quoted price and better to keep a record you can inspect. The market thesis.
LinkedIn ↗ · GitHub · [email protected]
05 / Start here

See your traffic on the tape.

Try it with Claude Code

No email, no registration, no signup. Just add two lines in your Claude Code settings. Then, you can inspect your traffic in the shared public dashboard.

Private account

Talk to us about a private beta for your production tasks. This includes deployment requirements, provider access and routing coverage.

Start the trial Discuss a private beta [email protected]