docs
// Features

Prefix Caching

Reuse the KV cache across requests that share a prefix. Lower TTFT on long system prompts, cheaper retrieval-heavy chats, and you write zero client code. Routing prefers the backend that already holds your prefix, and a stable X-Conversation-Id keeps a conversation on the backend that served its last request.

client_code
0 lines
on by default, every tier
cached_token_price
discounted
below the full per-token rate
routing_signal
prefix affinity
scored alongside latency and load
reported_in
prompt_tokens_details.cached_tokens
every chat completion response

Overview

Large language models process input tokens through a series of attention layers, producing intermediate key-value (KV) cache entries for each token. When two requests share the same prefix (for example, the same system prompt), the KV cache entries for that prefix are identical. Prefix caching avoids recomputing these shared entries: a backend that already holds them reuses them.

Erebine handles prefix caching automatically. No client-side configuration is needed. All requests benefit from prefix caching by default.

A model's owner can turn it off for every engine serving the model with the model's prefix_caching setting, part of its engine tuning.

Benefits

  • Reduced time-to-first-token (TTFT). Cached prefixes skip the prefill computation phase, reducing latency for the first generated token.
  • Lower compute cost. Fewer tokens need to be processed on the GPU, freeing capacity for other requests.
  • Higher throughput. Backends can serve more requests per second when prefill is partially skipped.

How It Works

Each backend's inference engine keeps the KV entries of the prompts it serves and reuses them when a later request on the same backend starts with the same tokens. It evicts the least recently used entries first, so a prefix in steady use stays warm while rarely used ones are reclaimed. This works on every backend without any per-request configuration.

Chat completions are placed by prefix. The router remembers which backend served the start of each of your project's prompts. When a request begins with a prefix that a backend able to take it has already served, the router scores that match alongside latency, load and cache pressure, so the warm backend usually wins with no header or client code. It is a preference, not a guarantee: a warm backend that is much busier or slower can lose to a cold one. Another project's prompts never count as a match.

To hold a conversation on one backend, send the same X-Conversation-Id header on every request of the conversation. The router keeps the conversation on the backend that served its last request while that backend can take it, so each turn reuses the prefix the last one left warm. When that backend is drained, at capacity or unhealthy, the request is placed by prefix like any other and the conversation continues where it lands. Realtime sessions (Realtime) keep their own routing: a session's turns stay on the backend that served its earlier turns while that backend can take them.

Prompt Optimization

While prefix caching is automatic, you can structure your prompts to maximize cache hit rates. The key principle is: put stable content at the beginning and variable content at the end.

Best Practices

Practice Why It Helps
Place system prompts first System prompts are typically identical across requests. Placing them first ensures the shared prefix is as long as possible.
Use consistent prompt templates Even small differences (extra whitespace, reworded instructions) create different cache keys and prevent cache reuse.
Keep shared context before variable content Place few-shot examples, RAG context, or conversation history after the system prompt but before the user's new message.
Avoid per-request timestamps or IDs in early messages Dynamic values in early messages invalidate all downstream cache entries, preventing any cache reuse.

Example: Optimized Prompt Structure

JSON cache_friendly
{ "messages": [ {"role": "system", "content": "You are a helpful customer support agent for Acme Corp. Always be polite and reference our return policy when relevant."}, {"role": "user", "content": "I want to return my order #12345."} ] }

In this example, the system message is identical across all customer support requests. A backend caches the KV entries for the system prompt and reuses them for every request it serves, regardless of the user's specific question.

Example: Suboptimal Prompt Structure

JSON cache_busting
{ "messages": [ {"role": "system", "content": "You are a helpful agent. Current time: 2025-01-15T10:30:00Z. Session: abc-123."}, {"role": "user", "content": "I want to return my order #12345."} ] }

Here, the timestamp and session ID in the system prompt change with every request, preventing any prefix caching. Move dynamic values to a later message or to the user message if possible.

The single biggest cache-killer is a dynamic value in early messages. A timestamp, request ID, user ID, or random nonce in the system prompt invalidates every downstream cache entry, even if the rest of the prompt is byte-identical. Keep dynamic values in user messages or trailing tool results, not in the system prompt.

Multi-Turn Conversations

In multi-turn conversations, the entire conversation history up to the current turn forms the prefix. As the conversation grows, more of the prompt is shared between turns. The first few turns may not benefit much from caching, but longer conversations see increasing cache hit rates because each new turn only adds content at the end of the existing prefix.

Cached Tokens in Usage

The usage object in chat completion responses includes a prompt_tokens_details field that reports how many input tokens were served from the KV cache:

JSON
{ "usage": { "prompt_tokens": 150, "completion_tokens": 42, "total_tokens": 192, "prompt_tokens_details": { "cached_tokens": 128, "audio_tokens": null } } }

In this example, 128 of the 150 prompt tokens were served from the KV cache, meaning the backend only needed to compute the remaining 22 tokens during prefill. The higher the cached_tokens ratio, the faster the time-to-first-token.

A cached_tokens value of 0 means no prefix cache was available for that request. This is normal for the first request with a given prefix or when the backend's KV cache has been evicted.

Cost Implications

On billable tiers, cached tokens are charged at a fraction of the full per-token rate. Each tier has a cached_token_cost_multiplier that determines the discount. On Self-Hosted cached input tokens are free. The Free tier does not meter cached tokens.

For example, if a request has 1,000 prompt tokens and 800 are served from cache, you are charged for 200 full-price tokens plus 800 tokens at the tier's cached rate, rather than 1,000 full-price tokens. Call the tier's full rate R and its cached-token multiplier m; the request costs 200R + 800mR, less than the 1000R you would have paid with no cache.

Figures are behind sign-in. Per-token rates and the cached-token multiplier are shown to signed-in accounts; the formula below is the same either way. For a quote without an account, mail contact@erebine.ai.

cost = (uncached_tokens / 1_000_000) * price_per_1m + (cached_tokens / 1_000_000) * price_per_1m * cached_token_cost_multiplier

The cached_tokens count is reported in usage.prompt_tokens_details.cached_tokens in every response, so you can monitor savings per request. See Service Tiers: Cached Token Pricing for which tiers apply a multiplier.

Troubleshooting

Most "cache isn't working" reports trace back to one of three causes. Check them in order before opening a support thread.

Symptom Likely cause What to check
cached_tokens is always 0 Dynamic value early in the prompt Inspect the first 1-2 messages. Strip timestamps, request IDs, and per-request metadata; move them to the user message.
cached_tokens only sometimes > 0 Prefix landed on a different backend Prefix affinity is a preference: under load a busier warm backend can lose to a cold one, and after a router restart the router no longer knows which backend served a prefix. Send the same X-Conversation-Id on every request of a conversation to hold it on one backend; it still moves when that backend is drained, at capacity or unhealthy. The ratio averages out across requests; treat any single response as a sample, not a verdict.
Was caching, now isn't Prompt template drift Diff the current prompt against a known-good run. Extra whitespace, rewrapped lines, or a reordered tool list all break the prefix hash.

Tips

  • Use streaming for perceived latency. Streaming delivers tokens as they are generated, improving perceived responsiveness even when total latency is unchanged. See the Streaming API.
  • Choose the right tier. Base, Plus, and Max carry interactive scheduling priority for latency-sensitive workloads. Bulk is the lowest metered rate and is scheduled behind them, which suits batch processing where latency is less critical. See Service Tiers.
  • Monitor cached_tokens. Track the cached_tokens field in usage responses over time. A consistently low ratio may indicate that your prompt templates are not stable enough for caching.
  • Avoid unnecessary prompt variation. Randomized few-shot example ordering, per-request metadata in system prompts, and dynamic timestamps all reduce cache effectiveness.