Prefix Caching
Reuse the KV cache across requests that share a prefix. Lower TTFT on long system prompts, cheaper retrieval-heavy chats, and you write zero client code. Routing prefers the backend that already holds your prefix, and a stable X-Conversation-Id keeps a conversation on the backend that served its last request.
- client_code
- 0 lines
- on by default, every tier
- cached_token_price
- discounted
- below the full per-token rate
- routing_signal
- prefix affinity
- scored alongside latency and load
- reported_in
prompt_tokens_details.cached_tokens- every chat completion response
Overview
Large language models process input tokens through a series of attention layers, producing intermediate key-value (KV) cache entries for each token. When two requests share the same prefix (for example, the same system prompt), the KV cache entries for that prefix are identical. Prefix caching avoids recomputing these shared entries: a backend that already holds them reuses them.
Erebine handles prefix caching automatically. No client-side configuration is needed. All requests benefit from prefix caching by default.
A model's owner can turn it off for every engine serving the model with the
model's prefix_caching setting, part of its
engine tuning.
Benefits
- Reduced time-to-first-token (TTFT). Cached prefixes skip the prefill computation phase, reducing latency for the first generated token.
- Lower compute cost. Fewer tokens need to be processed on the GPU, freeing capacity for other requests.
- Higher throughput. Backends can serve more requests per second when prefill is partially skipped.
How It Works
Each backend's inference engine keeps the KV entries of the prompts it serves and reuses them when a later request on the same backend starts with the same tokens. It evicts the least recently used entries first, so a prefix in steady use stays warm while rarely used ones are reclaimed. This works on every backend without any per-request configuration.
Chat completions are placed by prefix. The router remembers which backend served the start of each of your project's prompts. When a request begins with a prefix that a backend able to take it has already served, the router scores that match alongside latency, load and cache pressure, so the warm backend usually wins with no header or client code. It is a preference, not a guarantee: a warm backend that is much busier or slower can lose to a cold one. Another project's prompts never count as a match.
To hold a conversation on one backend, send the same
X-Conversation-Id header on every request of the conversation.
The router keeps the conversation on the backend that served its last
request while that backend can take it, so each turn reuses the prefix the
last one left warm. When that backend is drained, at capacity or unhealthy,
the request is placed by prefix like any other and the conversation
continues where it lands. Realtime sessions
(Realtime) keep their own routing: a
session's turns stay on the backend that served its earlier turns while
that backend can take them.
Prompt Optimization
While prefix caching is automatic, you can structure your prompts to maximize cache hit rates. The key principle is: put stable content at the beginning and variable content at the end.
Best Practices
| Practice | Why It Helps |
|---|---|
| Place system prompts first | System prompts are typically identical across requests. Placing them first ensures the shared prefix is as long as possible. |
| Use consistent prompt templates | Even small differences (extra whitespace, reworded instructions) create different cache keys and prevent cache reuse. |
| Keep shared context before variable content | Place few-shot examples, RAG context, or conversation history after the system prompt but before the user's new message. |
| Avoid per-request timestamps or IDs in early messages | Dynamic values in early messages invalidate all downstream cache entries, preventing any cache reuse. |
Example: Optimized Prompt Structure
{
"messages": [
{"role": "system", "content": "You are a helpful customer support agent for Acme Corp. Always be polite and reference our return policy when relevant."},
{"role": "user", "content": "I want to return my order #12345."}
]
}
In this example, the system message is identical across all customer support requests. A backend caches the KV entries for the system prompt and reuses them for every request it serves, regardless of the user's specific question.
Example: Suboptimal Prompt Structure
{
"messages": [
{"role": "system", "content": "You are a helpful agent. Current time: 2025-01-15T10:30:00Z. Session: abc-123."},
{"role": "user", "content": "I want to return my order #12345."}
]
}
Here, the timestamp and session ID in the system prompt change with every request, preventing any prefix caching. Move dynamic values to a later message or to the user message if possible.
The single biggest cache-killer is a dynamic value in early messages. A timestamp, request ID, user ID, or random nonce in the system prompt invalidates every downstream cache entry, even if the rest of the prompt is byte-identical. Keep dynamic values in user messages or trailing tool results, not in the system prompt.
Multi-Turn Conversations
In multi-turn conversations, the entire conversation history up to the current turn forms the prefix. As the conversation grows, more of the prompt is shared between turns. The first few turns may not benefit much from caching, but longer conversations see increasing cache hit rates because each new turn only adds content at the end of the existing prefix.
Cached Tokens in Usage
The usage object in chat completion responses includes a
prompt_tokens_details field that reports how many input tokens
were served from the KV cache:
{
"usage": {
"prompt_tokens": 150,
"completion_tokens": 42,
"total_tokens": 192,
"prompt_tokens_details": {
"cached_tokens": 128,
"audio_tokens": null
}
}
}
In this example, 128 of the 150 prompt tokens were served from the KV cache,
meaning the backend only needed to compute the remaining 22 tokens during
prefill. The higher the cached_tokens ratio, the faster
the time-to-first-token.
A cached_tokens value of 0 means no prefix cache was available
for that request. This is normal for the first request with a given prefix
or when the backend's KV cache has been evicted.
Cost Implications
On billable tiers, cached tokens are charged at a fraction of the full per-token
rate. Each tier has a cached_token_cost_multiplier that determines
the discount. On Self-Hosted
cached input tokens are free. The Free tier does not meter cached tokens.
For example, if a request has 1,000 prompt tokens and 800 are served
from cache, you are charged for 200 full-price tokens plus 800
tokens at the tier's cached rate, rather than 1,000 full-price
tokens. Call the tier's full rate R and
its cached-token multiplier m; the request costs
200R + 800mR, less than the 1000R you
would have paid with no cache.
Figures are behind sign-in. Per-token rates and the cached-token multiplier are shown to signed-in accounts; the formula below is the same either way. For a quote without an account, mail contact@erebine.ai.
cost = (uncached_tokens / 1_000_000) * price_per_1m
+ (cached_tokens / 1_000_000) * price_per_1m * cached_token_cost_multiplier
The cached_tokens count is reported in
usage.prompt_tokens_details.cached_tokens in every response, so you
can monitor savings per request. See
Service Tiers: Cached Token Pricing
for which tiers apply a multiplier.
Troubleshooting
Most "cache isn't working" reports trace back to one of three causes. Check them in order before opening a support thread.
| Symptom | Likely cause | What to check |
|---|---|---|
cached_tokens is always 0 |
Dynamic value early in the prompt | Inspect the first 1-2 messages. Strip timestamps, request IDs, and per-request metadata; move them to the user message. |
cached_tokens only sometimes > 0 |
Prefix landed on a different backend | Prefix affinity is a preference: under load a busier warm backend can
lose to a cold one, and after a router restart the router no longer
knows which backend served a prefix. Send the same
X-Conversation-Id on every request of a conversation to
hold it on one backend; it still moves when that backend is drained, at
capacity or unhealthy. The ratio averages out across requests; treat
any single response as a sample, not a verdict. |
| Was caching, now isn't | Prompt template drift | Diff the current prompt against a known-good run. Extra whitespace, rewrapped lines, or a reordered tool list all break the prefix hash. |
Tips
- Use streaming for perceived latency. Streaming delivers tokens as they are generated, improving perceived responsiveness even when total latency is unchanged. See the Streaming API.
- Choose the right tier. Base, Plus, and Max carry interactive scheduling priority for latency-sensitive workloads. Bulk is the lowest metered rate and is scheduled behind them, which suits batch processing where latency is less critical. See Service Tiers.
- Monitor cached_tokens. Track the
cached_tokensfield in usage responses over time. A consistently low ratio may indicate that your prompt templates are not stable enough for caching. - Avoid unnecessary prompt variation. Randomized few-shot example ordering, per-request metadata in system prompts, and dynamic timestamps all reduce cache effectiveness.