The Router
One OpenAI-compatible surface. Behind it, a dispatcher that places every request on an Erebine Inference Microservice (EIM). A prompt already warm on an EIM is scored on sixteen independent signals and sampled with A-Res weighted random; other requests follow the endpoint's standing placement or your weights, and are scored the same way when neither applies. Dispatch runs over a mutually-authenticated CurveZMQ mesh. Managed pools, self-hosted EIMs, and public cloud are one fleet.
Overview
Every Erebine inference request, chat, responses, embeddings, reranking, tool calls, arrives at the same router process that also serves the frontend page surface. The router holds a live picture of every connected EIM: which models are loaded, how hot their KV cache is, how recently each one answered, how close it is to your traffic, and whether the project sending the request is over or under its fair share.
How a request is placed depends on what the router already knows:
- Pinned conversation. A chat completion sent with
X-Conversation-Idgoes back to the EIM that served the conversation's last request while that EIM can take it. - Warm prompt. A chat completion whose prompt prefix an eligible EIM already holds is scored on every signal across the eligible EIMs, and a winner is sampled with a weighted-random algorithm that still leaves room for exploration. Realtime turns are always scored this way.
- Standing placement or weights. Any other request goes to the EIMs the endpoint is already placed on, or is split across the endpoint's EIMs by the weights you set.
- Neither. A request with no standing placement and no weights is scored like a warm one.
Dispatch runs over an authenticated, encrypted mesh with built-in backpressure. The whole decision happens before your first token, typically in under a millisecond at the router itself.
flowchart LR
Client["Client request
(chat / responses / tools)"]
Admit["Admission
auth + credits + rate limits"]
Score["Composite scoring
16 signals"]
Select["Weighted-random
sampling + exploration"]
Dispatch["Encrypted dispatch
over inference mesh"]
Stream["Streaming response
with credit-based flow control"]
Feedback["Post-request feedback
recalibrates the router"]
Place{"Warm prompt?
Standing placement
or weights?"}
Follow["Standing placement
or your weights"]
Client --> Admit
Admit --> Place
Place -- "warm, or neither" --> Score
Place -- "placement or weights" --> Follow
Score --> Select
Select --> Dispatch
Follow --> Dispatch
Dispatch --> Stream
Stream -.-> Feedback
Feedback -.-> Score
Composite Scoring
Picking an EIM by latency alone is a trap, the fastest EIM a millisecond ago may be the one whose KV cache just evicted your prompt, the one whose queue just filled, or the one in a region a continent away. The router blends 16 distinct signals into a single composite score, and computes it fresh for every request it scores.
The sixteen signals
- // affinity
- Model-cache affinity
- Is the model already loaded? Cold-loading a 70B model is measured in minutes; a warm EIM answers immediately.
- // prefix
- Prefix-cache match
- How much of your prompt's KV cache the EIM still holds from a prior request. More overlap, shorter prefill, faster first token. A bonus tier stacks on top when overlap is high and the prefix-match prediction has proven accurate for that EIM.
- // ewma
- Latency prediction
- A per-EIM, exponentially-weighted moving average of recent end-to-end latency. Weighted harder when an SLO is attached.
- // pressure
- KV-cache pressure
- EIMs near their KV ceiling are deprioritized and, past a threshold, excluded entirely.
- // health
- Worker health
- Heartbeat freshness, error rates, and probe results drive a continuous health score.
- // slots
- Worker utilization
- How many slots remain available right now.
- // region
- Region affinity
- Stay close to the caller when latency matters; spill out when capacity does.
- // fairness
- Per-project fairness
- Projects under their fair share are favored; a project past its concurrent-request threshold is scheduled behind the rest, never rejected.
- // tier
- Service-tier priority
- Priority requests outrank standard. Flex requests yield to both.
- // ownership
- Self-hosted ownership
- If your project owns the EIM, your requests get first pass at it.
- // slo
- SLO bonus
- EIMs consistently meeting your SLO win ties.
- // origin
- Internal vs. external
- Internal traffic gets a configurable boost to keep system jobs from being starved by user traffic.
- // batch
- Batch penalty
- Batch requests are weighted down so interactive traffic preempts them naturally.
- // staleness
- Metrics-staleness penalty
- EIMs whose telemetry is overdue are progressively distrusted until they re-report.
- // in-flight
- In-flight token rate
- Tokens still to be generated on an EIM's active requests, divided by its measured tokens per second. One long generation blocks more capacity than one short one, and raw slot counts cannot see that.
- // deadline
- Deadline tail risk
- When a request carries a deadline and an EIM's predicted latency plus its tail spread would overrun it, that EIM takes a hard penalty. Fast median, fat tail loses.
flowchart LR
subgraph Signals
S1["Model-cache affinity"]
S2["Prefix-cache match"]
S3["EWMA latency"]
S4["KV pressure"]
S5["Worker health"]
S6["Utilization"]
S7["Region affinity"]
S8["Per-project fairness"]
S9["Service tier"]
SA["...additional signals"]
end
Score{"Composite score
per EIM"}
S1 --> Score
S2 --> Score
S3 --> Score
S4 --> Score
S5 --> Score
S6 --> Score
S7 --> Score
S8 --> Score
S9 --> Score
SA --> Score
Weighted Selection & Exploration
A single best score is rarely the best answer. Always picking the top-scoring EIM concentrates traffic, accelerates KV evictions on the favorite, and prevents the router from learning that a newly connected EIM is actually faster. The router avoids this in two ways.
A-Res weighted-random sampling
Instead of picking the maximum score, the router draws from all
eligible EIMs using the A-Res algorithm (Efraimidis and
Spirakis). Each EIM's selection probability is proportional to
its composite score, so the best EIM wins most of the time.
An EIM that is 90% as good still wins a meaningful share of
requests. Load spreads naturally, KV caches stay warm across the
pool, and a high scorer that briefly stumbles has competitors
ready to absorb the traffic.
Epsilon-greedy exploration
A small slice of scored requests, five percent by default, bypasses the composite score entirely and chooses uniformly at random from the eligible pool. This keeps the router honest: newly connected EIMs, EIMs whose latency model is stale, and EIMs recovering from a degraded state all get sampled, measured, and folded back into the scoring model. The result is a router that keeps learning, instead of one that lock-steps to whatever was fastest an hour ago.
Prefix-Cache Affinity
The router keeps a per-EIM index of which prompt prefixes each EIM has recently served. When a chat completion or a realtime turn arrives, the router predicts, before dispatching, how much of the prompt's KV cache is likely already resident on each candidate, and prefers the candidates with the warmest match.
Self-calibrating accuracy
Every prediction is checked. After the request completes, the EIM reports how many prefix blocks it actually hit, and the router compares that to what it predicted. The per-EIM accuracy of those predictions is tracked with an exponential moving average and used to dampen or amplify future prefix-match scores. An EIM whose cache map turns out to be a poor predictor gets less benefit of the doubt; an EIM whose predictions are consistently right gets a strong-prefix bonus on top.
If accuracy collapses for any EIM, usually because its cache state diverged from what the router believed, the router resets that EIM's prefix index and rebuilds it from fresh evidence. No human intervention required.
sequenceDiagram
participant Router
participant Worker as EIM
Router->>Router: Predict prefix match for request
Router->>Worker: Dispatch
Worker-->>Router: Response + actual hit/total blocks
Router->>Router: EWMA-update prediction accuracy
Note over Router: Accuracy too low?
Reset this EIM's index.
The user-facing effect is simple: when your prompts share stable prefixes (a system prompt, a few-shot block, a long document quoted across many turns) the router quietly funnels those requests to the EIMs most likely to skip the prefill, without you having to pin or tag anything. See Prefix Caching for prompt-structure best practices.
Warm cache before weights
On an endpoint served by several EIMs you weight, the cache comes
first. A request whose prompt prefix an attached EIM already holds
is scored and goes to the warm EIM ahead of the weights, as does a
conversation held on one EIM with X-Conversation-Id.
The weights decide every request with no warm prefix. A drained
EIM (weight 0) takes no new requests either way, and no request
reaches an EIM the endpoint does not allow.
Per-Project Fairness
A shared router is only as fair as its weakest defense against a noisy neighbor. The router enforces fairness at three layers, so a project burning through its allowance cannot starve quieter projects sharing the same EIMs.
Concurrent-request fairness
A project can carry a concurrent-request threshold. It is set per project; there is none by default. Projects with fewer requests in flight are favored in EIM selection. A project at or over its threshold is scheduled behind other projects' requests when contention is high. Its requests are never rejected for crossing the threshold. Flex requests always yield to standard and priority. Soft fairness keeps low-volume projects responsive without ever leaving capacity idle.
Rate-limit fairness
Above the soft layer, sliding-window rate limits apply per project, with a graduated grace zone: as you approach your limit, response headers begin signaling slowdown so you can throttle cleanly. Cross the hard limit and requests are rejected with standard HTTP semantics.
Spend fairness
Every request passes through an atomic credit ledger before it can be dispatched. Credit holds are taken at admission, settled on completion, and serialized in the database so two concurrent requests can never both spend the last dollar. Spend caps are enforced consistently across every endpoint the router exposes.
Anti-starvation
Idle projects are never forgotten. Long-idle projects keep their fair-share position when traffic resumes, and a periodic reaper sweeps stale request accounting so a leaked counter cannot keep a project pinned to the back of the line indefinitely.
Secure EIM Protocol
Once an EIM is chosen, the request travels over Erebine's inference mesh, a purpose-built protocol that handles encryption, framing, flow control, and lease management without any of it being your problem.
CurveZMQ encryption
Every router-to-EIM connection is mutually authenticated and
end-to-end encrypted with CurveZMQ. Each EIM is identified by
the fingerprint of its public key, so the router can refuse to
talk to an EIM whose identity it has not previously enrolled.
Keys are rotated on a schedule and held in protected memory on
the router side.
MessagePack framing
All control-plane messages use a compact, typed binary format
(MessagePack with a two-byte type prefix). The router can
dispatch on message type without decoding the body, which keeps
the hot path cheap. The choice of MessagePack over Python pickle
or other dynamic formats also closes off a well-known class of
deserialization vulnerabilities.
Lease-bound EIMs
EIMs do not connect once and trust forever. Each EIM holds a short-lived lease that it must renew on a heartbeat. If a lease lapses, the router moves the EIM into a grace state and probes it directly; if probes fail, the EIM is evicted and its in-flight work is rerouted. The result: a partitioned or hung EIM vanishes from the eligible pool in seconds, not minutes.
Credit-based streaming backpressure
Streaming responses are governed by a credit-based flow control
protocol layered on top of ZeroMQ high-water marks. Each stream
has a window of bytes the sender is allowed to emit; the
receiver replenishes the window as it consumes data. If the
client slows down, the credit window contracts and the EIM
naturally pauses generation rather than buffering unbounded
tokens upstream. Memory stays predictable even under SSE
backpressure.
sequenceDiagram
participant Router
participant Agent as EIM
Router->>Agent: CURVE handshake (mutual auth)
Agent->>Router: Lease renewal (heartbeat)
Router->>Agent: Dispatch (MessagePack, typed)
Agent-->>Router: Stream chunks (within credit window)
Router->>Agent: Replenish credits as client consumes
Note over Router,Agent: Client slow?
Window contracts, EIM pauses.
Resilience & Self-Healing
A production router has to assume things will break and keep serving traffic anyway. The router carries the usual primitives , and a few less-usual ones.
- Circuit breakers, Sensitive subsystems (such as memory extraction) sit behind per-endpoint breakers. Consecutive failures open the circuit and short-circuit further calls for a cooldown window, so a sick downstream cannot cascade into the request path.
- Lease probes, EIMs in the grace state are actively pinged with round-trip latency measured. EIMs that recover are reinstated; EIMs that do not are evicted cleanly.
- Tier fallback & spillover, Per-EIM queues have bounded depth. When a tier saturates, requests can be resumed on adjacent capacity instead of deadlocking against a single hot EIM.
- Cold provisioning, When no warm EIM holds your model, eligible candidates can be warmed and routed around until the model is ready; cold-load orchestration is handled by the EIM fleet, not the router request path.
- Self-healing prefix index, An EIM whose prefix predictions stop being accurate gets its slice of the index flushed and rebuilt automatically.
- Stale-request reaper, A background sweep evicts request accounting that has been pending too long, preventing leaked counters from corrupting fairness over time.
- Enrollment rate limiting, The EIM enrollment routes pass through a per-source token-bucket limiter so a misbehaving network cannot flood the control plane. Enroll-initiate and enroll-complete are auth-gated by the standard middleware chain; the refresh route is rate-limited and verifies its bearer inline rather than through middleware auth.
Observability
The router is built for operators, not just for end users. Every request is observable end-to-end, every decision is auditable, and the cost of that observability is bounded.
- Distributed tracing, Standards-compliant
W3C traceparentheaders are extracted on ingress and propagated to downstream EIMs, so a request can be followed across the router, the EIM, any tool calls, and back, in your existing tracing stack. - Prometheus metrics with cardinality protection, All key router decisions are exported as Prometheus metrics. A built-in cardinality limiter caps the number of unique tenant and region labels and folds overflow into a shared bucket, so a runaway label cannot blow up your monitoring backend.
- Structured audit log, Approval gates and admin actions (EIM management, admin API, storage-tier and cache controls, maintenance operations, and chat administration) are written to a structured audit log in the database, queryable for compliance reporting.
- Per-request usage accounting, Tokens in, tokens out, cached tokens, and (where applicable) research and reasoning tokens are returned in the response payload so you can attribute cost to features and users without scraping logs.
In Operator Terms
- You write to one OpenAI-compatible API. The router decides, per request, where it actually runs.
- Your prompts stay warm. Prefix-cache affinity, model affinity, and region affinity all stack, without you having to pin or tag anything.
- Your neighbors cannot starve you. Soft fairness, hard rate limits, and atomic credit holds keep noisy tenants from stealing your capacity.
- Failures are contained. Circuit breakers, lease probes, tier spillover, and a self-healing affinity index keep one bad EIM from poisoning the pool.
- You can see what happened. W3C tracing across every hop, cardinality-safe Prometheus metrics, and a structured audit trail.
Bring your own EIMs, use our managed pools, or mix the two behind a single project. The router treats them as one tier-aware fleet, and gives every request a fair, fast, observable home.