docs
// Getting Started

The Router

One OpenAI-compatible surface. Behind it, a dispatcher that places every request on an Erebine Inference Microservice (EIM). A prompt already warm on an EIM is scored on sixteen independent signals and sampled with A-Res weighted random; other requests follow the endpoint's standing placement or your weights, and are scored the same way when neither applies. Dispatch runs over a mutually-authenticated CurveZMQ mesh. Managed pools, self-hosted EIMs, and public cloud are one fleet.

Overview

Every Erebine inference request, chat, responses, embeddings, reranking, tool calls, arrives at the same router process that also serves the frontend page surface. The router holds a live picture of every connected EIM: which models are loaded, how hot their KV cache is, how recently each one answered, how close it is to your traffic, and whether the project sending the request is over or under its fair share.

How a request is placed depends on what the router already knows:

  • Pinned conversation. A chat completion sent with X-Conversation-Id goes back to the EIM that served the conversation's last request while that EIM can take it.
  • Warm prompt. A chat completion whose prompt prefix an eligible EIM already holds is scored on every signal across the eligible EIMs, and a winner is sampled with a weighted-random algorithm that still leaves room for exploration. Realtime turns are always scored this way.
  • Standing placement or weights. Any other request goes to the EIMs the endpoint is already placed on, or is split across the endpoint's EIMs by the weights you set.
  • Neither. A request with no standing placement and no weights is scored like a warm one.

Dispatch runs over an authenticated, encrypted mesh with built-in backpressure. The whole decision happens before your first token, typically in under a millisecond at the router itself.

Composite Scoring

Picking an EIM by latency alone is a trap, the fastest EIM a millisecond ago may be the one whose KV cache just evicted your prompt, the one whose queue just filled, or the one in a region a continent away. The router blends 16 distinct signals into a single composite score, and computes it fresh for every request it scores.

The sixteen signals

// affinity
Model-cache affinity
Is the model already loaded? Cold-loading a 70B model is measured in minutes; a warm EIM answers immediately.
// prefix
Prefix-cache match
How much of your prompt's KV cache the EIM still holds from a prior request. More overlap, shorter prefill, faster first token. A bonus tier stacks on top when overlap is high and the prefix-match prediction has proven accurate for that EIM.
// ewma
Latency prediction
A per-EIM, exponentially-weighted moving average of recent end-to-end latency. Weighted harder when an SLO is attached.
// pressure
KV-cache pressure
EIMs near their KV ceiling are deprioritized and, past a threshold, excluded entirely.
// health
Worker health
Heartbeat freshness, error rates, and probe results drive a continuous health score.
// slots
Worker utilization
How many slots remain available right now.
// region
Region affinity
Stay close to the caller when latency matters; spill out when capacity does.
// fairness
Per-project fairness
Projects under their fair share are favored; a project past its concurrent-request threshold is scheduled behind the rest, never rejected.
// tier
Service-tier priority
Priority requests outrank standard. Flex requests yield to both.
// ownership
Self-hosted ownership
If your project owns the EIM, your requests get first pass at it.
// slo
SLO bonus
EIMs consistently meeting your SLO win ties.
// origin
Internal vs. external
Internal traffic gets a configurable boost to keep system jobs from being starved by user traffic.
// batch
Batch penalty
Batch requests are weighted down so interactive traffic preempts them naturally.
// staleness
Metrics-staleness penalty
EIMs whose telemetry is overdue are progressively distrusted until they re-report.
// in-flight
In-flight token rate
Tokens still to be generated on an EIM's active requests, divided by its measured tokens per second. One long generation blocks more capacity than one short one, and raw slot counts cannot see that.
// deadline
Deadline tail risk
When a request carries a deadline and an EIM's predicted latency plus its tail spread would overrun it, that EIM takes a hard penalty. Fast median, fat tail loses.

Weighted Selection & Exploration

A single best score is rarely the best answer. Always picking the top-scoring EIM concentrates traffic, accelerates KV evictions on the favorite, and prevents the router from learning that a newly connected EIM is actually faster. The router avoids this in two ways.

A-Res weighted-random sampling

Instead of picking the maximum score, the router draws from all eligible EIMs using the A-Res algorithm (Efraimidis and Spirakis). Each EIM's selection probability is proportional to its composite score, so the best EIM wins most of the time. An EIM that is 90% as good still wins a meaningful share of requests. Load spreads naturally, KV caches stay warm across the pool, and a high scorer that briefly stumbles has competitors ready to absorb the traffic.

Epsilon-greedy exploration

A small slice of scored requests, five percent by default, bypasses the composite score entirely and chooses uniformly at random from the eligible pool. This keeps the router honest: newly connected EIMs, EIMs whose latency model is stale, and EIMs recovering from a degraded state all get sampled, measured, and folded back into the scoring model. The result is a router that keeps learning, instead of one that lock-steps to whatever was fastest an hour ago.

Prefix-Cache Affinity

The router keeps a per-EIM index of which prompt prefixes each EIM has recently served. When a chat completion or a realtime turn arrives, the router predicts, before dispatching, how much of the prompt's KV cache is likely already resident on each candidate, and prefers the candidates with the warmest match.

Self-calibrating accuracy

Every prediction is checked. After the request completes, the EIM reports how many prefix blocks it actually hit, and the router compares that to what it predicted. The per-EIM accuracy of those predictions is tracked with an exponential moving average and used to dampen or amplify future prefix-match scores. An EIM whose cache map turns out to be a poor predictor gets less benefit of the doubt; an EIM whose predictions are consistently right gets a strong-prefix bonus on top.

If accuracy collapses for any EIM, usually because its cache state diverged from what the router believed, the router resets that EIM's prefix index and rebuilds it from fresh evidence. No human intervention required.

The user-facing effect is simple: when your prompts share stable prefixes (a system prompt, a few-shot block, a long document quoted across many turns) the router quietly funnels those requests to the EIMs most likely to skip the prefill, without you having to pin or tag anything. See Prefix Caching for prompt-structure best practices.

Warm cache before weights

On an endpoint served by several EIMs you weight, the cache comes first. A request whose prompt prefix an attached EIM already holds is scored and goes to the warm EIM ahead of the weights, as does a conversation held on one EIM with X-Conversation-Id. The weights decide every request with no warm prefix. A drained EIM (weight 0) takes no new requests either way, and no request reaches an EIM the endpoint does not allow.

Per-Project Fairness

A shared router is only as fair as its weakest defense against a noisy neighbor. The router enforces fairness at three layers, so a project burning through its allowance cannot starve quieter projects sharing the same EIMs.

Concurrent-request fairness

A project can carry a concurrent-request threshold. It is set per project; there is none by default. Projects with fewer requests in flight are favored in EIM selection. A project at or over its threshold is scheduled behind other projects' requests when contention is high. Its requests are never rejected for crossing the threshold. Flex requests always yield to standard and priority. Soft fairness keeps low-volume projects responsive without ever leaving capacity idle.

Rate-limit fairness

Above the soft layer, sliding-window rate limits apply per project, with a graduated grace zone: as you approach your limit, response headers begin signaling slowdown so you can throttle cleanly. Cross the hard limit and requests are rejected with standard HTTP semantics.

Spend fairness

Every request passes through an atomic credit ledger before it can be dispatched. Credit holds are taken at admission, settled on completion, and serialized in the database so two concurrent requests can never both spend the last dollar. Spend caps are enforced consistently across every endpoint the router exposes.

Anti-starvation

Idle projects are never forgotten. Long-idle projects keep their fair-share position when traffic resumes, and a periodic reaper sweeps stale request accounting so a leaked counter cannot keep a project pinned to the back of the line indefinitely.

Secure EIM Protocol

Once an EIM is chosen, the request travels over Erebine's inference mesh, a purpose-built protocol that handles encryption, framing, flow control, and lease management without any of it being your problem.

CurveZMQ encryption

Every router-to-EIM connection is mutually authenticated and end-to-end encrypted with CurveZMQ. Each EIM is identified by the fingerprint of its public key, so the router can refuse to talk to an EIM whose identity it has not previously enrolled. Keys are rotated on a schedule and held in protected memory on the router side.

MessagePack framing

All control-plane messages use a compact, typed binary format (MessagePack with a two-byte type prefix). The router can dispatch on message type without decoding the body, which keeps the hot path cheap. The choice of MessagePack over Python pickle or other dynamic formats also closes off a well-known class of deserialization vulnerabilities.

Lease-bound EIMs

EIMs do not connect once and trust forever. Each EIM holds a short-lived lease that it must renew on a heartbeat. If a lease lapses, the router moves the EIM into a grace state and probes it directly; if probes fail, the EIM is evicted and its in-flight work is rerouted. The result: a partitioned or hung EIM vanishes from the eligible pool in seconds, not minutes.

Credit-based streaming backpressure

Streaming responses are governed by a credit-based flow control protocol layered on top of ZeroMQ high-water marks. Each stream has a window of bytes the sender is allowed to emit; the receiver replenishes the window as it consumes data. If the client slows down, the credit window contracts and the EIM naturally pauses generation rather than buffering unbounded tokens upstream. Memory stays predictable even under SSE backpressure.

Resilience & Self-Healing

A production router has to assume things will break and keep serving traffic anyway. The router carries the usual primitives , and a few less-usual ones.

  • Circuit breakers, Sensitive subsystems (such as memory extraction) sit behind per-endpoint breakers. Consecutive failures open the circuit and short-circuit further calls for a cooldown window, so a sick downstream cannot cascade into the request path.
  • Lease probes, EIMs in the grace state are actively pinged with round-trip latency measured. EIMs that recover are reinstated; EIMs that do not are evicted cleanly.
  • Tier fallback & spillover, Per-EIM queues have bounded depth. When a tier saturates, requests can be resumed on adjacent capacity instead of deadlocking against a single hot EIM.
  • Cold provisioning, When no warm EIM holds your model, eligible candidates can be warmed and routed around until the model is ready; cold-load orchestration is handled by the EIM fleet, not the router request path.
  • Self-healing prefix index, An EIM whose prefix predictions stop being accurate gets its slice of the index flushed and rebuilt automatically.
  • Stale-request reaper, A background sweep evicts request accounting that has been pending too long, preventing leaked counters from corrupting fairness over time.
  • Enrollment rate limiting, The EIM enrollment routes pass through a per-source token-bucket limiter so a misbehaving network cannot flood the control plane. Enroll-initiate and enroll-complete are auth-gated by the standard middleware chain; the refresh route is rate-limited and verifies its bearer inline rather than through middleware auth.

Observability

The router is built for operators, not just for end users. Every request is observable end-to-end, every decision is auditable, and the cost of that observability is bounded.

  • Distributed tracing, Standards-compliant W3C traceparent headers are extracted on ingress and propagated to downstream EIMs, so a request can be followed across the router, the EIM, any tool calls, and back, in your existing tracing stack.
  • Prometheus metrics with cardinality protection, All key router decisions are exported as Prometheus metrics. A built-in cardinality limiter caps the number of unique tenant and region labels and folds overflow into a shared bucket, so a runaway label cannot blow up your monitoring backend.
  • Structured audit log, Approval gates and admin actions (EIM management, admin API, storage-tier and cache controls, maintenance operations, and chat administration) are written to a structured audit log in the database, queryable for compliance reporting.
  • Per-request usage accounting, Tokens in, tokens out, cached tokens, and (where applicable) research and reasoning tokens are returned in the response payload so you can attribute cost to features and users without scraping logs.

In Operator Terms

  • You write to one OpenAI-compatible API. The router decides, per request, where it actually runs.
  • Your prompts stay warm. Prefix-cache affinity, model affinity, and region affinity all stack, without you having to pin or tag anything.
  • Your neighbors cannot starve you. Soft fairness, hard rate limits, and atomic credit holds keep noisy tenants from stealing your capacity.
  • Failures are contained. Circuit breakers, lease probes, tier spillover, and a self-healing affinity index keep one bad EIM from poisoning the pool.
  • You can see what happened. W3C tracing across every hop, cardinality-safe Prometheus metrics, and a structured audit trail.

Bring your own EIMs, use our managed pools, or mix the two behind a single project. The router treats them as one tier-aware fleet, and gives every request a fair, fast, observable home.