docs
//

Two ways to serve inference: an Erebine Inference Microservice (EIM) you run on your hardware, or a shared EIM the platform runs on its own. Same router, same OpenAI-compatible surface, same dashboard. The choice is operational, not architectural.

Overview

When a request arrives at your endpoint, the router selects an EIM based on availability, capacity, and ownership.

  • Private EIMs: Owned and operated by you, running on your own infrastructure
  • Shared EIMs: Platform-managed EIMs available to all authorized users

Tip: You can use both EIM types together. Private EIMs handle your primary workloads while shared EIMs provide automatic fallback.

Quick Comparison

Aspect EIM (Private) Shared
Ownership User/Project Platform
Access Owner-only Any authorized user
Enrollment Join key required Pre-provisioned
Data isolation Single-tenant Multi-tenant
Control Full lifecycle management None
Cost User provides hardware Platform cost
Model caching Dedicated cache Shared pool
Fallback Can use shared as backup N/A

Tier Types

Every endpoint is assigned a service tier that determines pricing, limits, and scheduling priority. Rate limits are platform-wide, not per tier. The tier does not select hardware: one tier can hold NVIDIA, AMD, Intel, Apple and CPU EIMs side by side, and hardware eligibility is a per-endpoint requirement derived from the model, evaluated against each EIM's reported capability (see Hardware Requirements). The tier slug is used when configuring endpoints and when specifying supported_tiers on join keys or EIMs.

Slug Name Billing Tenancy Priority
free Free No charge Multi-tenant Lowest
bulk Bulk Per token, lowest metered rate Multi-tenant Lower
base Base Per token, entry interactive rate Multi-tenant Interactive
plus Plus Per token, mid-ladder interactive rate Multi-tenant Interactive
max Max Per token, top interactive rate Multi-tenant Interactive
self_hosted Self-Hosted Hourly per EIM, plus per uncached token Single-tenant Highest

Tier Descriptions

free

The entry-level tier for testing and evaluation: evaluation limits and the lowest scheduling priority. Projects on the free tier are capped on endpoints, on endpoints per task mode, and on uploaded models; a refused create names the cap. Unspent purchased credit or a live subscription removes the endpoint caps and the uploaded-model cap; free and referral credit does not, and the tier's max model size still applies.

bulk

Lowest metered rate. Scheduled behind the interactive tiers; built for volume and background work.

base

Entry metered rate for interactive traffic on the shared fleet.

plus

Mid-ladder metered rate for interactive traffic on the shared fleet. Same scheduling priority as base and max.

max

Top metered rate for interactive traffic on the shared fleet.

self_hosted

Bring-your-own-accelerator tier for private EIMs. You supply the hardware; the platform provides routing, monitoring, and management. Pricing is an hourly rate per EIM plus a per-token rate on the uncached tokens your EIMs serve; cached input tokens are free. Pricing lists its request timeout and max model size.

Note: Tier membership is explicit. An EIM serves exactly the tiers listed in its supported_tiers, inherited from the join key at enrollment; membership is not derived from the EIM's detected hardware. A GPU EIM enrolled against self_hosted serves GPU workloads for the endpoints bound to it; list a platform tier (for example plus) only if the EIM should also serve that tier's shared traffic. Tokens your own EIM serves bill at the Self-Hosted per-token rate, whatever tier the endpoint names.

SLA: No tier carries an uptime guarantee. The platform does not measure or enforce per-tier SLAs; a tier sets price, limits, and scheduling priority.

Private EIM Features

Private EIMs give you full control over your inference infrastructure while leveraging Erebine routing and management capabilities. One EIM drives one accelerator or many, and several EIMs can share a machine, so you decide how your accelerators are split.

Key Capabilities

  • Join key enrollment, Connect your hardware to the platform using secure, time-limited join keys created under System -> EIMs and EEMs in the dashboard or via the API. Join keys expire after at most 1 hour.
  • Full lifecycle management, Suspend, resume, and remove EIMs at any time. View logs and monitor health metrics in real time. EIMs transition through pending, active, disconnected, suspended, and dead states.
  • Region assignment, Assign EIMs to regions (free-form strings up to 24 ASCII characters, e.g. "us-east-1" or "datacenter-3") for geographic or logical organization. The region is inherited from the join key used at enrollment.
  • Dedicated model cache, Private EIMs maintain a private model cache not shared with other users, providing faster loading and data isolation.
  • Storage limits, Optionally configure a storage limit in bytes per EIM via storage_limit_enabled and storage_limit_bytes on the join key or EIM record. When disabled, EIMs can use unlimited storage up to disk capacity.
  • Tier support, Each EIM declares which service tiers it serves via supported_tiers, inherited from the join key at enrollment. Membership is explicit and not derived from the EIM's detected accelerator; an EIM with no listed tiers serves only the endpoints bound to it.

For detailed setup instructions, join key management, enrollment procedures, monitoring, and API examples, see the Private EIMs & Join Keys guide.

Shared EIM Features

Shared EIMs are platform-managed infrastructure available to all users.

No Enrollment Needed

Shared EIMs are pre-provisioned and ready to use. When you create an endpoint, it can immediately route to shared EIMs without any setup.

Platform-Managed

Erebine handles all operational aspects:

  • Provisioning and scaling
  • Health monitoring and recovery
  • Software updates and patches
  • Hardware maintenance

Multi-Tenant with Isolation

Shared EIMs serve requests from multiple projects. Request isolation is enforced through process-level sandboxing, memory isolation, network segmentation, and request authentication.

Security: Even on shared EIMs, your requests and data are isolated from other users. Each request runs in a sandboxed environment with memory cleared between requests.

Routing Behavior

The Router uses the following logic to select an EIM for each request:

  1. Hardware requirements gate: EIMs that do not satisfy a requirement declared on the endpoint (an architecture class or NVLink, with the model's VRAM floor and GPU count that come with either) are excluded before any preference ordering, regardless of tier membership. Whether the model fits an EIM's memory is checked per EIM when the EIM is bound and when the endpoint is placed on it
  2. User-owned EIMs take precedence: If you have active EIMs enrolled in the requested tier, they are used first
  3. Automatic fallback: When all your EIMs are busy or offline, requests can fall back to shared EIMs
  4. Regional affinity: Requests prefer EIMs in the same or nearby regions
  5. Load balancing: Multiple EIMs of the same type are load-balanced using a fair-queue scheduler with latency prediction
  6. Heartbeat freshness: EIMs whose last heartbeat is older than 30 seconds are treated as disconnected and excluded from routing

Note: Fallback to shared EIMs must be enabled in your endpoint settings. When disabled, requests will queue until an EIM becomes available or the queue timeout (30 seconds) is reached.

Frequently Asked Questions

Can I use both EIM types simultaneously?

Yes. You can have private EIMs for your primary workloads while using shared EIMs as a fallback. Configure this in your endpoint settings under "Fallback Options."

What happens if my EIM goes offline?

When an EIM disconnects, requests are routed to other available EIMs. If no EIMs are available and fallback is enabled, requests go to shared EIMs. If fallback is disabled, requests are queued until an EIM becomes available. See EIM Lifecycle for details.

How do I migrate from shared to private EIMs?

Generate a join key with New EIM under System -> EIMs and EEMs or via the API, deploy an EIM on your infrastructure, and start it with the join key. See the EIM Enrollment guide for step-by-step instructions.

Are my requests visible to other users on shared EIMs?

No. Even on shared EIMs, requests are isolated. Each request runs in a sandboxed environment, memory is cleared between requests, network access is restricted to your project, and logs and metrics are project-scoped.

What hardware do I need for self-hosting?

Requirements depend on your model and workload. GPU EIMs need an NVIDIA GPU with CUDA support, an AMD GPU with ROCm support, or an Intel GPU with oneAPI support. CPU EIMs need a modern x86_64 CPU. You also need sufficient RAM and disk space to load your models, a stable network connection, and a Linux environment compatible with vLLM.

Can I run multiple EIMs?

Yes. Running multiple EIMs provides redundancy (requests continue if one EIM fails), increased throughput (handle more concurrent requests), and geographic distribution (deploy EIMs closer to your users).

Does a higher tier mean different hardware?

No. A tier sets price, limits, and scheduling priority. Any tier can hold any mix of NVIDIA, AMD, Intel, Apple, and CPU EIMs, and one endpoint can be served by that mix. Hardware needs come from the model and the endpoint: the router checks per EIM that the model fits (GPU memory on a GPU EIM, parameter count on a CPU EIM), plus any architecture or NVLink requirement declared on the endpoint. On the platform tiers the GPU is shared among multiple projects with software-level isolation; for single-tenant capacity, run your own EIMs on self_hosted or configure exclusive binding on the endpoint.

How does /v1/enroll/refresh authenticate?

The refresh endpoint validates the rotation token presented in the request body rather than running through the standard authentication middleware. See Authentication for the full token-rotation flow.

What tier slug should I use when configuring a self-hosted EIM?

Use self_hosted as the tier slug for private EIMs that you enroll yourself. You can also set supported_tiers on the join key or EIM to allow the EIM to serve requests from additional tiers.