docs
// Infrastructure

EIM Deployment

An Erebine Inference Microservice (EIM): a Docker compose stack that runs vLLM behind the Erebine wire protocol on your NVIDIA GPU box. Your model, your hardware, your kernel. The router treats it as just another EIM once the join-key handshake succeeds.

image
ghcr.io/erebine/container-agents/eim-vllm-cuda:latest
compose
compose/compose.agent-nvidia.yaml
runtime uid:gid
5152:5152
enroll endpoint
POST /v1/enroll

Container images, compose files, and the macOS application all live in the public erebine/container-agents repository. The compose files are under compose/; images are published to ghcr.io/erebine/container-agents. Select the stack that matches your hardware:

Stack Image Compose File
NVIDIA CUDA (this page) ghcr.io/erebine/container-agents/eim-vllm-cuda:latest compose/compose.agent-nvidia.yaml
AMD ROCm ghcr.io/erebine/container-agents/eim-vllm-rocm:latest compose/compose.agent-amd-rocm.yaml
Intel GPU (OMIX) ghcr.io/erebine/container-agents/eim-vllm-omix:latest compose/compose.agent-intel-omix.yaml
AMD CPU (ZenDNN) ghcr.io/erebine/container-agents/eim-vllm-zendnn:latest compose/compose.agent-amd-cpu-zendnn.yaml
Intel / generic CPU ghcr.io/erebine/container-agents/eim-vllm-cpu:latest compose/compose.agent-cpu.yaml
Speech and image generation, NVIDIA CUDA ghcr.io/erebine/container-agents/eim-vllm-omni-cuda:latest Speech and Image Generation Image
Speech and image generation, AMD ROCm ghcr.io/erebine/container-agents/eim-vllm-omni-rocm:latest Speech and Image Generation Image
Speech and image generation, Intel GPU (OMIX) ghcr.io/erebine/container-agents/eim-vllm-omni-omix:latest Speech and Image Generation Image
Speech (Phonon), CPU ghcr.io/erebine/container-agents/eim-speech-cpu:latest Speech Image (CPU)
Speech (Phonon), NVIDIA CUDA ghcr.io/erebine/container-agents/eim-speech-cuda:latest Speech Image (CUDA)
System 1 decisions, CPU ghcr.io/erebine/container-agents/eim-laya-cpu:latest Laya Images
System 1 decisions, NVIDIA CUDA ghcr.io/erebine/container-agents/eim-laya-cuda:latest Laya Images
System 1 decisions, AMD ROCm ghcr.io/erebine/container-agents/eim-laya-rocm:latest Laya Images
Apple Silicon (macOS) native app (no container) EIM on macOS
AMD ROCm on Windows (WSL2) ghcr.io/erebine/container-agents/eim-vllm-rocm:latest compose/compose.agent-amd-rocm-wsl.yaml; see EIM on Windows

Prerequisites

Before deploying an EIM, ensure your infrastructure meets the following requirements.

Recommended Hardware

These values are advisory; the EIM does not enforce minimums at startup. Actual VRAM headroom is sized per-model by the EIM's VRAM estimator.

Component Advisory Minimum Recommended
GPU NVIDIA with 16GB VRAM NVIDIA A30/H100+ or RTX 3090+
System RAM 32GB 64GB+
Disk Space 100GB SSD 500GB+ NVMe SSD
Network 100 Mbps 1 Gbps+

One requirement is not advisory. On x86_64, every EIM and EEM image needs a CPU with the x86-64-v3 instruction set, AVX2, BMI2, FMA and MOVBE among it: the images are built on Red Hat Universal Base Image 10, whose C library does not start on an older CPU. Intel Core processors since Haswell and AMD processors since Excavator have it; some Pentium, Celeron and Atom models do not. A virtual machine must pass the flags through to the guest. grep -m1 -ow avx2 /proc/cpuinfo is a quick check: it prints avx2 on a CPU that has it.

Software Requirements

Software Version
Docker 24.0+
Docker Compose 2.20+
NVIDIA Driver 535+
NVIDIA Container Toolkit 1.14+

NVIDIA Container Toolkit Installation

Install the NVIDIA Container Toolkit to enable GPU access in Docker containers:

Ubuntu/Debian
distribution=$(. /etc/os-release;echo $ID$VERSION_ID) curl -fsSL https://nvidia.github.io/libnvidia-container/gpgkey | \ sudo gpg --dearmor -o /usr/share/keyrings/nvidia-container-toolkit-keyring.gpg curl -s -L https://nvidia.github.io/libnvidia-container/$distribution/libnvidia-container.list | \ sed 's#deb https://#deb [signed-by=/usr/share/keyrings/nvidia-container-toolkit-keyring.gpg] https://#g' | \ sudo tee /etc/apt/sources.list.d/nvidia-container-toolkit.list sudo apt-get update sudo apt-get install -y nvidia-container-toolkit sudo nvidia-ctk runtime configure --runtime=docker sudo systemctl restart docker

Verify GPU Access

bash
docker run --rm --gpus all nvidia/cuda:12.1-base-ubuntu22.04 nvidia-smi

Container Setup

EIM is distributed as a Docker image that bundles vLLM and the erebine-eim binary for model inference.

Deployment Paths

Path What it does
erectl deploy eim One command: hardware detection, data directories, kernel tuning, the compose stack, an optional systemd unit, the join key, the start and four verification gates. See erectl deploy.
compose by hand The recipe below, for hosts erectl cannot reach: pull the image, create the directories, set the environment, run compose.

An Erebine Execution Manager (EEM) coexists on the same host through erectl deploy eem, with its own deploy directory, unit, data directory and teardown.

Container Details

Property Value
Image ghcr.io/erebine/container-agents/eim-vllm-cuda:latest
Home Directory /var/lib/inference
Model Cache /var/lib/inference/.cache/erebine/models
Config Directory /var/lib/inference/.config/erebine

Pull the Image

bash
docker pull ghcr.io/erebine/container-agents/eim-vllm-cuda:latest

Intel GPU Image

ghcr.io/erebine/container-agents/eim-vllm-omix:latest runs vLLM on Intel Arc Pro B-series GPUs through the Intel Open Middleware Xe (OMIX) 0.4.1 driver stack, which the image carries. It is built for x86_64.

  • Host. A Linux kernel with the in-kernel xe driver bound to the GPU (lspci -k shows Kernel driver in use: xe), and the GPU firmware Intel requires for OMIX 0.4: GuC 70.65 or later from the host's linux-firmware, and the card's own firmware at release 775 or later. No Intel software is needed on the host.
  • Devices and groups. The container gets /dev/dri. Pass the host's render and video groups by number, as RENDER_GID and VIDEO_GID; the image has no render group, so the name fails.
  • Host memory. The kernel charges every Intel GPU buffer to host memory. Under the default vm.overcommit_memory=0 a single buffer larger than RAM plus swap is refused, and under vm.overcommit_memory=2 all of them share the host's commit limit. On a host with less RAM plus swap than GPU memory, set vm.overcommit_memory=1 (in a file under /etc/sysctl.d) to use the whole GPU; erectl deploy does this, erectl deploy eim --update does it on a host deployed before erectl did, and erectl doctor warns on a host where it is not 1. Otherwise the EIM lowers its GPU memory utilization until the KV cache fits under the limit, and logs why.
  • Model coverage. The image's attention kernels are compiled for a curated set of model families: Llama 2 and 3, Code Llama, Qwen 2, 2.5, 3, 3.5 and Qwen3-Next, Mistral, DeepSeek V2, V3 and R1, Kimi-VL, Falcon, gpt-oss, Gemma 3 and 4, GLM-4, Seed-OSS, Starcoder2, Phi-3.5, Phi-4 multimodal and VLM2Vec. Sliding-window attention for Qwen 2.5 and Llama 3.1 long-context windows is not included. For a model outside the set, the engine logs "kernel not compiled for this configuration" with the head size and either falls back to a much slower reference attention or stops.
  • GPU selection. Set ZE_AFFINITY_MASK (for example 0,1) to restrict the EIM to some of the host's GPUs. An index beyond the host's GPU count stops the GPU runtime.
  • Virtual machines. On a VM whose console is a virtio GPU, boot with drm_kms_helper.fbdev_emulation=0; without it, GPU enumeration can hang on the console framebuffer. On a VM with the GPU passed through, keep the GPU out of runtime suspend (write on to its power/control in sysfs, for example from a udev rule for devices bound to xe): a read of the GPU's firmware interface while it is suspended has paused such a VM. Mask fwupd.service, fwupd-refresh.service and fwupd-refresh.timer too, since fwupd's device scan makes that read.

Speech and Image Generation Image

ghcr.io/erebine/container-agents/eim-vllm-omni-cuda:latest is the NVIDIA CUDA image with vLLM-Omni 0.33.0rc1 added. It serves speech and image_generation endpoints and every task mode the CUDA image serves.

ghcr.io/erebine/container-agents/eim-vllm-omni-omix:latest is the Intel GPU (OMIX) image with the same vLLM-Omni release added. It serves speech and image_generation endpoints. It holds the transformers release vLLM-Omni requires, older than the one eim-vllm-omix holds, so it is for image and speech models; text models run on eim-vllm-omix.

ghcr.io/erebine/container-agents/eim-vllm-omni-rocm:latest is the AMD ROCm image with the same vLLM-Omni release added. It serves speech and image_generation endpoints and every task mode the ROCm image serves.

No CPU or Apple Silicon build of this image exists. Phonon speech models run on CPU in the Speech Image (CPU) and on NVIDIA GPUs in the Speech Image (CUDA).

  • Deploy. erectl deploy eim --task-mode image_generation (or speech) renders the EIM with the omni image for its hardware on nvidia-gpu, amd-gpu and intel-gpu; on cpu or amd-cpu, --task-mode speech alone renders the speech image instead. See erectl deploy.
  • Registration. The EIM registers speech and image_generation only when it finds vLLM-Omni installed. The router places an image generation endpoint only on EIMs that registered image_generation.
  • Models. Image models, their profiles and their settings are on Image Generation; the request API is on Images. Launch differences are on Image Generation Launches.

Speech Image (CPU)

ghcr.io/erebine/container-agents/eim-speech-cpu:latest serves speech endpoints for Phonon text-to-speech models on CPU, with no GPU and no vLLM. It runs ptts-openai-server 0.6.0, the open-source Phonon server, built from source. A CPU EIM serves Phonon faster than real time, so a speech endpoint needs no GPU, and the image leaves any GPU on the host alone.

  • Models. Phonon checkpoints only. The EIM registers speech for the model families the image carries, and the router places a Phonon model only on such an EIM. Other speech models run on the Speech and Image Generation Image.
  • Voices. The image carries no model weights and no voices; both arrive with the model's files. A voice pack imported into the model arrives the same way: the EIM syncs the new files and restarts the engine, which then reports the voices it loaded.
  • Deploy. erectl deploy eim --hardware-type cpu --engine native, or --task-mode speech alone, renders the EIM with this image (also on amd-cpu); it serves speech alone, so another task mode beside it is refused. See erectl deploy.
  • x86_64 CPUs. The image needs a CPU with the x86-64-v3 instruction set, AVX2 and FMA among it, as every image on Red Hat UBI 10 does: see Prerequisites.

Speech Image (CUDA)

ghcr.io/erebine/container-agents/eim-speech-cuda:latest serves speech endpoints for Phonon text-to-speech models on an NVIDIA GPU. It runs the same ptts-openai-server 0.6.0 as the Speech Image (CPU), built with CUDA 13.0, and carries the CUDA libraries the server loads. It is built for x86_64.

  • Host. An NVIDIA GPU of compute capability 8.0 (Ampere) or newer, an NVIDIA driver of release 580 or later, and the NVIDIA Container Toolkit, which gives the container the GPU and the host's driver library. The image has no CPU build: without an NVIDIA GPU the EIM refuses to start. Run the CPU image on such a host.
  • Models and voices. Phonon checkpoints only. The image carries no model weights and no voices; both arrive with the model's files, as on the CPU image.
  • GPU. Give the container the GPU it serves on, for example --gpus device=1 with docker run.
  • Placement. The GPU is opt-in. A Phonon endpoint runs here when you create it on a CUDA speech EIM (the Self-Hosted tab of the new-endpoint form picks the EIM) or when it requires a GPU (erectl endpoints create --required-gpu-count 1). Any other Phonon endpoint can land on a CPU or a GPU speech EIM, and the router prefers neither. The 31M-parameter model uses a small part of one GPU, so a MIG slice such as 1g.6gb on an A30 is the right size; without MIG the EIM holds a whole GPU.
  • Deploy. erectl deploy eim --hardware-type nvidia-gpu --engine native renders the EIM with this image in every GPU strategy, MIG included, and mounts no host CUDA toolkit: the server's kernels are PTX the driver compiles. --task-mode speech without --engine runs eim-vllm-omni-cuda instead. AMD and Intel GPUs have no speech image. See erectl deploy.

Laya Images

Three images run the Laya engine instead of vLLM. They serve System 1 decision endpoints only (task_mode: "systemone"; see System 1 Decisions) from any Laya-format checkpoint: the convaiinnovations/laya bundle or a fine-tune. Each installs every Python package from a hashed lock. The CPU and CUDA images are built for x86_64 and aarch64; the ROCm image is built for x86_64 only, because PyTorch publishes no ROCm build for aarch64.

Image Runs on Pins
ghcr.io/erebine/container-agents/eim-laya-cpu:latest Any x86_64 or aarch64 host, no GPU. AMD CPU hosts use this image. laya 0.4.30, PyTorch 2.14.1 (CPU), Python 3.12
ghcr.io/erebine/container-agents/eim-laya-cuda:latest NVIDIA GPUs, CUDA 13.0, the line the vLLM CUDA image uses. laya 0.4.30, PyTorch 2.14.1 (cu130), Python 3.12
ghcr.io/erebine/container-agents/eim-laya-rocm:latest AMD GPUs on x86_64 hosts: gfx908, gfx90a, gfx942, gfx950, gfx1030, gfx1100, gfx1101, gfx1102, gfx1103, gfx1150, gfx1151, gfx1200, and gfx1201. PyTorch has no kernels for other AMD GPUs. laya 0.4.30, PyTorch 2.14.1 (ROCm 7.14), Python 3.12

Each image sets EREBINE_EIM_ENGINE=laya. There is no compose file for them, and erectl deploy does not set them up; run the image with the volumes the compose stacks mount, giving the EIM its own model and config directories:

bash
sudo mkdir -p /data/erebine-laya/models /data/erebine-laya/config sudo chown -R 5152:5152 /data/erebine-laya docker run -d --name eim-laya-cpu --restart unless-stopped \ -v /data/erebine-laya/models:/var/lib/inference/.cache/erebine/models:Z \ -v /data/erebine-laya/config:/var/lib/inference/.config/erebine:Z \ -e EREBINE_EIM_JOIN_KEY=ejk_your_key_here \ ghcr.io/erebine/container-agents/eim-laya-cpu:latest

The :Z suffix gives each directory the container's SELinux label. Without it, on an SELinux host such as Rocky Linux, RHEL or Fedora, the EIM cannot write its enrollment state and never enrolls; Docker and Podman ignore the suffix where SELinux is off. With Podman, a --restart unless-stopped container comes back after a reboot only when podman-restart.service is enabled: sudo systemctl enable podman-restart.service.

For the CUDA image, add --gpus all, or --gpus device=1 to give it one GPU, and use the eim-laya-cuda image.

For the ROCm image, pass the GPU devices and the host's video and render groups by number (the image has no render group, so the name fails). To give it one GPU, add -e HIP_VISIBLE_DEVICES=1.

bash
docker run -d --name eim-laya-rocm --restart unless-stopped \ --device /dev/kfd --device /dev/dri \ --group-add "$(getent group video | cut -d: -f3)" \ --group-add "$(getent group render | cut -d: -f3)" \ -v /data/erebine-laya/models:/var/lib/inference/.cache/erebine/models:Z \ -v /data/erebine-laya/config:/var/lib/inference/.config/erebine:Z \ -e EREBINE_EIM_JOIN_KEY=ejk_your_key_here \ ghcr.io/erebine/container-agents/eim-laya-rocm:latest
  • Give a Laya EIM its own GPU: sharing one with a vLLM EIM is not supported.
  • No Intel GPU Laya image is built. On an Intel GPU host, eim-laya-cpu runs on CPU.
  • An image runs only on the GPU its PyTorch build drives: eim-laya-cuda on an AMD-only host, or eim-laya-rocm on an NVIDIA host, fails its launch with a non-retryable error. eim-laya-cpu runs on CPU on any host, with or without a GPU.
  • After enrollment, the EIM's details under System -> EIMs and EEMs show an Engine block: laya, the Laya version, systemone, and the device the model computes on.
  • Engine settings are on Advanced Configuration: Engines.

Enrollment Workflow

On first start, the EIM enrolls with the Erebine inference mesh using the join key. Enrollment state (including a signed EEM token) is persisted to /var/lib/inference/.config/erebine on the host-mounted config volume. On subsequent restarts the EIM reconnects automatically by exchanging its persisted refresh credential at /v1/enroll/refresh, so you can remove EREBINE_EIM_JOIN_KEY from your environment after the first successful enrollment.

If the config volume is wiped or credentials are rotated, re-supply a fresh join key from the dashboard. See Advanced Configuration for refresh and rotation details.

Environment Variables

Configure the EIM with environment variables. The following tables list the options needed for deployment. A model can state eleven vLLM launch settings of its own, its engine tuning; a variable or vLLM argument set on this EIM wins over the model's value (see Model Engine Tuning).

Earlier releases named every EIM variable EREBINE_AGENT_*; EREBINE_EIM_HOST_EIM_COUNT was EREBINE_AGENT_HOST_AGENT_COUNT. The old names still work as deprecated aliases. The EIM logs one warning at startup naming each old name it found, the new name wins when both are set, and an empty value counts as unset.

Required Variables

Variable Description
EREBINE_EIM_JOIN_KEY Join key for enrolling with the Erebine inference mesh. Obtain one with New EIM under System -> EIMs and EEMs. Required on first run only; after successful enrollment the EIM reconnects using its persisted EEM refresh credential and this variable can be removed.

EIM Configuration

Variable Default Description
EREBINE_EIM_MAX_CONCURRENT auto Optional ceiling for concurrent inference requests. When unset, the EIM auto-configures the ceiling from the resolved GPU count and model size.
EREBINE_EIM_OVERPROVISION_FACTOR 1.5 Admit slots per safe slot, at least 1.0. Safe slots are what the KV cache holds without preemption; the EIM accepts up to safe slots times this factor, capped by the ceiling above. EIMs serving an anchor are routed by safe slots.
EREBINE_EIM_LOG_LEVEL info Log level: trace, debug, info, warning, error
EREBINE_EIM_IMAGE_ACCELERATOR set by the image The accelerator the image's engine is built for: cuda, rocm, xpu, cpu or zendnn. Each vLLM image and each speech image sets it; leave it unset. A GPU image that finds no GPU it can use does not fall back to CPU: it reports itself failed, with the reason in its log, and takes no model. A CPU image runs on CPU even on a host with GPUs. Any other value stops the EIM at startup.

vLLM Configuration

The model the EIM serves is assigned from the Erebine dashboard during enrollment; there is no operator-supplied model id environment variable. The variables below tune the runtime around the assigned model.

Variable Default Description
EREBINE_EIM_MAX_MODEL_LEN unset Maximum sequence length. Defaults to the model's config.json max_position_embeddings when unset.
EREBINE_EIM_TENSOR_PARALLEL_SIZE 1 Tensor parallel size for multi-GPU. Must be set explicitly to use more than one GPU; there is no auto-detection from visible devices.
EREBINE_EIM_GPU_MEMORY_UTILIZATION 0.95 Fraction of each GPU's memory reserved for the engine (0.0-1.0). Set, it wins over a model's own gpu_memory_utilization; unset, a model's value applies, else 0.95.
SHM_SIZE 90g Docker Compose shm_size setting. Controls shared memory allocation for the container. Can be specified as a size string (e.g., 90g, 16g) or in bytes.
EREBINE_EIM_CUDA_VISIBLE_DEVICES unset (all visible GPUs) Specific GPU devices to use (comma-separated). When unset, the EIM defers to vLLM's device discovery, which uses all GPUs visible to the container.

Cache Configuration

Variable Default Description
EREBINE_EIM_MODEL_CACHE_PATH /var/lib/inference/.cache/erebine/models Local model cache directory
EREBINE_EIM_MODEL_CACHE_MAX_SIZE_GB 100 Maximum cache size in gigabytes

GPU Configuration

Configure GPU access and memory allocation. The defaults below assume one model per EIM and 95% memory utilization. A model may state its own GPU memory utilization; the variable, when set, wins on this EIM.

Single GPU Setup

.env
EREBINE_EIM_TENSOR_PARALLEL_SIZE=1 EREBINE_EIM_GPU_MEMORY_UTILIZATION=0.95 EREBINE_EIM_CUDA_VISIBLE_DEVICES=0

Multi-GPU Setup

For models that require multiple GPUs (tensor parallelism):

.env
EREBINE_EIM_TENSOR_PARALLEL_SIZE=2 EREBINE_EIM_GPU_MEMORY_UTILIZATION=0.95 EREBINE_EIM_CUDA_VISIBLE_DEVICES=0,1

Specific GPU Selection

To use specific GPUs (e.g., GPUs 2 and 3 on a 4-GPU system):

.env
EREBINE_EIM_TENSOR_PARALLEL_SIZE=2 EREBINE_EIM_CUDA_VISIBLE_DEVICES=2,3

Shared Memory Configuration

Larger models require more shared memory. Adjust SHM_SIZE in your .env file based on model size. The default of 90g is sufficient for most deployments:

Model Size Recommended SHM_SIZE
1-8B parameters 16g
13-34B parameters 32g
70B+ parameters 90g (default)

Model Storage

Configure persistent storage for downloaded models to avoid re-downloading on container restart.

Host Directory Setup

Create directories on the host for persistent model and configuration storage:

bash
sudo mkdir -p /data/erebine/models /data/erebine/config

Docker Compose with Volume Mounts

Fetch the compose file from the erebine/container-agents repository and start the EIM. The compose file mounts the host directories created above into the container so the model cache and enrollment state persist across restarts.

bash
# Clone the deployment repo (or download just the compose file you need) git clone https://github.com/erebine/container-agents.git cd container-agents/compose # Provide the join key (first run only) and start the NVIDIA EIM export EREBINE_EIM_JOIN_KEY=ejk_your_key_here docker compose -f compose.agent-nvidia.yaml up -d

Swap compose.agent-nvidia.yaml for compose.agent-amd-rocm.yaml, compose.agent-intel-omix.yaml, compose.agent-amd-cpu-zendnn.yaml, or compose.agent-cpu.yaml to match your accelerator. Each compose file documents its full environment-variable surface in header comments. On Windows under WSL2, an AMD GPU uses compose.agent-amd-rocm-wsl.yaml and an NVIDIA GPU compose.agent-nvidia.yaml; see EIM on Windows.

KV Cache Offload

The EIM ships with vLLM native CPU KV cache offloading enabled by default (a sliding share of system RAM, 25% on a 16 GiB host rising to 75% at 1 TiB). This reduces Time-to-First-Token (TTFT) for repeated prompt prefixes by spilling evicted KV blocks to host memory instead of recomputing them.

Tuning

See erebine-eim --help under --kv-offload-size-gb for tuning. Setting the environment variable EREBINE_EIM_KV_OFFLOAD_SIZE_GB=0 disables offload.

Verification

After starting your EIM, verify that it is running correctly.

Check Container Logs

bash
docker-compose logs -f agent

Successful enrollment emits a log line containing enrollment successful followed by the assigned EIM id. The EIM then transitions to ready and begins polling for work.

A log line containing GPU image found no GPU it can drive means the container cannot reach its GPU. The line names what each detection step found and the GPU devices the container can see. Start an NVIDIA container with its GPUs (--gpus or a CDI device), pass /dev/kfd and /dev/dri into an AMD container and /dev/dri into an Intel one, then restart it. Under WSL2 an AMD container takes /dev/dxg and /usr/lib/wsl/lib/libdxcore.so mounted at /usr/lib64/libdxcore.so instead (see EIM on Windows). The EIM shows as failed until then.

Confirm in the Dashboard

Open System -> EIMs and EEMs in the Erebine dashboard. The newly enrolled EIM appears with status online and reports its accelerator, assigned model, and last-seen timestamp. If the EIM does not appear within a minute of container start, recheck the join key and the /data/erebine/config volume permissions (UID/GID 5152).