docs
// Tools

erectl deploy

One command from a bare Linux host to enrolled Erebine Inference Microservices (EIMs) or an enrolled Erebine Execution Manager (EEM). erectl deploy eim and erectl deploy eem detect the container runtime (and, for EIMs, the hardware), mint a join key, generate the compose stack, optionally install systemd units, start the EIMs or the EEM, and verify enrollment end to end. With --runtime baremetal they install the EIM or EEM binary and run it under systemd instead.

Overview

The manual recipe in the container-agents compose files -- directories, sysctl, join key, the right file for your hardware -- is fully automated:

bash
sudo erectl deploy eim \
  --auto-hardware-detect --count 2 --service \
  --region us-east

The command plans first, shows you every host mutation it will make, and asks before proceeding. --dry-run prints the plan and every command it would run without touching the host, and is the one mode that does not need root. Re-running with the same flags converges: directories are kept, the compose stack reconciles, and a still-valid join key is reused rather than re-minted.

Kinds

The kind is required. An EIM stack and an EEM are different host mutations, and a root-privileged command does not guess between them: erectl deploy with no kind prints the two kinds, and the same flags with no kind in front of them are a usage error rather than a default.

KindDeploys
eimN EIMs running vLLM on GPU or CPU. Hardware detection, GPU strategies, MIG orchestration, kernel tuning.
eemOne EEM running tools. No GPU, no kernel tuning; it serves workspaces rather than models.

Both kinds coexist on one host. Each keeps its own deploy directory, systemd unit, data directory and active-deploy pointer, so deploying one never touches the other's stack, and each has its own --down and --update. Pointing --deploy-dir at the other kind's directory is refused rather than crossed.

A join key carries the role it was minted for. A key minted for an EIM cannot deploy an EEM or the reverse: erectl refuses locally before touching the host, and the router refuses the enrollment too.

Prerequisites

  • Linux with systemd, and root (via sudo).
  • podman (with podman-compose) or docker (with the compose v2 plugin). podman is preferred when both are present; override with --runtime. With neither, the command stops and points at --runtime baremetal; it never switches runtime on its own. On podman, also skopeo: the registry pre-flight checks the image with skopeo inspect before anything changes, and stops without it. docker needs none; it uses docker manifest inspect.
  • For eim on NVIDIA GPUs: the NVIDIA Container Toolkit. The command verifies it before changing anything. eem needs no GPU and no toolkit.
  • An erectl configuration with a management-scope API key (see erectl setup). Under sudo, root's configuration is the one that is read; pass --base-url and --api-key explicitly if root has none. A join key you bring with --join-key-file does not replace it: the command lists the project's join keys before it changes anything, and its verification gates read the enrollment and the router's list of connected EIMs or EEMs through the same API.
  • No EIM or EEM of the same kind already running under another unit. Every deploy, container or bare metal, refuses while any enabled or running systemd unit other than its own runs that kind's binary (erebine-eim or erebine-eem, under either that name or its earlier one, erebine-eim-agent or erebine-eem-agent), whatever the unit is called -- the deb and rpm packages ship such units -- because two instances of one kind on one host would enroll twice. systemctl disable --now the unit the error names and re-run.

Quick Start

Minting a join key needs one thing: the region. For eim the tier defaults to self_hosted, which makes the EIMs user-owned -- serving your project, not shared platform traffic; pass --tier to serve other tiers (list the valid slugs with erectl tiers). The router address defaults to tcp://<host>:5555 derived from your configuration's base_url; pass --router-addr when the router's mesh endpoint lives elsewhere. Alternatively, bring your own key with --join-key-file (or - for stdin) and skip minting entirely; the API key from Prerequisites is still needed.

bash
# Detect hardware, deploy one user-owned EIM, no systemd unit:
sudo erectl deploy eim --auto-hardware-detect --region us-east

# Preview only (no root needed):
erectl deploy eim --auto-hardware-detect --dry-run --region us-east

# Serve a specific tier instead of the self_hosted default:
sudo erectl deploy eim --auto-hardware-detect \
  --region us-east --tier plus

# Mesh endpoint on a different host than the API:
sudo erectl deploy eim --auto-hardware-detect \
  --region us-east --router-addr tcp://mesh.example.com:5555

One EEM, with a systemd unit so it comes back after a reboot:

bash
# Deploy, enroll and verify one EEM:
sudo erectl deploy eem --service --region us-east

# Preview only (no root needed):
erectl deploy eem --dry-run --region us-east

# Name the registration and the project explicitly:
sudo erectl deploy eem --service --region us-east \
  --registration-name eem-fleet-01 --project proj_ABC123

An omitted --project defaults to the project your base_url is scoped to; an omitted --registration-name defaults to eem-<hostname>, which must be unique within the project.

Join keys expire after one hour, so the command pulls images first and mints as late as possible. After a verified deploy it prints the exact command to revoke the key -- for both kinds, enrollment state persists in each instance's data directory, so the key is only needed once.

Bare Metal

No container runtime on the host, or no reason for one: --runtime baremetal installs the EIM or EEM binary and runs it under systemd, behind the same unit names, lifecycle flags and verification gates as a container deploy. It is never chosen automatically.

bash
# EIMs on the host, running your vLLM:
sudo erectl deploy eim --runtime baremetal --service \
  --auto-hardware-detect --region us-east --vllm-path /opt/vllm

# One EEM:
sudo erectl deploy eem --runtime baremetal --service --region us-east

--service is required: systemd is the only supervisor a bare-metal EIM or EEM has. Linux on x86_64 only, the one architecture every release channel publishes.

Install method

The release defaults to latest, the same default the container path uses for its image tag; pin one with --image-version 2.4.0. Bare metal needs release 2.4.0 or newer: an older pinned release is refused, and so is a brew tap that still pins one. The binary comes from the first of these that applies, or from --install-method:

  1. Already installed at the wanted version: reused, nothing downloaded.
  2. deb on hosts with apt, rpm on hosts with dnf, yum or zypper.
  3. brew, the erebine/tap/erebine-eim or erebine/tap/erebine-eem formula, run as the account that owns Homebrew. latest only: the tap pins one version. erectl installs and upgrades by the full name, which Homebrew takes as trusting the formula, so a host an earlier erectl set up under erebine-eim-agent or erebine-eem-agent moves to the new name on its next deploy. By hand, upgrade by the full name or run brew trust erebine/tap once.
  4. binary, the release asset, installed to /usr/local/bin as erebine-eim or erebine-eem, with the earlier name, erebine-eim-agent or erebine-eem-agent, beside it as a symlink. The brew formulas install the same pair; the deb and rpm packages keep the earlier name.

Every download lands in a fresh directory only root can reach, under /var/cache/erebine (a /var/cache/erebine that is not a root-owned directory only root can write is refused, not repaired), and is checked against the sha256 digest the release publishes before anything installs it; a mismatch stops the deploy, and the directory is removed either way. The release binary loads the distribution's libzmq and libzstd, and the EEM's libcurl too; the deb, rpm and brew installs pull them in, the release binary does not. For that method the deploy reads the loader cache (ldconfig -p) before anything changes and refuses while any is missing, naming the packages to install; a dry run shows the same finding as a warning. After the install the binary still has to start, so a library the cache could not vouch for fails the deploy with the packages to install instead of failing the first boot.

On RHEL, Rocky Linux and AlmaLinux zeromq and its libsodium come only from EPEL, for the rpm and for the release binary alike: enable it (dnf install epel-release) before the deploy, or the rpm install stops on unresolved dependencies and the release binary cannot load. The plan says so on those hosts.

The deb and rpm packages ship units of their own that run the same binary. The deploy never enables them, and refuses while any unit running the same binary is enabled or running, as every deploy does (see Prerequisites); it checks again after installing a package.

vLLM

The EIM runs your vLLM; erectl never installs Python. It uses vllm on PATH, or --vllm-path: the executable, or the Python environment that holds bin/vllm. Under sudo PATH is root's, so pass the flag. The version is read from the environment without starting Python. A version other than the one this release is built and tested with (vLLM 0.33.0) is a plan warning; no vLLM refuses the deploy.

The deploy writes the EIM images' vLLM and LMCache launchers into <deploy-dir>/bin, bound to your environment's interpreter, so a bare-metal EIM keeps the image's offline defaults and usage accounting. Nothing is written into your environment. Engine settings that belong to a particular build -- a tcmalloc LD_PRELOAD on CPU, ROCm tuning flags, CUDA_HOME for a host CUDA toolkit -- are yours to set in agents.local.env.

The EIMs' account must be able to read the environment. A venv under a home directory closed to other users is refused by name; /opt is the usual place.

Units and instances

erebine-agents.service is still the unit you operate. On bare metal it starts one erebine-agents@<n>.service per EIM, and each instance stops and restarts with it: systemctl stop erebine-agents stops every EIM, and journalctl -u 'erebine-agents@*' -f follows them all. The EEM runs as erebine-eem.service.

A deploy clears any failed instance, enables the stack unit and restarts it when it is not running or when a generated file or the binary changed; otherwise it starts only what is stopped, such as an instance that had failed, and leaves the running EIMs alone. It then fails unless the unit is active. Over a running stack, --restart restarts it regardless and --no-restart leaves the running EIMs as they are. The header comment erectl writes at the top of each file does not count as a change, so re-running with the flags in another order restarts nothing.

What a container gives each EIM, each instance gets explicitly:

ResourceBare metal
Accounterebine-eim (uid 5152) or erebine-eem (uid 5153), created when absent. The uids are the container images', so the data directories need no re-chown.
Home and state<data-dir>/agent<n>, the only directory the EIM can write. Models and the LMCache disk tier stay in agent<n>/models and agent<n>/lmcache, where a container deploy keeps them.
PortsMetrics on 127.0.0.1:9100+n; LMCache, when enabled, on 6555+n and 8090+n. The EEM keeps 127.0.0.1:9095 for metrics and health, so both kinds share a host without a clash.
DevicesThe GPU strategy's plan through CUDA_VISIBLE_DEVICES or HIP_VISIBLE_DEVICES; a MIG EIM's slice is looked up at every start.
/tmpPrivate to each instance.
Memory limitsLocked memory and address space unlimited (LimitMEMLOCK=infinity, LimitAS=infinity), where a container gets an unlimited memlock ulimit.
Priority limitLimitNICE=-4: the Swift runtime starts some worker threads at nice -2, which an account without CAP_SYS_NICE may do only under a nice limit that reaches it. It is a limit, not a capability.
CredentialsThe EEM's --credentials-dir is owned by root, readable by the EEM's group, and read-only in its unit, as the container mounts it.

EEM tools on bare metal

The EEM runs every tool as its own account, erebine-eem, inside a hardened unit (NoNewPrivileges=yes, a read-only system). It advertises only what that account can actually run, so four families need the operator's say-so:

  • Container tools (container_ps and the rest) need a runtime the account can use, checked with <runtime> info when the EEM starts. Rootless podman does not work in this unit, so podman on the host is not enough and the tools are not advertised. To serve them, give the account the docker socket and restart the EEM: sudo systemctl edit erebine-eem, add [Service] and SupplementaryGroups=docker, then sudo systemctl restart erebine-eem. Membership in the docker group is equivalent to root on that host.
  • linux_journalctl_read sees only the EEM account's own messages. To read other units' logs, add systemd-journal to the same SupplementaryGroups= line; the account can then read every unit's log, including anything a service writes there.
  • CRI and libvirt tools (crictl_ps, crictl_inspect, virsh_list) need the node's container runtime socket and the host's libvirt, checked with crictl info and virsh -c qemu:///system list when the EEM starts. On a stock Kubernetes node the CRI socket that /etc/crictl.yaml names (/run/containerd/containerd.sock, or /var/run/crio/crio.sock for CRI-O) is readable by root only, so the crictl tools are not advertised. To serve them, give the socket a group (for containerd, set gid under [grpc] in /etc/containerd/config.toml to that group's id and restart containerd) and add the group to the same SupplementaryGroups= line. For virsh_list, add libvirt. Either grant controls every container or virtual machine on the host.
  • dmesg needs to read the kernel log, checked when the EEM starts. Most distributions close the kernel ring buffer to unprivileged accounts (kernel.dmesg_restrict = 1, the default on Rocky Linux 10, RHEL 10 and Debian 12), so dmesg is not advertised unless the account can read the system journal. With systemd-journal on the same SupplementaryGroups= line, dmesg reads the current boot's kernel messages from the journal instead, in journalctl's line format, and a workspace that sets allowed_commands needs journalctl in the list for it.

erectl adds none of these groups for you. The checks run once per start, so after changing the host restart the EEM.

Units run hardened: read-only system, private /tmp, no new privileges. On AMD GPUs the EIMs join the host's video and render groups -- only the ones the host has. Device selection is by those variables, not a device cgroup, so it is not the isolation a pinned container gives.

Everything else is the container path's: sysctl and transparent hugepages, MIG carving and its boot unit, the device-readiness gate, the verification gates, --no-start, --update, --down and --revoke-key. The in-container device check is skipped: there is no container to look into. --update installs a newer release through the method the deploy used, the way a container update pulls a newer image, and --down keeps the installed binary and the account, as it keeps images. To change runtime, tear the running deployment down first; a deploy over it with the other runtime is refused.

Hardware Types

eim only; an EEM runs no model and needs no accelerator.

TypeImageDetected by
nvidia-gpueim-vllm-cudanvidia-smi lists a GPU
amd-gpueim-vllm-rocm/dev/kfd with a compute node; under WSL2, /dev/dxg and an amd-smi installed in the distribution that lists an AMD GPU (see Windows (WSL2))
intel-gpueim-vllm-omixa discrete GPU bound to the xe driver
amd-cpueim-vllm-zendnnAMD CPU, no GPU
cpueim-vllm-cpueverything else

Pass --hardware-type to skip detection. The plan output always states what was detected and why.

Speech and image generation endpoints run on vLLM-Omni on a GPU, and Phonon speech also runs on a CPU. Name the task modes the EIMs will serve with --task-mode (repeatable): speech or image_generation runs the omni image for the hardware: eim-vllm-omni-cuda on nvidia-gpu and eim-vllm-omni-rocm on amd-gpu, which serve every other vLLM task mode too, and eim-vllm-omni-omix on intel-gpu. On cpu or amd-cpu, --task-mode speech alone runs eim-speech-cpu, the speech image for Phonon models, which serves speech and nothing else; speech beside any other task mode there is refused. image_generation is refused on cpu and amd-cpu. On bare metal --task-mode speech and --task-mode image_generation are refused: install vllm-omni into the --vllm-path environment instead, and the EIM registers both modes when it finds it. --update keeps the image the deploy chose.

--engine native runs Phonon speech on the native engine instead of vLLM: eim-speech-cuda on nvidia-gpu, in every GPU strategy, and eim-speech-cpu on cpu and amd-cpu. It implies --task-mode speech and refuses any other task mode. amd-gpu and intel-gpu have no native engine build and are refused; on intel-gpu, --task-mode speech runs eim-vllm-omni-omix for the vLLM-Omni speech models, and either host can serve Phonon from its CPU with --hardware-type cpu or amd-cpu. Container deploys only. eim-speech-cuda needs no CUDA toolkit on the host, so the deploy mounts none. See Speech Image (CPU) and Speech Image (CUDA).

bash
# One Phonon EIM on an NVIDIA GPU:
sudo erectl deploy eim --hardware-type nvidia-gpu --engine native \
  --region us-east --service

GPU Strategies

eim only. With --count above one on a GPU host, --gpu-strategy picks the shape:

  • pinned (default): one GPU per EIM, tensor parallelism 1. No GPU-to-GPU interconnect needed -- the right shape for VFIO passthrough and for models that fit a single GPU. Higher aggregate throughput than one tensor-parallel group.
  • shared (NVIDIA only): every EIM sees every GPU with memory split evenly. For many small models per board; EIMs contend for the same devices.
  • mig (NVIDIA only): hard-partitioned MIG slices, one per EIM. Deterministic startup and real isolation; see below.

A single EIM (--count 1) always uses the canonical single-EIM shape with tensor parallelism derived from every visible GPU, whatever the strategy -- except mig, which always pins its slice.

When --count exceeds the GPUs on the host and no strategy was passed, the default falls back from pinned to shared automatically: all EIMs run against the available GPU(s) with vLLM memory limits split evenly (0.90/count each) -- multiple EIMs per GPU with no MIG required. The plan labels the fallback; an explicit --gpu-strategy pinned refuses instead.

MIG Orchestration

eim only. Compose files cannot create MIG instances; erectl can. --mig (shorthand for --hardware-type nvidia-gpu --gpu-strategy mig) with --mig-profile handles the whole lifecycle:

bash
sudo erectl deploy eim --mig --mig-profile 2g.12gb --count 2 --service \
  --region us-east
  • Verifies the boards support MIG and the requested profile fits, per GPU, before changing anything.
  • Adopts an existing matching layout as-is; refuses to destroy a different one without --mig-repartition.
  • Enables MIG mode where needed (never resetting a busy GPU), carves the instances, and regenerates the CDI spec.
  • Installs a boot service that re-creates the layout: MIG instances never survive a reboot, and on Hopper and newer boards MIG mode itself does not either. This is why mig requires --service.

Profile names are per board -- run nvidia-smi mig -lgip to see yours. Use the name (like 2g.12gb), never the numeric ID.

Kernel Settings

eim only; an EEM leaves the kernel alone. Every inference deploy ensures three settings in /etc/sysctl.d/99-erebine.conf and applies the file:

SettingValue
net.core.rmem_max, net.core.wmem_maxAt least 4194304
vm.max_map_countAt least 262144

Each is only ever raised. A value already higher in the file is kept, and vm.max_map_count is written only while the host runs below 262144: a host that runs higher, such as an Elasticsearch host at 1048576, is left alone, and the file never carries a value that would lower it. The plan shows the current value and the target, or that the host is already at or above it. The command warns when a file that sorts after 99-erebine.conf sets a lower value, because that file wins at boot. --no-sysctl skips the step on the single-EIM shapes that raise the buffers themselves; --down keeps the file.

On an Intel GPU host (--hardware-type intel-gpu) the deploy also writes vm.overcommit_memory = 1 to /etc/sysctl.d/99-erebine-intel-gpu.conf and applies it; a file that already holds it is not rewritten. The kernel charges every Intel GPU buffer to host memory, and under its default setting a single buffer larger than RAM plus swap is refused, so on a host with less RAM plus swap than GPU memory the EIM could not allocate its KV cache. With mode 1 every process may overcommit, so real memory exhaustion ends in the OOM killer instead of a refused allocation. --no-sysctl skips this file too; the EIM then lowers its GPU memory utilization to fit. --down keeps the file. The file sorts before 99-erebine.conf and 99-sysctl.conf, so a line in either, or in any file that sorts after it, wins at boot: when such a line sets another value, the command names the file and says to remove the line or set it to 1. A vm.overcommit_memory = 1 line you added to 99-erebine.conf yourself is kept.

Transparent Hugepages

eim only; an EEM leaves the kernel alone. Every inference deploy sets the host's transparent hugepage mode to madvise for both enabled and defrag, so hugepages and their compaction stalls are confined to memory that asks for them. The runtime value takes effect immediately, and the command makes it persistent:

  • Debian and Ubuntu with GRUB: adds transparent_hugepage=madvise to the kernel command line through a drop-in, /etc/default/grub.d/99-erebine-thp.cfg, then runs update-grub. /etc/default/grub is never edited.
  • RHEL, Rocky Linux, AlmaLinux, CentOS, and Oracle Linux: adds the same argument to every installed kernel with grubby.
  • Both: /etc/tmpfiles.d/99-erebine-thp.conf restores defrag at boot, because no kernel argument sets it.

A new kernel argument applies at the next reboot. The command says when one is needed and never reboots the host. Anything it will not change it reports with the exact commands to run, and the deploy continues: inside a container, on image-mode hosts (rpm-ostree kargs), on other distributions, or when a different transparent_hugepage= value is already configured. When the active tuned profile sets its own transparent hugepage value, the command warns and names the profile change that stops the override. Re-running changes nothing that is already in place; until the reboot, the Debian path runs update-grub again so an interrupted run is never mistaken for a finished one.

Pass --no-thp to skip the step on hosts that manage kernel arguments elsewhere.

Windows (WSL2)

On a WSL2 distribution the command detects WSL (a microsoft kernel, WSL's interop entry, or a WSL session variable) and deploys the shape each GPU needs there. Every GPU under WSL2 is WSL's /dev/dxg; there is no /dev/kfd or /dev/nvidia*, and the /dev/dri nodes WSL2 has are not the GPU. The plan says WSL was detected and what it changes. See EIM on Windows for the Windows side and the compose route.

TypeUnder WSL2
nvidia-gpuThe NVIDIA container toolkit's WSL mode, one device nvidia.com/gpu=all. The unit waits for /dev/dxg. More than one EIM takes --gpu-strategy shared; --mig and a pinned layout of more than one EIM are refused.
amd-gpu/dev/dxg only, /usr/lib/wsl/lib/libdxcore.so mounted read-only at /usr/lib64/libdxcore.so, unprivileged, PYTORCH_ALLOC_CONF=expandable_segments:False in agents.env. The unit waits for /dev/dxg and libdxcore.so. One EIM; --gpu-link and --no-sysctl are refused.
intel-gpuRefused.
cpu, amd-cpuUnchanged.

Before it changes anything, an AMD deploy checks that /dev/dxg exists, that WSL has mounted /usr/lib/wsl/lib/libdxcore.so, and that the kernel's release names both microsoft and WSL, and refuses with what to do: wsl --update, the current AMD driver, then wsl --shutdown. --service is refused while systemd is not running, with the /etc/wsl.conf lines that turn it on. --auto-hardware-detect finds an AMD GPU when ROCm is installed in the distribution and its amd-smi lists the GPU; otherwise pass --hardware-type amd-gpu.

sudo erectl deploy eim --hardware-type amd-gpu --service --region us-east

A host deployed with an earlier erectl has a unit that waits for /dev/kfd and restarts every two minutes, and its EIM has never started. Run the same deploy command again with this erectl. It rewrites the compose file, agents.env and the unit's device wait, and mints a fresh join key when the recorded one has expired (keys live one hour). --update rewrites the same files but reuses the recorded key, so it suits only a deploy less than an hour old.

EEM Options

The EEM has no hardware to detect and no kernel to tune. What it needs instead is an identity in the project and the directory it reads credentials from; its tools are compiled into the EEM:

FlagMeaning
--registration-nameThe EEM's name, unique per project (default eem-<hostname>)
--projectProject the EEM reports to (default: the project the base_url is scoped to). Refused under --update, which re-renders the recorded project
--credentials-dirCredential bundles, mounted read-only (default <data-dir>/credentials)
--data-dirData root holding credentials, state and config (default /data/erebine-eem)
--domainsDomain the EEM may serve (repeatable; default every domain its tools declare)
--disable-domainsDomain the EEM must not serve (repeatable)
--workspaceWorkspace the EEM declares, by its name in the project (repeatable). Refused under --update and --down
--refresh-unitBare metal: rewrite the installed erebine-eem.service with this erectl and restart on change, keeping everything else (see Upgrading a bare-metal EEM)

The three data subdirectories are created and handed to the EEM's own uid, so nothing in them ends up owned by root. On bare metal the credentials directory stays root's instead, readable by the EEM's group: without a container to mount it read-only, ownership is what keeps the EEM from writing it. The EEM writes its signing key, its CURVE key and its enrollment record into <data-dir>/config, which is why a restart costs no join-key slot.

--workspace seeds the workspaces file the EEM enforces, into that same config directory. The names are read from your project and resolved before anything on the host changes: a name the project does not have, or one that names a chat workspace, refuses the whole run rather than deploying an EEM that binds nothing. So does a workspace of a type Erebine does not describe: the deploy cannot name its tool domains, and a declaration with none admits every domain's tools, so deploy without it and declare it afterwards with erectl workspaces declare --domains. A declaration binds on the name, so that check is what stops a typo from silently serving nothing.

bash
sudo erectl deploy eem --service --region us-east \
  --workspace prod-k8s --workspace erebine-repo

Without the flag no file is written and the EEM keeps its single local workspace; the command prints the erectl workspaces declare line that writes one later. A code workspace arrives without repo_roots -- those name directories on this host, and the project has nowhere to put them -- so the deploy says which ones to fill in before the EEM serves them. The file is yours after the first write: an edit takes effect without a restart, and --update leaves it exactly as it found it.

Everything else -- the region, the key source, the endpoint, the unit, the timeouts -- comes from the common options both kinds share.

Systemd and Reboots

--service installs a unit per kind -- erebine-agents.service for eim, erebine-eem.service for eem -- so the stack starts at boot on every runtime (on bare metal the same names run the EIMs and the EEM directly). Without it: docker restarts the containers itself; podman only does so on 5.8.2 or newer with podman-restart.service enabled -- the plan output tells you exactly what your host will do.

Every unit the command installs, for both kinds and on every runtime, lifts the locked-memory and address-space limits (LimitMEMLOCK=infinity, LimitAS=infinity): vLLM and CUDA map very large virtual address ranges, and NCCL registers memory against the locked-memory limit. docker starts containers from its own daemon rather than from the unit, so every inference service in the generated compose also carries ulimits: memlock: -1 (soft and hard), which docker and podman both apply. Every service in both kinds' generated compose carries ulimits: nice: 24 (soft and hard), the raw form of the bare-metal LimitNICE=-4, so its worker threads take the priority the runtime asks for without a capability; --update writes it into an existing stack.

The unit is a oneshot that runs up -d and stays active; it does not hold the containers attached. Container output reaches the journal through the journald log driver both kinds' generated compose pins, so journalctl -u erebine-agents -f (or -u erebine-eem) follows it on docker as well as podman. systemctl restart erebine-agents bounces the containers; enrollment state persists, so EIMs relaunch without re-enrolling.

Stopping the service runs compose down, which removes the containers on purpose: left in place, the next up -d adopted the old container and kept serving the old image. Nothing in the data directory is touched -- keys, enrollment records and model state all survive -- so the next start creates fresh containers that come back as the same EIMs. A reboot brings everything back, including the MIG layout when the persistence unit is installed. After an NVIDIA driver update on a MIG host, run systemctl restart erebine-mig-setup.

Verification

A deploy is not finished when the containers are up. Both kinds hold the run open until the router agrees they are serving, and fail the deploy rather than reporting success over a stack that enrolled and then went quiet.

erectl deploy eim runs four gates: the join key's enrollment counter rises by the number of EIMs that needed enrolling; every container is running on two samples; the accelerator is visible inside each service (pinned multi-GPU and MIG shapes); and the router reports the new EIMs connected and heartbeating.

erectl deploy eem runs three:

  1. Enrollment. The join key's enrollment counter rises by one. Skipped when the EEM already holds an enrollment record, because it restores its identity and redeems no slot.
  2. Liveness. The container is running, sampled twice so a crash loop cannot pass between samples.
  3. Router health and capability manifest. The router reports this EEM -- matched by the worker id the EEM persisted, never a sibling on another host -- connected, heartbeating, and having published a capability manifest. An EEM that enrolls but publishes no manifest advertises no tools, so this gate treats it as a failed deploy and says so.

A verified deploy prints the command that revokes the join key. Run it: the enrollment is complete, and for both kinds the identity on disk is what brings them back.

Updates

--update re-renders the deployment that is recorded on the host with the erectl you are running, refreshes the image, and restarts only when something moved. It changes nothing about the shape, so every shape flag is refused by name -- including --project on eem, whose project comes from the record.

bash
sudo erectl deploy eim --update
sudo erectl deploy eem --update

An update over an unchanged stack re-renders nothing and restarts nothing, so it is safe to run on a schedule. --restart forces the bounce anyway; --no-restart applies the changes and leaves the stack running as it is.

An update sizes shared memory from the host's /dev/shm as it is now unless the deploy passed --shm-size, so a host whose /dev/shm changed gets the new size and the stack restarts to take it.

An eim update also brings the host settings the deploy makes up to what the erectl you are running makes: the kernel settings, transparent hugepages, on an Intel GPU host the udev rule that keeps the GPUs out of runtime suspend, fwupd masked, and vm.overcommit_memory = 1, and on an AMD GPU host in a VM the PCIe link you stated, if you stated one. A host deployed by an older erectl gets the settings it is missing. Before it asks to proceed, the update lists each setting it will change, with the file and the key, or says they are already in place; a setting that is already right is not touched, and files that override one at boot are named. The last line counts them: 4 file(s) changed; 2 host setting(s) changed; image unchanged; restarted. Sysctl changes are applied to the running kernel, not only written. When vm.overcommit_memory changes, the EIMs restart, because they size their KV cache when they start; the other settings need no restart. --no-sysctl and --no-thp skip their steps on an update too, and so does a deploy that was run with them. On bare metal an update also creates the service account again if it was deleted. An eem update changes no host setting.

On bare metal an update installs the release the deploy recorded (the newest, unless the deploy pinned one) only over an older binary. Before it asks to proceed it prints what it does to the binary: an upgrade from one release to another, nothing when that release is already installed, or a refusal. It refuses, changing nothing, when the installed binary is a newer release, such as a release candidate you installed yourself, and when it cannot tell which release the binary is: a binary replaced by hand since erectl installed it reports a build erectl has no release for. Release candidates sort before their release, so 2.10.4-rc5 is older than 2.10.4. Rewrite only the units with --refresh-unit and keep that binary, or pass --allow-downgrade to install the recorded release over it. A deb install is then downgraded through apt; an rpm or brew install cannot be downgraded in place, so remove it first.

Upgrading a bare-metal EEM

A bare-metal EEM runs from the erebine-eem.service its deploy wrote, and a newer EEM binary installed over it does not rewrite that unit. When a release changes the unit, rewrite it on each host with the newer erectl:

bash
sudo erectl deploy eem --refresh-unit --dry-run   # show the change, touch nothing
sudo erectl deploy eem --refresh-unit

--refresh-unit reads the installed deployment from the host and rewrites only the unit. The binary, the account, the data and credentials directories, the env files and the enrollment stay as they are; it needs no join key and downloads nothing. It prints the lines that change, asks before writing (--yes skips the question), reloads systemd, restarts the EEM and checks that it is running. A run with nothing to change writes nothing and restarts nothing; the header comment does not count as a change. Drop-ins under erebine-eem.service.d are kept and listed. It takes --deploy-dir, --runtime, --restart, --no-restart, --yes, --lock-wait and --dry-run, and refuses every other flag by name.

Install the new binary at the path the unit runs before you refresh, and the restart picks up both; when the unit is already current, --restart restarts onto the new binary anyway. --update rewrites the unit too, but also installs the release the deploy recorded, refusing to put it over a newer binary, and needs the join key still in eem.env; use --refresh-unit when you installed the binary yourself. A host with no bare-metal EEM, or one whose recorded deployment lacks what the unit is built from, is refused with what is missing, and nothing changes.

Upgrading bare-metal EIMs

Bare-metal EIMs run from the erebine-agents.service and erebine-agents@.service their deploy wrote, and a newer EIM binary installed over them does not rewrite those units. When a release changes them, rewrite them on each host with the newer erectl:

bash
sudo erectl deploy eim --refresh-unit --dry-run   # show the change, touch nothing
sudo erectl deploy eim --refresh-unit

--refresh-unit reads the installed deployment from the host -- the hardware type, GPU strategy, EIM count, data directory, account, groups and the vLLM environment -- and rewrites only the two units. The binary, your vLLM, the model caches, the MIG layout and its persistence unit, the env files, the launcher and every EIM's enrollment stay as they are; it needs no join key and downloads nothing. It prints the lines that change in each unit, asks before writing (--yes skips the question), reloads systemd, restarts erebine-agents.service, which restarts every EIM, and checks that each instance is running. A run with nothing to change writes nothing and restarts nothing; the header comment does not count as a change. Drop-ins under erebine-agents.service.d, erebine-agents@.service.d and any erebine-agents@<n>.service.d are kept and listed. It takes the same flags as the EEM's --refresh-unit and refuses every other flag by name.

Install the new binary at the path the units run before you refresh, and the restart picks up both; when the units are already current, --restart restarts onto the new binary anyway. Each EIM loads its model again from its cache after the restart. --update rewrites the units too, but also rewrites the env files and launchers, installs the release the deploy recorded, refusing to put it over a newer binary, and needs the join key still in agents.env; use --refresh-unit when you installed the binary yourself. A host with no bare-metal EIMs, or one whose recorded deployment lacks what the units are built from, or where a file they need is gone, is refused with what is missing, and nothing changes.

Teardown

bash
# Inference stack:
sudo erectl deploy eim --down              # stop and remove the stack
sudo erectl deploy eim --down --revoke-key # also revoke the join key

# EEM:
sudo erectl deploy eem --down
sudo erectl deploy eem --down --revoke-key

Each kind tears down only its own stack; the other keeps running. Teardown keeps what is expensive or shared: model caches and EIM and EEM identities under the data directory, the env file, the sysctl entries, the transparent hugepage settings, and the MIG layout (with the unwind recipe printed). For the EEM it also keeps the enrollment record beside its keys. Re-deploying after a teardown picks the identities back up without re-enrolling.

Common Options

Both kinds take these, and they mean the same thing for each:

FlagMeaning
--regionJoin-key region (required when minting)
--join-key, --join-key-fileBring your own key (- for stdin; prefer the file form, a value is visible in ps)
--join-key-nameDisplay name for the minted key
--enroll-endpointHTTP enrollment URL (default: the key's own claim, else the base_url origin)
--allow-insecure-enrollPermit a plain-HTTP, non-loopback enrollment endpoint
--enroll-timeoutSeconds to wait for the enrollment gate
--serviceInstall and enable the kind's systemd unit
--runtimepodman | docker | baremetal (default: podman, then docker; baremetal needs --service)
--install-methodBare metal: brew | deb | rpm | binary (default: reuse, then the host's package manager, then brew, then the release binary)
--deploy-dirGenerated-file home (default /etc/erebine/deploy for eim, /etc/erebine/deploy-eem for eem). This and every directory flag take an absolute path without spaces, quotes, backslashes, % or $: they are written into unit files as they are
--registry, --image-versionImage source override and tag pin; on bare metal --image-version pins the EIM or EEM release and --registry is refused
--max-concurrent, --log-levelEIM and EEM tunables
--no-startStop after artifacts, units and pull; the printed recipe finishes the deploy
--down, --revoke-keyTeardown, and revoking the recorded key with it
--update, --restart, --no-restartRe-render the recorded deployment; force or suppress the restart
--allow-downgradeWith --update on bare metal: install the recorded release even over a newer binary, or one erectl cannot tell (see Updates)
--lock-wait, --yes, --dry-runLocking, confirmation, preview

EIM Options

On top of the common options, the inference kind takes:

FlagMeaning
--countEIMs to deploy (default 1; 1 to 64). Above one, every EIM gets EREBINE_EIM_HOST_EIM_COUNT and sizes its default KV offload from that share of host RAM, so the EIMs do not each claim the whole host's RAM; an EREBINE_EIM_KV_OFFLOAD_SIZE_GB set in agents.local.env (or its pre-2.10 name, EREBINE_AGENT_KV_OFFLOAD_SIZE_GB) is used as set
--hardware-type / --auto-hardware-detectExactly one is required
--gpu-strategypinned | shared | mig
--mig, --mig-profile, --mig-repartitionMIG shorthand, profile name, layout replacement
--tierTier slug (repeatable; default self_hosted -- user-owned EIMs; list slugs with erectl tiers)
--router-addrRouter ZeroMQ address (repeatable; defaults to tcp://<base_url host>:5555)
--data-dirData root (default /data/erebine)
--shm-sizeShared memory (/dev/shm) per EIM container. Default: the size of the host's own /dev/shm, read at deploy time and rounded down to whole GiB (256g for a 256 GiB host /dev/shm; under 1 GiB, whole MiB). /dev/shm is tmpfs, so the size caps what each container can write there and sets nothing aside. Takes a compose size such as 64g, 512m or 1.5g. The EIM's own KV cache offload (EREBINE_EIM_KV_OFFLOAD_SIZE_GB) uses process RAM, not /dev/shm. Only a --kv-offloading-size passed straight to vLLM through EREBINE_EIM_VLLM_ARGS keeps its cache here: one file of the full size per data-parallel engine, so it needs size x DP + 1 GiB + DP x 424 MiB, and the EIM clamps it at launch. The plan shows the size and where it came from, warns when agents.local.env passes such an offload that does not fit at DP=2 (with the DP=1 and DP=2 limits), and falls back to 90g with a warning when the host's /dev/shm cannot be read. A single EIM, MIG aside, on docker or on a podman-compose release after 1.6.0 shares the host's IPC namespace and /dev/shm (ipc: host), so no shm_size is written there and the plan shows n/a; podman-compose 1.6.0 and earlier ignore ipc: host and get the size
--gpu-memory-utilization, --max-model-lenInference tunables
--overprovision-factorAdmit slots per safe slot, at least 1.0 (default 1.5). EIMs bound to an anchor are routed by safe slots
--gpu-linkOptional tuning, off by default, amd-gpu in a VM: the PCIe link of the slot the GPU sits in, as gen<N>x<W> (for example gen3x8), or auto to remove it. Not a stability fix. Taken by --update; refused for other hardware, on a host that is not a VM, and under WSL2. See AMD GPUs in a VM
--no-sysctl, --no-thp, --force-chownStep controls (bare metal refuses --no-sysctl and --shm-size)
--task-modeTask mode the EIMs will serve (repeatable). speech or image_generation picks the omni image, eim-vllm-omni-cuda on nvidia-gpu, eim-vllm-omni-rocm on amd-gpu or eim-vllm-omni-omix on intel-gpu; speech alone on cpu or amd-cpu picks eim-speech-cpu, and speech beside another mode there is refused (container deploys only); the other vLLM modes keep the hardware's image. systemone is refused: Laya EIMs are not rendered by erectl deploy. Refused under --update and --down. See Hardware Types
--enginevllm (default) or native. native runs the speech image for the hardware, eim-speech-cuda on nvidia-gpu or eim-speech-cpu on cpu and amd-cpu, and serves --task-mode speech alone; refused on amd-gpu, intel-gpu and bare metal, and under --update and --down. An explicit vllm takes no speech image, so --task-mode speech on a CPU host is refused with it. See Hardware Types
--vllm-pathBare metal: the vllm executable, or the Python environment holding bin/vllm (default: vllm on PATH)
--refresh-unitBare metal: rewrite the installed erebine-agents.service and erebine-agents@.service with this erectl and restart on change, keeping everything else (see Upgrading bare-metal EIMs)

Troubleshooting

  • Join key expired: keys live one hour. The command minimizes the window by pulling first; a very slow first pull can still cross it. Re-running is safe -- every step converges and a fresh key is minted.
  • Enrollment timeout with many EIMs: enrollment is rate limited per host. The command paces starts above five EIMs automatically; the timeout scales with --count and can be raised with --enroll-timeout.
  • EIMs enroll but never serve: the router-health gate catches this -- it means the ZeroMQ path to the join key's router addresses (--router-addr, or the default derived from base_url) is unreachable from the host. Check routing and firewalls toward the router's mesh port, and pass --router-addr explicitly if the mesh endpoint is not on the base_url host.
  • The join key was minted for the other kind: a key carries the role it was minted for, and a deploy of the wrong kind is refused before the host is touched. Mint the key from the matching dashboard tab, or drop --join-key-file and let the deploy mint the right one.
  • The EEM enrolled but published no manifest: it advertises no tools, so the third gate fails the deploy. Its tools are compiled into the EEM: check that --domains and --disable-domains leave some enabled, and read journalctl -u erebine-eem.
  • The EEM will not enroll again: it redeems a join key once and keeps the enrollment beside its keys, in agent-enrollment.json under its config directory. Remove that file to enroll again from a fresh key.
  • An EIM or EEM deleted on the router stays down: it leaves terminated-by-router in its config directory and waits instead of enrolling again; the same happens when its enrollment is revoked or refused as a clone. Re-running the deploy names the EIM or EEM, removes the file once it has started it with the run's key, and it enrolls with that key within a minute. With --no-start the printed steps end with removing the file; run that once the EIM or EEM is up with the new key. --update leaves the file, so an update does not bring back an EIM or EEM you deleted.
  • The join key is for another project than the EEM: the deploy warns, and the EEM enrolls into the key's project. The third gate fails while the EEM is still enrolled in the old project, naming both; the EEM's log says why it did not move, usually a key that expired or was revoked. Mint a new key and re-run, or re-run with a key for the old project to stay there. The EEM's entry in the old project stays behind, offline; remove it from that project's EEMs page.
  • The unit restarts every two minutes and the journal names a missing device: the unit's start waits 120 seconds for the GPU's devices, then fails, and systemd tries again 10 seconds later. Each missing path is named on its own line with the next step. On an AMD host, /dev/kfd missing means amdgpu has not loaded (lsmod | grep amdgpu, dmesg | grep -i amdgpu); under WSL2, /dev/dxg or /usr/lib/wsl/lib/libdxcore.so missing means WSL has no GPU support for the distribution (wsl --update and the current AMD driver on Windows, then wsl --shutdown).
  • MIG devices unresolvable after a driver update: run systemctl restart erebine-mig-setup to re-carve and refresh the CDI spec.
  • Another deploy holds the lock: the message names its pid, start time, and last completed step; pass --lock-wait to queue behind it. The lock is per host, so the two kinds queue behind each other rather than racing.
  • A recorded deployment belongs to the other kind: an explicit --deploy-dir that points at the other kind's directory is refused, naming the kind it found and the flag that fixes it.
  • No container runtime found: install podman or docker, or re-run with --runtime baremetal --service.
  • Another unit runs the binary: an enabled or running unit, such as the one a deb or rpm package ships, runs erebine-eim or erebine-eem (or the earlier erebine-eim-agent or erebine-eem-agent) and would be a second instance of the same kind. Run the systemctl disable --now command the error names, then re-run the deploy.
  • Bare metal: no vLLM found: under sudo, PATH is root's. Pass --vllm-path with the environment vLLM is installed in.
  • Bare metal: an EIM instance is failed: five failed starts inside ten minutes park an instance so a bad join key is not retried forever. Read journalctl -u erebine-agents@<n>, fix the cause, then systemctl reset-failed and systemctl restart erebine-agents.