Deploy Your First EEM
One command from a blank Linux host to an Erebine Execution Manager (EEM) enrolled with your router and serving an operational workspace. No GPU. erectl deploy eem creates the directories, mints an execution join key, generates and starts the stack, and verifies enrollment before it reports success. Ends at the first approved tool call.
Prerequisites
- A Linux host with
docker(orpodman) and Compose v2, and root on it viasudo. Minimum 1 CPU, 512 MB RAM. No GPU. Without a container runtime, deploy on bare metal instead (see Step 1). erectlinstalled on that host and configured against your router with an API key carrying themanagementscope -- see erectl. Undersudoit is root's configuration that is read, so either configure root or pass--base-urland--api-keyexplicitly.- A project. The EEM joins the project your
base_urlis scoped to unless you name another with--project. - Outbound HTTPS to the router URL (for enrollment) and outbound TCP to the router's CurveZMQ port (for the data plane).
- One of: a Kubernetes cluster, OpenStack, Docker daemon, or another target infrastructure the EEM should manage.
Step 1: Deploy the EEM
One command. It mints the join key itself, so there is no token to copy between machines and nothing to paste into a file.
sudo erectl deploy eem --service --region us-east
--service installs the systemd unit so
the EEM comes back after a reboot. The region is
what the minted key records; add
--registration-name to name the EEM
something other than the default
eem-<hostname>.
The command plans first and asks before it changes anything:
Erebine EEM Deploy Plan
=======================
1. Target shape
EEM (no GPU, no kernel tuning), runtime docker
registration name eem-fleet-01; project the base URL's project (read at step 1; override with --project)
compose: docker compose
image: ghcr.io/erebine/container-agents/eem:latest
2. Host mutations
data dirs: /data/erebine-eem/{credentials,state,config} (chown 5153)
artifacts: /etc/erebine/deploy-eem/{eem.compose.yaml 644, eem.env 600, eem.local.env 600 (seeded once), deploy-state.json 640}
workspaces: none declared (no file is written; --workspace seeds one)
units: erebine-eem.service
3. Join key
mint after pull (EEM key; keys live at most 1h; expiry starts then)
maxEnrollments 10; role execution; no tiers
router url resolved at step 5 (no --enroll-endpoint; the token's htp claim, else https://router.example.com from config base_url)
4. Start and verify
first enrollment (agent-enrollment.json absent; one join-key slot)
gates: enrollment delta, liveness, router health + capability manifest
timeout: 120s
5. Boot-restart truth
systemd starts the stack at boot
6. Listening surface
bridge network: the EEM listens on nothing host-side
7. Warnings
(none)
Proceed? (y/N)
Answer y and the seven steps run in
order:
[1/7] Creating data directories under /data/erebine-eem (uid 5153) ...
[2/7] Writing compose, env, state, pointer ...
[3/7] Installing the systemd unit ...
[4/7] Pulling ghcr.io/erebine/container-agents/eem:latest ...
[5/7] Resolving the join key ...
minted execution join key jk_01HX... (prefix ere-join..., expires in <= 1h)
[6/7] Starting the EEM ...
[7/7] Verifying ...
gate 1: 1 enrollment(s) confirmed
gate 2: all 1 container(s) running
gate 3: EEM us-east:agent-3f9c1d2b8a04 connected, heartbeating, capability manifest published
Deploy verified. Recommended next step: revoke the join key now that enrollment is complete: erectl eems join-keys jk_01HX... --revoke
Run the revoke command it prints; it asks before it
revokes, and --yes (or
--force) skips the question. The EEM redeems
its join key once and keeps the enrollment beside its
keys under /data/erebine-eem/config, so
restarts and re-deploys need no key at all.
Preview first: erectl deploy eem --dry-run --region us-east prints the same plan and every command it would run, touching nothing. It is the one mode that does not need root.
No container runtime on the host? Add
--runtime baremetal. The EEM binary
comes from the host's package manager, Homebrew or
the release, verified against the release's sha256,
and runs as erebine-eem.service under
its own erebine-eem account, from EEM
release 2.4.0 on. The data directory, the join key,
the gates and --down work as above; the
credentials directory stays root's, read-only to the
EEM.
sudo erectl deploy eem --runtime baremetal --service --region us-east
The EEM needs libzmq, libsodium, libzstd and
libcurl. The packages pull them in; for the release
binary the deploy checks them before it changes the
host and names any that are missing. On RHEL, Rocky
Linux and AlmaLinux, zeromq and libsodium come from
EPEL: run sudo dnf install epel-release
first. Details: bare metal.
On a Mac, the desktop app runs the EEM instead: see Desktop Execution Manager.
Step 2: Verify the EEM
The deploy already proved enrollment, liveness and
the published manifest. Two checks read the same
facts back. On the host, ask the EEM itself. It
answers on its loopback metrics listener, port
9095, inside the container:
sudo docker compose -p erebine-eem -f /etc/erebine/deploy-eem/eem.compose.yaml \
exec eem-agent curl -s http://127.0.0.1:9095/readyz
curl -s http://127.0.0.1:9095/readyz
{"ready":true,"enrolled":true,"hasBearerToken":true,"manifestLoaded":true,"leaseAcknowledged":true,"discoveryComplete":true,"draining":false}
A ready EEM answers HTTP 200 with
"ready":true; key order varies. A
not-ready EEM answers 503 with the same fields,
and the one that is false names what
blocks it. leaseRejection appears only
when the router refused the EEM's lease, and
carries the reason. On podman, use the compose
command the deploy plan printed.
From your workstation, list the EEMs:
erectl eems -o wide
+--------------------------------------+----------------------------+----------------------------+----------+----------+---------+----------------------+
| ID | NAME | WORKER_ID | STATUS | REGION | PRESEED | LAST_HEARTBEAT |
+--------------------------------------+----------------------------+----------------------------+----------+----------+---------+----------------------+
| 3f9c1d2b-8a04-4c1e-9b6f-0d2e7a1c5b90 | us-east:agent-3f9c1d2b8a04 | us-east:agent-3f9c1d2b8a04 | active | us-east | no | 2026-04-22T03:31:12Z |
+--------------------------------------+----------------------------+----------------------------+----------+----------+---------+----------------------+
1 row
ID is the EEM's UUID; Step 5 takes it.
Enrollment assigns the worker id and mirrors it into
NAME, so both columns carry the same
<region>:agent-<id> string --
not the --registration-name the deploy
passed, which is not stored. Pick your EEM out by
the worker id gate 3 printed. Then list the tools the
EEM published in its capability manifest -- this is
the catalogue callers dispatch against:
erectl exec tools -o wide
+------------------+-------------------------------------------------------------+
| Name | Description |
+------------------+-------------------------------------------------------------+
| k8s_drain_node | Drains a Kubernetes node for maintenance (evicts workloads) |
| k8s_get_pods | Lists pods in a Kubernetes namespace (JSON output) |
+------------------+-------------------------------------------------------------+
An empty list means the EEM advertises no tool. The
tools are built into the EEM, which leaves out a
tool whose command, such as kubectl, is
not installed where it runs, and every tool in a
domain EREBINE_EEM_DOMAINS or
EREBINE_EEM_DISABLE_DOMAINS turns off.
Step 3: Create an Operational Workspace
Execution tools run in an operational workspace. A
chat workspace has no scope and no EEM, so it
cannot run them. Create one from any workstation
with erectl configured:
erectl workspaces create --name prod-k8s --kind operational --type kubernetes
Workspace 'ws_01HX...' created successfully.
Workspace:
ID: ws_01HX...
Project ID: proj_01HX...
Name: prod-k8s
Kind: operational
Default: no
Types: kubernetes
Inference routing: default
Created: 2026-04-22T03:29:44Z
Updated: 2026-04-22T03:29:44Z
--type takes one or more of code,
kubernetes, container,
linux, openstack and
iac, comma-separated:
--type kubernetes,openstack makes one
workspace that serves both. The ID is the
workspace's ws_ id; Step 5 and
erectl exec invoke take it, and
erectl workspaces list --kind operational
prints it again.
On the web
On the Workspaces page, open New workspace and choose Operational. Project owners and operators get the choice; members create chat workspaces only.
- Name: the name the EEM declares, so it takes lowercase letters, digits and single hyphens, starts and ends with a letter or digit, and has at most 48 characters. The field checks it as you type.
- Types: one or more. A workspace
can serve
kubernetesandopenstacktogether, and a chat linked to it reaches both. - Credentials: a row for each type
that reads a credential, and
Add credential for any other
family a tool reads (
kubernetes,openstack,git_token). A stored row of a family no tool reads yet is marked: tools of that family are refused in the workspace, and Remove unbinds it. A row holds the name of a file you put on the EEM host in this workspace's credentials folder (see Workspace credentials); the secret itself never passes through the form. A row left empty binds nothing. - Description: optional.
A name, type or file name the router refuses is shown beside its field, and the form keeps what you typed. The workspace's Settings, under Operational, change the types, the credentials and the scope later.
Step 4: Declare It on the EEM
The EEM learns the workspaces it serves from its
workspaces.yaml; enrollment carries keys,
addresses and a token, never a workspace list. The
name of each declaration must equal the
router-side workspace name exactly -- that string is
the only axis the router matches on. Declare it one
of three ways.
With erectl, on the EEM host
sudo erectl workspaces declare --workspace prod-k8s
Wrote /data/erebine-eem/config/workspaces.yaml.
Added: prod-k8s
Replaced: -
Removed: -
Left alone: -
The command builds each declaration from the
router's own workspace row, so the name cannot
drift, and writes the file the deploy recorded. A
new declaration lists the tool domains its types
serve, kubernetes for
prod-k8s; set others, such as
--domains kubernetes,observability, or
edit the file. A workspace of a type Erebine does not
describe is refused until --domains
names its domains.
Repeat --workspace to declare more;
--remove <name> stops declaring
one. A refresh keeps the scope already in the file;
--reseed-scope takes the router's copy
instead.
At deploy
When the workspace exists before the EEM does,
name it on the deploy command. The deploy seeds the
file from the router the same way.
--update does not take
--workspace; on a running EEM, use
declare.
sudo erectl deploy eem --service --region us-east --workspace prod-k8s
In the desktop app
On a Mac, the Execution Manager in the desktop app creates and declares code workspaces, and Add from this project declares one that already exists. See Desktop Execution Manager.
By hand
The file lives in the config directory the EEM mounts:
workspaces:
- name: prod-k8s # MUST equal the router-side workspace name
types: [kubernetes]
description: Production cluster
domains: [kubernetes, observability]
scope:
namespaces: [default, kube-system]
rate_per_minute: 10
sudo chown 5153:5153 /data/erebine-eem/config/workspaces.yaml
The generated stack already points the EEM at this file. Nothing restarts: the EEM rebuilds its manifest from the file on every call, so a narrowed scope binds the very next tool it runs, and the router picks the change up within about one lease interval.
Step 5: Bind the EEM (Optional)
Without a binding, dispatch uses the first online
EEM in the project that advertises the workspace.
Bind to pin one. The command takes the workspace's
ws_ id from Step 3 and the EEM UUID
from Step 2, not names:
erectl workspaces bind-eem ws_01HX... 3f9c1d2b-8a04-4c1e-9b6f-0d2e7a1c5b90
erectl workspaces eems ws_01HX...
workspaces eems lists the bindings.
pause-eem stops dispatch to a bound
EEM and keeps the binding;
unbind-eem removes it.
Step 6: Link a Chat
A chat's tool calls reach an EEM only through the operational workspace the chat is linked to. Without the link, they reach no EEM. Link it per chat, by title or id:
erectl chats link-exec-workspace "Release triage" prod-k8s
linked "Release triage" (chat_01HX...) -> prod-k8s (ws_01HX...)
Or choose Execute in in the chat on the web, the desktop app or iOS. Only owners and operators set the link. See Connect a Chat to an Execution Agent.
Step 7: Run and Approve a Tool Call
In the linked chat, ask for something that reads,
such as the pods in default. The model
calls k8s_get_pods and it runs at once:
read tools need no approval.
Ask for something that writes or drains, such as
draining worker-3. The model calls
k8s_drain_node, and the chat shows an
approval card. The call waits until an owner or
operator decides it, in any of these places:
- the approval card in the chat;
- the waiting row on the Activity tab of the
Operations page,
/ops; - the Approvals screen of the desktop app or the iOS app;
erectl, from any workstation.
erectl approvals list -o wide
erectl approvals approve 7c1e2a90-4b3d-4f6e-8a21-5d9b0c3e7f14 --note "maintenance window"
+--------------------------------------+-------------+----------------+------------+----------------------+----------------------+
| ID | Risk | Tool | Workspace | Waiting since | Deadline |
+--------------------------------------+-------------+----------------+------------+----------------------+----------------------+
| 7c1e2a90-4b3d-4f6e-8a21-5d9b0c3e7f14 | destructive | k8s_drain_node | prod-k8s | 2026-04-22T03:40:02Z | 2026-04-22T04:40:02Z |
+--------------------------------------+-------------+----------------+------------+----------------------+----------------------+
1 row
erectl approvals reject <id> --reason "..."
refuses the call. An irreversible call also needs
--confirm "I approve".
erectl approvals watch follows new
requests from the terminal.
Write tools wait for approval by default.
Destructive and irreversible tools, and every tool
in the database domain, always wait,
whatever the workspace grants. See
Approvals.
From the CLI
erectl exec invoke runs a tool without a
chat. --workspace takes the
ws_ id from Step 3:
erectl exec invoke --workspace ws_01HX... --tool k8s_get_pods --args '{"namespace": "default"}'
Invocation '5b2f8e61-0c47-4d9a-b3e8-1a6d4f7c2e09' created (status queued).
Invocation Details:
ID: 5b2f8e61-0c47-4d9a-b3e8-1a6d4f7c2e09
Status: queued
The command submits the call and exits, printing the
invocation id and its status. Follow it with
erectl exec watch <id>. A call that
needs approval prints
status awaiting_approval and the
exec watch command to follow it; approve
it as above.
The same pattern holds in every domain. One read tool and one tool that waits for approval in each:
| Domain | Read tool | Write or destructive tool |
|---|---|---|
kubernetes |
k8s_get_pods |
k8s_drain_node (destructive) |
code |
code_grep |
code_edit (write) |
openstack |
openstack_server_list |
openstack_server_reboot (destructive) |
container |
container_ps |
container_restart (destructive) |
Your first EEM now serves an operational workspace. Next, author a chat template to bake in a repeatable procedure: Author a Chat Template.
Troubleshooting
See Error Codes for the full matrix and EEM Troubleshooting for symptoms. Common first-deploy issues:
join_key_role_mismatch: the join key was minted for an EIM. A key carries the role it was minted for, anderectl deploy eemrefuses it before the host is touched; the router refuses the enrollment too. Drop--join-key-fileand let the deploy mint an execution key, or mint one with New EEM on the EEM tab of System -> EIMs and EEMs.expired_tokenormax_enrollments: the join key expired or has enrolled its limit of EEMs. Re-run the deploy, which mints a fresh key.- A tool refuses with no scope: the chat is not linked, or the workspace is not declared on the EEM. See Step 4 and Step 6.
- The EEM will not enroll again:
it redeems a join key once and keeps the
enrollment in
agent-enrollment.jsonbeside its keys in the config directory. Remove that file to enroll again from a fresh key. A key for another project is redeemed: the EEM moves to that project. - No log output: the generated
stack pins the journald log driver, so the
container's output is in
journalctl -u erebine-eem -fon docker as well as podman.
To take the EEM down, keeping its data, keys and
enrollment: sudo erectl deploy eem
--down. To re-render the deployment with a
newer erectl and refresh the image:
sudo erectl deploy eem --update,
which restarts only when something actually moved.
Upgrading a Bare-Metal EEM
A bare-metal deploy writes
erebine-eem.service once. Installing a
newer EEM binary does not rewrite it, so when a
release changes the unit, rewrite it on each host
with the newer erectl:
sudo erectl deploy eem --refresh-unit --dry-run # show the change, touch nothing
sudo erectl deploy eem --refresh-unit
It reads the installed deployment from the host and
rewrites only the unit: the binary, the
erebine-eem account, the data and
credentials directories, the env files and the EEM's
enrollment stay as they are, and no join key is
needed. It prints the lines that change, asks before
writing (--yes skips the question),
reloads systemd, restarts the EEM and checks that it
is running. A run with nothing to change writes
nothing and restarts nothing, so it is safe to
repeat. Drop-ins added with
systemctl edit erebine-eem are kept and
listed.
Install the new binary first, at the path the unit
runs, then refresh: the restart picks up both. When
the unit is already current, add
--restart to restart onto the new
binary anyway. --update rewrites the unit
too, but also installs the release the deploy
recorded, and stops rather than put it over a newer
binary unless you pass --allow-downgrade;
use --refresh-unit when you installed the
binary yourself.
On a host with no bare-metal EEM, or one whose installed deployment does not record what the unit is built from, it stops and names what is missing without changing anything. Run the full deploy again in that case.
Manual Deployment (compose)
For hosts erectl cannot reach, the same
stack runs by hand from the public compose file. Mint
the join key with New EEM on the EEM tab
of System -> EIMs and EEMs first.
Clone the
erebine/container-agents
repository on the EEM host (or copy just the
compose/ directory) so the compose file
referenced below is available locally.
git clone https://github.com/erebine/container-agents.git
cd container-agents/compose
Create the host directories the EEM mounts for workspace credentials, runtime state, and its persistent signing key. The container runs as UID:GID 5153:5153.
sudo mkdir -p /data/erebine-eem/credentials \
/data/erebine-eem/state /data/erebine-eem/config
sudo chown -R 5153:5153 /data/erebine-eem
Create a .env file in the
compose/ directory. Docker Compose reads it
automatically for variable substitution. The EEM
refuses to start without the three required values.
# Required
EREBINE_EEM_JOIN_KEY=eyJhbGciOi...
EREBINE_ROUTER_URL=https://router.example.com
EREBINE_EEM_REGISTRATION_NAME=eem-fleet-01
# Image (pulls the published container image)
EEM_IMAGE=ghcr.io/erebine/container-agents/eem:latest
# Optional
EREBINE_EEM_LOG_LEVEL=info
EREBINE_EEM_MAX_CONCURRENT=20
Treat this file as a secret: the join token is a bearer credential. Set chmod 600 .env; never commit it to a repository and never copy it onto an operator workstation.
docker compose -f compose.agent-eem.yaml up -d
docker compose -f compose.agent-eem.yaml logs -f eem-agent
Expected messages on a first start, in order. Each
line also carries a timestamp, the level and its
metadata; Enrollment active carries the
assigned worker_id and
source=join_key.
EEM starting
Enrolling with router
Enrollment completed
Enrollment active
Announced on the control plane
EEM running
A later start restores the saved enrollment instead:
it logs Restored persisted enrollment in
place of Enrolling with router and
Enrollment completed, and
Enrollment active carries
source=persisted.
The registration name must be unique per project. The
EEM generates a persistent Ed25519 signing key on first
run and writes it under the mounted config
volume so CURVE rotation acks survive restarts; the
router address and CURVE public key arrive in the
enrollment response. The lease duration is owned by the
router server-side. The join key is consumed once, and
the enrollment is persisted beside the keys: do not
re-use the key even if the enrollment seems to have
failed.
Declaring a workspace on the manual path
Write the declarations into the config directory the
stack mounts -- the same file as in
Step 4, and the same rule that
name must equal the router-side workspace
name exactly:
sudo tee /data/erebine-eem/config/workspaces.yaml >/dev/null <<'YAML'
workspaces:
- name: prod-k8s # MUST equal the router-side workspace name
types: [kubernetes]
description: Production cluster
domains: [kubernetes, observability]
scope:
namespaces: [default, kube-system]
rate_per_minute: 10
YAML
sudo chown 5153:5153 /data/erebine-eem/config/workspaces.yaml
The compose file points the EEM at that file: its
environment: block sets
EREBINE_EEM_WORKSPACES_FILE to
/var/lib/eem/.config/erebine/workspaces.yaml,
the config mount. A copy without the key reads
/etc/erebine/workspaces.yaml, which the
stack does not mount; add the key and run
docker compose -f compose.agent-eem.yaml up -d
--force-recreate. Edits to the declarations
need no recreate: the EEM rebuilds its manifest
from the file on every call.
The router half is identical on both paths and runs
from any workstation with erectl
configured, not on the EEM host: create the
workspace as in Step 3,
optionally bind the EEM as in
Step 5, link a chat as in
Step 6, then run a tool call as
in Step 7.