docs
// EEM Guides

Deploy Your First EEM

One command from a blank Linux host to an Erebine Execution Manager (EEM) enrolled with your router and serving an operational workspace. No GPU. erectl deploy eem creates the directories, mints an execution join key, generates and starts the stack, and verifies enrollment before it reports success. Ends at the first approved tool call.

Prerequisites

  • A Linux host with docker (or podman) and Compose v2, and root on it via sudo. Minimum 1 CPU, 512 MB RAM. No GPU. Without a container runtime, deploy on bare metal instead (see Step 1).
  • erectl installed on that host and configured against your router with an API key carrying the management scope -- see erectl. Under sudo it is root's configuration that is read, so either configure root or pass --base-url and --api-key explicitly.
  • A project. The EEM joins the project your base_url is scoped to unless you name another with --project.
  • Outbound HTTPS to the router URL (for enrollment) and outbound TCP to the router's CurveZMQ port (for the data plane).
  • One of: a Kubernetes cluster, OpenStack, Docker daemon, or another target infrastructure the EEM should manage.

Step 1: Deploy the EEM

One command. It mints the join key itself, so there is no token to copy between machines and nothing to paste into a file.

bash
sudo erectl deploy eem --service --region us-east

--service installs the systemd unit so the EEM comes back after a reboot. The region is what the minted key records; add --registration-name to name the EEM something other than the default eem-<hostname>.

The command plans first and asks before it changes anything:

text
Erebine EEM Deploy Plan ======================= 1. Target shape EEM (no GPU, no kernel tuning), runtime docker registration name eem-fleet-01; project the base URL's project (read at step 1; override with --project) compose: docker compose image: ghcr.io/erebine/container-agents/eem:latest 2. Host mutations data dirs: /data/erebine-eem/{credentials,state,config} (chown 5153) artifacts: /etc/erebine/deploy-eem/{eem.compose.yaml 644, eem.env 600, eem.local.env 600 (seeded once), deploy-state.json 640} workspaces: none declared (no file is written; --workspace seeds one) units: erebine-eem.service 3. Join key mint after pull (EEM key; keys live at most 1h; expiry starts then) maxEnrollments 10; role execution; no tiers router url resolved at step 5 (no --enroll-endpoint; the token's htp claim, else https://router.example.com from config base_url) 4. Start and verify first enrollment (agent-enrollment.json absent; one join-key slot) gates: enrollment delta, liveness, router health + capability manifest timeout: 120s 5. Boot-restart truth systemd starts the stack at boot 6. Listening surface bridge network: the EEM listens on nothing host-side 7. Warnings (none) Proceed? (y/N)

Answer y and the seven steps run in order:

text
[1/7] Creating data directories under /data/erebine-eem (uid 5153) ... [2/7] Writing compose, env, state, pointer ... [3/7] Installing the systemd unit ... [4/7] Pulling ghcr.io/erebine/container-agents/eem:latest ... [5/7] Resolving the join key ... minted execution join key jk_01HX... (prefix ere-join..., expires in <= 1h) [6/7] Starting the EEM ... [7/7] Verifying ... gate 1: 1 enrollment(s) confirmed gate 2: all 1 container(s) running gate 3: EEM us-east:agent-3f9c1d2b8a04 connected, heartbeating, capability manifest published Deploy verified. Recommended next step: revoke the join key now that enrollment is complete: erectl eems join-keys jk_01HX... --revoke

Run the revoke command it prints; it asks before it revokes, and --yes (or --force) skips the question. The EEM redeems its join key once and keeps the enrollment beside its keys under /data/erebine-eem/config, so restarts and re-deploys need no key at all.

Preview first: erectl deploy eem --dry-run --region us-east prints the same plan and every command it would run, touching nothing. It is the one mode that does not need root.

No container runtime on the host? Add --runtime baremetal. The EEM binary comes from the host's package manager, Homebrew or the release, verified against the release's sha256, and runs as erebine-eem.service under its own erebine-eem account, from EEM release 2.4.0 on. The data directory, the join key, the gates and --down work as above; the credentials directory stays root's, read-only to the EEM.

bash
sudo erectl deploy eem --runtime baremetal --service --region us-east

The EEM needs libzmq, libsodium, libzstd and libcurl. The packages pull them in; for the release binary the deploy checks them before it changes the host and names any that are missing. On RHEL, Rocky Linux and AlmaLinux, zeromq and libsodium come from EPEL: run sudo dnf install epel-release first. Details: bare metal.

On a Mac, the desktop app runs the EEM instead: see Desktop Execution Manager.

Step 2: Verify the EEM

The deploy already proved enrollment, liveness and the published manifest. Two checks read the same facts back. On the host, ask the EEM itself. It answers on its loopback metrics listener, port 9095, inside the container:

bashcontainer runtime
sudo docker compose -p erebine-eem -f /etc/erebine/deploy-eem/eem.compose.yaml \ exec eem-agent curl -s http://127.0.0.1:9095/readyz
bashbare metal
curl -s http://127.0.0.1:9095/readyz
text
{"ready":true,"enrolled":true,"hasBearerToken":true,"manifestLoaded":true,"leaseAcknowledged":true,"discoveryComplete":true,"draining":false}

A ready EEM answers HTTP 200 with "ready":true; key order varies. A not-ready EEM answers 503 with the same fields, and the one that is false names what blocks it. leaseRejection appears only when the router refused the EEM's lease, and carries the reason. On podman, use the compose command the deploy plan printed.

From your workstation, list the EEMs:

bash
erectl eems -o wide
text
+--------------------------------------+----------------------------+----------------------------+----------+----------+---------+----------------------+ | ID | NAME | WORKER_ID | STATUS | REGION | PRESEED | LAST_HEARTBEAT | +--------------------------------------+----------------------------+----------------------------+----------+----------+---------+----------------------+ | 3f9c1d2b-8a04-4c1e-9b6f-0d2e7a1c5b90 | us-east:agent-3f9c1d2b8a04 | us-east:agent-3f9c1d2b8a04 | active | us-east | no | 2026-04-22T03:31:12Z | +--------------------------------------+----------------------------+----------------------------+----------+----------+---------+----------------------+ 1 row

ID is the EEM's UUID; Step 5 takes it. Enrollment assigns the worker id and mirrors it into NAME, so both columns carry the same <region>:agent-<id> string -- not the --registration-name the deploy passed, which is not stored. Pick your EEM out by the worker id gate 3 printed. Then list the tools the EEM published in its capability manifest -- this is the catalogue callers dispatch against:

bash
erectl exec tools -o wide
textexcerpt
+------------------+-------------------------------------------------------------+ | Name | Description | +------------------+-------------------------------------------------------------+ | k8s_drain_node | Drains a Kubernetes node for maintenance (evicts workloads) | | k8s_get_pods | Lists pods in a Kubernetes namespace (JSON output) | +------------------+-------------------------------------------------------------+

An empty list means the EEM advertises no tool. The tools are built into the EEM, which leaves out a tool whose command, such as kubectl, is not installed where it runs, and every tool in a domain EREBINE_EEM_DOMAINS or EREBINE_EEM_DISABLE_DOMAINS turns off.

Step 3: Create an Operational Workspace

Execution tools run in an operational workspace. A chat workspace has no scope and no EEM, so it cannot run them. Create one from any workstation with erectl configured:

bash
erectl workspaces create --name prod-k8s --kind operational --type kubernetes
text
Workspace 'ws_01HX...' created successfully. Workspace: ID: ws_01HX... Project ID: proj_01HX... Name: prod-k8s Kind: operational Default: no Types: kubernetes Inference routing: default Created: 2026-04-22T03:29:44Z Updated: 2026-04-22T03:29:44Z

--type takes one or more of code, kubernetes, container, linux, openstack and iac, comma-separated: --type kubernetes,openstack makes one workspace that serves both. The ID is the workspace's ws_ id; Step 5 and erectl exec invoke take it, and erectl workspaces list --kind operational prints it again.

On the web

On the Workspaces page, open New workspace and choose Operational. Project owners and operators get the choice; members create chat workspaces only.

  • Name: the name the EEM declares, so it takes lowercase letters, digits and single hyphens, starts and ends with a letter or digit, and has at most 48 characters. The field checks it as you type.
  • Types: one or more. A workspace can serve kubernetes and openstack together, and a chat linked to it reaches both.
  • Credentials: a row for each type that reads a credential, and Add credential for any other family a tool reads (kubernetes, openstack, git_token). A stored row of a family no tool reads yet is marked: tools of that family are refused in the workspace, and Remove unbinds it. A row holds the name of a file you put on the EEM host in this workspace's credentials folder (see Workspace credentials); the secret itself never passes through the form. A row left empty binds nothing.
  • Description: optional.

A name, type or file name the router refuses is shown beside its field, and the form keeps what you typed. The workspace's Settings, under Operational, change the types, the credentials and the scope later.

Step 4: Declare It on the EEM

The EEM learns the workspaces it serves from its workspaces.yaml; enrollment carries keys, addresses and a token, never a workspace list. The name of each declaration must equal the router-side workspace name exactly -- that string is the only axis the router matches on. Declare it one of three ways.

With erectl, on the EEM host

bash
sudo erectl workspaces declare --workspace prod-k8s
text
Wrote /data/erebine-eem/config/workspaces.yaml. Added: prod-k8s Replaced: - Removed: - Left alone: -

The command builds each declaration from the router's own workspace row, so the name cannot drift, and writes the file the deploy recorded. A new declaration lists the tool domains its types serve, kubernetes for prod-k8s; set others, such as --domains kubernetes,observability, or edit the file. A workspace of a type Erebine does not describe is refused until --domains names its domains. Repeat --workspace to declare more; --remove <name> stops declaring one. A refresh keeps the scope already in the file; --reseed-scope takes the router's copy instead.

At deploy

When the workspace exists before the EEM does, name it on the deploy command. The deploy seeds the file from the router the same way. --update does not take --workspace; on a running EEM, use declare.

bash
sudo erectl deploy eem --service --region us-east --workspace prod-k8s

In the desktop app

On a Mac, the Execution Manager in the desktop app creates and declares code workspaces, and Add from this project declares one that already exists. See Desktop Execution Manager.

By hand

The file lives in the config directory the EEM mounts:

/data/erebine-eem/config/workspaces.yaml
workspaces: - name: prod-k8s # MUST equal the router-side workspace name types: [kubernetes] description: Production cluster domains: [kubernetes, observability] scope: namespaces: [default, kube-system] rate_per_minute: 10
bash
sudo chown 5153:5153 /data/erebine-eem/config/workspaces.yaml

The generated stack already points the EEM at this file. Nothing restarts: the EEM rebuilds its manifest from the file on every call, so a narrowed scope binds the very next tool it runs, and the router picks the change up within about one lease interval.

Step 5: Bind the EEM (Optional)

Without a binding, dispatch uses the first online EEM in the project that advertises the workspace. Bind to pin one. The command takes the workspace's ws_ id from Step 3 and the EEM UUID from Step 2, not names:

bash
erectl workspaces bind-eem ws_01HX... 3f9c1d2b-8a04-4c1e-9b6f-0d2e7a1c5b90 erectl workspaces eems ws_01HX...

workspaces eems lists the bindings. pause-eem stops dispatch to a bound EEM and keeps the binding; unbind-eem removes it.

Step 6: Link a Chat

A chat's tool calls reach an EEM only through the operational workspace the chat is linked to. Without the link, they reach no EEM. Link it per chat, by title or id:

bash
erectl chats link-exec-workspace "Release triage" prod-k8s
text
linked "Release triage" (chat_01HX...) -> prod-k8s (ws_01HX...)

Or choose Execute in in the chat on the web, the desktop app or iOS. Only owners and operators set the link. See Connect a Chat to an Execution Agent.

Step 7: Run and Approve a Tool Call

In the linked chat, ask for something that reads, such as the pods in default. The model calls k8s_get_pods and it runs at once: read tools need no approval.

Ask for something that writes or drains, such as draining worker-3. The model calls k8s_drain_node, and the chat shows an approval card. The call waits until an owner or operator decides it, in any of these places:

  • the approval card in the chat;
  • the waiting row on the Activity tab of the Operations page, /ops;
  • the Approvals screen of the desktop app or the iOS app;
  • erectl, from any workstation.
bash
erectl approvals list -o wide erectl approvals approve 7c1e2a90-4b3d-4f6e-8a21-5d9b0c3e7f14 --note "maintenance window"
textapprovals list
+--------------------------------------+-------------+----------------+------------+----------------------+----------------------+ | ID | Risk | Tool | Workspace | Waiting since | Deadline | +--------------------------------------+-------------+----------------+------------+----------------------+----------------------+ | 7c1e2a90-4b3d-4f6e-8a21-5d9b0c3e7f14 | destructive | k8s_drain_node | prod-k8s | 2026-04-22T03:40:02Z | 2026-04-22T04:40:02Z | +--------------------------------------+-------------+----------------+------------+----------------------+----------------------+ 1 row

erectl approvals reject <id> --reason "..." refuses the call. An irreversible call also needs --confirm "I approve". erectl approvals watch follows new requests from the terminal.

Write tools wait for approval by default. Destructive and irreversible tools, and every tool in the database domain, always wait, whatever the workspace grants. See Approvals.

From the CLI

erectl exec invoke runs a tool without a chat. --workspace takes the ws_ id from Step 3:

bash
erectl exec invoke --workspace ws_01HX... --tool k8s_get_pods --args '{"namespace": "default"}'
text
Invocation '5b2f8e61-0c47-4d9a-b3e8-1a6d4f7c2e09' created (status queued). Invocation Details: ID: 5b2f8e61-0c47-4d9a-b3e8-1a6d4f7c2e09 Status: queued

The command submits the call and exits, printing the invocation id and its status. Follow it with erectl exec watch <id>. A call that needs approval prints status awaiting_approval and the exec watch command to follow it; approve it as above.

The same pattern holds in every domain. One read tool and one tool that waits for approval in each:

Domain Read tool Write or destructive tool
kubernetes k8s_get_pods k8s_drain_node (destructive)
code code_grep code_edit (write)
openstack openstack_server_list openstack_server_reboot (destructive)
container container_ps container_restart (destructive)

Your first EEM now serves an operational workspace. Next, author a chat template to bake in a repeatable procedure: Author a Chat Template.

Troubleshooting

See Error Codes for the full matrix and EEM Troubleshooting for symptoms. Common first-deploy issues:

  • join_key_role_mismatch: the join key was minted for an EIM. A key carries the role it was minted for, and erectl deploy eem refuses it before the host is touched; the router refuses the enrollment too. Drop --join-key-file and let the deploy mint an execution key, or mint one with New EEM on the EEM tab of System -> EIMs and EEMs.
  • expired_token or max_enrollments: the join key expired or has enrolled its limit of EEMs. Re-run the deploy, which mints a fresh key.
  • A tool refuses with no scope: the chat is not linked, or the workspace is not declared on the EEM. See Step 4 and Step 6.
  • The EEM will not enroll again: it redeems a join key once and keeps the enrollment in agent-enrollment.json beside its keys in the config directory. Remove that file to enroll again from a fresh key. A key for another project is redeemed: the EEM moves to that project.
  • No log output: the generated stack pins the journald log driver, so the container's output is in journalctl -u erebine-eem -f on docker as well as podman.

To take the EEM down, keeping its data, keys and enrollment: sudo erectl deploy eem --down. To re-render the deployment with a newer erectl and refresh the image: sudo erectl deploy eem --update, which restarts only when something actually moved.

Upgrading a Bare-Metal EEM

A bare-metal deploy writes erebine-eem.service once. Installing a newer EEM binary does not rewrite it, so when a release changes the unit, rewrite it on each host with the newer erectl:

bash
sudo erectl deploy eem --refresh-unit --dry-run # show the change, touch nothing sudo erectl deploy eem --refresh-unit

It reads the installed deployment from the host and rewrites only the unit: the binary, the erebine-eem account, the data and credentials directories, the env files and the EEM's enrollment stay as they are, and no join key is needed. It prints the lines that change, asks before writing (--yes skips the question), reloads systemd, restarts the EEM and checks that it is running. A run with nothing to change writes nothing and restarts nothing, so it is safe to repeat. Drop-ins added with systemctl edit erebine-eem are kept and listed.

Install the new binary first, at the path the unit runs, then refresh: the restart picks up both. When the unit is already current, add --restart to restart onto the new binary anyway. --update rewrites the unit too, but also installs the release the deploy recorded, and stops rather than put it over a newer binary unless you pass --allow-downgrade; use --refresh-unit when you installed the binary yourself.

On a host with no bare-metal EEM, or one whose installed deployment does not record what the unit is built from, it stops and names what is missing without changing anything. Run the full deploy again in that case.

Manual Deployment (compose)

For hosts erectl cannot reach, the same stack runs by hand from the public compose file. Mint the join key with New EEM on the EEM tab of System -> EIMs and EEMs first.

Clone the erebine/container-agents repository on the EEM host (or copy just the compose/ directory) so the compose file referenced below is available locally.

bash
git clone https://github.com/erebine/container-agents.git cd container-agents/compose

Create the host directories the EEM mounts for workspace credentials, runtime state, and its persistent signing key. The container runs as UID:GID 5153:5153.

bash
sudo mkdir -p /data/erebine-eem/credentials \ /data/erebine-eem/state /data/erebine-eem/config sudo chown -R 5153:5153 /data/erebine-eem

Create a .env file in the compose/ directory. Docker Compose reads it automatically for variable substitution. The EEM refuses to start without the three required values.

compose/.env
# Required EREBINE_EEM_JOIN_KEY=eyJhbGciOi... EREBINE_ROUTER_URL=https://router.example.com EREBINE_EEM_REGISTRATION_NAME=eem-fleet-01 # Image (pulls the published container image) EEM_IMAGE=ghcr.io/erebine/container-agents/eem:latest # Optional EREBINE_EEM_LOG_LEVEL=info EREBINE_EEM_MAX_CONCURRENT=20

Treat this file as a secret: the join token is a bearer credential. Set chmod 600 .env; never commit it to a repository and never copy it onto an operator workstation.

bash
docker compose -f compose.agent-eem.yaml up -d docker compose -f compose.agent-eem.yaml logs -f eem-agent

Expected messages on a first start, in order. Each line also carries a timestamp, the level and its metadata; Enrollment active carries the assigned worker_id and source=join_key.

text
EEM starting Enrolling with router Enrollment completed Enrollment active Announced on the control plane EEM running

A later start restores the saved enrollment instead: it logs Restored persisted enrollment in place of Enrolling with router and Enrollment completed, and Enrollment active carries source=persisted.

The registration name must be unique per project. The EEM generates a persistent Ed25519 signing key on first run and writes it under the mounted config volume so CURVE rotation acks survive restarts; the router address and CURVE public key arrive in the enrollment response. The lease duration is owned by the router server-side. The join key is consumed once, and the enrollment is persisted beside the keys: do not re-use the key even if the enrollment seems to have failed.

Declaring a workspace on the manual path

Write the declarations into the config directory the stack mounts -- the same file as in Step 4, and the same rule that name must equal the router-side workspace name exactly:

bash
sudo tee /data/erebine-eem/config/workspaces.yaml >/dev/null <<'YAML' workspaces: - name: prod-k8s # MUST equal the router-side workspace name types: [kubernetes] description: Production cluster domains: [kubernetes, observability] scope: namespaces: [default, kube-system] rate_per_minute: 10 YAML sudo chown 5153:5153 /data/erebine-eem/config/workspaces.yaml

The compose file points the EEM at that file: its environment: block sets EREBINE_EEM_WORKSPACES_FILE to /var/lib/eem/.config/erebine/workspaces.yaml, the config mount. A copy without the key reads /etc/erebine/workspaces.yaml, which the stack does not mount; add the key and run docker compose -f compose.agent-eem.yaml up -d --force-recreate. Edits to the declarations need no recreate: the EEM rebuilds its manifest from the file on every call.

The router half is identical on both paths and runs from any workstation with erectl configured, not on the EEM host: create the workspace as in Step 3, optionally bind the EEM as in Step 5, link a chat as in Step 6, then run a tool call as in Step 7.