docs
// Execution Management (EEM)

Tutorials

Three end-to-end walkthroughs that exercise the most-requested Erebine Execution Manager (EEM) workflows. Copy the commands and payloads verbatim, substitute your own project external ID, workspace ID, and EEM names. Assumes you finished Deploy Your First EEM.

1. Author Your First Chat Template (drain-node)

Goal: ship a chat template that encodes the drain-node runbook and proves the operational safety invariants during a live run.

Step 1, seed a file payload from the platform template. The CLI has no copy verb; export the existing template as JSON, rename it, and save it locally.

shell
erectl templates show drain-node -o json \ | jq '.name = "drain-node-acme"' \ > drain-node-acme.json

Step 2, edit the template. Open drain-node-acme.json. The file payload uses these top-level fields:

{
  "name": "drain-node-acme",
  "system_prompt": "...",
  "description": "Drain a Kubernetes node safely.",
  "allowed_tools": ["k8s_get_nodes", "k8s_get_pods",
                    "k8s_cordon_node", "k8s_drain_node",
                    "k8s_uncordon_node"],
  "approval_policy": "elevated",
  "applicable_workspace_types": ["kubernetes"]
}

Customize the system_prompt: the procedure should enumerate pre-conditions (get nodes and pods, identify DaemonSet pods), the cordon/drain/verify sequence, and the uncordon rollback on failure. Do not allow tools outside allowed_tools. The approval_policy enum is standard, elevated, or restricted.

Step 3, create the template at project scope.

shell
erectl templates create --file drain-node-acme.json

Step 4, apply the template. Applying opens a chat, pinned to the current template version, in a chat workspace. A chat cannot live in an operational workspace, so link the new chat to the operational one where its tools run:

shell
erectl templates apply drain-node-acme --workspace ws_01HX... # a chat workspace erectl chats link-exec-workspace <chat_id> kubernetes-staging

In the returned chat, ask to drain a test node; approve each gated tool call; verify the final summary says every pod was evicted and no PDBs were violated.

Step 5, promote to platform scope. Once the template has run cleanly in staging, promote it so every project in the deployment can apply it:

shell
erectl templates promote drain-node-acme --scope platform

The promote verb only accepts --scope platform; workspace-to-workspace copy is not a supported operation. To run the template in a second workspace, call erectl templates apply again and link the new chat to that workspace.

2. Deploy EEM on a Jumphost

Goal: run an EEM on a locked-down jumphost via docker compose, with credentials mounted read-only and a systemd supervisor.

Start from the published compose.agent-eem.yaml in the erebine/container-agents repository (see Deploy Your First EEM for the base setup and the compose/.env), then layer a hardening override and a systemd supervisor on top of it. Target credentials (kubeconfig, AWS) go in the base file's read-only /data/erebine-eem/credentials mount.

Hardening override. Save alongside the base file as compose/compose.override.yml. Docker Compose merges it onto the base eem-agent service: read-only root filesystem, all capabilities dropped, no privilege escalation, and a writable tmpfs.

services:
  eem-agent:
    read_only: true
    cap_drop:
      - ALL
    security_opt:
      - no-new-privileges:true
    tmpfs:
      - /tmp

Systemd supervisor. Save as /etc/systemd/system/erebine-eem.service so the stack returns after a reboot. WorkingDirectory points at the cloned compose/ directory so both files and the .env resolve.

[Unit]
Description=Erebine EEM (docker compose)
After=docker.service network-online.target
Requires=docker.service
Wants=network-online.target

[Service]
Type=oneshot
RemainAfterExit=yes
WorkingDirectory=/opt/erebine-public/compose
ExecStart=/usr/bin/docker compose -f compose.agent-eem.yaml -f compose.override.yml up -d
ExecStop=/usr/bin/docker compose -f compose.agent-eem.yaml -f compose.override.yml down
TimeoutStartSec=0

[Install]
WantedBy=multi-user.target

Bootstrap sequence. Mint a join key and set it (with the router URL and registration name) in compose/.env, then enable the unit.

shell
erectl eems join-keys --create \ --name eem-jumphost-01 \ --region us-east-1 \ --router-addr tcp://router.example.com:5555 # Put the printed Join Token in compose/.env as EREBINE_EEM_JOIN_KEY=... sudo systemctl daemon-reload sudo systemctl enable --now erebine-eem journalctl -u erebine-eem -f erectl eems

--name, --region, and at least one --router-addr are required by the join-key creation endpoint. erectl eems join-keys mints the execution role into the key: an EEM enrolling with a key minted for another role is refused. An execution key names no tiers.

The unit is now supervised. Docker-compose restart policy handles process-level crashes; systemd handles boot-time recovery; the router re-admits the EEM on reconnect using its CURVE key.

3. Connect a Slack Bot to Approval Webhooks

Goal: deliver pending approvals to a Slack channel, let reviewers approve/reject from a Slack message, and round-trip the decision back to the router.

Step 1, create a Slack app. In api.slack.com/apps, create an app, enable Incoming Webhooks for a channel, and enable Interactivity with a request URL pointing at your bot service (for example https://bots.example.com/slack/interact). Note the signing secret.

Step 2, register a webhook endpoint with the router.

shell
erectl webhooks --create \ --url https://bots.example.com/approvals \ --events 'exec.approval_requested,exec.approval_timed_out' \ --secret "$(openssl rand -hex 32)"

Webhook subscription is a flat command, there is no endpoints subgroup. The signing secret is supplied via --secret; generate one locally (for example with openssl rand -hex 32) and store it alongside the bot service so signature verification can find it.

Step 3, handle exec.approval_requested. On receipt, verify the X-Webhook-Signature header against the signing secret, then post a Slack message with Approve / Reject buttons. The button value carries the approval ID.

Step 4, handle the Slack interaction. Verify Slack's signature. Read the button value to get the approval ID, then call the router:

POST /:project_id/v1/exec/approvals/:id/approve
Authorization: Bearer <bot-api-key>
Content-Type: application/json

{
  "note": "Approved via Slack by <user>"
}

The router validates that the bot's API key carries the execution scope and that the key's user holds a role that may decide the call: only a project owner approves a destructive or irreversible call, and an owner or operator approves a read or write call. Issue the bot's key to a user with the role your approvals need.

Step 5, close the loop. The router does not emit an approval-resolved webhook today; once the bot's POST to /approve (or /reject) returns 200, update the Slack message with the final decision and disable the buttons to prevent double-actions. Subscribe to exec.approval_timed_out to handle the timeout case from the router side.

See Webhook Events for the full payload schema.