docs
// Execution Management (EEM)

Operator Guides Index

Task-first procedures, one operator, one outcome, five to fifteen minutes each. The order tracks the first-week lifecycle: deploy an Erebine Execution Manager (EEM), author a template against it, run the approval cadence that template drives, unwind a problem when one surfaces.

1. Deploy Your First EEM

Goal: take a fresh host with cluster credentials and get it serving tool calls inside an Erebine project in under five minutes.

  1. On the EEM host, put each workspace's credentials under credentials_dir/<workspace>/, one file per credential family, and name each in the workspace's credentials map: a kubeconfig as its kubernetes entry, a one-cloud clouds.yaml as its openstack entry. See Workspace credentials. A workspace without one runs on the EEM's own identity, so run one EEM per credential boundary.
  2. From your local workstation, mint a join key: erectl eems join-keys --create --name <name> --region <region> --router-addr tcp://<router-host>:5555. The command prints the join-key value; capture it once.
  3. Copy the value to the EEM host and export it as EREBINE_EEM_JOIN_KEY (legacy: EEM_JOIN_KEY) before launching the container. Keep it off the command line: the --join-key flag shows the key to every account on the host in ps.
  4. Run erectl eems, the new EEM should appear in the listing with a healthy status.
  5. Inspect what it published with erectl eems <eem-id>. The default handler renders the registered tool surface and last-heartbeat fields the EEM advertised.

The EEM is now reachable from any operational workspace bound to it. Bind one with erectl workspaces bind-eem <workspace-id> <eem-id>.

Env vars worth knowing: EREBINE_EEM_JOIN_KEY, EREBINE_EEM_ROUTER_URL, EREBINE_EEM_REGISTRATION_NAME, EREBINE_EEM_LOG_LEVEL, EREBINE_EEM_METRICS_PORT. The first four also answer to a legacy EEM_* spelling; the Environment Variables page lists every name.

2. Author a Chat Template

Goal: encode a recurring procedure (a runbook successor) as a chat template so operators and automation follow the same plan every time.

  1. Start from a known-good baseline: dump the nearest platform template to a local file with erectl templates show <template-id> -o json > local.json.
  2. Edit the JSON; the fields you change most are name, system_prompt, allowed_tools, and approval_policy.
  3. Write the system prompt as a numbered procedure. State the pre-conditions, the step sequence, the post-conditions, and what to do on failure. Keep it explicit about which steps MUST NOT be skipped.
  4. Restrict allowed_tools to the minimum set required. Less is safer.
  5. Create the new template from the edited file with erectl templates create --file local.json. To start a chat from it, run erectl templates apply <template-id> --workspace <chat-workspace-id>; the chat lands in that chat workspace. Link it to the operational workspace with erectl chats link-exec-workspace.
  6. Dry-run in a test workspace before letting the template touch production. The template-authoring tutorial walks through the drain-node example end to end.

3. Manage Approvals

Goal: configure who approves what, and in what cadence, for every operational workspace.

  1. List the open approvals operators are waiting on with erectl approvals list [--risk <level>].
  2. Inspect a single request, prompt, tool call, risk, timeout, with erectl approvals show <approval-id>.
  3. Stream new requests as they land: erectl approvals watch [--risk <level>] prints an SSE feed suitable for piping into a notifier.
  4. Decide a request from the CLI with erectl approvals approve <approval-id> --note <reason> or erectl approvals reject <approval-id> --reason <reason>. Reviewers may also act on the Operations page.
  5. Who may decide is set by project role, not by a policy: only an owner approves a destructive or irreversible call, and an owner or operator approves a read or write call. Approval-policy authoring (per-risk timeouts and escalation chains) lives in the workspace settings UI today; there is no erectl approvals policy subcommand. If your alert payload mentions retry_after seconds or returns a Retry-After header, wait that long before re-issuing the call, the router's rate-limit middleware sets both.

4. Respond to Alerts

Goal: triage a Prometheus alert the fleet emitted.

  1. Look at the alert label set. Every EEM alert carries project, workspace, and agent. The label set names the blast radius.
  2. Open /ops and filter Activity by the same workspace and EEM.
  3. Open the run. Its detail shows the arguments, the output and the timeline.
  4. If the EEM is misbehaving (wrong output, stale credentials), pause its binding with erectl workspaces pause-eem <workspace-id> <eem-id>. While paused, dispatch to that workspace refuses with workspace_binding_paused rather than rerouting. Resume with erectl workspaces resume-eem <workspace-id> <eem-id> once the host is healthy.
  5. If the target is misbehaving (cluster outage, cloud API throttle), pause every EEM bound to the workspace the same way until the upstream issue resolves. Cancel a running call with erectl exec cancel <id> or Cancel on the Operations page.
  6. Watch the error envelope. Common operator-visible codes include endpoint_inactive, model_provisioning, capacity_exceeded, backend_unavailable, and rate_limit_exceeded. A 403 with scope_insufficient means the caller's API key lacks a required scope; a 500 carrying a x-request-id response header should be forwarded to platform support together with the timestamp.
  7. When the alert resolves, Prometheus will mark the rule inactive. If it does not, cross-reference Troubleshooting for the specific error code.

5. Investigate an Execution

Goal: understand, in full, what a specific tool invocation did.

  1. Find the invocation ID. The user will usually paste it from the chat; the /ops dashboard exposes it via a copy button on every row.
  2. Inspect the call: erectl exec show <id>. The output names the tool, risk, status, latency and any error, then prints the parameters and the result. Artifacts produced by the call are referenced in the result; fetch them through the chat's artifact API (the x_read_artifact tool surface) or the dashboard artifact viewer, there is no erectl artifacts command group.
  3. For distributed traces, enable OTEL on the router with EREBINE_OTEL_ENABLED=true and EREBINE_OTEL_ENDPOINT=<collector> and query the configured tracing backend (Jaeger, Tempo, etc.) by request ID. There is no CLI wrapper for trace retrieval today.
  4. For post-incident review, capture the full JSON shape of the execution with erectl exec show <id> -o json and attach it to the incident record. The router's audit log has no retention window: nothing ages an execution's record out. In a zero-data-retention workspace the record keeps the tool name, outcome and timing only, and its arguments and result are dropped when the call finishes.

6. Rotate Credentials

Goal: rotate an EEM's tool credentials (or its CURVE keys) without dropping in-flight work.

  1. Tool credentials: replace the credential file on the EEM host (a new kubeconfig, a new ~/.aws/credentials) as Credential Rotation shows. The next tool call reads it; the EEM needs no restart, and a call already running finishes on the credential it read.
  2. CURVE keys: run erectl eems curve-rotate <eem-id>. The router asks the EEM, over its existing connection, to generate a fresh Curve25519 keypair. The EEM sends back the new public key, signed with its signing key; the router verifies it, records it, and tells the EEM to switch. The router keeps accepting the old key for a five-minute overlap window, and the audit log records the rotation.
  3. Join keys: expire immediately unused ones with erectl eems join-keys <id> --revoke. Reissue with a fresh erectl eems join-keys --create --name <name> --region <region> --router-addr tcp://<router-host>:5555 call.
  4. After any rotation, run erectl eems <eem-id> and confirm the EEM's heartbeat advanced. A fresh heartbeat is evidence the rotation took effect.