docs
// EEM Guides

Incident Response

Diagnose, contain, and recover from an Erebine Execution Manager (EEM) execution incident: a wrong tool result, a destructive call that bypassed approval, a stuck EEM, a misbehaving tool. Triage, contain, investigate, remediate, postmortem. Do not skip steps.

Triage

Within the first 5 minutes, answer three questions:

  1. What executions are affected? erectl exec history --from <iso> --to <iso> --status <s> lists the recent tool runs in a window. Cross-reference the approvals queue with erectl approvals list.
  2. What blast radius? Run erectl exec show <executionId> for each suspect execution. Look at the workspace, the tool, and the exit-code / output.
  3. Is the EEM still healthy? erectl health dashboard shows the project-wide service health. erectl eems <id> --events prints the lifecycle event stream for a single EEM.

Decide: contain locally (pause one workspace's binding to the EEM) or contain widely (pause every binding of every affected EEM).

Contain

The containment primitive is the binding pause. It stops dispatch from one workspace to one bound EEM and keeps the binding, so resuming restores exactly the routing that was in place. While paused, dispatch refuses with workspace_binding_paused rather than silently rerouting to another EEM. The workspace argument is its ws_ id; the EEM argument is its UUID.

bash
erectl workspaces pause-eem ws_01HX... 3f9c1d2b-8a04-4c1e-9b6f-0d2e7a1c5b90 \ --reason "INC-1234" erectl workspaces resume-eem ws_01HX... 3f9c1d2b-8a04-4c1e-9b6f-0d2e7a1c5b90

The --reason text is shown to anyone whose call hits the pause. To contain a whole workspace, pause every EEM bound to it. To contain an EEM, pause each of its bindings.

Cancel a running call: erectl exec cancel <id> or Cancel on the Operations page; the EEM stops it.

Investigate

Sources you will consult in every incident:

  1. Execution record. erectl exec show <executionId> reports the execution status, workspace, tool, and result metadata, and erectl exec watch <executionId> follows a live execution. There is no CLI surface for the audit log; use Export audit on the Operations page.
  2. Approval queue. erectl approvals list and erectl approvals show <id> surface stuck or rejected approvals. Destructive-call gating routes through this queue, so incidents often correlate with expired entries.
  3. EEM events. erectl eems <id> --events prints the lifecycle event stream for the suspected EEM.
  4. Structured logs on the EEM. sudo journalctl -u erebine-eem --since "30 minutes ago" --grep <executionId> on the EEM host.
  5. Distributed trace. With EREBINE_OTEL_ENABLED=true (and an EREBINE_OTEL_ENDPOINT configured) the trace backend holds the per-frame timing from router to EEM to tool to result. Tune sample rate via EREBINE_OTEL_SAMPLE_RATE.

Correlate by execution id. The execution record and traces share the same id; logs print it on every line.

Remediate

Typical remediation paths, from cheapest to most invasive:

  • Inspect the auto-fork-branch. If the tool call triggered auto-fork-branch, the parent chat has an unreached branch recorded immediately before the destructive call. Locate it with erectl chats branches list <chatId>. There is no CLI rollback verb today; consume the branch through the chat UI or document the manual recovery steps in the postmortem.
  • Corrective tool call. Run a read-only probe first, then a reverse mutation (e.g. kubectl apply of the pre-change manifest).
  • Rotate credentials. If the incident indicates credential leak, see Credential Rotation.
  • Rotate CURVE keys. If the incident indicates CURVE-key leak, run erectl eems curve-rotate <eem-id> [--reason <text>].
  • Remove the EEM. Use erectl eems <eem-id> --delete for permanent removal.

Postmortem

After containment, a written postmortem is expected for every incident that altered production state unexpectedly. Minimum fields:

  • Incident ID and window.
  • Affected invocations (with IDs).
  • Root cause (template, tool, credential, or model misbehavior).
  • Detection path and time-to-detect.
  • Remediation actions.
  • Preventive follow-ups (chat template update, tool risk reclassification, approval policy tightening).

Attach an execution archive to the postmortem. Generate one with erectl exports create --type executions --since <ts> --until <ts>, then poll with erectl exports status and pull the bundle via erectl exports download.