docs
// EEM Guides

Incident Response

Diagnose, contain, and recover from an EEM execution incident: a wrong tool result, a destructive call that bypassed approval, a stuck agent, a misbehaving tool. Triage, contain, investigate, remediate, postmortem. Do not skip steps.

Triage

Within the first 5 minutes, answer three questions:

  1. What executions are affected? erectl exec list returns the recent execution set. There are no built-in time-window or status filters; narrow the result client-side (for example by piping to jq) or cross-reference the approvals queue with erectl approvals list.
  2. What blast radius? Run erectl exec show <executionId> for each suspect execution. Look at the workspace, the tool, and the exit-code / output. Use erectl exec raw <executionId> for the raw output blob.
  3. Is the EEM agent still healthy? erectl health dashboard shows the project-wide service health. erectl agents <id> --events prints the lifecycle event stream for a single agent.

Decide: contain locally (one workspace via a scoped agent pause) or contain globally (pause every EEM agent).

Contain

The containment primitive is per-agent: erectl agents pause-exec <agentId>. To scope the pause to a single workspace's dispatch on that agent, pass --workspace <name>. There is no workspace-wide pause command; contain a workspace by pausing each agent that serves it.

bash
erectl agents pause-exec agt_eem_fleet_01 \ --workspace kubernetes-prod

A paused agent accepts no new tool invocations on the scoped workspace; in-flight calls complete. Record the incident id in your tracker; the CLI does not accept a reason flag, so attach it to the postmortem instead.

If the agent itself is the problem, omit --workspace to pause all of its dispatch:

bash
erectl agents pause-exec agt_eem_fleet_01

The agent's lease remains valid but it no longer receives dispatch frames; the router queues any pending invocations for redispatch once you run erectl agents resume-exec <agentId>.

Investigate

Sources you will consult in every incident:

  1. Execution record. erectl exec show <executionId> reports the execution status, workspace, tool, and result metadata. erectl exec raw <executionId> returns the raw output blob, and erectl exec watch <executionId> follows a live execution. There is no CLI surface for the immutable audit log; view audit_logs rows through the operator dashboard.
  2. Approval queue. erectl approvals list and erectl approvals show <id> surface stuck or rejected approvals. Destructive-call gating routes through this queue, so incidents often correlate with expired entries.
  3. Agent events. erectl agents <id> --events prints the lifecycle event stream for the suspected agent.
  4. Structured logs on the agent. sudo journalctl -u erebine-eem --since "30 minutes ago" --grep <executionId> on the agent host.
  5. Distributed trace. With EREBINE_OTEL_ENABLED=true (and an EREBINE_OTEL_ENDPOINT configured) the trace backend holds the per-frame timing from router to agent to tool to result. Tune sample rate via EREBINE_OTEL_SAMPLE_RATE.

Correlate by execution id. The execution record and traces share the same id; logs print it on every line.

Remediate

Typical remediation paths, from cheapest to most invasive:

  • Inspect the auto-fork-branch. If the tool call triggered auto-fork-branch, the parent chat has an unreached branch recorded immediately before the destructive call. Locate it with erectl chats branches list <chatId>. There is no CLI rollback verb today; consume the branch through the chat UI or document the manual recovery steps in the postmortem.
  • Corrective tool call. Run a read-only probe first, then a reverse mutation (e.g. kubectl apply of the pre-change manifest).
  • Rotate credentials. If the incident indicates credential leak, see Credential Rotation.
  • Rotate CURVE keys. If the incident indicates CURVE-key leak, run erectl agents curve-rotate <agentId> [--reason <text>].
  • Revoke the agent. Use erectl agents revoke-credentials <agentId> to invalidate the agent's credentials while keeping the record, or erectl agents <agentId> --delete for permanent removal.

Postmortem

After containment, a written postmortem is expected for every incident that altered production state unexpectedly. Minimum fields:

  • Incident ID and window.
  • Affected invocations (with IDs).
  • Root cause (template, tool, credential, or model misbehavior).
  • Detection path and time-to-detect.
  • Remediation actions.
  • Preventive follow-ups (chat template update, tool risk reclassification, approval policy tightening).

Attach an execution archive to the postmortem. Generate one with erectl exports create --type executions --since <ts> --until <ts>, then poll with erectl exports status and pull the bundle via erectl exports download.