Incident Response
Diagnose, contain, and recover from an Erebine Execution Manager (EEM) execution incident: a wrong tool result, a destructive call that bypassed approval, a stuck EEM, a misbehaving tool. Triage, contain, investigate, remediate, postmortem. Do not skip steps.
Triage
Within the first 5 minutes, answer three questions:
- What executions are affected?
erectl exec history --from <iso> --to <iso> --status <s>lists the recent tool runs in a window. Cross-reference the approvals queue witherectl approvals list. - What blast radius? Run
erectl exec show <executionId>for each suspect execution. Look at the workspace, the tool, and the exit-code / output. - Is the EEM still healthy?
erectl health dashboardshows the project-wide service health.erectl eems <id> --eventsprints the lifecycle event stream for a single EEM.
Decide: contain locally (pause one workspace's binding to the EEM) or contain widely (pause every binding of every affected EEM).
Contain
The containment primitive is the binding pause. It
stops dispatch from one workspace to one bound EEM
and keeps the binding, so resuming restores exactly
the routing that was in place. While paused,
dispatch refuses with
workspace_binding_paused rather than
silently rerouting to another EEM. The workspace
argument is its ws_ id; the EEM
argument is its UUID.
erectl workspaces pause-eem ws_01HX... 3f9c1d2b-8a04-4c1e-9b6f-0d2e7a1c5b90 \
--reason "INC-1234"
erectl workspaces resume-eem ws_01HX... 3f9c1d2b-8a04-4c1e-9b6f-0d2e7a1c5b90
The --reason text is shown to anyone
whose call hits the pause. To contain a whole
workspace, pause every EEM bound to it. To contain
an EEM, pause each of its bindings.
Cancel a running call: erectl exec cancel <id>
or Cancel on the Operations page; the EEM stops it.
Investigate
Sources you will consult in every incident:
- Execution record.
erectl exec show <executionId>reports the execution status, workspace, tool, and result metadata, anderectl exec watch <executionId>follows a live execution. There is no CLI surface for the audit log; use Export audit on the Operations page. - Approval queue.
erectl approvals listanderectl approvals show <id>surface stuck or rejected approvals. Destructive-call gating routes through this queue, so incidents often correlate with expired entries. - EEM events.
erectl eems <id> --eventsprints the lifecycle event stream for the suspected EEM. - Structured logs on the EEM.
sudo journalctl -u erebine-eem --since "30 minutes ago" --grep <executionId>on the EEM host. - Distributed trace. With
EREBINE_OTEL_ENABLED=true(and anEREBINE_OTEL_ENDPOINTconfigured) the trace backend holds the per-frame timing from router to EEM to tool to result. Tune sample rate viaEREBINE_OTEL_SAMPLE_RATE.
Correlate by execution id. The execution record and traces share the same id; logs print it on every line.
Remediate
Typical remediation paths, from cheapest to most invasive:
- Inspect the auto-fork-branch.
If the tool call triggered auto-fork-branch,
the parent chat has an unreached branch
recorded immediately before the destructive
call. Locate it with
erectl chats branches list <chatId>. There is no CLI rollback verb today; consume the branch through the chat UI or document the manual recovery steps in the postmortem. - Corrective tool call. Run a
read-only probe first, then a reverse mutation
(e.g.
kubectl applyof the pre-change manifest). - Rotate credentials. If the incident indicates credential leak, see Credential Rotation.
- Rotate CURVE keys. If the
incident indicates CURVE-key leak, run
erectl eems curve-rotate <eem-id> [--reason <text>]. - Remove the EEM. Use
erectl eems <eem-id> --deletefor permanent removal.
Postmortem
After containment, a written postmortem is expected for every incident that altered production state unexpectedly. Minimum fields:
- Incident ID and window.
- Affected invocations (with IDs).
- Root cause (template, tool, credential, or model misbehavior).
- Detection path and time-to-detect.
- Remediation actions.
- Preventive follow-ups (chat template update, tool risk reclassification, approval policy tightening).
Attach an execution archive to the postmortem.
Generate one with erectl exports create
--type executions --since <ts> --until
<ts>, then poll with
erectl exports status and pull the
bundle via erectl exports download.