Operator Guides Index
Task-first procedures, one operator, one outcome, five to fifteen minutes each. The order tracks the first-week lifecycle: deploy an Erebine Execution Manager (EEM), author a template against it, run the approval cadence that template drives, unwind a problem when one surfaces.
1. Deploy Your First EEM
Goal: take a fresh host with cluster credentials and get it serving tool calls inside an Erebine project in under five minutes.
- On the EEM host, put each workspace's credentials under
credentials_dir/<workspace>/, one file per credential family, and name each in the workspace'scredentialsmap: a kubeconfig as itskubernetesentry, a one-cloudclouds.yamlas itsopenstackentry. See Workspace credentials. A workspace without one runs on the EEM's own identity, so run one EEM per credential boundary. - From your local workstation, mint a join key:
erectl eems join-keys --create --name <name> --region <region> --router-addr tcp://<router-host>:5555. The command prints the join-key value; capture it once. - Copy the value to the EEM host and export it as
EREBINE_EEM_JOIN_KEY(legacy:EEM_JOIN_KEY) before launching the container. Keep it off the command line: the--join-keyflag shows the key to every account on the host inps. - Run
erectl eems, the new EEM should appear in the listing with a healthy status. - Inspect what it published with
erectl eems <eem-id>. The default handler renders the registered tool surface and last-heartbeat fields the EEM advertised.
The EEM is now reachable from any operational workspace bound
to it. Bind one with
erectl workspaces bind-eem <workspace-id> <eem-id>.
Env vars worth knowing:
EREBINE_EEM_JOIN_KEY,
EREBINE_EEM_ROUTER_URL,
EREBINE_EEM_REGISTRATION_NAME,
EREBINE_EEM_LOG_LEVEL,
EREBINE_EEM_METRICS_PORT. The first four
also answer to a legacy EEM_* spelling;
the Environment
Variables page lists every name.
3. Manage Approvals
Goal: configure who approves what, and in what cadence, for every operational workspace.
- List the open approvals operators are waiting on with
erectl approvals list [--risk <level>]. - Inspect a single request, prompt, tool call, risk,
timeout, with
erectl approvals show <approval-id>. - Stream new requests as they land:
erectl approvals watch [--risk <level>]prints an SSE feed suitable for piping into a notifier. - Decide a request from the CLI with
erectl approvals approve <approval-id> --note <reason>orerectl approvals reject <approval-id> --reason <reason>. Reviewers may also act on the Operations page. - Who may decide is set by project role, not by a
policy: only an owner approves a
destructiveorirreversiblecall, and an owner or operator approves areadorwritecall. Approval-policy authoring (per-risk timeouts and escalation chains) lives in the workspace settings UI today; there is noerectl approvals policysubcommand. If your alert payload mentionsretry_afterseconds or returns aRetry-Afterheader, wait that long before re-issuing the call, the router's rate-limit middleware sets both.
4. Respond to Alerts
Goal: triage a Prometheus alert the fleet emitted.
- Look at the alert label set. Every EEM alert carries
project,workspace, andagent. The label set names the blast radius. - Open
/opsand filter Activity by the same workspace and EEM. - Open the run. Its detail shows the arguments, the output and the timeline.
- If the EEM is misbehaving (wrong output, stale
credentials), pause its binding with
erectl workspaces pause-eem <workspace-id> <eem-id>. While paused, dispatch to that workspace refuses withworkspace_binding_pausedrather than rerouting. Resume witherectl workspaces resume-eem <workspace-id> <eem-id>once the host is healthy. - If the target is misbehaving (cluster outage, cloud API
throttle), pause every EEM bound to the workspace the
same way until the upstream issue resolves. Cancel a
running call with
erectl exec cancel <id>or Cancel on the Operations page. - Watch the error envelope. Common operator-visible codes
include
endpoint_inactive,model_provisioning,capacity_exceeded,backend_unavailable, andrate_limit_exceeded. A 403 withscope_insufficientmeans the caller's API key lacks a required scope; a 500 carrying ax-request-idresponse header should be forwarded to platform support together with the timestamp. - When the alert resolves, Prometheus will mark the rule inactive. If it does not, cross-reference Troubleshooting for the specific error code.
5. Investigate an Execution
Goal: understand, in full, what a specific tool invocation did.
- Find the invocation ID. The user will usually paste it
from the chat; the
/opsdashboard exposes it via a copy button on every row. - Inspect the call:
erectl exec show <id>. The output names the tool, risk, status, latency and any error, then prints the parameters and the result. Artifacts produced by the call are referenced in the result; fetch them through the chat's artifact API (thex_read_artifacttool surface) or the dashboard artifact viewer, there is noerectl artifactscommand group. - For distributed traces, enable OTEL on the router with
EREBINE_OTEL_ENABLED=trueandEREBINE_OTEL_ENDPOINT=<collector>and query the configured tracing backend (Jaeger, Tempo, etc.) by request ID. There is no CLI wrapper for trace retrieval today. - For post-incident review, capture the full JSON shape of
the execution with
erectl exec show <id> -o jsonand attach it to the incident record. The router's audit log has no retention window: nothing ages an execution's record out. In a zero-data-retention workspace the record keeps the tool name, outcome and timing only, and its arguments and result are dropped when the call finishes.
6. Rotate Credentials
Goal: rotate an EEM's tool credentials (or its CURVE keys) without dropping in-flight work.
- Tool credentials: replace the
credential file on the EEM host (a new kubeconfig, a new
~/.aws/credentials) as Credential Rotation shows. The next tool call reads it; the EEM needs no restart, and a call already running finishes on the credential it read. - CURVE keys: run
erectl eems curve-rotate <eem-id>. The router asks the EEM, over its existing connection, to generate a fresh Curve25519 keypair. The EEM sends back the new public key, signed with its signing key; the router verifies it, records it, and tells the EEM to switch. The router keeps accepting the old key for a five-minute overlap window, and the audit log records the rotation. - Join keys: expire immediately unused ones
with
erectl eems join-keys <id> --revoke. Reissue with a fresherectl eems join-keys --create --name <name> --region <region> --router-addr tcp://<router-host>:5555call. - After any rotation, run
erectl eems <eem-id>and confirm the EEM's heartbeat advanced. A fresh heartbeat is evidence the rotation took effect.