docs
// Tools

erectl eims & eems

Operate EIMs and EEMs from a shell: enroll them with join keys, read their event history and live logs, and rotate an EEM's CURVE key. Project-scoped, context-aware.

Overview

An Erebine Inference Microservice (EIM) serves models on accelerators for your project. It registers with the platform, receives model assignments through the sync system, and reports health through heartbeats. An Erebine Execution Manager (EEM) runs tool and shell executions for your workspaces.

Two command groups manage them. Each lists only its own kind:

  • erectl eims, the project's EIMs: list, get, create, update, delete and events (default), plus join-keys, set-location and logs.
  • erectl eems, the project's EEMs: list, get, update, delete, events, disable, enable and reset (default), plus join-keys, logs, manifest and curve-rotate.

A command that takes an id acts on that id. The listing is what the group scopes: erectl eims shows rows enrolled as inference, erectl eems rows enrolled as execution.

To stop dispatch to an EEM for one workspace, use erectl workspaces pause-eem.

The default action of each group and of join-keys is chosen by flag: the action is determined by which flag is present, and only one action flag may be specified at a time. The remaining subcommands are positional verbs.

Deployment-aware: eims and eems resolve the base URL and API key from the active erectl deployment (the project id is derived from base_url). Requests go to the router management API under /v1/management/. Use erectl config use <name> to switch deployments; override a single invocation with --context, --base-url, or --api-key.

Pagination flags: --limit <n> and --after <cursor> apply to every listing (eims, eems, --events, join-keys). Per-command tables below list only command-specific flags.

Global flag: --dry-run is a global erectl flag (not specific to delete or revoke). When present on a mutating subcommand it prints the intended action without contacting the router.

Destructive subcommands write router state. Reach for --dry-run first, --force only after.

eims usage pattern

Positional arguments documented below as <eim-id> are required unless flagged otherwise.

bash
erectl eims # List EIMs erectl eims <id> # Show EIM details erectl eims --create --name <name> \ --worker-id <wid> --public-key <b64> --region <r> # Create an EIM record erectl eims <id> --update --name <name> # Update an EIM erectl eims <id> --delete # Delete an EIM erectl eims <id> --events # List EIM events erectl eims <id> --disable|--enable|--reset # Park, unpark or reset an EIM erectl eims set-location <id> --country US # Set the serving location erectl eims logs <id> # Stream the EIM's console logs erectl eims join-keys # List inference join keys

eims list

List the project's EIMs. This is the default action when no ID or action flag is given.

bash
erectl eims erectl eims --limit 50 erectl eims --after 3f9c1d2b-8a04-4c1e-9b6f-0d2e7a1c5b90

Accepts the shared pagination flags --limit and --after (see Overview).

Output columns: ID, NAME, WORKER_ID, STATUS, REGION, PRESEED, LAST_HEARTBEAT. The STATUS column is color-coded; PRESEED shows whether the EIM may register models from its local disk.

eims get

Show details of one EIM: worker ID, status, region, last heartbeat, slots (safe slots the KV cache holds and admit slots the EIM accepts, e.g. 11 safe / 16 admit), supported tiers, model preseed permission, and description.

bash
erectl eims 3f9c1d2b-8a04-4c1e-9b6f-0d2e7a1c5b90 erectl eims 3f9c1d2b-8a04-4c1e-9b6f-0d2e7a1c5b90 -o json

eims create

Create an EIM record. --name, --worker-id, --public-key, and --region are all required. EIMs normally enroll themselves with a join key; an administrator can also pre-create the record by supplying the worker identifier and the EIM's base64-encoded Curve25519 public key directly. EEMs enroll only through join keys, so erectl eems has no --create.

bash
# Minimal creation (all four flags are required) erectl eims --create \ --name "gpu-node-1" \ --worker-id w-001 \ --public-key <base64-curve25519-public-key> \ --region us-east # With capacity, description, and local-disk model imports allowed erectl eims --create \ --name "gpu-node-2" \ --worker-id w-002 \ --public-key <base64-curve25519-public-key> \ --region us-east \ --max-concurrent 8 \ --description "H100 EIM for production" \ --supported-tiers base,plus \ --allow-preseed

Options

Option Description
--name <name> EIM name (required)
--worker-id <id> Mesh worker identifier (required)
--public-key <b64> Base64-encoded Curve25519 public key of the EIM (required)
--region <region> Deployment region, e.g. us-east, eu-west (required)
--max-concurrent <n> Maximum number of concurrent inference requests
--description <text> Human-readable description of the EIM
--supported-tiers <tiers> Comma-separated service tiers this EIM handles (list slugs with erectl tiers)
--allow-preseed Let the EIM register models imported from its local disk. Off unless given.

eims update

Update an existing EIM. At least one of the supported fields must be provided.

bash
erectl eims 3f9c1d2b-8a04-4c1e-9b6f-0d2e7a1c5b90 --update --name "gpu-node-1-renamed" erectl eims 3f9c1d2b-8a04-4c1e-9b6f-0d2e7a1c5b90 --update --max-concurrent 16 erectl eims 3f9c1d2b-8a04-4c1e-9b6f-0d2e7a1c5b90 --update --description "Updated H100 EIM" erectl eims 3f9c1d2b-8a04-4c1e-9b6f-0d2e7a1c5b90 --update --status suspended erectl eims 3f9c1d2b-8a04-4c1e-9b6f-0d2e7a1c5b90 --update --region us-west erectl eims 3f9c1d2b-8a04-4c1e-9b6f-0d2e7a1c5b90 --update --supported-tiers bulk,base erectl eims 3f9c1d2b-8a04-4c1e-9b6f-0d2e7a1c5b90 --update --allow-preseed erectl eims 3f9c1d2b-8a04-4c1e-9b6f-0d2e7a1c5b90 --update --no-allow-preseed

Options

Option Description
--name <name> New EIM name
--description <text> New description
--max-concurrent <n> New maximum concurrent request count
--region <region> New deployment region
--status <value> New status value (e.g. active, suspended)
--supported-tiers <tiers> Comma-separated supported service tiers (list slugs with erectl tiers)
--allow-preseed, --no-allow-preseed Grant or revoke permission to register models imported from the EIM's local disk. Unset leaves the value alone.

The worker identifier and the public key are fixed at enrollment; re-enroll to change them.

eims delete

Delete an EIM from the project. A confirmation prompt is shown by default.

bash
# With confirmation prompt erectl eims 3f9c1d2b-8a04-4c1e-9b6f-0d2e7a1c5b90 --delete # Skip confirmation erectl eims 3f9c1d2b-8a04-4c1e-9b6f-0d2e7a1c5b90 --delete --force # Dry run (--dry-run is a global erectl flag) erectl eims 3f9c1d2b-8a04-4c1e-9b6f-0d2e7a1c5b90 --delete --dry-run

--force skips the confirmation prompt. --dry-run is a global erectl flag (see Overview).

A deleted EIM stays down. It leaves terminated-by-router in its config directory, and while that file is there it does not enroll again with the join key it was deployed with: the container waits and its log says why. To bring it back, run erectl deploy for that host again, which removes the file and enrolls the EIM again, or remove the file yourself and give the EIM a valid join key.

eims events

List event history for one EIM. Events include lifecycle changes, errors, and operational events.

bash
erectl eims 3f9c1d2b-8a04-4c1e-9b6f-0d2e7a1c5b90 --events erectl eims 3f9c1d2b-8a04-4c1e-9b6f-0d2e7a1c5b90 --events --limit 50 erectl eims 3f9c1d2b-8a04-4c1e-9b6f-0d2e7a1c5b90 --events --after evt_xyz

Accepts the shared pagination flags --limit and --after (see Overview); the cursor is an event ID.

Output columns: ID, EVENT_TYPE, DETAILS, CREATED.

eims set-location

Record the country and region an EIM serves from. A marketplace listing publishes the pair as its datacenter. Neither value is derived: nothing on the wire proves where a host sits, so the operator states it.

bash
erectl eims set-location 3f9c1d2b-8a04-4c1e-9b6f-0d2e7a1c5b90 --country US erectl eims set-location 3f9c1d2b-8a04-4c1e-9b6f-0d2e7a1c5b90 --region us-east erectl eims set-location 3f9c1d2b-8a04-4c1e-9b6f-0d2e7a1c5b90 --country DE --region eu-central

Options

Option Description
--country <code> ISO 3166-1 alpha-2 country, matched case-insensitively and stored uppercase. Unassigned codes are refused, including XX.
--region <region> Deployment region label

Give at least one of the two. An omitted value keeps the stored one.

eims logs

Stream an EIM's console logs until Ctrl-C. Recent buffered lines print first, then live output.

bash
erectl eims logs 3f9c1d2b-8a04-4c1e-9b6f-0d2e7a1c5b90 erectl eims logs 3f9c1d2b-8a04-4c1e-9b6f-0d2e7a1c5b90 --raw

--raw prints only the log text, without the timestamp and source prefix.

eems usage pattern

Positional arguments documented below as <eem-id> are required unless flagged otherwise.

bash
erectl eems # List EEMs erectl eems <id> # Show EEM details erectl eems <id> --update --name <name> # Update an EEM erectl eems <id> --delete # Delete an EEM erectl eems <id> --events # List EEM events erectl eems <id> --disable # Park an EEM; it stays enrolled erectl eems <id> --enable # Return a disabled EEM to service erectl eems <id> --reset # Drop the EEM's state and restart it erectl eems logs <id> # Stream the EEM's console logs erectl eems manifest <id> # Show the EEM's tools and workspaces erectl eems curve-rotate <id> # Rotate the EEM's CURVE key erectl eems workspaces <id> # List the EEM's workspaces erectl eems add-workspace <id> <workspace> # Bind a workspace and declare it erectl eems remove-workspace <id> <workspace> # Unbind a workspace and remove it erectl eems policy <id> [settings] # Show or set the EEM's approval policy erectl eems join-keys # List execution join keys

To deploy an EEM on a host, see erectl deploy eem.

eems list

List the project's EEMs. This is the default action when no ID or action flag is given.

bash
erectl eems erectl eems --limit 50 erectl eems -o json

Accepts the shared pagination flags --limit and --after (see Overview).

eems get

Show details of one EEM: worker ID, status, region, last heartbeat, and description. With -o json the record also carries agent_role and the hash of the capability manifest the EEM advertised.

bash
erectl eems 7d41a0c2-5b9e-4f13-8c2a-1e6b3f9d0a47 erectl eems 7d41a0c2-5b9e-4f13-8c2a-1e6b3f9d0a47 -o json

eems update

Update an existing EEM. At least one field must be provided. The options match eims update except --allow-preseed: an EEM runs tools, not models, so it has no models to import.

bash
erectl eems 7d41a0c2-5b9e-4f13-8c2a-1e6b3f9d0a47 --update --name "eem-build-1" erectl eems 7d41a0c2-5b9e-4f13-8c2a-1e6b3f9d0a47 --update --description "CI tool runner" erectl eems 7d41a0c2-5b9e-4f13-8c2a-1e6b3f9d0a47 --update --status suspended

eems delete

Delete an EEM from the project. A confirmation prompt is shown by default; --force skips it.

bash
erectl eems 7d41a0c2-5b9e-4f13-8c2a-1e6b3f9d0a47 --delete erectl eems 7d41a0c2-5b9e-4f13-8c2a-1e6b3f9d0a47 --delete --force erectl eems 7d41a0c2-5b9e-4f13-8c2a-1e6b3f9d0a47 --delete --dry-run

A deleted EEM stays down. It leaves terminated-by-router in its config directory, and while that file is there it does not enroll again with the join key it was deployed with: it waits and its log says why. To bring it back, run erectl deploy eem for that host again, which removes the file and enrolls the EEM again, or remove the file yourself and start the EEM with a valid join key. An EEM in the Erebine desktop app stops instead, and starts again when you start it.

eems disable, enable and reset

The same three actions as the Disable, Enable and Reset buttons on the EEMs dashboard, with the same effect and the same audit record. Owners and operators only; a member's key is refused with 403.

  • --disable parks the EEM. It stays enrolled and connected, refuses tool calls, and no workspace dispatches to it until it is enabled. A suspended or already disabled EEM is refused with 409.
  • --enable returns a disabled EEM to service. An EEM that is not disabled is refused with 409.
  • --reset tells the EEM to drop its state and restart. Its enrollment stays. A disabled EEM comes back still disabled.

--reason is recorded with --disable and --reset in the EEM's events and the audit log. --dry-run prints the action and sends nothing; -o json prints {"id": "<eem-id>", "status": "disabled"} (enabled, or dispatched for a reset). erectl eims takes the same three flags for an EIM.

bash
erectl eems 7d41a0c2-5b9e-4f13-8c2a-1e6b3f9d0a47 --disable --reason "host patching" erectl eems 7d41a0c2-5b9e-4f13-8c2a-1e6b3f9d0a47 --enable erectl eems 7d41a0c2-5b9e-4f13-8c2a-1e6b3f9d0a47 --reset --dry-run

eems events

List event history for one EEM, newest first: each enrollment (enrolled, with reenrollment=true when the same host enrolled again), key rotations (agent.key_rotated), and each disable, enable and reset. Output columns: ID, EVENT_TYPE, DETAILS, CREATED.

bash
erectl eems 7d41a0c2-5b9e-4f13-8c2a-1e6b3f9d0a47 --events erectl eems 7d41a0c2-5b9e-4f13-8c2a-1e6b3f9d0a47 --events --limit 20

eems logs

Stream an EEM's console logs until Ctrl-C. Recent buffered lines print first, then live output. --raw drops the timestamp and source prefix. When the stream is refused (for example an id that is not one of the project's EEMs) or fails, the command prints the reason and exits non-zero; Ctrl-C ends it with status 0.

bash
erectl eems logs 7d41a0c2-5b9e-4f13-8c2a-1e6b3f9d0a47 erectl eems logs 7d41a0c2-5b9e-4f13-8c2a-1e6b3f9d0a47 --raw

eems manifest

Show the capability manifest Erebine holds for an EEM: the tools it advertised and the workspaces it declares, which is what tool calls are dispatched against. The summary counts the tools by domain and lists each declared workspace with its types, domains ((all) when it names none) and credentials (family=file, one per credential family). A warning names each declared workspace that matches no workspace in the project; calls for it go to the first EEM advertising the tool rather than to a bound one, so fix the name in the EEM's workspaces.yaml or create the workspace.

bash
erectl eems manifest 7d41a0c2-5b9e-4f13-8c2a-1e6b3f9d0a47 erectl eems manifest 7d41a0c2-5b9e-4f13-8c2a-1e6b3f9d0a47 -o json | jq '.manifest.tools | length'

With -o json it prints {"manifest": ..., "warnings": [...]}: the manifest as stored, and an empty warnings list when there are none. When no manifest is held for the EEM (it is not one of this project's EEMs, has not connected since the last restart, or has advertised nothing) the command says so and exits non-zero.

eems curve-rotate

Initiate a CURVE (Curve25519) control-plane key rotation for an enrolled EEM. The router instructs the EEM to generate a fresh key pair and waits for a signed acknowledgement over the existing authenticated channel before promoting the new key. Owner-only.

bash
erectl eems curve-rotate 7d41a0c2-5b9e-4f13-8c2a-1e6b3f9d0a47 erectl eems curve-rotate 7d41a0c2-5b9e-4f13-8c2a-1e6b3f9d0a47 --reason "post-incident"

Arguments and Options

Name Description
<eem-id> The EEM's UUID, as erectl eems lists it.
--reason <text> Free-text reason recorded in the audit log. Defaults to scheduled.

On success the command prints the rotation request id, the dispatch status, and a monitor URL. Track progress with erectl eems <eem-id> --events.

eems workspaces

List an EEM's workspaces with both states. Bound means calls in the workspace go to this EEM; declared means the EEM's own workspaces.yaml declares it, which is what the EEM enforces. A workspace runs only when it is both. A declared name that matches no workspace in the project shows - as its workspace. Owners and operators only.

bash
erectl eems workspaces 7d41a0c2-5b9e-4f13-8c2a-1e6b3f9d0a47 erectl eems workspaces 7d41a0c2-5b9e-4f13-8c2a-1e6b3f9d0a47 -o json

Under -o json each row carries workspace_id, name, workspace_types (a list), bound, declared, paused, paused_reason, remote_editable, declare_command and remove_command.

eems add-workspace and remove-workspace

add-workspace binds a workspace to the EEM at once. When the EEM's host set EREBINE_EEM_ALLOW_REMOTE_DECLARATIONS, the EEM also declares it in its workspaces.yaml: a new declaration takes the workspace's scope, and an existing one keeps the scope in the file. Otherwise the command prints the erectl workspaces declare line to run on the EEM's host. An EEM that runs a release from before workspaces took several types cannot read the declaration this sends, so for it the command asks you to upgrade the EEM and erectl on its host instead. remove-workspace does the reverse: it unbinds, then removes the declaration or prints the line that does. Owners and operators only.

bash
erectl eems add-workspace 7d41a0c2-5b9e-4f13-8c2a-1e6b3f9d0a47 prod-k8s erectl eems remove-workspace 7d41a0c2-5b9e-4f13-8c2a-1e6b3f9d0a47 ws_3kq9x2

Arguments

Name Description
<eem-id> EEM id.
<workspace> The operational workspace's ws_ id or name.

The command reports the binding and the declaration separately. It exits non-zero when the EEM was asked to change its file and did not (it was unreachable, did not answer in time, refused, or could not write the file); the printed host command makes the change by hand.

eems policy

Shows or sets the EEM's own approval policy: whether a person or the project's decider answers this EEM's waiting calls in a workspace that has no policy of its own. Without settings it prints the policy, or says the EEM has none and its workspaces' and the project's policies apply. With settings it creates the policy, or changes only the settings given and keeps the rest. Owners and operators only. The settings, and the order in which workspace, EEM and project policies apply, are on erectl approvals policies.

bash
erectl eems policy 7d41a0c2-5b9e-4f13-8c2a-1e6b3f9d0a47 erectl eems policy 7d41a0c2-5b9e-4f13-8c2a-1e6b3f9d0a47 --mode auto --max-auto-risk write erectl eems policy 7d41a0c2-5b9e-4f13-8c2a-1e6b3f9d0a47 --reject-threshold 0.9 --low-confidence reject

Options

Option Description
--mode human|autohuman: a person decides every call. auto: the decider answers first.
--max-auto-risk writeThe highest risk the decider may approve: write only. It never approves a destructive or irreversible call.
--approve-threshold-write <0..1|default>The decider's confidence needed to approve a write call (default 0.92).
--reject-threshold <0..1|default>The decider's confidence needed to reject a call (default 0.85).
--low-confidence human|rejectWhat happens to a call the decider is unsure about.

Delete the EEM's policy with erectl approvals policies delete <policy-id>; the id is on erectl eems policy <id>.

Join Keys Subcommand

Join keys are limited-use enrollment tokens that let a new EIM or EEM register itself with the platform. After the token is consumed by the configured maximum number of enrollments (or after the TTL elapses) it can no longer be used.

Each group mints keys for its own kind: erectl eims join-keys mints and lists inference keys, erectl eems join-keys execution keys. The role is minted into the key, and an enrollment that asserts a different role is refused.

The router management API does not expose a standalone GET-by-id endpoint for join keys. Running erectl eims join-keys <id> without an action flag prints a notice and falls back to the list view; query the listing (optionally with --include-terminal) or the event history for per-key state.

bash
erectl eims join-keys # List inference join keys erectl eims join-keys --create \ --name <name> --region <r> \ --router-addr tcp://router:5555 \ --supported-tiers self_hosted # Create an inference join key erectl eems join-keys --create \ --name <name> --region <r> \ --router-addr tcp://router:5555 # Create an execution join key erectl eims join-keys <id> --revoke # Revoke a join key

join-keys list

List the join keys of the group's kind. By default only live keys are returned; pass --include-terminal to also surface revoked, fully-enrolled, or expired keys.

bash
erectl eims join-keys erectl eims join-keys --limit 50 erectl eems join-keys --include-terminal

Options

Option Description
--include-terminal Include revoked, fully-enrolled, and expired join keys in the output

Also accepts the shared pagination flags --limit and --after (see Overview).

Output columns: ID, NAME, STATUS, ROLE, REGION, ENROLLMENTS (current), MAX, EXPIRES. ROLE is the role the key was minted for.

join-keys create

Create a new join key. --name and --region are required. --router-addr defaults to the address erectl deploy uses, tcp://<base URL host>:5555, which the dry run and the result print. An inference key, minted by erectl eims join-keys, also needs --supported-tiers: every EIM it enrolls inherits those tiers, and a key that names none is refused. An execution key, minted by erectl eems join-keys, may name none. The token value is shown once at creation; save it immediately as it will not be shown again. With -o json the command prints the API response, with the token under token.

bash
# Minimal inference join key erectl eims join-keys --create \ --name gpu-pool \ --region us-east \ --router-addr tcp://router:5555 \ --supported-tiers self_hosted # Multiple router addresses, enrollment cap, and TTL erectl eims join-keys --create \ --name gpu-pool-eu \ --region eu-west \ --router-addr tcp://router-a:5555 \ --router-addr tcp://router-b:5555 \ --max-enrollments 5 \ --ttl-seconds 86400 \ --supported-tiers base,plus # A key that enrolls EEMs, for the router at the base URL's host erectl eems join-keys --create \ --name eem-pool \ --region us-east

Options

Option Description
--name <name> Join key display name (required)
--region <region> Region to scope the join key to (required)
--router-addr <addr> Router address the enrolled EIM or EEM connects to. Repeatable. Defaults to tcp://<base URL host>:5555, the address erectl deploy uses.
--max-enrollments <n> Maximum number of enrollments the key admits (10 by default)
--ttl-seconds <n> Join key time-to-live in seconds
--supported-tiers <tiers> Comma-separated service tiers every enrolled EIM inherits. Required for an inference key; optional for an execution key. Unknown tier slugs are refused. List valid slugs with erectl tiers.

The role is minted into the key. An enrollment whose asserted role differs from the role its key was minted for is refused. Mint an EEM's key with erectl eems join-keys --create; an inference key will not enroll one.

Security: The join token is shown only once at creation. Store it securely and distribute it only to the hosts you are enrolling.

join-keys revoke

Revoke a join key, immediately preventing any further enrollments using it. The key id is enough; either group revokes it. A confirmation prompt is shown by default.

bash
# With confirmation prompt erectl eims join-keys jk_abc123 --revoke # Skip confirmation erectl eims join-keys jk_abc123 --revoke --force # Dry run (--dry-run is a global erectl flag) erectl eems join-keys jk_def456 --revoke --dry-run

--force skips the confirmation prompt. --dry-run is a global erectl flag (see Overview).

Examples

Enroll a New EIM

bash
# Create a join key for the new EIM erectl eims join-keys --create \ --name new-eim \ --region us-east \ --router-addr tcp://router:5555 \ --supported-tiers self_hosted \ --max-enrollments 1 # Start the EIM with the printed token; it uses the token to self-register # (erectl deploy eim mints and passes the key for you) # Verify the EIM appeared erectl eims

Inspect and Monitor an EIM

bash
# List all EIMs erectl eims # Get full details erectl eims 3f9c1d2b-8a04-4c1e-9b6f-0d2e7a1c5b90 # View recent events erectl eims 3f9c1d2b-8a04-4c1e-9b6f-0d2e7a1c5b90 --events --limit 20 # Follow its console erectl eims logs 3f9c1d2b-8a04-4c1e-9b6f-0d2e7a1c5b90

Temporarily Park an EEM

bash
# Stop dispatching to the EEM in one workspace, keeping the binding erectl workspaces pause-eem <ws_id> 7d41a0c2-5b9e-4f13-8c2a-1e6b3f9d0a47 --reason "node upgrade" # Resume when ready erectl workspaces resume-eem <ws_id> 7d41a0c2-5b9e-4f13-8c2a-1e6b3f9d0a47

Rotate an EEM's CURVE Key

bash
erectl eems curve-rotate 7d41a0c2-5b9e-4f13-8c2a-1e6b3f9d0a47 --reason "scheduled-rotation" erectl eems 7d41a0c2-5b9e-4f13-8c2a-1e6b3f9d0a47 --events --limit 20

Clean Up Unused Join Keys

Live listings hide revoked, expired, and fully-enrolled keys by default; --include-terminal surfaces them.

bash
# List join keys of each kind, including terminal states erectl eims join-keys --include-terminal erectl eems join-keys --include-terminal # Revoke any keys that are no longer needed erectl eims join-keys jk_abc123 --revoke --force

Background reading: EIM and EEM roles (what each runs and what it can do). The pagination below returns to the erectl reference.