docs
// EEM Guides

Credential Rotation

Routine and emergency rotation for the four credentials an Erebine Execution Manager (EEM) touches.

Workspace credentials, the Curve25519 data-plane keypair, API keys used by clients calling /v1/exec/*, and the short-lived join keys used at enrollment. Each has its own discipline.

Summary

Four credentials, four postures. Skim the strip before drilling in.

  • // workspace Long-lived or STS, on disk under /var/lib/erebine/credentials/. Replace the file. The next call reads it.
  • // curve Curve25519 keypair, data-plane identity of the EEM. erectl eems curve-rotate.
  • // api Client-side keys for /v1/exec/*. Lives on the caller, not the EEM. erectl keys --create then --delete.
  • // join TTL-bounded tokens that admit up to their enrollment limit of EEMs. Not rotated; bounded. Cap TTL. Revoke unconsumed on suspicion.

Press Alt+C while a code block is focused (Tab through to one) to copy it. No Meta/Cmd binding, so the OS clipboard chord is untouched.

Workspace Credentials

Workspace credentials are files on the EEM host, read as the user the EEM runs as (erebine under the packaged erebine-eem-agent unit). Nothing caches them: a tool's command-line client starts afresh on every call and reads its own configuration then, and a file in the credentials directory is read by the call that uses it. Rotation is one step: replace the file. The next call reads it; there is nothing to reload.

The credentials directory is credentials_dir, default /var/lib/erebine/credentials. It holds one directory per workspace, named as in workspaces.yaml, and one file per credential family the workspace binds, named in its credentials map: <credentials_dir>/<workspace>/<name>. The EEM refuses a file that is a symlink, is empty, or is readable by anyone but its owner: make the EEM's user the owner and set mode 0600 or 0400. Under erectl deploy the directory is --credentials-dir on the host (default <data-dir>/credentials), which the EEM can read and not write; replace the file there.

bashreplace in place
# On the EEM host. Write the new file beside the old one, then rename it # over the old one, so no call reads a half-written credential. erebine is # the packaged unit's user; use the user your EEM runs as. dir=/var/lib/erebine/credentials/prod-k8s sudo install -m 0600 -o erebine -g erebine /tmp/new-kubeconfig "$dir/kubeconfig.new" sudo mv "$dir/kubeconfig.new" "$dir/kubeconfig"

A call already running finishes on the credential it read; the next one uses the new file. The EEM needs no restart, and its unit has no reload action.

To check credentials without reading them back, invoke the eem_test_credential tool in the workspace. With no arguments it checks every credential the workspace binds and reports one status per family; with a file's name it checks that one file:

bashcheck
# From an operator workstation: erectl exec invoke --workspace ws_a1b2c3d4 --tool eem_test_credential --args '{}' erectl exec invoke --workspace ws_a1b2c3d4 --tool eem_test_credential \ --args '{"credential_name": "kubeconfig"}'

Each file answers configured, or names the fault: not_configured, insecure_permissions, symlink_refused, empty or invalid_reference. Checked together, the workspace answers configured when every file is, needs_attention naming the families to fix, or none_bound when it binds none.

For short lived tokens (e.g. cloud STS), refresh the file on a timer before expiry, the same way. The EEM does not refresh tokens itself; it consumes what the operator provides.

Least-privilege database role

The database tools (psql_query, psql_tables, mysql_query) connect as whatever role their credential authenticates as, and they run tenant SQL that has already cleared owner approval. The read-only session stops writes; it does not stop file egress or server-side program execution. Provision a dedicated role with read on the intended tables and no other privilege, and point it at a read database that carries no file, program, or network extensions.

Replace the placeholder role, database, schema, and table names before running. Run PostgreSQL as a superuser or the database owner; run MySQL as an account that can create users and grant.

Read stays read: the role can return any row in the tables you grant. Grant SELECT on the specific tables the workspace needs, not a whole schema, and never on a shared multi-tenant table. Use a dedicated read database, not a copy of your application database, so no unaudited view, function, or extension is reachable.

sqlpostgresql
-- Login role with every privileged attribute off. NOSUPERUSER is what -- keeps COPY ... TO PROGRAM and server file access out of reach. CREATE ROLE eem_ro LOGIN NOSUPERUSER NOCREATEDB NOCREATEROLE NOREPLICATION NOBYPASSRLS NOINHERIT CONNECTION LIMIT 8 PASSWORD 'REPLACE_WITH_SECRET'; -- Read-only sessions by default, and a pinned search_path. ALTER ROLE eem_ro SET default_transaction_read_only = on; ALTER ROLE eem_ro SET search_path = pg_catalog, reporting; -- Strip ambient PUBLIC access on this dedicated read database. REVOKE ALL ON DATABASE eem_db FROM PUBLIC; GRANT CONNECT ON DATABASE eem_db TO eem_ro; REVOKE ALL ON SCHEMA public FROM PUBLIC; -- Grant only the exact tables. Do not use ALL TABLES or default privileges: -- both sweep in foreign tables and owner-rights views. GRANT USAGE ON SCHEMA reporting TO eem_ro; GRANT SELECT ON reporting.orders, reporting.customers TO eem_ro; -- Remove ambient EXECUTE so no definer function runs with its owner's rights. -- Run as the role that owns the readable objects. REVOKE EXECUTE ON ALL FUNCTIONS IN SCHEMA reporting FROM PUBLIC; REVOKE EXECUTE ON ALL PROCEDURES IN SCHEMA reporting FROM PUBLIC; -- Before service, confirm: no role memberships, no foreign tables, no -- definer/untrusted-language functions, no file/program/network extensions. SELECT rolname, rolsuper, rolcanlogin FROM pg_roles WHERE rolname = 'eem_ro'; SELECT c.relname FROM pg_class c JOIN pg_namespace n ON n.oid = c.relnamespace WHERE n.nspname = 'reporting' AND c.relkind = 'f'; -- expect zero rows SELECT extname FROM pg_extension; -- no dblink/file_fdw/postgres_fdw -- Never grant: SUPERUSER, pg_execute_server_program, pg_read_server_files, -- pg_write_server_files, pg_read_all_data, or USAGE on any foreign wrapper/server.
sqlmysql
-- Dedicated account, host-pinned to the agent's egress address (never all-hosts), -- TLS required, connection count capped. CREATE USER 'eem_ro'@'AGENT_EGRESS_ADDR' IDENTIFIED BY 'REPLACE_WITH_SECRET' REQUIRE SSL WITH MAX_USER_CONNECTIONS 8; -- Grant only the exact tables. FILE is a global-only privilege, so a SELECT -- grant can never carry it. Do not use ALL PRIVILEGES. GRANT SELECT ON app_db.orders TO 'eem_ro'@'AGENT_EGRESS_ADDR'; GRANT SELECT ON app_db.customers TO 'eem_ro'@'AGENT_EGRESS_ADDR'; -- Confirm the grant list is exactly USAGE + those SELECTs, and that no -- mandatory role adds privileges the account did not receive directly. SHOW GRANTS FOR 'eem_ro'@'AGENT_EGRESS_ADDR'; SELECT @@global.mandatory_roles, @@global.activate_all_roles_on_login; -- Server-side file gates a read-only session does not cover. Expect -- secure_file_priv = NULL and local_infile = OFF (MySQL 8.4 defaults). SELECT @@global.secure_file_priv, @@global.local_infile; -- Never grant: FILE, PROCESS, SUPER, SHUTDOWN, RELOAD, EVENT, TRIGGER, -- CREATE/ALTER ROUTINE, EXECUTE, CREATE TEMPORARY TABLES, CREATE USER, -- GRANT OPTION, or any admin dynamic privilege.

Place the credential where the client reads it under a clean environment: PostgreSQL reads ~/.pgpass in the EEM user's home, backed by a pg_hba.conf entry; MySQL reads the [client] section of /etc/my.cnf. Environment variables do not reach the tool subprocess.

Block the client shell-out: the MySQL client can run an EEM-host command with system or \!, outside any database grant. Run the client with --skip-system-command (or --binary-mode), and confirm the server has no operating-system command functions registered before you trust the read-only role.

CURVE Keys

The Curve25519 keypair is the data-plane identity of the EEM. Rotate on compliance schedule, on suspected compromise, or after a host rebuild.

bashrotate-curve
# From an operator workstation: erectl eems curve-rotate 3f9c1d2b-8a04-4c1e-9b6f-0d2e7a1c5b90 --reason "scheduled"

The rotation is on-demand, not scheduled. The router instructs the EEM to generate a fresh keypair and waits for a signed acknowledgement over the existing authenticated channel before promoting the new key. The router keeps accepting the old key for a five-minute overlap window. The EEM ID is the EEM's UUID (e.g. 3f9c1d2b-8a04-4c1e-9b6f-0d2e7a1c5b90); the --reason string is recorded in the audit log and defaults to scheduled.

Expected transcript:

textrequest accepted
Rotation '7f0e...' dispatched to EEM '3f9c1d2b-8a04-4c1e-9b6f-0d2e7a1c5b90'. Status: pending Monitor: GET /v1/management/agents/3f9c1d2b-8a04-4c1e-9b6f-0d2e7a1c5b90/events Check progress with: erectl eems 3f9c1d2b-8a04-4c1e-9b6f-0d2e7a1c5b90 --events

Run the printed erectl eems ... --events command to observe the acknowledgement and promotion. The rotation is audit-logged with operator, reason, and request id.

If status resolves to failed, the recorded reason will be one of: acknowledgement timeout, signature mismatch, or revoked EEM. Treat any of these as a containment trigger and pivot to Incident Response before retrying.

API Keys

API keys live on the caller side, not on the EEM host. Rotation uses the management scope:

bashrotate-api-key
erectl keys --create --name fleet-ops-2026q2 --scopes execution erectl keys <old-key-id> --delete

Create the new key before revoking the old one so callers have a clean cutover. The revoke is immediate, plan the client rollout before calling it.

Join Keys

Join keys are not rotated; they are bounded. Each is a TTL-bounded token that admits up to its enrollment limit of EEMs (10 by default, --max-enrollments to change it). The operational handle is the TTL: cap it to the shortest value that still lets operations complete the enrollment.

bashissue-join-key
erectl eems join-keys --create \ --name eem-fleet-02 \ --region us-east \ --router-addr tcp://router:5555 \ --ttl-seconds 600 # cap as low as operations allows

erectl eems join-keys mints an execution key: the role is minted into the key, and an EEM enrolling with a key minted by erectl eims join-keys is refused. An execution key names no tiers.

A leaked join key is contained by its TTL, but the window is not zero. Revoke unconsumed keys immediately on suspicion -- the TTL is a backstop, not a substitute for revocation.

Unconsumed and terminal (revoked, used, expired) keys can be listed, and individual keys revoked by id:

bashlist + revoke
erectl eems join-keys # active keys erectl eems join-keys --include-terminal # include revoked/used/expired erectl eems join-keys jk_abc123 --revoke --force

Recommended Cadence

CredentialRoutine cadenceEmergency
Workspace credentials (long-lived) 90 days Within 1 hour of suspected leak
Workspace credentials (STS / short-lived) Per token TTL Per token TTL (no faster)
CURVE keys 180 days Within 15 minutes of suspected leak
API keys 180 days Within 15 minutes
Join keys Expires; revoke unused keys Revoke unconsumed immediately

These are defaults. Compliance regimes (PCI, HIPAA, FedRAMP) override with tighter requirements; follow the stricter of the two.

Adjacent Runbooks