Credential rotation

Summary

Four credentials, four postures. Skim the strip before drilling in.

// workspace Long-lived or STS, on disk under /var/lib/erebine/credentials/. Replace file, reload service.
// curve Curve25519 keypair, data-plane identity of the EEM. erectl agents curve-rotate.
// api Client-side keys for /v1/exec/*. Lives on the caller, not the EEM. erectl keys create then revoke.
// join Single-use, TTL-bounded JWTs. Not rotated; bounded. Cap TTL. Revoke unconsumed on suspicion.

Press Alt+C while a code block is focused (Tab through to one) to copy it. No Meta/Cmd binding, so the OS clipboard chord is untouched.

Workspace Credentials

Workspace credential files live under /var/lib/erebine/credentials/ and are referenced by credentials_ref in the manifest. Rotation is a two-step: replace the file on disk, then reload.

bashinstall + reload

# On the EEM host:
sudo install -m 0600 -o root /tmp/new-kubeconfig \
    /var/lib/erebine/credentials/kubernetes-prod.yaml
sudo systemctl reload erebine-eem

                

The reload re-opens the credential file without interrupting in-flight invocations. The invocation that was already authenticated with the old credential completes; the next one picks up the new file.

For short lived tokens (e.g. cloud STS), set a filesystem watcher or timer to refresh the file before expiry. The EEM does not refresh tokens itself; it consumes what the operator provides.

Least-privilege database role

The database tools (psql_query, psql_tables, mysql_query) connect as whatever role their credential authenticates as, and they run tenant SQL that has already cleared owner approval. The read-only session stops writes; it does not stop file egress or server-side program execution. Provision a dedicated role with read on the intended tables and no other privilege, and point it at a read database that carries no file, program, or network extensions.

Replace the placeholder role, database, schema, and table names before running. Run PostgreSQL as a superuser or the database owner; run MySQL as an account that can create users and grant.

Read stays read: the role can return any row in the tables you grant. Grant SELECT on the specific tables the workspace needs, not a whole schema, and never on a shared multi-tenant table. Use a dedicated read database, not a copy of your application database, so no unaudited view, function, or extension is reachable.

sqlpostgresql

-- Login role with every privileged attribute off. NOSUPERUSER is what
-- keeps COPY ... TO PROGRAM and server file access out of reach.
CREATE ROLE eem_ro
    LOGIN NOSUPERUSER NOCREATEDB NOCREATEROLE
    NOREPLICATION NOBYPASSRLS NOINHERIT
    CONNECTION LIMIT 8
    PASSWORD 'REPLACE_WITH_SECRET';

-- Read-only sessions by default, and a pinned search_path.
ALTER ROLE eem_ro SET default_transaction_read_only = on;
ALTER ROLE eem_ro SET search_path = pg_catalog, reporting;

-- Strip ambient PUBLIC access on this dedicated read database.
REVOKE ALL ON DATABASE eem_db FROM PUBLIC;
GRANT  CONNECT ON DATABASE eem_db TO eem_ro;
REVOKE ALL ON SCHEMA public FROM PUBLIC;

-- Grant only the exact tables. Do not use ALL TABLES or default privileges:
-- both sweep in foreign tables and owner-rights views.
GRANT USAGE ON SCHEMA reporting TO eem_ro;
GRANT SELECT ON reporting.orders, reporting.customers TO eem_ro;

-- Remove ambient EXECUTE so no definer function runs with its owner's rights.
-- Run as the role that owns the readable objects.
REVOKE EXECUTE ON ALL FUNCTIONS  IN SCHEMA reporting FROM PUBLIC;
REVOKE EXECUTE ON ALL PROCEDURES IN SCHEMA reporting FROM PUBLIC;

-- Before service, confirm: no role memberships, no foreign tables, no
-- definer/untrusted-language functions, no file/program/network extensions.
SELECT rolname, rolsuper, rolcanlogin FROM pg_roles WHERE rolname = 'eem_ro';
SELECT c.relname FROM pg_class c
    JOIN pg_namespace n ON n.oid = c.relnamespace
    WHERE n.nspname = 'reporting' AND c.relkind = 'f';   -- expect zero rows
SELECT extname FROM pg_extension;                        -- no dblink/file_fdw/postgres_fdw

-- Never grant: SUPERUSER, pg_execute_server_program, pg_read_server_files,
-- pg_write_server_files, pg_read_all_data, or USAGE on any foreign wrapper/server.
                

sqlmysql

-- Dedicated account, host-pinned to the agent's egress address (never all-hosts),
-- TLS required, connection count capped.
CREATE USER 'eem_ro'@'AGENT_EGRESS_ADDR'
    IDENTIFIED BY 'REPLACE_WITH_SECRET'
    REQUIRE SSL
    WITH MAX_USER_CONNECTIONS 8;

-- Grant only the exact tables. FILE is a global-only privilege, so a SELECT
-- grant can never carry it. Do not use ALL PRIVILEGES.
GRANT SELECT ON app_db.orders    TO 'eem_ro'@'AGENT_EGRESS_ADDR';
GRANT SELECT ON app_db.customers TO 'eem_ro'@'AGENT_EGRESS_ADDR';

-- Confirm the grant list is exactly USAGE + those SELECTs, and that no
-- mandatory role adds privileges the account did not receive directly.
SHOW GRANTS FOR 'eem_ro'@'AGENT_EGRESS_ADDR';
SELECT @@global.mandatory_roles, @@global.activate_all_roles_on_login;

-- Server-side file gates a read-only session does not cover. Expect
-- secure_file_priv = NULL and local_infile = OFF (MySQL 8.4 defaults).
SELECT @@global.secure_file_priv, @@global.local_infile;

-- Never grant: FILE, PROCESS, SUPER, SHUTDOWN, RELOAD, EVENT, TRIGGER,
-- CREATE/ALTER ROUTINE, EXECUTE, CREATE TEMPORARY TABLES, CREATE USER,
-- GRANT OPTION, or any admin dynamic privilege.
                

Place the credential where the client reads it under a clean environment: PostgreSQL reads ~/.pgpass in the agent user's home, backed by a pg_hba.conf entry; MySQL reads the [client] section of /etc/my.cnf. Environment variables do not reach the tool subprocess.

Block the client shell-out: the MySQL client can run an agent-host command with system or \!, outside any database grant. Run the client with --skip-system-command (or --binary-mode), and confirm the server has no operating-system command functions registered before you trust the read-only role.

CURVE Keys

The Curve25519 keypair is the data-plane identity of the EEM. Rotate on compliance schedule, on suspected compromise, or after a host rebuild.

bashrotate-curve

# From an operator workstation:
erectl agents curve-rotate agt_abc123 --reason "scheduled"

The rotation is on-demand, not scheduled. The router instructs the agent to generate a fresh keypair and waits for a signed acknowledgement over the existing authenticated channel before promoting the new key. The agent ID is the router identifier (e.g. agt_abc123); the --reason string is recorded in the audit log and defaults to scheduled.

Expected transcript:

textrequest accepted

request_id: 7f0e...
agent_id:   agt_abc123
status:     pending
monitor:    /v1/management/agents/agt_abc123/events

                

Monitor the events stream at the printed URL to observe the acknowledgement and promotion. The rotation is audit-logged with operator, reason, and request id.

If status resolves to failed, the recorded reason will be one of: acknowledgement timeout, signature mismatch, or revoked agent. Treat any of these as a containment trigger and pivot to Incident Response before retrying.

API Keys

API keys live on the caller side, not on the EEM host. Rotation uses the management scope:

bashrotate-api-key

erectl keys create --scope execution --name fleet-ops-2026q2
erectl keys revoke --name fleet-ops-2026q1

Create the new key before revoking the old one so callers have a clean cutover. The revoke is immediate, plan the client rollout before calling it.

Join Keys

Join keys are not rotated; they are bounded. Each is a single-use, TTL-bounded JWT, consumed on first enrollment and never reusable. The operational handle is the TTL: cap it to the shortest value that still lets operations complete the enrollment.

bashissue-join-key

erectl agents join-keys --create \
    --name eem-fleet-02 \
    --region us-east \
    --router-addr tcp://router:5555 \
    --ttl-seconds 600   # cap as low as operations allows

                

The --agent-role option accepts only inference or execution; EEM execution agents use execution. Omit it to accept the router default.

A leaked join key is contained by its TTL, but the window is not zero. Revoke unconsumed keys immediately on suspicion -- the TTL is a backstop, not a substitute for revocation.

Unconsumed and terminal (revoked, used, expired) keys can be listed, and individual keys revoked by id:

bashlist + revoke

erectl agents join-keys                       # active keys
erectl agents join-keys --include-terminal    # include revoked/used/expired
erectl agents join-keys jk_abc123 --revoke --force

                

Recommended Cadence

Credential	Routine cadence	Emergency
Workspace credentials (long-lived)	90 days	Within 1 hour of suspected leak
Workspace credentials (STS / short-lived)	Per token TTL	Per token TTL (no faster)
CURVE keys	180 days	Within 15 minutes of suspected leak
API keys	180 days	Within 15 minutes
Join keys	N/A (single-use)	Revoke unconsumed immediately

These are defaults. Compliance regimes (PCI, HIPAA, FedRAMP) override with tighter requirements; follow the stricter of the two.

Credential Rotation

Summary

Workspace Credentials

Least-privilege database role

CURVE Keys

API Keys

Join Keys

Recommended Cadence

Adjacent Runbooks