Credential rotation

Summary

Four credentials, four postures. Skim the strip before drilling in.

// workspace Long-lived or STS, on disk under /var/lib/erebine/credentials/. Replace the file. The next call reads it.
// curve Curve25519 keypair, data-plane identity of the EEM. erectl agents curve-rotate.
// api Client-side keys for /v1/exec/*. Lives on the caller, not the EEM. erectl keys create then revoke.
// join Single-use, TTL-bounded JWTs. Not rotated; bounded. Cap TTL. Revoke unconsumed on suspicion.

Press Alt+C while a code block is focused (Tab through to one) to copy it. No Meta/Cmd binding, so the OS clipboard chord is untouched.

Workspace Credentials

Workspace credentials are files on the EEM host, read as the user the agent runs as (erebine under the packaged erebine-eem-agent unit). Nothing caches them: a tool's command-line client starts afresh on every call and reads its own configuration then, and a file in the credentials directory is read by the call that uses it. Rotation is one step: replace the file. The next call reads it; there is nothing to reload.

The credentials directory is credentials_dir, default /var/lib/erebine/credentials. It holds one directory per workspace, named as in workspaces.yaml, and one file per credential, named by the workspace's credentials_ref: <credentials_dir>/<workspace>/<name>. The agent refuses a file that is a symlink, is empty, or is readable by anyone but its owner: make the agent's user the owner and set mode 0600 or 0400. Under erectl deploy the directory is --credentials-dir on the host (default <data-dir>/credentials), which the agent can read and not write; replace the file there.

bashreplace in place

# On the EEM host. Write the new file beside the old one, then rename it
# over the old one, so no call reads a half-written credential. erebine is
# the packaged unit's user; use the user your agent runs as.
dir=/var/lib/erebine/credentials/prod-k8s
sudo install -m 0600 -o erebine -g erebine /tmp/new-kubeconfig "$dir/kubeconfig.new"
sudo mv "$dir/kubeconfig.new" "$dir/kubeconfig"

                

A call already running finishes on the credential it read; the next one uses the new file. The agent needs no restart, and its unit has no reload action.

To check a credential without reading it back, invoke the eem_test_credential tool in the workspace with the file's name:

bashcheck

# From an operator workstation:
erectl exec invoke --workspace ws_a1b2c3d4 --tool eem_test_credential \
    --args '{"credential_name": "kubeconfig"}'

                

It answers configured, or names the fault: not_configured, insecure_permissions, symlink_refused, empty or invalid_reference.

For short lived tokens (e.g. cloud STS), refresh the file on a timer before expiry, the same way. The EEM does not refresh tokens itself; it consumes what the operator provides.

Least-privilege database role

The database tools (psql_query, psql_tables, mysql_query) connect as whatever role their credential authenticates as, and they run tenant SQL that has already cleared owner approval. The read-only session stops writes; it does not stop file egress or server-side program execution. Provision a dedicated role with read on the intended tables and no other privilege, and point it at a read database that carries no file, program, or network extensions.

Replace the placeholder role, database, schema, and table names before running. Run PostgreSQL as a superuser or the database owner; run MySQL as an account that can create users and grant.

Read stays read: the role can return any row in the tables you grant. Grant SELECT on the specific tables the workspace needs, not a whole schema, and never on a shared multi-tenant table. Use a dedicated read database, not a copy of your application database, so no unaudited view, function, or extension is reachable.

sqlpostgresql

-- Login role with every privileged attribute off. NOSUPERUSER is what
-- keeps COPY ... TO PROGRAM and server file access out of reach.
CREATE ROLE eem_ro
    LOGIN NOSUPERUSER NOCREATEDB NOCREATEROLE
    NOREPLICATION NOBYPASSRLS NOINHERIT
    CONNECTION LIMIT 8
    PASSWORD 'REPLACE_WITH_SECRET';

-- Read-only sessions by default, and a pinned search_path.
ALTER ROLE eem_ro SET default_transaction_read_only = on;
ALTER ROLE eem_ro SET search_path = pg_catalog, reporting;

-- Strip ambient PUBLIC access on this dedicated read database.
REVOKE ALL ON DATABASE eem_db FROM PUBLIC;
GRANT  CONNECT ON DATABASE eem_db TO eem_ro;
REVOKE ALL ON SCHEMA public FROM PUBLIC;

-- Grant only the exact tables. Do not use ALL TABLES or default privileges:
-- both sweep in foreign tables and owner-rights views.
GRANT USAGE ON SCHEMA reporting TO eem_ro;
GRANT SELECT ON reporting.orders, reporting.customers TO eem_ro;

-- Remove ambient EXECUTE so no definer function runs with its owner's rights.
-- Run as the role that owns the readable objects.
REVOKE EXECUTE ON ALL FUNCTIONS  IN SCHEMA reporting FROM PUBLIC;
REVOKE EXECUTE ON ALL PROCEDURES IN SCHEMA reporting FROM PUBLIC;

-- Before service, confirm: no role memberships, no foreign tables, no
-- definer/untrusted-language functions, no file/program/network extensions.
SELECT rolname, rolsuper, rolcanlogin FROM pg_roles WHERE rolname = 'eem_ro';
SELECT c.relname FROM pg_class c
    JOIN pg_namespace n ON n.oid = c.relnamespace
    WHERE n.nspname = 'reporting' AND c.relkind = 'f';   -- expect zero rows
SELECT extname FROM pg_extension;                        -- no dblink/file_fdw/postgres_fdw

-- Never grant: SUPERUSER, pg_execute_server_program, pg_read_server_files,
-- pg_write_server_files, pg_read_all_data, or USAGE on any foreign wrapper/server.
                

sqlmysql

-- Dedicated account, host-pinned to the agent's egress address (never all-hosts),
-- TLS required, connection count capped.
CREATE USER 'eem_ro'@'AGENT_EGRESS_ADDR'
    IDENTIFIED BY 'REPLACE_WITH_SECRET'
    REQUIRE SSL
    WITH MAX_USER_CONNECTIONS 8;

-- Grant only the exact tables. FILE is a global-only privilege, so a SELECT
-- grant can never carry it. Do not use ALL PRIVILEGES.
GRANT SELECT ON app_db.orders    TO 'eem_ro'@'AGENT_EGRESS_ADDR';
GRANT SELECT ON app_db.customers TO 'eem_ro'@'AGENT_EGRESS_ADDR';

-- Confirm the grant list is exactly USAGE + those SELECTs, and that no
-- mandatory role adds privileges the account did not receive directly.
SHOW GRANTS FOR 'eem_ro'@'AGENT_EGRESS_ADDR';
SELECT @@global.mandatory_roles, @@global.activate_all_roles_on_login;

-- Server-side file gates a read-only session does not cover. Expect
-- secure_file_priv = NULL and local_infile = OFF (MySQL 8.4 defaults).
SELECT @@global.secure_file_priv, @@global.local_infile;

-- Never grant: FILE, PROCESS, SUPER, SHUTDOWN, RELOAD, EVENT, TRIGGER,
-- CREATE/ALTER ROUTINE, EXECUTE, CREATE TEMPORARY TABLES, CREATE USER,
-- GRANT OPTION, or any admin dynamic privilege.
                

Place the credential where the client reads it under a clean environment: PostgreSQL reads ~/.pgpass in the agent user's home, backed by a pg_hba.conf entry; MySQL reads the [client] section of /etc/my.cnf. Environment variables do not reach the tool subprocess.

Block the client shell-out: the MySQL client can run an agent-host command with system or \!, outside any database grant. Run the client with --skip-system-command (or --binary-mode), and confirm the server has no operating-system command functions registered before you trust the read-only role.

CURVE Keys

The Curve25519 keypair is the data-plane identity of the EEM. Rotate on compliance schedule, on suspected compromise, or after a host rebuild.

bashrotate-curve

# From an operator workstation:
erectl agents curve-rotate agt_abc123 --reason "scheduled"

The rotation is on-demand, not scheduled. The router instructs the agent to generate a fresh keypair and waits for a signed acknowledgement over the existing authenticated channel before promoting the new key. The agent ID is the router identifier (e.g. agt_abc123); the --reason string is recorded in the audit log and defaults to scheduled.

Expected transcript:

textrequest accepted

request_id: 7f0e...
agent_id:   agt_abc123
status:     pending
monitor:    /v1/management/agents/agt_abc123/events

                

Monitor the events stream at the printed URL to observe the acknowledgement and promotion. The rotation is audit-logged with operator, reason, and request id.

If status resolves to failed, the recorded reason will be one of: acknowledgement timeout, signature mismatch, or revoked agent. Treat any of these as a containment trigger and pivot to Incident Response before retrying.

API Keys

API keys live on the caller side, not on the EEM host. Rotation uses the management scope:

bashrotate-api-key

erectl keys create --scope execution --name fleet-ops-2026q2
erectl keys revoke --name fleet-ops-2026q1

Create the new key before revoking the old one so callers have a clean cutover. The revoke is immediate, plan the client rollout before calling it.

Join Keys

Join keys are not rotated; they are bounded. Each is a single-use, TTL-bounded JWT, consumed on first enrollment and never reusable. The operational handle is the TTL: cap it to the shortest value that still lets operations complete the enrollment.

bashissue-join-key

erectl agents join-keys --create \
    --name eem-fleet-02 \
    --region us-east \
    --router-addr tcp://router:5555 \
    --ttl-seconds 600   # cap as low as operations allows

                

The --agent-role option accepts only inference or execution; EEM execution agents use execution. Omit it to accept the router default.

A leaked join key is contained by its TTL, but the window is not zero. Revoke unconsumed keys immediately on suspicion -- the TTL is a backstop, not a substitute for revocation.

Unconsumed and terminal (revoked, used, expired) keys can be listed, and individual keys revoked by id:

bashlist + revoke

erectl agents join-keys                       # active keys
erectl agents join-keys --include-terminal    # include revoked/used/expired
erectl agents join-keys jk_abc123 --revoke --force

                

Recommended Cadence

Credential	Routine cadence	Emergency
Workspace credentials (long-lived)	90 days	Within 1 hour of suspected leak
Workspace credentials (STS / short-lived)	Per token TTL	Per token TTL (no faster)
CURVE keys	180 days	Within 15 minutes of suspected leak
API keys	180 days	Within 15 minutes
Join keys	N/A (single-use)	Revoke unconsumed immediately

These are defaults. Compliance regimes (PCI, HIPAA, FedRAMP) override with tighter requirements; follow the stricter of the two.

Credential Rotation

Summary

Workspace Credentials

Least-privilege database role

CURVE Keys

API Keys

Join Keys

Recommended Cadence

Adjacent Runbooks