Credential Rotation
Routine and emergency rotation for the four credentials an Erebine Execution Manager (EEM) touches.
Workspace credentials, the Curve25519 data-plane keypair, API keys used by clients calling /v1/exec/*, and the short-lived join keys used at enrollment. Each has its own discipline.
Summary
Four credentials, four postures. Skim the strip before drilling in.
-
// workspace
Long-lived or STS, on disk under
/var/lib/erebine/credentials/. Replace the file. The next call reads it. -
// curve
Curve25519 keypair, data-plane identity of the EEM.
erectl eems curve-rotate. -
// api
Client-side keys for
/v1/exec/*. Lives on the caller, not the EEM.erectl keys --createthen--delete. - // join TTL-bounded tokens that admit up to their enrollment limit of EEMs. Not rotated; bounded. Cap TTL. Revoke unconsumed on suspicion.
Press Alt+C while a code block is focused (Tab through to one) to copy it. No Meta/Cmd binding, so the OS clipboard chord is untouched.
Workspace Credentials
Workspace credentials are files on the EEM host,
read as the user the EEM runs as
(erebine under the packaged
erebine-eem-agent unit). Nothing caches
them: a tool's command-line client starts afresh on
every call and reads its own configuration then, and
a file in the credentials directory is read by the
call that uses it. Rotation is one step: replace the
file. The next call reads it; there is nothing to reload.
The credentials directory is
credentials_dir, default
/var/lib/erebine/credentials. It holds one
directory per workspace, named as in
workspaces.yaml, and one file per
credential family the workspace binds, named in its
credentials map:
<credentials_dir>/<workspace>/<name>.
The EEM refuses a file that is a symlink, is empty,
or is readable by anyone but its owner: make the
EEM's user the owner and set mode
0600 or 0400. Under
erectl deploy the directory is
--credentials-dir on the host (default
<data-dir>/credentials), which the
EEM can read and not write; replace the file there.
# On the EEM host. Write the new file beside the old one, then rename it
# over the old one, so no call reads a half-written credential. erebine is
# the packaged unit's user; use the user your EEM runs as.
dir=/var/lib/erebine/credentials/prod-k8s
sudo install -m 0600 -o erebine -g erebine /tmp/new-kubeconfig "$dir/kubeconfig.new"
sudo mv "$dir/kubeconfig.new" "$dir/kubeconfig"
A call already running finishes on the credential it read; the next one uses the new file. The EEM needs no restart, and its unit has no reload action.
To check credentials without reading them back, invoke
the eem_test_credential tool in the
workspace. With no arguments it checks every credential
the workspace binds and reports one status per family;
with a file's name it checks that one file:
# From an operator workstation:
erectl exec invoke --workspace ws_a1b2c3d4 --tool eem_test_credential --args '{}'
erectl exec invoke --workspace ws_a1b2c3d4 --tool eem_test_credential \
--args '{"credential_name": "kubeconfig"}'
Each file answers configured, or names the
fault: not_configured,
insecure_permissions,
symlink_refused, empty or
invalid_reference. Checked together, the
workspace answers configured when every
file is, needs_attention naming the
families to fix, or none_bound when it
binds none.
For short lived tokens (e.g. cloud STS), refresh the file on a timer before expiry, the same way. The EEM does not refresh tokens itself; it consumes what the operator provides.
Least-privilege database role
The database tools (psql_query,
psql_tables, mysql_query) connect as
whatever role their credential authenticates as, and they run
tenant SQL that has already cleared owner approval. The read-only
session stops writes; it does not stop file egress or server-side
program execution. Provision a dedicated role with read on the
intended tables and no other privilege, and point it at a read
database that carries no file, program, or network extensions.
Replace the placeholder role, database, schema, and table names before running. Run PostgreSQL as a superuser or the database owner; run MySQL as an account that can create users and grant.
Read stays read: the role can return any row in
the tables you grant. Grant SELECT on the specific
tables the workspace needs, not a whole schema, and never on a
shared multi-tenant table. Use a dedicated read database, not a copy
of your application database, so no unaudited view, function, or
extension is reachable.
-- Login role with every privileged attribute off. NOSUPERUSER is what
-- keeps COPY ... TO PROGRAM and server file access out of reach.
CREATE ROLE eem_ro
LOGIN NOSUPERUSER NOCREATEDB NOCREATEROLE
NOREPLICATION NOBYPASSRLS NOINHERIT
CONNECTION LIMIT 8
PASSWORD 'REPLACE_WITH_SECRET';
-- Read-only sessions by default, and a pinned search_path.
ALTER ROLE eem_ro SET default_transaction_read_only = on;
ALTER ROLE eem_ro SET search_path = pg_catalog, reporting;
-- Strip ambient PUBLIC access on this dedicated read database.
REVOKE ALL ON DATABASE eem_db FROM PUBLIC;
GRANT CONNECT ON DATABASE eem_db TO eem_ro;
REVOKE ALL ON SCHEMA public FROM PUBLIC;
-- Grant only the exact tables. Do not use ALL TABLES or default privileges:
-- both sweep in foreign tables and owner-rights views.
GRANT USAGE ON SCHEMA reporting TO eem_ro;
GRANT SELECT ON reporting.orders, reporting.customers TO eem_ro;
-- Remove ambient EXECUTE so no definer function runs with its owner's rights.
-- Run as the role that owns the readable objects.
REVOKE EXECUTE ON ALL FUNCTIONS IN SCHEMA reporting FROM PUBLIC;
REVOKE EXECUTE ON ALL PROCEDURES IN SCHEMA reporting FROM PUBLIC;
-- Before service, confirm: no role memberships, no foreign tables, no
-- definer/untrusted-language functions, no file/program/network extensions.
SELECT rolname, rolsuper, rolcanlogin FROM pg_roles WHERE rolname = 'eem_ro';
SELECT c.relname FROM pg_class c
JOIN pg_namespace n ON n.oid = c.relnamespace
WHERE n.nspname = 'reporting' AND c.relkind = 'f'; -- expect zero rows
SELECT extname FROM pg_extension; -- no dblink/file_fdw/postgres_fdw
-- Never grant: SUPERUSER, pg_execute_server_program, pg_read_server_files,
-- pg_write_server_files, pg_read_all_data, or USAGE on any foreign wrapper/server.
-- Dedicated account, host-pinned to the agent's egress address (never all-hosts),
-- TLS required, connection count capped.
CREATE USER 'eem_ro'@'AGENT_EGRESS_ADDR'
IDENTIFIED BY 'REPLACE_WITH_SECRET'
REQUIRE SSL
WITH MAX_USER_CONNECTIONS 8;
-- Grant only the exact tables. FILE is a global-only privilege, so a SELECT
-- grant can never carry it. Do not use ALL PRIVILEGES.
GRANT SELECT ON app_db.orders TO 'eem_ro'@'AGENT_EGRESS_ADDR';
GRANT SELECT ON app_db.customers TO 'eem_ro'@'AGENT_EGRESS_ADDR';
-- Confirm the grant list is exactly USAGE + those SELECTs, and that no
-- mandatory role adds privileges the account did not receive directly.
SHOW GRANTS FOR 'eem_ro'@'AGENT_EGRESS_ADDR';
SELECT @@global.mandatory_roles, @@global.activate_all_roles_on_login;
-- Server-side file gates a read-only session does not cover. Expect
-- secure_file_priv = NULL and local_infile = OFF (MySQL 8.4 defaults).
SELECT @@global.secure_file_priv, @@global.local_infile;
-- Never grant: FILE, PROCESS, SUPER, SHUTDOWN, RELOAD, EVENT, TRIGGER,
-- CREATE/ALTER ROUTINE, EXECUTE, CREATE TEMPORARY TABLES, CREATE USER,
-- GRANT OPTION, or any admin dynamic privilege.
Place the credential where the client reads it under a clean
environment: PostgreSQL reads ~/.pgpass in the EEM
user's home, backed by a pg_hba.conf entry; MySQL reads
the [client] section of /etc/my.cnf.
Environment variables do not reach the tool subprocess.
Block the client shell-out: the MySQL client can
run an EEM-host command with system or
\!, outside any database grant. Run the client with
--skip-system-command (or --binary-mode),
and confirm the server has no operating-system command functions
registered before you trust the read-only role.
CURVE Keys
The Curve25519 keypair is the data-plane identity of the EEM. Rotate on compliance schedule, on suspected compromise, or after a host rebuild.
# From an operator workstation:
erectl eems curve-rotate 3f9c1d2b-8a04-4c1e-9b6f-0d2e7a1c5b90 --reason "scheduled"
The rotation is on-demand, not scheduled. The
router instructs the EEM to generate a fresh
keypair and waits for a signed acknowledgement
over the existing authenticated channel before
promoting the new key. The router keeps accepting
the old key for a five-minute overlap window. The
EEM ID is the EEM's UUID (e.g.
3f9c1d2b-8a04-4c1e-9b6f-0d2e7a1c5b90); the
--reason string is recorded in the
audit log and defaults to scheduled.
Expected transcript:
Rotation '7f0e...' dispatched to EEM '3f9c1d2b-8a04-4c1e-9b6f-0d2e7a1c5b90'.
Status: pending
Monitor: GET /v1/management/agents/3f9c1d2b-8a04-4c1e-9b6f-0d2e7a1c5b90/events
Check progress with: erectl eems 3f9c1d2b-8a04-4c1e-9b6f-0d2e7a1c5b90 --events
Run the printed erectl eems ... --events
command to observe the acknowledgement and promotion. The
rotation is audit-logged with operator, reason,
and request id.
If status resolves to
failed, the recorded reason will be
one of: acknowledgement timeout, signature
mismatch, or revoked EEM. Treat any of these
as a containment trigger and pivot to
Incident
Response before retrying.
API Keys
API keys live on the caller side, not on the EEM host. Rotation uses the management scope:
erectl keys --create --name fleet-ops-2026q2 --scopes execution
erectl keys <old-key-id> --delete
Create the new key before revoking the old one so callers have a clean cutover. The revoke is immediate, plan the client rollout before calling it.
Join Keys
Join keys are not rotated; they are bounded. Each
is a TTL-bounded token that admits up to its
enrollment limit of EEMs (10 by default,
--max-enrollments to change it). The operational
handle is the TTL: cap it to the shortest value
that still lets operations complete the enrollment.
erectl eems join-keys --create \
--name eem-fleet-02 \
--region us-east \
--router-addr tcp://router:5555 \
--ttl-seconds 600 # cap as low as operations allows
erectl eems join-keys mints an execution
key: the role is minted into the key, and an EEM
enrolling with a key minted by
erectl eims join-keys is refused. An
execution key names no tiers.
A leaked join key is contained by its TTL, but the window is not zero. Revoke unconsumed keys immediately on suspicion -- the TTL is a backstop, not a substitute for revocation.
Unconsumed and terminal (revoked, used, expired) keys can be listed, and individual keys revoked by id:
erectl eems join-keys # active keys
erectl eems join-keys --include-terminal # include revoked/used/expired
erectl eems join-keys jk_abc123 --revoke --force
Recommended Cadence
| Credential | Routine cadence | Emergency |
|---|---|---|
| Workspace credentials (long-lived) | 90 days | Within 1 hour of suspected leak |
| Workspace credentials (STS / short-lived) | Per token TTL | Per token TTL (no faster) |
| CURVE keys | 180 days | Within 15 minutes of suspected leak |
| API keys | 180 days | Within 15 minutes |
| Join keys | Expires; revoke unused keys | Revoke unconsumed immediately |
These are defaults. Compliance regimes (PCI, HIPAA, FedRAMP) override with tighter requirements; follow the stricter of the two.