Configuration

SwitchTender is one binary with subcommands. This page lists every command, flag, and environment variable.

Licensed features

Almost everything on this page runs on Community, which needs no license. Six features are licensed, and separately an install is capped on how many approval policies it holds at once. A flag or request that turns on a licensed feature is refused outright rather than quietly ignored, so an install never believes it has a control it does not have.

license status prints the tier this install runs and when a license lapses. Tiers and prices are at https://switchtender.com/pricing.

Environment variables

Every command that takes --db defaults to switchtender.db in the current directory. Only serve, init, demo, restore, and an import create a database there. The commands that work on an existing install, such as token, user, examples, backup, and the audit commands, refuse a SQLite path that does not exist and name the path they looked at. Opening it would create an empty database the server never reads, and the command would report success. An import that starts a new database says so, with the --db a server needs to read it.

init

Bootstraps a new deployment. It creates the database and the first admin account, writes an environment file, and optionally a systemd unit. Run it once, then start serve. Running it again, with --force to rewrite the environment file or after deleting it, leaves an admin account that already exists as it is, password included.

The admin password comes from SWITCHTENDER_ADMIN_PASSWORD, or is generated and printed once when that variable is unset.

serve

Runs the HTTP API, the in-process executor, the scheduler, the retention sweeper, and the web UI.

Retention windows accept a whole number of days with a d suffix, such as 30d, or Go duration syntax such as 720h.

AI providers

The advisory AI features run against one provider, chosen with --ai-provider: ollama for a local model, or anthropic and openai for a cloud model with SWITCHTENDER_AI_KEY. The provider is off until set, and no feature ever executes anything the provider suggests.

Cloud models see automation content: commands, playbook names, failed-run logs, and host drift. Because that content is security-adjacent, a model with strict safety classifiers can decline a benign request as a false positive. When the Anthropic model is a Fable or Mythos model, SwitchTender opts into server-side fallbacks, so a declined request is retried on claude-opus-4-8 in the same call and the feature keeps working. A Fable model also requires that the account keep 30-day data retention, or the API rejects every request.

desktop

Runs SwitchTender as a local desktop application. It serves on a private loopback port, stores its data in a per-user directory, and opens the web UI in the default browser. It takes no flags. Set SWITCHTENDER_DESKTOP_NO_BROWSER to skip opening a browser. See Desktop for packaging a macOS app or a Windows installer.

worker

Leases pending runs from the shared store and executes them. Point it and a server at the same database and they compete for work. Or start it with --server and it leases runs from the control node over the mesh relay, with no database access of its own.

worker key

switchtender worker key new --out FILE generates a delivery key for a relay worker pool. It writes the private key to --out, mode 0600, and never over an existing file, then prints the public key and its key id as one JSON line:

{"key_id":"f4e8a76a457da19cf3742de37986d439","delivery_key":"x25519:x9f9/kkB6Jq3Wqht/7TWzDGUeTRfbABrj1CO+8/6hHc=","file":"/etc/switchtender/delivery.key"}

switchtender worker key public FILE prints the same line for a key file that already exists, and refuses one another account can read. The file is a standard PKCS #8 X25519 private key, so openssl genpkey -algorithm X25519 -out FILE makes an equivalent one, and openssl pkey -in FILE -pubout -outform DER | tail -c 32 | base64 prints the base64 that follows x25519:.

ansible

Installs, lists, and removes the managed Ansible runtime: a Python virtual environment holding one pinned ansible-core release, installed with pip in hash-checking mode from a lock this binary carries. Runs use it when no --ansible-bin is configured. A python3 the release supports must be installed.

• ansible install [--version <release>] [--python <path>] [--wheels <dir>] [--index-url <url>] installs a release and makes it the one runs use. --version takes a release such as 2.21.4 or a minor version such as 2.21, and defaults to the newest supported release. --python names the interpreter to build with, and defaults to python3, then python3.N, on PATH. --wheels installs offline from a directory of wheels, relative to where the command runs, checked against the same hashes. --index-url downloads from a mirror instead of PyPI. pip's own environment variables and configuration files are not read. Installing an installed release verifies it and makes it the one runs use, reinstalling nothing.
• ansible list prints the installed releases and which one is current.
• ansible remove [--version <release>] removes one release, or every release and the directory when nothing else is in it.
• ansible lock [--version <release>] prints the requirements lock a release installs from.

Every ansible subcommand takes --dir for the runtime directory, which defaults to SWITCHTENDER_ANSIBLE_RUNTIME_DIR, then ansible/ in the data directory of --db, and --pretty for indented JSON.

token

Manages API tokens. A public bind on an empty database mints an initial admin token at startup.

• token new --name <label> [--user <username>] [--ttl <duration>] [--agent] mints a token, printed once. A zero TTL never expires.
• --user binds the token to an account, and the token carries that account's role. A token minted without --user is unscoped and acts as admin, so bind every token you hand to a person, a service, or an AI agent. See running an agent.
• --agent marks the token as held by an AI agent rather than a person. It caps the token at operator whatever role its account holds, so an agent can launch and propose work but can never manage identity, access, or secrets, and can never approve its own held run. Every action it takes is recorded in the chain as actor_type: agent with the account it acts for beside it, which is what actor_kind: agent policy rules match on. Every run such a token asks for waits for a person's approval unless a rule with effect: exempt covers it, as Agent runs are held by default describes. It requires --user, so the chain always records the human the agent acts for. Without one the command refuses. Mint every agent token with it: a token without --agent is indistinguishable from a person's in the record, and gets no agent hold.
• token list lists tokens without their secrets.
• token revoke <id> deletes a token.

All token subcommands take --db and the global --pretty flag for indented JSON.

user

Manages accounts with roles: admin, operator, and viewer.

• user new <username> --role <role> creates an account. The password comes from SWITCHTENDER_PASSWORD or a prompt, never an argument.
• user list lists accounts.
• user delete <id> deletes an account. Its tokens stop working.

All user subcommands take --db.

license

Shows or installs this install's license. No license is Community, which is complete in itself. What each tier covers is listed at https://switchtender.com/pricing.

• license status shows the tier this install runs and when a license lapses.
• license install <file> verifies a license file and installs it beside the database.

A license is read from the file SWITCHTENDER_LICENSE names, or from switchtender-license.json in the same directory as the database. A license this install cannot parse or that has lapsed reads as Community rather than failing the server, so an expiry never takes an install down.

assess

Reports what an automation export holds and what governing it would change, without writing anything and without a database. It reads an export the same way import does, then grades every template it found through the same risk and reversibility graders the product uses at run time, so a number in an assessment cannot disagree with what a run would later say.

switchtender assess awx awx-export.json
switchtender assess chef chef-nodes.json

Formats are awx, semaphore, chef, and puppet. An AWX-format export also covers Ansible Automation Platform, Tower, and Ascender.

An export saved as UTF-8 with a byte order mark, or as UTF-16 the way Windows PowerShell writes a file by default, is read the same as plain UTF-8, here, by import, and by the assessment page.

The output has three parts: what the import would create, what does not survive the move as it was, and what changes about how it is governed. Everything that does not come across is listed, one line each. The third part names the templates that cannot be undone, the ones carrying a destructive signal, the credentials more than one template shares, and the templates targeting no stored inventory.

Grades are a floor rather than a measurement. An Ansible template keeps its work in a playbook, and at assessment time that playbook is in a repository nothing has fetched, so reading one can only ever raise a grade. The report says how many templates that applies to rather than leaving the numbers looking more settled than they are.

import

Migrates from AWX, Semaphore, Chef, Puppet, Rundeck, Jenkins, or cron. Which objects each one carries across is in what each source brings over.

• import awx <export.json> [--awx-template-ids <list.json>] [--apply] brings projects, inventories static and dynamic, credential shells, job templates and workflows, surveys, and schedules. --awx-template-ids takes AWX's job template list, GET /api/v2/job_templates/ saved as JSON, repeatable for each page. A job template that accepted provisioning callbacks is then bound to its AWX id, so it also answers at its old AWX callback address. An export that carries each job template's id needs no list. See the AWX-compatible callback address.
• import semaphore <export.json> [--apply] brings the same kinds from a Semaphore export, apart from the dynamic inventory sources AWX alone carries.
• import chef <nodes.json> [--apply] brings one inventory of the fleet: every node a host, grouped by its chef_environment and by every role[...] in its run list, carrying the ohai facts that identify a machine, with ipaddress also set as ansible_host. Accepts an array of node documents, a single node, or an object keyed by node name. Cookbooks and recipes are not imported and the recipes seen are named in the report.
• import puppet <nodes.json|facts.json|nodes.txt> [--apply] brings one inventory of the fleet, grouped by environment, from a PuppetDB nodes query, a PuppetDB facts query, or the plain certname list puppet node list prints. Deactivated and expired nodes are left out and counted. Manifests and modules are not imported.
• import rundeck <jobs.yaml|project-archive.zip> [--inventory <name>] [--apply] brings templates, surveys, and schedules from either a job export or a project archive, told apart by content. An archive brings one project as well, but only when its source control configuration names a repository this can reach. Neither artifact carries a node definition, so no inventory is imported from either and --inventory names the hosts its jobs target.
• import jenkins <JENKINS_HOME|jobs-dir|job-dir|config.xml> [--inventory <name>] [--apply] brings templates, surveys, and schedules from freestyle jobs. Jenkins picks an agent by label, so --inventory names the machines.
• import cron <crontab-file> [--inventory <name>] [--system] [--apply] brings schedules alone, one per crontab line, each carrying its own one-step bash pipeline rather than a template. --system parses the six-field /etc/crontab form, whose user column sits before the command. That step runs on the SwitchTender host, not on the machine the crontab came from.

All seven take --db for the target database. Without --apply the command only reports what it would create. Rundeck, Jenkins, and cron import no credentials, so a job that needed a login needs one built by hand afterward.

audit

Audit trail tools.

• audit bundle emits the chain as a signed LoomSeal bundle. Anyone verifies it offline with the open loomseal verifier, or in a browser, without trusting the server that produced it.
• audit anchor has a public timestamp authority sign the current head, so a chain that has lost its tail no longer reaches its anchor.
• audit receipt <seq:link> redeems a receipt the server issued, proving that entry is still in the chain.
• audit report renders the period's change register as a self-contained HTML evidence report.
• audit run <id> emits one run's evidence dossier as a self-contained HTML document.

Flags on the subcommands:

• audit bundle --limit <n> carries only the newest N entries. The default carries the whole chain.
• audit anchor --type <kind> picks the anchor: rfc3161 for a signed timestamp, or git or https for one checked by fetching it. --ref is the authority URL for rfc3161, otherwise the URL a verifier fetches. --tree anchors the Merkle root over the whole chain, which is the coordinate a sparse receipt proves membership against.
• audit report --from <when> --to <when> bounds the change register's period. --from defaults to ninety days before --to, and --to defaults to now and is exclusive.

receipt

Writes a signed receipt for one finished run, which a third party verifies offline with verify. A receipt is the chain segment from the request that created the run through the entry recording what it did, signed with this install's key and carrying any anchors that fix its position.

Publish the key fingerprint the command prints so a verifier can pin it.

verify

Verifies a receipt written by receipt. It trusts nothing this server says: it recomputes every chain link from the receipt's own claims, checks the signature covers the exact bytes, and confirms any anchors name an entry the receipt holds. It reads only the file, reaches no database and no network, and does not run the server, so a relying party can check a receipt on a machine that has never seen this install.

switchtender verify run.receipt --pubkey sha256:...

Pass --pubkey with the fingerprint the producer published to tie the result to a key obtained out of band. Without it the receipt is checked against the key it names, which proves it was not altered but not who signed it, and the verdict says so: INTACT, BUT UNIDENTIFIED rather than VERIFIED. A --pubkey given an empty value, which is what a failed key fetch leaves behind, is refused rather than treated as no pin.

witness

Watches a server's span beat feed from outside it. A chain proves what it holds was not altered, but not that nothing was removed from the end, because the process running the chain also decides what gets written down. A witness on another machine remembers what the feed served, keeps that memory in a signed checkpoint, and raises a finding when a beat goes missing, an already-witnessed beat comes back rewritten, or the head regresses. Run it where the server's operator has no hand.

The server has to be writing beats for there to be anything to watch, and beats are off by default. Start the server with --span-cadence, for example --span-cadence 60s. Against a server with beats off, the witness reports empty_feed, and a --once run exits nonzero.

• witness serve watches many servers from one process and answers auditors with countersigned attestations.
• witness verify-attestation verifies an attestation offline against a pinned witness key.

witness serve

Runs the witness. Its flags are checked against that subcommand rather than witness itself.

witness verify-attestation

Verifies one attestation offline, for the relying party. It reads the JSON, recomputes the signature, and prints the verdict with the signer named in both published forms: signed_by is the raw hex public key the attestation carries, and signed_by_key_id is the sha256 key id the witness prints at startup and serves from its API.

switchtender witness verify-attestation attestation.json --pubkey sha256:...

The verdict carries pinned. Without a pin, ok means only that the document is internally consistent, which an attestation a forger signed with their own key also is, so an unpinned verdict carries a note saying so. An empty --pubkey is refused.

Pass --pubkey with the key you pinned out of band. Either form is accepted, since the key id is what the witness tells operators to publish and the hex key is what the document carries. Without it the check proves only that the attestation is internally consistent, which a forger with their own key satisfies trivially.

mcp

Serves the Model Context Protocol over stdio, so an agent can list templates, propose a run, and read what happened. Every tool call is an ordinary authenticated API request carrying the token given here, so it passes the same authorization, the same approval policy, and the same fail-closed audit append as a request from a person. A run it proposes waits for a person's approval unless a rule with effect: exempt covers it, as agent runs are held by default describes. See Agents.

export SWITCHTENDER_MCP_TOKEN=swt_...
switchtender mcp --server https://switchtender.internal

The token is read from SWITCHTENDER_MCP_TOKEN, falling back to SWITCHTENDER_TOKEN. Prefer the environment variable to a flag, whose value is visible in the host's process list. The command refuses to start on an admin token. There is deliberately no approve tool, so an agent cannot release its own work, and no credential, account, token, grant, or policy tool, so it cannot widen its own reach.

demo

Seeds a fresh database with sample data and real runs, then serves it read-only, so a public instance is safe to expose. It needs ansible on the PATH to run the sample playbooks.

Seeding runs real playbooks and takes a couple of minutes, which is a visible gap if a public demo reseeds in place. The two flags split that work in half so it can happen off to the side:

switchtender demo --db next.db --seed-only     # build the next database, serving continues
mv next.db demo.db                             # swap it in
switchtender demo --db demo.db --no-seed       # serves the prepared data in under a second

A host wired this way reseeds without a visible outage, since the running instance keeps answering the whole time the replacement is being built.

examples

Seeds a handful of starter templates, so a first launch works on the spot rather than opening on an empty list. They use the Bash tool and print or read something local, needing no project, inventory, or credential. Delete them once you have your own.

Run it against the same database serve uses. It skips a template whose name is already present, so it is safe to run twice. Takes --db.

backup

Writes an encrypted, portable backup of the control-plane configuration and secrets. The whole file is sealed with the deployment encryption key, so it stays confidential and tamper-evident, and it restores into either the SQLite or the PostgreSQL backend. See Backup and restore.

Run history and the audit chain are not included. The audit chain has its own signed export, through audit bundle.

restore

Reads a backup and upserts its objects into the store by id. It needs the same encryption key the backup was written with, and it never deletes objects absent from the file.

version

Prints the SwitchTender version.

--verify fetches this version's published binary hashes and verifies the running executable.

help and completion

Both are the standard Cobra built-ins. help prints usage for any command, and completion emits a shell completion script for bash, zsh, fish, or PowerShell.

Output flags

JSON goes to stdout compact by default. Four commands take --pretty to indent it. It is not a global flag, so passing it elsewhere is an error rather than a no-op.

Separation between organizations is opt-in

Organizations, teams, and grants exist, but with --strict-grants off an object nobody has granted falls back to the caller's global role. On that default any operator may use any project, inventory, or credential in the install, whichever organization it belongs to. That is deliberate: a small team should not have to grant every object before anything works, and an upgrade should not lock people out of what they were already using.

It does mean an install with several organizations on it is not separated until --strict-grants is on. If you are running work for more than one team, more than one customer, or anything where one group must not reach another's credentials, turn it on and grant deliberately. Objects created before you do carry no grants, so plan to assign them.

A refusal from these rules says so: the body reads forbidden: this object requires a grant you do not hold, which is the same sentence whether the object is delegated elsewhere, ungranted under strict grants, or does not exist. It tells you a grant is what is missing rather than a role or an account, and nothing about the object itself.

What --strict-grants decides is only the default for an object nobody has granted. An object that carries a grant is access-controlled in both modes: writing a grant on it is what declares it so, and a caller who does not hold one is denied whether or not strict grants are on. Every listing answers the same rule the by-id read does, so a run you are refused by name is also absent from the run list, the fleet and drift views, the host pages, and the change log.

One consequence is worth knowing before you write your first grant. Install-wide figures that carry no per-row id are served whole only to a caller nothing is hidden from, because such a total is every other group's volume in one number. On an install with no grants at all that is everybody, which is the common case and is unaffected. It stays unaffected by a grant on a template or a worker queue, since neither can hide a run. What changes it is a grant on a project, an inventory, or a credential that the caller does not hold.

For a caller in that position each surface answers differently, and the difference is worth knowing before you point a dashboard at one:

The empty /metrics body is deliberate: a scraper reads a 5xx as the install being down and pages somebody. It does mean an alert written against a series going to zero will not fire for a scrape token that has become restricted, because the series is gone rather than zero. Scrape with a token that holds everything, or alert on the scrape's own staleness as well as its values.

Directory sign-in and existing accounts

An account records what created it: an administrator, or the directory that provisioned it. A directory identity signs in to an account only when that account is its own, so an identity provider asserting the username of a local administrator is refused rather than handed that administrator's account and role.

This matters most when the username does not come from a stable subject. With --jwt-username-claim email or a SAML username attribute pointing at an email, against an issuer that lets a user set their own address, matching on username alone was account takeover. OIDC has always refused the equivalent when the provider does not vouch for the address.

An account created before this was recorded carries no source, so it cannot be told apart from a local one of the same name. Those are still signed in to, because refusing them would lock out every directory user on upgrade, and each is logged so an administrator can set the source and remove the ambiguity.

Attention thresholds

The overview's Needs attention panel, GET /v1/attention, sorts waiting work into four answers to what is stopping it: a lost worker, no worker for its queue, an approval, or a block behind another run. Five thresholds decide when work shows as blocked and when each answer alerts. An alert is listed by the doctor and told, once per condition, to the server-wide chat channels, webhooks, ntfy, and email, to a run's own targets of those kinds, and to the named targets attached for the attention event, on every kind including PagerDuty. The API reference describes the answer.

Each value is a Go duration such as 90s, 15m, or 4h, or off. Zero is refused rather than read as off. The file states the organization's defaults and may override them for one organization on the install, one queue, or one template, and the most specific wins: a template over a queue, a queue over an organization, an organization over the defaults. Each override replaces only what it states. The default queue is written as "".

defaults:
  alert_no_worker: 10m
  alert_approval: 8h
orgs:
  org_ab12cd34ef56:
    alert_approval: 2h
queues:
  prod:
    alert_no_worker: 2m
    blocked_after: 5m
  "":
    alert_blocked: 30m
templates:
  tpl_0123456789ab:
    alert_approval: 30m

An unknown key, a value that is not a duration, and blocked_after: off are refused when the server starts, so a typo never silently leaves an alert on its default or off.

Confining relay workers to their queues

A relay worker runs in a segment the control node cannot reach, which means the least trusted machine in the estate holds a worker token. With a single --worker-token, that machine may name any queue it likes and lease from it, so a compromised host in a DMZ can take a production run and execute it with production credentials.

--worker-pools binds each token to the queues it may serve. The file stores the SHA-256 of each token, never the token itself, the same way a webhook secret is stored:

workers:
  - name: dmz
    token_sha256: 9f2c...           # sha256 of that pool's bearer token
    queues: [dmz]
  - name: production
    token_sha256: 41ab...
    queues: [prod, canary]

A pool that declares no queues may lease from all of them, which is the single-token shape stated out loud. A pool that declares queues is refused anything else, including the default queue when it names none, so confinement cannot be escaped by omission.

Generate a digest with printf %s "$TOKEN" | shasum -a 256. A malformed file stops the server rather than falling back to no confinement, because an install that believes it is segmented and is not is worse than one that refuses to start.

That confines the lease side. The submit side is confined by granting the queue, which is a grantable object named queue:<name>:

curl -X POST localhost:8080/v1/grants \
  -H "Authorization: Bearer $ST_TOKEN" \
  -d '{"subject": "team_sre", "object": "queue:prod", "access": "use"}'

A queue nobody has granted follows the same rule every other object does: the global role decides, unless --strict-grants is on, in which case an ungranted queue is refused. Granting a queue makes it access-controlled, so only the subjects named may route work to it. The grant is checked wherever a queue is chosen: on a run, on a template, on an inventory, and at launch against the template's own queue.

An install that has not turned strict grants on can gate a queue with a rule instead, since a policy matches on queue:

policies:
  - name: hold anything headed for production
    queue: prod
    require_distinct_approver: true

A policy with queue holds only what is routed to that queue, and a run on a named queue executes only when a worker serving that queue claims it. The control node's own runner takes unqueued runs alone. On an install without relay workers, a queued run therefore waits as pending indefinitely, even after approval. To gate runs the server itself executes, write the policy without queue.

Delivering secrets to relay workers

A relay worker holds no database and no encryption key, so on its own it cannot open a credential, and a run that needs one fails on it. A pool opts in to sealed delivery by registering a delivery key. When one of its workers claims a run, the control node opens that run's secrets and nothing else: its credentials and those of the inventory it targets, the registry login its image is pulled with, its custom credential types' field values, from which the worker renders their files, its secret survey answers, and, for a federated credential, an identity token minted for the run. It seals them to the pool's key, bound to that claim, and sends them with the claim. The worker opens them in memory when the run first needs one, writes them into the run's private directory exactly as a worker with database access does, and removes them when the run ends, whether it succeeded, failed, was canceled, timed out, or never started.

To opt a pool in:

1. Generate its key, on a worker or wherever you then copy the private key from:

   switchtender worker key new --out /etc/switchtender/delivery.key
   

2. Register the printed public key on the pool in the worker pool file every control node reads with --worker-pools:

   workers:
     - name: dmz
       token_sha256: 9f2c...
       queues: [dmz]
       delivery_key: x25519:x9f9/kkB6Jq3Wqht/7TWzDGUeTRfbABrj1CO+8/6hHc=
   

3. Restart each control node so it reads the file, and start every worker of the pool with the private key:

   switchtender worker --server https://switchtender.example.com --queue dmz \
     --delivery-key /etc/switchtender/delivery.key
   

The rules it follows:

• Delivery is off until a pool registers a key. A pool without one is sent nothing, and a run that needs a secret fails on its workers with a message naming the pool and how to register a key, which is what every relay worker did before delivery existed.
• Only a pool bound to explicit queues is sent secrets. A pool's queues are the runs it can claim, and so the runs whose secrets its key unlocks, and a pool with no queues could claim anything. A pool file that gives a key to a pool with no queues, or the same key to two pools, stops the server at startup.
• A worker refuses a key file another account can read, the way ssh refuses an unprotected identity file, and refuses to start with one.
• Credential sources resolve on the control node, so a worker in an isolated segment needs no route to Vault or a cloud secret manager. A dynamic secret minted for the run is revoked when the worker reports the run finished, or within about thirty seconds of its claim ending any other way: a worker that died, a run the janitor interrupted or requeued, or a finish recorded through another replica.
• For a Vault dynamic secret the control node also records a revoke handle in the database: the lease id and the Vault address, sealed with the encryption key, beside the run and the credential that minted it. The secret itself is never stored, and the handle is never logged or returned by the API. Every control node with the encryption key sweeps these records, so a secret is revoked once its claim ends even when the node that minted it was stopped or restarted in between. The revoke uses the credential's token, or VAULT_TOKEN for a credential that names the Vault in VAULT_ADDR, and is refused when the credential now names another Vault. A revoke that fails is retried with backoff until the lease would have expired anyway.
• Two kinds of dynamic source give no handle. A dynamic source a plugin adds, whether compiled in through the SDK or loaded as an external plugin, is revoked only by the control node that minted it, so when that node stops first its secret expires on its own lifetime. AWS STS credentials cannot be revoked early at all and always expire on their own lifetime.
• A federated credential's token is minted on the control node, whose signing key never leaves it, for the mode the worker will execute: a plan for a dry run or for an apply a plan-content rule plans first, and an apply otherwise. A worker about to execute in the other mode refuses the token.
• Anything that goes wrong is refused rather than worked around. A secret that does not open on the control node, a delivery that cannot be recorded in the audit trail, a delivery sealed to a key the worker does not hold, or one that does not open for the claim fails the run with the reason. A run never executes with a secret missing.

The seal is HPKE (RFC 9180) in base mode with DHKEM(X25519, HKDF-SHA256), HKDF-SHA256, and ChaCha20-Poly1305, from the Go standard library. HPKE was chosen over NaCl's anonymous sealed box because it authenticates associated data, so the binding is checked by the decryption itself. Each delivery uses a fresh ephemeral key, and the run id, a hash of the claim's lease, the worker's lease name, the pool, the key id, and a random delivery id are all authenticated. So:

• A worker of another pool holds another key and cannot open the delivery.
• A delivery presented for another run, for a later claim of the same run, or by another worker does not open.
• A worker opens a given delivery once and refuses it if it is presented again.
• A worker opens a delivery only after the control node has accepted its fenced start under the same lease, so a claim answer replayed after its run started or finished elsewhere is never decrypted. The start is refused once the janitor has taken the claim back, and to a worker that never claimed the run.
• Only ciphertext crosses the relay, so a proxy or load balancer that logs relay traffic logs nothing readable without the pool's private key.

The worker holds the opened values only until it has written them where the tool reads them, then drops them. Go cannot overwrite a string in place, so what is dropped is the reference, and the decrypted bytes themselves are zeroed. Disable core dumps on worker hosts, as on any host that runs credentialed work, and see Secrets for how the run's private directory is handled.

Rotating a pool's key

A worker can hold several keys, and a delivery names the key it was sealed to, so a rotation needs no moment where runs fail:

1. Generate the new key and install it on every worker of the pool beside the old one, restarting each with both: --delivery-key old.key --delivery-key new.key.
2. Replace the pool's delivery_key with the new public key in the worker pool file on every control node, and restart them. Deliveries are sealed to the new key from then on, and every worker opens them.
3. Remove the old key from the workers and restart them with the new one alone.

Each control node logs, at startup, the key id every pool registered, and each worker logs the key ids it holds, so the step a fleet has reached shows in the logs. Every delivery's audit record names the key id it was sealed to.

When a key may be compromised, replace the pool's delivery_key and restart the control nodes first, then install the new key on the workers. Until a worker has it, credentialed runs on that pool fail rather than run. Treat the secrets of runs the pool executed as exposed and rotate them at their source, and rotate the pool's worker token too, since whoever read the key file could read the token beside it.

What a relay worker writes into the audit trail

A relay worker runs where the control node cannot see it, so two moments are recorded: the run leaving for that machine, and the outcome coming back. A run whose secrets were delivered records a third, between them, naming which pool and worker received which credential ids and secret answer names, under which key, and never a value:

RELAY  /relay/claim/run_4f21a9                       pool:dmz worker:build-dmz-01
RELAY  /relay/delivered/run_4f21a9/key/f4e8a76a457da19cf3742de37986d439/credentials/cred_1a2b3c,cred_4d5e6f/answers/db_password
                                                     pool:dmz worker:build-dmz-01
RELAY  /relay/finished/run_4f21a9/succeeded          pool:dmz worker:build-dmz-01

Captured output, structured events, per-host and per-task summaries, and heartbeats are not recorded. They are the content and liveness of a run that is already stored on the run itself, they arrive several times a second, and writing each into a hash chain would drown the record it exists to make readable.

The claim and finish records do not fail closed, unlike every mutation through the API. Refusing a worker's report because the audit store is unhealthy does not un-finish the run. It loses the outcome of work that already ran on real hosts, so a failure to record is logged loudly instead. The delivery record is the exception: it is written before anything is sent, and a delivery that cannot be recorded is not sent, so the run fails on the worker with that reason.