HTTP API

Every endpoint the server exposes. The API is served under the /v1 base path. The web UI at /ui/, along with /healthz, /readyz, /metrics, /.well-known/loomseal.json, the workload identity federation documents under /.well-known/, the OpenID Connect and SAML sign-in routes, the webhook /hooks path, and the /relay worker path, is unversioned. The root redirects to the UI. A template's provisioning callback, POST /v1/templates/{id}/callback, is versioned and carries no token: the host config key in its body is the credential. The same holds for the one AWX path served, a template's AWX-compatible callback address, POST /api/v2/job_templates/{awx id}/callback/. Nothing else under /api/ is served.

/.well-known/loomseal.json is deliberately public and unauthenticated: it publishes this install's signing key and identifiers so a relying party can pin the key from a channel independent of any bundle it is handed.

/.well-known/openid-configuration and /.well-known/jwks.json are public and unauthenticated for the same kind of reason: a cloud verifying a run's identity token fetches them and has no account here. They hold public keys only, and answer 404 until --federation-issuer is set. See Federated cloud credentials.

Licensed endpoints

Almost every endpoint below runs on Community, which needs no license. These answer 403 without one, with a message naming the feature rather than failing in some subtler way:

Two more are enforced somewhere other than the request:

• /relay is served whatever the license. The gate is on the other end: switchtender worker --server refuses to start without Team, and so does a worker sharing the database: every worker process is gated.
• The directory sign-in routes need Pro, enforced at startup rather than per request. Configuring any of OIDC, SAML, LDAP, or JWT without a license refuses the server at startup, so those routes are either licensed or absent.

A streamed export whose status line has already been sent cannot report a later failure with a status code. The run event NDJSON download and the run log download therefore end with a {"export_incomplete":true,"reason":"..."} line when they stop early, so a short file is never mistaken for a whole one.

When a run needs a credential that is not ready

A launch, a rerun, a retry, a relaunch, and a webhook answer 409 when the run needs a credential that has no secret yet, which every imported credential is until someone sets one, or a credential whose sealed secret does not open under this server's encryption key and salt. The error names the credential and the fix: set its secret, or restore the key and salt it was sealed with.

When a launch cannot start for its configuration

Every path that launches work sorts its errors the same way and answers each kind to suit who sent it. A configuration error is one someone must fix before the launch can succeed: an inventory that is not a valid inventory, names more hosts than one inventory may, or needs Ansible this server lacks, or a project, inventory, or credential that no longer exists. A run submission, a template launch, a pipeline, a drift reconcile, and an AI proposal answer it 400 with the whole reason, and a rerun answers it 409.

A webhook trigger, a pull request plan, and a provisioning callback answer it 409 and name the error, as in this template cannot launch: invalid inventory document, and no more. The detail can quote what the inventory holds, and the sender is not its owner. The server log keeps the detail, and launching the template by hand shows it. A webhook's refusal is also its trigger's last_error, and a pull request plan's is a comment on the pull request. A constructed inventory that names an input that no longer exists, and an input of a composed inventory that does not parse, are configuration errors too. Those three paths answer them 409 and name the error, as in inventory not found or composed inventory could not be resolved.

Content that could not be read is an inventory whose content was fetched from its content source (a command, Vault, or a secret manager) or read by ansible-inventory, and the read failed. A person's launch answers it 400 with the whole reason, and a rerun 409, so the person can see whether the source refused or was down. A webhook trigger, a pull request plan, and a provisioning callback answer 502 with this template cannot launch: inventory content could not be read, and no more. The same read fails for a refused request and for an outage, and a sender that retries a 502 gets past an outage. A webhook's answer is also its trigger's last_error, and a pull request plan gets no comment for it. A /switchtender apply comment gets one reply, This apply could not start: inventory content could not be read. Nothing was applied., which tells the commenter to comment again once the content can be read.

A failure of the server or its stores while composing an inventory, such as a store that did not answer, is the server's failure, and so is a snapshot the server could not seal or reopen when the run was submitted. A person gets 500 with the detail kept to the server log, a rerun included, which answered a snapshot it could not seal 409. The webhook trigger and the provisioning callback answer 502 with could not launch the template, and a pull request plan 502 with could not plan the pull request.

How long a submission can take

A submission answers once the approval gate has judged the run, and for a Terraform or OpenTofu dry run whose configuration calls registry or remote modules, the gate downloads those modules first. POST /v1/runs, a template launch, a rerun, and an agent's submission through MCP wait for that download, which --module-fetch-timeout bounds at two minutes by default, and then for the scan, so allow a client that long before it gives up. A download that runs out of time does not fail the submission: the plan is recorded as unclassified, and held or refused like any other plan the gate could not read in full. A webhook does not wait that long, since its sender would stop listening first: POST /hooks/{token} answers within five seconds, with accepted when the launch is still going.

When the chain itself refuses a bundle

GET /v1/audit/bundle recomputes the whole chain and holds it against every anchor recorded over it before any window is applied. A chain this server checked and rejected is a finding, not a fault, so it answers 409 rather than 500, and a caller can tell the two apart without reading prose. A 500 is left to a real fault here, such as a store that will not read, and a limit that is not a count stays a 400.

{
  "error":  "entry 3 does not recompute (sequence 3)",
  "reason": "chain_break",
  "broke_at": 3,
  "broke_seq": 3,
  "count": 9
}

The coordinates are the same ones GET /v1/audit/verify reports, so the two answers agree. A windowed request is refused for a break anywhere in the chain, not only inside the window: a bundle signed over a window sitting past a break would attest to entries this install cannot stand behind.

Opening a live stream

EventSource cannot set a header, so GET /v1/runs/{id}/stream takes a short-lived ticket in the query string instead of a bearer token. Mint one over the ordinary authenticated route and open the stream with it:

curl -X POST -H "Authorization: Bearer $ST_TOKEN" \
  localhost:8080/v1/runs/run_abc/stream-ticket
# {"ticket":"...","expires_in":30}

A ticket opens that one run, works once, and expires in thirty seconds. The reason is that a URL is not private: a reverse proxy logs the full request line by default, so a session token in the query string reaches every access log downstream. A ticket in the same place is worth almost nothing.

An install running open, with no tokens, needs no ticket and the plain path works.

Account profiles

An account carries an optional profile alongside its role: full_name, email, phone, title, links, and notes. All of them are optional, so an account created by the CLI or provisioned over single sign-on stays valid with none of them set. title is descriptive and grants nothing; role alone decides what an account may do.

curl -X PUT https://switchtender.example.com/v1/users/user_9f2c \
  -H "Authorization: Bearer $ST_TOKEN" \
  -H 'Content-Type: application/json' \
  -d '{
    "username": "ada",
    "role": "operator",
    "full_name": "Ada Lovelace",
    "email": "ada@example.com",
    "phone": "+1 555 0100",
    "title": "Platform Engineer",
    "links": ["https://wiki.example.com/people/ada"],
    "notes": "review each quarter"
  }'

The profile is replaced wholesale on update, so send the profile you want to end up with rather than only the parts that changed. An omitted field clears.

A profile is personal data and is treated as such. Only an admin may read it. /v1/users requires the admin role and is not delegable by a manage grant. The values are never written to the logs, and a rejection names the offending field without echoing it. Each single-line field is capped at 320 characters, notes at 2000, and an account may carry at most eight links. A link must be an http or https address. Any other scheme is refused, because the admin page renders links as anchors.

Template run timeout

A template may cap how long its launches are allowed to execute with timeout, a whole number of seconds. It is accepted on create and update, and every launch of the template carries it onto the run, whether the launch came from the API, a schedule, or a webhook trigger.

curl -X POST https://switchtender.example.com/v1/templates \
  -H "Authorization: Bearer $ST_TOKEN" \
  -H 'Content-Type: application/json' \
  -d '{
    "name": "nightly database backup",
    "playbook": "plays/backup.yml",
    "timeout": 5400
  }'

Zero, or the field omitted, leaves launches on the server default set by --run-timeout, so a template saved before this field existed is unchanged. A run that exceeds its timeout is canceled and finalized as failed. A launch cannot raise the cap. The template's value is what applies.

Naming what a run targets

Two fields name a target and they are not interchangeable.

inventory_id names a stored inventory, the kind the UI creates and the one almost every caller wants. inventory is a path to an inventory file already on the server, for a run whose inventory is managed outside this product.

Sending a stored inventory's name in inventory is read as a path. Ansible exits zero when a host pattern matches nothing, so a run aimed at a path that does not exist touches no host at all. Its recap names no host, and the run is recorded as failed with the reason on its error, since a clean exit over nothing is not a success.

Ansible run controls

A run submission and a template both accept the Ansible controls that used to require a hand-built command. They ride onto a run the same way from the API, a schedule, or a webhook trigger, and a retry keeps them.

curl -X POST https://switchtender.example.com/v1/runs \
  -H "Authorization: Bearer $ST_TOKEN" \
  -H 'Content-Type: application/json' \
  -d '{
    "playbook": "plays/deploy.yml",
    "inventory_id": "inv_3f9c1b7a2e04",
    "limit": "canary01",
    "tags": ["web", "config"],
    "skip_tags": ["reboot"],
    "forks": 25,
    "verbosity": 2,
    "diff_mode": true
  }'

These apply to the Ansible tool. The other tools ignore them, so a Bash or Terraform template that carries one is unaffected.

A run submission also accepts extra_vars, an object of variables injected into the run. Ansible receives them the way --extra-vars supplies them, and a plugin tool reads them as its input. A template carries its own extra_vars and a launch may merge more over them.

Saved workflows

A template may carry steps, a pipeline graph, instead of a single tool. Such a template is a saved workflow: every path that fires a template, a launch, a schedule, or a webhook trigger, runs the graph as a pipeline, and the template's survey answers and extra vars reach every step. A workflow template sets no top-level playbook, command, tool, shards, or Ansible controls, since each step names its own. The graph is validated when the template is saved, so a cycle or an unknown dependency is refused then rather than on every launch.

A step with "type": "approval" is an approval step: the workflow waits there until an admin approves or denies it. It takes a description and an approval_timeout in seconds, and a step names it in if_denied to run when it is denied or times out. GET /v1/approvals lists the steps waiting, and POST /v1/runs/{id}/approve or /reject with the step's id decides one, optionally with {"state_digest": "..."} from the listing so the decision binds to what was shown. The same call with the workflow's id is refused with 409, naming the step and its call. A workflow reaching an approval step is announced with its own event, workflow.step_awaiting_approval, which names the step and what each answer runs next, so a receiver can tell it from a whole run held for approval. The concepts page describes the semantics.

curl -X POST https://switchtender.example.com/v1/templates   -H "Authorization: Bearer $ST_TOKEN"   -H 'Content-Type: application/json'   -d '{
    "name": "build and ship",
    "inventory_id": "inv_3f9c1b7a2e04",
    "steps": [
      {"name": "build", "tool": "bash", "command": "make release"},
      {"name": "deploy", "playbook": "deploy.yml", "depends_on": ["build"]}
    ]
  }'

Approver reasons

An approval or a denial may carry the approver's reason, for a held run and a workflow approval step alike. It is optional unless a policy's require_reason asks for one, and it holds at most 1,000 characters.

curl -X POST https://switchtender.example.com/v1/runs/run_abc123/approve \
  -H "Authorization: Bearer $ST_TOKEN" -H 'Content-Type: application/json' \
  -d '{"reason": "approved, change window confirmed with the database team"}'

The reason passes through the secret masker before anything records it, the same masker run output and pull request comments pass through. When the masker changes the text, nothing is recorded: the answer is 409 with masked_reason, the text as it would be kept, and the decision goes through when it is sent again with masked_reason set to exactly that text. A reason is never refused for looking like a secret. The interface shows the masked text and asks the approver to confirm it, and labels the field "Stored as audit evidence. Don't include secrets or personal data." with "Masked for known secrets before storage. Treat it as permanent audit data." beneath it.

The masked text is stored with a random 32-byte value beside the decision. The chain entry for the decision commits a hiding commitment to it, never the text: sha256: and the hex SHA-256 of the RFC 8785 canonical JSON object {"event": <decision id>, "random": <hex of the random value>, "reason": <masked text>}, where the decision id is the id of the chain entry that records the decision and is committed in its body as decision_id. The request that carried the reason is recorded with the reason withheld from its fingerprint, so the text as typed is committed nowhere.

GET /v1/runs/{id}/decisions lists the run's decisions with their reasons and random values, the separation-of-duties evaluation of a decision on an agent's run, corrections, and redactions. The evidence dossier and a contiguous receipt carry the same text and random value beside each decision, so anybody holding the receipt recomputes the commitment offline with no secret, and switchtender verify does exactly that and fails a receipt whose reason does not open it.

A reason is never edited. POST /v1/runs/{id}/decisions/{decision}/corrections with {"text": "..."} appends a correction, masked and committed the same way under its own chain entry, and the dossier and the interface show it under the decision it corrects. An admin may redact a reason or a correction with POST /v1/runs/{id}/decisions/{record}/redact and {"category": "personal_data"}, or secret or other. Redaction is a privacy action, not an edit: it removes the text and the random value together and appends a chain entry recording who redacted it, when, and the category, referencing the decision. The commitment stays on the chain and can never be opened again. Receipts already issued keep what they disclosed.

A reason is audit data, subject to your organization's retention policy. It is held as long as its run: when retention removes the run, it removes the decision records with it, and the commitment that stays on the chain can no longer be opened. A rejected run's error reads "rejected by an approver" and a denied step's "denied by an approver". The reason stays in the decision record, where a redaction can reach it, and is never copied into a run's error or its committed outcome.

Fact cache

A template may set use_fact_cache, the setting AWX calls the same thing. Every launch then writes the facts earlier runs gathered for the inventory's hosts into a private directory that Ansible's jsonfile cache plugin reads, and stores what the run gathers when it ends, so a play with gather_facts: false still sees each host's facts. fact_cache_timeout is how many seconds cached facts stay fresh enough to serve, and zero serves them however old they are, which is the AWX default. Both need inventory_id, because facts are kept per stored inventory host, and both are omitted-means-unchanged on an update.

curl -X PUT https://switchtender.example.com/v1/templates/tpl_9d41c2 \
  -H "Authorization: Bearer $ST_TOKEN" \
  -H 'Content-Type: application/json' \
  -d '{
    "name": "patch web",
    "playbook": "plays/patch.yml",
    "inventory_id": "inv_3f9c1b7a2e04",
    "use_fact_cache": true,
    "fact_cache_timeout": 86400
  }'

With the cache on, a run's spec records "fact_cache": {"timeout_seconds": N}. An approval binds to it and a receipt discloses it, so a run whose fact cache setting changes after it was approved is refused, and a held run shows the setting to its approver. With the cache off the field is absent, so the digest of every such run is what it always was.

GET /v1/inventories/{id}/facts lists the cached hosts and GET /v1/inventories/{id}/facts/{host} returns one host's document. Reading either takes the operator role and read on the inventory, and a host whose gathering run the caller may not read is left out. A server started with --fact-cache-admin-only answers 403 to anyone below admin. Cached facts never enter the audit chain, a receipt, an evidence bundle, or a backup, and a document larger than one mebibyte is not kept. See the Ansible guide for how the cache behaves during a run.

Provisioning callbacks

A template may set allow_callbacks, which lets a host in the template's stored inventory launch the template against itself, usually from a boot script. The host posts the template's host config key to /v1/templates/{id}/callback. The server matches the address the request came from to one host in the inventory and launches the template limited to that host. The launch passes through the approval policies like any other.

Mint the key once, and keep it, because it is never shown again:

curl -X POST https://switchtender.example.com/v1/templates/tpl_9d41c2/callback-key \
  -H "Authorization: Bearer $ST_TOKEN"

{
  "template": "tpl_9d41c2",
  "host_config_key": "hck_...",
  "callback_path": "/v1/templates/tpl_9d41c2/callback"
}

The host then calls back the same way an AWX host does:

curl -k -f -i -H 'Content-Type: application/json' -X POST \
  -d '{"host_config_key": "hck_..."}' \
  https://switchtender.example.com/v1/templates/tpl_9d41c2/callback

A form body, host_config_key=hck_..., is accepted too. The answers are:

Turning allow_callbacks off revokes the key. A template also carries callback_limit, which says what a callback does with the template's limit: intersect, the default, launches only for a calling host the limit selects, and replace launches for any matched host, the way AWX does. Both are omitted-means-unchanged on an update.

A template an import bound to its AWX job template id is served with awx_job_template_id and, once a host has called through the AWX-compatible address, awx_callback_called_at. Neither can be set. awx_callback turns that address on or off, and can be turned on only for a template an import bound. See the Ansible guide for how a host is matched and how the launch is recorded, and the migration guide for the AWX-compatible address.

What a request may carry

Text holding the NUL byte, or text that is not valid UTF-8, in the request path or a query parameter is refused with 400 before the request is routed, so it never reaches a store. A JSON body is held to the same rule: a string holding the NUL character, written \u0000, is refused with 400 naming the field, such as steps[1].name. An import is checked the same way before it writes anything: an export holding such text in any object it would create is refused with 400 naming the object and the field.

Some values are stored under an index and have a bound. Past it the request is refused with 400 stating the bound.

List responses

Every list response is an envelope: the rows under a name, count for how many were returned, and total for how many exist. They agree on any ordinary install.

The run list pages, because run history grows without bound: limit (default 200, maximum 1000) and offset, with has_more and next_offset to continue.

The configuration lists page the same way: users, tokens, templates, schedules, triggers, notification targets, organizations and organization members take limit (default and maximum 1000) and offset, and carry has_more and next_offset. They also take id, repeatable, to read only the named objects. total counts the objects this caller may read.

Survey field constraints

A template survey field accepts bounds beyond its type, checked at launch before any answer becomes an extra var. A field also takes an optional help string shown beneath its prompt, and a multiline type for a block of text such as a set of variables or a note.

A launch that violates a constraint is refused with the field it failed, and no run is submitted.

The shape itself, which strict decoding refuses to guess at:

{
  "survey": [
    {"var": "release", "label": "Release tag", "type": "text",
     "required": true, "pattern": "^v[0-9]+\\.[0-9]+\\.[0-9]+$",
     "help": "The tag to deploy, such as v2.1.0"},
    {"var": "batch", "label": "Hosts per batch", "type": "int",
     "default": 5, "min": 1, "max": 50},
    {"var": "environment", "label": "Environment", "type": "choice",
     "required": true, "choices": ["staging", "production"]},
    {"var": "notes", "label": "Change notes", "type": "multiline",
     "max_length": 2000}
  ]
}

type is one of text, multiline, secret, int, choice, or bool, and integer, boolean, string, textarea, and password are read as int, bool, text, multiline, and secret. var names the extra var the answer becomes, and it is the only field besides type that every entry must carry. An unknown key is refused rather than ignored, so a survey that almost parses is reported instead of silently losing a field.

A launch answers the survey under answers, keyed by each field's var, with an int answered as a number and a bool as true or false:

{"answers": {"release": "v2.1.0", "batch": 10, "environment": "staging"}}

Secret fields

A secret field asks for a password, a token, or any other answer that must not be kept as text. AWX calls the same thing a password field. The answer is checked against the field's bounds like a text answer, then sealed with the credential key before anything is stored, the same AES-256-GCM sealing a credential's secret gets. What follows from that:

• The run keeps only the sealed answer. Its extra_vars never hold it, and a run read through the API, the UI, an export, or an agent's MCP tools lists the variable under sealed_vars by name only, which says an answer was supplied and nothing about what it was.
• The executor opens the answer inside the execution that uses it and hands it to the tool the way every other answer arrives: as an Ansible extra var, in SWITCHTENDER_VARS and its own SWITCHTENDER_VAR_<name> entry for a script, or as a TF_VAR_ entry for Terraform and OpenTofu. A secret answer of the same name wins over a template extra var.
• The answer is masked in the run's log and its events wherever the tool prints it, whatever the variable is called, the way a credential's secret is. A value shorter than four characters is not masked, the same floor credentials have.
• The audit chain, the receipt, the dossier, and the SIEM stream record that a secret answer was supplied, by variable name in the run's spec, and never its value or its sealed form.
• An approval covers which sealed answer runs, not only that one was given. A run created with a secret answer lists, under sealed_var_digests, the SHA-256 of each answer's sealed form, fixed when the run is created and part of the spec an approval binds and a receipt discloses. A digest of ciphertext sealed under the server's key says nothing about the answer. The executor opens a sealed answer only while it still matches its digest, so an answer swapped for another under the same name after the run was approved fails the run with the reason and never reaches the tool. A run created before digests existed carries none and keeps the spec it had.
• Launching a template that asks a secret question needs SWITCHTENDER_ENCRYPTION_KEY. Without one the launch is refused rather than storing the answer as text.
• A relay worker receives a sealed answer only when its pool registered a delivery key: the control node opens the answer at claim and seals it, with the run's credentials, to that pool's key, bound to the claim. A run carrying one on a pool with no key fails on the relay worker with the reason instead of running with the answer missing. See Delivering secrets to relay workers.

A secret field may carry a default. It is sealed when the template is saved, and every read of the template, an admin's included, shows it as "[redacted]" to say that a default is set. Send that mask back unchanged on an edit and the stored default is kept, send a new value to replace it, or send an empty string to clear it. A launch that leaves the field unanswered uses the sealed default without it ever being opened outside the execution. In the launch dialog the field is a password input that is never filled in, and leaving it empty is what uses the default.

A relaunch, a rerun, a retry, a shard of a split, and every step of a saved workflow carry the sealed answer of the run they come from, which is what AWX does on relaunch, held to the digest that run was created with. Retention removes the sealed answer with the run it belongs to. An agent may answer a secret question on a template it is allowed to launch, and no caller, agent or person, can read an answer back.

Launches nobody answers

A schedule, a webhook trigger, a pull request plan, and a provisioning callback launch a template with nobody there to answer its survey, so each one fills it in the same way:

• Every question takes its default. A plain question's default becomes an extra var, over a template extra var of the same name, and a secret question's sealed default rides onto the run still sealed, in place of any template extra var of that name.
• A required question takes its default too, and the default is checked against the question's rules the way an answer is. An empty default does not count as one.
• A required question with no default, or with a default its rules refuse, has no answer, so the launch is refused and no run is created. The refusal names every such question and says to give it a default or launch the template by hand.

Where the refusal is recorded depends on what fired:

The questions are the variable names, comma separated. The refusal is recorded in place of a fire, so the trail never shows a fire that ran nothing. On a push trigger, last_error says why the most recent delivery started no run, whatever the reason. On a review trigger it records a plan refused for its survey, and the pull request hears about every other refusal. Either clears when a delivery starts a run. The Doctor reports an enabled schedule whose template cannot be answered as broken, before its first fire.

Per-template notifications

A template or a run submission may carry notifications, a list of targets that receive its terminal state in addition to the server-wide channels, and hear when a rule holds it for approval. Each target names a kind and the field that kind is addressed by, plus an optional on_failure that limits the target to failed runs, which also leaves it out of holds. A hold reaches webhook, chat, ntfy, and email targets, and never a PagerDuty, Grafana, or Twilio one: a hold asks for a decision, and those report incidents.

curl -X POST https://switchtender.example.com/v1/templates \
  -H "Authorization: Bearer $ST_TOKEN" \
  -H 'Content-Type: application/json' \
  -d '{
    "name": "prod deploy",
    "playbook": "plays/deploy.yml",
    "notifications": [
      {"kind": "slack", "url": "https://hooks.slack.com/services/T0/B1/x"},
      {"kind": "pagerduty", "key": "R0UTINGKEY", "on_failure": true},
      {"kind": "email", "to": "oncall@example.com, lead@example.com"}
    ]
  }'

A webhook, a target or one named with --notify-webhook, receives one body shape: {"event": "run.finished", "run": {...}} when a run reaches a terminal state, and the same with "event": "run.held" when a rule holds it for a person to decide on. Two more events use the same shape:

The chat channels, ntfy, and email say the same in words: a step's message names the workflow, the step, what it asks, and what approving and denying each run, and an alert leads with its summary and who can act. A step waiting reaches the same channels a hold does. An alert reaches the chat channels, webhooks, ntfy, and email, and is emailed whatever --notify-on says, since it reports a problem. A run's own PagerDuty, Grafana, and Twilio targets hear neither, the same rule holds follow.

A malformed target is refused at create or update with the field it lacks, not dropped at delivery. A Twilio or email target names only a recipient. The account credentials stay in server flags, so a template never carries them. On read, webhook URLs, PagerDuty routing keys, and Grafana tokens come back masked. An edit that echoes the mask back keeps the stored value.

Notification targets

A notification target is a channel defined once and attached to many objects, the way AWX attaches a notification template. It takes the same kinds and fields as a per-template notification: webhook, slack, mattermost, rocketchat, discord, teams, and ntfy with a url, pagerduty with a key, grafana with a url and a key, and twilio and email with a to. Each target has a name, an optional description, and an optional org_id whose members may use it.

curl -X POST https://switchtender.example.com/v1/notifications \
  -H "Authorization: Bearer $ST_TOKEN" \
  -H 'Content-Type: application/json' \
  -d '{"name": "ops slack", "kind": "slack", "url": "https://hooks.slack.com/services/T0/B1/x"}'

The url and key are sealed with the install's encryption key before they are stored, so creating a target that carries either needs SWITCHTENDER_ENCRYPTION_KEY and SWITCHTENDER_ENCRYPTION_SALT and is refused with 409 without them. Neither is ever returned. A read answers url as a hint masked to its scheme and host, key_set when a key is stored, and needs_secret for a target an import created without the secret it needs. An edit that sends a blank or masked url or key keeps the stored one, so a target can be renamed without its secret being entered again, and a stored secret never has to travel back through a form to be kept.

A read also carries delivery, the target's delivery status. Its state is needs_secret for a target waiting for its secret, configured for one that has delivered nothing yet, healthy when its latest finished delivery arrived, retrying while a failed attempt waits for its next one, and failing when its latest finished delivery failed after its retries. Latest means the one that finished last, so a delivery retried for minutes counts from when it finished, not from when it was recorded. Beside it are missing, the times of the latest delivery and failure, the latest failure's reason, and how many failed in the last seven days. A target waiting for its secret keeps every part an import carried, such as a Grafana instance's address, and missing names what it still lacks: url, key, or both. An edit that sends only those parts finishes it. An edit that leaves one still missing is saved, and the target keeps waiting.

An attachment ties a target to an object for one event:

curl -X POST https://switchtender.example.com/v1/notifications/ntf_ab12cd34ef56/attachments \
  -H "Authorization: Bearer $ST_TOKEN" \
  -H 'Content-Type: application/json' \
  -d '{"object_kind": "template", "object_id": "tpl_abc123", "event": "failure"}'

When a top-level run reaches an event, every target attached for that event to the template it came from, the schedule that fired it, its project, its organization, or the organization that owns its template is told, once, however many of those it is attached through. The template is found through the run's source, so a target on a template hears the runs its schedules and triggers fire, and a rerun's runs as well. A split's shards and a pipeline's steps are announced through their parent rather than one by one.

A read that fails while the template is being found, of the schedule, trigger, or run that names it, or of the template to learn its organization, counts as a database that cannot be read. The event is asked for again and is not recorded for the other targets alone, as described under Delivery order. A template that no longer exists is different: it has no targets left to find, and asking again finds nothing more. Its targets are passed over, the others are told, and the server logs a warning that names the run and what could not be found. That includes a template whose schedule, trigger, or rerun origin was deleted after the run started.

A target is delivered with the same formatters and the same refusal of private addresses as every other channel, and with SWITCHTENDER_EGRESS_PROXY set it leaves through that proxy, after its address is checked, as every channel does. A hold reaches a PagerDuty, Grafana, or Twilio target never, the same rule a template's own targets follow. The rule is applied to the target as it stands when the message goes out, so a target changed to one of those kinds after a hold was recorded for it skips the hold, and the delivery is recorded as skipped with why. A target attached for approval hears a whole run held and a workflow waiting at an approval step alike, each as its own event, run.held or workflow.step_awaiting_approval. A target attached for attention hears an alert on every kind, a PagerDuty, Grafana, or Twilio target included, since attaching one for that event is the request to be paged. A PagerDuty incident takes the alert's id as its dedup key, so one condition opens one incident. A started event reaches a webhook as "event": "run.started", and a webhook target's body also carries delivery: {"id": "ntf_ab12cd34ef56/run_x1y2/2", "sequence": 2}, with note added when an earlier message failed. The same id arrives as the Idempotency-Key header.

Delivery order

For each run and notification target, SwitchTender attempts delivery in the run's event order. A failed notification does not prevent later ones from being attempted after its retries run out.

Each event is recorded as the run reaches it, with the run's next sequence number, in the same transaction that queues one delivery per attached target. A database that refuses the write for a moment, or cannot be read to find the targets, is asked again for up to five seconds. A run's end is also marked owed in the same database write that ends the run, whatever ends it: the run's own server, a worker's report, the lease sweep, a cancel, or a decision. Every server records an end still owed a few seconds later, so a server that stops right after a run ends, or a database that refused the end for longer, leaves the end late rather than lost. A run's end is recorded once, however many times it is announced.

A run's start and its hold are marked owed the same way. A start is owed in the write that moves a run into running, whether a worker claims it, a coordinator starts it, a decision starts it, or a workflow resumes after an approval step. A hold is owed in the write that creates a run held, or moves it into a hold from a state that was not one, including a workflow parked at an approval step. A decision that takes a held run and lets it go again is not a new hold. Every server records a start or hold still owed a few seconds later. A parked workflow's hold is recorded as each approval step it waits at, the way the workflow announces them. A run's start, its own hold, and each approval step's hold are recorded once, however many times they are announced.

When the sweep finds that the run has already moved past a start or hold that is still owed, because it ended, was decided, or moved on, the event is not sent. Telling a target that a run started after it ended, or asking a person to decide a run somebody already decided, would arrive after the messages that followed it. The event is recorded on the run as skipped, with the reason, for example not sent: the run had already ended, failed, before its start could be announced. It shows at GET /v1/runs/{id}/notifications like every other skipped delivery.

A delivery waits until every earlier delivery to the same target for the same run has finished. Targets never wait for each other: a server attempts at most four deliveries to one target at once and keeps attempting the others' beside them, so a target that never answers holds only its own share. Steps of a workflow that run beside each other are not put in an order between themselves: an approval step's message follows the workflow's start and every step it comes after, and the workflow's end follows all of them.

A failed attempt is tried four more times over about three minutes. A target that refuses a message outright, with any 4xx but 408, 425, or 429, or a channel this server has no transport for, is not retried. An attempt cut short because its server is stopping is not counted, and the next server to claim the delivery makes it. When the attempts run out the delivery is kept as failed, and the next message to that target about that run carries the note "An earlier notification about this run to this target could not be delivered." The failure shows on the run at GET /v1/runs/{id}/notifications and on its page, in the target's delivery status and at GET /v1/notifications/{id}/deliveries, and in the doctor. A target waiting for its secret when an event happens is recorded as skipped, with why.

Deliveries are idempotent on the target, the run, and the event's sequence number, so a retry, a restart, or several servers delivering from one database never queue or claim the same delivery twice. Events live in the database, so a delivery pending when a server stops is made by the next process to start on it. Nothing further is promised: not the order a receiver sees messages in, not that each arrives exactly once, and not that any arrives.

This is how named targets are delivered. The server-wide --notify-* channels and a template's own notifications list keep their direct, best-effort delivery.

A schedule's skipped fire is announced the same way, to the targets attached for skipped to the schedule, the template it fires, or its organization, and to the targets attached to the schedule itself for failure. An AWX import never attaches anything for skipped, and whoever watches a schedule's failures is who needs to know it stopped reaching hosts. A target attached only for started, success, or approval does not hear it. Each channel says skipped: no hosts matched and never failed, and like a hold a skip never reaches a PagerDuty, Grafana, or Twilio target. A webhook receives "event": "schedule.skipped" with a run-shaped body that names the schedule as its source_id and its inventory as inventory_id, carries kind skipped_fire, and has no id, because no run was started.

A target's secrets are opened at delivery and never written onto the run, so they stay out of the run record, its receipt, and its evidence, where a template's inline targets are stored on the run. The server-wide channels and per-template notifications keep working beside targets.

Deleting a template, a workflow, a schedule, a project, or an organization removes its attachments in the same transaction, and leaves the targets. The delete's own audit entry states the cleanup, the count and the targets, in its path, such as /v1/templates/tpl_abc123?notification_attachments=2&notification_targets=ntf_a,ntf_b, rather than as entries of its own. A delete that finds the attachments changed after its entry was written deletes nothing and answers 409. The doctor reports any attachment whose object or target is gone, from whatever path it was left behind.

Attaching asks two things: use of the target, and management of the object, which is admin, or a manage grant on a template or project, or admin of the organization that owns it. Attaching to a schedule or an organization is admin work, the same as editing one. A grant on ntf_<id> scopes a target the way a grant on a credential scopes it.

What needs attention

GET /v1/attention answers the question an operator asks of work that is not moving: what is stopping it. Every unfinished run, workflow, split, and schedule falls under at most one of four answers, its main blocker, and counts says how many fall under each, so the four add up to the items listed:

An item in several conditions at once, such as a workflow with one branch waiting at an approval step while another branch's worker was lost, shows the first of them in the order above as its main blocker, the rest under badges, and one line in interaction on how they bear on each other. since and waiting_seconds measure the time in the current blocker rather than since the run was created: a run approved after a long hold has waited for a worker only since the approval released it, and one whose last worker went away has waited since that worker last reported. A retried workflow step, a dependency, and a run holding a slot are given as the reason on an item, not counted on their own.

Each item's main condition carries reason, who_can_act, and next. For a missing worker it carries queue and eligible_workers, for a lost one worker, last_seen, and reclaim_at, the automatic reclaim the view counts down to, for a blocked item holder_run_id, and for an approval approval: whether it is a whole run or a workflow_step, for a step whether the workflow's other_branches_running or it is paused, the decision_id approve and reject take, who may approve, and on_approve and on_deny saying what each answer runs next.

alert_after_seconds is the threshold that applies to the item and alerting reports it has passed it, which is also when the doctor lists it and when the attached targets are told, once per condition, as a run.needs_attention event. The defaults alert on a lost worker only when the lease sweep has not reclaimed its run within two lease periods, a minute, on a missing worker after 15 minutes, on a blocked item after 15 minutes, and never on an approval, since an approval waiting is the gate working. --attention-file changes them for the organization, an organization on the install, a queue, or a template. See the configuration reference. thresholds in the answer gives the install-wide values in seconds, zero meaning off.

The view lists only what the caller may read, under the rules the run list and the schedule list apply. There is no way to put a run back in the queue by hand from it. A worker that stopped reporting may only be cut off from the store rather than gone, and a run requeued from under it could execute twice, so the lease sweep's reclaim, which waits for the lease to expire first, is the only way a lost worker's run moves.

curl -s "https://switchtender.example.com/v1/attention?blocker=no_worker" \
  -H "Authorization: Bearer $ST_TOKEN"

Schedule timezone

A schedule reads its cron expression in its timezone, an IANA name such as America/New_York or Europe/Berlin. With one set, 0 2 * * * fires at 02:00 in that zone and follows its daylight-saving shifts, so a nightly window stays put across the year. The field is accepted on create and update and applies to the same expression the preview endpoint renders. A create or an import that names no zone takes the server's local zone and writes its name onto the schedule, so every server of a highly available group reads it alike, and the preview answers that zone as timezone. A schedule an earlier release stored without a zone is read in UTC, and the first server of this release to open the database writes UTC onto it.

spring_forward says what a schedule does with a time the clocks skip when they go forward: jump runs it when the clock jumps (the default for cron), later runs it after the clock change, the length of the jump later (the default for rrule, as AWX does), and skip skips that day. The preview takes the same parameter and answers spring_gap for the next night it matters.

curl -X POST https://switchtender.example.com/v1/schedules \
  -H "Authorization: Bearer $ST_TOKEN" \
  -H 'Content-Type: application/json' \
  -d '{"cron":"0 2 * * *","timezone":"America/New_York","template_id":"tpl_abc123"}'

Schedule recurrence rules

A schedule may carry rrule, an RFC 5545 recurrence, in place of cron: a DTSTART and one or more RRULE lines, with any EXRULE, RDATE, and EXDATE lines, separated by line breaks or spaces. A TZID on the DTSTART sets the zone, and a create or update that sends no timezone takes it from there. A schedule that sends both cron and rrule, a rule whose zone disagrees with timezone, or a rule with no fire left after now is refused with 400 and a message naming the fault.

curl -X POST https://switchtender.example.com/v1/schedules \
  -H "Authorization: Bearer $ST_TOKEN" \
  -H 'Content-Type: application/json' \
  -d '{"rrule":"DTSTART;TZID=America/New_York:20260102T170000\nRRULE:FREQ=MONTHLY;BYMONTH=3,6,9,12;BYDAY=-1FR","template_id":"tpl_abc123"}'

GET /v1/schedules/preview takes rrule in place of cron and answers next with up to five fires. A rule bounded by COUNT or UNTIL with fewer left answers what it has and "finished": true, and a rule whose DTSTART names a zone answers that zone as timezone. The schedule tutorial lists the parts a rule takes and how daylight-saving changes are handled.

Fleet view windows

/v1/fleet and /v1/tasks take a window, the number of recent runs per host or per task the view considers, and /v1/hosts/{host}/runs takes a limit. All three default to 10. The window is capped at 100 and the host history limit at 500. A larger value is answered with the cap, and the response echoes the window it actually used. The caps exist because the per-host and per-task summaries are kept when their runs are deleted, so on a long-lived fleet the tables hold a row for every host of every run, and every row a window admits becomes an element of the answer.

The same tables are bounded by count rather than by age. --retain-history keeps the newest N summaries for each host and each task and drops the rest, so a host's outcome history still outlives its runs without the tables growing forever. N is never allowed below 500, the deepest window these endpoints will answer, so trimmed history is history no request could have reached.

curl -s "https://switchtender.example.com/v1/fleet?window=30" \
  -H "Authorization: Bearer $ST_TOKEN"

Relay endpoints

With --worker-token set, the server also serves the mesh relay under /relay: the execution path an outbound worker started with --server uses instead of a database connection. Every call presents the worker bearer token.

Each report call is bounded twice. It presents the per-claim capability the claim response issued, so it can only write to the run this worker holds, and one call carries at most a few thousand items, so a worker cannot force an unbounded decode on the control node. The worker sends a wide run's evidence in several calls rather than losing it to that cap. Host facts are bounded further: a worker may write facts only for hosts its run has already reported results for, so nothing can be recorded about a machine no run claims to have touched.

What that does and does not give you: a worker authors its own results, so a worker you do not trust can still describe its own run untruthfully. What it cannot do is reach past that run into the recorded state of the rest of the fleet. Give a queue only to workers you would let touch the hosts that queue targets.