Skip to main content

How a request runs

Each change and each read of a host is a job. A client does not send a request to an agent. It sends the request to the controller. The controller puts a job on the queue for the agents that the request targets, and waits for their results.

Before a job is queued​

The controller refuses a request before it makes a job in these cases:

  • The token is missing or not valid (401).
  • The token does not give the permission of the operation (403). Each operation needs one permission, for example sysctl:write. Security tells how a token gets permissions.
  • The target is not a known agent, a label, _any or _all (400).
  • The input breaks a rule, for example a missing field or a key that is not the name of a kernel parameter (400).

A refused request makes no job. If it had a valid token, it makes an audit entry.

Targeting​

--target on the CLI, or {hostname} in the route, selects the agents that receive the job:

TargetSends the job to
web-01That agent, by hostname or machine ID
_anyOne agent that is free. The agents share a queue group
_allAll agents
group:web.devEach agent whose group label is web.dev or starts with it

The controller changes a hostname to the machine ID of the agent before it publishes. Thus an agent continues to receive its work when its hostname changes.

A label has parts with dots between them. An agent subscribes to each prefix of each label. For example, an agent with group: web.dev.us-east receives work for group:web, group:web.dev and group:web.dev.us-east. An agent can have up to five labels. Each prefix of each label is a JetStream consumer. Thus a large fleet with many labels needs an external NATS cluster.

The job​

The controller writes the job to a KV bucket, and the job stays there. The stream carries only a notification that points to the job. Reads and changes go on different subjects, jobs.query.* and jobs.modify.*. Thus an agent can process them differently. When PKI is on, the controller signs the job. The agent checks the signature before it runs the job.

The agent records each step as a status event on the job. Then it runs the operation and writes its result to the response bucket. osapi does not store the status of a job. It calculates the status from the events and the responses when you read the job:

StatusMeaning
submittedThe controller put the job on the queue
acknowledgedAn agent received the job
startedThe agent runs the job
completedThe job ran and succeeded
failedThe job ran and did not succeed
skippedThe host cannot do the job, for example a Debian operation on macOS
timeoutNo answer in controller.api.job_timeout

timeout is not failed. It does not tell if the operation ran. The agent can be gone, or it can still do the work and answer soon. A job sent to many agents also reports processing while some agents still run it, and partial_failure when some agents failed.

If an agent receives a job and does not acknowledge it, the agent receives the job again, up to agent.consumer.max_deliver times. After that, the delivery advisory of the job goes to the dead-letter stream, JOBS-DLQ. You can examine it there.

The result​

The controller waits for the result. It returns the result, with the ID of the job, on the same request:

{
"job_id": "550e8400-e29b-41d4-a716-446655440000",
"results": [
{
"hostname": "web-01",
"status": "ok",
"key": "vm.swappiness",
"changed": true
}
]
}

Each result is a list with one row for each host, for one host or for one hundred. Each row has the name of the host and its status: ok, failed, skipped or timeout. The last three also have error. A change reports changed. It is false when the host already had the requested state. Thus the same request, when you run it a second time, changes nothing.

A broadcast waits for each agent that it expects. If an agent does not answer in the timeout, the result gets a timeout row for it. Thus the response always tells what happened on each host that the request targeted.

Jobs after the request​

The job and its results stay in NATS after the request returns, for nats.kv.ttl:

osapi client job list --status failed
osapi client job get --id <job_id>
osapi client job retry --id <job_id>
osapi client job delete --id <job_id>

No command makes a job directly. The endpoint and the command of each operation make its job, and they check the input for that operation.

Design details​

  • The job system: the subjects, the KV-first design, the statuses, the retries and broadcast.
  • Permissions: how osapi selects the permission of an operation.