Skip to main content

Observability

Each of the three processes reports through the same four channels: traces, metrics, logs and a heartbeat in the registry. One trace ID follows a request from the command that sent it to the agent that ran it.

Traces​

osapi uses OpenTelemetry. A trace starts in the CLI and goes with the work:

  1. The CLI starts a span and sends traceparent with the request.
  2. The HTTP middleware of the controller continues the trace. The NATS client puts the trace context in the headers of the job notification.
  3. The agent continues the trace in a job.process span. A method of a behavior adds a behavior.<name>.<method> span below it. That span has the behavior, the method, the result, whether the host changed, and the input fields that the behavior marks for traces.

Tracing is off by default (telemetry.tracing). If you turn it on without an exporter, each log line has trace_id and span_id. Then one grep shows a request from start to end. The otlp exporter sends spans over gRPC to a backend, for example Jaeger or Tempo. The stdout exporter prints them. --debug turns on tracing without an exporter.

Metrics​

Each process can run a metrics server on its own port. The server has /metrics in the Prometheus format, and the /health and /health/ready probes. The metrics server of the controller is on 9090 by default. agent.metrics and nats.server.metrics set the servers of the agent and the NATS server. osapi records metrics through OpenTelemetry and exports them with the osapi_ prefix, together with the Go runtime and process metrics. Watch these metrics first:

MetricFromWatch for
osapi_component_upAll0: the process is not ready
osapi_subsystem_upAll0 on a subsystem, for example the notifier
osapi_jobs_created_totalControllerThe request rate
osapi_jobs_processed_totalAgentThe failed part, by status
osapi_job_duration_secondsAgentSlow operations
osapi_heartbeat_age_secondsAgentAn increase: the agent cannot reach NATS
osapi_behavior_calls_totalAgentCalls for each behavior, method and result
osapi_behavior_duration_secondsAgentThe latency of each method
osapi_behavior_changes_totalAgentHow often a write changed a host
osapi_nats_client_operation_duration_secondsAllSlow or failed calls to NATS, by bucket

Metrics lists each metric and its labels.

Logs​

Each process writes logs with the structured logger of Go, one line for each event. Each line names its subsystem, for example agent.heartbeat, controller.api or provider.sysctl. All lines use the same field names: hostname, job_id, path and error. With tracing on, each line also has the trace ID and the span ID.

Debug lines show each operation that the agent dispatches, and each change that it skips because the host already has the state. Warnings show a state that is not normal but still operates. An operation that fails, for example a sysctl write that the kernel did not apply, is an error and fails the job.

Health and the registry​

Each component writes a heartbeat to the registry every 10 seconds. The heartbeat gives the state of each part of the component:

  • the API, the metrics, the notifier and tracing of the controller
  • the heartbeat and the metrics of the agent
  • the NATS server

If a component does not write its entry again in nats.registry.ttl, the entry expires, and osapi shows the component as unreachable.

EndpointTellsToken needed
/api/healthThe controller process runsNo
/api/health/readyThe controller can reach NATS and its KV bucketsNo
/api/health/statusEach component and its parts, on all hostshealth:read

osapi client health status shows the last endpoint. osapi client agent list shows each agent with its conditions.

Conditions and notifications​

Each component examines its conditions at each heartbeat, for example memory, disk and load pressure on an agent. It compares them with the limits in agent.conditions, and writes them with the heartbeat.

The controller watches the registry. It makes a notification when a condition starts (fired) or stops (resolved), or when the heartbeat of a component expires (unreachable). It sends the notification of an active condition again at an interval, so that the notification stays visible. Today, notifications go to the log.

You can also drain an agent. A drained agent does not receive new work, but the process continues to run. The drain and its timeline stay in the agent state bucket.

Design details​