Skip to main content

Building a behavior

A behavior is one thing that osapi manages on a host, for example kernel parameters. You write it one time, as a Go definition. osapi builds these parts from the definition: the API routes, the dispatch of the agent, the CLI commands, the OpenAPI spec, the docs pages and the telemetry. It also builds the tests of each layer above the provider. A new behavior is one directory under internal/behaviors/ and one import line. osapi behavior create writes both.

osapi serves a behavior under /api and osapi client node, next to the hand-written domains that did not move yet. A behavior that replaces a hand-written domain deletes that domain in the same change: the two never run side by side. internal/behaviors/sysctl is the reference. osapi behavior spec get lists each option, and go doc ./internal/behavior shows the Go API, with examples that run as tests.

The layout​

Every behavior has the same files, so a reader, a reviewer or an agent finds the same thing in the same place. osapi behavior create --name <name> --description "..." writes them, as a working behavior that keeps a value in a file. The name is the route, the command and the docs page. The Go package and its directory are the name too, unless --package names a different one. A name that is a Go keyword needs --package: --name package --package packages serves /package from internal/behaviors/packages/. The scaffold refuses the keyword, and does not change the name for you.

internal/behaviors/<name>/
behavior.go behavior_public_test.go the definition; validates it and runs the examples
methods.go methods_public_test.go one function per method, on a mocked provider
provider.go the Provider interface each platform implements
debian.go debian_public_test.go a platform's provider, one file per platform
settings.go optional: the behavior's own settings, see below
types.go input and output types, with their tags
mocks/ the provider's generated double
gen/ generated by go generate; never edit
group.go the behavior as an endpoint group, with a handler for each method
api.go api_public_test.go the route table, from the group
decode.go each method's decoder, from the group
api_<method>_public_test.go a method's HTTP tests
cli.go cli_public_test.go the commands; each example run through them
api.yaml this behavior's part of the OpenAPI spec

Each file with code has a test with its name. The provider of a platform is <platform>.go. The definition declares it with behavior.Platform. On a platform without a provider, the methods report skipped.

osapi behavior create also registers the behavior with one blank import in internal/behaviors/all/all.go. Run go generate ./internal/behaviors/<name>/mocks ./internal/behaviors/all after it, for the mock and the gen/ directory. Nothing else outside the directory changes: the agent builds the provider from what the definition declares.

behavior.go: the definition​

var Behavior = behavior.Register(behavior.Define(
"sysctl", "Kernel parameters, written to sysctl.d files and applied",
behavior.Node,
behavior.API,
behavior.CLI,
behavior.Read("get", Get).Key("key").
Describe("Get a kernel parameter's current value").
Example("One parameter", GetInput{Key: "vm.swappiness"},
Entry{Key: "vm.swappiness", Value: "10"}),
behavior.Write("create", Create).
Describe("Manage a kernel parameter and apply it").
Explain("Writes osapi-<key>.conf to behaviors.sysctl.dir and applies it ...").
Example("Turn on IP forwarding",
CreateInput{Key: "net.ipv4.ip_forward", Value: "1"},
MutationResult{Key: "net.ipv4.ip_forward", Changed: true}),
))
OptionWhat it does
behavior.NodeThe work runs on a host, through the job system. Required.
behavior.Under("g")Nests it under a group: /api/node/{hostname}/g/<name>, osapi client node g <name>.
behavior.APIServes it over REST and puts it in the OpenAPI spec and the API reference.
behavior.CLIGenerates its commands. Their pages in the CLI reference come from the commands. Needs API, because the commands call it.
Read(name, fn)A method that reads the host: GET, the <name>:read permission.
Write(name, fn)A method that changes it: POST, or PUT for update and DELETE for delete.
.Key("field")Puts that input field in the path: /sysctl/{key}.
.Describe("...")One line: the command's short help, the operation's summary, the first line of the CLI page.
.Explain("...")Optional longer text: what it touches on the host and what running it twice does. It is the command's long help.
.Example(doc, in, out)A sample call. Its command line goes in the examples of the command, under a comment with doc. The tests send it and check out.
behavior.Config(v)The behavior's own settings, with v as their defaults. See below.
behavior.Platform(p, f)How the provider is built on platform p: debian, darwin, or linux for every Linux host without a provider of its own. On any other, the methods report skipped.
behavior.Sudo("prog")A program the provider runs with sudo. The agent's preflight checks it.
behavior.Runs("prog")A program the provider runs without sudo. The agent warns at startup when it is missing.
behavior.Writes("...")The paths the provider writes, removes or changes the mode of, as patterns. "*" is any path. The examples fail on a path that is not declared.
behavior.KeepsStateThe provider keeps a record in the file-state bucket. The examples fail on a record without it.
behavior.Validator(...)A validate tag of the behavior's own: the check, its schema and its message.
behavior.Requires(b...)Behaviors it is built on. See below.
behavior.InternalOnly other behaviors use it: no route, no command, no permissions.

A behavior with neither API nor CLI is reachable only through the job system or another behavior.

The name of a method decides more than its label. The HTTP method, the permission and the command come from it. Thus use list, get, create, update and delete when they fit.

settings.go: the behavior's own config​

Most behaviors need none. One that does declares a struct, with the same tags as osapi's config, and passes its defaults to behavior.Config:

type Settings struct {
Dir string `mapstructure:"dir" doc:"Directory the agent writes osapi's kernel parameter files to." validate:"required,startswith=/"`
}

behavior.Define("sysctl", ..., behavior.Config(Settings{Dir: "/etc/sysctl.d"}), ...)

The settings are in the osapi config under behaviors.<name>. osapi config list shows them. OSAPI_BEHAVIORS_<NAME>_<FIELD> and the osapi agent start flag --<name>-<field> set them. config validate checks their validate tags. A provider builder reads them with host.Settings[Settings](h). Settings are the fixed configuration of the agent. A value that changes with each call goes in the input of the method.

Built on other behaviors​

A behavior that needs another's work declares it, and calls the other's methods as plain Go calls on the same host, inside the same job:

behavior.Requires(managedfile.Behavior),
behavior.Platform("debian", newDebian),

func newDebian(h host.Host) (Provider, error) {
files := host.Require[managedfile.Provider](h, managedfile.Behavior)
if files == nil {
return nil, provider.ErrUnsupported // managedfile cannot run here
}
...
}

// in the provider
behavior.Call(ctx, managedfile.Behavior, managedfile.Update, d.files,
managedfile.UpdateInput{Path: path, Content: c})

behavior.Call runs the method inside its own span, so a trace shows behavior.managedfile.update under behavior.sysctl.create. A direct call of managedfile.Update works, but the trace has no step for it, so the tests of the behavior fail it.

The agent and the fake host both build the provider of a required behavior before the behavior that requires it. A behavior that only other behaviors use is behavior.Internal. It has examples and tests like the others, and a docs page that tells what uses it. It has no route, command or permissions. behavior validate fails when a requirement is not registered, when requirements make a cycle, and when an internal behavior has nothing that requires it. sysctl requires managedfile, and it is the example.

A behavior is internal when an operator would never call it, and it is a piece of work that other behaviors share, such as writing a file. osapi behavior create writes a behavior that an operator calls. Make shared work internal when a second behavior needs it.

A behavior declares the paths it writes, also when a required behavior does the write. managedfile declares behavior.Writes("*"), because it writes the path its caller names, and sysctl declares /etc/sysctl.d/osapi-*.conf. Thus osapi behavior get --name sysctl shows the files sysctl changes, and the spans under its own:

Telemetry:
Span: behavior.sysctl.<method>
Child spans: behavior.managedfile.<method>, exec sysctl, exec chown, exec stat, file.write, file.remove, file.chmod, kv.get, kv.put
Writes: /etc/sysctl.d/osapi-*.conf

The behaviors that a behavior calls, and the ones that they call, are its chain. osapi behavior chain get --name <name> shows each call that its examples make, and osapi behavior chain validate checks the chain rules. The Chains reference has how to build one, and each building block of the build.

types.go: the contract​

The tags of the input and output types hold all of the contract. Each field of an input becomes a request field, a CLI flag and a spec property. Each field of an output becomes a JSON field and a table column.

TagOnBecomes
json:"key"bothThe name everywhere: request, flag (--key), column (KEY), --json field
validate:"..."inputChecked by the API before a job is queued; the spec's constraints
doc:"..."bothThe field's description in the spec, the flag's help and the docs pages
render:"..."outputHow a column shows: - hides it, bytes, age or status
telemetry:"attr"inputRecords the field on the method's trace span. The field that .Key names is recorded without the tag.
template:"vars"inputA Go template, whose .Vars are in the input field vars (a map[string]string). The API refuses one that does not parse, or that reads a field or a var it does not have. osapi behavior spec get lists what a template reads.
secret:"true"inputA secret: redacted from the audit log, write-only in the spec, never traced

A write's output has a changed bool. The table shows it as the row's status, changed or ok, rather than as a column. An output that is a slice becomes one row per element.

The definition declares a validate rule of its own with behavior.Validator(tag, check, schema, message). The server runs check before it puts a job on the queue. The spec has schema. A value that fails shows "<value> <message>". A tag without a schema fails the spec generation. Thus the spec never checks less than the server.

With CLI, each input field is a flag. A flag holds a string, a bool, an int, a []string, a map[string]string or a []byte, a file. The CLI owns that list and its generator builds each one, so an int64 or a float64 fails osapi behavior validate.

The generated layers need each field in a form they can build. osapi behavior validate fails a definition that breaks one of these rules:

RuleRequires
nameThe behavior and its group are lowercase letters and digits, with no dash. A Go keyword, such as package, takes osapi behavior create --package
inputEach input field has a json name and a doc, no default (the server does not apply it), no array, and is not named hostname or target
optionalAn optional field of a write's body, validate:"omitempty,...", is json:"name,omitempty" too, so the client does not send "". The endpoint layer owns it
keyA key is validate:"required": it is a path segment
outputEach output field has a json name and no array, and each render tag is one the output knows. The CLI owns the render values
schemaA struct that an output holds has a name that no other behavior's type has, because one spec names every behavior's types

Each rule that another layer owns is asked of that layer. Flags and render values are the CLI's, and the optional rule is the endpoint layer's. Thus behavior validate and the other layer cannot disagree.

methods.go and provider.go​

The services a behavior uses, in the build-a-behavior skill, lists what the agent hands a provider (filesystem, commands, files osapi owns, secrets, settings), how the tests fake each, and which existing provider to copy for each kind of state.

Each method takes the provider, the method's input type and a context, and returns its output type:

func Get(ctx context.Context, p Provider, in GetInput) (Entry, error)

Methods change the input for the provider, and the provider does the work. Provider is the interface of each platform. The definition tells how to build the provider of each platform:

behavior.Platform("debian", newDebian),

func newDebian(h host.Host) (Provider, error) {
return &debian{fs: h.FS, path: host.Settings[Settings](h).Path}, nil
}

The agent gives host.Host to the builder when the agent starts. It has the platform, the filesystem, the command runner, the file state store, the hostname and the settings of the behavior. The agent writes to the log the behaviors that it built a provider for. On a platform that the behavior does not declare, or when the builder returns provider.ErrUnsupported, its methods report skipped.

What every behavior gets​

Written once in the platform, so a behavior writes none of it:

  • Routes: the behavior as an endpoint group, in gen/group.go, which the endpoint server serves as it serves a group of the controller. The route checks the permission, decodes the request and checks its tags, the target included. The generated handler of the method gives the input to internal/behavior/dispatch, which sends the job, does the broadcast to _all, _any or a label, and makes the {job_id, results[]} envelope.
  • Commands with typed flags, the theme's table and --json.
  • Telemetry: a behavior.<name>.<method> span under the job span of the agent, and osapi_behavior_calls_total, osapi_behavior_duration_seconds and osapi_behavior_changes_total, through the OpenTelemetry provider of the agent.
  • Permissions <name>:read and <name>:write. The built-in roles give them: admin and write give both, and read gives the read permission. osapi token roles get --name <role> shows them.
  • Audit entries, through the same middleware as every route.

Testing​

A behavior's examples are its tests. Each one can say what the host looks like first and what the call changes there:

Example("Turn on IP forwarding",
CreateInput{Key: "net.ipv4.ip_forward", Value: "1"},
MutationResult{Key: "net.ipv4.ip_forward", Changed: true},
behavior.Wrote(conf("net.ipv4.ip_forward"), "net.ipv4.ip_forward = 1\n"),
behavior.Ran("sysctl", "-p", conf("net.ipv4.ip_forward")),
),
OptionSays
behavior.After(input)Run the method that takes input first
behavior.Prints(cmd, out)The command line cmd prints out
behavior.PrintsAfter(ran, cmd, out)Once ran has run, cmd prints out: state a program keeps, such as a package query after the install
behavior.Exists(path, text)The file is there before the call
behavior.OnPlatform(platforms...)The example runs only on these platforms, for an answer that is the platform's own, such as the OS name
behavior.Wrote(path, text)The call leaves the file holding exactly text
behavior.Removed(path)The call leaves the file gone
behavior.Ran(prog, args...)The call runs that command, with or without sudo

From those, each behavior gets these tests, and you do not write them:

LayerWhereChecks
Providerbehaviortest.Run, in behavior_public_test.goEach example on a fake host, per platform: its output and effects; a write declares effects and reports changed: false run twice; a failing command or store call never panics
APIgen/api_<method>_public_test.go, generatedEach example through the real handler and middleware with the job client mocked, every response checked against the generated spec: the example on one host and broadcast; a host that skips, fails or times out; the job client failing; a 404 for a keyed method; a 400 for a bad target, each missing required field, a body that is not JSON, a field of the wrong type and each refusal the API can see; a 401 and a 403
Routesgen/api_public_test.go, generatedEvery method's HTTP method, path and permission
CLIgen/cli_public_test.go, generatedEach example's flags, given to the generated command, send the example's input
Wiringinternal/behaviors/all/all_public_test.goEvery registered behavior validates, every route is mounted and every command attached
Live servertest/integration/behavior_test.goEvery unkeyed read against a real agent; the write examples too with OSAPI_INTEGRATION_WRITE_BEHAVIORS=1; CI's Integration step runs them as root, with OSAPI_INTEGRATION_WRITE_BEHAVIORS=1, on its throwaway Linux runner, each from its item deleted; a behavior with no delete gets its item back afterwards, read with get and written with update, so a hostname example leaves the runner its own name. A write example sets its starting state with After, which a live host can run

The generated HTTP tests are tables of rows. Each row has the request, the token, the call that the job client expects and its answer, and the expected status and body. internal/behavior/dispatch/servertest runs them. Fix a failing generated test in the definition or in the generator, never in gen/.

The fake host is internal/behavior/behaviortest. It has an in-memory filesystem, a command runner and an in-memory file state store. The command runner records what ran and prints what Prints says. The provider of each platform also has its own test, <platform>_public_test.go, for what an example cannot show, for example the edge cases of a parser or a filesystem error.

The harness also probes each string input field of each example, one at a time, with three hostile values:

  • a path that goes out of its directory (../../../../osapi-probe)
  • a value that starts a new line
  • a value that looks like an option (--osapi-probe)

A probe passes when the validate tags of the field refuse it, as the API does, or when the method fails, or when it changes nothing that it must not change. A probe fails when it writes a file outside the directories that the example writes, puts its line in a file with other content, gets to a command as an option without a -- before it, or gets to a sh -c script. The harness does not probe an internal behavior, because its input comes from other behaviors.

A method also declares what it refuses, with .Fails(doc, input, errorText) and the same host options: an input that it rejects, or a host that it cannot change.

Fails("A parameter osapi stopped managing",
UpdateInput{Key: "vm.swappiness", Value: "10"}, "not managed",
behavior.After(CreateInput{Key: "vm.swappiness", Value: "60"}),
behavior.After(DeleteInput{Key: "vm.swappiness"}),
),

The provider test checks that each refusal fails with that text. The API test checks that a refusal whose input breaks a validate tag gets a 400, and that no job goes on the queue. The CLI page of the method lists the refusals under "Result".

On the fake host, each filesystem, command and store call of the example fails in turn. Each store read returns a damaged record in turn. The method must return an error and must not panic. With its examples and refusals, the test of sysctl reaches all of its Debian provider. The only exception is a JSON encoding error that a struct of strings cannot cause.

The checks​

go generate ./internal/behaviors/all # write gen/ and the spec; just cli-docs-generate writes the pages
just cli-docs-generate # write the CLI pages of the commands
osapi behavior spec get # each part of the DSL, and when to use it
osapi behavior validate # the definition's rules, each with its fix
osapi behavior get --name sysctl # surfaces, routes, schemas, as registered
just behavior-gen-check # fails when generated files are stale
just behavior-cover-check # fails when a behavior is below 100% coverage
just behavior-scaffold-check # fails when a new behavior fails its tests

osapi behavior validate checks each definition against its rules. It exits 1 when one fails, and names the behavior, the method, the rule and the fix. The tests of each behavior run its examples on a fake host, and each failure ends with its fix too. osapi behavior validate --help lists each rule and each test check, from the same tables that the checks use. Thus the list cannot be out of date. osapi behavior spec get lists each part of the DSL, the generated paths and what the platform cannot do yet.

just test runs behavior-gen-check. Thus a behavior that changed without go generate fails CI. It also runs behavior-cover-check. Thus a behavior whose examples and tests miss a line fails CI too. It also runs behavior-scaffold-check, which makes a behavior with osapi behavior create in a copy of the checkout and runs its tests. Thus the behavior that you start from passes each check.

A behavior that replaces a hand-written domain changes the joined spec. The API contract test, TestJoinedSpec in internal/apicontract, then fails, and its golden diff is the change to the API: write it again with -update and read it.

Generated, never edited​

FileFrom
internal/behaviors/<name>/gen/*.gothe definition, its types and its examples, built with jen
internal/behaviors/<name>/gen/api.yamlgen/group.go, by the endpoint layer's spec builder in internal/behavior/gen/group
internal/controller/api/gen/api.yamlthe hand-written specs and every gen/api.yaml, by just generate
internal/behaviors/<name>/mocks/provider.gen.gothe Provider interface, by mockgen
internal/behaviors/all/gen/gen.goevery behavior with API or CLI
docs/docs/sidebar/reference/behaviors/<name>/<name>.mdosapi behavior get and method get, as markdown, by just cli-docs-generate
docs/docs/sidebar/usage/cli/client/...the generated commands, by just cli-docs-generate

gen/cli.go puts the docs of a method on its command. .Describe is the Short of the command, .Explain is its Long, and the examples are its Example. The reference.Permission annotation has the permission of the method. The reference.Result annotation has a table of what each example does on the host, and a table of the refusals. The command of the behavior has a reference.Notes annotation, with a link to the page of the behavior. Thus just cli-docs-generate writes the page of a behavior command as it writes the page of a hand-written command. CLI reference has the pages.

internal/behavior/gen writes them. It builds Go with jen (github.com/dave/jennifer), not with text templates. Thus the output always parses, and jen manages its imports. goimports puts the imports in the repository's three groups. The output gets no other format: each file starts with a Code generated line, so golangci-lint skips it in the format check and in the lint. It builds the spec with the openapi3 types of kin-openapi, the library that oapi-codegen uses to read specs. A test loads and validates the result. To change a generated file, change the behavior or the generator, and run go generate.