Building a behavior
A behavior is one thing that osapi manages on a host, for example kernel
parameters. You write it one time, as a Go definition. osapi builds these parts
from the definition: the API routes, the dispatch of the agent, the CLI
commands, the OpenAPI spec, the docs pages and the telemetry. It also builds the
tests of each layer above the provider. A new behavior is one directory under
internal/behaviors/ and one import line. osapi behavior create writes both.
osapi serves a behavior under /api and osapi client node, next to the
hand-written domains that did not move yet. A behavior that replaces a
hand-written domain deletes that domain in the same change: the two never run
side by side. internal/behaviors/sysctl is the reference.
osapi behavior spec get lists each option, and go doc ./internal/behavior
shows the Go API, with examples that run as tests.
The layout
Every behavior has the same files, so a reader, a reviewer or an agent finds the
same thing in the same place.
osapi behavior create --name <name> --description "..." writes them, as a
working behavior that keeps a value in a file. The name is the route, the
command and the docs page. The Go package and its directory are the name too,
unless --package names a different one. A name that is a Go keyword needs
--package: --name package --package packages serves /package from
internal/behaviors/packages/. The scaffold refuses the keyword, and does not
change the name for you.
internal/behaviors/<name>/
behavior.go behavior_public_test.go the definition; validates it and runs the examples
methods.go methods_public_test.go one function per method, on a mocked provider
provider.go the Provider interface each platform implements
debian.go debian_public_test.go a platform's provider, one file per platform
settings.go optional: the behavior's own settings, see below
types.go input and output types, with their tags
mocks/ the provider's generated double
gen/ generated by go generate; never edit
group.go the behavior as an endpoint group, with a handler for each method
api.go api_public_test.go the route table, from the group
decode.go each method's decoder, from the group
api_<method>_public_test.go a method's HTTP tests
cli.go cli_public_test.go the commands; each example run through them
api.yaml this behavior's part of the OpenAPI spec
Each file with code has a test with its name. The provider of a platform is
<platform>.go. The definition declares it with behavior.Platform. On a
platform without a provider, the methods report skipped.
osapi behavior create also registers the behavior with one blank import in
internal/behaviors/all/all.go. Run
go generate ./internal/behaviors/<name>/mocks ./internal/behaviors/all after
it, for the mock and the gen/ directory. Nothing else outside the directory
changes: the agent builds the provider from what the definition declares.
behavior.go: the definition
var Behavior = behavior.Register(behavior.Define(
"sysctl", "Kernel parameters, written to sysctl.d files and applied",
behavior.Node,
behavior.API,
behavior.CLI,
behavior.Read("get", Get).Key("key").
Describe("Get a kernel parameter's current value").
Example("One parameter", GetInput{Key: "vm.swappiness"},
Entry{Key: "vm.swappiness", Value: "10"}),
behavior.Write("create", Create).
Describe("Manage a kernel parameter and apply it").
Explain("Writes osapi-<key>.conf to behaviors.sysctl.dir and applies it ...").
Example("Turn on IP forwarding",
CreateInput{Key: "net.ipv4.ip_forward", Value: "1"},
MutationResult{Key: "net.ipv4.ip_forward", Changed: true}),
))
| Option | What it does |
|---|---|
behavior.Node | The work runs on a host, through the job system. Required. |
behavior.Under("g") | Nests it under a group: /api/node/{hostname}/g/<name>, osapi client node g <name>. |
behavior.API | Serves it over REST and puts it in the OpenAPI spec and the API reference. |
behavior.CLI | Generates its commands. Their pages in the CLI reference come from the commands. Needs API, because the commands call it. |
Read(name, fn) | A method that reads the host: GET, the <name>:read permission. |
Write(name, fn) | A method that changes it: POST, or PUT for update and DELETE for delete. |
.Key("field") | Puts that input field in the path: /sysctl/{key}. |
.Describe("...") | One line: the command's short help, the operation's summary, the first line of the CLI page. |
.Explain("...") | Optional longer text: what it touches on the host and what running it twice does. It is the command's long help. |
.Example(doc, in, out) | A sample call. Its command line goes in the examples of the command, under a comment with doc. The tests send it and check out. |
behavior.Config(v) | The behavior's own settings, with v as their defaults. See below. |
behavior.Platform(p, f) | How the provider is built on platform p: debian, darwin, or linux for every Linux host without a provider of its own. On any other, the methods report skipped. |
behavior.Sudo("prog") | A program the provider runs with sudo. The agent's preflight checks it. |
behavior.Runs("prog") | A program the provider runs without sudo. The agent warns at startup when it is missing. |
behavior.Writes("...") | The paths the provider writes, removes or changes the mode of, as patterns. "*" is any path. The examples fail on a path that is not declared. |
behavior.KeepsState | The provider keeps a record in the file-state bucket. The examples fail on a record without it. |
behavior.Validator(...) | A validate tag of the behavior's own: the check, its schema and its message. |
behavior.Requires(b...) | Behaviors it is built on. See below. |
behavior.Internal | Only other behaviors use it: no route, no command, no permissions. |
A behavior with neither API nor CLI is reachable only through the job system
or another behavior.
The name of a method decides more than its label. The HTTP method, the
permission and the command come from it. Thus use list, get, create,
update and delete when they fit.
settings.go: the behavior's own config
Most behaviors need none. One that does declares a struct, with the same tags as
osapi's config, and passes its defaults to behavior.Config:
type Settings struct {
Dir string `mapstructure:"dir" doc:"Directory the agent writes osapi's kernel parameter files to." validate:"required,startswith=/"`
}
behavior.Define("sysctl", ..., behavior.Config(Settings{Dir: "/etc/sysctl.d"}), ...)
The settings are in the osapi config under behaviors.<name>.
osapi config list shows them. OSAPI_BEHAVIORS_<NAME>_<FIELD> and the
osapi agent start flag --<name>-<field> set them. config validate checks
their validate tags. A provider builder reads them with
host.Settings[Settings](h). Settings are the fixed configuration of the agent.
A value that changes with each call goes in the input of the method.
Built on other behaviors
A behavior that needs another's work declares it, and calls the other's methods as plain Go calls on the same host, inside the same job:
behavior.Requires(managedfile.Behavior),
behavior.Platform("debian", newDebian),
func newDebian(h host.Host) (Provider, error) {
files := host.Require[managedfile.Provider](h, managedfile.Behavior)
if files == nil {
return nil, provider.ErrUnsupported // managedfile cannot run here
}
...
}
// in the provider
behavior.Call(ctx, managedfile.Behavior, managedfile.Update, d.files,
managedfile.UpdateInput{Path: path, Content: c})
behavior.Call runs the method inside its own span, so a trace shows
behavior.managedfile.update under behavior.sysctl.create. A direct call of
managedfile.Update works, but the trace has no step for it, so the tests of
the behavior fail it.
The agent and the fake host both build the provider of a required behavior
before the behavior that requires it. A behavior that only other behaviors use
is behavior.Internal. It has examples and tests like the others, and a docs
page that tells what uses it. It has no route, command or permissions.
behavior validate fails when a requirement is not registered, when
requirements make a cycle, and when an internal behavior has nothing that
requires it. sysctl requires managedfile, and it is the example.
A behavior is internal when an operator would never call it, and it is a piece
of work that other behaviors share, such as writing a file.
osapi behavior create writes a behavior that an operator calls. Make shared
work internal when a second behavior needs it.
A behavior declares the paths it writes, also when a required behavior does the
write. managedfile declares behavior.Writes("*"), because it writes the path
its caller names, and sysctl declares /etc/sysctl.d/osapi-*.conf. Thus
osapi behavior get --name sysctl shows the files sysctl changes, and the spans
under its own:
Telemetry:
Span: behavior.sysctl.<method>
Child spans: behavior.managedfile.<method>, exec sysctl, exec chown, exec stat, file.write, file.remove, file.chmod, kv.get, kv.put
Writes: /etc/sysctl.d/osapi-*.conf
The behaviors that a behavior calls, and the ones that they call, are its chain.
osapi behavior chain get --name <name> shows each call that its examples make,
and osapi behavior chain validate checks the chain rules. The
Chains reference has how to build one, and each building block
of the build.
types.go: the contract
The tags of the input and output types hold all of the contract. Each field of an input becomes a request field, a CLI flag and a spec property. Each field of an output becomes a JSON field and a table column.
| Tag | On | Becomes |
|---|---|---|
json:"key" | both | The name everywhere: request, flag (--key), column (KEY), --json field |
validate:"..." | input | Checked by the API before a job is queued; the spec's constraints |
doc:"..." | both | The field's description in the spec, the flag's help and the docs pages |
render:"..." | output | How a column shows: - hides it, bytes, age or status |
telemetry:"attr" | input | Records the field on the method's trace span. The field that .Key names is recorded without the tag. |
template:"vars" | input | A Go template, whose .Vars are in the input field vars (a map[string]string). The API refuses one that does not parse, or that reads a field or a var it does not have. osapi behavior spec get lists what a template reads. |
secret:"true" | input | A secret: redacted from the audit log, write-only in the spec, never traced |
A write's output has a changed bool. The table shows it as the row's status,
changed or ok, rather than as a column. An output that is a slice becomes
one row per element.
The definition declares a validate rule of its own with
behavior.Validator(tag, check, schema, message). The server runs check
before it puts a job on the queue. The spec has schema. A value that fails
shows "<value> <message>". A tag without a schema fails the spec generation.
Thus the spec never checks less than the server.
With CLI, each input field is a flag. A flag holds a string, a bool, an
int, a []string, a map[string]string or a []byte, a file. The CLI owns
that list and its generator builds each one, so an int64 or a float64 fails
osapi behavior validate.
The generated layers need each field in a form they can build.
osapi behavior validate fails a definition that breaks one of these rules:
| Rule | Requires |
|---|---|
name | The behavior and its group are lowercase letters and digits, with no dash. A Go keyword, such as package, takes osapi behavior create --package |
input | Each input field has a json name and a doc, no default (the server does not apply it), no array, and is not named hostname or target |
optional | An optional field of a write's body, validate:"omitempty,...", is json:"name,omitempty" too, so the client does not send "". The endpoint layer owns it |
key | A key is validate:"required": it is a path segment |
output | Each output field has a json name and no array, and each render tag is one the output knows. The CLI owns the render values |
schema | A struct that an output holds has a name that no other behavior's type has, because one spec names every behavior's types |
Each rule that another layer owns is asked of that layer. Flags and render
values are the CLI's, and the optional rule is the endpoint layer's. Thus
behavior validate and the other layer cannot disagree.
methods.go and provider.go
The services a behavior uses, in the build-a-behavior skill, lists what the agent hands a provider (filesystem, commands, files osapi owns, secrets, settings), how the tests fake each, and which existing provider to copy for each kind of state.
Each method takes the provider, the method's input type and a context, and returns its output type:
func Get(ctx context.Context, p Provider, in GetInput) (Entry, error)
Methods change the input for the provider, and the provider does the work.
Provider is the interface of each platform. The definition tells how to build
the provider of each platform:
behavior.Platform("debian", newDebian),
func newDebian(h host.Host) (Provider, error) {
return &debian{fs: h.FS, path: host.Settings[Settings](h).Path}, nil
}
The agent gives host.Host to the builder when the agent starts. It has the
platform, the filesystem, the command runner, the file state store, the hostname
and the settings of the behavior. The agent writes to the log the behaviors that
it built a provider for. On a platform that the behavior does not declare, or
when the builder returns provider.ErrUnsupported, its methods report
skipped.
What every behavior gets
Written once in the platform, so a behavior writes none of it:
- Routes: the behavior as an endpoint group, in
gen/group.go, which the endpoint server serves as it serves a group of the controller. The route checks the permission, decodes the request and checks its tags, the target included. The generated handler of the method gives the input tointernal/behavior/dispatch, which sends the job, does the broadcast to_all,_anyor a label, and makes the{job_id, results[]}envelope. - Commands with typed flags, the theme's table and
--json. - Telemetry: a
behavior.<name>.<method>span under the job span of the agent, andosapi_behavior_calls_total,osapi_behavior_duration_secondsandosapi_behavior_changes_total, through the OpenTelemetry provider of the agent. - Permissions
<name>:readand<name>:write. The built-in roles give them:adminandwritegive both, andreadgives the read permission.osapi token roles get --name <role>shows them. - Audit entries, through the same middleware as every route.
Testing
A behavior's examples are its tests. Each one can say what the host looks like first and what the call changes there:
Example("Turn on IP forwarding",
CreateInput{Key: "net.ipv4.ip_forward", Value: "1"},
MutationResult{Key: "net.ipv4.ip_forward", Changed: true},
behavior.Wrote(conf("net.ipv4.ip_forward"), "net.ipv4.ip_forward = 1\n"),
behavior.Ran("sysctl", "-p", conf("net.ipv4.ip_forward")),
),
| Option | Says |
|---|---|
behavior.After(input) | Run the method that takes input first |
behavior.Prints(cmd, out) | The command line cmd prints out |
behavior.PrintsAfter(ran, cmd, out) | Once ran has run, cmd prints out: state a program keeps, such as a package query after the install |
behavior.Exists(path, text) | The file is there before the call |
behavior.OnPlatform(platforms...) | The example runs only on these platforms, for an answer that is the platform's own, such as the OS name |
behavior.Wrote(path, text) | The call leaves the file holding exactly text |
behavior.Removed(path) | The call leaves the file gone |
behavior.Ran(prog, args...) | The call runs that command, with or without sudo |
From those, each behavior gets these tests, and you do not write them:
| Layer | Where | Checks |
|---|---|---|
| Provider | behaviortest.Run, in behavior_public_test.go | Each example on a fake host, per platform: its output and effects; a write declares effects and reports changed: false run twice; a failing command or store call never panics |
| API | gen/api_<method>_public_test.go, generated | Each example through the real handler and middleware with the job client mocked, every response checked against the generated spec: the example on one host and broadcast; a host that skips, fails or times out; the job client failing; a 404 for a keyed method; a 400 for a bad target, each missing required field, a body that is not JSON, a field of the wrong type and each refusal the API can see; a 401 and a 403 |
| Routes | gen/api_public_test.go, generated | Every method's HTTP method, path and permission |
| CLI | gen/cli_public_test.go, generated | Each example's flags, given to the generated command, send the example's input |
| Wiring | internal/behaviors/all/all_public_test.go | Every registered behavior validates, every route is mounted and every command attached |
| Live server | test/integration/behavior_test.go | Every unkeyed read against a real agent; the write examples too with OSAPI_INTEGRATION_WRITE_BEHAVIORS=1; CI's Integration step runs them as root, with OSAPI_INTEGRATION_WRITE_BEHAVIORS=1, on its throwaway Linux runner, each from its item deleted; a behavior with no delete gets its item back afterwards, read with get and written with update, so a hostname example leaves the runner its own name. A write example sets its starting state with After, which a live host can run |
The generated HTTP tests are tables of rows. Each row has the request, the
token, the call that the job client expects and its answer, and the expected
status and body. internal/behavior/dispatch/servertest runs them. Fix a
failing generated test in the definition or in the generator, never in gen/.
The fake host is internal/behavior/behaviortest. It has an in-memory
filesystem, a command runner and an in-memory file state store. The command
runner records what ran and prints what Prints says. The provider of each
platform also has its own test, <platform>_public_test.go, for what an example
cannot show, for example the edge cases of a parser or a filesystem error.
The harness also probes each string input field of each example, one at a time, with three hostile values:
- a path that goes out of its directory (
../../../../osapi-probe) - a value that starts a new line
- a value that looks like an option (
--osapi-probe)
A probe passes when the validate tags of the field refuse it, as the API does,
or when the method fails, or when it changes nothing that it must not change. A
probe fails when it writes a file outside the directories that the example
writes, puts its line in a file with other content, gets to a command as an
option without a -- before it, or gets to a sh -c script. The harness does
not probe an internal behavior, because its input comes from other behaviors.
A method also declares what it refuses, with .Fails(doc, input, errorText) and
the same host options: an input that it rejects, or a host that it cannot
change.
Fails("A parameter osapi stopped managing",
UpdateInput{Key: "vm.swappiness", Value: "10"}, "not managed",
behavior.After(CreateInput{Key: "vm.swappiness", Value: "60"}),
behavior.After(DeleteInput{Key: "vm.swappiness"}),
),
The provider test checks that each refusal fails with that text. The API test
checks that a refusal whose input breaks a validate tag gets a 400, and that
no job goes on the queue. The CLI page of the method lists the refusals under
"Result".
On the fake host, each filesystem, command and store call of the example fails in turn. Each store read returns a damaged record in turn. The method must return an error and must not panic. With its examples and refusals, the test of sysctl reaches all of its Debian provider. The only exception is a JSON encoding error that a struct of strings cannot cause.
The checks
go generate ./internal/behaviors/all # write gen/ and the spec; just cli-docs-generate writes the pages
just cli-docs-generate # write the CLI pages of the commands
osapi behavior spec get # each part of the DSL, and when to use it
osapi behavior validate # the definition's rules, each with its fix
osapi behavior get --name sysctl # surfaces, routes, schemas, as registered
just behavior-gen-check # fails when generated files are stale
just behavior-cover-check # fails when a behavior is below 100% coverage
just behavior-scaffold-check # fails when a new behavior fails its tests
osapi behavior validate checks each definition against its rules. It exits 1
when one fails, and names the behavior, the method, the rule and the fix. The
tests of each behavior run its examples on a fake host, and each failure ends
with its fix too. osapi behavior validate --help lists each rule and each test
check, from the same tables that the checks use. Thus the list cannot be out of
date. osapi behavior spec get lists each part of the DSL, the generated paths
and what the platform cannot do yet.
just test runs behavior-gen-check. Thus a behavior that changed without
go generate fails CI. It also runs behavior-cover-check. Thus a behavior
whose examples and tests miss a line fails CI too. It also runs
behavior-scaffold-check, which makes a behavior with osapi behavior create
in a copy of the checkout and runs its tests. Thus the behavior that you start
from passes each check.
A behavior that replaces a hand-written domain changes the joined spec. The API
contract test, TestJoinedSpec in internal/apicontract, then fails, and its
golden diff is the change to the API: write it again with -update and read it.
Generated, never edited
| File | From |
|---|---|
internal/behaviors/<name>/gen/*.go | the definition, its types and its examples, built with jen |
internal/behaviors/<name>/gen/api.yaml | gen/group.go, by the endpoint layer's spec builder in internal/behavior/gen/group |
internal/controller/api/gen/api.yaml | the hand-written specs and every gen/api.yaml, by just generate |
internal/behaviors/<name>/mocks/provider.gen.go | the Provider interface, by mockgen |
internal/behaviors/all/gen/gen.go | every behavior with API or CLI |
docs/docs/sidebar/reference/behaviors/<name>/<name>.md | osapi behavior get and method get, as markdown, by just cli-docs-generate |
docs/docs/sidebar/usage/cli/client/... | the generated commands, by just cli-docs-generate |
gen/cli.go puts the docs of a method on its command. .Describe is the
Short of the command, .Explain is its Long, and the examples are its
Example. The reference.Permission annotation has the permission of the
method. The reference.Result annotation has a table of what each example does
on the host, and a table of the refusals. The command of the behavior has a
reference.Notes annotation, with a link to the page of the behavior. Thus
just cli-docs-generate writes the page of a behavior command as it writes the
page of a hand-written command. CLI reference has the
pages.
internal/behavior/gen writes them. It builds Go with jen
(github.com/dave/jennifer), not with text templates. Thus the output always
parses, and jen manages its imports. goimports puts the imports in the
repository's three groups. The output gets no other format: each file starts
with a Code generated line, so golangci-lint skips it in the format check and
in the lint. It builds the spec with the openapi3 types of kin-openapi, the
library that oapi-codegen uses to read specs. A test loads and validates the
result. To change a generated file, change the behavior or the generator, and
run go generate.