system-agent¶

system-agent is what every VM runs for the fleet's own sake: it ships the VM's metrics and logs to VictoriaMetrics on ci01, writes its DNS records, and offers its container logs to a central Dozzle. It carries no Traefik, so it goes on a VM whether or not that VM serves a web interface.

Every host lists it, and no host runs it while in bootstrap mode. See In bootstrap mode.

What it runs¶

vmagent
vlagent
vector
cadvisor
dozzle-agent
dockns
socket-proxy

Service	Does
`vmagent`	Scrapes metrics on the VM and sends them to vmauth, the way in to the metric and log stores on ci01
`vlagent`	Sends the logs vector hands it to vmauth on ci01
`vector`	Collects container logs, the host's journal, `.log` files under `/var/log`, syslog, and Traefik's access log
`cadvisor`	Measures each container's use of CPU, memory, disk, and network
`dozzle-agent`	Serves the VM's container logs on port 7007, for a Dozzle server to read
`dockns`	Writes a DNS record, in the network's own DNS or at Cloudflare, for each container that carries its labels
`socket-proxy`	Gives vector, dozzle-agent, and dockns a filtered view of the Docker API

The Stack in Komodo carries the host's name, such as system-agent-id01. The project is system, so the containers are named system-vmagent, system-vector, and so on. The Dozzle agent is system-dozzle-agent.

vmagent scrapes five targets once a minute: itself, vlagent, cadvisor, the host's Node Exporter, and the Traefik on the same VM. It looks the Traefik up by name, so a VM without one has no target and reports no failure.

vmagent and vlagent each keep up to 100 MB of unsent data on disk while ci01 is unreachable, and send it when ci01 is back.

The journal is where a host keeps its own logs, the kernel's and the logins' included, so it is vector's main source for the host.

vector listens for syslog on UDP port 5140. It reads Traefik's access log from /opt/docker/logs/traefik/traefik, the folder the VM's Traefik stack writes to, and finds nothing there on a VM without one.

In bootstrap mode¶

The stack's setup.yaml says it needs vmauth, which is on ci01. A host in bootstrap mode has that need unmet, so the run leaves the stack out. See What it changes.

Two things follow until the host leaves bootstrap mode. Its metrics and logs are not shipped, and nothing writes its DNS records. See What you see in bootstrap mode.

The stack first deploys when the fleet leaves bootstrap mode. See Leaving bootstrap mode.

Values it reads¶

Key	Reads	Kind	Described under
`VMAUTH_USER`	`GLOBAL_VMAUTH_USER`	Variable	Telemetry
`VMAUTH_PASS`	`GLOBAL_VMAUTH_PASS`	Secret	Telemetry
`VMAUTH_HOST`	`GLOBAL_VMAUTH_HOST`	Variable	Telemetry
`DOCKNS_WAN_IP`	`DOCKNS_WAN_IP`	Variable	dockns
`DOCKNS_CF_API_KEY`	`DOCKNS_CF_API_KEY`	Secret	dockns
`DOCKNS_CF_ACCOUNT_ID`	`DOCKNS_CF_ACCOUNT_ID`	Secret	dockns
`DOCKNS_CF_ZONE_ID`	`DOCKNS_CF_ZONE_ID`	Secret	dockns
`DOCKNS_UNIFI_HOST`	`DOCKNS_UNIFI_HOST`	Variable	dockns
`DOCKNS_UNIFI_API_KEY`	`DOCKNS_UNIFI_API_KEY`	Secret	dockns

It also reads the 19 operational defaults, as every stack does.

The three telemetry values are the one login every agent uses. They exist from the time ci01 is built. See VictoriaMetrics (ci01).

The six dockns values are created when the fleet leaves bootstrap mode. See Leaving bootstrap mode.

dockns values	Needed on
`DOCKNS_UNIFI_HOST`, `DOCKNS_UNIFI_API_KEY`	Every VM. dockns writes no internal record today, so the two are staged for later. See Not yet confirmed
The three `DOCKNS_CF_` values and `DOCKNS_WAN_IP`	A VM that hosts something the internet reaches

On a VM with nothing public, set the second row's four keys to blank under komodo_stack_env in the inventory:

komodo_stack_env:
  system-agent:
    DOCKNS_WAN_IP: ""
    DOCKNS_CF_API_KEY: ""
    DOCKNS_CF_ACCOUNT_ID: ""
    DOCKNS_CF_ZONE_ID: ""

See Blanking a reference. The same place points a second site's hosts at that site's UniFi console. See dockns.

What the host needs¶

The project is system, so the stack's folders sit under /opt/docker/volumes/system and /opt/docker/logs/system. Both are mode 0750, owned by the admin account with group 101000.

Folder	Owner	Mode
`/opt/docker/volumes/system/vmagent-data`	`101000:101000`	Default
`/opt/docker/volumes/system/vlagent-data`	`101000:101000`	Default
`/opt/docker/volumes/system/vector-data`	`101000:101000`	Default
`/opt/docker/volumes/system/dockns-data`	`100000:100000`	Default

Port	Allowed from	Comment
`5140/tcp`	The internal subnet	Vector syslog
`5140/udp`	The internal subnet	Vector syslog
`7007/tcp`	The internal subnet	Dozzle agent

The host's NixOS configuration creates the folders and opens the ports when the host is deployed. It reads them from the host's file in the private repo, nixos/hosts/<host>.json, which nixos-sync.yml writes from the stack's setup.yaml. See Host layout.

dockns runs as root inside its container, which the host sees as 100000. The other three folders belong to 101000. See UID offsets.

vmagent also mounts two files that the host's NixOS configuration puts in place for Node Exporter, outside the stack's own setup:

File	Holds
`/etc/node-exporter/node_exporter.crt`	Node Exporter's self-signed certificate
`/etc/node-exporter/scrape-password`	This host's scrape password, owned by `101000` and mode `0400`

The host makes the password and the certificate the first time it boots, keeps them on its persistent disk, and copies them into /etc/node-exporter before Docker starts. The password never leaves the host, so no Komodo Secret holds it. A host's vmagent scrapes its own Node Exporter and no other.

The host's configuration opens port 9100 as well, which is where vmagent reaches Node Exporter.

Verify¶

In Komodo, the system-agent-<host> Stack shows as running with seven services.

On the host, list the Stack's containers. Komodo names the Compose project after the Stack, not after PROJECT_NAME:

docker compose -p system-agent-<host> ps

Every container that has a health check shows healthy in the STATUS column.

The agents show as healthy whether or not anything reaches ci01, so check the far end. In Grafana on ci01, run this query with your host in place of id01:

up{instance=~".*id01.*"}

The result has a series with the value 1 for each of vmagent, vlagent, cadvisor, and node. A VM with a Traefik has one more, for the job traefik-traefik.

A missing node series with the rest present means vmagent did not read the scrape password. Check that /etc/node-exporter/scrape-password is a file and not a folder. Docker creates a folder at a mount's source when nothing is there.

For logs, open VictoriaLogs on ci01 and filter on stream_name.

`stream_name` starts with	Comes from
`container-`	A container's output
A systemd unit's name	The journal
`host-`	A `.log` file under `/var/log`
`syslog-`	A device sending syslog to port 5140

Traefik's access log has no stream_name. vector sends it with RouterName and ServiceName as its stream fields, so filter on RouterName:* to find it. Lines appear once the VM's Traefik has routed a request.

Each access log line is stored under the time its request started, not the time the line reached ci01. vector's transform writes that start time to the field timestamp, and its Traefik sink names the same field as _time_field. See Search Traefik's access log.

Data worth keeping¶

Folder under `/opt/docker/volumes/system`	Holds
`dockns-data`	`record-db.json`, dockns's own list of the records it has written

The other three folders are buffers and checkpoints. Losing them loses at most what had not been sent.

Not yet confirmed¶

The stack has not been deployed on any host. These are the points most likely to need work.

The stack on ci01. Its agents send to the vmauth on their own host, by the name Traefik answers on. That path has not been tried.
dockns and the labels. Three containers in fleet-stacks carry dockns labels (ntfy, Stalwart, and Bulwark), and those labels name a server called technitium. This stack gives dockns two servers, cloudflare and unifi. Until the labels change and the other containers gain them, dockns writes no internal record.
The Traefik scrape. vmagent shares no Docker network with the Traefik stack, and Traefik publishes no metrics port on the host.
The time on Traefik's access log lines. The sink's _time_field now names the field the transform writes, and no line has been sent to VictoriaLogs to see which time it is stored under.
Syslog over TCP. The stack publishes port 5140 for TCP and UDP and the firewall allows both. vector's syslog source listens on UDP alone.