Automation and monitoring (ci01)¶

ci01 runs what the rest of the fleet is operated with: Semaphore, which runs every build after the handover, the VictoriaMetrics backend every host reports to, and the fleet's notifications, mail relay, and uptime checks.

It is the second VM built, and the last one built from the control shell. It comes straight after km01 because Semaphore is what takes over from the shell, and every later host is built from it. See The handover.

One run builds the host and deploys its three stacks. The first deploy downloads about twenty images, so allow the run up to a quarter of an hour. Each stack has a page of its own for its values, its checks, and its first sign-in.

Stack	Provides	Page
semaphore-server	Semaphore, and the database that holds OpenTofu's state	Semaphore (ci01)
victoriametrics-server	Metrics, logs, traces, alerting, and Grafana	VictoriaMetrics (ci01)
core-infra	Notifications, the mail relay, and uptime checks	Core infrastructure (ci01)

ci01's first build is part of the foundation. Build ci01 on The first run sends you to this page and has you follow it from step 1 to step 5. This page does not send you back there: its last section sends you on to the foundation's next page.

Status: written, not yet run.

Prerequisites¶

km01 is built and Komodo deploys its own Stack, through step 5 of The first run.
The operational defaults and the Traefik values exist in Komodo, from Setting up Komodo.
The login of an account at an SMTP relay, such as your mail provider's submission service. Core infrastructure (ci01) says how to run without one.

1. Describe the host¶

In the private repo's hosts.yml, add ci01 to the docker_host group. The first three keys are the ones km01's entry explains:

    ci01:
      ansible_host: 172.16.7.121
      serverHostname: "ci01"
      docker_stacks:
        - system-agent
        - traefik-agent
        - semaphore-server
        - victoriametrics-server
        - core-infra
      komodo_stack_env:
        core-infra:
          POSTFIX_RELAYHOST: "[smtp.myah-mitchell.com]:587"
          POSTFIX_RELAYHOST_USERNAME: "relay@myah-mitchell.com"

The list above is what the host runs once the fleet has left bootstrap mode. While docker_stacks_bootstrap: true is set, the run deploys the host without system-agent and traefik-agent, and with traefik-bootstrap in their place. nixos-sync.yml and komodo-sync.yml make that swap when they write the host's files, so the list itself never changes. See Bootstrap mode.

victoriametrics-server is the backend alone, so ci01 reports nothing about itself until system-agent is deployed, when the fleet leaves bootstrap mode.

komodo_stack_env is new here. It sets keys of one stack's Environment for this host, under the stack's folder name, and holds settings only, never a secret. See Stack values.

The two Postfix keys say where the fleet's mail is handed on and under which account. Replace both with your relay's host and port and your account's login, and keep the square brackets. See Core infrastructure (ci01) for what each key does.

In opentofu/prod.tfvars, add its VM inside vms. The fields are the ones km01's entry explains:

  ci01 = {
    server       = "vh01"
    vm_id        = 7121
    cores        = 4
    memory_mb    = 8192
    vlan_id      = 7
    ipv4_address = "172.16.7.121/24"
    ipv4_gateway = "172.16.7.1"
    dns_servers  = ["172.16.7.1"]
    tags         = ["docker"]
    extra_disks = {
      persist = { interface = "scsi2", size_gb = 100 }
    }
  }

Semaphore and core-infra are light. victoriametrics-server is what the four cores and 8 GB are for, with seven services that include three databases and Grafana.

The 100 GB disk is for those databases. Metrics are kept for 60 days with no cap on size. Logs and traces are kept for a year, each capped at 5 GB.

ci01's address is long-lived. A stack that sends mail has it in GLOBAL_EMAIL_HOST, which Authentik is the first to read, and Proxmox sends its notifications to it.

Then generate ci01's files: its SSH host keys, its NixOS file, and its Komodo file.

From ~/src/fleet-ansible, with the private repo and fleet-nixos checked out next to it, and the deploy key in SOPS_AGE_KEY or SOPS_AGE_KEY_FILE.

A host with no file under secrets/host-keys/ in the private repo gets its SSH host keys first:

nix run ../fleet-nixos#new-host-key -- --fleet ../fleet-private <host>

The command writes secrets/host-keys/<host>.yaml, adds the host to .sops.yaml, and encrypts secrets/fleet.yaml again so that the host can read it. For a host that has its keys already, it changes nothing. See Host keys.

Then write the NixOS files and the Komodo files:

ansible-playbook -i ../fleet-private/hosts.yml nixos-sync.yml
ansible-playbook -i ../fleet-private/hosts.yml komodo-sync.yml

Add what changed, and read the diff before it is committed:

git -C ../fleet-private add hosts.yml group_vars/ opentofu/ komodo/ nixos/ secrets/ .sops.yaml
git -C ../fleet-private diff --cached

The diff of nixos/hosts/<host>.json shows the host's network, its features, and the folders, seed files and ports its stacks need. The diff of komodo/stacks/<host>.toml shows the Stacks the next run deploys and every value in their Environment. The files under secrets/ are encrypted, so their diff shows that a value changed and not what it changed to.

Then commit and push:

git -C ../fleet-private commit -m "Describe <host>"
git -C ../fleet-private push

The run builds the host from the files git tracks in the private repo, and Komodo reads the pushed copy. The run stops at a host whose committed files differ from what the inventory gives.

2. Stage the values¶

Create every value ci01's stacks read, in Komodo, before the run. Start at Semaphore's values. Each section ends with a link to the next, and the third sends you back to Check the values below.

Order	Section	Creates
1	Semaphore's values	Ten Secrets
2	VictoriaMetrics' values	Two Variables and one Secret
3	Core infrastructure's values	One Secret

Check the values¶

The run does not check that a value exists. A missing one reaches the container as literal text, and the stack fails in a way that does not name the cause. See How a stack gets its values.

In Komodo, open Settings > Variables and search for SEMAPHORE, VMAUTH, and POSTFIX in turn. The three searches list ten, three, and one name.

3. Run the build¶

On the first build Semaphore does not exist yet, so the run comes from the control shell. Use the Command line tab.

SemaphoreCommand line

In Semaphore, run the site Template with Target set to ci01.

From ~/src/fleet-ansible, with the environment file loaded.

On the first build, add -e vms_backend=local to the command below. The option keeps OpenTofu's state in a file in the shell, since the state database is on ci01 itself. Build ci01 shows the whole command.

On any later run, prepare the shell first. See Running from a shell again.

ansible-playbook -i ../fleet-private/hosts.yml site.yml \
  -e target=ci01

The run creates the VM, installs NixOS on it, deploys its configuration, and has Komodo deploy four Stacks: traefik-bootstrap-ci01, semaphore-server, victoriametrics-server, and core-infra.

The first deploy pulls about twenty images. The run waits up to fifteen minutes for a host's Stacks, and a run that gives up while images are still downloading passes when run again.

Manual steps, instead of site.yml

Create the VM by hand¶

The run's first stage creates the VM through OpenTofu, from the host's entry in opentofu/prod.tfvars. By hand, on the Proxmox host, with the values from that same entry:

qm create <vmid> --name <host> --ostype l26 \
  --cpu x86-64-v2-AES --cores <cores> --numa 1 \
  --memory <memory> --balloon 0 \
  --scsihw virtio-scsi-single \
  --scsi0 <vm-storage>:<os-size>,discard=on,ssd=1,iothread=1 \
  --scsi1 <vm-storage>:<docker-size>,discard=on,ssd=1,iothread=1 \
  --scsi2 <vm-storage>:<persist-size>,discard=on,ssd=1,iothread=1 \
  --ide2 local:iso/fleet-nixos-installer.iso,media=cdrom \
  --ide0 <vm-storage>:cloudinit \
  --ipconfig0 ip=<address>,gw=<gateway> \
  --nameserver <dns-server> \
  --net0 virtio,bridge=vmbr0,tag=<vlan-id> \
  --boot 'order=scsi0;ide2' \
  --serial0 socket --vga serial0 \
  --agent enabled=1 --onboot 1 \
  --tags 'nixos;<tags>'
qm start <vmid>

Placeholder	Value
`<vmid>`	The entry's `vm_id`
`<host>`	The entry's name, which is the host's name in the inventory
`<cores>`, `<memory>`	The entry's `cores` and `memory_mb`
`<vm-storage>`	The Proxmox storage that holds VM disks, `local-zfs` unless the server's entry sets `datastore_id`
`<os-size>`, `<docker-size>`	The entry's `os_disk_gb` and `docker_disk_gb` in GB, `20` and `40` when it sets neither
`<persist-size>`	The persist disk's `size_gb`
`<address>`, `<gateway>`	The entry's `ipv4_address`, with its prefix length, and `ipv4_gateway`
`<dns-server>`	The entry's `dns_servers`, separated by spaces inside one pair of quotes when there are several
`<vlan-id>`	The entry's `vlan_id`
`<tags>`	The entry's `tags`, separated by semicolons

An entry with no dns_servers takes no --nameserver option, and one with no vlan_id takes no ,tag= part.

The three disks are blank. The VM finds nothing to boot on scsi0, so it boots the installer ISO on ide2, and the installer takes its address from the cloud-init drive on ide0.

The ISO has to be on the Proxmox host first. See The installer ISO.

Create all three disks before the first start. The install partitions scsi0 and scsi1 and formats scsi2, and it stops when one is missing.

A VM made this way is not in OpenTofu's state, and it does not carry the tofu tag that marks the VMs OpenTofu manages. Leave its entry out of opentofu/prod.tfvars, or a later run tries to create a second VM under the same name and address.

Install NixOS without the rest of the run¶

The run's second and third stages wait for the VM's SSH port, install NixOS when the VM answers as the installer, and deploy the host's configuration. The same three commands of the flake do it from a shell.

From ~/src/fleet-ansible, with the host's files committed in the private repo, the deploy key in SOPS_AGE_KEY or SOPS_AGE_KEY_FILE, and the fleet's SSH key in ~/.ssh/config. See The fleet's SSH key.

Ask the VM what it runs:

nix run ../fleet-nixos#host-state -- <address>

It prints installer for a VM that booted the ISO. Then install:

nix run ../fleet-nixos#install-host -- --fleet ../fleet-private <host> <address>

Warning

The install wipes scsi0 and scsi1. It formats scsi2 only when the disk is blank, and it refuses any machine that is not the installer.

The last line reads install-host: <host> is installed and is rebooting. Repeat the host-state command until it prints installed, then deploy the configuration:

nix run ../fleet-nixos#deploy-host -- --fleet ../fleet-private <host> <address>

<host> is the host's name in the inventory, and <address> is its ansible_host. The configuration is the whole base build: accounts, SSH, the firewall, Docker, the persistent disk, Periphery, Node Exporter, and the folders, seed files and ports of every stack in the host's file. See The commands.

For traefik-bootstrap:

A host deployed with the stack in its file has these folders already. The commands make them on a host whose file does not list the stack.

sudo install -d -o $USER -g 101000 -m 0750 \
  /opt/docker/logs/traefik /opt/docker/volumes/traefik
sudo install -d -o 101000 -g 101000 -m 0755 /opt/docker/logs/traefik/traefik
sudo install -d -o 101000 -g 101000 /opt/docker/volumes/traefik/traefik-certs
sudo install -d -o 101000 -g 101000 /opt/docker/volumes/traefik/traefik-plugins

No command opens the stack's ports, because the firewall changes only through the host's configuration. The ports below are open once the host's file, nixos/hosts/<host>.json, holds them: nixos-sync.yml writes that file in the private repo, and a deploy of the host applies it. See Host layout.

Port	Allowed from
`80/tcp`	Anywhere
`443/tcp`	Anywhere
`8443/tcp`	Anywhere

For semaphore-server:

A host deployed with the stack in its file has these folders and seed files already. The commands make them on a host whose file does not list the stack.

sudo install -d -o $USER -g 101000 -m 0750 \
  /opt/docker/logs/semaphore /opt/docker/volumes/semaphore
sudo install -d -o 101001 -g 101001 /opt/docker/volumes/semaphore/semaphore-data
sudo install -d -o 101001 -g 101001 /opt/docker/volumes/semaphore/semaphore-config
sudo install -d -o 101001 -g 101001 /opt/docker/volumes/semaphore/semaphore-tmp
sudo install -d -o 101001 -g 101001 /opt/docker/volumes/semaphore/nix-data
sudo install -d -o 100000 -g 100000 -m 0755 /opt/docker/volumes/semaphore/postgres-initdb
sudo install -d -o 100000 -g 100000 /opt/docker/volumes/semaphore/postgres-data
sudo install -d -o 100000 -g 100000 /opt/docker/volumes/semaphore/postgres-backup-data

sudo test -e /opt/docker/volumes/semaphore/postgres-initdb/10-tofu-state.sh || sudo curl -fsSL \
  -o /opt/docker/volumes/semaphore/postgres-initdb/10-tofu-state.sh \
  https://raw.githubusercontent.com/myah-mitchell/fleet-stacks/main/containers/semaphore/config/postgres-initdb/10-tofu-state.sh
sudo chown 100000:100000 /opt/docker/volumes/semaphore/postgres-initdb/10-tofu-state.sh
sudo chmod 0755 /opt/docker/volumes/semaphore/postgres-initdb/10-tofu-state.sh

For victoriametrics-server:

A host deployed with the stack in its file has these folders already. The commands make them on a host whose file does not list the stack.

sudo install -d -o $USER -g 101000 -m 0750 \
  /opt/docker/logs/victoriametrics /opt/docker/volumes/victoriametrics
sudo install -d -o 101000 -g 101000 /opt/docker/volumes/victoriametrics/victoriametrics-data
sudo install -d -o 101000 -g 101000 /opt/docker/volumes/victoriametrics/victorialogs-data
sudo install -d -o 101000 -g 101000 /opt/docker/volumes/victoriametrics/victoriatraces-data
sudo install -d -o 101000 -g 101000 /opt/docker/volumes/victoriametrics/grafana-data

For core-infra:

A host deployed with the stack in its file has these folders and seed files already. The commands make them on a host whose file does not list the stack.

sudo install -d -o $USER -g 101000 -m 0750 \
  /opt/docker/logs/core /opt/docker/volumes/core
sudo install -d -o 101000 -g 101000 /opt/docker/volumes/core/ntfy-data
sudo install -d -o 101000 -g 101000 -m 0700 /opt/docker/volumes/core/mailrise-secrets
sudo install -d -o 100000 -g 100000 /opt/docker/volumes/core/postfix-data
sudo install -d -o 101000 -g 101000 /opt/docker/volumes/core/mailpit-data
sudo install -d -o 101000 -g 101000 /opt/docker/volumes/core/blackbox-exporter-config
sudo install -d -o 101000 -g 101000 /opt/docker/volumes/core/uptime-kuma-data

sudo test -e /opt/docker/volumes/core/mailrise-secrets/mailrise.conf || sudo curl -fsSL \
  -o /opt/docker/volumes/core/mailrise-secrets/mailrise.conf \
  https://raw.githubusercontent.com/myah-mitchell/fleet-stacks/main/containers/mailrise/config/mailrise.conf.example
sudo chown 101000:101000 /opt/docker/volumes/core/mailrise-secrets/mailrise.conf
sudo chmod 0600 /opt/docker/volumes/core/mailrise-secrets/mailrise.conf
sudo test -e /opt/docker/volumes/core/blackbox-exporter-config/blackbox.yml || sudo curl -fsSL \
  -o /opt/docker/volumes/core/blackbox-exporter-config/blackbox.yml \
  https://raw.githubusercontent.com/myah-mitchell/fleet-stacks/main/containers/blackbox-exporter/config/blackbox.yml.example
sudo chown 101000:101000 /opt/docker/volumes/core/blackbox-exporter-config/blackbox.yml

No command opens the stack's ports, because the firewall changes only through the host's configuration. The ports below are open once the host's file, nixos/hosts/<host>.json, holds them: nixos-sync.yml writes that file in the private repo, and a deploy of the host applies it. See Host layout.

Port	Allowed from
`8025/tcp`	The internal subnet
`25/tcp`	The internal subnet

Deploy the Stacks by hand¶

The run's last stage has Komodo read the host's Stacks from komodo/stacks/<host>.toml in the private repo and deploy them. To create and deploy one in Komodo's UI instead, follow Deploying a stack by hand.

A Stack made by hand under the name the run would give it is taken over by the next run, not duplicated.

4. Verify¶

The run ends with ansible's recap, one line of counts for the host. failed=0 and unreachable=0 mean every stage finished, and the last stage only finishes once Komodo reports every one of the host's Stacks as running.

In Komodo's UI, open Resources > Servers. The host shows as connected.

Open Resources > Stacks and filter by the host's Server. Each Stack shows as Running.

If the run stopped instead, its last message names the stage and what it stopped at. Fix the cause and run it again. Every stage is safe to repeat.

Then check each stack on its own page. Start at Verify Semaphore. Each check ends with a link to the next, and the third sends you to step 5 of this page.

Order	Stack	Check
1	semaphore-server	Verify Semaphore
2	victoriametrics-server	Verify VictoriaMetrics
3	core-infra	Verify core infrastructure

5. Sign in and finish each service¶

Three of the services have no account until you make one, one has a default password to replace, and Semaphore's nix has no sops yet. Do these before moving on.

Start at Semaphore's sign-in. Each row ends with a link to the next, and the third sends you to What's next on this page.

Order	Service	What to do
1	Semaphore	Sign in with the admin account from its Secrets, then add sops to its nix
2	Grafana	Sign in and replace the default password
3	ntfy, mailrise, and Uptime Kuma	Steps 3 to 6 of Core infrastructure

What's next¶

On the first build, configure Semaphore to run the playbook the shell has been running. See The Semaphore project, the foundation's next page.

After the foundation, the next host is id01. See Identity (id01).

Not yet confirmed¶

The whole page. ci01 has not been built by the run.
Whether fifteen minutes is enough for the first deploy of all four Stacks on a host with no images. The nix image is among them, and its service copies the whole of /nix before Semaphore starts.
A run from Semaphore that redeploys semaphore-server. See The handover.