Applications (ap01)¶
ap01 is the host for whatever you want the fleet to run. Nothing in the plan depends on it, so it is built last, and this page is also the worked example of adding a host of your own.
The page puts two stacks on ap01. The first, dozzle-server, exists in fleet-stacks already and only has to be listed. The second is one you write yourself, from the repo's template. Skip step 3 when every stack you want exists.
Every step is the same for any host. Where the page says ap01, a host of your own has its own name, address, and stacks, and each step says what to change. At the end you have a VM that runs the stacks you listed, built by one run, and you have done every part of adding a host once.
Status: written, not yet run.
Prerequisites¶
- The foundation is finished, through The handover.
- The fleet has left bootstrap mode. See Leaving bootstrap mode.
- For step 3, a GitHub account, and a machine with git, Python 3, PyYAML, and Docker Compose.
Placeholders¶
| Placeholder | Value |
|---|---|
<github-login> |
Your GitHub username |
<image> |
The name of the image your stack runs, in lower case, such as vaultwarden |
<stack> |
The new stack's folder name, such as vaultwarden-server |
<project> |
The new stack's project name, which prefixes its containers and folders, such as vaultwarden |
<key-prefix> |
The image's name in upper case, which prefixes its keys, such as VAULTWARDEN |
1. Describe the host¶
Pick a name, an address, and a VMID. The fleet names a host after its role, two letters and a number, and these pages build the VMID from the VLAN and the address's last part.
A host is described in two files of the private repo. The inventory, hosts.yml, says what the host is called and which stacks it runs. OpenTofu's variables file, opentofu/prod.tfvars, says how big its VM is.
In hosts.yml, add ap01 to the docker_host group:
ap01:
ansible_host: 172.16.7.151
serverHostname: "ap01"
docker_stacks:
- system-agent
- traefik-agent
Every VM lists system-agent. A VM that serves a web interface lists traefik-agent as well.
ap01 is on the internal network, so it takes the gateway the docker_host group sets. A host on another network sets network_gateway in its own entry, as bh01 does.
ap01 is built after the fleet has left bootstrap mode, so the run deploys the list as it is written. For a host added earlier than that, see Bootstrap mode.
In opentofu/prod.tfvars, add its VM inside vms:
ap01 = {
server = "vh01"
vm_id = 7151
cores = 2
memory_mb = 4096
vlan_id = 7
ipv4_address = "172.16.7.151/24"
ipv4_gateway = "172.16.7.1"
dns_servers = ["172.16.7.1"]
tags = ["docker"]
extra_disks = {
persist = { interface = "scsi2", size_gb = 20 }
}
}
Two cores, 4 GB, and 20 GB are enough for the two stacks on this page. Size your own host for what it runs.
The persistent disk holds every stack's data and can grow later, but never shrink. See Growing a disk.
For a host of your own, change these and keep the rest as it is:
| In the example | For a host of your own |
|---|---|
ap01, as the key in hosts.yml, in serverHostname, and as the key in prod.tfvars |
The host's name, the same in all three places |
172.16.7.151, in ansible_host and in ipv4_address |
A free address on the host's network. The copy in prod.tfvars carries the prefix length |
vm_id |
A VMID that is free in Proxmox |
vlan_id, ipv4_gateway, dns_servers |
The VLAN, gateway, and DNS server of the host's network |
server |
The Proxmox server that holds the VM, by its name under servers |
cores, memory_mb, size_gb |
What the host's stacks need |
docker_stacks |
system-agent, traefik-agent for a host that serves a web interface, then the host's own stacks, which steps 2 and 3 add |
The address and the gateway are written in both files, and the run stops when the two disagree.
See Describing a host for every key in both entries, and km01's page for a walk through the common ones.
2. Add a stack that exists¶
Choose the stack from Stacks and read its page before listing it. Two sections of the page decide what the host needs.
| Section of the stack page | What to check |
|---|---|
| Values it reads | Every Variable and Secret in the table exists in Komodo before the run |
| What the host needs | A stack that needs a Traefik on its host is why traefik-agent is in the list. A port open to the internal subnet only needs docker_stacks_internal_subnet, which the docker_host group sets already |
Add the stack to ap01's list by its folder name:
dozzle-server reads no Variable or Secret of its own and opens no port. Traefik routes dozzle.ap01.home.myah-mitchell.com to it.
To set a key of the stack's komodo.env for this host, add komodo_stack_env to the entry. komodo.env is the file in a stack's folder that becomes the Stack's Environment in Komodo.
dozzle-server runs no agent of its own. It reads each host, ap01 included, from the Dozzle agent in that host's system-agent, and takes the list from DOZZLE_REMOTE_AGENT, as addresses with the port, separated by commas:
komodo_stack_env:
dozzle-server:
DOZZLE_REMOTE_AGENT: "172.16.7.151:7007,172.16.7.101:7007,172.16.7.121:7007"
system-agent:
DOCKNS_CF_API_KEY: ""
DOCKNS_CF_ACCOUNT_ID: ""
DOCKNS_CF_ZONE_ID: ""
DOCKNS_WAN_IP: ""
A host's own komodo_stack_env replaces the one the docker_host group sets, and the two are not merged. That is why the system-agent block from Leaving bootstrap mode is repeated here. See Stack values for the rules a value follows.
For a host of your own, list the stacks you chose in place of dozzle-server. Add a komodo_stack_env block only when a stack's page names a key that differs by host, and repeat the group's system-agent block in it when you do.
3. Add a stack of your own¶
A stack is a folder under stacks/ in fleet-stacks that holds one hand-written file, compose.yaml. Each service in it extends a container defined under containers/, and scripts/build.py generates the rest of the folder from those containers. See the Docker Compose primer for how the two folders fit together.
This step is the same on any host, since a stack belongs to the repo and not to a host. Only List it on the host names ap01. The example values in the placeholder table are for Vaultwarden, and every <image>, <stack>, <project>, and <key-prefix> below takes the names of your own application.
File in stacks/<stack> |
Written by | Holds |
|---|---|---|
compose.yaml |
You | The project name, the networks, and one extends per service |
komodo.env |
build.py |
The Stack's Environment in Komodo |
setup.yaml |
build.py |
The folders, seed files, and open ports the host's configuration takes from the stack |
README.md |
build.py |
The same host setup as tables, with commands for the folders and files |
build.py also writes a .env file with values for local testing. It is ignored by git and never reaches Komodo.
Get a repo you can push to¶
Skip this when you can push to the fleet-stacks repo the fleet uses.
On GitHub, open myah-mitchell/fleet-stacks, click Fork, and clone the fork. Then point the run and Komodo at it, in the vars of the docker_host group in hosts.yml:
vars:
docker_stacks_repo_url: "https://github.com/<github-login>/fleet-stacks.git"
komodo_stacks_repo: "<github-login>/fleet-stacks"
Keep the fork public. The repo holds no secret, and neither the run nor Komodo then needs a credential to read it.
Both values apply to every host in the group, so every host's Stacks move to the fork on its next run.
Add the container¶
From the root of the fleet-stacks checkout, copy the container template to a folder named after the image:
File in containers/<image> |
Edit it to hold |
|---|---|
compose.yaml |
One service, named with a leading dot, such as .<image> |
komodo.env |
The keys only this container reads |
setup.yaml |
What the container needs on the host |
stack-README.md, testing.env |
Text for the stack's README, and values for local testing |
In compose.yaml, replace imageName and IMAGENAME throughout, and set the image and the port Traefik forwards to. Delete what the container does not use: the command, the published port, the volumes, the kop-public labels, and the DockNS labels. Keep the block of defaults above the service as it is.
The template's route uses chain-no-auth@file, which never asks for a sign-in. For a service that Authentik should guard, use the line dozzle's container uses. See the auth chain for what the two chains do.
Then give the key a line in the container's komodo.env, so the run can fill it in:
In setup.yaml, list each folder the container mounts, with the owner as the host sees it. The container's UID 1000 is 101000 on the host. See UID offsets.
A container with Traefik labels also says that it needs a Traefik on its host:
A port the container publishes on the host gets an entry under firewall as well. A service reached through Traefik publishes none. The template's own setup.yaml shows every kind of entry, and scripts/project-layout.md in fleet-stacks describes every key.
Add the stack¶
Copy the stack template, and give the new stack its project name and its service:
In stacks/<stack>/compose.yaml, set the project name in the comment near the top. build.py reads the name from that line.
Then add the service under the services: key:
The template declares four networks: proxy, frontend, backend, and socket_proxy. Comment out the ones no service joins, as dozzle-server does with backend.
A stack with a database adds one more service that extends containers/postgres, and the application's container names it under needs_stack in its setup.yaml. authentik-server is the fullest example in the repo.
Generate and check¶
Generate the stack's files:
The output ends with Build complete. The script stops with a message when a setup.yaml has an unknown key, or when a container needs a service that nothing in the stack provides.
Check that Compose accepts the result:
The command prints nothing when the stack is valid.
Commit the container folder and the stack folder together, generated files included, and push. The repo's CI fails when the committed files differ from what build.py generates.
Give it its values¶
A line in a container's komodo.env takes a fixed value, stays blank, or references a Komodo Variable or Secret by name:
#= Stack Specific Settings
#== <image>
<key-prefix>_HOSTNAME: <image>
<key-prefix>_ADMIN_PASSWORD: [[<key-prefix>_ADMIN_PASSWORD]]
Every secret is a reference, written as the name in double square brackets. Nothing secret is written in the repo, and a value that differs by host goes in the inventory's komodo_stack_env. See How a stack gets its values.
The stacks in the repo name a value after the service that owns it, and start the name with GLOBAL_ when more than one stack reads it. Keep a generated secret alphanumeric: 48 characters for a database password and 96 for any other.
List it on the host¶
Add the stack to ap01's docker_stacks, after the ones from step 2. For a host of your own, this is that host's list:
A stack that runs on more than one host needs the host's name on the end of its Stack, since Komodo wants Stack names unique. Add it to komodo_stacks_per_host in the docker_host group's vars. The value replaces the role's default, so repeat the six names the default holds:
komodo_stacks_per_host:
- system-agent
- traefik-agent
- traefik-bootstrap
- crowdsec-agent
- dozzle-agent
- victoriametrics-agent
- <stack>
Add it to these pages¶
This step is for whoever keeps a copy of this site. The tables on a stack page are generated from fleet-stacks. From the docs checkout, with fleet-stacks checked out next to it:
The script writes four snippets under snippets/generated/<stack> and the register in Variables and Secrets. It stops when the new komodo.env references a name that scripts/fleet-register.yaml does not describe. Add the name there, with its kind and what it holds, and run the script again.
Then write stacks/<stack>.md in the shape of the other stack pages, and add it to the site's navigation. The register links to that page for every value the stack reads.
4. Stage the values¶
Create every Variable and Secret the host's stacks read, in Komodo, before the run. See Creating one for the clicks.
dozzle-server needs none. Your own stack needs one for each reference from Give it its values.
For a host of your own, go through the Values it reads table on the page of each stack it lists.
A value that is missing does not stop the run. It reaches the container as the literal text of the reference. See How a stack gets its values.
Then generate ap01's files: its SSH host keys, its NixOS file, and its Komodo file. For a host of your own, put its name wherever the commands say <host>.
From ~/src/fleet-ansible, with the private repo and fleet-nixos checked out next to it, and the deploy key in SOPS_AGE_KEY or SOPS_AGE_KEY_FILE.
A host with no file under secrets/host-keys/ in the private repo gets its SSH host keys first:
The command writes secrets/host-keys/<host>.yaml, adds the host to .sops.yaml, and encrypts secrets/fleet.yaml again so that the host can read it. For a host that has its keys already, it changes nothing. See Host keys.
Then write the NixOS files and the Komodo files:
ansible-playbook -i ../fleet-private/hosts.yml nixos-sync.yml
ansible-playbook -i ../fleet-private/hosts.yml komodo-sync.yml
Add what changed, and read the diff before it is committed:
git -C ../fleet-private add hosts.yml group_vars/ opentofu/ komodo/ nixos/ secrets/ .sops.yaml
git -C ../fleet-private diff --cached
The diff of nixos/hosts/<host>.json shows the host's network, its features, and the folders, seed files and ports its stacks need. The diff of komodo/stacks/<host>.toml shows the Stacks the next run deploys and every value in their Environment. The files under secrets/ are encrypted, so their diff shows that a value changed and not what it changed to.
Then commit and push:
The run builds the host from the files git tracks in the private repo, and Komodo reads the pushed copy. The run stops at a host whose committed files differ from what the inventory gives.
Both playbooks read each stack's setup.yaml from the repo's main branch as pushed. Push the commit from step 3 before you generate.
After a move to a fork, the diff also shows the new repo in every other host's Komodo file. Commit those with it.
5. Run the build¶
In Semaphore, run the site Template with Target set to ap01.
From ~/src/fleet-ansible, in a shell prepared for runs after the handover. See Running from a shell again.
The run creates the VM, installs NixOS on it, deploys its configuration, and has Komodo deploy system-agent-ap01, traefik-agent-ap01, dozzle-server, and your own Stack.
For a host of your own, the only change is the target: the host's name in Target, or after target=. See How a host is built for what each stage of the run does.
Manual steps, instead of site.yml
Create the VM by hand¶
The run's first stage creates the VM through OpenTofu, from the host's entry in opentofu/prod.tfvars. By hand, on the Proxmox host, with the values from that same entry:
qm create <vmid> --name <host> --ostype l26 \
--cpu x86-64-v2-AES --cores <cores> --numa 1 \
--memory <memory> --balloon 0 \
--scsihw virtio-scsi-single \
--scsi0 <vm-storage>:<os-size>,discard=on,ssd=1,iothread=1 \
--scsi1 <vm-storage>:<docker-size>,discard=on,ssd=1,iothread=1 \
--scsi2 <vm-storage>:<persist-size>,discard=on,ssd=1,iothread=1 \
--ide2 local:iso/fleet-nixos-installer.iso,media=cdrom \
--ide0 <vm-storage>:cloudinit \
--ipconfig0 ip=<address>,gw=<gateway> \
--nameserver <dns-server> \
--net0 virtio,bridge=vmbr0,tag=<vlan-id> \
--boot 'order=scsi0;ide2' \
--serial0 socket --vga serial0 \
--agent enabled=1 --onboot 1 \
--tags 'nixos;<tags>'
qm start <vmid>
| Placeholder | Value |
|---|---|
<vmid> |
The entry's vm_id |
<host> |
The entry's name, which is the host's name in the inventory |
<cores>, <memory> |
The entry's cores and memory_mb |
<vm-storage> |
The Proxmox storage that holds VM disks, local-zfs unless the server's entry sets datastore_id |
<os-size>, <docker-size> |
The entry's os_disk_gb and docker_disk_gb in GB, 20 and 40 when it sets neither |
<persist-size> |
The persist disk's size_gb |
<address>, <gateway> |
The entry's ipv4_address, with its prefix length, and ipv4_gateway |
<dns-server> |
The entry's dns_servers, separated by spaces inside one pair of quotes when there are several |
<vlan-id> |
The entry's vlan_id |
<tags> |
The entry's tags, separated by semicolons |
An entry with no dns_servers takes no --nameserver option, and one with no vlan_id takes no ,tag= part.
The three disks are blank. The VM finds nothing to boot on scsi0, so it boots the installer ISO on ide2, and the installer takes its address from the cloud-init drive on ide0.
The ISO has to be on the Proxmox host first. See The installer ISO.
Create all three disks before the first start. The install partitions scsi0 and scsi1 and formats scsi2, and it stops when one is missing.
A VM made this way is not in OpenTofu's state, and it does not carry the tofu tag that marks the VMs OpenTofu manages. Leave its entry out of opentofu/prod.tfvars, or a later run tries to create a second VM under the same name and address.
Install NixOS without the rest of the run¶
The run's second and third stages wait for the VM's SSH port, install NixOS when the VM answers as the installer, and deploy the host's configuration. The same three commands of the flake do it from a shell.
From ~/src/fleet-ansible, with the host's files committed in the private repo, the deploy key in SOPS_AGE_KEY or SOPS_AGE_KEY_FILE, and the fleet's SSH key in ~/.ssh/config. See The fleet's SSH key.
Ask the VM what it runs:
It prints installer for a VM that booted the ISO. Then install:
Warning
The install wipes scsi0 and scsi1. It formats scsi2 only when the disk is blank, and it refuses any machine that is not the installer.
The last line reads install-host: <host> is installed and is rebooting. Repeat the host-state command until it prints installed, then deploy the configuration:
<host> is the host's name in the inventory, and <address> is its ansible_host. The configuration is the whole base build: accounts, SSH, the firewall, Docker, the persistent disk, Periphery, Node Exporter, and the folders, seed files and ports of every stack in the host's file. See The commands.
For system-agent:
A host deployed with the stack in its file has these folders already. The commands make them on a host whose file does not list the stack.
sudo install -d -o $USER -g 101000 -m 0750 \
/opt/docker/logs/system /opt/docker/volumes/system
sudo install -d -o 101000 -g 101000 /opt/docker/volumes/system/vmagent-data
sudo install -d -o 101000 -g 101000 /opt/docker/volumes/system/vlagent-data
sudo install -d -o 101000 -g 101000 /opt/docker/volumes/system/vector-data
sudo install -d -o 100000 -g 100000 /opt/docker/volumes/system/dockns-data
No command opens the stack's ports, because the firewall changes only through the host's configuration. The ports below are open once the host's file, nixos/hosts/<host>.json, holds them: nixos-sync.yml writes that file in the private repo, and a deploy of the host applies it. See Host layout.
| Port | Allowed from |
|---|---|
5140/tcp |
The internal subnet |
5140/udp |
The internal subnet |
7007/tcp |
The internal subnet |
For traefik-agent:
A host deployed with the stack in its file has these folders already. The commands make them on a host whose file does not list the stack.
sudo install -d -o $USER -g 101000 -m 0750 \
/opt/docker/logs/traefik /opt/docker/volumes/traefik
sudo install -d -o 101000 -g 101000 -m 0755 /opt/docker/logs/traefik/traefik
sudo install -d -o 101000 -g 101000 /opt/docker/volumes/traefik/traefik-certs
sudo install -d -o 101000 -g 101000 /opt/docker/volumes/traefik/traefik-plugins
No command opens the stack's ports, because the firewall changes only through the host's configuration. The ports below are open once the host's file, nixos/hosts/<host>.json, holds them: nixos-sync.yml writes that file in the private repo, and a deploy of the host applies it. See Host layout.
| Port | Allowed from |
|---|---|
80/tcp |
Anywhere |
443/tcp |
Anywhere |
8443/tcp |
Anywhere |
For dozzle-server:
A host deployed with the stack in its file has these folders already. The commands make them on a host whose file does not list the stack.
For a stack of your own, the same commands are in the README.md that build.py wrote into its folder.
Deploy the Stacks by hand¶
The run's last stage has Komodo read the host's Stacks from komodo/stacks/<host>.toml in the private repo and deploy them. To create and deploy one in Komodo's UI instead, follow Deploying a stack by hand.
A Stack made by hand under the name the run would give it is taken over by the next run, not duplicated.
6. Verify¶
The run ends with ansible's recap, one line of counts for the host. failed=0 and unreachable=0 mean every stage finished, and the last stage only finishes once Komodo reports every one of the host's Stacks as running.
In Komodo's UI, open Resources > Servers. The host shows as connected.
Open Resources > Stacks and filter by the host's Server. Each Stack shows as Running.
If the run stopped instead, its last message names the stage and what it stopped at. Fix the cause and run it again. Every stage is safe to repeat.
The dozzle-server Stack has one service:
Open https://dozzle.ap01.home.myah-mitchell.com in a browser. The name needs a DNS record pointing at ap01, or an entry in your own hosts file.
Authentik asks for a sign-in first. Dozzle then lists ap01's containers, and those of every host named in DOZZLE_REMOTE_AGENT.
For a host of your own, the first check is the same for every host. After it, follow the Verify section on the page of each stack the host runs, and open the names under its Hostnames.
What's next¶
ap01 is the last host in the running order.
For a further host, the short form of this page is Adding a host.
Not yet confirmed¶
- The whole page. ap01 has not been built by the run, and no stack has been added through these steps.
- dozzle-server reading its own host. It reaches ap01's agent at ap01's address and port 7007, from inside a Docker network, and whether the firewall rule for the internal subnet admits that has not been tried. Docker publishes the port with rules of its own, so the request may never pass the chain the host's rule is in. See dozzle-server.
- The sign-in in front of Dozzle. It depends on the chain that Leaving bootstrap mode sets up, which has not been run.
- A run against a fork.
docker_stacks_repo_urlandkomodo_stacks_repoare ordinary role defaults, and setting them in the inventory has not been tried. - The format of
DOZZLE_REMOTE_AGENT. It follows Dozzle's own documentation and has not been tried in this fleet.