Fleet bootstrap¶
These pages build a fleet of Docker hosts on Proxmox, starting with nothing but the Proxmox host. Every VM is created blank by OpenTofu, installed with NixOS from one flake, and given its stacks by Komodo, all in a single ansible run started from Semaphore.
New to these tools¶
Start with The fleet at a glance. It shows what you are building, which tool does which job, and how a host goes from nothing to running services, with no step to follow.
Each tool has a primer under Tools, and each recurring term has an entry in the glossary. The build pages link to both where a tool or a term first appears, so you can read them as you meet them and need not read them all first.
Once the terms are familiar, read Conventions. It is the fleet-stacks repo's short page of rules for how hosts, stacks, and values are named and where a secret may be written down, and every page here assumes them.
How the section is laid out¶
| Part | Holds | Read it |
|---|---|---|
| Concepts | The fleet at a glance, then the network and DNS the fleet expects, what the run does, the NixOS flake, the private repo and its secrets, bootstrap mode, the host's layout, and the register of every value | Once, before the first host |
| The foundation | Proxmox, the installer ISO, km01, ci01, Komodo, and Semaphore, built from a shell | Once, in order |
| Hosts | One page per host, in the order below | One per build |
| Stacks | Reference for every stack: what it runs, reads, and needs | When you need a fact |
| Procedures | Work that is not tied to one host | When the occasion comes |
Running order¶
Every page in the table has the status "Written, not yet run". See What the statuses mean.
| Order | Page | Builds |
|---|---|---|
| 1 | The control shell | The first control node, the age keys, and the private repo's first contents |
| 2 | Proxmox and the installer ISO | The installer ISO and the API token |
| 3 | The first run with Komodo (km01) | km01 and Komodo Core |
| 4 | Setting up Komodo | Komodo's users, keys, sync, and first values |
| 5 | Automation and monitoring (ci01) | ci01 and its three stacks |
| 6 | The Semaphore project | Semaphore's Project and the site Template |
| 7 | The handover | Semaphore as the control node |
| 8 | Identity (id01) | Authentik |
| 9 | Certificates (pk01) | step-ca |
| 10 | Traefik hub (tf01) | The internal hub and its Redis |
| 11 | DMZ edge (bh01) | The public edge and the tunnel |
| 12 | Leaving bootstrap mode | The real Traefik, sign-in, and telemetry on every host |
| 13 | Mail (mx01) | Mailboxes. Optional |
| 14 | Applications (ap01) | A host of your own, as a worked example |
Rows 1 to 7 are the foundation, and The foundation explains why they come in that order. From row 8 on, every host is one run of the site Template.
ci01 has three more pages, one for each of its stacks: Semaphore, VictoriaMetrics, and Core infrastructure. The ci01 page sends you through them.
id01 comes before pk01, tf01, and bh01 because the others can wait. Nothing asks Authentik for a sign-in, and nothing publishes a route to tf01, until the fleet leaves bootstrap mode. bh01 needs tf01 first, because its Redis replicates tf01's.
The fleet leaves bootstrap mode at row 12, with every host it was waiting for up. mx01 and ap01 are built after that, in normal mode, as any host added later is.
mx01 is optional, and no other host waits on it. The service mail every stack sends goes through Postfix on ci01 with or without mx01. Once mx01 is live, Postfix relays that mail through Stalwart.
What the statuses mean¶
"Written, not yet run" means the page was assembled from the playbooks, the flake, the compose files, and the generated facts, and checked against them. Nobody has followed it against a real host, and every page has that status. Treat every UI label and every wait in such a page as needing confirmation, and correct the page as you go.
Each page lists what it could not confirm under Not yet confirmed.
Bootstrap mode¶
Most stacks expect three things another host provides: a sign-in from Authentik on id01, a telemetry backend on ci01, and the hub on tf01. None exists when the first hosts are built.
The fleet starts in bootstrap mode, where each host runs a stand-in Traefik that depends on no other host, and leaves the mode in one procedure when the hosts it was waiting for are up. See Bootstrap mode.
Procedures¶
These are for a fleet that is already built. Each one opens with what it does, when you need it, and what it puts at risk.
| Page | Covers |
|---|---|
| Adding a host | The short form of every host page, for a host of your own |
| Leaving bootstrap mode | Moving every host to the real Traefik, the sign-in chain, and telemetry |
| Deploying a stack by hand | Creating a Stack in Komodo without the run |
| Rebuilding a VM | Installing a host again and keeping its persistent disk |
| Growing a disk | Making a host's disk larger |
| Updating the fleet | Moving the hosts to newer versions of NixOS and its packages |
What is not covered¶
These pages build the fleet's VMs. They do not install Proxmox itself, and they do not cover a backup server or any other machine that is not one of these VMs.
Conventions and Project layout describe the fleet-stacks repo itself, so they stay in that repo.