The foundation¶
The foundation is everything that has to exist before a host can be built by running one Template in Semaphore: the fleet's keys and secrets, Proxmox with the installer ISO, Komodo on km01, and Semaphore on ci01. Follow these pages once, in order, when the fleet is built from nothing.
At the end you have two running hosts, km01 and ci01, and a Semaphore that builds every other host from one form. Most of the work is commands in a shell on your own machine. Two pages are done in a web interface, Komodo's and Semaphore's. This is the stretch of the guide with the most manual steps and the most new tools, and each page links a tool to its primer where you first meet it.
Status: written, not yet run. See what is not yet confirmed.
Before you start¶
The pages assume a few things exist that the fleet does not build. Gather them first, since some take a day to arrive.
| You need | Used for | First needed |
|---|---|---|
| A server with Proxmox VE installed, and its root password | Running every VM | Proxmox and the installer ISO |
| Two VLANs on your router and switch, Servers and DMZ, each with a subnet and a gateway, and both carried to the Proxmox host. See What to set up before the build | The internal network and the DMZ | Proxmox and the installer ISO |
| A GitHub account, with an SSH key that can push to it | The private repo | The control shell |
| A password manager | The keys and secrets that outlive the shell | The control shell |
| A domain whose DNS is on Cloudflare, and the right to create an API token for it | Certificates from Let's Encrypt, and public hostnames | Setting up Komodo |
| An account at an SMTP relay, such as your mail provider's | Mail the fleet's services send | Automation and monitoring (ci01) |
| A DNS server you can add records to, or your own machine's hosts file. See Names | Opening each web interface by name | The first run |
Later hosts add two more. id01 can use a free MaxMind account, and leaving bootstrap mode as written needs a UniFi console that serves the fleet's DNS.
The values in these pages¶
Every address, name, and domain on these pages is an example from one fleet, and yours differ. Replace these wherever they appear:
| On the page | Stands for |
|---|---|
172.16.7.x, VLAN 7 |
Your internal network, the Servers VLAN. Its gateway is 172.16.7.1 here |
172.16.8.x, VLAN 8 |
Your DMZ |
172.16.1.11 |
Your Proxmox host, on the MGMT VLAN |
myah-mitchell.com and home.myah-mitchell.com |
Your domain, and the subdomain for internal names |
myah-mitchell in a GitHub address of the private repo |
Your GitHub account. The three public repos stay as written |
MYMI, mm, mmadmin |
Your own short name and the accounts made from it |
The host names, such as km01 and ci01, can stay as they are.
The problem it solves¶
Komodo deploys every stack, and Komodo is itself a stack on km01. Semaphore runs every build, and Semaphore is itself a stack on ci01. Neither can build the host it lives on before it exists.
A shell on your own machine stands in for Semaphore until ci01 is up. It runs the same playbook against the same private repo, so km01 and ci01 come out the same as every host built after them.
Komodo Core is started by hand one time, on a km01 that the run has already built, and the next run hands that stack over to Komodo.
Nothing built here is temporary except a few files in the shell, which the last page deletes.
The order of the foundation¶
The diagram shows the seven steps in order, which of them the shell does in Semaphore's place, and where the shell hands over. The route lists every page and section in the order you visit them.
flowchart TD
subgraph shell[The shell on your machine, standing in for Semaphore]
s1[1. Tools, keys, and first secrets]
s2[2. Installer ISO and API token on Proxmox]
s3[3. First run of km01, then Komodo Core started by hand]
s5[5. Second run of km01, then ci01 in one run]
end
s1 --> s2
s2 --> s3
s3 --> k4[4. Komodo set up in its UI]
k4 --> s5
s5 --> u6[6. Semaphore configured in its UI]
u6 --> h7[7. The handover: the state moves to ci01, the shell is cleaned]
h7 --> sem[Semaphore on ci01 builds every later host]The shell comes first, because everything after it is a file the shell writes or a command the shell runs. The keys and the secrets are made there, before any host, because the installer ISO is built from the fleet's file and that file holds the keys.
Proxmox comes next. The run creates a VM from the ISO, so the ISO has to be on the Proxmox host, and the token has to exist, before the first run.
km01 is built in two runs with Komodo's setup between them. The first run stops before the Komodo stage, because there is no Komodo to hand the stacks to. Core is started by hand, set up, and the second run hands the stacks over. ci01 follows in one run. The first run's page has a background block on why, under Create km01.
Semaphore is set up last, and the handover moves the state to it. Until then the shell is the control node.
The steps¶
| Step | Page | Done from |
|---|---|---|
| 1. Set the shell up as a control node, with the keys and the secrets | The control shell | The shell |
| 2. Describe Proxmox, build the installer ISO, and create an API token | Proxmox and the installer ISO | The shell and the Proxmox host |
| 3. Create km01 and start Komodo Core | The first run | The shell and km01 |
| 4. Set Komodo up | Setting up Komodo | Komodo's UI |
| 5. Run km01 in full, then build ci01 | The first run | The shell |
| 6. Configure Semaphore | The Semaphore project | Semaphore's UI |
| 7. Move the state, prove Semaphore, clean the shell | The handover | The shell and Semaphore |
Every host after that is one run of the site Template. See Fleet bootstrap for the order.
The route¶
The seven steps cross between pages more often than the table above shows, because km01 and ci01 have pages of their own. This table is the whole foundation in the order it is done, one row for each stop. Keep it open in a second tab, and find your row whenever a page sends you somewhere else.
| Stop | Go to | What you do there | Then |
|---|---|---|---|
| 1 | The control shell | Steps 1 to 8: the tools, the checkouts, the keys, the environment file, and the first secrets | Stop 2 |
| 2 | Describe the Proxmox host | Step 1, then the first sentence of step 2 | Stop 3 |
| 3 | km01's inventory entry | Copy km01's entry and the group's vars into hosts.yml. Stop at The VM entry |
Stop 4 |
| 4 | Write the fleet's file | The rest of step 2, then steps 3 to 5: the ISO, the API token, and the token in the shell | Stop 5 |
| 5 | Describe km01 | The first sentence of step 1 | Stop 6 |
| 6 | km01's VM entry | Copy the VM entry into opentofu/prod.tfvars |
Stop 7 |
| 7 | Describe km01 | The rest of step 1, then steps 2 and 3: the run of km01 without Komodo, and Core started by hand | Stop 8 |
| 8 | Setting up Komodo | Steps 1 to 8, in Komodo's UI | Stop 9 |
| 9 | Run km01 in full | Step 5: the commit and the run of km01 in full. Then read step 6, which holds the command for stop 12 | Stop 10 |
| 10 | Describe ci01 | Step 1: ci01's entries and its generated files | Stop 11 |
| 11 | Semaphore's values, then VictoriaMetrics', then core infrastructure's | Create the fourteen values ci01's stacks read | Stop 12 |
| 12 | Check the values | The check, then step 3 with the command from Build ci01, then the start of step 4 | Stop 13 |
| 13 | Verify Semaphore, then VictoriaMetrics, then core infrastructure | Check each stack | Stop 14 |
| 14 | Semaphore's sign-in and sops, then Grafana, then ntfy to Uptime Kuma | Sign in to each service and finish it. These are the rows of step 5 of ci01's page | Stop 15 |
| 15 | The Semaphore project | Steps 1 to 8, in Semaphore's UI | Stop 16 |
| 16 | The handover | Steps 1 to 5: move the state, prove Semaphore, clean the shell | Identity (id01) |
Each page says the same at the point where it sends you away: which section to go to, and which one to come back to. One more page is opened for its tables and not followed: the register of Variables and Secrets, from stop 8.
What stays by hand¶
- Installing Proxmox, and the shell's own tools.
- Starting Komodo Core the first time, in step 3.
- The setup inside Komodo and Semaphore, in steps 4 and 6.
- Each application's own first-run work, such as Authentik's first login and the step-ca key ceremony.
None of these makes one VM differ from another. Every VM is created from the same ISO by OpenTofu and built from the same flake by the same playbook.
What the shell holds¶
From step 1 until the handover, the shell holds the fleet's SSH private key, the deploy age key, the Proxmox API token, the state passphrase, Komodo's API secret, and OpenTofu's state file. Treat the machine as you would a password manager until step 7 removes them.
Together those files can log in to every host as root, create and delete VMs, and decrypt every secret in the private repo. Each page says what a key or a token is for where it has you create it.
Read first¶
The fleet at a glance shows which tool does which job, and is the place to start if the tools are new to you. How a host is built explains the run these pages start. The private repo explains the files they have you write.