Adding a host¶
This page adds one more VM to a fleet that is already built. A new host is two entries in the private repo, its SSH host keys, two generated files, and one run.
Use it when you want a host for a new job and every stack it runs exists already in fleet-stacks. It is the short form of every host page, with the steps and little of the reasoning. Applications (ap01) is the worked example: the same steps with real values, and a stack written from nothing.
The run touches no host but the new one. The one shared thing that changes is secrets/fleet.yaml, which step 3 encrypts again so the new host can read it.
Status: written, not yet run.
Prerequisites¶
- The foundation is finished, through The handover.
- A free address on the host's VLAN, and a free VMID on the Proxmox server. The VMID is the number Proxmox knows a VM by.
- The control shell, with the fleet-ansible repo, the private repo and fleet-nixos checked out next to each other, and the deploy key in
SOPS_AGE_KEYorSOPS_AGE_KEY_FILE. See The control shell.
The installer ISO on the Proxmox host serves every host, so a new host needs no new ISO.
<host> stands for the new host's name throughout.
1. Describe the host¶
In the private repo's hosts.yml, add the host to the docker_host group, with its address, its hostname, and the stacks it runs. In opentofu/prod.tfvars, add its VM inside vms, under the same name.
The first file is the inventory, which says what the host is and runs. The second is the tfvars file, which says how big its VM is and where it lives. See Describing a host for both entries and every key in them.
List the stacks the host runs once the fleet is finished. While the fleet is in bootstrap mode the generated files swap some of them, and the list stays as it is. See Bootstrap mode.
The address, the prefix length and the gateway in the tfvars entry have to match the inventory's ansible_host, network_prefix_length and network_gateway. When the entry has dns_servers, they have to match network_dns, which defaults to the gateway. The run stops when one differs.
2. Set what differs on this host¶
Skip this step when every stack runs with the values fleet-stacks gives it.
To set a key of a stack's Environment for this host, add komodo_stack_env to its entry in hosts.yml. The Environment is the list of values Komodo hands the stack when it deploys it. See Stack values for the form and the rules.
To drop a reference the host does not use, set the key to blank in the same place. See Blanking a reference.
3. Make the host's keys¶
Make the host's SSH host keys with new-host-key. From ~/src/fleet-ansible:
nix run ../fleet-nixos#new-host-key -- --fleet ../fleet-private <host>
git -C ../fleet-private status --short
The status shows one new file, secrets/host-keys/<host>.yaml, and two changed ones, .sops.yaml and secrets/fleet.yaml.
Run the command once per host. A second run keeps the keys it finds and changes nothing, so the same command at the start of the next step is harmless.
Background: why the keys are made before the host exists
An SSH host key is what a machine proves its identity with when you connect to it. Most systems make one at first boot, so the first connection has to accept a key nobody has seen.
The fleet makes the keys first and keeps them, encrypted, in the private repo. The install puts them on the host's persistent disk, so the run already knows which key the new host must answer with, and refuses any other.
The same key does a second job. The host's age key is derived from its ed25519 host key, which is how the host decrypts the fleet's secrets from its first boot. That is why the command also changes .sops.yaml and secrets/fleet.yaml: the new host joins the list of keys that file is encrypted for. See Host keys and the sops primer.
4. Generate the host's files¶
From ~/src/fleet-ansible, with the private repo and fleet-nixos checked out next to it, and the deploy key in SOPS_AGE_KEY or SOPS_AGE_KEY_FILE.
A host with no file under secrets/host-keys/ in the private repo gets its SSH host keys first:
The command writes secrets/host-keys/<host>.yaml, adds the host to .sops.yaml, and encrypts secrets/fleet.yaml again so that the host can read it. For a host that has its keys already, it changes nothing. See Host keys.
Then write the NixOS files and the Komodo files:
ansible-playbook -i ../fleet-private/hosts.yml nixos-sync.yml
ansible-playbook -i ../fleet-private/hosts.yml komodo-sync.yml
Add what changed, and read the diff before it is committed:
git -C ../fleet-private add hosts.yml group_vars/ opentofu/ komodo/ nixos/ secrets/ .sops.yaml
git -C ../fleet-private diff --cached
The diff of nixos/hosts/<host>.json shows the host's network, its features, and the folders, seed files and ports its stacks need. The diff of komodo/stacks/<host>.toml shows the Stacks the next run deploys and every value in their Environment. The files under secrets/ are encrypted, so their diff shows that a value changed and not what it changed to.
Then commit and push:
The run builds the host from the files git tracks in the private repo, and Komodo reads the pushed copy. The run stops at a host whose committed files differ from what the inventory gives.
The diff shows two new files beside the host's keys: nixos/hosts/<host>.json, and komodo/stacks/<host>.toml with a [[stack]] block for each Stack the run deploys.
5. Stage the values¶
Each stack's page lists the Variables and Secrets it reads. See Stacks.
Create the ones that do not exist yet, in Komodo, before the run. See Creating one for the clicks, and Variables and Secrets for what each one holds.
The deploy does not stop for a missing one. A reference with nothing behind it reaches the container as literal text.
6. Run the build¶
In Semaphore, run the site Template with Target set to the host's name.
From ~/src/fleet-ansible, in a shell prepared for runs after the handover. See Running from a shell again.
The run creates the VM, installs NixOS on it, deploys its configuration, and has Komodo deploy its Stacks. See How a host is built for each stage.
Running it again is safe. A stage that finds its work done changes nothing, so after fixing a failure, start the same run again.
7. Verify¶
The run ends with ansible's recap, one line of counts for the host. failed=0 and unreachable=0 mean every stage finished, and the last stage only finishes once Komodo reports every one of the host's Stacks as running.
In Komodo's UI, open Resources > Servers. The host shows as connected.
Open Resources > Stacks and filter by the host's Server. Each Stack shows as Running.
If the run stopped instead, its last message names the stage and what it stopped at. Fix the cause and run it again. Every stage is safe to repeat.
Each stack's page says how to check the stack itself.
What's next¶
Give the host's names a DNS record, or an entry in your own hosts file, before opening a web interface on it.
A host added while the fleet is in bootstrap mode leaves it with the rest. See Leaving bootstrap mode.
A host seldom stays as it was built. These pages cover what comes after.
| To | Read |
|---|---|
| Put another stack that exists on the host | Adding a stack to a host |
| Write a stack of your own | Writing a new stack |
| See both done with real values | Applications (ap01) |
Not yet confirmed¶
No host has been added by these steps. The commands were run against a copy of the private repo with placeholder secrets, and every host in it evaluates. Confirm these on the first real host and correct this page.
- The run from a blank VM to a deployed host, with the installer answering at the host's address.
new-host-keyin a private repo whose.sops.yamlalready names other hosts, followed by a run of every host that sharessecrets/fleet.yaml.