Rebuilding a VM¶

A rebuild installs a host again from its configuration and keeps the host's data. It is how a host gets a clean operating system.

Use it when a host's operating system is damaged or in doubt and going back a generation does not help. For a bad update alone, see Rolling back, which is quicker and wipes nothing.

The host is down from the reset until the run finishes, and every container image is pulled again. The data is on a disk the install leaves alone, and a snapshot of that disk, taken in step 1, is the way back if it does not.

Part	Done by
Stopping the containers and taking the snapshot	You, on the host and on the Proxmox host
Sending the host back to the installer	You, with `reset-host` in a shell
Installing NixOS and deploying the host's configuration	The run
Deploying the host's Stacks	The run, through Komodo

km01 and ci01 each need more. Read Rebuilding km01 or Rebuilding ci01 before step 1 for those two. When the VM itself is what is broken, see Replacing the VM. For more than one host, see Order and pacing.

Status: written, not yet run.

What a rebuild keeps¶

The VM stays as it is: the install wipes the root disk and the Docker disk, and it leaves the persistent disk alone. See Three disks.

The diagram shows the three disks, and where what is on each comes from after a rebuild.

flowchart LR
  repo["The private repo and the flake"]
  komodo["Komodo Core on km01"]
  subgraph vm["The VM, which stays"]
    os["Root disk, scsi0. Wiped and installed again"]
    docker["Docker disk, scsi1. Wiped and left empty"]
    persist["Persistent disk, scsi2. Kept"]
  end
  repo -->|the host's configuration| os
  repo -->|the SSH host keys, written again| persist
  komodo -->|the Stacks, with every image pulled again| docker
  persist -->|Periphery's key, so the same Server| komodo

The host keeps its identity. Its SSH host keys come from the private repo's secrets and sit on the persistent disk, so the host answers with the keys it had. Periphery's key is on the same disk, so Komodo sees the Server it knew.

Each stack's data, config, and logs are on the persistent disk too, so a Stack that is deployed again finds them where they were.

Prerequisites¶

The control shell, with the fleet's SSH key in ~/.ssh/config, as in Create the fleet's SSH key. The shell is needed because reset-host has no Template in Semaphore. See The control shell.
The host's files in the private repo are generated, committed and pushed. See After a change.
A run of the host with Dry Run ticked in Semaphore, or with --check on the command line, ends with failed=0. This is check mode. It stops at files that are out of date and at a configuration that does not evaluate, while the host is still up, and does not reach the host.
The host is healthy and has a recent backup.
No container version changes in the same session. A newer version that has already written to the data can stop a rollback from being clean.

Placeholders¶

Placeholder	Value
`<host>`	The host being rebuilt, such as `id01`
`<address>`	The host's address, `ansible_host` in the inventory
`<vmid>`	The VM's ID, `vm_id` in its tfvars entry
`<volume>`	The storage and volume name of the VM's `scsi2` disk, found in step 1
`<dataset>`	The ZFS dataset behind that volume, found in step 1
`<admin>`	The admin account, `abbr_name` followed by `admin`, such as `mmadmin`
`<os-volume>`	The storage and volume name at the start of the VM's `scsi0` line
`<old-vmid>`	The VMID of the VM being replaced
`<new-vmid>`	A free VMID for the replacement VM

1. Snapshot the persistent disk¶

Stop the host's containers, then take a snapshot of the persistent disk on the Proxmox host. The snapshot is the point a rollback can return to, and the containers stop first so that the data is at rest when it is taken and when the host goes down.

Log in to the host as the admin account:

ssh <admin>@<address>

docker ps -q | xargs -r docker stop
sync
docker ps

The last list is empty. Periphery is a service of the host and not a container, so Komodo still shows the Server as connected.

On the Proxmox host, find the disk's volume, then its dataset:

qm config <vmid> | grep '^scsi2'
pvesm path <volume>

<volume> is the storage and volume name at the start of the scsi2 line, such as local-zfs:vm-7131-disk-2. The second command prints /dev/zvol/<dataset>.

zfs snapshot <dataset>@pre-rebuild
zfs list -t snapshot <dataset>

The list shows <dataset>@pre-rebuild. A ZFS snapshot records the disk as it is at that moment, and it is made at once. It is kept on the Proxmox host, outside the VM, so nothing the install does can reach it.

2. Send the host back to the installer¶

Warning

The command ends the host's operating system. From here the host is down until the run in step 3 finishes, and everything on its root disk and its Docker disk is lost.

From ~/src/fleet-ansible:

nix run ../fleet-nixos#reset-host -- --yes-wipe <host> <address>

The command reads the name the host gives itself in /etc/fleet-host, and wipes nothing when it is not <host>. Then it overwrites the start of the root disk and restarts the VM, which finds nothing to boot on that disk and boots the installer ISO. The last line reads:

reset-host: wiped the start of the OS disk of <host>, which now reboots into the installer

Ask the VM what it runs, and repeat the command until it prints installer:

nix run ../fleet-nixos#host-state -- <address>

3. Run the host¶

SemaphoreCommand line

In Semaphore, run the site Template with Target set to the host's name.

From ~/src/fleet-ansible, in a shell prepared for runs after the handover. See Running from a shell again.

ansible-playbook -i ../fleet-private/hosts.yml site.yml \
  -e target=<host>

Stage	What it does on a host that was reset
Create the VM	Nothing. The VM is the one OpenTofu knows
Wait	Finds the installer's SSH port open
NixOS	Finds the installer, and installs. The install makes the root disk and the Docker disk anew, keeps the filesystem on the persistent disk, installs NixOS, and restarts the VM. Then the stage deploys the host's configuration
Deploy the stacks	Has Komodo deploy each of the host's Stacks. The Docker disk is empty, so every image is pulled again

The command is the one that built the host, with nothing added for a rebuild. The host comes back as itself. See What a rebuild keeps.

4. Verify¶

The run ends with ansible's recap, one line of counts for the host. failed=0 and unreachable=0 mean every stage finished, and the last stage only finishes once Komodo reports every one of the host's Stacks as running.

In Komodo's UI, open Resources > Servers. The host shows as connected.

Open Resources > Stacks and filter by the host's Server. Each Stack shows as Running.

If the run stopped instead, its last message names the stage and what it stopped at. Fix the cause and run it again. Every stage is safe to repeat.

Log in to the host and check that the data is where it was:

ssh <admin>@<address>

findmnt /srv/persist
sudo ls /opt/docker/volumes /opt/docker/logs
sudo ls /srv/persist/host/komodo /srv/persist/host/ssh

The disk is mounted, both folders hold the project folders the host had, and the last command lists periphery.key and the SSH host keys.

In Komodo, the host is the Server it was before, with the same Stacks. If the Server stays down, read Periphery's log on the host:

sudo journalctl -u komodo-periphery -n 50

Check each service the way its stack's page does. See Stacks.

5. Remove the snapshot¶

Wait until you trust the host. A few days is reasonable. Then, on the Proxmox host:

zfs destroy <dataset>@pre-rebuild

The snapshot holds every block it covers on the pool until it is destroyed, so do not leave it there. As the host's data changes, the snapshot's share of the pool grows.

Rolling back¶

A rebuild has no old operating system to return to. The host's system is whatever its configuration builds, so a rollback is about the data and about the configuration.

To have the data as it was before the rebuild, return the persistent disk to the snapshot. The command discards everything written to the disk since. On the Proxmox host:

qm shutdown <vmid>
zfs rollback <dataset>@pre-rebuild
qm start <vmid>

Then run the host again as in step 3, so that Komodo deploys the Stacks on the data that came back.

When the rebuilt host is wrong because of a change to its configuration, undo the change where it was made: revert the commit in the private repo or in fleet-nixos, generate the host's files again, and run the host. See Rolling back.

A host that does not answer¶

reset-host needs a host that answers over SSH. For one that does not, do from the Proxmox host what the command does from inside. Find the volume of the root disk and stop the VM:

qm config <vmid> | grep '^scsi0'
qm stop <vmid>

Warning

The next command destroys the start of a disk. Run it against the volume on the scsi0 line of this VM and no other.

dd if=/dev/zero of=$(pvesm path <os-volume>) bs=1M count=16 conv=fsync
qm start <vmid>

<os-volume> is the storage and volume name at the start of the scsi0 line. The VM boots the installer, and step 3 installs it. qm stop is a power cut for the containers, so expect each database to recover as it would from one.

Replacing the VM¶

This section is for a VM whose hardware definition is what is broken, where an install on the same VM would not help. OpenTofu creates a new VM under the host's name, the persistent disk moves to it, and the run installs it. The steps in between are by hand, because OpenTofu refuses to destroy a VM and knows nothing about moving a disk.

It needs a shell prepared for runs after the handover, with the tunnel to the state database open, and a free VMID for the new VM. See Running from a shell again. <old-vmid> and <new-vmid> stand for the two VMs.

Step	Why
Read the state	To work with OpenTofu by hand, outside the run
Shut the old VM down	So its disk is at rest, and its name and address are free
Take the old VM out of the state	So OpenTofu creates a VM for the host again
Describe and create the new VM	A new VM, left off
Move the persistent disk	The data goes to the new VM before its first boot
Start the new VM and run the host	The run installs it as it does any new host
Remove the old VM	Only when the new one is trusted

OpenTofu's state is its record of what it has created. It decides what to do by comparing that record with the tfvars file, which is why two of the steps edit one or the other.

Read the state¶

Make a fresh checkout of the fleet-opentofu repo, give it the tfvars file, and list what the state holds:

rm -rf /tmp/ansible-fleet-opentofu-checkout
git clone --depth 1 https://github.com/myah-mitchell/fleet-opentofu \
  /tmp/ansible-fleet-opentofu-checkout
cd /tmp/ansible-fleet-opentofu-checkout/envs/prod
cp ~/src/fleet-private/opentofu/prod.tfvars private.auto.tfvars
tofu init
tofu state list

The list has one line for each VM, and the host's is among them:

module.vm["<host>"].proxmox_virtual_environment_vm.this

Shut the old VM down¶

On the Proxmox host:

qm shutdown <old-vmid>
qm set <old-vmid> --name <host>-old --onboot 0
qm status <old-vmid>

The answer is status: stopped. The containers stop with the VM and the persistent disk unmounts cleanly, which is what makes it safe to move. Take the snapshot now, with the commands step 1 runs on the Proxmox host and <old-vmid> as the VM.

The new name keeps the two VMs apart, because OpenTofu looks up where a VM runs by its name. Turning off the start at boot keeps a restart of the Proxmox host from starting the old VM on the address the new one uses.

Take the old VM out of the state¶

Remove the host's entry from the state. While the state holds the old VM, OpenTofu sees the host as built and creates nothing.

Removing the entry changes nothing in Proxmox. It only makes OpenTofu forget the VM. In the shell, in the folder of the checkout:

tofu state rm 'module.vm["<host>"].proxmox_virtual_environment_vm.this'
tofu state list

The host's line is gone from the list, and every other VM's line is still there.

Describe the new VM¶

In the private repo's opentofu/prod.tfvars, change two lines of the host's entry. Set vm_id to the new VMID, and add started = false:

  id01 = {
    server       = "vh01"
    vm_id        = 9131
    started      = false
    cores        = 4

Leave the rest as it is, and correct whatever was wrong with the old VM's definition. started = false has OpenTofu create the VM and leave it off, so the disk can be swapped before the first boot.

From ~/src/fleet-ansible, commit and push, then give the new file to the OpenTofu checkout:

git -C ../fleet-private add opentofu/prod.tfvars
git -C ../fleet-private commit -m "Replace the VM of <host>"
git -C ../fleet-private push
cp ../fleet-private/opentofu/prod.tfvars \
  /tmp/ansible-fleet-opentofu-checkout/envs/prod/private.auto.tfvars

Create the new VM¶

In the shell, in the folder of the checkout, read the plan for this one VM:

tofu plan -target='module.vm["<host>"]'

The plan ends with 1 to add, 0 to change, 0 to destroy. Anything else means the state or the tfvars file is not what this page expects, so stop and read the plan. Then apply it, and answer yes:

tofu apply -target='module.vm["<host>"]'

On the Proxmox host, check the result:

qm status <new-vmid>
qm config <new-vmid> | grep -E '^(name|scsi2)'

The VM is stopped, its name is the host's, and it has a scsi2 disk. That disk is new and blank.

Move the persistent disk¶

On the Proxmox host, take the blank disk off the new VM:

qm set <new-vmid> --delete scsi2
qm config <new-vmid> | grep '^unused'

The second command shows the detached disk, as unused0 on a VM with no other unused disk. Delete it under the name that line gives:

qm set <new-vmid> --delete unused0

Warning

The second delete destroys a volume. Check the VMID in both commands before running them. The old VM's disk is the only copy of the data outside the backup and the snapshot.

Move the old VM's disk across:

qm disk move <old-vmid> scsi2 --target-vmid <new-vmid> --target-disk scsi2
qm config <new-vmid> | grep '^scsi2'

The line carries discard=on, iothread=1, and ssd=1. If the move dropped them, set them again with the volume name that line prints:

qm set <new-vmid> --scsi2 <volume>,discard=on,ssd=1,iothread=1

Start the new VM and run the host¶

In opentofu/prod.tfvars, remove the started = false line. Commit and push, and copy the file into the OpenTofu checkout as before. Read the plan once more:

cd /tmp/ansible-fleet-opentofu-checkout/envs/prod
tofu plan -target='module.vm["<host>"]'

The plan ends with 0 to add, 1 to change, 0 to destroy, and the change is started. A plan that wants to change or add a disk has noticed the swap. Stop there, and see Not yet confirmed.

Then run the host from the Command line tab of step 3, and verify it as in step 4. The new VM's root disk is blank, so it boots the installer, and the run installs it as it does any new host.

Remove the old VM¶

Wait until you trust the new VM. Then, on the Proxmox host:

qm config <old-vmid> | grep '^scsi2' || echo "safe to destroy"

Warning

Go on only if the answer is safe to destroy. qm destroy deletes every disk the VM owns, and an old VM that still has a scsi2 line owns the data.

qm destroy <old-vmid>
zfs destroy <dataset>@pre-rebuild

After the move the disk may be under another name, so find the dataset again as in step 1, with <new-vmid> as the VM.

To give up on the new VM before that, shut it down, move the disk back with the two VMIDs swapped in the qm disk move command, and give the old VM its name and its start at boot again. Do not run the host while OpenTofu's state describes the new VM.

Replacing ci01's VM¶

Copy the state into a file in the shell before the old VM is shut down. The state database is on ci01, so it goes down with the old VM. After Read the state, with the tunnel open:

mkdir -p ~/.local/state/fleet-opentofu
cat > backend_override.tf <<EOF
terraform {
  backend "local" {
    path = "$HOME/.local/state/fleet-opentofu/prod.tfstate"
  }
}
EOF
tofu init -migrate-state -force-copy
tofu state list

The list is the same as before, read from the file this time. Close the tunnel, and add -e vms_backend=local to the ansible-playbook command, which keeps the run on the file the tofu commands use.

When ci01's Stacks run again, put the state back. Open the tunnel as The handover does, reading the container's address again since it may have changed. Then:

cd /tmp/ansible-fleet-opentofu-checkout/envs/prod
rm backend_override.tf
tofu init -migrate-state -force-copy
tofu state list

The list shows every VM. Until the state is back, do not run ci01 from Semaphore, which reads the old state.

Rebuilding km01¶

Komodo Core runs on km01, so nothing in the fleet can be deployed while km01 is down. Every other host keeps running its containers. Their Periphery agents reconnect when Core is back.

Follow steps 1 and 2 as written. Then:

Run km01 from the shell with the last stage left out, because that stage asks Core to deploy and Core is not running yet. The command is the one in step 3 with --skip-tags komodo added.
Start Core by hand, as The first run does. The file with Core's database credentials is on the persistent disk, so leave its values as they are. Core starts with the credentials its database already has.
Run km01 again with no option added. Komodo takes over Core's containers as it did on the first build, and deploys the rest of km01's Stacks.
Remove your checkout of fleet-stacks from km01.

Skip Setting up Komodo. The admin account, the keys, the Resource Sync, and every Variable and Secret are in Core's database, on the persistent disk.

Core's own keypair is under komodo-keys on the same disk, so every host still trusts it. See What to keep safe.

Rebuilding ci01¶

ci01 holds Semaphore and the database with OpenTofu's state. Both are down from the reset until ci01's Stacks run again, so the run comes from a shell and leaves OpenTofu out.

Before the reset¶

Prepare the shell while ci01 is up. It needs the fleet's SSH key, the deploy key in SOPS_AGE_KEY or SOPS_AGE_KEY_FILE, and KOMODO_API_KEY and KOMODO_API_SECRET. It needs neither the tunnel nor anything OpenTofu reads.

Do the run with --check from this shell, with the tags below, before step 1.

The run¶

Follow steps 1 and 2 as written. In step 3, name the stages to run, and give the run ci01's node and VMID:

ansible-playbook -i ../fleet-private/hosts.yml site.yml \
  -e target=ci01 \
  -e '{"vms_vm": {"node_name": "vh01", "vm_id": 7121}}' \
  --tags wait,nixos,komodo

The first stage is left out. The VM has not changed, so OpenTofu has nothing to do, and its state stays in the database on the persistent disk.

The install needs to know which VM it is installing, to read the installer's host key through Proxmox. The first stage normally notes that from OpenTofu, in a variable named vms_vm. With the stage left out, the second -e supplies it.

Use the Proxmox node the VM runs on and the vm_id from ci01's entry in opentofu/prod.tfvars. Without it, the run stops at the install and asks for the vms tag.

After the run¶

Verify ci01 as in step 4. Then prove Semaphore with a run against km01 and one against ci01, as in Prove Semaphore. The second one is the first run since the rebuild that reads the state.

Semaphore's Project, its Template, and the secrets in its Variable Group are in its database, so they are there as before.

Order and pacing¶

Rebuild one host at a time. Start with the host that costs least to lose, and finish with ci01 and then km01. A rebuild that goes wrong on a small host shows what to fix before it happens on Semaphore or on Core.

What's next¶

Clean the shell when the work is done. See Clean the shell.

Not yet confirmed¶

No host has been rebuilt by these steps. Each point below came from the code and from the tools' documentation. Confirm them on the first real rebuild and correct this page.

reset-host. That overwriting the start of the root disk and the restart that follows leave the VM in the installer has not been tried.
The restart that reset-host asks for is the kernel's own, which unmounts nothing. reset-host stops the containers and Docker and runs sync first for that reason, and the page stops the containers before the snapshot too. Whether the persistent disk then mounts without a repair has not been tried.
A snapshot of the persistent disk while the VM runs. The filesystem is mounted when the snapshot is taken, so a rollback returns to a filesystem that replays its journal at the next mount.
That the install leaves a persistent disk with a filesystem alone. install-host formats the disk only when blkid finds nothing on it.
That Periphery reconnects with its saved key, and that Core accepts the onboarding key from a Server name it already knows.
The device names the host's configuration mounts by, drive-scsi1 and drive-scsi2. Compare them with ls -l /dev/disk/by-id on a real VM.
A host that does not answer. The dd command does what reset-host does, from the Proxmox host, and has not been run.
Replacing the VM. The state records the blank disk OpenTofu created, and the VM then carries another volume under the same scsi2. The plan in Start the new VM and run the host shows whether the provider reads that as a change. If it does, the likely fix is a run of tofu apply -refresh-only for the VM, which has not been tried.
started = false on a new VM, the change to started in a later apply, and tofu state rm against the real state database.
Whether qm disk move with a target VM renames the volume without copying it, and whether it keeps discard, ssd, and iothread.
Rebuilding km01. Starting Core by hand against an existing database, and Komodo taking the containers over afterwards, follow the first build and have not been done on a rebuilt host.
Rebuilding ci01. A run with --tags wait,nixos,komodo has not been tried, and neither has Semaphore's first start on a rebuilt ci01. Passing vms_vm with -e follows from what the install reads, the VM's node_name and vm_id, and has not been tried either.
Replacing ci01's VM. Copying the state out of the database, and -force-copy replacing a state the database already holds, have not been tried.