Certificates (pk01)¶
pk01 runs step-ca, the fleet's internal certificate authority (CA). A CA signs certificates for the fleet's own names, and a machine that trusts the CA's root certificate accepts every certificate it signs. pk01 is built now so that the CA's root key is made, backed up, and taken offline before anything depends on it.
At the end of this page the CA is running, its root key exists only in two encrypted backups that you hold, and every host trusts its root certificate.
Its own stack is step-ca-server, the smallest in the fleet: one service and no database. pk01 is also the one host built in two runs. The CA reads its password from a file on the host, and the file has to be there before the first start, so the first run stops short of the deploy.
Nothing in the fleet asks this CA for a certificate yet. Traefik gets its certificates from Let's Encrypt, and the hosts' SSH keys are static. See Certificates from Let's Encrypt.
Status: written, not yet run.
Prerequisites¶
- The foundation is finished, through The handover.
- Two durable places to keep the root key's backup, physically apart from each other, such as an encrypted USB stick at home and a second one elsewhere.
- A machine outside the fleet with
ageand thestepcommand line installed, to test the backup on. - Time to go from step 5 to step 7 in one sitting. The root key sits on a running host between the two.
1. Describe the host¶
In the private repo's hosts.yml, add pk01 to the docker_host group:
pk01:
ansible_host: 172.16.7.141
serverHostname: "pk01"
komodo_stacks_manage: false
docker_stacks:
- system-agent
- traefik-agent
- step-ca-server
komodo_stack_env:
step-ca-server:
STEPCA_CA_NAME: "MM"
The list above is what the host runs once the fleet has left bootstrap mode. While docker_stacks_bootstrap: true is set, the run deploys the host without system-agent and traefik-agent, and with traefik-bootstrap in their place. nixos-sync.yml and komodo-sync.yml make that swap when they write the host's files, so the list itself never changes. See Bootstrap mode.
The first three keys are the ones every host has. See km01's entry for those. Two keys are new on this page.
komodo_stacks_manage: false turns off the run's last stage for this host, the one where Komodo deploys the stacks. The first run then builds pk01 and deploys nothing. Step 5 takes the line out again.
komodo_stack_env sets a value in a stack's environment for this host alone. See Stack values.
STEPCA_CA_NAME is the name the CA is created under, and it shows in every certificate the CA issues. Set it to your organisation's abbreviation. step-ca adds a space and Root CA or Intermediate CA to it, so MM gives a root certificate named MM Root CA and an intermediate named MM Intermediate CA. Choose it now, since changing it later means a new CA and new certificates everywhere.
In opentofu/prod.tfvars, add its VM inside vms:
pk01 = {
server = "vh01"
vm_id = 7141
cores = 2
memory_mb = 2048
vlan_id = 7
ipv4_address = "172.16.7.141/24"
ipv4_gateway = "172.16.7.1"
dns_servers = ["172.16.7.1"]
tags = ["docker"]
extra_disks = {
persist = { interface = "scsi2", size_gb = 10 }
}
}
The fields are the same as for every VM, and the OpenTofu primer explains them. Only the name, the VMID, the address, and the sizes differ from km01's.
One service with no database needs little, so pk01 is the smallest VM in the fleet. The CA's whole state is its configuration, its keys, and its record of what it issued, and 10 GB holds them.
pk01 is on the internal VLAN, and nothing publishes it to the internet. A CA for internal names has no reason to answer from outside.
Then generate pk01's files: its SSH host keys, its NixOS file, and its Komodo file.
From ~/src/fleet-ansible, with the private repo and fleet-nixos checked out next to it, and the deploy key in SOPS_AGE_KEY or SOPS_AGE_KEY_FILE.
A host with no file under secrets/host-keys/ in the private repo gets its SSH host keys first:
The command writes secrets/host-keys/<host>.yaml, adds the host to .sops.yaml, and encrypts secrets/fleet.yaml again so that the host can read it. For a host that has its keys already, it changes nothing. See Host keys.
Then write the NixOS files and the Komodo files:
ansible-playbook -i ../fleet-private/hosts.yml nixos-sync.yml
ansible-playbook -i ../fleet-private/hosts.yml komodo-sync.yml
Add what changed, and read the diff before it is committed:
git -C ../fleet-private add hosts.yml group_vars/ opentofu/ komodo/ nixos/ secrets/ .sops.yaml
git -C ../fleet-private diff --cached
The diff of nixos/hosts/<host>.json shows the host's network, its features, and the folders, seed files and ports its stacks need. The diff of komodo/stacks/<host>.toml shows the Stacks the next run deploys and every value in their Environment. The files under secrets/ are encrypted, so their diff shows that a value changed and not what it changed to.
Then commit and push:
The run builds the host from the files git tracks in the private repo, and Komodo reads the pushed copy. The run stops at a host whose committed files differ from what the inventory gives.
2. Stage the values¶
step-ca-server reads no Variable or Secret of its own, so there is nothing to create in Komodo.
The one secret it needs is the CA password, which is a file on pk01 and never passes through Komodo. Step 4 creates it.
3. Run the build¶
In Semaphore, run the site Template with Target set to pk01.
From ~/src/fleet-ansible, in a shell prepared for runs after the handover. See Running from a shell again.
The run creates the VM, installs NixOS on it, and deploys its configuration, which makes the folders its stacks need. It ends there, with no Stack deployed.
The recap line for pk01 shows failed=0 and unreachable=0. In Komodo's UI, under Resources > Servers, pk01 shows as connected, and no Stack is attached to it.
Manual steps, instead of site.yml
Create the VM by hand¶
The run's first stage creates the VM through OpenTofu, from the host's entry in opentofu/prod.tfvars. By hand, on the Proxmox host, with the values from that same entry:
qm create <vmid> --name <host> --ostype l26 \
--cpu x86-64-v2-AES --cores <cores> --numa 1 \
--memory <memory> --balloon 0 \
--scsihw virtio-scsi-single \
--scsi0 <vm-storage>:<os-size>,discard=on,ssd=1,iothread=1 \
--scsi1 <vm-storage>:<docker-size>,discard=on,ssd=1,iothread=1 \
--scsi2 <vm-storage>:<persist-size>,discard=on,ssd=1,iothread=1 \
--ide2 local:iso/fleet-nixos-installer.iso,media=cdrom \
--ide0 <vm-storage>:cloudinit \
--ipconfig0 ip=<address>,gw=<gateway> \
--nameserver <dns-server> \
--net0 virtio,bridge=vmbr0,tag=<vlan-id> \
--boot 'order=scsi0;ide2' \
--serial0 socket --vga serial0 \
--agent enabled=1 --onboot 1 \
--tags 'nixos;<tags>'
qm start <vmid>
| Placeholder | Value |
|---|---|
<vmid> |
The entry's vm_id |
<host> |
The entry's name, which is the host's name in the inventory |
<cores>, <memory> |
The entry's cores and memory_mb |
<vm-storage> |
The Proxmox storage that holds VM disks, local-zfs unless the server's entry sets datastore_id |
<os-size>, <docker-size> |
The entry's os_disk_gb and docker_disk_gb in GB, 20 and 40 when it sets neither |
<persist-size> |
The persist disk's size_gb |
<address>, <gateway> |
The entry's ipv4_address, with its prefix length, and ipv4_gateway |
<dns-server> |
The entry's dns_servers, separated by spaces inside one pair of quotes when there are several |
<vlan-id> |
The entry's vlan_id |
<tags> |
The entry's tags, separated by semicolons |
An entry with no dns_servers takes no --nameserver option, and one with no vlan_id takes no ,tag= part.
The three disks are blank. The VM finds nothing to boot on scsi0, so it boots the installer ISO on ide2, and the installer takes its address from the cloud-init drive on ide0.
The ISO has to be on the Proxmox host first. See The installer ISO.
Create all three disks before the first start. The install partitions scsi0 and scsi1 and formats scsi2, and it stops when one is missing.
A VM made this way is not in OpenTofu's state, and it does not carry the tofu tag that marks the VMs OpenTofu manages. Leave its entry out of opentofu/prod.tfvars, or a later run tries to create a second VM under the same name and address.
Install NixOS without the rest of the run¶
The run's second and third stages wait for the VM's SSH port, install NixOS when the VM answers as the installer, and deploy the host's configuration. The same three commands of the flake do it from a shell.
From ~/src/fleet-ansible, with the host's files committed in the private repo, the deploy key in SOPS_AGE_KEY or SOPS_AGE_KEY_FILE, and the fleet's SSH key in ~/.ssh/config. See The fleet's SSH key.
Ask the VM what it runs:
It prints installer for a VM that booted the ISO. Then install:
Warning
The install wipes scsi0 and scsi1. It formats scsi2 only when the disk is blank, and it refuses any machine that is not the installer.
The last line reads install-host: <host> is installed and is rebooting. Repeat the host-state command until it prints installed, then deploy the configuration:
<host> is the host's name in the inventory, and <address> is its ansible_host. The configuration is the whole base build: accounts, SSH, the firewall, Docker, the persistent disk, Periphery, Node Exporter, and the folders, seed files and ports of every stack in the host's file. See The commands.
For traefik-bootstrap:
A host deployed with the stack in its file has these folders already. The commands make them on a host whose file does not list the stack.
sudo install -d -o $USER -g 101000 -m 0750 \
/opt/docker/logs/traefik /opt/docker/volumes/traefik
sudo install -d -o 101000 -g 101000 -m 0755 /opt/docker/logs/traefik/traefik
sudo install -d -o 101000 -g 101000 /opt/docker/volumes/traefik/traefik-certs
sudo install -d -o 101000 -g 101000 /opt/docker/volumes/traefik/traefik-plugins
No command opens the stack's ports, because the firewall changes only through the host's configuration. The ports below are open once the host's file, nixos/hosts/<host>.json, holds them: nixos-sync.yml writes that file in the private repo, and a deploy of the host applies it. See Host layout.
| Port | Allowed from |
|---|---|
80/tcp |
Anywhere |
443/tcp |
Anywhere |
8443/tcp |
Anywhere |
For step-ca-server:
A host deployed with the stack in its file has these folders already. The commands make them on a host whose file does not list the stack.
4. Create the CA password¶
Log in to pk01 as the admin account and write 32 random bytes to the password file:
head -c32 /dev/urandom | base64 \
| sudo tee /opt/docker/volumes/step-ca/step-ca-secrets/password > /dev/null
sudo chown 101000:101000 /opt/docker/volumes/step-ca/step-ca-secrets/password
sudo chmod 600 /opt/docker/volumes/step-ca/step-ca-secrets/password
Check the file and its owner:
The listing shows one file, password, owned by 101000 with mode -rw-------. 101000 is the container's own user as the host sees it. See UID offsets.
Copy the password into your password manager, and keep a second copy on paper or on a disk of its own, beside the root key's backup:
Warning
This password encrypts the root key and the intermediate key. Once the root key is offline, a lost password means the backup can never be opened.
The file is on the host because the container mounts it read-only from there. When the file is missing, Docker puts an empty folder in its place, and step-ca fails to start.
5. Deploy the CA¶
In hosts.yml, remove the komodo_stacks_manage: false line from pk01. Commit and push the change:
git -C ../fleet-private add hosts.yml
git -C ../fleet-private commit -m "Deploy pk01's stacks"
git -C ../fleet-private push
komodo_stacks_manage is in neither of pk01's generated files, so there is nothing to generate again.
Run the build a second time, the same way as in step 3. This time the run has Komodo deploy two Stacks: traefik-bootstrap-pk01 and step-ca-server.
On its first start step-ca creates a root key, an intermediate key, and the SSH certificate authority's keys, all encrypted with the password from step 4. It also turns on its ACME endpoint, the interface a program such as Traefik uses to ask for a certificate unattended.
Manual steps, instead of site.yml
Deploy the Stacks by hand¶
The run's last stage has Komodo read the host's Stacks from komodo/stacks/<host>.toml in the private repo and deploy them. To create and deploy one in Komodo's UI instead, follow Deploying a stack by hand.
A Stack made by hand under the name the run would give it is taken over by the next run, not duplicated.
6. Verify¶
The run ends with ansible's recap, one line of counts for the host. failed=0 and unreachable=0 mean every stage finished, and the last stage only finishes once Komodo reports every one of the host's Stacks as running.
In Komodo's UI, open Resources > Servers. The host shows as connected.
Open Resources > Stacks and filter by the host's Server. Each Stack shows as Running.
If the run stopped instead, its last message names the stage and what it stopped at. Fix the cause and run it again. Every stage is safe to repeat.
The step-ca-server Stack has one service:
On pk01, ask the CA for its health:
The command prints ok. It is the same check the container's own health check runs.
Ask again through Traefik, from a machine on the internal network:
The reply is {"status":"ok"}. The name needs a DNS record pointing at pk01, or an entry in your own hosts file. -k is needed while pk01 is in bootstrap mode, because Traefik serves its own self-signed certificate.
The CA's route never asks for a sign-in, in either mode. step-ca checks each request itself, and the clients that renew certificates unattended could not pass a sign-in page.
7. Take the root key offline¶
Move the root key off pk01 in one sitting, through the five parts below: copy it out, encrypt it, store two copies, remove it from the host, and test a copy.
The root key and the intermediate key are both on pk01 now. The CA only needs the intermediate to issue certificates. The root signs a new intermediate when one is needed, so it belongs somewhere no running host can reach.
Background: why the CA has two keys, and what losing each one costs
Every host is told to trust one thing, the root certificate. A certificate is accepted when a chain of signatures leads from it back to that root, so whoever holds the root key can make a certificate for any name, and the whole fleet accepts it. The only cure for a stolen root key is a new CA, with a new root certificate on every host.
The root key is therefore used for one job. It signs the intermediate's certificate, and is then put away. The intermediate key does the daily signing, so it has to stay on pk01, where the password file unlocks it each time the CA starts.
That split limits what a break-in on pk01 costs. A stolen intermediate is replaced by signing a new one with the root key from the backup, and the root certificate on every host stays as it is. With the root key still on pk01, the same break-in would cost the whole CA.
The step-ca primer explains certificate chains in general.
Copy the key out¶
On pk01, copy the key and the root certificate out of the container into a folder only you can read:
mkdir -m 700 ~/root-key && cd ~/root-key
docker cp step-ca-step-ca:/home/step/secrets/root_ca_key .
docker cp step-ca-step-ca:/home/step/certs/root_ca.crt .
Print the root certificate's fingerprint, and store it with the backup:
Encrypt it a second time¶
Encrypt the key under a passphrase with age. pk01 has no age of its own, so run it out of nixpkgs:
The command downloads age the first time it runs, and installs nothing on pk01.
age asks for the passphrase twice. Use a new passphrase, different from the CA password, and write it down somewhere that does not depend on the fleet being up.
The key file is already encrypted under the CA password. With a passphrase of its own on top, a backup stays closed to someone who finds it together with only one of the two.
Store two copies¶
Copy root_ca_key.age and root_ca.crt to both places from the prerequisites. Either copy alone is enough to recover from.
root_ca.crt is public. Keep a copy within reach as well, for step 8.
Remove it from pk01¶
Delete the working copies, then the key inside the container:
shred -u ~/root-key/root_ca_key ~/root-key/root_ca_key.age
docker exec step-ca-step-ca rm /home/step/secrets/root_ca_key
Restart the CA and check that it runs on the intermediate alone:
docker restart step-ca-step-ca
docker exec step-ca-step-ca step ca health --ca-url https://127.0.0.1:9000
The command prints ok.
Test the backup¶
On the machine outside the fleet, with one of the two copies, decrypt the key and sign a throwaway intermediate with it. Signing an intermediate is the one job the root key has, so this is the test that counts:
age -d -o root_ca_key root_ca_key.age
step certificate create "Test intermediate" test.crt test.key \
--ca ./root_ca.crt --ca-key ./root_ca_key --profile intermediate-ca
age asks for the passphrase, and step asks for the CA password, then for a new password for the test key. A certificate written to test.crt proves the backup, the passphrase, and the CA password all work.
Delete what the test made:
8. Give the fleet the root certificate¶
In the private repo's group_vars/all/private.yml, add the root certificate to ca_certificates. This is what makes the hosts accept certificates the CA signs:
ca_certificates:
- name: Home_Internal_CA
content: |
-----BEGIN CERTIFICATE-----
...
-----END CERTIFICATE-----
Paste the contents of root_ca.crt in place of the three dots.
The certificate reaches the hosts through nixos/fleet.json. From ~/src/fleet-ansible, write that file again, then commit and push:
ansible-playbook -i ../fleet-private/hosts.yml nixos-sync.yml
git -C ../fleet-private add group_vars/all/private.yml nixos/
git -C ../fleet-private commit -m "Trust the internal CA"
git -C ../fleet-private push
A host has the certificate in its own trust store from its next deploy. Run each host that is already built one more time, as in step 3 with that host as the target.
A host built from here on has the certificate from its first boot. See After a change.
A trust store is the list of root certificates the programs on a machine accept. This step covers the programs running on the host. A container has a trust store of its own and is not changed.
What's next¶
Build tf01, the Traefik hub. See Traefik hub (tf01).
Two later pieces of work depend on this CA, and neither is part of the bootstrap: a Traefik resolver that gets certificates for internal names from it, and SSH certificates in place of the fleet's static key. For the key, see Leaving bootstrap mode.
Not yet confirmed¶
- The whole page. pk01 has not been built by the run.
- The first start. step-ca's image has not been started with the password mounted read-only at the path the image keeps its own copy in. If the container stops once, after creating the CA, and then restarts cleanly, that is the cause.
- The names of the two certificates. That step-ca adds a space and
Root CAorIntermediate CAtoSTEPCA_CA_NAMEis taken from fleet-stacks, and no CA has been created with the valueMM. - The run with
komodo_stacks_manage: false. The role skips its tasks when the value is false, and that has been read in the code and not run. agethroughnix shellon pk01. The command needs pk01 to reach the NixOS binary cache, and it has not been run on a host.- The root certificate in a host's trust store. The list reaches the host's configuration through
nixos/fleet.json, which has been evaluated and not deployed. - The ACME endpoint and the SSH certificate authority. Both are turned on at first start, and nothing has asked either for a certificate.
- The root certificate inside containers. The trust step reaches the host only.