Komodo (km01)¶

km01 runs Komodo Core, which deploys every stack in the fleet, its own included. Every other host runs Komodo's agent, Periphery, which connects to Core and does the deploying on its host. See Core and Periphery.

km01 is the first VM built, because no other host can be given its stacks until Core is there to deploy them. At the end of this page km01 is a NixOS host that Komodo manages, with Komodo's UI answering on its own hostname.

Its own stack is komodo-server. km01's first build is part of the foundation, because Komodo cannot deploy the stack it runs in before it has started. See The problem it solves. The foundation uses this page twice, both times for step 1, and runs the build from The first run. Every run after the first is the ordinary one in step 3.

Status: written, not yet run.

Prerequisites¶

For the first build, you came here from one of two steps of the foundation, and each sends you to one section of step 1. Write the fleet's file uses The inventory entry, and Describe km01 uses The VM entry.
For any later run, a km01 that Komodo already manages.

1. Describe the host¶

A host is described in two files of the private repo: the inventory, hosts.yml, and OpenTofu's variables file, opentofu/prod.tfvars.

The inventory entry¶

In hosts.yml, add km01 to the docker_host group:

    km01:
      ansible_host: 172.16.7.101
      serverHostname: "km01"
      docker_stacks:
        - system-agent
        - traefik-agent
        - komodo-server

Key	Holds
`ansible_host`	The host's address. Ansible connects to it, and the host is given it as its static address
`serverHostname`	The hostname the host is given, and the name of its Server in Komodo
`docker_stacks`	The stacks the host runs, each by its folder name under `stacks/` in the fleet-stacks repo

Every host page has an entry of this shape. The later pages explain only what they add to it.

The list above is what the host runs once the fleet has left bootstrap mode. While docker_stacks_bootstrap: true is set, the run deploys the host without system-agent and traefik-agent, and with traefik-bootstrap in their place. nixos-sync.yml and komodo-sync.yml make that swap when they write the host's files, so the list itself never changes. See Bootstrap mode.

km01 is the group's first host, so give the group its vars with it. Every later host page adds a host and leaves these as they are. The foundation adds one flag to them, docker_stacks_bootstrap: true, in Describe km01.

docker_host:
  vars:
    NIXOS: true
    network_gateway: "172.16.7.1"
    docker_stacks_internal_subnet: "172.16.7.0/24"
    FIREWALL: true
    DOCKER: true
    KOMODO: true
    NODE_EXPORTER: true

Key	Holds
`NIXOS`	Marks a host as one the run installs NixOS on
`network_gateway`	The gateway of the internal network. A host on another network sets its own
`docker_stacks_internal_subnet`	The subnet a port is opened to when a stack opens it to the internal network only
`FIREWALL`, `DOCKER`, `KOMODO`, `NODE_EXPORTER`	What every host in the group runs: the firewall, Docker, Periphery, and Node Exporter

Put together, the group reads as below. It sits at the top level of hosts.yml, beside all and pve_host. Each later host goes under hosts, at the same depth as km01.

docker_host:
  hosts:
    km01:
      ansible_host: 172.16.7.101
      serverHostname: "km01"
      docker_stacks:
        - system-agent
        - traefik-agent
        - komodo-server
  vars:
    NIXOS: true
    network_gateway: "172.16.7.1"
    docker_stacks_internal_subnet: "172.16.7.0/24"
    FIREWALL: true
    DOCKER: true
    KOMODO: true
    NODE_EXPORTER: true

See Describing a host for the keys a host can set beside these.

On the first build, this is as far as Write the fleet's file needs. Go back to it now, and leave the VM entry below for Describe km01.

The VM entry¶

In opentofu/prod.tfvars, add its VM inside vms:

  km01 = {
    server       = "vh01"
    vm_id        = 7101
    cores        = 2
    memory_mb    = 4096
    vlan_id      = 7
    ipv4_address = "172.16.7.101/24"
    ipv4_gateway = "172.16.7.1"
    dns_servers  = ["172.16.7.1"]
    tags         = ["docker"]
    extra_disks = {
      persist = { interface = "scsi2", size_gb = 20 }
    }
  }

OpenTofu creates the VM in Proxmox from this entry. The file is its tfvars file, and every host page has an entry with these fields.

Field	Holds
`server`	The Proxmox server the VM is created on, by its name under `servers` in the same file
`vm_id`	The VM's ID in Proxmox. These pages use the VLAN followed by the last part of the address in three digits
`cores`, `memory_mb`	The VM's size
`vlan_id`	The VLAN the VM's network card is on: `7` is the internal network in these pages
`ipv4_address`, `ipv4_gateway`, `dns_servers`	The network the installer starts with. The address carries its prefix length here
`tags`	Tags shown on the VM in Proxmox, added to the two OpenTofu always sets
`extra_disks`	The persistent disk, always on `scsi2`, with its size in GB

The OS disk and the Docker disk are not in the entry because their defaults, 20 GB and 40 GB, suit every host in these pages. See Describing a host for the optional fields, and The Proxmox servers for vh01.

Core, FerretDB, Postgres, and the backup container are light together, so two cores and 4 GB are enough. Their data and the dumps the backup keeps are small, and 20 GB holds them.

km01's address is long-lived. It is in komodo_core_address, which every host's Periphery dials.

Background: why the address is written in both files

The two files are read by different tools at different moments. OpenTofu reads the tfvars entry when it creates the VM, and passes the address to the installer through the VM's cloud-init drive. That is what lets the run reach a machine with nothing installed on it.

The inventory is what the installed host is built from. NixOS gives the host ansible_host as its static address, and ansible connects to the same address.

Nothing copies one file into the other, so the run compares them. It stops when the address, the prefix length, the gateway, or the nameservers differ between the two.

On the first build, go back to Describe km01 now. It sets two more values and then generates km01's files. After a later change to either entry, generate the files again. See After a change.

2. Stage the values¶

A stack takes its settings and credentials from Variables and Secrets held in Komodo, which have to exist before the run that deploys the stack. See How a stack gets its values.

komodo-server reads two Secrets, KOMODO_DB_USERNAME and KOMODO_DB_PASSWORD. They are the login Core's database was first started with, and Core has to keep presenting the same one.

They are created during the first build, from the file Core was started with. See the database Secrets.

A later run needs nothing staged.

3. Run the build¶

The first build does not use this step. Its runs are Create km01 and Run km01 in full on The first run. The tabs below are for every run after those.

SemaphoreCommand line

In Semaphore, run the site Template with Target set to km01.

From ~/src/fleet-ansible, in a shell prepared for runs after the handover. See Running from a shell again.

ansible-playbook -i ../fleet-private/hosts.yml site.yml \
  -e target=km01

When a run changes the definition of the komodo-server Stack, Komodo redeploys the containers Core runs in. The UI drops for up to a minute, and the run waits for it to come back.

Manual steps, instead of site.yml

Create the VM by hand¶

The run's first stage creates the VM through OpenTofu, from the host's entry in opentofu/prod.tfvars. By hand, on the Proxmox host, with the values from that same entry:

qm create <vmid> --name <host> --ostype l26 \
  --cpu x86-64-v2-AES --cores <cores> --numa 1 \
  --memory <memory> --balloon 0 \
  --scsihw virtio-scsi-single \
  --scsi0 <vm-storage>:<os-size>,discard=on,ssd=1,iothread=1 \
  --scsi1 <vm-storage>:<docker-size>,discard=on,ssd=1,iothread=1 \
  --scsi2 <vm-storage>:<persist-size>,discard=on,ssd=1,iothread=1 \
  --ide2 local:iso/fleet-nixos-installer.iso,media=cdrom \
  --ide0 <vm-storage>:cloudinit \
  --ipconfig0 ip=<address>,gw=<gateway> \
  --nameserver <dns-server> \
  --net0 virtio,bridge=vmbr0,tag=<vlan-id> \
  --boot 'order=scsi0;ide2' \
  --serial0 socket --vga serial0 \
  --agent enabled=1 --onboot 1 \
  --tags 'nixos;<tags>'
qm start <vmid>

Placeholder	Value
`<vmid>`	The entry's `vm_id`
`<host>`	The entry's name, which is the host's name in the inventory
`<cores>`, `<memory>`	The entry's `cores` and `memory_mb`
`<vm-storage>`	The Proxmox storage that holds VM disks, `local-zfs` unless the server's entry sets `datastore_id`
`<os-size>`, `<docker-size>`	The entry's `os_disk_gb` and `docker_disk_gb` in GB, `20` and `40` when it sets neither
`<persist-size>`	The persist disk's `size_gb`
`<address>`, `<gateway>`	The entry's `ipv4_address`, with its prefix length, and `ipv4_gateway`
`<dns-server>`	The entry's `dns_servers`, separated by spaces inside one pair of quotes when there are several
`<vlan-id>`	The entry's `vlan_id`
`<tags>`	The entry's `tags`, separated by semicolons

An entry with no dns_servers takes no --nameserver option, and one with no vlan_id takes no ,tag= part.

The three disks are blank. The VM finds nothing to boot on scsi0, so it boots the installer ISO on ide2, and the installer takes its address from the cloud-init drive on ide0.

The ISO has to be on the Proxmox host first. See The installer ISO.

Create all three disks before the first start. The install partitions scsi0 and scsi1 and formats scsi2, and it stops when one is missing.

A VM made this way is not in OpenTofu's state, and it does not carry the tofu tag that marks the VMs OpenTofu manages. Leave its entry out of opentofu/prod.tfvars, or a later run tries to create a second VM under the same name and address.

Install NixOS without the rest of the run¶

The run's second and third stages wait for the VM's SSH port, install NixOS when the VM answers as the installer, and deploy the host's configuration. The same three commands of the flake do it from a shell.

From ~/src/fleet-ansible, with the host's files committed in the private repo, the deploy key in SOPS_AGE_KEY or SOPS_AGE_KEY_FILE, and the fleet's SSH key in ~/.ssh/config. See The fleet's SSH key.

Ask the VM what it runs:

nix run ../fleet-nixos#host-state -- <address>

It prints installer for a VM that booted the ISO. Then install:

nix run ../fleet-nixos#install-host -- --fleet ../fleet-private <host> <address>

Warning

The install wipes scsi0 and scsi1. It formats scsi2 only when the disk is blank, and it refuses any machine that is not the installer.

The last line reads install-host: <host> is installed and is rebooting. Repeat the host-state command until it prints installed, then deploy the configuration:

nix run ../fleet-nixos#deploy-host -- --fleet ../fleet-private <host> <address>

<host> is the host's name in the inventory, and <address> is its ansible_host. The configuration is the whole base build: accounts, SSH, the firewall, Docker, the persistent disk, Periphery, Node Exporter, and the folders, seed files and ports of every stack in the host's file. See The commands.

For traefik-bootstrap:

A host deployed with the stack in its file has these folders already. The commands make them on a host whose file does not list the stack.

sudo install -d -o $USER -g 101000 -m 0750 \
  /opt/docker/logs/traefik /opt/docker/volumes/traefik
sudo install -d -o 101000 -g 101000 -m 0755 /opt/docker/logs/traefik/traefik
sudo install -d -o 101000 -g 101000 /opt/docker/volumes/traefik/traefik-certs
sudo install -d -o 101000 -g 101000 /opt/docker/volumes/traefik/traefik-plugins

No command opens the stack's ports, because the firewall changes only through the host's configuration. The ports below are open once the host's file, nixos/hosts/<host>.json, holds them: nixos-sync.yml writes that file in the private repo, and a deploy of the host applies it. See Host layout.

Port	Allowed from
`80/tcp`	Anywhere
`443/tcp`	Anywhere
`8443/tcp`	Anywhere

For komodo-server:

A host deployed with the stack in its file has these folders and seed files already. The commands make them on a host whose file does not list the stack.

sudo install -d -o $USER -g 101000 -m 0750 \
  /opt/docker/logs/komodo /opt/docker/volumes/komodo
sudo install -d -o 101000 -g 101000 /opt/docker/volumes/komodo/ferretdb-data
sudo install -d -o 100000 -g 100000 /opt/docker/volumes/komodo/postgres-data
sudo install -d -o 100000 -g 100000 /opt/docker/volumes/komodo/postgres-backup-data
sudo install -d -o 101000 -g 101000 /opt/docker/volumes/komodo/komodo-keys
sudo install -d -o 101000 -g 101000 /opt/docker/volumes/komodo/komodo-backups
sudo install -d -o 101000 -g 101000 /opt/docker/volumes/komodo/komodo-sync
sudo install -d -o 101000 -g 101000 /opt/docker/volumes/komodo/komodo-cache
sudo install -d -o 101000 -g 101000 -m 0700 /opt/docker/volumes/komodo/komodo-secrets

sudo test -e /opt/docker/volumes/komodo/komodo-secrets/core.config.toml || sudo curl -fsSL \
  -o /opt/docker/volumes/komodo/komodo-secrets/core.config.toml \
  https://raw.githubusercontent.com/myah-mitchell/fleet-stacks/main/containers/komodo/config/core.config.toml.example
sudo chown 101000:101000 /opt/docker/volumes/komodo/komodo-secrets/core.config.toml
sudo chmod 0600 /opt/docker/volumes/komodo/komodo-secrets/core.config.toml

No command opens the stack's ports, because the firewall changes only through the host's configuration. The ports below are open once the host's file, nixos/hosts/<host>.json, holds them: nixos-sync.yml writes that file in the private repo, and a deploy of the host applies it. See Host layout.

Port	Allowed from
`9120/tcp`	Anywhere

Start Core by hand¶

Core is the one stack Komodo cannot deploy while Core is down. Start it with Compose, as The first run does. On a km01 that has been built before, the environment file is already on the persistent disk, and build.py keeps every value that is set in it.

Deploy the Stacks by hand¶

The run's last stage has Komodo read the host's Stacks from komodo/stacks/<host>.toml in the private repo and deploy them. To create and deploy one in Komodo's UI instead, follow Deploying a stack by hand.

A Stack made by hand under the name the run would give it is taken over by the next run, not duplicated.

4. Verify¶

The run ends with ansible's recap, one line of counts for the host. failed=0 and unreachable=0 mean every stage finished, and the last stage only finishes once Komodo reports every one of the host's Stacks as running.

In Komodo's UI, open Resources > Servers. The host shows as connected.

Open Resources > Stacks and filter by the host's Server. Each Stack shows as Running.

If the run stopped instead, its last message names the stage and what it stopped at. Fix the cause and run it again. Every stage is safe to repeat.

The komodo-server Stack has four services:

ferretdb
postgres
postgres-backup
komodo

Open Komodo through Traefik, at https://komodo.km01.home.myah-mitchell.com. The name needs a DNS record pointing at km01, or an entry in your own hosts file.

Note

In bootstrap mode the browser warns about the certificate. Traefik serves its own self-signed one until the host leaves bootstrap mode, so accept the warning for now.

The direct address, http://172.16.7.101:9120, keeps working in both modes. It is the one every Periphery and the run itself use.

What to keep safe¶

Two things on km01's persistent disk cost the most to lose.

Path	Holds
`/opt/docker/volumes/komodo/komodo-keys`	Core's keypair. Without it every host has to be onboarded again
`/opt/docker/volumes/komodo/komodo-server.env`	The credentials Postgres was first started with, used to start Core by hand on a rebuilt km01

Every Variable and Secret in Komodo lives in Core's database, under postgres-data, and the backup container writes a dump of it each day to postgres-backup-data.

What's next¶

On the first build, go back to the step that sent you here: Write the fleet's file after the inventory entry, or Describe km01 after the VM entry. The route lists every stop of the foundation in order.

ci01 is the host built after km01. See Automation and monitoring (ci01).

Not yet confirmed¶

The whole page. km01 has not been built by the run.
A run that redeploys the komodo-server Stack: that Periphery finishes the deploy after Core's container stops, and that the run's wait outlasts the time Core is away. See The first run.
Starting Core by hand on a rebuilt km01, from the environment file the persistent disk kept.