DMZ edge (bh01)¶

bh01 is the fleet's edge, and the only host that carries web traffic from the internet. It reaches Cloudflare through a tunnel it opens outward, so no port is forwarded to it and no public address points at it.

Its own stack is traefik-dmz: a Traefik, a Redis that copies the one on tf01, and cloudflared, the tunnel's connector. bh01 sits on the DMZ VLAN, a network of its own for the hosts the internet can reach. It is the first host whose traffic to the rest of the fleet crosses your router's firewall.

bh01 comes after tf01 because its Redis copies tf01's, and it is the last host the fleet waits for before it leaves bootstrap mode. At the end of this page the tunnel is connected and the copy is in step with tf01. No hostname is public yet. Publishing a hostname does that, after bootstrap mode.

The page has more manual steps than the hosts before it, for two reasons. The router's firewall is yours to configure, and the tunnel's credentials come from your Cloudflare account, so the run can supply neither.

Status: written, not yet run.

Prerequisites¶

tf01 is built, because bh01's Redis copies tf01's. See Traefik hub (tf01).
A Cloudflare account that holds the zone for your domain.
An admin machine with the cloudflared CLI installed. It must not be bh01, and the machine the control shell runs on will do.
A DMZ VLAN on the router, carried by the Proxmox bridge the VMs use, with firewall rules you control.

Placeholders¶

Placeholder	Value
`<admin>`	The admin account, `abbr_name` followed by `admin`, such as `mmadmin`
`<tunnel-id>`	The tunnel's ID, printed in step 4
`<hostname>`	A public name to publish, such as `ntfy.myah-mitchell.com`

1. Describe the host¶

In the private repo's hosts.yml, add bh01 to the docker_host group:

    bh01:
      ansible_host: 172.16.8.111
      network_gateway: "172.16.8.1"
      serverHostname: "bh01"
      komodo_stacks_manage: false
      docker_stacks:
        - system-agent
        - traefik-dmz

Two keys are new here, beside the ones km01's entry explains.

network_gateway is the DMZ's gateway. The docker_host group sets the internal network's gateway, so a host on the DMZ sets its own. See the group's values.

The host's DNS server is the same address as its gateway unless network_dns names another.

traefik-dmz includes traefik-agent, so the list does not name it.

While docker_stacks_bootstrap: true is set, the run leaves out system-agent and deploys traefik-dmz alone. traefik-dmz is a Traefik itself, so the run adds no traefik-bootstrap. See Bootstrap mode.

komodo_stacks_manage: false turns off the run's last stage for this host, the one where Komodo deploys the stacks. The first run then builds bh01 and deploys nothing. cloudflared cannot start before its credentials are on the host, and step 7 takes the line out again once they are.

In opentofu/prod.tfvars, add its VM inside vms:

  bh01 = {
    server       = "vh01"
    vm_id        = 8111
    cores        = 4
    memory_mb    = 8192
    vlan_id      = 8
    ipv4_address = "172.16.8.111/24"
    ipv4_gateway = "172.16.8.1"
    dns_servers  = ["172.16.8.1"]
    tags         = ["docker"]
    extra_disks = {
      persist = { interface = "scsi2", size_gb = 10 }
    }
  }

The VLAN, the address, the gateway, and the DNS server all come from the DMZ. bh01 is the first host where they differ from the internal ones. The run stops when the address or the gateway here differs from the one in hosts.yml.

Every request from the internet passes through this Traefik, and the four cores and 8 GB leave room for that. The persistent disk holds certificates and the tunnel's two files, so 10 GB is more than enough.

Then generate bh01's files: its SSH host keys, its NixOS file, and its Komodo file.

From ~/src/fleet-ansible, with the private repo and fleet-nixos checked out next to it, and the deploy key in SOPS_AGE_KEY or SOPS_AGE_KEY_FILE.

A host with no file under secrets/host-keys/ in the private repo gets its SSH host keys first:

nix run ../fleet-nixos#new-host-key -- --fleet ../fleet-private <host>

The command writes secrets/host-keys/<host>.yaml, adds the host to .sops.yaml, and encrypts secrets/fleet.yaml again so that the host can read it. For a host that has its keys already, it changes nothing. See Host keys.

Then write the NixOS files and the Komodo files:

ansible-playbook -i ../fleet-private/hosts.yml nixos-sync.yml
ansible-playbook -i ../fleet-private/hosts.yml komodo-sync.yml

Add what changed, and read the diff before it is committed:

git -C ../fleet-private add hosts.yml group_vars/ opentofu/ komodo/ nixos/ secrets/ .sops.yaml
git -C ../fleet-private diff --cached

The diff of nixos/hosts/<host>.json shows the host's network, its features, and the folders, seed files and ports its stacks need. The diff of komodo/stacks/<host>.toml shows the Stacks the next run deploys and every value in their Environment. The files under secrets/ are encrypted, so their diff shows that a value changed and not what it changed to.

Then commit and push:

git -C ../fleet-private commit -m "Describe <host>"
git -C ../fleet-private push

The run builds the host from the files git tracks in the private repo, and Komodo reads the pushed copy. The run stops at a host whose committed files differ from what the inventory gives.

2. Open the path across the boundary¶

Allow these four paths on the router, between the DMZ and the internal VLAN. The run itself needs the first two.

From	To	Port	Used for
The control node, ci01 at `172.16.7.121`	bh01	`22/tcp`	The run installs and deploys over SSH
bh01	km01, `172.16.7.101`	`9120/tcp`	Periphery dials Komodo Core
bh01	tf01, `172.16.7.111`	`6379/tcp`	The Redis copy, and bh01's own route publisher
bh01	Every internal host that serves a web interface	`443/tcp`	Published routes, the sign-in on id01, and telemetry to ci01

A run from a shell needs the first rule for the shell's own address.

bh01 also needs the internet, outbound only. Ports 80 and 443 carry NixOS packages, images, the fleet-stacks repo, Let's Encrypt, and Cloudflare's API. Port 7844, over both TCP and UDP, carries the tunnel.

Allow nothing else from the DMZ inward. A host in the DMZ that is taken over can reach whatever these rules leave open.

Background: the path of a public request, and why it needs these openings

The diagram follows one request from a browser on the internet to a service on an internal host, and shows which connections cross the router.

flowchart LR
  browser[Browser on the internet] -->|HTTPS to the public name| cf[Cloudflare]
  subgraph dmz[bh01, in the DMZ]
    cloudflared[cloudflared]
    bhtraefik[Traefik on bh01]
    copy[Redis copy]
  end
  subgraph internal[Internal network]
    redis[Redis on tf01]
    own[Traefik on the service's host]
    service[The service's container]
  end
  cloudflared -->|opens the tunnel, outbound on port 7844| cf
  cf -->|the request, down the tunnel| cloudflared
  cloudflared -->|ingress rule| bhtraefik
  copy -->|the routes| bhtraefik
  copy -->|connects to port 6379 and copies| redis
  bhtraefik -->|the request, port 443| own
  own --> service

Nothing connects to bh01 from the internet. cloudflared dials out to Cloudflare and keeps that connection open, and Cloudflare sends each request back down it. That is why the router forwards no port, and why bh01's only rule towards the internet is outbound.

Two connections cross from the DMZ to the internal network, and both start on bh01. The Redis copy dials tf01 on port 6379 to learn the routes. Traefik on bh01 then sends each request on to the Traefik on the service's own host, on port 443, which is the address this page assumes the route holds. The other two rows of the table are not on the request's path. They let the run and Komodo manage bh01 as they manage every host.

The copy means bh01 answers from the routes it last saw when tf01 is unreachable. See The edge on bh01 and the tunnel for how this fits with the hub.

The DMZ's DNS server has to resolve the fleet's internal names. bh01 finds tf01 by the value of TRAEFIK_KOP_REDIS_SERVER, which is tf01.home.myah-mitchell.com in these pages. Nothing writes DNS records in bootstrap mode, so add a record for that name by hand, pointing at 172.16.7.111.

Admit the DMZ on tf01¶

The router is one of two firewalls on this path. tf01 has a firewall of its own, and it opens the Redis port to docker_stacks_internal_subnet only. bh01 is outside that subnet.

In hosts.yml, open that one port on tf01 to bh01's address as well:

    tf01:
      ansible_host: 172.16.7.111
      serverHostname: "tf01"
      docker_stacks_port_sources:
        - port: 6379
          proto: tcp
          sources: ["172.16.8.111/32"]
      docker_stacks:
        - system-agent
        - traefik-server

The rule is part of tf01's configuration, so it changes in the inventory. A rule added on tf01 by hand is gone after the next deploy or reboot. See What the run overwrites.

Only port 6379 is opened to bh01. Every other internal port on tf01 stays closed to the DMZ. The entry has to name a port that a stack on tf01 opens to the internal subnet, and the build stops if it does not.

From ~/src/fleet-ansible, write tf01's file again, then commit and push:

ansible-playbook -i ../fleet-private/hosts.yml nixos-sync.yml
git -C ../fleet-private add hosts.yml nixos/
git -C ../fleet-private commit -m "Open tf01's Redis to bh01"
git -C ../fleet-private push

Run tf01 again, the same way as in its own build. Then log in to tf01 and read the rules:

sudo iptables -S nixos-fw | grep -E -e '--dport 6379 '
sudo iptables -S DOCKER-USER | grep -E -e '--ctorigdstport 6379 '

nixos-fw has two lines that end in -j nixos-fw-accept, one for the internal subnet and one for 172.16.8.111/32. DOCKER-USER has a line for 172.16.8.111/32 that ends in -j RETURN, followed by the line that drops everything outside the internal subnet.

The two chains are the host's own rules and the rules for ports Docker publishes. See The firewall.

3. Stage the values¶

Nothing new is needed. traefik-dmz reads seven values, and all seven exist already, from Setting up Komodo.

This Traefik uses the Cloudflare token and the Let's Encrypt address for real certificates, in bootstrap mode too. tf01 was the first host to use them, so a tf01 with a trusted certificate has proven both.

The tunnel's credentials are a file on bh01's disk, not a value in Komodo. The next step creates them.

4. Create the tunnel¶

On the admin machine, sign in to Cloudflare and create the tunnel:

cloudflared tunnel login
cloudflared tunnel create home-edge

The first command opens a browser, where you authorise the zone for your domain. The second prints the tunnel's ID and writes the credentials to ~/.cloudflared/<tunnel-id>.json.

home-edge names this site's edge. A second site would get a tunnel of its own.

This is a tunnel whose settings live in a file on bh01, not in Cloudflare's dashboard. The hostnames it accepts are then part of the host, and step 6 writes the file.

Confirm the tunnel exists:

cloudflared tunnel list

The list includes home-edge with the same ID.

Warning

Anything that holds the credentials file can serve traffic for your hostnames. Keep it out of every repo, and delete every copy but the one on bh01 and the one in your password manager.

5. Run the build¶

SemaphoreCommand line

In Semaphore, run the site Template with Target set to bh01.

From ~/src/fleet-ansible, in a shell prepared for runs after the handover. See Running from a shell again.

ansible-playbook -i ../fleet-private/hosts.yml site.yml \
  -e target=bh01

The run creates the VM, installs NixOS on it, and deploys its configuration, which makes the folders its stack needs and puts cloudflared's example config in place. It ends there, with no Stack deployed.

The recap line for bh01 shows failed=0 and unreachable=0. In Komodo's UI, under Resources > Servers, bh01 shows as connected, which proves Periphery reached km01 across the boundary.

Manual steps, instead of site.yml

Create the VM by hand¶

The run's first stage creates the VM through OpenTofu, from the host's entry in opentofu/prod.tfvars. By hand, on the Proxmox host, with the values from that same entry:

qm create <vmid> --name <host> --ostype l26 \
  --cpu x86-64-v2-AES --cores <cores> --numa 1 \
  --memory <memory> --balloon 0 \
  --scsihw virtio-scsi-single \
  --scsi0 <vm-storage>:<os-size>,discard=on,ssd=1,iothread=1 \
  --scsi1 <vm-storage>:<docker-size>,discard=on,ssd=1,iothread=1 \
  --scsi2 <vm-storage>:<persist-size>,discard=on,ssd=1,iothread=1 \
  --ide2 local:iso/fleet-nixos-installer.iso,media=cdrom \
  --ide0 <vm-storage>:cloudinit \
  --ipconfig0 ip=<address>,gw=<gateway> \
  --nameserver <dns-server> \
  --net0 virtio,bridge=vmbr0,tag=<vlan-id> \
  --boot 'order=scsi0;ide2' \
  --serial0 socket --vga serial0 \
  --agent enabled=1 --onboot 1 \
  --tags 'nixos;<tags>'
qm start <vmid>

Placeholder	Value
`<vmid>`	The entry's `vm_id`
`<host>`	The entry's name, which is the host's name in the inventory
`<cores>`, `<memory>`	The entry's `cores` and `memory_mb`
`<vm-storage>`	The Proxmox storage that holds VM disks, `local-zfs` unless the server's entry sets `datastore_id`
`<os-size>`, `<docker-size>`	The entry's `os_disk_gb` and `docker_disk_gb` in GB, `20` and `40` when it sets neither
`<persist-size>`	The persist disk's `size_gb`
`<address>`, `<gateway>`	The entry's `ipv4_address`, with its prefix length, and `ipv4_gateway`
`<dns-server>`	The entry's `dns_servers`, separated by spaces inside one pair of quotes when there are several
`<vlan-id>`	The entry's `vlan_id`
`<tags>`	The entry's `tags`, separated by semicolons

An entry with no dns_servers takes no --nameserver option, and one with no vlan_id takes no ,tag= part.

The three disks are blank. The VM finds nothing to boot on scsi0, so it boots the installer ISO on ide2, and the installer takes its address from the cloud-init drive on ide0.

The ISO has to be on the Proxmox host first. See The installer ISO.

Create all three disks before the first start. The install partitions scsi0 and scsi1 and formats scsi2, and it stops when one is missing.

A VM made this way is not in OpenTofu's state, and it does not carry the tofu tag that marks the VMs OpenTofu manages. Leave its entry out of opentofu/prod.tfvars, or a later run tries to create a second VM under the same name and address.

Install NixOS without the rest of the run¶

The run's second and third stages wait for the VM's SSH port, install NixOS when the VM answers as the installer, and deploy the host's configuration. The same three commands of the flake do it from a shell.

From ~/src/fleet-ansible, with the host's files committed in the private repo, the deploy key in SOPS_AGE_KEY or SOPS_AGE_KEY_FILE, and the fleet's SSH key in ~/.ssh/config. See The fleet's SSH key.

Ask the VM what it runs:

nix run ../fleet-nixos#host-state -- <address>

It prints installer for a VM that booted the ISO. Then install:

nix run ../fleet-nixos#install-host -- --fleet ../fleet-private <host> <address>

Warning

The install wipes scsi0 and scsi1. It formats scsi2 only when the disk is blank, and it refuses any machine that is not the installer.

The last line reads install-host: <host> is installed and is rebooting. Repeat the host-state command until it prints installed, then deploy the configuration:

nix run ../fleet-nixos#deploy-host -- --fleet ../fleet-private <host> <address>

<host> is the host's name in the inventory, and <address> is its ansible_host. The configuration is the whole base build: accounts, SSH, the firewall, Docker, the persistent disk, Periphery, Node Exporter, and the folders, seed files and ports of every stack in the host's file. See The commands.

For traefik-dmz:

A host deployed with the stack in its file has these folders and seed files already. The commands make them on a host whose file does not list the stack.

sudo install -d -o $USER -g 101000 -m 0750 \
  /opt/docker/logs/traefik /opt/docker/volumes/traefik
sudo install -d -o 101000 -g 101000 -m 0755 /opt/docker/logs/traefik/traefik
sudo install -d -o 101000 -g 101000 /opt/docker/volumes/traefik/traefik-certs
sudo install -d -o 101000 -g 101000 /opt/docker/volumes/traefik/traefik-plugins
sudo install -d -o 101000 -g 101000 /opt/docker/volumes/traefik/cloudflared-config
sudo install -d -o 101000 -g 101000 -m 0700 /opt/docker/volumes/traefik/cloudflared-secrets

sudo test -e /opt/docker/volumes/traefik/cloudflared-config/config.yml || sudo curl -fsSL \
  -o /opt/docker/volumes/traefik/cloudflared-config/config.yml \
  https://raw.githubusercontent.com/myah-mitchell/fleet-stacks/main/containers/cloudflared/config/config.yml.example
sudo chown 101000:101000 /opt/docker/volumes/traefik/cloudflared-config/config.yml

No command opens the stack's ports, because the firewall changes only through the host's configuration. The ports below are open once the host's file, nixos/hosts/<host>.json, holds them: nixos-sync.yml writes that file in the private repo, and a deploy of the host applies it. See Host layout.

Port	Allowed from
`80/tcp`	Anywhere
`443/tcp`	Anywhere
`8443/tcp`	Anywhere

6. Install the tunnel's files¶

From the admin machine, copy the credentials to bh01:

scp ~/.cloudflared/<tunnel-id>.json <admin>@172.16.8.111:

Log in to bh01 and move the file into place:

sudo install -o 101000 -g 101000 -m 0600 ~/<tunnel-id>.json \
  /opt/docker/volumes/traefik/cloudflared-secrets/<tunnel-id>.json
rm ~/<tunnel-id>.json

Open the config the run put in place:

sudoedit /opt/docker/volumes/traefik/cloudflared-config/config.yml

Put the tunnel's ID in the first two lines, and remove the example rule for vault.myah-mitchell.com. The file then reads:

tunnel: <tunnel-id>
credentials-file: /etc/cloudflared/secrets/<tunnel-id>.json

ingress:
  - service: http_status:404

Each entry under ingress is a rule that says where cloudflared sends the requests for one hostname. The last rule answers any hostname no other rule matches. It stays, and it stays last, because cloudflared reads the rules in order.

Give the file back to the container's user, in case the editor changed its owner:

sudo chown 101000:101000 \
  /opt/docker/volumes/traefik/cloudflared-config/config.yml

Check both files:

sudo ls -ln /opt/docker/volumes/traefik/cloudflared-secrets \
  /opt/docker/volumes/traefik/cloudflared-config

Each folder holds one file owned by 101000, and the credentials show mode -rw-------.

A host's configuration puts the example there only when no file is in its place, so these edits survive every later run and a rebuild of bh01.

7. Deploy the edge¶

In hosts.yml, remove the komodo_stacks_manage: false line from bh01. Commit and push the change:

git -C ../fleet-private add hosts.yml
git -C ../fleet-private commit -m "Deploy bh01's stacks"
git -C ../fleet-private push

komodo_stacks_manage is in neither of bh01's generated files, so there is nothing to generate again.

Run the build a second time, the same way as in step 5. This time the run has Komodo deploy one Stack, traefik-dmz.

Manual steps, instead of site.yml

Deploy the Stacks by hand¶

The run's last stage has Komodo read the host's Stacks from komodo/stacks/<host>.toml in the private repo and deploy them. To create and deploy one in Komodo's UI instead, follow Deploying a stack by hand.

A Stack made by hand under the name the run would give it is taken over by the next run, not duplicated.

8. Verify¶

The run ends with ansible's recap, one line of counts for the host. failed=0 and unreachable=0 mean every stage finished, and the last stage only finishes once Komodo reports every one of the host's Stacks as running.

In Komodo's UI, open Resources > Servers. The host shows as connected.

Open Resources > Stacks and filter by the host's Server. Each Stack shows as Running.

If the run stopped instead, its last message names the stage and what it stopped at. Fix the cause and run it again. Every stage is safe to repeat.

The traefik-dmz Stack has eight services:

traefik
error-pages
socket-proxy
logrotate
socket-proxy-rw
traefik-kop
redis
cloudflared

On bh01, confirm the Redis copy is in step with tf01:

docker exec traefik-redis redis-cli info replication

The output includes role:slave and master_link_status:up. slave is Redis's word for a copy, and tf01's Redis is the master. A copy that cannot reach tf01 reports down and keeps serving what it last had, so check this now.

Confirm the tunnel is connected:

docker ps --filter name=traefik-cloudflared --format '{{.Status}}'

The status ends with (healthy). cloudflared's health check passes only once the tunnel has a connection to Cloudflare.

Confirm the firewall:

sudo iptables -S nixos-fw | grep -E -e '--dport (80|443|8443) '

Three rules show, one for each of Traefik's ports. None names a source address after -s, and each ends in -j nixos-fw-accept.

9. Open the dashboard¶

Open https://traefik.bh01.home.myah-mitchell.com:8443 in a browser. The name needs a DNS record pointing at bh01, or an entry in your own hosts file.

The certificate is from Let's Encrypt, so the browser shows no warning.

The dashboard opens with no sign-in while the fleet is in bootstrap mode. Nothing on the internet can reach port 8443 on bh01, since nothing forwards to it, but every machine in the DMZ can.

While the fleet is in bootstrap mode¶

Part of bh01	In bootstrap mode	After it
Certificate	Let's Encrypt, trusted	The same
Tunnel	Connected, with no hostname on it	Carries each published hostname
Redis copy	In step with tf01, which holds no routes	Holds every published route
Dashboard	No sign-in	Authentik first
Metrics and logs	Not shipped. system-agent is left out	Shipped to ci01

bh01 can prove the tunnel and the copy in bootstrap mode, and it cannot publish anything. A route reaches bh01 only when the route publisher on the service's own host writes it to tf01's Redis. The publisher is part of traefik-agent, and a host in bootstrap mode runs traefik-bootstrap in its place.

Publishing a hostname¶

Do this after the fleet has left bootstrap mode, once for each public name.

A service can be published only when its container carries kop-public labels. In fleet-stacks today those are ntfy, Stalwart, and Bulwark.

Two lists decide what the internet reaches, and a name has to be on both. The labels put the route on bh01's Traefik, by way of tf01's Redis and the copy, and the steps here put the name on the tunnel. See How a route reaches the hub.

On the admin machine, create the public DNS record:

cloudflared tunnel route dns home-edge <hostname>

On bh01, add a rule for the name to config.yml, above the catch-all:

ingress:
  - hostname: <hostname>
    service: http://traefik-traefik:80

  - service: http_status:404

Every rule points at bh01's own Traefik, which decides where the request goes. traefik-traefik is that container's name, because every Traefik stack uses the project name traefik.

Restart cloudflared so that it reads the file:

docker restart traefik-cloudflared

Open https://<hostname> from outside your network, such as from a phone with Wi-Fi off. The service answers.

Warning

A hostname on the tunnel is on the internet from the moment its DNS record exists. Publish only a service that signs its users in itself, or one whose route uses the Authentik chain.

A public name has no sub-domain in it. ntfy.myah-mitchell.com is public, and ntfy.home.myah-mitchell.com is the internal name for the same service.

What's next¶

Every host the fleet was waiting for is up. Leave bootstrap mode next. See Leaving bootstrap mode.

mx01 is the host after that, and it is optional. See Mail (mx01).

Not yet confirmed¶

The whole page. bh01 has not been built by the run.
The rules on tf01 for bh01's address. The inventory value reaches tf01's file and the rules the flake writes from it were read in an evaluation of tf01's configuration on 2026-10-03, and have not been deployed. Whether DOCKER-USER holds for a published port depends on Docker keeping the chain, as the host layout says.
The Redis provider on bh01. Traefik sends the Redis password to the local copy, and the copy is started with no password of its own. Redis may refuse that login.
Ingress rules that point at port 80. Traefik's entrypoint there redirects to HTTPS, and no published route listens on it. The rules may have to point at port 443 instead.
The address the route publisher writes for a published service. The page assumes the service's own host, on port 443.
Port 7844 for the tunnel, which comes from Cloudflare's documentation.
Publishing Authentik. auth.myah-mitchell.com is its public name, and authentik-server carries no kop-public labels, so bh01 has no route for it yet.