The handover¶

The shell has done its work. This page moves OpenTofu's state from the shell's file into the database on ci01, proves that Semaphore can run the fleet, and removes the secrets from the shell.

At the end Semaphore is the fleet's control node, and your own machine holds nothing that can reach a host. The work is a handful of commands in the shell, two runs started from Semaphore's UI, and some careful copying into a password manager. The two runs change nothing on the hosts, so neither one installs anything.

Status: written, not yet run. The state move was tried with OpenTofu 1.12.6 against a local Postgres and read back with 1.9.0, the version in Semaphore's image. See Not yet confirmed for the rest.

What moves where¶

The shell holds four things a run needs. Three of them were copied into Semaphore on the previous page. The fourth, the state, moves on this page. The diagram shows where each one ends up before the shell's copy is deleted.

flowchart LR
  subgraph shell[The shell]
    state[State file]
    sshkey[Fleet SSH private key]
    agekey[Deploy age key]
    env[Environment file]
  end
  subgraph semaphore[Semaphore on ci01]
    db[tofu_state database]
    keystore[Key Store]
    group[Variable Group secrets]
  end
  state -->|moved in step 2| db
  sshkey -->|copied on the previous page| keystore
  agekey -->|copied on the previous page| group
  env -->|copied on the previous page| group
  sshkey -->|stored in step 4| vault[Your password manager]
  agekey -->|stored in step 4| vault
  env -->|stored in step 4| vault

The state is the one thing with a single copy, which is why it is moved and read back before anything is deleted. The password manager gets the rest because Semaphore never shows a stored secret again.

Prerequisites¶

Semaphore has its Project and the site Template, from The Semaphore project.
The shell has ~/.config/fleet/env loaded, and the state file from the first run is where the run left it.
You have a password manager, or another encrypted store, to keep four secrets in.

Placeholders¶

Placeholder	Value
`<admin>`	The admin account, `abbr_name` followed by `admin`, such as `mmadmin`
`<tofu-state-password>`	The `tofu` role's password, the Komodo Secret `SEMAPHORE_TOFU_STATE_PASSWORD`
`<postgres-ip>`	The database container's address inside ci01, found in step 1

1. Open a tunnel to the state database¶

Log in to ci01 and read the database container's address:

ssh <admin>@172.16.7.121

docker inspect semaphore-postgres \
  -f '{{ (index .NetworkSettings.Networks "semaphore_backend").IPAddress }}'

Log out. In a second terminal in the shell, open the tunnel and leave it running:

ssh -N -L 15432:<postgres-ip>:5432 <admin>@172.16.7.121

The command prints nothing and does not return. While it runs, port 15432 in the shell reaches the database.

The database listens on a network inside ci01 and nowhere else, so the shell cannot connect to it directly. -L has SSH carry a port of the shell's to the container through ci01, and -N keeps the session open without running a command there.

2. Move the state¶

In the first terminal, point OpenTofu at the tunnel. Enter the password when asked, which is the value of the Komodo Secret SEMAPHORE_TOFU_STATE_PASSWORD:

read -rs -p "Password of the tofu role: " tofuPassword; echo
export PG_CONN_STR="postgres://tofu:${tofuPassword}@localhost:15432/tofu_state?sslmode=disable"

Remove the setting that keeps the state in a file, and let OpenTofu copy the state to the backend the repo declares. A backend is the place OpenTofu keeps its state. The first run left a checkout of the fleet-opentofu repo behind, and vms_backend=local wrote that setting into it:

cd /tmp/ansible-fleet-opentofu-checkout/envs/prod
rm backend_override.tf
tofu init -migrate-state -force-copy

The output includes Successfully configured the backend "pg"!. Read the state back from the database:

tofu state list

The list includes one line for each VM the shell created:

module.vm["ci01"].proxmox_virtual_environment_vm.this
module.vm["km01"].proxmox_virtual_environment_vm.this

The state is encrypted the same way in both places, with the passphrase in TF_ENCRYPTION. -migrate-state is what copies it, and -force-copy answers the question OpenTofu would otherwise ask before copying.

If the checkout is gone¶

/tmp does not always survive a restart of the machine. The state file is in your home folder and does. Make the checkout again, pointed at the file, then go back to the rm above:

git clone --depth 1 https://github.com/myah-mitchell/fleet-opentofu \
  /tmp/ansible-fleet-opentofu-checkout
cd /tmp/ansible-fleet-opentofu-checkout/envs/prod
cat > backend_override.tf <<EOF
terraform {
  backend "local" {
    path = "$HOME/.local/state/fleet-opentofu/prod.tfstate"
  }
}
EOF
tofu init

3. Prove Semaphore¶

In Semaphore, run the site Template with Target set to km01. The run uses nothing from the shell, so a pass shows that Semaphore holds everything it needs.

The run passes through every stage and leaves km01 as it was. Its recap shows failed=0 and unreachable=0 for km01. changed is not zero: the nixos stage reports a change each time it deploys, and the komodo stage each time it syncs, whether or not the host ends up different.

The stage	Proves that Semaphore
`vms`	Reads the state, decrypts it, and reaches the Proxmox API
`wait`	Reaches km01's SSH port
`nixos`	Decrypts the private repo with the deploy key, clones fleet-nixos, and reaches km01 over SSH with the fleet's key
`komodo`	Reaches Komodo's API with the service user's key

A run that fails in the vms stage with a message about creating a VM did not find the state. Check PG_CONN_STR and TF_ENCRYPTION in the Variable Group, then run it again. Proxmox refuses a second VM under an ID that is taken, so the failed run has created nothing.

Run the Template again with Target set to ci01.

Close the tunnel by pressing Ctrl+C in the second terminal.

4. Store what outlives the shell¶

Semaphore holds every secret the run needs, and shows none of them again. Semaphore also lives on ci01, so the day ci01 is rebuilt, a shell has to do it.

Put these four in your password manager before the next step deletes them:

Secret	Where it is
The fleet's SSH private key	`~/.ssh/fleet-ansible`
The deploy age key	`~/.config/fleet/deploy.key`
The run's secrets, state passphrase included	`~/.config/fleet/env`, the whole file
The `tofu` role's password	The Komodo Secret `SEMAPHORE_TOFU_STATE_PASSWORD`

The admin age key is in the password manager already, from The control shell. It opens every file the deploy key opens, so it is the way back in if the deploy key is ever lost. Check it is there.

Warning

Without the state passphrase nobody can read the state, in the database or in a backup of it. Every VM would have to be imported again.

5. Clean the shell¶

Delete the state, the checkouts the run made, the environment file, and the two keys:

rm -rf ~/.local/state/fleet-opentofu /tmp/ansible-fleet-opentofu-checkout /tmp/ansible-fleet-nixos-checkout
rm ~/.config/fleet/env ~/.config/fleet/deploy.key
rm ~/.ssh/fleet-ansible ~/.ssh/fleet-ansible.pub

Remove the Match block for the fleet's key from ~/.ssh/config. Left in place, it points at a file that is gone, and every login to a host fails until it is taken out.

Close every terminal that loaded the environment file. The values stay in a terminal's memory until it exits.

The checkouts under ~/src can stay. The private repo's secrets are encrypted, and without the deploy key the shell cannot read them.

Background: why the shell's files are deleted

Between them, the files can log in to every host as root, create VMs on Proxmox, decrypt every secret in the private repo, and read the state. That was needed while the shell was the control node. It is not needed now, and a machine you also read mail and browse on is a poor place to leave it.

Semaphore holds the same secrets on a host that does one job, behind a sign-in, and shows none of them again. Keeping a second live copy in the shell would double the places a key can leak from, and the shell's copy is the one nobody would notice being read.

The copies in the password manager are for the day a shell is needed again. They are encrypted at rest and take a deliberate act to bring back, which is the difference between them and a file in your home folder.

What's next¶

The foundation is finished. Every host from here is described from a shell and then built by one run of the site Template, in the order the running order gives. The next host is id01. See Identity (id01).

Describing a host makes its keys and writes its files, and that needs an age key that opens the secrets: the deploy key or the admin key. Restore one to the shell for that step, as in Running from a shell again, and clean the shell again afterwards.

The two sections below are reference for later, and no part of the handover: The state database and Running from a shell again.

The state database¶

The state lives in tofu_state, a second database on Semaphore's Postgres. A role named tofu owns that database and nothing else, so Semaphore's own login cannot read it.

A script in the stack's postgres-initdb folder creates both, on the first start of an empty data folder and never again. Changing the role's password later is an ALTER ROLE by hand, followed by the new value in PG_CONN_STR.

The backup container dumps both databases. Restore tofu_state on its own. Restoring the whole server from a dump rolls the state back with everything else, and an old state makes OpenTofu try to create VMs that exist.

Running from a shell again¶

Three cases need a shell after the handover: Semaphore is down, ci01 itself is being rebuilt, or a run needs an option the Template does not pass.

Set the shell up as The control shell does, without creating a key or a passphrase. Restore the fleet's SSH key, the deploy age key, and the environment file from your password manager, to the paths the table in step 4 gives, and put the Match block back in ~/.ssh/config.

With ci01 up, the state stays in the database. Open the tunnel from step 1, and add this line to the environment file with the tofu role's password in it:

export PG_CONN_STR="postgres://tofu:<tofu-state-password>@localhost:15432/tofu_state?sslmode=disable"

Then run the command from the host page's Command line tab as it is written. Do not add vms_backend=local. That option starts a second, empty state, and OpenTofu then tries to create every VM in the run again.

With ci01 down, the database is down with it. See Rebuilding ci01 for how the state is taken out before the rebuild and put back after it.

Clean the shell again when the work is done, as in step 5.

Not yet confirmed¶

The tunnel. The host can reach a container on an internal Docker network by its address, and the host's sshd does not turn TCP forwarding off, but the two have not been tried together on ci01.
The state move against the real database. It was tried against a local Postgres, with a state that held no Proxmox VM.
A first run of tofu from inside Semaphore.
What the vms stage does when it finds no state. The page's claim that Proxmox refuses the duplicate ID follows from how Proxmox treats VM IDs, and has not been provoked.
The nixos stage from Semaphore: sops with the key from SOPS_AGE_KEY, the clone of fleet-nixos, and the flake's ssh calls with the key from Semaphore's agent.
The recap of a run that changes nothing on the host. The nixos and komodo stages report a change by design, and the count has not been seen from a real run.
A run against ci01 from Semaphore, which is a run against the host Semaphore is on. A deploy that restarts Semaphore's own container, or a sync that redeploys semaphore-server, should stop the run partway, and pass when run again.