# Debugging Notes

Live notes from debugging the NixOS K8s QEMU image build and boot process.

## 2026-08-21: kubelet preStart wipes /opt/cni/bin, breaking all new pods

### Symptom
- Cluster healthy for ~11h, then every newly created pod stuck in
  `ContainerCreating` with `FailedCreatePodSandBox`: `plugin type="cilium-cni"
  failed (add): failed to find plugin "cilium-cni" in path [/opt/cni/bin]`.
- Existing pods unaffected; nodes Ready; Cilium pods Running.
- `/opt/cni/bin/cilium-cni` missing although `install-cilium-cni.service`
  had succeeded at boot (exit 0).

### Root cause
Two interacting behaviors:
1. The NixOS kubernetes module's kubelet `ExecStartPre` runs
   `rm /opt/cni/bin/* || true` on **every kubelet start**, then re-links only
   the NixOS `cniPluginPackages`. Any foreign binary is destroyed.
2. `certmgr` periodically refreshes kubelet certs ("spec DNS name has
   changed") and restarts kubelet via `systemctl restart kubelet.service`.

So each cert rotation wiped `cilium-cni`, and the boot-time-only extraction
service never ran again.

### Fix
Make `install-cilium-cni` self-healing (`common.nix`):
- Added a systemd `.path` unit (`systemd.paths.install-cilium-cni`) with
  `PathModified = /opt/cni/bin` triggering the service on any directory change.
- Service fast-exits when the binary exists; otherwise extracts it from the
  imported Cilium image via `ctr images mount` as before.
Boot-time `wantedBy` retained so first-boot extraction still waits out the
image-import window.

### Technique
- Pod stuck in `ContainerCreating` ≠ image problem: read pod events
  (`kubectl describe pod`) before checking images.
- When a file mysteriously disappears, check whether any unit's ExecStartPre
  recreates its directory: `journalctl -u <unit>` around the file's mtime, and
  read the actual store-path script
  (`cat $(systemctl show -p ExecStartPre --value <unit> | cut -d';' -f1)`).

---

## 2026-08-21: Pre-flight validating rendered manifests without mutating

### Technique
Before switching an application to a new deployment tool (e.g. Nixidy), validate
the rendered output against a live API without side effects:

```bash
kubectl apply --dry-run=server -f <manifest>   # per file; exit code matters
```

This catches schema validation errors, missing API groups, and admission
rejections that client-side dry-runs miss. Remember `KUBECONFIG` must be set in
the same shell or kubectl silently falls back to `http://localhost:8080`.

---

## 2026-07-11: Image builds but VM never boots via EFI

### Symptom
- `nix build .#k8s-master-0` produces a valid 3.4GB QCOW2 EFI image
- Booting with OVMF + systemd-boot: zero serial output, SSH never connects
- VM process stays alive, port 2222 is always open (QEMU user-networking opens it immediately regardless of guest state)
- After 4+ minutes, still no SSH and no serial output

### What works
- `nixos-rebuild build-vm` boots fine (SSH connects, auth denied since no key)
- Direct kernel boot (`-kernel`/`-initrd` flags) from the same system closure starts the kernel

### Root cause (identified, not yet fixed)
The `qemu-guest.nix` profile imported at `flake.nix:15`:
```nix
qemuGuestModule = {
  imports = [ "${toString nixpkgs}/nixos/modules/profiles/qemu-guest.nix" ];
  ...
};
```
This adds 9p virtio filesystem configuration to the initrd. When booting as a standalone
disk image (no 9p virtio hosts), the initrd tries to mount `nix-store`, `shared`, `xchg`
via 9p, fails, drops to emergency shell with locked root — system never reaches multi-user.

Direct kernel boot output showing the failure:
```
9pnet_virtio: no channels available for device nix-store
FAILED Failed to mount /sysroot/nix/.ro-store
Dependency failed for /sysroot/nix/store
You are in emergency mode. After logging in, type "journalctl -xb" to view system logs.
Cannot open access to console, the root account is locked.
```

### ESP is valid
Parsed the GPT + FAT32 ESP manually. Contents:
- Partition 1: ESP, LBA 16384-526335, 249MB
- Partition 2: Linux filesystem, 5.5GB
- ESP has BOOTX64.EFI, systemd-boot 261, NixOS directory, kernel, initrd, config entries
- loader.conf: `#timeout 3`, `#console-mode keep` (commented out = defaults)
- systemd-boot binary contains expected strings, version 261

### OVMF serial behavior
OVMF does NOT output to serial when using `-nographic`. Firmware/bootloader messages
only appear on VGA framebuffer. To see OVMF output you need:
- `-vnc :N` or a display, OR
- `-debugcon file:/path/to/log -global isa-debugcon.iobase=0x402` for UEFI debug log

This means you cannot debug EFI boot failures over serial alone.

### `-cpu host` required
The default QEMU CPU (`qemu64`) lacks features the kernel requires. Use `-cpu host`
when booting with KVM. Without it, the kernel prints:
```
This kernel requires the following features not present on the CPU:
Unable to boot - please use a kernel appropriate for your CPU.
```

### OVMF vars lock after first use
QEMU locks the OVMF VARS file after a boot session. Must copy fresh from nixpkgs
each time and `chmod +w`:
```bash
OVMF_FD=$(nix eval --raw nixpkgs#OVMF-xen.fd)
rm -f /tmp/k8s-vars.fd
cp -f "$OVMF_FD/FV/OVMF_VARS.fd" /tmp/k8s-vars.fd
chmod +w /tmp/k8s-vars.fd
```
OVMF CODE file is fine with `readonly=on`.

### QEMU port forwarding is misleading
QEMU user-networking hostfwd opens the host port immediately when QEMU starts,
regardless of guest state. An open port does NOT mean the guest has booted or sshd
is running. Only use actual SSH connection attempts (or serial output) to determine
boot status.

### `timesyncd.enable` needs `lib.mkDefault`
The `vmVariant` NixOS module sets `services.timesyncd.enable = false`. Our config
set it to `true` without priority, causing a conflict when building `build-vm`:
```
The option `virtualisation.vmVariant.services.timesyncd.enable' has conflicting definition values
```
Fix: use `lib.mkDefault true` in `common.nix:54`.

### `nixos-rebuild build-vm` test command
This is the quickest way to verify the config boots:
```bash
cd kubernetes/
QEMU_NET_OPTS="hostfwd=tcp::2222-:22" nixos-rebuild build-vm --flake .#k8s-master-0
# Find the run script in the nix store, run it
# SSH with: ssh -o StrictHostKeyChecking=no -p 2222 root@localhost
# (auth will fail — no password/key configured — but sshd running = success)
```
Note: this bypasses the bootloader (uses `-kernel`/`-initrd` directly) and includes
9p virtio shares, so it doesn't test the standalone image boot path.

---

## 2026-08-19: IPv6-only Cilium bootstrap deadlock

### Symptom
- Cilium init containers repeatedly failed with timeouts to `https://[fd03::1]:443`.
- Nodes had IPv6 addresses, but all nodes remained `NotReady` and CoreDNS stayed Pending.

### Root cause
The Kubernetes Service ClusterIP is not routable until Cilium programs the service
datapath. Cilium's own init container therefore cannot use the in-cluster API Service
to bootstrap an IPv6-only cluster.

### Fix
Set Cilium `k8sServiceHost: k8s-master-0` and `k8sServicePort: 6443`, using the
direct IPv6-resolved master address during bootstrap.

### Technique
Inspect the `config` init-container log with `kubectl logs -n kube-system <cilium-pod>
-c config --previous` and check whether it is dialing `fd03::1` or the master address.

---

## 2026-08-19: CoreDNS IPv6 certificate and forwarding loop

### Symptom
- CoreDNS crashed with `plugin/loop`.
- CoreDNS could not watch Kubernetes resources because the API certificate did not
  include `fd03::1`.

### Root cause
The API Service is IPv6 ClusterIP `fd03::1`, but easyCerts did not derive that SAN
from the IPv6 service CIDR. CoreDNS also forwarded external queries to the node's
`/etc/resolv.conf`, which points at the CoreDNS Service itself.

### Fix
Add `fd03::1` through `apiserver.extraSANs` and set CoreDNS `forward` to explicit
IPv6 upstream resolvers instead of `/etc/resolv.conf`.

---

## 2026-08-19: Offline image archive reference mismatch

### Symptom
- `ctr images import` succeeded, but kubelet still attempted to pull Cilium images
  by their digest and failed because the nodes have no internet.

### Root cause
The old Podman archive preserved tags but not the digest-qualified references emitted
by the Helm chart. Podman also collapsed multiple `podman save` inputs into one image
with multiple tags on this host.

### Fix
Deploy Cilium's imported version tags with `image.useDigest`, `envoy.image.useDigest`,
and `operator.image.useDigest` disabled. Build one Docker archive per image and import
each archive independently on every node.

---

## 2026-08-19: Offline Rook manifests and Ceph convergence

### Symptom
- Rook installation failed because the master tried to fetch GitHub manifests from
  the offline IPv6-only network.
- Ceph remained `Progressing` while the dashboard module hung creating its admin user.

### Fix
Download Rook CRD, common, CSI-operator, and operator manifests on the control
machine, copy them to the master, and apply the local files. Disable the optional Ceph
dashboard for this storage-only test cluster so Rook can reach `HEALTH_OK`.

---

## 2026-08-19: ULA IPv6 internet egress requires host NAT66

### Symptom
- The host could ping public IPv6, while nodes routed through `fd7a::1` but public
  IPv6 ping and HTTPS failed.

### Root cause
The libvirt bridge provides private ULA addressing but does not provide IPv6 NAT for
the guest subnet. Public upstream routing cannot accept `fd00::/8` source addresses.

### Fix
Configure host NixOS `networking.nat.enableIPv6 = true` with `internalIPv6s =
["fd00::/8"]` and the active external interface, then activate the host system.

### Verification
Run `just verify`; it checks public IPv6 reachability from every node and from a
temporary IPv6 HTTPS test pod.

---

## 2026-07-11: OVMF-xen.fd does not work with QEMU/KVM

### Root cause (RESOLVED)
**OVMF-xen.fd is a Xen-specific firmware variant.** It does not initialize VGA output
when run under QEMU/KVM, so the VM produces zero serial/VGA output and never boots.

**Fix:** Use `nix eval --raw 'nixpkgs#OVMF.fd'` — the standard OVMF variant:
```bash
OVMF_FD=$(nix eval --raw 'nixpkgs#OVMF.fd')
# CODE: $OVMF_FD/FV/OVMF_CODE.fd  (use with readonly=on)
# VARS: $OVMF_FD/FV/OVMF_VARS.fd (copy to /tmp, chmod +w, use as read-write)
```

### Verification
With `OVMF.fd`, the VM boots through:
- UEFI firmware → systemd-boot menu → NixOS initrd → switch-root → login prompt
- All K8s services start: apiserver, controller-manager, scheduler, proxy, kubelet, containerd
- Serial console (`ttyS0`) shows full boot log and login prompt
- `kube-addon-manager` fails (non-blocking — expected on first boot before CNI is configured)

### Complete working QEMU test command
```bash
OVMF_FD=$(nix eval --raw 'nixpkgs#OVMF.fd')
rm -f /tmp/k8s-vars.fd
cp -f "$OVMF_FD/FV/OVMF_VARS.fd" /tmp/k8s-vars.fd
chmod +w /tmp/k8s-vars.fd

qemu-system-x86_64 \
  -enable-kvm -m 8192 -smp 4 -cpu host \
  -drive if=pflash,format=raw,readonly=on,file=$OVMF_FD/FV/OVMF_CODE.fd \
  -drive if=pflash,format=raw,file=/tmp/k8s-vars.fd \
  -drive file=/tmp/k8s-test.qcow2,if=virtio,format=qcow2 \
  -netdev user,id=net0,hostfwd=tcp::2222-:22 \
  -device virtio-net-pci,netdev=net0 \
  -nographic
```

### SSH authentication
The VM boots and sshd runs with:
```nix
services.openssh.enable = true;
services.openssh.settings = {
  PermitRootLogin = "yes";
  PasswordAuthentication = false;
};
```
SSH connects successfully but requires an authorized key.

**SSH key setup:**
- Public key stored at `kubernetes/ssh-public-key` (git-tracked)
- Read via `builtins.readFile ./ssh-public-key` in `flake.nix`
- Passed to NixOS configs as `_module.args.sshPublicKey`
- Used in `common.nix`: `users.users.root.openssh.authorizedKeys.keys = [ sshPublicKey ];`
- To rotate: regenerate key, update `kubernetes/ssh-public-key`, rebuild images

### kube-addon-manager fails on first boot
The addon manager service fails with:
```
FAILED] Failed to start Kubernetes addon manager
```
This is expected — it tries to create pods before CNI is configured.

**Cilium install plan:**
- Currently: install via Helm post-boot (`cilium install`)
- Future: integrate into NixOS addon manager for auto-install
  - Generate Cilium manifests via Helm at build time
  - Add to `services.kubernetes.addons.cilium.manifests`
  - Addon manager applies them automatically on first boot
  - Needs investigation: does NixOS kubernetes module support CRDs via addons?

---

## 2026-07-12: API server port mismatch (master vs workers)

### Symptom
- Master kubelet registers successfully, workers cannot reach apiserver
- `systemctl is-active kubelet` shows "activating" on workers, "active" on master
- certmgr logs: `authentication error. Giving up without retries` / `invalid token`

### Investigation technique: trace NixOS module source
The NixOS module source lives in the Nix store. Find it:
```bash
# On the running VM:
find /nix/store -path '*kubernetes*' -name '*.nix' 2>/dev/null | grep kubelet | head -5

# Grep for relevant config patterns:
grep -rn 'apiserverAddress\|masterAddress\|securePort' /nix/store/.../nixos/modules/services/cluster/kubernetes/
```

Key finding in `default.nix:345`:
```nix
services.kubernetes.apiserverAddress = lib.mkDefault "https://${ 
  if cfg.apiserver.advertiseAddress != null then
    cfg.apiserver.advertiseAddress   # ← uses IP only, no port!
  else
    "${cfg.masterAddress}:${toString cfg.apiserver.securePort}"
}";
```

The kubelet kubeconfig (in `kubelet.nix:417`):
```nix
services.kubernetes.kubelet.kubeconfig.server = mkDefault top.apiserverAddress;
```

### Root cause
Setting `apiserver.advertiseAddress = "192.168.122.10"` (without port) causes `apiserverAddress` to be `https://192.168.122.10` — which defaults to port 443. But workers get `https://192.168.122.10:${securePort}` (6443). Neither port works for all kubeconfigs.

### Fix
Remove `advertiseAddress` from `master.nix`. Module constructs the full URL from `masterAddress:securePort`.

### Investigation technique: verify actual running config
```bash
# Check what flags kube-apiserver actually got:
systemctl cat kube-apiserver | grep -E 'secure-port|advertise'

# Check what port it's listening on:
ss -tlnp | grep -E '6443|443'

# Evaluate NixOS option value at build time (on host):
nix eval --json '.#nixosConfigurations.k8s-master-0.config.services.kubernetes.apiserver.securePort'

# Check NixOS module option defaults:
grep -A5 'securePort' /nix/store/.../apiserver.nix
```

---

## 2026-07-12: Worker cert distribution (cfssl API token)

### Symptom
Workers boot, certmgr starts, but immediately logs:
```
level=error msg="authentication error. Giving up without retries"
level=error msg="spec .../kubeletClient.json: continuing despite failed initial validation
  due to while fetching new keyPair: ... {"code":400,"message":"invalid token"}"
```
Kubelet and kube-proxy stay in "activating" state.

### Investigation technique: trace token flow
```bash
# Check where the token lives on master:
ls -la /var/lib/kubernetes/secrets/apitoken.secret
# → lrwxrwxrwx ... -> /var/lib/cfssl/apitoken.secret  (symlink!)

cat /var/lib/kubernetes/secrets/apitoken.secret
# → c2cd48beae7d52947534a6e273393ea4  (random, generated at boot)

# Check workers:
ls -la /var/lib/kubernetes/secrets/apitoken.secret
# → -rw------- ... (regular file, not symlink — no local cfssl)
```

### Investigation technique: read NixOS activation scripts
```bash
# Find the activation script source:
grep -n 'apitoken\|cfsslAPIToken' /nix/store/.../pki.nix

# Key lines:
# L35: cfsslAPITokenBaseName = "apitoken.secret";
# L36: cfsslAPITokenPath = "${config.services.cfssl.dataDir}/${cfsslAPITokenBaseName}";
# L37: certmgrAPITokenPath = "${top.secretsPath}/${cfsslAPITokenBaseName}";
# L193: if [ ! -f "${cfsslAPITokenPath}" ]; then ... random generation ...
# L215: if [ -f "${cfsslAPITokenPath}" ]; then ln -fs ... (symlink creation)
```

Token flow:
1. Master activation: generates random 32-char hex at `/var/lib/cfssl/apitoken.secret`
2. Master bootstrap service: symlinks to `/var/lib/kubernetes/secrets/apitoken.secret`
3. Worker bootstrap: creates empty file at secrets path
4. Workers need the token manually pasted in

### Fix
Pre-bake a fixed token into the NixOS image for all nodes:
1. Define `k8sApiToken` in `flake.nix`
2. Pass as `_module.args.k8sApiToken`
3. Write via `system.activationScripts` with `deps = [ "kubernetes-pki" ]`

### Warning: activation script ordering
`lib.mkAfter` does NOT control ordering between different activation scripts — it only appends text to the same script. To run after another script, use:
```nix
# WRONG — runs in unknown order relative to kubernetes-pki:
system.activationScripts.k8s-apitoken = lib.mkAfter ''...'';

# CORRECT — runs after kubernetes-pki:
system.activationScripts.k8s-apitoken = {
  text = ''...'';
  deps = [ "kubernetes-pki" ];
};
```

### Investigation technique: batch service status check
```bash
# Quick check all K8s services across all nodes:
for ip in 192.168.122.{10,11,12,13}; do
  echo "=== $ip ==="
  SSH_AUTH_SOCK= ssh -i "$SSH_KEY" root@$ip "systemctl is-active kubelet kube-proxy certmgr"
done
```

---

## 2026-07-12: General debugging techniques

### SSH into QEMU VMs
Bitwarden SSH agent blocks key-based auth. Always prefix with `SSH_AUTH_SOCK=`:
```bash
SSH_KEY="/home/file_magic/Projects/github/dotfiles/kubernetes/ssh-key"
SSH_AUTH_SOCK= ssh -i "$SSH_KEY" root@192.168.122.10
```

### Known hosts after redeploy
New VMs get new host keys. Clean up before SSH:
```bash
for ip in 192.168.122.{10,11,12,13}; do ssh-keygen -R "$ip" 2>/dev/null; done
```

### Read VM screen (when serial console fails)
```bash
virsh -c qemu:///system qemu-monitor-command k8s-master-0 --hmp "screendump /tmp/screen.ppm"
nix-shell -p tesseract --run "tesseract /tmp/screen.ppm stdout"
```

### libvirt VM lifecycle
Must use `qemu:///system` (not plain `virsh` which connects to user session):
```bash
# List running VMs:
virsh -c qemu:///system list --name

# Full cleanup sequence:
virsh -c qemu:///system destroy $vm        # force stop
virsh -c qemu:///system undefine $vm --nvram  # remove config + NVRAM
virsh -c qemu:///system vol-delete --pool default $vol.qcow2  # remove disk

# Note: --remove-all-storage fails with NVRAM; use --nvram instead
```

### OVMF VARS template
Must be exact size (540672 bytes for OVMF.fd) and zeroed:
```bash
OVMF_VARS_TEMPLATE=$(nix eval --raw nixpkgs#OVMF.fd)/FV/OVMF_VARS.fd
truncate -s "$(stat -c%s "$OVMF_VARS_TEMPLATE")" /tmp/empty-vars.fd
```

### tofulibvirt provider quirks
- `ovmf_code_path` is a directory; actual firmware at `${path}/FV/OVMF_CODE.fd`
- `libvirt_volume` uses `create.content.url` not `source`
- `os.firmware = "efi"` uses provider's own OVMF (may have stale vars) — use manual `loader`/`nv_ram` instead
- `features.acpi` is Boolean, not list
- `devices.graphics` VNC: remove explicit `port` when using `auto_port = true`
- `disk.driver.type` must be `"qcow2"` (provider defaults to `"raw"`)

### tofu binary location
```bash
/nix/store/2q5865pn7ygz8n1bjdzgsxxagbhpvisw-opentofu-1.12.3/bin/tofu
```

---

## 2026-07-13: Workers stuck "activating" — missing kubelet-client certs

### Symptom
- Master fully operational: cfssl, apiserver, kubelet all active
- Workers: certmgr active, kubelet/kube-proxy "activating" indefinitely
- `kubelet-client.pem` and `kubelet-client.pem` missing in `/var/lib/kubernetes/secrets/`
- certmgr log: `authentication error` → `invalid token`

### Root cause: `install -o cfssl` fails on workers
The `k8s-write-apitoken` service used `install -o cfssl` unconditionally:
```
install: invalid user 'cfssl'
```
Workers don't have the `cfssl` user — only the master does (created by the NixOS cfssl module). The service exits with error, token never gets written.

### Root cause: missing `/var/lib/cfssl` directory
Even after adding `id cfssl` guard, the service failed:
```
install: cannot create regular file '/var/lib/cfssl/apitoken.secret': No such file or directory
```
The `/var/lib/cfssl` directory is created by cfssl's preStart, but our service runs BEFORE cfssl (via `before = [ "cfssl.service" ]`).

### Fix
```nix
script = ''
  TOKEN_FILE=/tmp/k8s-apitoken.tmp
  printf '%s' "${k8sApiToken}" > "$TOKEN_FILE"
  # Master: write to cfssl dataDir (owned by cfssl user)
  if id cfssl >/dev/null 2>&1; then
    mkdir -p /var/lib/cfssl
    install -o cfssl -m 400 "$TOKEN_FILE" /var/lib/cfssl/apitoken.secret
  fi
  # All nodes: write to secrets path
  mkdir -p /var/lib/kubernetes/secrets
  install -m 600 "$TOKEN_FILE" /var/lib/kubernetes/secrets/apitoken.secret
  rm -f "$TOKEN_FILE"
'';
```

Key points:
- `id cfssl` guard prevents failure on workers
- `mkdir -p /var/lib/cfssl` before install (directory doesn't exist yet)
- `printf` + temp file instead of process substitution (systemd doesn't handle `<(...)`)

---

## 2026-07-13: Workers in disk-pressure — image too small

### Symptom
- All workers had `node.kubernetes.io/disk-pressure:NoSchedule` taint
- Root filesystem at 95% usage (4.7G / 5.3G)
- CoreDNS couldn't schedule anywhere
- `1.5G /var/lib/containerd` + `3.2G /nix` filled the 5.75 GiB image

### Root cause
NixOS image builder default `virtualisation.diskSize = "auto"` produced a 5.75 GiB image. With containerd images, NixOS system, and K8s components, that's too small.

### Fix
```nix
qemuGuestModule = {
  # ...
  virtualisation.diskSize = 20480; # 20 GiB in MiB
};
```

### Note on tofu/libvirt volume sizing
- `libvirt_volume` with `create.content.url` does NOT support `size` parameter
- Must set the disk size at NixOS image build time, not in tofu
- `qemu-img info result/*.qcow2` shows `virtual size` to verify

---

## 2026-07-13: Cilium CNI binary missing on workers after reboot

### Symptom
- Cilium pods Running, but `ip route` shows no Cilium routes
- `cilium-cni` binary missing from `/opt/cni/bin/` on workers
- Cilium init container writes to `hostPath`, but NixOS manages `/opt/cni/bin` with its own symlinks

### Technique: check CNI binary presence
```bash
# On each node:
ls -la /opt/cni/bin/cilium-cni 2>/dev/null || echo "MISSING"
```

### Fix
NixOS manages `/opt/cni/bin` with symlinks; Cilium's `hostPath` writes get lost. Solution:
1. Extract `cilium-cni` from the locally imported Cilium image with `ctr images mount`.
2. Run that extraction from a retrying systemd service; workers do not have an admin
   kubeconfig, so `kubectl cp` is not a valid node-local solution.

The service must retry until the Cilium image is imported. An earlier version exited
successfully after seeing an unrelated kube-system container, leaving every worker
without the plugin and preventing all new pod sandboxes from starting.

### Verification
`just verify` passed on 2026-08-19 with all nodes Ready, node IPv6 egress, and pod IPv6
HTTPS egress.

---

## 2026-08-19: IPv6 egress verification image curl arguments

### Symptom
- The IPv6 egress test pod started but exited with curl reporting
  `option --connect-timeout=8: is unknown`.

### Root cause
The curl build in the offline Ceph image accepts the timeout value as a separate
argument, not in the GNU-style `--option=value` form.

### Fix
Pass `--connect-timeout` and `8` as separate Kubernetes container arguments.

### Verification
The corrected test completed successfully through the host NAT66 path, and `just verify`
passed with `failed=0`.

---

## 2026-08-19: IPv6 host-to-local pod probes restart CoreDNS and CSI

### Symptom
- CoreDNS repeatedly restarted with exit code 0 after kubelet liveness probes to
  pod IPv6 addresses on port 10054 timed out.
- The ceph-csi controller manager repeatedly restarted after probes to pod IPv6
  port 8081 timed out.
- Cilium agents remained Running, while `cilium-health` reported host-to-local
  endpoint timeouts.

### Root cause
The IPv6 host path through `cilium_host` could not reliably deliver kubelet
health probes to local pod addresses. Opening Cilium's node health port 4240 did
not fix probes to the pod IPs, and enabling per-endpoint routes did not change
the behavior.

### Fix
- Run CoreDNS in the node network namespace with `ClusterFirstWithHostNet`.
- Run the CSI controller manager in the node network namespace with
  `ClusterFirstWithHostNet`, one replica, and a `Recreate` strategy so an
  unavailable replacement cannot leave an old ReplicaSet running.
- Allow ingress TCP 10054 and 8081 for the host-network health probes, and TCP
  and UDP 10053 for host-network CoreDNS DNS traffic.

### Verification
After a clean `just down` and `just up`, verification passed with `failed=0`.
CoreDNS and the CSI controller remained `1/1` with zero restarts after three
minutes. Cilium health endpoint warnings remain a separate observability issue.

---

## 2026-07-13: Rook-Ceph slow commands and mon quorum loss

### Symptom
- `ceph` commands hang (health, status, osd tree)
- mons fall out of quorum intermittently
- csi-rbdplugin pods in CrashLoopBackOff

### Root cause
containerd's `LimitNOFILE` (set to 1048576) causes Ceph daemons to open excessive file descriptors, slowing I/O and causing timeouts.

### Fix
```nix
systemd.services.containerd.serviceConfig.LimitNOFILE = lib.mkForce null;
```

---

## 2026-07-13: Rook-Ceph OSDs not found — no available devices

### Symptom
- OSD prepare pod runs but finds no usable devices
- `ceph osd tree` shows no OSDs
- OSD prepare log: skipping `sda1` (boot), `sda2` (rootfs) — all partitions in use

### Technique: check available block devices
```bash
lsblk -o NAME,SIZE,TYPE,MOUNTPOINT
# sda  20G  disk
# ├─sda1  1G  part /boot
# └─sda2 19G  part /nix
# No unused block devices → no OSDs possible
```

### Fix
Add a dedicated 40GB data disk per worker in Terraform (`libvirt_volume.worker_data`),
attached as `/dev/sdc`. The separate `/dev/sdb` 30GB disk is mounted at
`/var/lib/containerd`; NixOS does not partition the Ceph disk, and Rook-Ceph provisions
it as a raw OSD.

---

## 2026-07-13: CephConnection CRD missing — Rook v1.20.2 requires csi-operator.yaml

### Symptom
- Rook operator log: `no matches for kind "CephConnection" in version "csi.ceph.io/v1"`
- `crds.yaml` from Rook doesn't include `CephConnection`

### Fix
Apply `csi-operator.yaml` from Rook v1.20.2 examples (provides `CephConnection` CRD):
```bash
kubectl apply -f https://raw.githubusercontent.com/rook/rook/v1.20.2/deploy/examples/csi-operator.yaml
```

---

## 2026-07-13: Ceph version rejected — Rook v1.20.2 requires Squid

### Symptom
- OSD pods log: `failed the ceph version check: the version does not meet the minimum version "19.2.0-0 squid"`
- Using `quay.io/ceph/ceph:v18.2.0` (Reef)

### Fix
Rook v1.20.2 requires Ceph v19.2.0 (Squid):
```yaml
cephVersion:
  image: quay.io/ceph/ceph:v19.2.0
```

---

## 2026-07-13: NixOS kernel modules for Rook-Ceph

### Symptom
- Rook-Ceph CSI pods fail to mount RBD volumes
- Missing kernel modules: `rbd`, `ceph`

### Fix
Add to `common.nix`:
```nix
boot.kernelModules = [ "rbd" "ceph" ];
```

Also mount kernel modules and `/nix` into CSI plugin volumes (NixOS doesn't use standard FHS paths):
```nix
# In Rook-Ceph ConfigMap:
CSI_RBD_PLUGIN_VOLUME = ''
  - name: lib-modules
    hostPath:
      path: /run/current-system/kernel-modules/lib/modules/
  - name: host-nix
    hostPath:
      path: /nix
'';
CSI_RBD_PLUGIN_VOLUME_MOUNT = ''
  - name: lib-modules
    mountPath: /lib/modules
    readOnly: true
  - name: host-nix
    mountPath: /nix
    readOnly: true
'';
```

---

## 2026-07-13: Ansible kubernetes.core modules don't support async

### Symptom
- Tasks with `async: 300` / `poll: 10` fail silently or timeout
- `kubernetes.core.helm`, `kubernetes.core.k8s`, `kubernetes.core.k8s_info` ignore async directives

### Fix
Remove all `async`/`poll` from kubernetes.core tasks. Use `register` + `until` + `retries` + `delay` for polling instead.

---

## 2026-07-13: Ansible `role_path` undefined with `import_tasks`

### Symptom
- Tasks using `{{ role_path }}/files/...` fail with `undefined variable`
- Happens when using `import_tasks` (flat task files, no roles)

### Fix
Use `{{ playbook_dir }}` instead of `{{ role_path }}` for local file paths.

---

## 2026-07-13: KUBECONFIG must be set at play level for kubectl

### Symptom
- `kubectl` commands in tasks fail with `Unable to connect to the server`
- No kubeconfig in task environment

### Fix
Set at play level:
```yaml
- hosts: master
  environment:
    KUBECONFIG: /root/.kube/config
  tasks:
    - name: Setup kubeconfig
      ansible.builtin.shell: |
        mkdir -p /root/.kube
        cp /etc/kubernetes/cluster-admin.kubeconfig /root/.kube/config
      args:
        creates: /root/.kube/config
```

---

## 2026-07-13: Cilium pod label is `k8s-app`, not `app.kubernetes.io/name`

### Symptom
- `kubectl get pods -l app.kubernetes.io/name=cilium` returns nothing
- Cilium pods use `k8s-app=cilium` label

### Fix
```bash
kubectl get pods -l k8s-app=cilium -n kube-system
```

---

## 2026-07-13: SCP from master to workers fails — remote path resolution

### Symptom
- `ansible.builtin.copy` with `src: "{{ playbook_dir }}/..."` works on master but fails on workers
- `{{ playbook_dir }}` resolves on the control machine, but the file doesn't exist on the remote

### Fix
Use `ansible.builtin.fetch` to pull the file to the control machine, then `ansible.builtin.copy` with `delegate_to`:
```yaml
- name: Fetch from master
  ansible.builtin.fetch:
    src: /opt/cni/bin/cilium-cni
    dest: /tmp/cilium-cni
    flat: true
  delegate_to: "{{ inventory_hostname }}"

- name: Copy to workers
  ansible.builtin.copy:
    src: /tmp/cilium-cni
    dest: /opt/cni/bin/cilium-cni
    mode: "0755"
  loop: "{{ groups['workers'] }}"
  delegate_to: "{{ item }}"
```

---

## 2026-07-13: Ansible `values_files` doesn't template Jinja2

### Symptom
- Helm values file sent to remote with raw Jinja2 variables (`{{ master_ip }}`)
- Helm install fails because the template was never rendered

### Fix
Use `values:` dict (inline templated values) instead of `values_files:` for values that contain Jinja2 variables.

---

## 2026-07-14: CSI staging path does not exist on node

### Symptom
- PVC stays Pending, pod stuck in ContainerCreating
- CSI node plugin logs: `staging path does not exist on node`
- kubelet passes staging path `/var/lib/kubernetes/plugins/kubernetes.io/csi/.../globalmount`

### Root cause
NixOS defaults kubelet `--root-dir` to `/var/lib/kubernetes`. CSI node plugin containers mount `/var/lib/kubelet/plugins` as a hostPath. When kubelet passes a staging path under `/var/lib/kubernetes/plugins/...`, the CSI container can't see it because it only mounts `/var/lib/kubelet/plugins`.

### Fix
```nix
services.kubernetes.dataDir = "/var/lib/kubelet";
```
This makes kubelet use `--root-dir=/var/lib/kubelet`, matching what CSI containers expect.

### Technique: verify kubelet root-dir
```bash
cat /proc/$(pgrep kubelet)/cmdline | tr "\0" "\n" | grep root-dir
# Should show: --root-dir=/var/lib/kubelet
```

---

## 2026-07-14: clusterDns wrong on workers

### Symptom
- CoreDNS pods running but worker pods can't resolve service names
- Workers get `clusterDns = ["10.0.0.254"]` but CoreDNS ClusterIP is `10.96.0.254`

### Root cause
Workers don't set `apiserver.serviceClusterIpRange`. The NixOS module computes `clusterDns` from the default CIDR `10.0.0.0/24`, giving `10.0.0.254`. The master's actual service CIDR is `10.96.0.0/12`, putting kube-dns at `10.96.0.254`.

### Fix
```nix
# In common.nix
services.kubernetes.addons.dns.clusterIp = "10.96.0.254";
```
Remove `clusterDns` from `master.nix` and `worker.nix` (let common.nix handle it).

---

## 2026-07-14: CSI operator not deploying node plugin DaemonSets

### Symptom
- CephCluster Ready, but no CSI node plugins on workers
- Volumes provisioned but can't be staged/attached
- `ROOK_USE_CSI_OPERATOR: "false"` in operator ConfigMap

### Root cause
With `ROOK_USE_CSI_OPERATOR: "false"`, the Rook operator only creates the CSI controller, not node plugin DaemonSets. The CSI operator is responsible for creating Driver CRDs which deploy DaemonSets.

### Fix
```yaml
ROOK_USE_CSI_OPERATOR: "true"
```
Apply `csi-operator.yaml` BEFORE Driver/OperatorConfig CRDs (CRD ordering dependency).

---

## 2026-07-14: CSI operator CRD ordering dependency

### Symptom
- Applying Driver/OperatorConfig CRDs fails: `Failed to find exact match for csi.ceph.io/v1.OperatorConfig`

### Root cause
The `csi-operator.yaml` defines CRDs (like `CephConnection`, `OperatorConfig`, `Driver`) that other resources reference. If you apply resources before the CRDs exist, Kubernetes can't validate them.

### Fix
Apply `csi-operator.yaml` first, then apply Driver CRDs, ImageSet ConfigMap, and OperatorConfig.

### Technique: extract resources from multi-document YAML
```python
# In Ansible shell task:
import yaml
with open('/tmp/rook-operator.yaml') as f:
    docs = list(yaml.safe_load_all(f))
# Extract Deployment separately, CSI resources separately, skip operator ConfigMap
csi_resources = [d for d in docs if d and d.get('kind') != 'Deployment'
                 and not (d.get('kind') == 'ConfigMap'
                          and d.get('metadata', {}).get('name') == 'rook-ceph-operator-config')]
```

---

## 2026-07-14: StorageClass missing CSI secret references

### Symptom
- PVC Pending, CSI provisioner log: `provided secret is empty`
- StorageClass has no `csi.storage.k8s.io/provisioner-secret-name` parameters

### Root cause
The CSI provisioner needs Kubernetes secrets to authenticate with Ceph for creating/deleting RBD images. Without the secret references in the StorageClass, it sends empty credentials.

### Fix
Add all four secret references to the StorageClass:
```yaml
parameters:
  csi.storage.k8s.io/provisioner-secret-name: rook-csi-rbd-provisioner
  csi.storage.k8s.io/provisioner-secret-namespace: rook-ceph
  csi.storage.k8s.io/node-stage-secret-name: rook-csi-rbd-node
  csi.storage.k8s.io/node-stage-secret-namespace: rook-ceph
```

### Warning: StorageClass parameters are immutable
You must delete and recreate the StorageClass to change parameters:
```bash
kubectl delete sc rook-ceph-block
# Then reapply with correct parameters
```

---

## 2026-07-14: CSI provisioner DeadlineExceeded followed by "operation already exists"

### Symptom
- First CreateVolume attempt times out: `rpc error: code = DeadlineExceeded desc = context deadline exceeded`
- Subsequent retries fail: `rpc error: code = Aborted desc = an operation with the given Volume ID already exists`
- PVC stays Pending indefinitely

### Root cause
The CSI driver successfully creates the RBD image on Ceph, but the operation takes >2.5 minutes (the provisioner sidecar's default deadline). The sidecar times out and reports failure, but the volume IS created on Ceph. The internal operation tracker still holds the in-progress state, so all retries get `Aborted`.

### Fix
Restart the CSI controller pod(s) to clear the stuck operation tracker:
```bash
kubectl -n rook-ceph delete pod rook-ceph.rbd.csi.ceph.com-ctrlplugin-<pod-name>
```
The StatefulSet/ReplicaSet recreates the pod with a fresh operation tracker. The PVC will be retried automatically.

### Why it was slow
On first boot, Ceph OSDs and monitors are still initializing. The CSI driver may need to wait for OSDs to be ready before it can create images. Subsequent volume creations are fast (<1 second) once Ceph is warmed up.

---

## 2026-07-14: CSI attacher leader lease held by deleted pod

### Symptom
- VolumeAttachment stays `ATTACHED=false`
- Pod stuck in ContainerCreating after PVC is Bound
- CSI attacher sidecar log: `Attempting to acquire leader lease...` indefinitely
- Lease holder is a deleted pod name

### Root cause
When CSI controller pods are deleted/restarted, the leader lease for the attacher sidecar may still be held by the old pod. The lease expires after 137 seconds (`leaseDurationSeconds`), but during that window the new pod can't attach volumes.

### Fix
Wait for the lease to expire (~2-3 minutes), or delete both CSI controller pods to force simultaneous lease expiry and re-election:
```bash
kubectl -n rook-ceph delete pod -l app=rook-ceph-rbdplugin-provisioner
```

### Technique: check lease state
```bash
kubectl -n rook-ceph get lease external-attacher-leader-rook-ceph-rbd-csi-ceph-com \
  -o jsonpath="holder={.spec.holderIdentity} renew={.spec.renewTime}"
```

---

## 2026-07-14: /lib/modules symlink needed for CSI node plugins on NixOS

### Symptom
- CSI node plugins fail to load kernel modules
- NixOS stores modules at `/run/current-system/kernel-modules/lib/modules/` (non-standard path)

### Fix
Add systemd tmpfiles rule in `common.nix`:
```nix
systemd.tmpfiles.rules = [
  "L /lib/modules - - - - /run/current-system/kernel-modules/lib/modules"
];
```

---

## 2026-08-13: Boot-blocker diagnosis without a working serial console (13-inch-thin-cannon)

### Symptom
- Installed 13-inch-thin-cannon boots to a LUKS prompt, passphrase is **accepted** (no
  re-prompt), then the serial console goes silent. SSH never comes up (watched to 240 s).
- Serial log frozen at ~3.3 KB (prompt + masked bullets). VGA screendump looks
  near-black to a human eye. Looks exactly like a kernel hang.

### Technique: QEMU monitor screendump + tesseract OCR (the breakthrough)
When the serial console is silent but the kernel is alive, read the **VGA screen**
via the QEMU monitor and OCR it. This is the non-libvirt analogue of the existing
`virsh qemu-monitor-command screendump` trick:

```bash
# 1. Screendump from the QEMU monitor TCP port (MONITOR_PORT=5555):
printf 'screendump /tmp/opencode/screen.ppm\n' | timeout 3 nc 127.0.0.1 5555
#    (or: bash -c 'exec 3<>/dev/tcp/127.0.0.1/5555; echo "screendump /tmp/opencode/screen.ppm" >&3')

# 2. OCR with tesseract (path is a pinned nix-store binary):
/nix/store/2ac32mdm2rdcy5fx6pma5jckpnfa2pkz-tesseract-5.5.2/bin/tesseract \
  /tmp/opencode/screen.ppm stdout --psm 6

# 3. Optional: scale up 2x first (tesseract needs ~10px glyphs):
/nix/store/clx4xcbrg16dhzyzm2faxjddfk9rz979-imagemagick-7.1.2-27/bin/magick \
  /tmp/opencode/screen.ppm -resize 200% -colorspace Gray /tmp/opencode/screen2x.ppm
```

What OCR revealed (the serial console never printed these):
```
[FAILED] Failed to start Find NixOS closure.
[DEPEND] Dependency failed for Initrd Default Target.
You are in emergency mode. After logging in, type "journalctl -xb"...
```
The system was **NOT hung** — it had dropped to the initrd emergency shell on VGA.
Root cause chain: the disk had **no `/nix` btrfs subvolume**, so the Nix store lived
inside `/root`, which the impermanence rollback wipes and replaces with `root-blank`
every boot. `initrd-find-nixos-closure` (`RequiresMountsFor = /sysroot/nix/store`,
nixpkgs `initrd.nix:688`) had no mount unit for that path → dependency failed →
emergency shell. Fix: a persistent `/nix` subvolume (disko.nix) + `neededForBoot = true`
(→ `x-initrd.mount`) so the initrd mounts the store before the closure finder runs.

### Technique: serial SysRq trigger to prove kernel liveness
When "no output after unlock" looks like a hang, send a SysRq over the serial console
via the QEMU monitor to prove the kernel is alive:

```bash
printf 'sendkey ctrl-alt-sysrq-t\n' | timeout 3 nc 127.0.0.1 5555
# 40 s later the kernel echoed to serial:  "sysrq: Show State"
```
`sysrq: Show State` is the expected serial-side header. Note the **task dump itself
went to VGA (tty0), not serial** — the `console=ttyS0 console=tty0` kernel param sends
the dump to both, but systemd had already switched the display path, so OCR the VGA
screendump to read the dump. Combined with a QEMU `info registers` (three samples,
`HLT=1` in EFLAGS = idle loop, not a spin/panic), this distinguishes "hung kernel"
from "boot failed, console not echoing".

### Technique: PPM → ASCII renderer for quick screen eyeball
`/tmp/opencode/ppm2ascii.sh` renders a QEMU screendump as ASCII art with color-coded
glyph blocks, letting you "see" the VGA layout in a terminal without opening the image
(helpful to find the region OCR should target and to confirm text-vs-blank).

## 2026-08-13: Intermittent initrd watchdog reboot (13-inch-thin-cannon)

### Symptom
- The **first** cold boot of the installed 13-inch-thin-cannon VM sometimes dies
  ~1-2 s into the initrd (right after the LUKS prompt is unlocked). Serial log:
  ```
  [   10.827277] kvm_intel: VMX not supported by CPU 7
  [   14.599504] watchdog: watchdog0: watchdog did not stop!
  [   14.836988] watchdog: watchdog0: watchdog did not stop!
  [   15.322382] watchdog: watchdog0: watchdog did not stop!
  [   15.328746] reboot: Restarting system
  ```
  The VM reboots itself; the next boot always survives. Intermittent — the same
  disk/binary sometimes boots straight through to SSH (verified working).

### Root cause
- QEMU's **q35 machine emulates an Intel ICH9 TCO watchdog** (`iTCO_wdt` driver).
  Real 13-inch-thin-cannon hardware has no such device, so this is a harness
  artifact, not a config bug.
- Kernel commit `1ae3e78c` ("watchdog: iTCO_wdt: No need to stop the timer in
  probe") stopped the driver from clearing NO_REBOOT / stopping the TCO timer at
  probe. QEMU ≥ 8.0 enables the TCO reboot-pin strap by default (commit `a6b6414`),
  so an armed TCO genuinely resets/reboots the guest.
- During the delayed initrd (the LUKS prompt holds the kernel ~10 s), the watchdog
  device gets opened and released before the driver can stop it → kernel watchdog
  core prints `watchdog did not stop!` and falls through to `emergency_restart()`
  (`reboot: Restarting system`). The LUKS delay is why the *first* boot dies — the
  prompt pushes the release past the TCO window.
- Intermittency: the QEMU TCO timer state at power-on varies, and a fast initrd
  beats the window (verified: surviving boots log `iTCO_wdt ... initialized.
  heartbeat=30 sec`).

### Fix (status: verdict — NOT a QEMU TCO hardware reset, 2026-08-13)
- Tested: booting with `-global ICH9-LPC.noreboot=on` (disables the q35
  ICH9-TCO/chipset reset) **does NOT prevent the reboot** — the boot still died
  with `watchdog: watchdog0: watchdog did not stop!` ×3 → `reboot: Restarting
  system`. Therefore the reboot is a **guest-initiated software reboot**
  (kernel `emergency_restart()` from the watchdog device-release path when the
  HW_RUNNING TCO can't be stopped), not a QEMU TCO hardware reset. The flag only
  ever guards the chipset-reset side, which is not the path being taken.
- Open question (not yet root-caused): **who opens/closes `/dev/watchdog` in the
  initrd** around the kvm module load, triggering the three release attempts.
  Suspected: systemd/udev probing the freshly-registered watchdog device during
  the delayed (LUKS-held) initrd.
- Candidate fixes, in preference order (all still unapplied):
  1. Harness tolerance: the `boot` phase already does responsive unlock + a long
     SSH wait, so it survives the one-time self-reboot (next boot always
     survives). Verify with repeated `just boot` runs.
  2. Guest-side: `boot.blacklistedKernelModules = [ "iTCO_wdt" ]` so the driver
     never probes (RHEL-documented mitigation). Diverges from real hardware
     config, so only use if the harness-tolerance route is insufficient.
  3. Real-hardware note: 13-inch-thin-cannon has no TCO; this bug is purely a
     QEMU harness artifact and will not occur on physical hardware.
- **Status update (2026-08-13, end of session):** harness tolerance confirmed
  sufficient — fresh install + 3× repeated cold `just boot` runs all green
  (only the expected wireguard WARN). No kernel blacklist needed.
- **Related but distinct:** a *separate* intermittent "SSH dead on some boots"
  failure was root-caused to a **systemd ordering cycle** (swapfile service
  `Before=swap.target` → systemd deleted `run-wrappers.mount` → missing
  `unix_chkpwd` → sshd PAM denial). Full write-up in
  `13-inch-thin-cannon/docs/iso-build.md` (Known issues & gotchas).

### Technique: distinguishing the reboot type (tested 2026-08-13)
- `-global ICH9-LPC.noreboot=on`: stops TCO/hardware-chipset resets. Applied
  and the reboot **persisted** → proves the reboot is a guest software
  `emergency_restart()`, not the emulated TCO.
- `-no-reboot`: VM exits instead of restarting on a guest-initiated reboot
  (kernel `emergency_restart`/reboot syscall). If a reboot survives the `noreboot`
  flag but is caught by `-no-reboot`, it is a software reboot, not TCO.

---

## 2026-08-14: Thin-cannon password provisioning and test harness

### Symptom

- The ISO built and installed successfully, but the installed `file_magic`
  password-login check failed.
- Key-based SSH worked, while the password test was initially misleading: the
  account was present but locked, or the client closed during authentication.

### Root cause

- The host uses `users.mutableUsers = true` by default in the shared hardened
  user module. `disko-install` creates users before activation, so activation
  preserved the install-time locked shadow entry instead of applying the
  declarative password hash.
- The password target originally passed its stdin directly to `agenix` after
  reading the two password prompts. When driven by a pipe, those reads
  consumed the input and agenix encrypted an empty value instead of the
  generated hash.
- The repository intentionally ignores `secrets/*.age`, so pure flakes omit
  the encrypted password file and `builtins.pathExists` evaluates false. No
  password secret is present in a fresh ISO; it must be provisioned out of band
  after the target host has generated its SSH host key.
- The installed host is rebuilt from the baked flake by `disko-install`; an
  impure `builtins.getEnv` value used while building the ISO is not automatically
  available to that later rebuild.
- Password experiments also crossed two independent shell boundaries: the
  installed user uses Nushell, while the test commands use POSIX shell syntax.
  Commands such as `2>&1` therefore cannot be sent directly through that user's
  login shell.

### Fix and current policy

- Set `users.mutableUsers = false` on `13-inch-thin-cannon` so the declarative
  agenix-backed yescrypt hash is applied during installation.
- Store the generated hash in a mode-600 temporary file before invoking
  agenix, so both interactive and piped password entry encrypt the same hash.
- Keep encrypted `.age` files out of version control; provision them on the
  target host after installation.
- Keep production SSH key-only. Provision the user's password with
  `just set-file-magic-password`, which creates
  `secrets/file_magic-password.age`; never put a plaintext password in Nix
  configuration.
- The harness logs in with the existing `file_magic` key and runs installed-user
  commands through Bash, leaving Nushell as the production login shell.
- The temporary password-authentication experiment was removed. Repeated QEMU
  runs showed that its `sshpass` result was not a reliable end-to-end signal;
  the source of truth for ISO health is the key-based `iso-test.sh` verification.
- On 2026-08-17, piped password provisioning followed by `just up` passed all
  live, install, and boot phases.
- On 2026-08-17, provisioning the password encrypted to the VM's generated SSH
  host key decrypted successfully and password SSH login passed; restoring
  key-only SSH then rejected password login while key login still passed.

### Useful techniques

- Check whether a declarative account is locked with `passwd -S <user>`.
- Inspect the generated user definition with the system's `*-users-groups.json`
  closure path, without printing password hashes.
- When debugging installed-user remote commands, invoke POSIX commands through
  `bash -c` rather than relying on the user's interactive shell parser.

---

## OVMF Path Discovery

After NixOS rebuild, the OVMF path changes. Find it:
```bash
virsh dumpxml k8s-master-0 | grep -i ovmf
# Use the path from the <loader> element

# Or build the flake's pinned OVMF (realizes the path, see 2026-08-01 entry):
nix build --no-link --print-out-paths .#ovmf
# -> /nix/store/<hash>-OVMF-<ver>-fd  (contains FV/OVMF_CODE.fd + FV/OVMF_VARS.fd)
```
Do NOT use `nix eval --raw nixpkgs#OVMF.fd` — it prints the store path but does not
realize it, so `FV/OVMF_CODE.fd` does not exist unless OVMF was previously built.

## virsh UEFI VM Management

```bash
# Undefine UEFI VMs requires --nvram flag (no path argument):
virsh -c qemu:///system undefine k8s-master-0 --nvram

# List VMs and their state:
virsh -c qemu:///system list --all
```

---

## 2026-08-01: `tofu destroy` leaves k8s-* disks/networks behind

### Symptom
- After `just up` then `just down`, libvirt still had orphaned resources: an active
  `k8s-1` NAT network with autostart, while the tofu state was empty and no domains/volumes
  were tracked.
- `tofu destroy` itself correctly deletes tracked volumes (verified: 13 resources,
  including `worker_data` raw disks, fully removed). The leftovers were resources that
  had drifted out of the tofu state (interrupted/failed applies, or switching
  `CLUSTER_INDEX`/`WORKER_COUNT` between runs).

### Root cause
1. **State drift.** Volumes/domains are named without a cluster index
   (`k8s-master-0.qcow2` in the shared `default` pool) while the network IS index-scoped
   (`k8s-${index}`). If an apply is interrupted or fails after libvirt objects are created
   but before tofu records them, or if you switch `CLUSTER_INDEX`/`WORKER_COUNT`, objects
   exist in libvirt but are invisible to `tofu destroy` — so they are never deleted.
2. **`nix eval --raw nixpkgs#OVMF.fd` returns an unrealized store path.** `tofu apply`
   then fails at `libvirt_domain` creation ("Failed to Start Domain") after the disks and
   network were already created, leaving a half-applied state and forcing manual cleanup.
   This is what caused the initial orphaned disks: a `just up` whose apply died partway.

### Fix
- Root cause was the OVMF path: it is now realized via the flake
  `nix build --no-link --print-out-paths .#ovmf` (new `ovmf` package output in
  `flake.nix`) instead of `nix eval --raw`. Applies no longer fail at domain start, so
  tofu state stays consistent and `tofu destroy` removes everything.
- `just down` stays a pure `tofu destroy` (no virsh side-commands): tofu state is the
  single source of cleanup. A virsh prune was tried as a workaround but removed —
  the goal is that a failure like this should be fixed at its root, not papered over
  with separate manual cleanup.

### Technique notes (not currently used)
- `virsh vol-list <pool>` does **not** support `--name` (unlike `list`/`net-list`).
  Parse the name column: `virsh vol-list --pool default | awk 'NR>2 && $1 != "" {print $1}'`.
- `virsh undefine <dom> --nvram` also removes the OVMF NVRAM file.
- When `tofu destroy` reports nothing destroyed but libvirt still has `k8s-*` objects,
  the state was drifted by a failed apply; re-apply once (or recover state) so destroy
  can track them again.

---

## 2026-08-01: ansible fails with missing `/var/lib/kubelet/secrets/cluster-admin.pem` on fresh cluster

### Symptom
- `just up` succeeded through deploy + wait-ssh, then the Cilium Helm task failed:
  `kubernetes cluster unreachable: invalid configuration: unable to read client-cert
  /var/lib/kubelet/secrets/cluster-admin.pem: no such file or directory`.
- The certs existed minutes later — this is a boot race, not a config error.

### Root cause
The NixOS `kubernetes` module with `easyCerts = true` generates the cluster PKI at boot:
the master's `cfssl` service + `kube-certmgr-bootstrap` create `/var/lib/kubelet/secrets/*.pem`.
The ansible playbook only waited for SSH (`wait_for_connection`), so on a fast-booting
VM (SSH ready in ~10s) ansible ran helm before cfssl finished generating certs.

### Fix
`ansible/site.yml` gained a "Wait for Kubernetes API server" play before Cilium install:
`kubectl get --raw /readyz` with `KUBECONFIG=/etc/kubernetes/cluster-admin.kubeconfig`,
retried 60 × 5s until rc==0 (per AGENTS.md: `until` + `retries` + `delay`). This waits
for both the certs AND the apiserver to be healthy. Use `ansible.builtin.command` +
task-level `environment:` (not `shell`) to satisfy ansible-lint's
`command-instead-of-shell` rule.

### Technique: SSH around stale known_hosts
After redeploys, the VM host key changes and ssh aborts on the fingerprint mismatch
(see the `Offending RSA key` warning). Use `-o UserKnownHostsFile=/dev/null` for manual
diagnostic SSH; the justfile already sets `-o StrictHostKeyChecking=no`.

---

## 2026-08-02: kube-controller-manager crash loop "New CIDR set failed; the node CIDR size is too big"

### Symptom
- After enabling dual-stack, `kube-controller-manager` crash-looped: systemd
  `RestartSec=30s` + RestartSec, leader-election events flapped every ~40s, and **no
  pods were created at all** (deployments/daemonsets had 0 ready, `kubectl get pods -n
  kube-system` = empty). No Cilium pods → nodes stuck NotReady.
- Controller-manager log: `Error initializing a controller ... err="New CIDR set
  failed; the node CIDR size is too big" controller="node-ipam-controller"`.

### Root cause
The k8s `nodeipam` controller splits each `--cluster-cidr` into per-node subnets with
`cidrset.NewCIDRSet`:

```go
if (clusterCIDR.IP.To4() == nil) && (subNetMaskSize-clusterMaskSize > clusterSubnetMaxDiff) {
    return nil, ErrCIDRSetSubNetTooBig   // clusterSubnetMaxDiff = 16
}
```

For **IPv6, node mask − cluster mask must be ≤ 16**. Default node mask is `/64`, so the
IPv6 cluster CIDR must be ≥ `/48`. My first config used `fd04::/64` cluster CIDR with
`--node-cidr-mask-size-ipv6=112` (diff 48) → crash.

### Fix
Set the v6 node-IPAM cluster CIDR to `/48` (`fd04::/48`) and drop the explicit
`--node-cidr-mask-size-ipv6` (defaults to `/64`, diff 16 ≤ 16).

### Technique
- Crash-looping control plane = "no pods anywhere", which looks like a scheduling
  problem but is actually the ReplicaSet/deployment controllers never starting.
  Diagnose the controller first (`journalctl -u kube-controller-manager`), not the pods.
- The k8s IPv6 `/48` constraint: [cidr_set.go](https://github.com/kubernetes/kubernetes/blob/v1.36.2/pkg/controller/nodeipam/ipam/cidrset/cidr_set.go)
  comment — "the subnet mask for IPv6 cluster cidr needs to be >= 48".

---