nix
nix and nixos configs forked from filemagic.tngl.sh/nix
Star
Fork
Watch
nix kubernetes docs debugging.md
54 kB · 1363 lines
Markdown
at main

Debugging Notes #

Live notes from debugging the NixOS K8s QEMU image build and boot process.

2026-08-21: kubelet preStart wipes /opt/cni/bin, breaking all new pods #

Symptom #

  • Cluster healthy for ~11h, then every newly created pod stuck in ContainerCreating with FailedCreatePodSandBox: plugin type="cilium-cni" failed (add): failed to find plugin "cilium-cni" in path [/opt/cni/bin].
  • Existing pods unaffected; nodes Ready; Cilium pods Running.
  • /opt/cni/bin/cilium-cni missing although install-cilium-cni.service had succeeded at boot (exit 0).

Root cause #

Two interacting behaviors:

  1. The NixOS kubernetes module's kubelet ExecStartPre runs rm /opt/cni/bin/* || true on every kubelet start, then re-links only the NixOS cniPluginPackages. Any foreign binary is destroyed.
  2. certmgr periodically refreshes kubelet certs ("spec DNS name has changed") and restarts kubelet via systemctl restart kubelet.service.

So each cert rotation wiped cilium-cni, and the boot-time-only extraction service never ran again.

Fix #

Make install-cilium-cni self-healing (common.nix):

  • Added a systemd .path unit (systemd.paths.install-cilium-cni) with PathModified = /opt/cni/bin triggering the service on any directory change.
  • Service fast-exits when the binary exists; otherwise extracts it from the imported Cilium image via ctr images mount as before. Boot-time wantedBy retained so first-boot extraction still waits out the image-import window.

Technique #

  • Pod stuck in ContainerCreating ≠ image problem: read pod events (kubectl describe pod) before checking images.
  • When a file mysteriously disappears, check whether any unit's ExecStartPre recreates its directory: journalctl -u <unit> around the file's mtime, and read the actual store-path script (cat $(systemctl show -p ExecStartPre --value <unit> | cut -d';' -f1)).

2026-08-21: Pre-flight validating rendered manifests without mutating #

Technique #

Before switching an application to a new deployment tool (e.g. Nixidy), validate the rendered output against a live API without side effects:

kubectl apply --dry-run=server -f <manifest>   # per file; exit code matters

This catches schema validation errors, missing API groups, and admission rejections that client-side dry-runs miss. Remember KUBECONFIG must be set in the same shell or kubectl silently falls back to http://localhost:8080.

2026-07-11: Image builds but VM never boots via EFI #

Symptom #

  • nix build .#k8s-master-0 produces a valid 3.4GB QCOW2 EFI image
  • Booting with OVMF + systemd-boot: zero serial output, SSH never connects
  • VM process stays alive, port 2222 is always open (QEMU user-networking opens it immediately regardless of guest state)
  • After 4+ minutes, still no SSH and no serial output

What works #

  • nixos-rebuild build-vm boots fine (SSH connects, auth denied since no key)
  • Direct kernel boot (-kernel/-initrd flags) from the same system closure starts the kernel

Root cause (identified, not yet fixed) #

The qemu-guest.nix profile imported at flake.nix:15:

qemuGuestModule = {
  imports = [ "${toString nixpkgs}/nixos/modules/profiles/qemu-guest.nix" ];
  ...
};

This adds 9p virtio filesystem configuration to the initrd. When booting as a standalone disk image (no 9p virtio hosts), the initrd tries to mount nix-store, shared, xchg via 9p, fails, drops to emergency shell with locked root — system never reaches multi-user.

Direct kernel boot output showing the failure:

9pnet_virtio: no channels available for device nix-store
FAILED Failed to mount /sysroot/nix/.ro-store
Dependency failed for /sysroot/nix/store
You are in emergency mode. After logging in, type "journalctl -xb" to view system logs.
Cannot open access to console, the root account is locked.

ESP is valid #

Parsed the GPT + FAT32 ESP manually. Contents:

  • Partition 1: ESP, LBA 16384-526335, 249MB
  • Partition 2: Linux filesystem, 5.5GB
  • ESP has BOOTX64.EFI, systemd-boot 261, NixOS directory, kernel, initrd, config entries
  • loader.conf: #timeout 3, #console-mode keep (commented out = defaults)
  • systemd-boot binary contains expected strings, version 261

OVMF serial behavior #

OVMF does NOT output to serial when using -nographic. Firmware/bootloader messages only appear on VGA framebuffer. To see OVMF output you need:

  • -vnc :N or a display, OR
  • -debugcon file:/path/to/log -global isa-debugcon.iobase=0x402 for UEFI debug log

This means you cannot debug EFI boot failures over serial alone.

-cpu host required #

The default QEMU CPU (qemu64) lacks features the kernel requires. Use -cpu host when booting with KVM. Without it, the kernel prints:

This kernel requires the following features not present on the CPU:
Unable to boot - please use a kernel appropriate for your CPU.

OVMF vars lock after first use #

QEMU locks the OVMF VARS file after a boot session. Must copy fresh from nixpkgs each time and chmod +w:

OVMF_FD=$(nix eval --raw nixpkgs#OVMF-xen.fd)
rm -f /tmp/k8s-vars.fd
cp -f "$OVMF_FD/FV/OVMF_VARS.fd" /tmp/k8s-vars.fd
chmod +w /tmp/k8s-vars.fd

OVMF CODE file is fine with readonly=on.

QEMU port forwarding is misleading #

QEMU user-networking hostfwd opens the host port immediately when QEMU starts, regardless of guest state. An open port does NOT mean the guest has booted or sshd is running. Only use actual SSH connection attempts (or serial output) to determine boot status.

timesyncd.enable needs lib.mkDefault #

The vmVariant NixOS module sets services.timesyncd.enable = false. Our config set it to true without priority, causing a conflict when building build-vm:

The option `virtualisation.vmVariant.services.timesyncd.enable' has conflicting definition values

Fix: use lib.mkDefault true in common.nix:54.

nixos-rebuild build-vm test command #

This is the quickest way to verify the config boots:

cd kubernetes/
QEMU_NET_OPTS="hostfwd=tcp::2222-:22" nixos-rebuild build-vm --flake .#k8s-master-0
# Find the run script in the nix store, run it
# SSH with: ssh -o StrictHostKeyChecking=no -p 2222 root@localhost
# (auth will fail — no password/key configured — but sshd running = success)

Note: this bypasses the bootloader (uses -kernel/-initrd directly) and includes 9p virtio shares, so it doesn't test the standalone image boot path.

2026-08-19: IPv6-only Cilium bootstrap deadlock #

Symptom #

  • Cilium init containers repeatedly failed with timeouts to https://[fd03::1]:443.
  • Nodes had IPv6 addresses, but all nodes remained NotReady and CoreDNS stayed Pending.

Root cause #

The Kubernetes Service ClusterIP is not routable until Cilium programs the service datapath. Cilium's own init container therefore cannot use the in-cluster API Service to bootstrap an IPv6-only cluster.

Fix #

Set Cilium k8sServiceHost: k8s-master-0 and k8sServicePort: 6443, using the direct IPv6-resolved master address during bootstrap.

Technique #

Inspect the config init-container log with kubectl logs -n kube-system <cilium-pod> -c config --previous and check whether it is dialing fd03::1 or the master address.

2026-08-19: CoreDNS IPv6 certificate and forwarding loop #

Symptom #

  • CoreDNS crashed with plugin/loop.
  • CoreDNS could not watch Kubernetes resources because the API certificate did not include fd03::1.

Root cause #

The API Service is IPv6 ClusterIP fd03::1, but easyCerts did not derive that SAN from the IPv6 service CIDR. CoreDNS also forwarded external queries to the node's /etc/resolv.conf, which points at the CoreDNS Service itself.

Fix #

Add fd03::1 through apiserver.extraSANs and set CoreDNS forward to explicit IPv6 upstream resolvers instead of /etc/resolv.conf.

2026-08-19: Offline image archive reference mismatch #

Symptom #

  • ctr images import succeeded, but kubelet still attempted to pull Cilium images by their digest and failed because the nodes have no internet.

Root cause #

The old Podman archive preserved tags but not the digest-qualified references emitted by the Helm chart. Podman also collapsed multiple podman save inputs into one image with multiple tags on this host.

Fix #

Deploy Cilium's imported version tags with image.useDigest, envoy.image.useDigest, and operator.image.useDigest disabled. Build one Docker archive per image and import each archive independently on every node.

2026-08-19: Offline Rook manifests and Ceph convergence #

Symptom #

  • Rook installation failed because the master tried to fetch GitHub manifests from the offline IPv6-only network.
  • Ceph remained Progressing while the dashboard module hung creating its admin user.

Fix #

Download Rook CRD, common, CSI-operator, and operator manifests on the control machine, copy them to the master, and apply the local files. Disable the optional Ceph dashboard for this storage-only test cluster so Rook can reach HEALTH_OK.

2026-08-19: ULA IPv6 internet egress requires host NAT66 #

Symptom #

  • The host could ping public IPv6, while nodes routed through fd7a::1 but public IPv6 ping and HTTPS failed.

Root cause #

The libvirt bridge provides private ULA addressing but does not provide IPv6 NAT for the guest subnet. Public upstream routing cannot accept fd00::/8 source addresses.

Fix #

Configure host NixOS networking.nat.enableIPv6 = true with internalIPv6s = ["fd00::/8"] and the active external interface, then activate the host system.

Verification #

Run just verify; it checks public IPv6 reachability from every node and from a temporary IPv6 HTTPS test pod.

2026-07-11: OVMF-xen.fd does not work with QEMU/KVM #

Root cause (RESOLVED) #

OVMF-xen.fd is a Xen-specific firmware variant. It does not initialize VGA output when run under QEMU/KVM, so the VM produces zero serial/VGA output and never boots.

Fix: Use nix eval --raw 'nixpkgs#OVMF.fd' — the standard OVMF variant:

OVMF_FD=$(nix eval --raw 'nixpkgs#OVMF.fd')
# CODE: $OVMF_FD/FV/OVMF_CODE.fd  (use with readonly=on)
# VARS: $OVMF_FD/FV/OVMF_VARS.fd (copy to /tmp, chmod +w, use as read-write)

Verification #

With OVMF.fd, the VM boots through:

  • UEFI firmware → systemd-boot menu → NixOS initrd → switch-root → login prompt
  • All K8s services start: apiserver, controller-manager, scheduler, proxy, kubelet, containerd
  • Serial console (ttyS0) shows full boot log and login prompt
  • kube-addon-manager fails (non-blocking — expected on first boot before CNI is configured)

Complete working QEMU test command #

OVMF_FD=$(nix eval --raw 'nixpkgs#OVMF.fd')
rm -f /tmp/k8s-vars.fd
cp -f "$OVMF_FD/FV/OVMF_VARS.fd" /tmp/k8s-vars.fd
chmod +w /tmp/k8s-vars.fd

qemu-system-x86_64 \
  -enable-kvm -m 8192 -smp 4 -cpu host \
  -drive if=pflash,format=raw,readonly=on,file=$OVMF_FD/FV/OVMF_CODE.fd \
  -drive if=pflash,format=raw,file=/tmp/k8s-vars.fd \
  -drive file=/tmp/k8s-test.qcow2,if=virtio,format=qcow2 \
  -netdev user,id=net0,hostfwd=tcp::2222-:22 \
  -device virtio-net-pci,netdev=net0 \
  -nographic

SSH authentication #

The VM boots and sshd runs with:

services.openssh.enable = true;
services.openssh.settings = {
  PermitRootLogin = "yes";
  PasswordAuthentication = false;
};

SSH connects successfully but requires an authorized key.

SSH key setup:

  • Public key stored at kubernetes/ssh-public-key (git-tracked)
  • Read via builtins.readFile ./ssh-public-key in flake.nix
  • Passed to NixOS configs as _module.args.sshPublicKey
  • Used in common.nix: users.users.root.openssh.authorizedKeys.keys = [ sshPublicKey ];
  • To rotate: regenerate key, update kubernetes/ssh-public-key, rebuild images

kube-addon-manager fails on first boot #

The addon manager service fails with:

FAILED] Failed to start Kubernetes addon manager

This is expected — it tries to create pods before CNI is configured.

Cilium install plan:

  • Currently: install via Helm post-boot (cilium install)
  • Future: integrate into NixOS addon manager for auto-install
    • Generate Cilium manifests via Helm at build time
    • Add to services.kubernetes.addons.cilium.manifests
    • Addon manager applies them automatically on first boot
    • Needs investigation: does NixOS kubernetes module support CRDs via addons?

2026-07-12: API server port mismatch (master vs workers) #

Symptom #

  • Master kubelet registers successfully, workers cannot reach apiserver
  • systemctl is-active kubelet shows "activating" on workers, "active" on master
  • certmgr logs: authentication error. Giving up without retries / invalid token

Investigation technique: trace NixOS module source #

The NixOS module source lives in the Nix store. Find it:

# On the running VM:
find /nix/store -path '*kubernetes*' -name '*.nix' 2>/dev/null | grep kubelet | head -5

# Grep for relevant config patterns:
grep -rn 'apiserverAddress\|masterAddress\|securePort' /nix/store/.../nixos/modules/services/cluster/kubernetes/

Key finding in default.nix:345:

services.kubernetes.apiserverAddress = lib.mkDefault "https://${ 
  if cfg.apiserver.advertiseAddress != null then
    cfg.apiserver.advertiseAddress   # ← uses IP only, no port!
  else
    "${cfg.masterAddress}:${toString cfg.apiserver.securePort}"
}";

The kubelet kubeconfig (in kubelet.nix:417):

services.kubernetes.kubelet.kubeconfig.server = mkDefault top.apiserverAddress;

Root cause #

Setting apiserver.advertiseAddress = "192.168.122.10" (without port) causes apiserverAddress to be https://192.168.122.10 — which defaults to port 443. But workers get https://192.168.122.10:${securePort} (6443). Neither port works for all kubeconfigs.

Fix #

Remove advertiseAddress from master.nix. Module constructs the full URL from masterAddress:securePort.

Investigation technique: verify actual running config #

# Check what flags kube-apiserver actually got:
systemctl cat kube-apiserver | grep -E 'secure-port|advertise'

# Check what port it's listening on:
ss -tlnp | grep -E '6443|443'

# Evaluate NixOS option value at build time (on host):
nix eval --json '.#nixosConfigurations.k8s-master-0.config.services.kubernetes.apiserver.securePort'

# Check NixOS module option defaults:
grep -A5 'securePort' /nix/store/.../apiserver.nix

2026-07-12: Worker cert distribution (cfssl API token) #

Symptom #

Workers boot, certmgr starts, but immediately logs:

level=error msg="authentication error. Giving up without retries"
level=error msg="spec .../kubeletClient.json: continuing despite failed initial validation
  due to while fetching new keyPair: ... {"code":400,"message":"invalid token"}"

Kubelet and kube-proxy stay in "activating" state.

Investigation technique: trace token flow #

# Check where the token lives on master:
ls -la /var/lib/kubernetes/secrets/apitoken.secret
# → lrwxrwxrwx ... -> /var/lib/cfssl/apitoken.secret  (symlink!)

cat /var/lib/kubernetes/secrets/apitoken.secret
# → c2cd48beae7d52947534a6e273393ea4  (random, generated at boot)

# Check workers:
ls -la /var/lib/kubernetes/secrets/apitoken.secret
# → -rw------- ... (regular file, not symlink — no local cfssl)

Investigation technique: read NixOS activation scripts #

# Find the activation script source:
grep -n 'apitoken\|cfsslAPIToken' /nix/store/.../pki.nix

# Key lines:
# L35: cfsslAPITokenBaseName = "apitoken.secret";
# L36: cfsslAPITokenPath = "${config.services.cfssl.dataDir}/${cfsslAPITokenBaseName}";
# L37: certmgrAPITokenPath = "${top.secretsPath}/${cfsslAPITokenBaseName}";
# L193: if [ ! -f "${cfsslAPITokenPath}" ]; then ... random generation ...
# L215: if [ -f "${cfsslAPITokenPath}" ]; then ln -fs ... (symlink creation)

Token flow:

  1. Master activation: generates random 32-char hex at /var/lib/cfssl/apitoken.secret
  2. Master bootstrap service: symlinks to /var/lib/kubernetes/secrets/apitoken.secret
  3. Worker bootstrap: creates empty file at secrets path
  4. Workers need the token manually pasted in

Fix #

Pre-bake a fixed token into the NixOS image for all nodes:

  1. Define k8sApiToken in flake.nix
  2. Pass as _module.args.k8sApiToken
  3. Write via system.activationScripts with deps = [ "kubernetes-pki" ]

Warning: activation script ordering #

lib.mkAfter does NOT control ordering between different activation scripts — it only appends text to the same script. To run after another script, use:

# WRONG — runs in unknown order relative to kubernetes-pki:
system.activationScripts.k8s-apitoken = lib.mkAfter ''...'';

# CORRECT — runs after kubernetes-pki:
system.activationScripts.k8s-apitoken = {
  text = ''...'';
  deps = [ "kubernetes-pki" ];
};

Investigation technique: batch service status check #

# Quick check all K8s services across all nodes:
for ip in 192.168.122.{10,11,12,13}; do
  echo "=== $ip ==="
  SSH_AUTH_SOCK= ssh -i "$SSH_KEY" root@$ip "systemctl is-active kubelet kube-proxy certmgr"
done

2026-07-12: General debugging techniques #

SSH into QEMU VMs #

Bitwarden SSH agent blocks key-based auth. Always prefix with SSH_AUTH_SOCK=:

SSH_KEY="/home/file_magic/Projects/github/dotfiles/kubernetes/ssh-key"
SSH_AUTH_SOCK= ssh -i "$SSH_KEY" root@192.168.122.10

Known hosts after redeploy #

New VMs get new host keys. Clean up before SSH:

for ip in 192.168.122.{10,11,12,13}; do ssh-keygen -R "$ip" 2>/dev/null; done

Read VM screen (when serial console fails) #

virsh -c qemu:///system qemu-monitor-command k8s-master-0 --hmp "screendump /tmp/screen.ppm"
nix-shell -p tesseract --run "tesseract /tmp/screen.ppm stdout"

libvirt VM lifecycle #

Must use qemu:///system (not plain virsh which connects to user session):

# List running VMs:
virsh -c qemu:///system list --name

# Full cleanup sequence:
virsh -c qemu:///system destroy $vm        # force stop
virsh -c qemu:///system undefine $vm --nvram  # remove config + NVRAM
virsh -c qemu:///system vol-delete --pool default $vol.qcow2  # remove disk

# Note: --remove-all-storage fails with NVRAM; use --nvram instead

OVMF VARS template #

Must be exact size (540672 bytes for OVMF.fd) and zeroed:

OVMF_VARS_TEMPLATE=$(nix eval --raw nixpkgs#OVMF.fd)/FV/OVMF_VARS.fd
truncate -s "$(stat -c%s "$OVMF_VARS_TEMPLATE")" /tmp/empty-vars.fd

tofulibvirt provider quirks #

  • ovmf_code_path is a directory; actual firmware at ${path}/FV/OVMF_CODE.fd
  • libvirt_volume uses create.content.url not source
  • os.firmware = "efi" uses provider's own OVMF (may have stale vars) — use manual loader/nv_ram instead
  • features.acpi is Boolean, not list
  • devices.graphics VNC: remove explicit port when using auto_port = true
  • disk.driver.type must be "qcow2" (provider defaults to "raw")

tofu binary location #

/nix/store/2q5865pn7ygz8n1bjdzgsxxagbhpvisw-opentofu-1.12.3/bin/tofu

2026-07-13: Workers stuck "activating" — missing kubelet-client certs #

Symptom #

  • Master fully operational: cfssl, apiserver, kubelet all active
  • Workers: certmgr active, kubelet/kube-proxy "activating" indefinitely
  • kubelet-client.pem and kubelet-client.pem missing in /var/lib/kubernetes/secrets/
  • certmgr log: authentication error → invalid token

Root cause: install -o cfssl fails on workers #

The k8s-write-apitoken service used install -o cfssl unconditionally:

install: invalid user 'cfssl'

Workers don't have the cfssl user — only the master does (created by the NixOS cfssl module). The service exits with error, token never gets written.

Root cause: missing /var/lib/cfssl directory #

Even after adding id cfssl guard, the service failed:

install: cannot create regular file '/var/lib/cfssl/apitoken.secret': No such file or directory

The /var/lib/cfssl directory is created by cfssl's preStart, but our service runs BEFORE cfssl (via before = [ "cfssl.service" ]).

Fix #

script = ''
  TOKEN_FILE=/tmp/k8s-apitoken.tmp
  printf '%s' "${k8sApiToken}" > "$TOKEN_FILE"
  # Master: write to cfssl dataDir (owned by cfssl user)
  if id cfssl >/dev/null 2>&1; then
    mkdir -p /var/lib/cfssl
    install -o cfssl -m 400 "$TOKEN_FILE" /var/lib/cfssl/apitoken.secret
  fi
  # All nodes: write to secrets path
  mkdir -p /var/lib/kubernetes/secrets
  install -m 600 "$TOKEN_FILE" /var/lib/kubernetes/secrets/apitoken.secret
  rm -f "$TOKEN_FILE"
'';

Key points:

  • id cfssl guard prevents failure on workers
  • mkdir -p /var/lib/cfssl before install (directory doesn't exist yet)
  • printf + temp file instead of process substitution (systemd doesn't handle <(...))

2026-07-13: Workers in disk-pressure — image too small #

Symptom #

  • All workers had node.kubernetes.io/disk-pressure:NoSchedule taint
  • Root filesystem at 95% usage (4.7G / 5.3G)
  • CoreDNS couldn't schedule anywhere
  • 1.5G /var/lib/containerd + 3.2G /nix filled the 5.75 GiB image

Root cause #

NixOS image builder default virtualisation.diskSize = "auto" produced a 5.75 GiB image. With containerd images, NixOS system, and K8s components, that's too small.

Fix #

qemuGuestModule = {
  # ...
  virtualisation.diskSize = 20480; # 20 GiB in MiB
};

Note on tofu/libvirt volume sizing #

  • libvirt_volume with create.content.url does NOT support size parameter
  • Must set the disk size at NixOS image build time, not in tofu
  • qemu-img info result/*.qcow2 shows virtual size to verify

2026-07-13: Cilium CNI binary missing on workers after reboot #

Symptom #

  • Cilium pods Running, but ip route shows no Cilium routes
  • cilium-cni binary missing from /opt/cni/bin/ on workers
  • Cilium init container writes to hostPath, but NixOS manages /opt/cni/bin with its own symlinks

Technique: check CNI binary presence #

# On each node:
ls -la /opt/cni/bin/cilium-cni 2>/dev/null || echo "MISSING"

Fix #

NixOS manages /opt/cni/bin with symlinks; Cilium's hostPath writes get lost. Solution:

  1. Extract cilium-cni from the locally imported Cilium image with ctr images mount.
  2. Run that extraction from a retrying systemd service; workers do not have an admin kubeconfig, so kubectl cp is not a valid node-local solution.

The service must retry until the Cilium image is imported. An earlier version exited successfully after seeing an unrelated kube-system container, leaving every worker without the plugin and preventing all new pod sandboxes from starting.

Verification #

just verify passed on 2026-08-19 with all nodes Ready, node IPv6 egress, and pod IPv6 HTTPS egress.

2026-08-19: IPv6 egress verification image curl arguments #

Symptom #

  • The IPv6 egress test pod started but exited with curl reporting option --connect-timeout=8: is unknown.

Root cause #

The curl build in the offline Ceph image accepts the timeout value as a separate argument, not in the GNU-style --option=value form.

Fix #

Pass --connect-timeout and 8 as separate Kubernetes container arguments.

Verification #

The corrected test completed successfully through the host NAT66 path, and just verify passed with failed=0.

2026-08-19: IPv6 host-to-local pod probes restart CoreDNS and CSI #

Symptom #

  • CoreDNS repeatedly restarted with exit code 0 after kubelet liveness probes to pod IPv6 addresses on port 10054 timed out.
  • The ceph-csi controller manager repeatedly restarted after probes to pod IPv6 port 8081 timed out.
  • Cilium agents remained Running, while cilium-health reported host-to-local endpoint timeouts.

Root cause #

The IPv6 host path through cilium_host could not reliably deliver kubelet health probes to local pod addresses. Opening Cilium's node health port 4240 did not fix probes to the pod IPs, and enabling per-endpoint routes did not change the behavior.

Fix #

  • Run CoreDNS in the node network namespace with ClusterFirstWithHostNet.
  • Run the CSI controller manager in the node network namespace with ClusterFirstWithHostNet, one replica, and a Recreate strategy so an unavailable replacement cannot leave an old ReplicaSet running.
  • Allow ingress TCP 10054 and 8081 for the host-network health probes, and TCP and UDP 10053 for host-network CoreDNS DNS traffic.

Verification #

After a clean just down and just up, verification passed with failed=0. CoreDNS and the CSI controller remained 1/1 with zero restarts after three minutes. Cilium health endpoint warnings remain a separate observability issue.

2026-07-13: Rook-Ceph slow commands and mon quorum loss #

Symptom #

  • ceph commands hang (health, status, osd tree)
  • mons fall out of quorum intermittently
  • csi-rbdplugin pods in CrashLoopBackOff

Root cause #

containerd's LimitNOFILE (set to 1048576) causes Ceph daemons to open excessive file descriptors, slowing I/O and causing timeouts.

Fix #

systemd.services.containerd.serviceConfig.LimitNOFILE = lib.mkForce null;

2026-07-13: Rook-Ceph OSDs not found — no available devices #

Symptom #

  • OSD prepare pod runs but finds no usable devices
  • ceph osd tree shows no OSDs
  • OSD prepare log: skipping sda1 (boot), sda2 (rootfs) — all partitions in use

Technique: check available block devices #

lsblk -o NAME,SIZE,TYPE,MOUNTPOINT
# sda  20G  disk
# ├─sda1  1G  part /boot
# └─sda2 19G  part /nix
# No unused block devices → no OSDs possible

Fix #

Add a dedicated 40GB data disk per worker in Terraform (libvirt_volume.worker_data), attached as /dev/sdc. The separate /dev/sdb 30GB disk is mounted at /var/lib/containerd; NixOS does not partition the Ceph disk, and Rook-Ceph provisions it as a raw OSD.

2026-07-13: CephConnection CRD missing — Rook v1.20.2 requires csi-operator.yaml #

Symptom #

  • Rook operator log: no matches for kind "CephConnection" in version "csi.ceph.io/v1"
  • crds.yaml from Rook doesn't include CephConnection

Fix #

Apply csi-operator.yaml from Rook v1.20.2 examples (provides CephConnection CRD):

kubectl apply -f https://raw.githubusercontent.com/rook/rook/v1.20.2/deploy/examples/csi-operator.yaml

2026-07-13: Ceph version rejected — Rook v1.20.2 requires Squid #

Symptom #

  • OSD pods log: failed the ceph version check: the version does not meet the minimum version "19.2.0-0 squid"
  • Using quay.io/ceph/ceph:v18.2.0 (Reef)

Fix #

Rook v1.20.2 requires Ceph v19.2.0 (Squid):

cephVersion:
  image: quay.io/ceph/ceph:v19.2.0

2026-07-13: NixOS kernel modules for Rook-Ceph #

Symptom #

  • Rook-Ceph CSI pods fail to mount RBD volumes
  • Missing kernel modules: rbd, ceph

Fix #

Add to common.nix:

boot.kernelModules = [ "rbd" "ceph" ];

Also mount kernel modules and /nix into CSI plugin volumes (NixOS doesn't use standard FHS paths):

# In Rook-Ceph ConfigMap:
CSI_RBD_PLUGIN_VOLUME = ''
  - name: lib-modules
    hostPath:
      path: /run/current-system/kernel-modules/lib/modules/
  - name: host-nix
    hostPath:
      path: /nix
'';
CSI_RBD_PLUGIN_VOLUME_MOUNT = ''
  - name: lib-modules
    mountPath: /lib/modules
    readOnly: true
  - name: host-nix
    mountPath: /nix
    readOnly: true
'';

2026-07-13: Ansible kubernetes.core modules don't support async #

Symptom #

  • Tasks with async: 300 / poll: 10 fail silently or timeout
  • kubernetes.core.helm, kubernetes.core.k8s, kubernetes.core.k8s_info ignore async directives

Fix #

Remove all async/poll from kubernetes.core tasks. Use register + until + retries + delay for polling instead.

2026-07-13: Ansible role_path undefined with import_tasks #

Symptom #

  • Tasks using {{ role_path }}/files/... fail with undefined variable
  • Happens when using import_tasks (flat task files, no roles)

Fix #

Use {{ playbook_dir }} instead of {{ role_path }} for local file paths.

2026-07-13: KUBECONFIG must be set at play level for kubectl #

Symptom #

  • kubectl commands in tasks fail with Unable to connect to the server
  • No kubeconfig in task environment

Fix #

Set at play level:

- hosts: master
  environment:
    KUBECONFIG: /root/.kube/config
  tasks:
    - name: Setup kubeconfig
      ansible.builtin.shell: |
        mkdir -p /root/.kube
        cp /etc/kubernetes/cluster-admin.kubeconfig /root/.kube/config
      args:
        creates: /root/.kube/config

2026-07-13: Cilium pod label is k8s-app, not app.kubernetes.io/name #

Symptom #

  • kubectl get pods -l app.kubernetes.io/name=cilium returns nothing
  • Cilium pods use k8s-app=cilium label

Fix #

kubectl get pods -l k8s-app=cilium -n kube-system

2026-07-13: SCP from master to workers fails — remote path resolution #

Symptom #

  • ansible.builtin.copy with src: "{{ playbook_dir }}/..." works on master but fails on workers
  • {{ playbook_dir }} resolves on the control machine, but the file doesn't exist on the remote

Fix #

Use ansible.builtin.fetch to pull the file to the control machine, then ansible.builtin.copy with delegate_to:

- name: Fetch from master
  ansible.builtin.fetch:
    src: /opt/cni/bin/cilium-cni
    dest: /tmp/cilium-cni
    flat: true
  delegate_to: "{{ inventory_hostname }}"

- name: Copy to workers
  ansible.builtin.copy:
    src: /tmp/cilium-cni
    dest: /opt/cni/bin/cilium-cni
    mode: "0755"
  loop: "{{ groups['workers'] }}"
  delegate_to: "{{ item }}"

2026-07-13: Ansible values_files doesn't template Jinja2 #

Symptom #

  • Helm values file sent to remote with raw Jinja2 variables ({{ master_ip }})
  • Helm install fails because the template was never rendered

Fix #

Use values: dict (inline templated values) instead of values_files: for values that contain Jinja2 variables.

2026-07-14: CSI staging path does not exist on node #

Symptom #

  • PVC stays Pending, pod stuck in ContainerCreating
  • CSI node plugin logs: staging path does not exist on node
  • kubelet passes staging path /var/lib/kubernetes/plugins/kubernetes.io/csi/.../globalmount

Root cause #

NixOS defaults kubelet --root-dir to /var/lib/kubernetes. CSI node plugin containers mount /var/lib/kubelet/plugins as a hostPath. When kubelet passes a staging path under /var/lib/kubernetes/plugins/..., the CSI container can't see it because it only mounts /var/lib/kubelet/plugins.

Fix #

services.kubernetes.dataDir = "/var/lib/kubelet";

This makes kubelet use --root-dir=/var/lib/kubelet, matching what CSI containers expect.

Technique: verify kubelet root-dir #

cat /proc/$(pgrep kubelet)/cmdline | tr "\0" "\n" | grep root-dir
# Should show: --root-dir=/var/lib/kubelet

2026-07-14: clusterDns wrong on workers #

Symptom #

  • CoreDNS pods running but worker pods can't resolve service names
  • Workers get clusterDns = ["10.0.0.254"] but CoreDNS ClusterIP is 10.96.0.254

Root cause #

Workers don't set apiserver.serviceClusterIpRange. The NixOS module computes clusterDns from the default CIDR 10.0.0.0/24, giving 10.0.0.254. The master's actual service CIDR is 10.96.0.0/12, putting kube-dns at 10.96.0.254.

Fix #

# In common.nix
services.kubernetes.addons.dns.clusterIp = "10.96.0.254";

Remove clusterDns from master.nix and worker.nix (let common.nix handle it).

2026-07-14: CSI operator not deploying node plugin DaemonSets #

Symptom #

  • CephCluster Ready, but no CSI node plugins on workers
  • Volumes provisioned but can't be staged/attached
  • ROOK_USE_CSI_OPERATOR: "false" in operator ConfigMap

Root cause #

With ROOK_USE_CSI_OPERATOR: "false", the Rook operator only creates the CSI controller, not node plugin DaemonSets. The CSI operator is responsible for creating Driver CRDs which deploy DaemonSets.

Fix #

ROOK_USE_CSI_OPERATOR: "true"

Apply csi-operator.yaml BEFORE Driver/OperatorConfig CRDs (CRD ordering dependency).

2026-07-14: CSI operator CRD ordering dependency #

Symptom #

  • Applying Driver/OperatorConfig CRDs fails: Failed to find exact match for csi.ceph.io/v1.OperatorConfig

Root cause #

The csi-operator.yaml defines CRDs (like CephConnection, OperatorConfig, Driver) that other resources reference. If you apply resources before the CRDs exist, Kubernetes can't validate them.

Fix #

Apply csi-operator.yaml first, then apply Driver CRDs, ImageSet ConfigMap, and OperatorConfig.

Technique: extract resources from multi-document YAML #

# In Ansible shell task:
import yaml
with open('/tmp/rook-operator.yaml') as f:
    docs = list(yaml.safe_load_all(f))
# Extract Deployment separately, CSI resources separately, skip operator ConfigMap
csi_resources = [d for d in docs if d and d.get('kind') != 'Deployment'
                 and not (d.get('kind') == 'ConfigMap'
                          and d.get('metadata', {}).get('name') == 'rook-ceph-operator-config')]

2026-07-14: StorageClass missing CSI secret references #

Symptom #

  • PVC Pending, CSI provisioner log: provided secret is empty
  • StorageClass has no csi.storage.k8s.io/provisioner-secret-name parameters

Root cause #

The CSI provisioner needs Kubernetes secrets to authenticate with Ceph for creating/deleting RBD images. Without the secret references in the StorageClass, it sends empty credentials.

Fix #

Add all four secret references to the StorageClass:

parameters:
  csi.storage.k8s.io/provisioner-secret-name: rook-csi-rbd-provisioner
  csi.storage.k8s.io/provisioner-secret-namespace: rook-ceph
  csi.storage.k8s.io/node-stage-secret-name: rook-csi-rbd-node
  csi.storage.k8s.io/node-stage-secret-namespace: rook-ceph

Warning: StorageClass parameters are immutable #

You must delete and recreate the StorageClass to change parameters:

kubectl delete sc rook-ceph-block
# Then reapply with correct parameters

2026-07-14: CSI provisioner DeadlineExceeded followed by "operation already exists" #

Symptom #

  • First CreateVolume attempt times out: rpc error: code = DeadlineExceeded desc = context deadline exceeded
  • Subsequent retries fail: rpc error: code = Aborted desc = an operation with the given Volume ID already exists
  • PVC stays Pending indefinitely

Root cause #

The CSI driver successfully creates the RBD image on Ceph, but the operation takes >2.5 minutes (the provisioner sidecar's default deadline). The sidecar times out and reports failure, but the volume IS created on Ceph. The internal operation tracker still holds the in-progress state, so all retries get Aborted.

Fix #

Restart the CSI controller pod(s) to clear the stuck operation tracker:

kubectl -n rook-ceph delete pod rook-ceph.rbd.csi.ceph.com-ctrlplugin-<pod-name>

The StatefulSet/ReplicaSet recreates the pod with a fresh operation tracker. The PVC will be retried automatically.

Why it was slow #

On first boot, Ceph OSDs and monitors are still initializing. The CSI driver may need to wait for OSDs to be ready before it can create images. Subsequent volume creations are fast (<1 second) once Ceph is warmed up.

2026-07-14: CSI attacher leader lease held by deleted pod #

Symptom #

  • VolumeAttachment stays ATTACHED=false
  • Pod stuck in ContainerCreating after PVC is Bound
  • CSI attacher sidecar log: Attempting to acquire leader lease... indefinitely
  • Lease holder is a deleted pod name

Root cause #

When CSI controller pods are deleted/restarted, the leader lease for the attacher sidecar may still be held by the old pod. The lease expires after 137 seconds (leaseDurationSeconds), but during that window the new pod can't attach volumes.

Fix #

Wait for the lease to expire (~2-3 minutes), or delete both CSI controller pods to force simultaneous lease expiry and re-election:

kubectl -n rook-ceph delete pod -l app=rook-ceph-rbdplugin-provisioner

Technique: check lease state #

kubectl -n rook-ceph get lease external-attacher-leader-rook-ceph-rbd-csi-ceph-com \
  -o jsonpath="holder={.spec.holderIdentity} renew={.spec.renewTime}"

Symptom #

  • CSI node plugins fail to load kernel modules
  • NixOS stores modules at /run/current-system/kernel-modules/lib/modules/ (non-standard path)

Fix #

Add systemd tmpfiles rule in common.nix:

systemd.tmpfiles.rules = [
  "L /lib/modules - - - - /run/current-system/kernel-modules/lib/modules"
];

2026-08-13: Boot-blocker diagnosis without a working serial console (13-inch-thin-cannon) #

Symptom #

  • Installed 13-inch-thin-cannon boots to a LUKS prompt, passphrase is accepted (no re-prompt), then the serial console goes silent. SSH never comes up (watched to 240 s).
  • Serial log frozen at ~3.3 KB (prompt + masked bullets). VGA screendump looks near-black to a human eye. Looks exactly like a kernel hang.

Technique: QEMU monitor screendump + tesseract OCR (the breakthrough) #

When the serial console is silent but the kernel is alive, read the VGA screen via the QEMU monitor and OCR it. This is the non-libvirt analogue of the existing virsh qemu-monitor-command screendump trick:

# 1. Screendump from the QEMU monitor TCP port (MONITOR_PORT=5555):
printf 'screendump /tmp/opencode/screen.ppm\n' | timeout 3 nc 127.0.0.1 5555
#    (or: bash -c 'exec 3<>/dev/tcp/127.0.0.1/5555; echo "screendump /tmp/opencode/screen.ppm" >&3')

# 2. OCR with tesseract (path is a pinned nix-store binary):
/nix/store/2ac32mdm2rdcy5fx6pma5jckpnfa2pkz-tesseract-5.5.2/bin/tesseract \
  /tmp/opencode/screen.ppm stdout --psm 6

# 3. Optional: scale up 2x first (tesseract needs ~10px glyphs):
/nix/store/clx4xcbrg16dhzyzm2faxjddfk9rz979-imagemagick-7.1.2-27/bin/magick \
  /tmp/opencode/screen.ppm -resize 200% -colorspace Gray /tmp/opencode/screen2x.ppm

What OCR revealed (the serial console never printed these):

[FAILED] Failed to start Find NixOS closure.
[DEPEND] Dependency failed for Initrd Default Target.
You are in emergency mode. After logging in, type "journalctl -xb"...

The system was NOT hung — it had dropped to the initrd emergency shell on VGA. Root cause chain: the disk had no /nix btrfs subvolume, so the Nix store lived inside /root, which the impermanence rollback wipes and replaces with root-blank every boot. initrd-find-nixos-closure (RequiresMountsFor = /sysroot/nix/store, nixpkgs initrd.nix:688) had no mount unit for that path → dependency failed → emergency shell. Fix: a persistent /nix subvolume (disko.nix) + neededForBoot = true (→ x-initrd.mount) so the initrd mounts the store before the closure finder runs.

Technique: serial SysRq trigger to prove kernel liveness #

When "no output after unlock" looks like a hang, send a SysRq over the serial console via the QEMU monitor to prove the kernel is alive:

printf 'sendkey ctrl-alt-sysrq-t\n' | timeout 3 nc 127.0.0.1 5555
# 40 s later the kernel echoed to serial:  "sysrq: Show State"

sysrq: Show State is the expected serial-side header. Note the task dump itself went to VGA (tty0), not serial — the console=ttyS0 console=tty0 kernel param sends the dump to both, but systemd had already switched the display path, so OCR the VGA screendump to read the dump. Combined with a QEMU info registers (three samples, HLT=1 in EFLAGS = idle loop, not a spin/panic), this distinguishes "hung kernel" from "boot failed, console not echoing".

Technique: PPM → ASCII renderer for quick screen eyeball #

/tmp/opencode/ppm2ascii.sh renders a QEMU screendump as ASCII art with color-coded glyph blocks, letting you "see" the VGA layout in a terminal without opening the image (helpful to find the region OCR should target and to confirm text-vs-blank).

2026-08-13: Intermittent initrd watchdog reboot (13-inch-thin-cannon) #

Symptom #

  • The first cold boot of the installed 13-inch-thin-cannon VM sometimes dies ~1-2 s into the initrd (right after the LUKS prompt is unlocked). Serial log:
    [   10.827277] kvm_intel: VMX not supported by CPU 7
[   14.599504] watchdog: watchdog0: watchdog did not stop!
[   14.836988] watchdog: watchdog0: watchdog did not stop!
[   15.322382] watchdog: watchdog0: watchdog did not stop!
[   15.328746] reboot: Restarting system
    The VM reboots itself; the next boot always survives. Intermittent — the same disk/binary sometimes boots straight through to SSH (verified working).

Root cause #

  • QEMU's q35 machine emulates an Intel ICH9 TCO watchdog (iTCO_wdt driver). Real 13-inch-thin-cannon hardware has no such device, so this is a harness artifact, not a config bug.
  • Kernel commit 1ae3e78c ("watchdog: iTCO_wdt: No need to stop the timer in probe") stopped the driver from clearing NO_REBOOT / stopping the TCO timer at probe. QEMU ≥ 8.0 enables the TCO reboot-pin strap by default (commit a6b6414), so an armed TCO genuinely resets/reboots the guest.
  • During the delayed initrd (the LUKS prompt holds the kernel ~10 s), the watchdog device gets opened and released before the driver can stop it → kernel watchdog core prints watchdog did not stop! and falls through to emergency_restart() (reboot: Restarting system). The LUKS delay is why the first boot dies — the prompt pushes the release past the TCO window.
  • Intermittency: the QEMU TCO timer state at power-on varies, and a fast initrd beats the window (verified: surviving boots log iTCO_wdt ... initialized. heartbeat=30 sec).

Fix (status: verdict — NOT a QEMU TCO hardware reset, 2026-08-13) #

  • Tested: booting with -global ICH9-LPC.noreboot=on (disables the q35 ICH9-TCO/chipset reset) does NOT prevent the reboot — the boot still died with watchdog: watchdog0: watchdog did not stop! ×3 → reboot: Restarting system. Therefore the reboot is a guest-initiated software reboot (kernel emergency_restart() from the watchdog device-release path when the HW_RUNNING TCO can't be stopped), not a QEMU TCO hardware reset. The flag only ever guards the chipset-reset side, which is not the path being taken.
  • Open question (not yet root-caused): who opens/closes /dev/watchdog in the initrd around the kvm module load, triggering the three release attempts. Suspected: systemd/udev probing the freshly-registered watchdog device during the delayed (LUKS-held) initrd.
  • Candidate fixes, in preference order (all still unapplied):
    1. Harness tolerance: the boot phase already does responsive unlock + a long SSH wait, so it survives the one-time self-reboot (next boot always survives). Verify with repeated just boot runs.
    2. Guest-side: boot.blacklistedKernelModules = [ "iTCO_wdt" ] so the driver never probes (RHEL-documented mitigation). Diverges from real hardware config, so only use if the harness-tolerance route is insufficient.
    3. Real-hardware note: 13-inch-thin-cannon has no TCO; this bug is purely a QEMU harness artifact and will not occur on physical hardware.
  • Status update (2026-08-13, end of session): harness tolerance confirmed sufficient — fresh install + 3× repeated cold just boot runs all green (only the expected wireguard WARN). No kernel blacklist needed.
  • Related but distinct: a separate intermittent "SSH dead on some boots" failure was root-caused to a systemd ordering cycle (swapfile service Before=swap.target → systemd deleted run-wrappers.mount → missing unix_chkpwd → sshd PAM denial). Full write-up in 13-inch-thin-cannon/docs/iso-build.md (Known issues & gotchas).

Technique: distinguishing the reboot type (tested 2026-08-13) #

  • -global ICH9-LPC.noreboot=on: stops TCO/hardware-chipset resets. Applied and the reboot persisted → proves the reboot is a guest software emergency_restart(), not the emulated TCO.
  • -no-reboot: VM exits instead of restarting on a guest-initiated reboot (kernel emergency_restart/reboot syscall). If a reboot survives the noreboot flag but is caught by -no-reboot, it is a software reboot, not TCO.

2026-08-14: Thin-cannon password provisioning and test harness #

Symptom #

  • The ISO built and installed successfully, but the installed file_magic password-login check failed.
  • Key-based SSH worked, while the password test was initially misleading: the account was present but locked, or the client closed during authentication.

Root cause #

  • The host uses users.mutableUsers = true by default in the shared hardened user module. disko-install creates users before activation, so activation preserved the install-time locked shadow entry instead of applying the declarative password hash.
  • The password target originally passed its stdin directly to agenix after reading the two password prompts. When driven by a pipe, those reads consumed the input and agenix encrypted an empty value instead of the generated hash.
  • The repository intentionally ignores secrets/*.age, so pure flakes omit the encrypted password file and builtins.pathExists evaluates false. No password secret is present in a fresh ISO; it must be provisioned out of band after the target host has generated its SSH host key.
  • The installed host is rebuilt from the baked flake by disko-install; an impure builtins.getEnv value used while building the ISO is not automatically available to that later rebuild.
  • Password experiments also crossed two independent shell boundaries: the installed user uses Nushell, while the test commands use POSIX shell syntax. Commands such as 2>&1 therefore cannot be sent directly through that user's login shell.

Fix and current policy #

  • Set users.mutableUsers = false on 13-inch-thin-cannon so the declarative agenix-backed yescrypt hash is applied during installation.
  • Store the generated hash in a mode-600 temporary file before invoking agenix, so both interactive and piped password entry encrypt the same hash.
  • Keep encrypted .age files out of version control; provision them on the target host after installation.
  • Keep production SSH key-only. Provision the user's password with just set-file-magic-password, which creates secrets/file_magic-password.age; never put a plaintext password in Nix configuration.
  • The harness logs in with the existing file_magic key and runs installed-user commands through Bash, leaving Nushell as the production login shell.
  • The temporary password-authentication experiment was removed. Repeated QEMU runs showed that its sshpass result was not a reliable end-to-end signal; the source of truth for ISO health is the key-based iso-test.sh verification.
  • On 2026-08-17, piped password provisioning followed by just up passed all live, install, and boot phases.
  • On 2026-08-17, provisioning the password encrypted to the VM's generated SSH host key decrypted successfully and password SSH login passed; restoring key-only SSH then rejected password login while key login still passed.

Useful techniques #

  • Check whether a declarative account is locked with passwd -S <user>.
  • Inspect the generated user definition with the system's *-users-groups.json closure path, without printing password hashes.
  • When debugging installed-user remote commands, invoke POSIX commands through bash -c rather than relying on the user's interactive shell parser.

OVMF Path Discovery #

After NixOS rebuild, the OVMF path changes. Find it:

virsh dumpxml k8s-master-0 | grep -i ovmf
# Use the path from the <loader> element

# Or build the flake's pinned OVMF (realizes the path, see 2026-08-01 entry):
nix build --no-link --print-out-paths .#ovmf
# -> /nix/store/<hash>-OVMF-<ver>-fd  (contains FV/OVMF_CODE.fd + FV/OVMF_VARS.fd)

Do NOT use nix eval --raw nixpkgs#OVMF.fd — it prints the store path but does not realize it, so FV/OVMF_CODE.fd does not exist unless OVMF was previously built.

virsh UEFI VM Management #

# Undefine UEFI VMs requires --nvram flag (no path argument):
virsh -c qemu:///system undefine k8s-master-0 --nvram

# List VMs and their state:
virsh -c qemu:///system list --all

2026-08-01: tofu destroy leaves k8s-* disks/networks behind #

Symptom #

  • After just up then just down, libvirt still had orphaned resources: an active k8s-1 NAT network with autostart, while the tofu state was empty and no domains/volumes were tracked.
  • tofu destroy itself correctly deletes tracked volumes (verified: 13 resources, including worker_data raw disks, fully removed). The leftovers were resources that had drifted out of the tofu state (interrupted/failed applies, or switching CLUSTER_INDEX/WORKER_COUNT between runs).

Root cause #

  1. State drift. Volumes/domains are named without a cluster index (k8s-master-0.qcow2 in the shared default pool) while the network IS index-scoped (k8s-${index}). If an apply is interrupted or fails after libvirt objects are created but before tofu records them, or if you switch CLUSTER_INDEX/WORKER_COUNT, objects exist in libvirt but are invisible to tofu destroy — so they are never deleted.
  2. nix eval --raw nixpkgs#OVMF.fd returns an unrealized store path. tofu apply then fails at libvirt_domain creation ("Failed to Start Domain") after the disks and network were already created, leaving a half-applied state and forcing manual cleanup. This is what caused the initial orphaned disks: a just up whose apply died partway.

Fix #

  • Root cause was the OVMF path: it is now realized via the flake nix build --no-link --print-out-paths .#ovmf (new ovmf package output in flake.nix) instead of nix eval --raw. Applies no longer fail at domain start, so tofu state stays consistent and tofu destroy removes everything.
  • just down stays a pure tofu destroy (no virsh side-commands): tofu state is the single source of cleanup. A virsh prune was tried as a workaround but removed — the goal is that a failure like this should be fixed at its root, not papered over with separate manual cleanup.

Technique notes (not currently used) #

  • virsh vol-list <pool> does not support --name (unlike list/net-list). Parse the name column: virsh vol-list --pool default | awk 'NR>2 && $1 != "" {print $1}'.
  • virsh undefine <dom> --nvram also removes the OVMF NVRAM file.
  • When tofu destroy reports nothing destroyed but libvirt still has k8s-* objects, the state was drifted by a failed apply; re-apply once (or recover state) so destroy can track them again.

2026-08-01: ansible fails with missing /var/lib/kubelet/secrets/cluster-admin.pem on fresh cluster #

Symptom #

  • just up succeeded through deploy + wait-ssh, then the Cilium Helm task failed: kubernetes cluster unreachable: invalid configuration: unable to read client-cert /var/lib/kubelet/secrets/cluster-admin.pem: no such file or directory.
  • The certs existed minutes later — this is a boot race, not a config error.

Root cause #

The NixOS kubernetes module with easyCerts = true generates the cluster PKI at boot: the master's cfssl service + kube-certmgr-bootstrap create /var/lib/kubelet/secrets/*.pem. The ansible playbook only waited for SSH (wait_for_connection), so on a fast-booting VM (SSH ready in ~10s) ansible ran helm before cfssl finished generating certs.

Fix #

ansible/site.yml gained a "Wait for Kubernetes API server" play before Cilium install: kubectl get --raw /readyz with KUBECONFIG=/etc/kubernetes/cluster-admin.kubeconfig, retried 60 × 5s until rc==0 (per AGENTS.md: until + retries + delay). This waits for both the certs AND the apiserver to be healthy. Use ansible.builtin.command + task-level environment: (not shell) to satisfy ansible-lint's command-instead-of-shell rule.

Technique: SSH around stale known_hosts #

After redeploys, the VM host key changes and ssh aborts on the fingerprint mismatch (see the Offending RSA key warning). Use -o UserKnownHostsFile=/dev/null for manual diagnostic SSH; the justfile already sets -o StrictHostKeyChecking=no.

2026-08-02: kube-controller-manager crash loop "New CIDR set failed; the node CIDR size is too big" #

Symptom #

  • After enabling dual-stack, kube-controller-manager crash-looped: systemd RestartSec=30s + RestartSec, leader-election events flapped every ~40s, and no pods were created at all (deployments/daemonsets had 0 ready, kubectl get pods -n kube-system = empty). No Cilium pods → nodes stuck NotReady.
  • Controller-manager log: Error initializing a controller ... err="New CIDR set failed; the node CIDR size is too big" controller="node-ipam-controller".

Root cause #

The k8s nodeipam controller splits each --cluster-cidr into per-node subnets with cidrset.NewCIDRSet:

if (clusterCIDR.IP.To4() == nil) && (subNetMaskSize-clusterMaskSize > clusterSubnetMaxDiff) {
    return nil, ErrCIDRSetSubNetTooBig   // clusterSubnetMaxDiff = 16
}

For IPv6, node mask − cluster mask must be ≤ 16. Default node mask is /64, so the IPv6 cluster CIDR must be ≥ /48. My first config used fd04::/64 cluster CIDR with --node-cidr-mask-size-ipv6=112 (diff 48) → crash.

Fix #

Set the v6 node-IPAM cluster CIDR to /48 (fd04::/48) and drop the explicit --node-cidr-mask-size-ipv6 (defaults to /64, diff 16 ≤ 16).

Technique #

  • Crash-looping control plane = "no pods anywhere", which looks like a scheduling problem but is actually the ReplicaSet/deployment controllers never starting. Diagnose the controller first (journalctl -u kube-controller-manager), not the pods.
  • The k8s IPv6 /48 constraint: cidr_set.go comment — "the subnet mask for IPv6 cluster cidr needs to be >= 48".