Build images — nix build .#images creates QCOW2 images for all 4 nodes
Deploy VMs — tofu destroy + tofu apply creates libvirt VMs from images
Wait for SSH — Polls every node over its IPv6 inventory address until SSH is available
Import offline images — Pulls images on the control machine, copies per-image archives to every node, and imports them into containerd
Install Cilium — CNI with eBPF masquerade (replaces kube-proxy)
Install Rook-Ceph — Storage operator, CephCluster, CSI driver, StorageClass
Verify — Asserts all pods Running, all services Ready

Offline Images #

The nodes have no external registry access. just up runs just offline-images after SSH is available. That target builds the fixed-output .#offline-images derivation, then runs the Ansible import playbook against its Nix store output.

To build and import the bundle manually, after just deploy and just wait-ssh:

bundle=$(nix build --no-link --print-out-paths .#offline-images)
cd ansible
ansible-playbook playbooks/import-images.yml \
  -e offline_image_bundle="$bundle"

The image definitions are in offline-images.nix. Each image is pinned by registry manifest digest and fixed archive hash, and the derivation produces one zstd-compressed Docker archive per image. The first build needs registry access; later builds use the Nix store or a binary cache. The build runs through Nix's unprivileged skopeo fetcher and does not need a Podman daemon or /etc/containers/policy.json.

The Kubernetes pod sandbox image is pinned to registry.k8s.io/pause:3.10 in common.nix and is included in the same bundle. CoreDNS is built and seeded locally by the NixOS Kubernetes module; kube-proxy is disabled because Cilium replaces it.

The generated archives can require several gigabytes of local storage. They live in the Nix store and are generated input for the Ansible import step.

The bundle can be built before any cluster exists:

just offline-images-build

This only needs registry access during the first build. Run just offline-images after the VMs are available to copy the already-built archives to the nodes.

Discover Helm Images #

List the images a chart will render before deploying a cluster:

just helm-images ./path/to/chart --values ./path/to/values.yaml

Additional Helm template flags can be passed after the chart, for example --set image.tag=... or --kube-version 1.32.0. The command renders the chart locally and extracts images from regular, init, and ephemeral containers. The output can then be pinned and added to offline-images.nix.

See docs/nixidy.md for the proposed Nixidy migration and offline deployment boundary.

SSH into Master #

just ssh

Note: Bitwarden SSH agent blocks key auth. The justfile handles this with IdentityAgent=none.

Verify Cluster #

just verify

Clean Up #

just down

Environment Variables #

CLUSTER_INDEX=1 just up   # Different cluster subnet (default: 0)
WORKER_COUNT=5 just up    # More workers (default: 3)

Single Node Testing (QEMU) #

cd kubernetes
nix build .#k8s-master-0

OVMF_FD=$(nix build --no-link --print-out-paths nixpkgs#OVMF.fd)
cp -f "$OVMF_FD/FV/OVMF_VARS.fd" /tmp/k8s-vars.fd
chmod +w /tmp/k8s-vars.fd

qemu-system-x86_64 \
  -enable-kvm -m 4096 -smp 2 -cpu host \
  -drive if=pflash,format=raw,readonly=on,file=$OVMF_FD/FV/OVMF_CODE.fd \
  -drive if=pflash,format=raw,file=/tmp/k8s-vars.fd \
  -drive file=result/k8s-master-0.qcow2,if=virtio,format=qcow2 \
  -netdev user,id=net0,hostfwd=tcp::2222-:22 \
  -device virtio-net-pci,netdev=net0 \
  -nographic

# SSH in (from another terminal)
SSH_AUTH_SOCK= ssh -i kubernetes/ssh-key -p 2222 root@localhost

Kubernetes Cluster on NixOS + KVM/QEMU #

Prerequisites #

Quick Start #

What just up Does #