unikraft-postgres #
PostgreSQL 16.4 as a Unikraft unikernel, ported from
unikraft-cloud/examples's postgres
to build for arm64 and boot under bsdkrun — with a real postmaster
serving SQL over TCP, not single-user mode.
./build.sh # host arch; or: ./build.sh x86_64
bsdkrun unikraft . --mem 2048 --port 5432:5432 \
--cmdline "elfloader -- /usr/local/bin/postgres -D /var/lib/postgresql/data"
$ psql -h 127.0.0.1 -p 5432 -U postgres postgres
postgres=# CREATE TABLE t (id INT PRIMARY KEY, name TEXT);
postgres=# INSERT INTO t VALUES (1, 'unikraft-on-bsdkrun');
postgres=# SELECT name FROM t;
name
---------------------
unikraft-on-bsdkrun
(1 row)
Status #
| what | state |
|---|---|
| Builds and boots (arm64) | works |
| Postmaster starts and listens on 5432 | works |
Spawns child processes (posix_spawn + execve) |
works |
| WAL recovery, checkpointer, background writer | works |
| Clients over the forwarded port, one backend each | works — 10/10 sequential connections served |
postgres --single (SQL on the console) |
works |
Both architectures. arm64 on macOS/Hypervisor.framework, x86_64 under KVM in
.github/workflows/e2e-unikraft-examples.yml, where the job is strict: it
creates a table, inserts a row and reads it back, so a green run means the
postmaster spawned a backend process and that backend planned and executed SQL.
How a forking server runs on a kernel with no fork #
Upstream's Kraftfile names runtime: base-compat:latest, a Unikraft Cloud
runtime. This example runs on stock Unikraft, where there is no fork() at all
— lib/posix-process/clone.c rejects any clone that does not share the address
space, because there is only ever one address space:
ERR: [libposix_process] CLONE_VM not set: Multiple address spaces are not supported
What Unikraft does have is vfork() + execve(), and its own documentation
points at the way through
(lib/posix-process/README.md):
Fortunately it is common that applications spawn new processes by calling
fork()immediately followed by anexecve()[…] Applications can use theposix_spawn()libc function that spawns a process usingvfork()in a safe way.
PostgreSQL already has exactly that mode. EXEC_BACKEND — what it uses on
Windows, and supports on Unix for testing — makes the postmaster fork and
immediately exec postgres --forkchild for every child, handing the child
its state through a file instead of through inherited memory. Every
fork_process() call site in the tree is inside #else /* !EXEC_BACKEND */, so
building -DEXEC_BACKEND leaves none behind, and
exec-backend-posix-spawn.patch turns the one remaining fork+exec into the
posix_spawn() that Unikraft supports.
So the shape is: CONFIG_APPELFLOADER_MULTIPROCESS on the kernel side (which
pulls in multiprocess, signals and an init process), -DEXEC_BACKEND plus four
small patches on the PostgreSQL side. The cluster is still initialised at image
build time, because initdb runs postgres --boot and postgres --single as
children and there is no reason to do at boot what can be done once.
What it took: six Unikraft fixes #
All are in ../../library/unikraft-base/patches/apply.sh, and none of them is
about PostgreSQL being unusual — each is a place where the guest is not Linux.
Three of them are kernel bugs no other example here exercises: two mean
execve() had never worked on arm64 at all, and one meant no
posix_spawn() child could survive on x86_64.
| # | what was wrong |
|---|---|
| 16 | signalfd is defined but not on the shim's legacy list, and arm64 has no __NR_signalfd — so CONFIG_LIBPOSIX_PROCESS_SIGNALFD could not be built. PostgreSQL 13+ reads signals through a file descriptor and exits with FATAL: signalfd() failed without it. |
| 18 | execve() sets every field of the binfmt loader's argument struct except argc/envc, which it leaves as stack garbage. A loader that believes them walks off the end of argv and dereferences whatever it finds. Nothing in-tree reads those counts, so upstream never noticed. |
| 19 | arm64 execve() enters the new program at the wrong register. It writes the entry point to LR, but ukarch_execenv_load() restores ELR_EL1 from the PC slot and leaves through eret — so the new program started at whatever was in freshly allocated stack memory, i.e. 0. x86_64 sets RIP correctly, which is why an x86_64-only upstream never saw it. |
| 20 | A signal the current thread has blocked is run in it anyway. Process-directed delivery looks for any thread that does not block the signal, then calls do_deliver() — which builds the frame on the current context. PostgreSQL blocks every signal while it installs handlers and only then initialises the latch they use, so a SIGCHLD arriving in that window ran handle_pm_child_exit_signal → SetLatch(NULL). |
| 22 | A vfork child woke up with the kernel's TLS as its userland TLS. child->tlsp = child->uktlsp — but vfork means the child is the parent for a moment, thread pointer included, and everything musl's posix_spawn child touches before execve() is TLS-relative (stack canary, errno, pthread self). arm64 absorbed it (thread pointer names the start of the block); x86_64 names the end, so the canary read at %fs+0x28 landed past the allocation and the child died before its first syscall — silently, because its death woke the vfork parent and the status pipe read EOF, which posix_spawn reports as success. |
| 21 | setsid() was a flat return -EPERM ("we have a single session with a single process" — written before multiprocess existed). Every PostgreSQL child calls it and dies with FATAL: setsid() failed. The neighbouring getsid() already reports UNIKRAFT_SID to anyone who asks, so refusing was inconsistent as well as fatal. |
Findings 19, 20 and 22 are worth restating, because they are not about
PostgreSQL: any multiprocess application on arm64 hits 19 on its first
execve(); any daemon that blocks signals around its own startup hits 20; and
anything calling posix_spawn() on x86_64 hits 22, which is every libc's
system() and popen() too.
What it took: four PostgreSQL patches #
| file | why |
|---|---|
allow-root.patch |
Upstream's, trimmed to the two programs this image ships. A unikernel has one user, uid 0, and nothing to drop privileges to. |
no-sysv-ipc.patch |
There is no System V IPC, so shmget() answers ENOSYS. Anonymous memory is not a downgrade here: one page table means an anonymous mapping is shared between the postmaster and its children by construction, at the same address — which is exactly what the segment was for. The second hunk is the child side, where re-attaching is a no-op. |
dsm-single-address-space.patch |
Dynamic shared memory cannot be turned off (PostgreSQL 12 removed none) and maps a file MAP_SHARED, which Unikraft supports only for anonymous mappings. The rewrite demotes the file from storage to a name: it records the address of an anonymous mapping, and attaching becomes a lookup. |
exec-backend-posix-spawn.patch |
The fork+exec in internal_forkexec() becomes one posix_spawn(). Hand-writing vfork() here would be undefined behaviour — fork_process() assigns to globals that would land in the parent's memory — and musl's posix_spawn() is precisely this sequence done safely. |
exec-backend-self-path.patch |
The postmaster execs itself, so skip find_other_exec() — which locates a postgres beside argv[0] and version-checks it by running "<path>" -V through popen(): a /bin/sh plus a second postgres before the server has done anything. In a unikernel there is exactly one postgres and it is the one running. This is also what unblocked startup: with those two extra process lifetimes gone, the startup child no longer hits a heap-consistency abort during WAL recovery (see below). |
Three more things are configuration rather than patches, all in the
Dockerfile:
- No Unix-domain socket. Binding one ends in
chmod(), which the guest does not implement (could not create any Unix-domain sockets). Clients arrive over the forwarded TCP port anyway. max_stack_depth = 1MB, against the 2 MiB stacks the Kraftfile sets — see below.- Background workers off. Each one is another process to spawn; the aux processes the postmaster always starts are not covered by those settings and do run.
Other things worth knowing #
The root filesystem may not contain hard links. ramfs_link is literally
vfscore_vop_eperm, and PostgreSQL's bundled timezone database is built almost
entirely out of them, so the cpio extractor stops the boot dead. build.sh
replaces each with a private copy — and has to do it after docker export,
because BuildKit deduplicates identical files into hard links as it writes each
COPY: a tree that leaves the build stage with none arrives in the image with
245.
RLIMIT_STACK describes the wrong stack. getrlimit() answers
__STACK_SIZE, the thread stack size, while the application runs on the stack
app-elfloader allocated for it. PostgreSQL subtracts a fixed 512 KiB from the
reported value and refuses to start if what is left is under max_stack_depth,
whose minimum is 100 KiB — so with the default 64 KiB no value is accepted:
DETAIL: "max_stack_depth" must not exceed -448kB.
The Kraftfile raises both stacks to 2 MiB: the reported one and the real one, since raising only the report hands PostgreSQL a limit it could overrun.
kraftwill not change a symbol in an already-generated.config. After editing kconfig in the Kraftfile,rm .config.postgres_fc-*or the build silently keeps the old values.
There is still a shell in the image even though nothing invokes it at boot
any more (exec-backend-self-path.patch removed the popen that needed it) —
it stays because archive recovery and COPY FROM PROGRAM also go through
/bin/sh, and at 800 KiB busybox is cheap. It links against this image's musl
and — the part that matters — is a PIE, which execve() requires on a
single-address-space kernel: the loader has to place it somewhere other than
where the caller is running.
libkrun appends words to the application's argv (earlycon=, tsi_hijack,
a bare --), because they land after the -- stop sequence. It does not bite
here: they arrive after -D <dir>, and PostgreSQL's getopt consumes the --.
It does bite postgres --single, which takes a database name as a non-option
argument — omit the name there and the stray word is absorbed instead.
Debugging a guest crash #
The e2e workflow resolves the addresses in a Unikraft crash banner to symbols
(.github/scripts/symbolicate-unikraft-crash.py), which is how patch 22 was
found: the banner's rip became [ld-musl-x86_64.so.1] pthread_exit+0x17f,
the thread-list unlink, over a "self" pointer in the kernel heap.
Two knobs in the Kraftfile make that work, both off by default because they
are noisy:
CONFIG_APPELFLOADER_DEBUG: 'y'— the loader'sloaded to 0x…-0x…lines, which the symbolicator needs to attribute an application-range address to postgres rather than musl and compute the offset.CONFIG_LIBSYSCALL_SHIM_STRACE: 'y'— every syscall with arguments and result. This is what showed the postmaster's spawn sequence ending inclone(CLONE_VM|CLONE_VFORK…)with nothing after it.
postgres --single — the whole server in one process, SQL on stdin, no
postmaster and no children — also works on the same image, from a terminal
(piped stdin does not reach the guest; the console reads EOF immediately).
Differences from upstream #
No runtime: base-compat:latest — that runtime is Unikraft Cloud's and is
not published for arm64, so this Kraftfile builds library/base from source
like the other examples here, plus multiprocess, signalfd and larger stacks.
No wrapper.sh — upstream boots into a bash port of docker-library's
entrypoint, which runs initdb, starts a temporary server through pg_ctl and
creates the database with psql. All of that happens in the Dockerfile here, so
the guest starts where upstream's wrapper finishes.
No pg_ukc_scaletozero — a Unikraft Cloud scale-to-zero integration, with
nothing to do with booting.
A much smaller build — ICU, readline, zlib, lz4, libxml and libxslt are configured out, about 40 MiB, doubled: a Unikraft root filesystem is resident twice (embedded in the kernel image and unpacked into ramfs).
--cmdline is required #
bsdkrun does not read the Kraftfile's cmd for a locally-built kernel. The
format is <argv0> -- <application argv>; everything before -- is parsed as
kernel library parameters and the first word is skipped, so the leading
placeholder is not optional.