`hosts/` #

Four ways to write one program so that a foreign host can drive it, each run from five languages and eight bindings. Everything here is self-contained: nothing under hosts/ imports from the rest of the repository, and the four directories do not import from each other. Where they share code, it is copied, so that any one of them can be read — or handed to a colleague — on its own.

A larger program, in two of these styles, is one level up in scale/; the per-step and end-to-end measurements of every host below are in bench/.

The program #

The same greeter as everywhere else in this repo: a loop with four waits of different kinds (a line of input, a lookup, a sleep, a counter) and fire-and-forget writes between them. Two participants, alice and bob, run concurrently. In the three async experiments a third, carol, runs the fanout program: the same vocabulary, but two waits at a time (driver::join), so a batch carries two requests and the host answers them in whichever order it likes — the thing the request ids exist for, and the thing sans-io's one-witness machine cannot do. Every host in this directory produces the same conversation; what differs is who does the waiting and on which thread.

The four #

The table contrasts the two ends; effects/ and coeffects/ sit between them and share the capabilities/ column's mechanism exactly — same driver, same host_ffi, same wire, different program.

	`sans-io/`	the three async styles
the program is	a hand-written state enum with typed `Token<R>` witnesses	an `async fn` written against five capability traits, implementing `Run`
its host API is	`start() → effects`, `answer(token, value) → effects`, `is_finished()`	the same, by design
who wrote the state machine	you	the compiler
the state is	plain data at every wait: movable, `Send`, serialisable (`serde` feature); checkpoint/resume tested	a pinned future within a turn; plain data between turns in principle (the struct's fields), which the driver deliberately does not expose — a crossing per turn for little
waits outstanding	at most one	any number (fan-out) — hence request ids on the wire; `fanout` exercises it in every host
the driver	`rust/driver`: `Token<R>` + `Answer<M>`, no runtime — the machine is already steppable	`rust/driver`: `local` (`Rc`, thread-affine, zero synchronisation) or `sync` (`Arc` + one lock, `Send`)

sans-io/ is the control. The other three are the design under test: direct-style code, no runtime, the same host-facing shape as sans-io — and the sync driver is the effectory runner from the rest of the repo, remade here in ~250 lines so the two threading models can be read side by side.

Layout, per directory #

<experiment>/
├── ABI.md                  the contract every host below assumes; the only thing that makes them "generic"
├── include/                the C ABI as a header, generated by cbindgen from rust/cdylib (see include/README.md)
├── rust/
│   ├── core/               the program (no_std)
│   ├── driver/             the mechanism the program is written against, generic over programs (no_std)
│   │                         async: Run, local + sync drivers, wire traits + codec, the coeffect path (Request/Asked/Ask)
│   │                         sans-io: Token<R>, Machine, Answer<M> — the witness, without a runtime
│   ├── host_ffi/           typed Machine (start / answer_* → Seen) with a byte layer (step) over it; tables — safe Rust
│   │                         sans-io: Held, the start-once / one-witness guard the typed Machine sits on
│   ├── wire/               the host-side vocabulary and its encoding (coeffects: Full + Quiet; capabilities: Effect; effects: none — core is the wire)
│   ├── harness/            the tokio world the native host runs in: scripted input, actors, tagged output — style-free, identical ×4
│   ├── cdylib/             the C-ABI skin: new / step / free / buf_free      → lib<experiment>_ffi.so
│   ├── nif/                the erl_nif skin, by hand                          → lib<experiment>_nif.so
│   ├── rustler/            the erl_nif skin, generated by Rustler             → lib<experiment>_rustler.so
│   ├── pyo3/               a Python extension module, generated by PyO3       → <experiment>_pyo3.so
│   └── tokio/              a native Rust host, for contrast: the per-style part only — a loop over a driver, or trait impls over the harness
├── java/       abi/  panama/  jextract/     hand-written descriptors · generated from include/*.h
├── python/     abi/  ctypes/  pyo3/         bytes over the C ABI · Python objects from Rust
├── lua/        abi/  ffi/                   LuaJIT ffi; the driver is a coroutine
├── go/         abi/  cgo/                   cgo over include/*.h
└── elixir/     abi/  nif/  rustler/         hand-rolled entry · generated by Rustler

Beside the four: scale/ — a larger program (a library desk: three actors, eleven waits, struct replies, a shared helper, fan-out) in capabilities and sans-io with one host each, to test the review's Scaling claims; see scale/README.md. And bench/ — nanoseconds and allocations per step for every Rust layer of every experiment and for every host binding, with the sleeps answered at once; raw samples, charts and percentiles under bench/results/. See bench/README.md; the short version is that the whole Rust stack from state machine to byte layer costs ~180 ns per step, and the cheapest host crossing costs nearly that again.

Each language directory has two halves. <lang>/abi/ is ABI.md in that language — codes, byte primitives, library loading and the out-buffer protocol, a scheduler shaped by the language's threading — and would be the same for any program speaking the contract. <lang>/<binding>/ is this program's host over one binding technique: its codec (which tag means which effect), its host (scripted input, lookup table, counter, tagged stdout), its driver (the step loop), and main. Where a language has two hosts, they sit side by side and share abi/ — and the diff between them is the point (see below).

The split mirrors the Rust side, where rust/host_ffi is the mechanism behind the ABI and rust/{cdylib,nif} are skins over it. The abi/ packages are byte-identical across all four; check-sync.sh proves it.

run.sh in each <lang>/<binding>/ builds the Rust artefact and runs the host. All of it works from nix develop at the repo root. check-sync.sh verifies the copied mechanism (rust/{driver,host_ffi} across the async three, rust/harness and every <lang>/abi across all four) has not drifted between experiments, and with --headers (needs cbindgen; exits 2 without it) that each include/*.h matches its crate.

What the hosts show #

The interesting variable is threading. A handle into the local driver is valid only on the thread that made it (its table is a thread_local!, because an Rc cannot live in a static). A handle into the sync driver, or into a sans-io machine, is valid from any thread, one at a time. Each language picks the shape it naturally has, and the output tags every line with the thread that produced it:

host	binding	threading it has	driver it uses	what you see in the tags
Java	Panama	executors: a pool	`sync`	steps hop `pool-1 … pool-4`
Go	cgo	goroutines: migrate across OS threads unless `LockOSThread`	`sync`; `--local` pins two shards	tids wander; with `--local`, alice and bob each keep one tid
Elixir	NIF	processes on scheduler threads; a NIF runs on whichever has the process, and nothing can pin it	`sync`, the only option	scheduler ids change run to run — the host that forces `Send`
Python	ctypes	OS threads under the GIL: Python code is serialised anyway	`local`, one loop	everything on `loop-1`
Lua	LuaJIT `ffi`	none — one interpreter state, coroutines only	`local`, the only sensible option	everything on the one thread; drivers are coroutines yielding on `Sleep`
Rust (`tokio/`)	—	work-stealing tasks	neither: tokio polls the `async fn` directly	thread ids wander

Read across the row for capabilities and you have the whole argument: the Rc driver is the smallest runtime that can suspend an async fn, and it fits Python and Lua exactly; Java, Go, and the BEAM want migration, and the Send driver is what that costs. Sans-io never has the question, because its state is plain data — which is the thing it made you write by hand.

But a library should ship one driver #

Building all eight hosts on both drivers settled something the greeter alone could not. Effects here are pull-only — the host calls in, Rust never calls out, no callback is ever registered — so no foreign value enters a Rust context, and the only !Send thing in the tree is the Rc that local puts there itself. Python's objects are !Send in Python; Lua's state is !Send in Lua; Rust sees bytes. The one exception is self-inflicted: the PyO3 Greeter is #[pyclass(unsendable)] because it wraps local::Driver, and over sync it would be an ordinary Send class.

So a library extracted from this should ship sync only, and document the way to hold a !Send thing as !Send shell, Send actors, effects between — the foreign values stay in the wrapper the runtime hands you anyway, and the actor crate holds nothing but its outbox:

flowchart LR
  subgraph shell["NOT-Send shell &mdash; whatever the runtime hands you"]
    direction TB
    JS["wasm_bindgen struct<br/>PyO3 class &middot; NIF resource<br/><br/>holds JsValue, PyObject,<br/>lua_State, callbacks"]
    LOOP["host loop:<br/>perform the effect,<br/>answer by token"]
  end

  subgraph core["Send core &mdash; the actor crate"]
    direction TB
    D["Driver<br/>mailbox, one Box::pin"]
    A["actor: async fn<br/>holds an outbox,<br/>nothing foreign"]
  end

  JS -- "inbound: an ordinary call.<br/>Send permits moving, not calling" --> D
  D --> A
  A -- "outbound: emits an effect" --> LOOP
  LOOP -- "the answer, by token" --> D

  classDef notsend fill:#fdeaea,stroke:#c0504d,color:#222
  classDef sendy fill:#e9f1fb,stroke:#4f81bd,color:#222
  class shell notsend
  class core sendy

Inbound is free: a !Send wrapper may hold a Send core and call straight into it, because Send is permission to move a value rather than to call it. Outbound is the effect: the actor cannot hold the foreign value, so it emits a request and the shell answers — the general confine-and-proxy pattern with the channel and the delegate task removed, since the driver's mailbox is the channel and the host loop is the delegate. The payoff is not on the !Send side: with the actors Send, the same actor crate also runs on a host with a thread pool.

The capability traits make that swap invisible — Ctx is a direct call in one build and a proxy in another, and core cannot tell. rust/harness's TokioCtx is already this shape for Count and Lookup.

The price is ~45 ns per step, a quarter of the byte layer and 1–10% of a host crossing. Two things are not free, though: on targets without CAS (thumbv6m and friends) spin::Mutex will not build and the Rc driver is the only one that compiles, so local survives as the portability path behind a feature flag; and sharded schedulers stop being a correctness requirement and become a performance choice, with BUSY turning from an impossibility into a live path every host must handle. The full argument is in analysis/hosts-review.md, under "Ship one driver".

Both drivers stay here, because the two threading models side by side are one of the findings.

Manual and generated, side by side #

The primary host in each language is deliberately manual: ctypes, hand-written Panama descriptors, a hand-rolled erl_nif entry (rust/nif, with the OTP header layouts copied in), cgo. These are not the idiomatic choices, and they exist because the claim under test is the raw one from ABI.md: any host can decode these bytes with a byte buffer and no library. Six languages doing exactly that, with the same new / step / free and the same codec in each, is the evidence.

The idiomatic choices sit beside them, so the diff is readable:

language	manual	generated	what the generator is	what disappears	what appears
Python	`ctypes/`	`pyo3/`	Rust defines Python types (`#[pyclass]`) via CPython's embedding API	`codec`, `native`, codes → exceptions	a Rust crate that knows Python; a build needing an interpreter; a `.so` only Python loads
Elixir	`nif/`	`rustler/`	Rust defines a NIF via macros; effects encode as terms	`codec`, byte protocol → `{:ok, effects}`	a `rustler` dependency; the NIF API version pinned by feature
Java	`panama/`	`jextract/`	Java tool emits bindings from the C header (`include/*.h`, itself generated by `cbindgen`)	hand-typed `FunctionDescriptor`s, `abi/Library.java`	a JDK ≥ 22 for the final FFM API; a generate step
Go	—	`cgo/`	cgo is a header-driven generator; it includes `include/*.h`	the hand-typed preamble	nothing — this was always the shape

Two different kinds of generator are in that table. PyO3 and Rustler replace the ABI with the host language's own object model — possible because CPython and the BEAM have embedding APIs that let native code define first-class objects. jextract and cgo consume the ABI: they generate the binding, but the codec stays, because a header can say int32_t step(uint64_t, const uint8_t*, size_t, uint8_t**, size_t*) and not what the bytes mean. The JVM and Go have no embedding API, so the first kind does not exist for them; UniFFI (Mozilla) is the nearest thing — a Rust-side generator emitting Kotlin/Swift/Python/Go source over a C ABI with serialised buffers, which is to say this design, generated. It is discussed in ../analysis/hosts-prior-art.md rather than built here.

Every skin — C ABI, hand-rolled NIF, Rustler, PyO3 — sits on the same typed layer, host_ffi::Machine (or sansio_wire::Machine): start() and answer_str(id, …) return the effects as Seen values, tokens already swapped for ids and the kind check already done. The byte skins add a codec on top; the object-model skins map Seen to a Python enum or an Erlang tuple. None of them does its own bookkeeping, and adding all four touched neither core nor driver.

Two of the generated variants turn a threading argument into a type. PyO3's #[pyclass(unsendable)] on the local driver's Greeter makes PyO3 enforce thread affinity — it checks the thread on every call and on drop. Rustler's ResourceArc<T> requires T: Send + Sync, so the local driver does not compile there and the sync one must sit behind a Mutex: the BEAM's threading model, stated by the borrow checker.

Where this came from #

../analysis/hosts-prior-art.md traces each part to its precedents — tagless final, embedded-hal-async, and Tower for the program shape, sans-io's tokens and Kotlin's Continuation<T> for the typed waits, zlib/SQLite/quiche for the C ABI, SWIG through UniFFI and Diplomat for the bindings — and states the four things it could not find elsewhere.

Where to start reading #

capabilities/rust/core/src/lib.rs — the program, 60 lines, no mention of any runtime. Then effects/rust/core and coeffects/rust/core — the same program, owning its vocabulary, and stating it as requirements.
capabilities/rust/driver/src/drive.rs — why an async fn needs a mailbox and sans-io does not; then local.rs and sync.rs side by side.
capabilities/ABI.md, then any <lang>/<binding>/driver.* — they are the same loop in six syntaxes.
capabilities/rust/nif/src/lib.rs — the whole erl_nif ABI, if you have never seen one.

hosts/ #