ceetuber
Star
Fork
Watch
ceetuber docs WALKTHROUGH.md
8.7 kB · 84 lines
Markdown
at main

From face tracking to a soft little ghost #

Ceetuber separates the tracking device, shared pose relay, and avatar modules. The default is a procedural rectangle; the ghost below demonstrates a specialized adapter. See avatar modules to create another character. The phone uses ARKit, Node receives UDP and broadcasts pose events, and each browser renders its own copy of the character.

iPhone ARKit → UDP JSON → PhoneTracker → shared pose → SSE → Three.js
Windows XInput helper → NativeController ────┘                ↓
browser sliders / Gamepad API → local pose or HTTP relay    OBS alpha

The hosted demo has no Node receiver. Browser input drives that page directly; hosting the renderer does not expose phone LAN telemetry to the website.

1. Reduce tracking to useful numbers #

FaceSender.swift runs an ARSession with ARFaceTrackingConfiguration. Each tracked face supplies:

  • Head rotation: inverse camera transform × face transform, converted to a normalized quaternion. This measures the face relative to the phone rather than the room.
  • Eye direction: average the forward axes of leftEyeTransform and rightEyeTransform, then derive horizontal and vertical angles. The transforms are head-local, separating eye gaze from head turning.
  • Blinks: independent eyeBlinkLeft and eyeBlinkRight blend-shape coefficients.

Send those values as small JSON UDP datagrams at ARKit's frame rate. No images or face mesh leave the phone. This implementation uses one combined eye direction, not calibrated screen-point gaze or independent vergence. It does not map mouth expressions.

Apple references: ARFaceAnchor, ARFaceTrackingConfiguration, Network.

2. Validate, center, and map #

phone-tracker.mjs checks packet size, schema, session and sequence, finite values, quaternion length, gaze bounds, and blink range. It accepts private IPv4 and loopback sources and pins the first sender IP until Release phone. This avoids accidental source mixing; it is not authentication.

The first tracked frame establishes a neutral head quaternion and gaze pair. Center captures a new neutral:

relative rotation = inverse(center rotation) × current rotation
gaze offset = current gaze − center gaze

The relative quaternion becomes XYZ Euler angles in degrees. Clamp raw yaw to ±30° and raw pitch/roll to ±20°, then apply main and per-axis gains. Clamping before gain makes a higher gain expand output travel. Gaze similarly normalizes the centered angles, clamps to ±1, and scales. While either eye is mostly closed, hold gaze to avoid blink-related jumps.

The character pose is seven numbers: yaw, pitch, roll, gazeX, gazeY, blinkL, blinkR. Face loss neutralizes the pose; missing packets time out after 1.2 seconds. A new sender session resets calibration. See the wire protocol.

3. Read controls outside browser focus #

Browser gamepads are convenient for the public demo, but access depends on browser support, interaction, and focus. native-controller.mjs starts xinput.ps1, which polls Windows XInput and emits newline-delimited JSON. Node normalizes axes and buttons and uses the same padPose mapping as the browser.

serve.mjs owns the shared pose. Explicit source selection disables the other native source, and active phone/controller input takes priority over stale manual posts. The /api/pose Server-Sent Events stream lets several renderers subscribe without owning separate tracking connections. OBS only needs this stream and assets.

HTTP binds to 127.0.0.1. Mutation endpoints check origin, content type, and body size. Settings saves normalize known fields and atomically replace a disk file. Calibration stays in memory because it belongs to the current camera position.

4. Smooth, then deform #

controls.mjs smooths with exponential decay using elapsed seconds rather than a fixed fraction per frame. Blinks have a faster response. deformation.mjs applies several layers:

Head: preserve the bone's rest quaternion and multiply by local rotation. glTF axis conversion is already in the rest transforms; another conversion here would twist the rig.

Eyes: at startup, bake each button's rigid skin binding into head-parented geometry, preserving Blink morphs. At runtime, translate it in the face plane and bilinearly sample a depth grid from rig.json to keep it near the drape. Leave the relative quaternion at identity. A button translates like a drawn-on eye instead of rotating like an eyeball.

Sheet: six weighted ring bones distribute head motion around the neck pivot. A height-based blend leaves the hem planted and follows the head increasingly toward the top. Critically damped springs delay part of each ring's response. Their exact analytic step avoids integration drift; skin weights provide continuity. This is procedural deformation, not fabric folding, cloth collision, or self-contact.

Hat: retain the weighted chain and rigid base; progressively delay hat03 through hat07. Hat gain amplifies the difference between immediate head motion and delayed tail motion. When the head stops, the tail settles toward its authored shape.

5. Give hanging objects physical state #

ornament-physics.mjs creates twenty pendulums. Each has persistent world position and velocity, a moving attachment, tether length, collision radius, and mass weighting. Motion comes from gravity, acceleration, inertia, and contacts rather than canned jostle curves.

The accumulator advances at 120 fixed steps per second. Substeps integrate motion, enforce tether lengths and sphere contacts, and approximate sheet contact with a local plane. Twelve constraint iterations improve convergence. The resulting offset rotates each bone, keeping ball, cap, and loop together.

The pom becomes a twenty-first body. Its rigid skin is converted into a mesh under a hinge attached to hat07; fluff follows the hinge. An angular spring pulls toward the authored direction, resistance grows near a cone boundary, and projection enforces the final maximum angle. An implicit spring step supports the wide stiffness range.

Air damping, gravity, angular range, and soft-edge start are separate controls. An 8% edge start begins resistance at 8% of maximum wobble, not an extra 8°. Zero range locks the pom to the moving attachment. Contacts and bounds can dissipate energy even with zero air damping.

Fixed substeps make normal motion comparable at 30, 60, and 120 render FPS, not bit-identical. Sampling, moving anchors, and capped stalls still matter. Large anchor teleports reset simulation. This small character solver is not a general rigid-body engine.

6. Recover Blender detail #

Procedural Blender shader nodes do not transfer directly to glTF. The bake recipe makes 2048-pixel color, tangent-normal, and roughness maps for cotton, felt, and lining. Normal maps approximate fine displacement. Sampled real fiber geometry restores fuzz and loose threads at silhouettes.

appearance.mjs restores materials and replaces ornament surfaces with smoother skinned spheres. A generated 512-pixel HDR studio cubemap supplies bright shapes for metal to reflect. Individual 256-pixel probes capture the nearby ghost; one ornament updates every 125 ms, or eight updates per second total across twenty ornaments. Ornaments are hidden during capture to avoid self-reflection.

These are rasterized approximations, not path tracing. Disable detailed reflections to use the shared studio environment and reduce GPU work. The dense hero model and cube captures can be expensive on mobile; production avatars may need lower geometry/texture budgets and fewer probes.

7. Composite through alpha #

app.mjs owns scene setup, UI wiring, source selection, and rendering. The ?output=1 page has a transparent canvas and hides the tester, background, and floor. OBS composites alpha directly without green spill or chroma key.

The seven-value pose boundary is the useful extension point: another tracker can produce it, another character can consume it, and the local relay can feed several renderers. Begin a remix there before changing physics or materials.