Video is the most important media format on the web, but content-addressed systems don't know what to do with it. DASL gives us DRISL for deterministic serialization of structured data ([[drisl]]) and CIDs for content identifiers ([[cid]]). But MP4 files — the dominant container format for video — resist content-addressing. Run the same video through ffmpeg twice with identical settings and you'll get different bytes and no stable CID. MUXL solves this problem by introducing a deterministic, highly-restrictive subset of ISO-BMFF.

This format attempts to fulfill three design goals: our format should be concatenatable, self-contained, and maximally-compatible. Achieving all three of these goals is impossible for a single MP4 format due to restrictions in the spec. To resolve this, MUXL introduces two presentation formats, which bridge the gap between our base format and media playback software. These presentation formats are prepended onto the canonical MUXL segments themselves, providing maximal compatibility while leaving the underlying format intact.

Design goals

• Concatenatable: it's really nice if our segments can be naively combined by adding new bytes to the end. It lets us use BLAKE-3 hashing ([[blake3]]) and BDASL CIDs ([[bdasl]]) for verifiable transfer of subsets of the input.
• Self-contained: Every piece of media needs to be self-contained, independently playable and interpretable without other external data. (HLS spells this property EXT-X-INDEPENDENT-SEGMENTS.) This is important to make our segments self-certifying and prevent bad actors from distorting the data.
• Maximally-compatible: We don't want to invent a new bespoke media format. A long video should present to users as one big MP4 file.

Base Format: MUXL fragments

The foundational format of MUXL is the MUXL fragment. Incoming frames are stripped of all metadata and wrapped in a CMAF format inspired by moq-lite's Hang CMAF format ([[hang]]). The MP4 atoms are:

[moof][mdat]

Once incoming data is wrapped like this, it's minted and may never change. All derived formats work with these; this is the "minting" process for MUXL. Each fragment has exactly two pieces of metadata: a decoding timestamp and a sequence number.

Segmentation Format: MUXL segments

Unfortunately, MUXL fragments can't be played without some additional metadata. The raw video data is present, but the media data, such as the resolution, orientation, and pixel format is absent. We borrow another term from Hang ([[hang]]) and call this the "catalog data".

Traditionally in MP4 and CMAF, this would be provided from "init segment" data: basically an MP4 file header containing a moov atom that identifies the media type. Usually, this init segment is provided out-of-band in an HLS playlist. That doesn't work for us because it means our segments are no longer self-contained. And doing it in-band doesn't work either: an MP4 file cannot contain multiple ftyp and moov atoms, so we'd be breaking concatenatability.

We've instead opted to include the catalog data as CBOR and prepend it to every segment using BMFF's path for generic extensibility: the uuid atom. This makes a small size tradeoff; each MUXL segment for a stream now contains redundant catalog data, but this is necessary to make the segments self-contained. As a bonus, we can use the presence of the uuid atom as a segmentation heuristic; when you encounter a uuid atom in a MUXL stream you know you've encountered a new self-contained set of fragments.

When minting new segments that contain video, it is encouraged to cut all tracks based on video keyframe boundaries to facilitate efficient playback. Video segments should not be minted without keyframe data. Audio-only segments without a reference video track should segment at 1-second intervals.

So, this gives us the definition of a MUXL segment: a uuid atom containing the catalog data followed by some MUXL fragments for one track of the input stream. To make this as minimal and deterministic as possible, we encode it with DRISL CBOR ([[drisl]]). So, a MUXL segment becomes:

[uuid-muxl][moof][mdat][moof][mdat][moof][mdat]....

This segment is now a self-contained .m4s file, suitable as a signing target and also freely concatenatable with other MUXL segments. Multiple tracks may be combined in a single blob by interleaving segments by timestamp.

Catalog {
  video?: { renditions: { <name>: VideoConfig }, display?, rotation?, flip? }
  audio?: { renditions: { <name>: AudioConfig } }
}

Self-certifying Format: S2PA

The full methodology for signing is part of the C2PA and S2PA ([[s2pa]]) specs and exceeds this documentation; all we need to know is that they're going to include another uuid atom containing the signing and provenance data. So, our format is now:

[uuid-c2pa][uuid-muxl][moof][mdat][moof][mdat]....

This constitutes the only optionality in the MUXL spec: segments may be prepended with additional uuid atoms as required by the use case. This allows the MUXL spec itself to stay minimal while providing the necessary extension points.

Presentation Formats: MP4 and fMP4

What we have now are .m4s files: CMAF fragment data alongside the catalog data necessary to play them. But we just invented this format so no video players are actually capable of playing it yet. To facilitate actual playback, we introduce two presentation formats: fMP4 and Flat MP4. Both presentation formats work by reading the catalog data from the segments and synthesizing an MP4 header that is prepended to the MUXL segments. The bytes of the segments themselves remain unaltered and are easily recoverable by stripping all of the generated metadata through to the first uuid atom. Because neither format actually modifies the canonical MUXL segments, any hash or signature over that segment stays valid as soon as you discard the presentation header.