Something went wrong. Try again.
Mirror of https://github.com/ssbc/scuttlebutt-protocol-guide
Something went wrong. Try again.
scuttlebutt-protocol-guide index.html
155 kB · 2072 lines
HTML
at master
123456789101112131415161718192021222324252627282930313233343536373839404142434445464748495051525354555657585960616263646566676869707172737475767778798081828384858687888990919293949596979899100101102103104105106107108109110111112113114115116117118119120121122123124125126127128129130131132133134135136137138139140141142143144145146147148149150151152153154155156157158159160161162163164165166167168169170171172173174175176177178179180181182183184185186187188189190191192193194195196197198199200201202203204205206207208209210211212213214215216217218219220221222223224225226227228229230231232233234235236237238239240241242243244245246247248249250251252253254255256257258259260261262263264265266267268269270271272273274275276277278279280281282283284285286287288289290291292293294295296297298299300301302303304305306307308309310311312313314315316317318319320321322323324325326327328329330331332333334335336337338339340341342343344345346347348349350351352353354355356357358359360361362363364365366367368369370371372373374375376377378379380381382383384385386387388389390391392393394395396397398399400401402403404405406407408409410411412413414415416417418419420421422423424425426427428429430431432433434435436437438439440441442443444445446447448449450451452453454455456457458459460461462463464465466467468469470471472473474475476477478479480481482483484485486487488489490491492493494495496497498499500501502503504505506507508509510511512513514515516517518519520521522523524525526527528529530531532533534535536537538539540541542543544545546547548549550551552553554555556557558559560561562563564565566567568569570571572573574575576577578579580581582583584585586587588589590591592593594595596597598599600601602603604605606607608609610611612613614615616617618619620621622623624625626627628629630631632633634635636637638639640641642643644645646647648649650651652653654655656657658659660661662663664665666667668669670671672673674675676677678679680681682683684685686687688689690691692693694695696697698699700701702703704705706707708709710711712713714715716717718719720721722723724725726727728729730731732733734735736737738739740741742743744745746747748749750751752753754755756757758759760761762763764765766767768769770771772773774775776777778779780781782783784785786787788789790791792793794795796797798799800801802803804805806807808809810811812813814815816817818819820821822823824825826827828829830831832833834835836837838839840841842843844845846847848849850851852853854855856857858859860861862863864865866867868869870871872873874875876877878879880881882883884885886887888889890891892893894895896897898899900901902903904905906907908909910911912913914915916917918919920921922923924925926927928929930931932933934935936937938939940941942943944945946947948949950951952953954955956957958959960961962963964965966967968969970971972973974975976977978979980981982983984985986987988989990991992993994995996997998999100010011002100310041005100610071008100910101011101210131014101510161017101810191020102110221023102410251026102710281029103010311032103310341035103610371038103910401041104210431044104510461047104810491050105110521053105410551056105710581059106010611062106310641065106610671068106910701071107210731074107510761077107810791080108110821083108410851086108710881089109010911092109310941095109610971098109911001101110211031104110511061107110811091110111111121113111411151116111711181119112011211122112311241125112611271128112911301131113211331134113511361137113811391140114111421143114411451146114711481149115011511152115311541155115611571158115911601161116211631164116511661167116811691170117111721173117411751176117711781179118011811182118311841185118611871188118911901191119211931194119511961197119811991200120112021203120412051206120712081209121012111212121312141215121612171218121912201221122212231224122512261227122812291230123112321233123412351236123712381239124012411242124312441245124612471248124912501251125212531254125512561257125812591260126112621263126412651266126712681269127012711272127312741275127612771278127912801281128212831284128512861287128812891290129112921293129412951296129712981299130013011302130313041305130613071308130913101311131213131314131513161317131813191320132113221323132413251326132713281329133013311332133313341335133613371338133913401341134213431344134513461347134813491350135113521353135413551356135713581359136013611362136313641365136613671368136913701371137213731374137513761377137813791380138113821383138413851386138713881389139013911392139313941395139613971398139914001401140214031404140514061407140814091410141114121413141414151416141714181419142014211422142314241425142614271428142914301431143214331434143514361437143814391440144114421443144414451446144714481449145014511452145314541455145614571458145914601461146214631464146514661467146814691470147114721473147414751476147714781479148014811482148314841485148614871488148914901491149214931494149514961497149814991500150115021503150415051506150715081509151015111512151315141515151615171518151915201521152215231524152515261527152815291530153115321533153415351536153715381539154015411542154315441545154615471548154915501551155215531554155515561557155815591560156115621563156415651566156715681569157015711572157315741575157615771578157915801581158215831584158515861587158815891590159115921593159415951596159715981599160016011602160316041605160616071608160916101611161216131614161516161617161816191620162116221623162416251626162716281629163016311632163316341635163616371638163916401641164216431644164516461647164816491650165116521653165416551656165716581659166016611662166316641665166616671668166916701671167216731674167516761677167816791680168116821683168416851686168716881689169016911692169316941695169616971698169917001701170217031704170517061707170817091710171117121713171417151716171717181719172017211722172317241725172617271728172917301731173217331734173517361737173817391740174117421743174417451746174717481749175017511752175317541755175617571758175917601761176217631764176517661767176817691770177117721773177417751776177717781779178017811782178317841785178617871788178917901791179217931794179517961797179817991800180118021803180418051806180718081809181018111812181318141815181618171818181918201821182218231824182518261827182818291830183118321833183418351836183718381839184018411842184318441845184618471848184918501851185218531854185518561857185818591860186118621863186418651866186718681869187018711872187318741875187618771878187918801881188218831884188518861887188818891890189118921893189418951896189718981899190019011902190319041905190619071908190919101911191219131914191519161917191819191920192119221923192419251926192719281929193019311932193319341935193619371938193919401941194219431944194519461947194819491950195119521953195419551956195719581959196019611962196319641965196619671968196919701971197219731974197519761977197819791980198119821983198419851986198719881989199019911992199319941995199619971998199920002001200220032004200520062007200820092010201120122013201420152016201720182019202020212022202320242025202620272028202920302031203220332034203520362037203820392040204120422043204420452046204720482049205020512052205320542055205620572058205920602061206220632064206520662067206820692070207120722073<!doctype html><html><head> <meta charset="utf-8"/> <title>Scuttlebutt Protocol Guide</title> <link rel="icon" href="img/favicon.png"/> <link rel="stylesheet" href="css/fonts.css"/> <link rel="stylesheet" href="css/guide.css"/></head><body> <header> <div class="title"> <a href="index.html"> <img src="img/logo.png" alt=""/> <span>Scuttlebutt Protocol Guide</span> </a> </div> <nav> <a href="https://github.com/ssbc/scuttlebutt-protocol-guide"><div>Repo</div></a> </nav> </header> <main> <h1>Scuttlebutt Protocol Guide</h1> <p class="subtitle">How Scuttlebutt peers find and talk to each other</p> <hr/> <aside class="kicker section-kicker"> <h2 id="toc">Contents</h2> </aside> <div class="toc"> <div class="col"> <div><a class="s1" href="#introduction">Introduction</a></div> <div><a class="s1" href="#keys-and-identities">Keys and identities</a></div> <div class="section"> <div><a class="s1" href="#discovery">Discovery</a></div> <div class="s2"> <div><a href="#local-network">Local network</a></div> <div><a href="#invite-code">Invite code</a></div> <div><a href="#pub-message">Pub message</a></div> </div> </div> <div class="section"> <div><a class="s1" href="#peer-connections">Peer connections</a></div> <div class="s2"> <div><a href="#handshake">Handshake</a></div> <div><a href="#box-stream">Box stream</a></div> <div><a href="#rpc-protocol">RPC protocol</a></div> </div> </div> </div> <div class="col"> <div class="section"> <div><a class="s1" href="#feeds">Feeds</a></div> <div class="s2"> <div><a href="#structure">Structure</a></div> <div><a href="#message-format">Message format</a></div> <div><a href="#createHistoryStream">createHistoryStream</a></div> <div><a href="#ebt-replication">EBT replication</a></div> </div> </div> <div class="section"> <div><a class="s1" href="#blobs">Blobs</a></div> <div class="s2"> <div><a href="#fetching">Fetching</a></div> <div><a href="#want-and-have">Want and have</a></div> </div> </div> </div> <div class="col"> <div class="section"> <div><a class="s1" href="#following">Following</a></div> <div class="s2"> <div><a href="#follow-graph">Follow graph</a></div> </div> </div> <div class="section"> <div><a class="s1" href="#pubs">Pubs</a></div> <div class="s2"> <div><a href="#invites">Invites</a></div> </div> </div> <div class="section"> <div><a class="s1" href="#rooms">Rooms</a></div> <div class="s2"> <div><a href="#rooms-tunnel">Tunnel</a></div> <div><a href="#rooms-joining">Joining</a></div> </div> </div> </div> <div class="col"> <div class="section"> <div><a class="s1" href="#private-messages">Private messages</a></div> <div class="s2"> <div><a href="#encrypting">Encrypting</a></div> <div><a href="#decrypting">Decrypting</a></div> </div> </div> </div> </div> <aside class="kicker section-kicker"> <h2 id="toc">Archived</h2> </aside> <div class="toc"> <div class="col"> <div class="section"> <div><a class="s1" href="#metafeeds">Metafeeds</a></div> <div class="s2"> <div><a href="#bendy-butt">Structure</a></div> <div><a href="#metafeed-contentSection">Content setion</a></div> <div><a href="#metafeed-announcing">Announcing</a></div> <div><a href="#metafeed-use-cases">Use Cases</a></div> <div><a href="#metafeed-partial-replication">Partial Replication</a></div> <div><a href="#metafeed-fusion-identity">Fusion Identity</a></div> <div><a href="#metafeed-network-identity">Network Identity</a></div> </div> </div> </div> </div> <hr/>
<p id="introduction" class="first"><a href="https://www.scuttlebutt.nz/">Scuttlebutt</a> is a protocol for building decentralized applications that work well offline and that no one person can control. Because there is no central server, Scuttlebutt clients connect to their peers to exchange information. This guide describes the protocols used to communicate within the Scuttlebutt network.</p> <p>Scuttlebutt is a flexible protocol, capable of supporting many different types of applications. One of its first applications was as a social network. This guide has a slight focus on how to use Scuttlebutt for social networking, but many of the explanations will still be useful if want to use it for something completely different, or are just curious how it works.</p>
<h2 id="cryptography">Cryptography</h2>
<h3 id="keys-and-identities">Keys and identities</h3> <p>The first thing a user needs to participate in Scuttlebutt is an identity. An identity is an Ed25519 key pair and typically represents a person, a device, a server or a bot. It’s normal for a person to have several Scuttlebutt identities.</p> <p>Upon starting for the first time, Scuttlebutt clients will automatically generate an Ed25519 key pair and save it in the user’s home folder under <code>.ssb/secret</code>.</p> <img src="img/identity_keypair.png" style="height: 80px;" alt="The Scuttlebutt identity is a long-term Ed25519 key pair." /> <p>Because identities are long and random, no coordination or permission is required to create a new one, which is essential to the network’s design.</p> <p>Later, a user can choose to give themselves a nickname or avatar to make themselves easier to refer to. Over time nicknames may change but identities stay the same. If a user loses their secret key or has it stolen they will need to generate a new identity and tell people to use their new one instead.</p> <p>The public key of an identity is presented to users and transmitted in some parts of the network protocol using this format:</p> <img src="img/format_public_key.png" style="height: 80px;" alt="@FCX/tsDLpubCPKKfIrw4gc+SQkHcaD17s7GI6i/ziWY=.ed25519 where everything but the @ prefix and .ed25519 suffix is the public-key, base64-encoded." /> <aside style="position: relative; top: 6px;"> <p>Throughout the protocol all instances of base64 are the variant that uses <code>+</code> and <code>/</code>. The final padding <code>=</code> is also required.</p> </aside> <p>The beginning <code>@</code> sign signifies that this represents a public key rather than a message or blob, which start with <code>%</code> and <code>&</code>. Each identity has an associated <a href="#feeds">feed</a>, which is a list of all the messages posted by that identity. This is why the identity is also called a <em>feed ID</em>.</p>
<h3 id="cryptographic-primitives">Cryptographic primitives</h3>
<p>The Scuttlebutt protocol relies on NaCl/libsodium's cryptobox primitives. This guide uses the following:</p>
<dl> <dt><code>nacl_scalarmult(n, p)</code></dt> <dd>This is <a href="https://doc.libsodium.org/advanced/scalar_multiplication">Libsodium's scalar multiplication function</a>, which takes two scalars (usually public and/or secret keys). It has the useful property that, given two key pairs <code>(pk1, sk1)</code> and <code>(pk2, sk2)</code>, <code>nacl_scalarmult(sk1, pk2) == nacl_scalarmult(sk2, pk1)</code>, which allows shared secret derivation between peers who know each other's public key. More on this later.</dd>
<dt><code>nacl_auth(msg, key)</code> and <code>assert_nacl_auth_verify(authenticator, msg, key)</code></dt> <dd>This functions are <a href="https://doc.libsodium.org/public-key_cryptography/authenticated_encryption">Libsodium's message authentication function</a>. The former takes a message and returns a 32-bytes authenticator, that acts as a detacted signature of the message. The latter verifies this authenticator is indeed valid for the given message and key; and errors if they don't.</dd>
<dt><code>nacl_secret_box(msg, nonce, key)</code> and <code>assert_nacl_secretbox_open(ciphertext, nonce, key)</code></dt> <dd>These function are based on <a href="https://doc.libsodium.org/secret-key_cryptography/secretbox">Libsodium's crypto_secretbox_easy and crypto_secretbox_open_easy function</a>, which use symmetric cryptography to, respectively, encrypt+authenticate, and verify+decrypt a message using a nonce and a shared secret.</dd>
<dt><code>nacl_sign_detached(msg, key)</code> and <code>assert_nacl_sign_verify_detached(sig, msg, key)</code></dt> <dd>The former is computed from <a href="https://doc.libsodium.org/public-key_cryptography/public-key_signatures">Libsodium's signature functions</a>. Unlike the usual Libsodium/NaCl functions, they work with signatures in independent buffers, rather than concatenated with the msg.</dd>
<dt><code>pk_to_curve25519(ed25519_pk)</code> and <code>sk_to_curve25519(ed25519_sk)</code></dt> <dd>These functions convert Ed25519 keys (used for cryptobox) to Curve25519 (aka X25519) keys, used for signing. They are <a href="https://doc.libsodium.org/advanced/ed25519-curve25519">implemented by Libsodium as <code>crypto_sign_ed25519_pk_to_curve25519</code> and <code>crypto_sign_ed25519_sk_to_curve25519</code></a>, respectively.</dd> </dl>
<h2 id="discovery">Discovery</h2> <div> <p>After a user has generated their identity they need to find some peers to connect to. To connect to a peer you need its public key and its address using any of the protocol it supports. Typically with TCP/IP, you would need its IP address and port, but the Scuttlebutt protocol is not restricted to TCP/IP as transport. The Scuttlebutt protocol currently has three methods for peers to discover each other.</p>
<h3 id="local-network">Local network</h3> <p>Peers constantly broadcast UDP packets on their local network advertising their presence. The body of each packet is a string containing the peer’s IP address, port and base64-encoded public key (without <code>@</code> or <code>.ed25519</code>):</p> </div> <aside class="impl"> <img class="icon" src="img/impl.png" alt=""/> <h5>Implementations</h5> <div class="lang">JS</div> <div class="vs"><a href="https://github.com/dominictarr/broadcast-stream/blob/master/index.js">broadcast-stream</a></div> <div class="vs"><a href="https://github.com/ssbc/ssb-local/blob/master/index.js">ssb-local</a></div> <div class="lang">Java</div> <div class="vs"><a href="https://github.com/apache/incubator-tuweni/blob/master/scuttlebutt-discovery/src/main/java/org/apache/tuweni/scuttlebutt/discovery/ScuttlebuttLocalDiscoveryService.java">LocalDiscoveryService</a></div> </aside>
<div> <figure class="packets"> <div class="packet"> <div class="header"> <span class="key">Source IP</span><span class="value">192.168.1.123</span> <span class="key">Source port</span><span class="value">8008</span> <span class="key">Destination IP</span><span class="value">255.255.255.255</span> <span class="key">Destination port</span><span class="value">8008</span> </div> <img src="img/format_udp_broadcast.png" style="height: 40px;" alt="net:192.168.1.123:8008:~shs:FCX/tsDLpubCPKKfIrw4gc+SQkHcaD17s7GI6i/ziWY="/> </div> </figure> <p>This message format can be any valid <a href="https://github.com/ssb-js/multiserver#address-format">multiserver address</a>. In local networks, this is usually a <code>net</code> address with an IPv4 or IPv6 address.</p> <p>Current implementations broadcast one of these packets every second. When a peer sees another peer’s broadcast packet they can connect to exchange messages. Some clients show nearby peers in the user interface so that the user can see other people on the same network as them.</p> </div> <aside> <p>UDP source and destination ports are set to the same port number that the peer is listening on TCP for peer connections (normally 8008).</p> </aside>
<h3 id="invite-code">Invite code</h3> <p><a href="#invites">Invite codes</a> help new users get connected to their first <a href="#pubs">pub</a>, which is a Scuttlebutt peer that is publicly accessible over the internet. An invite code contains a pub’s domain name, port and public key.</p> <p>They also contain a secret key that the user can <a href="#redeeming-invites">redeem</a> to make the pub <a href="#following">follow</a> them back. This lets the new user see messages posted by other members of the pub and share their own messages. Invite codes are the most common way for new users to get started on Scuttlebutt.</p> <p>Pub operators can distribute invite codes any way they see fit, for example by posting them on existing social networks. Some pubs have a web page that anybody can visit to generate an invite code.</p>
<h3 id="pub-message">Pub message</h3> <p>Users can post a message to their own <a href="#feeds">feed</a> advertising a pub:</p> <aside class="kicker"> <p>Here the user <code>@FCX/ts…</code> is advertising that they know of pub <code>@VJM7w1…</code> along with the pub’s domain name and port.</p> </aside> <pre><code>{ "author": "@FCX/tsDLpubCPKKfIrw4gc+SQkHcaD17s7GI6i/ziWY=.ed25519", "content": { "type": "pub", "address": { "host": "one.butt.nz", "port": 8008, "key": "@VJM7w1W19ZsKmG2KnfaoKIM66BRoreEkzaVm/J//wl8=.ed25519" } }, …}</code></pre> <p>When others see this message they can make a note that this pub exists and connect to it in the future.</p> <p>Pub messages are a useful way to find additional peers if you already know a few. Obviously this doesn’t work for new users who don’t know anyone else yet and therefore can’t see any pub messages.</p>
<h2 id="peer-connections">Peer connections</h2> <p>Once a Scuttlebutt client has discovered the IP address, port number and public key of a peer they can connect via TCP to ask for updates and exchange messages.</p>
<h3 id="handshake">Handshake</h3> <div> <p>The connection begins with a 4-step handshake to authenticate each peer and set up an encrypted channel.</p> <img src="img/message_flow.png" style="height: 350px;" alt="Message 1: Client hello (sent by the client). Message 2: Server hello. Message 3: Client authenticate. Message 4: Server accept" /> </div> <aside class="impl"> <img class="icon" src="img/impl.png" alt=""/> <h5>Implementations</h5> <div class="lang">JS</div> <div class="vs"><a href="https://github.com/auditdrivencrypto/secret-handshake/blob/master/protocol.js">protocol.js</a></div> <div class="vs"><a href="https://github.com/auditdrivencrypto/secret-handshake/blob/master/crypto.js">crypto.js</a></div> <div class="lang">Py</div> <div class="vs"><a href="https://github.com/pferreir/PySecretHandshake/blob/master/secret_handshake/crypto.py">crypto.py</a></div> <div class="lang">Go</div> <div class="vs"><a href="https://github.com/cryptoscope/secretstream/blob/ad7542b0cbda422a1ea3de7efa62a514672a2c88/secrethandshake/state.go">state.go</a></div> <div class="vs"><a href="https://github.com/cryptoscope/secretstream/blob/ad7542b0cbda422a1ea3de7efa62a514672a2c88/secrethandshake/conn.go">conn.go</a></div> <div class="lang">C</div> <div class="vs"><a href="https://github.com/AljoschaMeyer/shs1-c/blob/master/src/shs1.c">shs1.c</a></div> <div class="vs"><a href="https://git.scuttlebot.io/%25133ulDgs%2FoC1DXjoK04vDFy6DgVBB%2FZok15YJmuhD5Q%3D.sha256/blob/fd953a1e72b4b16e6e5a74bcf2f893dbf1407ce4/sbotc.c">sbotc.c</a></div> <div class="lang">Java</div> <div class="vs"><a href="https://github.com/apache/incubator-tuweni/blob/master/scuttlebutt-handshake/src/main/java/org/apache/tuweni/scuttlebutt/handshake/SecureScuttlebuttHandshakeClient.java">HandshakeClient</a></div> <div class="vs"><a href="https://github.com/apache/incubator-tuweni/blob/master/scuttlebutt-handshake/src/main/java/org/apache/tuweni/scuttlebutt/handshake/SecureScuttlebuttHandshakeServer.java">HandshakeServer</a></div> </aside> <p>The handshake uses the <a href="https://dominictarr.github.io/secret-handshake-paper/shs.pdf">Secret Handshake key exchange</a> which is designed to have these security properties:</p> <ul> <li>After a successful handshake the peers have verified each other’s public keys.</li> <li>The handshake produces a shared secret that can be used with a bulk encryption cypher for exchanging further messages.</li> <li>The client must know the server’s public key before connecting. The server learns the client’s public key during the handshake.</li> <li>Once the client has proven their identity the server can decide they don’t want to talk to this client and disconnect without confirming their own identity.</li> <li>A man-in-the-middle cannot learn the public key of either peer.</li> <li>Both peers need to know a key that represents the particular Scuttlebutt network they wish to connect to, however a man-in-the-middle can’t learn this key from the handshake. If the handshake succeeds then both ends have confirmed that they wish to use the same network.</li> <li>Past handshakes cannot be replayed. Attempting to replay a handshake will not allow an attacker to discover or confirm guesses about the participants’ public keys.</li> <li>Handshakes provide forward secrecy. Recording a user’s network traffic and then later stealing their secret key will not allow an attacker to decrypt their past handshakes.</li> </ul> <aside> <p><strong>Client</strong> is the computer initiating the TCP connection and <strong>server</strong> is the computer receiving it. Once the handshake is complete this distinction goes away.</p> </aside>
<h4 id="starting-keys">Starting keys</h4> <p>Upon starting the handshake, the client and server know these keys:</p> <img src="img/starting_keys.png" style="height: 590px;" alt="Both the client and servers know: their own long term key pair, their own ephemeral key pair, and the network's (private) identifier. Additionally, the client knows the server's long term public key."/>
<h4 id="client-hello">1. Client hello</h4> <img src="img/client_hello.png" style="height: 120px;" alt="The client sends their own ephemeral public key, hmac-authenticated using the network identifier"/>
<figure class="left-right code"> <h5 class="left">Client sends <span class="msgsize">(64 bytes)</span></h5> <h5 class="right">Server verifies</h5> <pre class="left"><code>concat( nacl_auth( msg: client_ephemeral_pk, key: network_identifier ), client_ephemeral_pk)</code></pre> <pre class="right"><code>assert(length(msg1) == 64)
client_hmac = first_32_bytes(msg1)client_ephemeral_pk = last_32_bytes(msg1)
assert_nacl_auth_verify( authenticator: client_hmac, msg: client_ephemeral_pk, key: network_identifier)</code></pre> </figure>
<div> <p>First the client sends their <img class="inline-key" src="img/key_little_a_public.png" alt="public"/> generated ephemeral key. Also included is an hmac that indicates the client wishes to use their key with this specific instance of the Scuttlebutt network.</p> <p>The <img class="inline-key" src="img/key_big_n.png" alt=""/> network identifier is a fixed key. On the main Scuttlebutt network it is the following 32-byte sequence:</p> <div class="fixed-key binary"> <span>d4</span><span>a1</span><span>cb</span><span>88</span><span>a6</span><span>6f</span><span>02</span><span>f8</span><span>db</span><span>63</span><span>5c</span><span>e2</span><span>64</span><span>41</span><span>cc</span><span>5d</span> <span>ac</span><span>1b</span><span>08</span><span>42</span><span>0c</span><span>ea</span><span>ac</span><span>23</span><span>08</span><span>39</span><span>b7</span><span>55</span><span>84</span><span>5a</span><span>9f</span><span>fb</span> </div> <p>Changing the key allows separate networks to be created, for example private networks or testnets. An eavesdropper cannot extract the network identifier directly from what is sent over the wire, although they could confirm a guess that it is the main Scuttlebutt network because that identifier is publicly known.</p> <p>The server stores the client’s ephemeral public key and uses the hmac to verify that the client is using the same network identifier.</p> </div> <aside> <p><strong>hmac</strong> is a function that allows verifying that a message came from someone who knows the same secret key as you. In this case the network identifier is used as the secret key.</p> <p>Both the message creator and verifier have to know the same message and secret key for the verification to succeed, but the secret key is not revealed to an eavesdropper. </p> <p>Throughout the protocol, all instances of hmac use HMAC-SHA-512-256 (which is the first 256 bits of HMAC-SHA-512).</p> </aside>
<h4 id="server-hello">2. Server hello</h4> <img src="img/server_hello.png" style="height: 120px;" alt="The server sends their own ephemeral public key, hmac-authenticated using the network identifier"/>
<figure class="left-right code"> <h5 class="left">Client verifies</h5> <h5 class="right">Server sends <span class="msgsize">(64 bytes)</span></h5> <pre class="left"><code>assert(length(msg2) == 64)
server_hmac = first_32_bytes(msg2)server_ephemeral_pk = last_32_bytes(msg2)
assert_nacl_auth_verify( authenticator: server_hmac, msg: server_ephemeral_pk, key: network_identifier)</code></pre> <pre class="right"><code>concat( nacl_auth( msg: server_ephemeral_pk, key: network_identifier ), server_ephemeral_pk)</code></pre> </figure>
<p>The server responds with their own <img class="inline-key" src="img/key_little_b_public.png" alt="public"/> ephemeral public key and hmac. The client stores the key and verifies that they are also using the same network identifier.</p>
<h4 id="shared-secret-derivation-1">Shared secret derivation</h4> <img src="img/shared_secret_derivation_1.png" style="height: 220px;" alt="Each derivation uses one public key (their peer's) and one secret key (their own). The resultting shared secrets are identical between server and client."/>
<figure class="left-right code"> <h5 class="left">Client computes</h5> <h5 class="right">Server computes</h5> <pre class="left"><code>shared_secret_ab = nacl_scalarmult( client_ephemeral_sk, server_ephemeral_pk)
shared_secret_aB = nacl_scalarmult( client_ephemeral_sk, pk_to_curve25519(server_longterm_pk))</code></pre> <pre class="right"><code>shared_secret_ab = nacl_scalarmult( server_ephemeral_sk, client_ephemeral_pk)
shared_secret_aB = nacl_scalarmult( sk_to_curve25519(server_longterm_sk), client_ephemeral_pk)</code></pre> </figure>
<div> <p>Now that ephemeral keys have been exchanged, both ends use them to derive a shared secret <img class="inline-key" src="img/key_little_a_little_b.png" alt=""/> using scalar multiplication.</p> <p>The client and server each combine their own ephemeral secret key with the other’s ephemeral public key to produce the same shared secret on both ends. An eavesdropper doesn’t know either secret key so they can’t generate the shared secret. A man-in-the-middle could swap out the ephemeral keys in Messages 1 and 2 for their own keys, so the shared secret <img class="inline-key" src="img/key_little_a_little_b.png" alt=""/> alone is not enough for the client and server to know that they are talking to each other and not a man-in-the-middle.</p> <p>Because the client already knows the <img class="inline-key" src="img/key_big_b_public.png" alt=""/> server’s long term public key, both ends derive a second secret <img class="inline-key" src="img/key_little_a_big_b.png" alt="using the client's ephemeral key pair (either the public or the secret key) and the server's permanent key pair (respectively either the secret or private key)"/> that will allow the client to send a message that only the real server can read and not a man-in-the-middle.</p> </div> <aside> <p><strong>Scalar multiplication</strong> is a function for deriving shared secrets from a pair of secret and public Curve25519 keys.</p> <p>The order of arguments matters. In the NaCl API the secret key is provided first.</p> <p>Note that long term keys are Ed25519 and must first be converted to Curve25519.</p> </aside>
<h4 id="client-accept">3. Client authenticate</h4> <img src="img/client_authenticate.png" style="height: 330px;" alt="The client computes a detached signature of the network identifier, the server's long-term public key, and a hash of the shared secret; signed with its permanent secret key. They add their permanent public key, and encrypt both so that they can only be opened by someone knowing the network identifier and both shared secrets; then send the cyphertext to the server." />
<figure class="left-right code"> <h5 class="left">Client computes</h5> <h5 class="right">Server verifies</h5> <pre class="left"><code>detached_signature_A = nacl_sign_detached( msg: concat( network_identifier, server_longterm_pk, sha256(shared_secret_ab) ), key: client_longterm_sk)</code></pre> <pre class="right" style="grid-row: span 3;"><code>msg3_plaintext = assert_nacl_secretbox_open( ciphertext: msg3, nonce: 24_bytes_of_zeros, key: sha256( concat( network_identifier, shared_secret_ab, shared_secret_aB ) ))
assert(length(msg3_plaintext) == 96)
detached_signature_A = first_64_bytes(msg3_plaintext)client_longterm_pk = last_32_bytes(msg3_plaintext)
assert_nacl_sign_verify_detached( sig: detached_signature_A, msg: concat( network_identifier, server_longterm_pk, sha256(shared_secret_ab) ), key: client_longterm_pk)</code></pre> <h5 class="left">Client sends <span class="msgsize">(112 bytes)</span></h5> <pre class="left"><code>nacl_secret_box( msg: concat( detached_signature_A, client_longterm_pk ), nonce: 24_bytes_of_zeros, key: sha256( concat( network_identifier, shared_secret_ab, shared_secret_aB ) ))</code></pre> </figure>
<div> <p>The client reveals their identity to the server by sending their <img class="inline-key" src="img/key_big_a_public.png" alt=""/> long term public key. The client also makes a signature using their <img class="inline-key" src="img/key_big_a_secret.png" alt=""/> long term secret key. By signing the keys used earlier in the handshake the client proves their identity and confirms that they do indeed wish to be part of this handshake.</p> <p>The client’s message is enclosed in a secret box to ensure that only the server can read it. Upon receiving it, the server opens the box, stores the client’s long term public key and verifies the signature.</p> <p>An all-zero nonce is used for the secret box. The secret box construction requires that all secret boxes using a particular key must use different nonces. It’s important to get this detail right because reusing a nonce will allow an attacker to recover the key and encrypt or decrypt any secret boxes using that key. Using a zero nonce is allowed here because this is the only secret box that ever uses the key <span class="key-formula">sha256(concat( <img class="inline-key" src="img/key_big_n.png" alt=""/>, <img class="inline-key" src="img/key_little_a_little_b.png" alt=""/>, <img class="inline-key" src="img/key_little_a_big_b.png" alt=""/>))</span>.</p> </div> <aside> <p><strong>Detached signatures</strong> do not contain a copy of the message that was signed, only a tag that allows verifying the signature if you already know the message.</p> <p>Here it is okay because the server knows all the information needed to reconstruct the message that the client signed.</p> </aside>
<h4 id="shared-secret-derivation-2">Shared secret derivation</h4> <img src="img/shared_secret_derivation_2.png" style="height: 220px;" alt="The client computes a new shared secret from their permanent secret key and the server's ephemeral public key. The server computes the same shared secret from the client's permanent public key and their own ephemeral secret key."/>
<figure class="left-right code"> <h5 class="left">Client computes</h5> <h5 class="right">Server computes</h5> <pre class="left"><code>shared_secret_Ab = nacl_scalarmult( sk_to_curve25519(client_longterm_sk), server_ephemeral_pk)</code></pre> <pre class="right"><code>shared_secret_Ab = nacl_scalarmult( server_ephemeral_sk, pk_to_curve25519(client_longterm_pk))</code></pre> </figure>
<p>Now that the server knows the <img class="inline-key" src="img/key_big_a_public.png" alt=""/> client’s long term public key, another shared secret <img class="inline-key" src="img/key_big_a_little_b.png" alt=""/> is derived by both ends. The server uses this shared secret to send a message that only the real client can read and not a man-in-the-middle.</p>
<h4 id="server-accept">4. Server accept</h4> <img src="img/server_accept.png" style="height: 400px;" alt="The server signs the network identifier, the previous detached signature, the client's permanent secret key, and the hash of the first shared secret, with their permanent secret key, as a new detached signature. They encrypt it so that they can only be opened by someone knowing the network identifier and all three shared secrets; then send the cyphertext to the client." />
<figure class="left-right code"> <h5 class="left">Client verifies</h5> <h5 class="right">Server computes</h5> <pre class="left" style="grid-row: span 3;"><code>detached_signature_B = assert_nacl_secretbox_open( ciphertext: msg4, nonce: 24_bytes_of_zeros, key: sha256( concat( network_identifier, shared_secret_ab, shared_secret_aB, shared_secret_Ab ) ))
assert_nacl_sign_verify_detached( sig: detached_signature_B, msg: concat( network_identifier, detached_signature_A, client_longterm_pk, sha256(shared_secret_ab) ), key: server_longterm_pk)</code></pre> <pre class="right"><code>detached_signature_B = nacl_sign_detached( msg: concat( network_identifier, detached_signature_A, client_longterm_pk, sha256(shared_secret_ab) ), key: server_longterm_sk)</code></pre> <h5 class="right">Server sends <span class="msgsize">(80 bytes)</span></h5> <pre class="right"><code>nacl_secret_box( msg: detached_signature_B, nonce: 24_bytes_of_zeros, key: sha256( concat( network_identifier, shared_secret_ab, shared_secret_aB, shared_secret_Ab ) ))</code></pre> </figure> <p>The server accepts the handshake by signing a message using their <img class="inline-key" src="img/key_big_b_secret.png" alt=""/> long term secret key. It includes a copy of the client’s previous signature. The server’s signature is enclosed in a secret box using all of the shared secrets.</p> <p>Upon receiving it, the client opens the box and verifies the server’s signature.</p> <p>Similarly to the previous message, this secret box also uses an all-zero nonce because it is the only secret box that ever uses the key <span class="key-formula">sha256(concat( <img class="inline-key" src="img/key_big_n.png"/>, <img class="inline-key" src="img/key_little_a_little_b.png" alt=""/>, <img class="inline-key" src="img/key_little_a_big_b.png" alt=""/>, <img class="inline-key" src="img/key_big_a_little_b.png" alt=""/>))</span>.</p>
<h4 id="handshake-complete">Handshake complete</h4> <img src="img/final_shared_secret.png" style="height: 40px;" alt="" /> <p>At this point the handshake has succeeded. The client and server have proven their identities to each other.</p> <p>The shared secrets established during the handshake are used to set up a pair of box streams for securely exchanging further messages.</p>
<h3 id="box-stream">Box stream</h3> <div> <p>Box stream is the bulk encryption protocol used to exchange messages following the handshake until the connection ends. It is designed to protect messages from being read or modified by a man-in-the-middle.</p> <p>Each message in a box stream has a header and body. The header is always 34 bytes long and says how long the body will be.</p> <img src="img/box_stream_overview.png" style="height: 140px;" alt="A stream is made of alternating headers (34 bytes) and bodies (1 to 4096 bytes); ending with a body followed by a 34-bytes 'goodbye' header"/>
<h4 id="sending">Sending</h4> <p>Sending a message involves encrypting the body of the message and preparing a header for it. Two secret boxes are used; one to protect the header and another to protect the body.</p> </div> <aside class="impl"> <img class="icon" src="img/impl.png" alt=""/> <h5>Implementations</h5> <div class="lang">JS</div> <div class="vs"><a href="https://github.com/dominictarr/pull-box-stream/blob/master/index.js">pull-box-stream</a></div> <div class="lang">Py</div> <div class="vs"><a href="https://github.com/pferreir/PySecretHandshake/blob/master/secret_handshake/boxstream.py">boxstream.py</a></div> <div class="lang">Go</div> <div class="vs"><a href="https://github.com/cryptoscope/secretstream/blob/ad7542b0cbda422a1ea3de7efa62a514672a2c88/boxstream/box.go">box.go</a></div> <div class="vs"><a href="https://github.com/cryptoscope/secretstream/blob/ad7542b0cbda422a1ea3de7efa62a514672a2c88/boxstream/unbox.go">unbox.go</a></div> <div class="lang">C</div> <div class="vs"><a href="https://github.com/AljoschaMeyer/box-stream-c/blob/master/src/box-stream.c">box-stream.c</a></div> <div class="vs"><a href="https://git.scuttlebot.io/%25133ulDgs%2FoC1DXjoK04vDFy6DgVBB%2FZok15YJmuhD5Q%3D.sha256/blob/fd953a1e72b4b16e6e5a74bcf2f893dbf1407ce4/sbotc.c">sbotc.c</a></div> <div class="lang">Java</div> <div class="vs"><a href="https://github.com/apache/incubator-tuweni/blob/master/scuttlebutt-handshake/src/main/java/org/apache/tuweni/scuttlebutt/handshake/SecureScuttlebuttStream.java">Stream</a></div> </aside>
<img src="img/box_stream_send.png" class="pagewidth" style="height: 670px;" alt="The plaintext message body is enclosed in a secret box using the key and nonce shown below. Secret boxes put a 16-byte tag onto the front of messages so that tampering can be detected when the box is opened. This tag is sliced off the body and put inside the header. A temporary header is made of the body length (a two-bytes big-endian integer) and th previous tag. This temporary header is then encrypted too, including its own (16-bytes) authentication tag, producing a 16+2+16 bytes header."/>
<h4 id="receiving">Receiving</h4> <p>Receiving a message involves reading the header to find out how long the body is then reassembling and opening the body secret box.</p> <img src="img/box_stream_receive.png" class="pagewidth" style="height: 560px;" alt="Read the first 34 bytes. This is the secret box containing the header. Open this box, extract the body length and body authentication tag. Read the number of bytes specified in the header. Join the body authentication tag and encrypted body back together, open it, and read the secret text."/>
<h4 id="goodbye">Goodbye</h4> <p>The stream ends with a special “goodbye” header. Because the goodbye header is authenticated it allows a receiver to tell the difference between the connection genuinely being finished and a man-in-the-middle forcibly resetting the underlying TCP connection.</p> <img src="img/box_stream_goodbye.png" class="pagewidth" style="height: 230px;" alt="The 'goodbye' header is made of 18 bytes of zero, encrypted in a secret box (with a header authenticated tag like other headers)."/> <p>When a receiver opens a header and finds that it contains all zeros then they will know that the connection is finished.</p>
<h4 id="keys-and-nonces">Keys and nonces</h4> <p>Two box streams are used at the same time when Scuttlebutt peers communicate. One is for client-to-server messages and the other is for server-to-client messages. The two streams use different keys and starting nonces for their secret boxes.</p> <img src="img/box_stream_params.png" class="pagewidth" style="height: 400px;" alt="The secret box key is made of a double-sha256 hash of the network identifier and three shared secrets, followed by either the server's permanent public key (for Client to Server) or the client's permanent public key (for Server to Client), both hashed again with sha256. The starting nonces are respectively the first 24 bytes of server's or the client's ephemeral public key, hmac-authenticated with the network identifier."/> <p>The starting nonce is used for the first header in the stream (“secret box 1” in the above figures), then incremented for the first body (“secret box 2”), then incremented for the next header and so on.</p>
<h3 id="rpc-protocol">RPC protocol</h3> <aside class="impl kicker"> <img class="icon" src="img/impl.png" alt=""/> <h5>Implementations</h5> <div class="lang">JS</div> <div><a href="https://github.com/ssbc/packet-stream-codec/blob/master/index.js">packet-stream-codec</a></div> <div class="lang">Py</div> <div class="vs"><a href="https://github.com/pferreir/pyssb/blob/master/ssb/packet_stream.py">packet_stream.py</a></div> <div class="vs"><a href="https://github.com/pferreir/pyssb/blob/master/ssb/muxrpc.py">muxrpc.py</a></div> <div class="lang">Go</div> <div class="vs"><a href="https://github.com/cryptoscope/go-muxrpc/tree/601b7be81ee6b2bd6f32b1247e4688537f696794/codec">codec</a></div> <div class="vs"><a href="https://github.com/cryptoscope/go-muxrpc/blob/601b7be81ee6b2bd6f32b1247e4688537f696794/rpc.go">rpc.go</a></div> <div class="lang">C</div> <div class="vs"><a href="https://git.scuttlebot.io/%25133ulDgs%2FoC1DXjoK04vDFy6DgVBB%2FZok15YJmuhD5Q%3D.sha256/blob/fd953a1e72b4b16e6e5a74bcf2f893dbf1407ce4/sbotc.c">sbotc.c</a></div> <div class="lang">Java</div> <div class="vs"><a href="https://github.com/apache/incubator-tuweni/blob/master/scuttlebutt-rpc/src/main/java/org/apache/tuweni/scuttlebutt/rpc/RPCCodec.java">RPCCodec</a></div> </aside> <div> <p>Scuttlebutt peers make requests to each other using an RPC protocol. Typical requests include asking for the latest messages in a particular feed or requesting a blob.</p> <p>The RPC protocol can interleave multiple requests so that a slow request doesn’t block following ones. It also handles long-running asynchronous requests for notifying when an event occurs and streams that deliver multiple responses over time.</p> <p>Similar to the box stream protocol, the RPC protocol consists of 9-bytes headers followed by variable-length bodies. There is also a 9-bytes goodbye message which is just a zeroed out header.</p> <img src="img/rpc_overview.png" style="height: 155px;" alt=""/> </div> <aside> <p><strong>Remote procedure calls</strong> are where a computer exposes a set of procedures that another computer can call over the network.</p> <p>The requester tells the responder the name of the procedure they wish to call along with any arguments. The responder performs the action and returns a value back to the requester.</p> </aside> <p>Both peers make requests to each other at the same time using the pair of box streams that have been established. The box streams protect the RPC protocol from eavesdropping and tampering.</p> <img src="img/rpc_alignment.png" style="height: 230px;" alt=""/> <aside style="position: relative; top: 36px;"> <p>RPC messages are not necessarily aligned to box stream boxes.</p> <p>Multiple RPC messages may be put inside one box or a single RPC message may be split over several boxes.</p> </aside>
<h4 id="header-structure">Header structure</h4> <p>RPC headers contain a set of flags to say what type of message it is, a field specifying its length and a request number which allows matching requests with their responses when there are several active at the same time.</p> <img src="img/rpc_header.png" style="height: 750px;" alt="Headers are made of (in network order): 4 zero bits, a stream bit (1 = 'message is part of a stream'), a end/error bit (1 = 'message is the last in its stream or an error), and a 2-bits body type (00 = binary, 01 = UTF-8 string, 10 = JSON), the body length (4 bytes unsigned big-endian), and the request number (4 bytes signed big-endial)."/>
<h4 id="request-format">Request format</h4> <p>To make an RPC request, send a JSON message containing the name of the procedure you wish to call, the type of procedure and any arguments.</p> <p>The name is a list of strings. For a top-level procedure like <em>createHistoryStream</em> the list only has one element: <code>["createHistoryStream"]</code>. Procedures relating to blobs are grouped in the blobs namespace, for example to use <em>blobs.get</em> send the list: <code>["blobs", "get"]</code>.</p> <div> <p>There are three types of procedure used when Scuttlebutt peers talk to each other:</p> <ul> <li><em>Source</em> procedures return multiple responses over time and are used for streaming data or continually notifying when new events occur. When making one of these requests, the stream flag in the RPC header must be set.</li> <li><em>Duplex</em> procedures are similar to <em>source</em> procedures but allow <em>multiple requests</em> as well as multiple responses over time. The many request events in a duplex utilize the same request number, and the stream flag must be set.</li> <li><em>Async</em> procedures return a single response. Async responses can arrive quickly or arrive much later in response to a one-off event.</li> </ul> <p>For each procedure in the RPC protocol you must already know whether it is source or async and correctly specify this in the request body.</p> </div> <aside> <p>The reference Scuttlebot implementation also has other internal procedures and procedure types which are used by graphical user interfaces like Patchwork.</p> <p>This guide only covers the procedures that are publicly available to other Scuttlebutt peers.</p> </aside>
<h4 id="source-example">Source example</h4> <p>This RPC message shows an example of a <em>createHistoryStream</em> request:</p>
<aside class="kicker" style="align-self: start; position: relative; top: 58px;"> <p>JSON messages don’t have indentation or whitespace when sent over the wire.</p> </aside> <figure class="request-response"> <div class="request"> <div class="header"> <span class="key">Request number</span><span class="value">1</span> <span class="key">Body type</span><span class="value">JSON</span> <span class="key">Stream</span><span class="value">Yes</span> <span class="key">End/err</span><span class="value">No</span> </div> <pre><code>{ "name": ["createHistoryStream"], "type": "source", "args": [{"id": "@FCX/tsDLpubCPKKfIrw4gc+SQkHcaD17s7GI6i/ziWY=.ed25519"}]}</code></pre> </div> <img class="request-arrow" src="img/arrow.png" alt=""/> </figure>
<div> <p><em>createHistoryStream</em> is how Scuttlebutt peers ask each other for a list of messages posted by a particular feed. It has one argument that is a JSON dictionary specifying more options about the request. <em>id</em> is the only required option and says which feed you are interested in.</p> <p>Because this is the first RPC request, the request number is 1. The next request made by this peer will be numbered 2. The other peer will also use request number 1 for their first request, but the peers can tell these apart because they know whether they sent or received each request.</p> </div> <p>Now the responder begins streaming back responses:</p>
<figure class="request-response"> <img class="response-arrow" src="img/arrow.png" alt=""/> <div class="response"> <div class="header"> <span class="key">Request number</span><span class="value">-1</span> <span class="key">Body type</span><span class="value">JSON</span> <span class="key">Stream</span><span class="value">Yes</span> <span class="key">End/err</span><span class="value">No</span> </div> <pre><code>{ "key": "%XphMUkWQtomKjXQvFGfsGYpt69sgEY7Y4Vou9cEuJho=.sha256", "value": { "previous": null, "author": "@FCX/tsDLpubCPKKfIrw4gc+SQkHcaD17s7GI6i/ziWY=.ed25519", "sequence": 1, "timestamp": 1514517067954, "hash": "sha256", "content": { "type": "post", "text": "This is the first post!" }, "signature": "QYOR/zU9dxE1aKBaxc3C0DJ4gRyZtlMfPLt+CGJcY73sv5abKK Kxr1SqhOvnm8TY784VHE8kZHCD8RdzFl1tBA==.sig.ed25519" }, "timestamp": 1514517067956}</code></pre> </div> <img class="response-arrow" src="img/arrow.png" alt=""/> <div class="response"> <div class="header"> <span class="key">Request number</span><span class="value">-1</span> <span class="key">Body type</span><span class="value">JSON</span> <span class="key">Stream</span><span class="value">Yes</span> <span class="key">End/err</span><span class="value">No</span> </div> <pre><code>{ "key": "%R7lJEkz27lNijPhYNDzYoPjM0Fp+bFWzwX0SmNJB/ZE=.sha256", "value": { "previous": "%XphMUkWQtomKjXQvFGfsGYpt69sgEY7Y4Vou9cEuJho=.sha256", "author": "@FCX/tsDLpubCPKKfIrw4gc+SQkHcaD17s7GI6i/ziWY=.ed25519", "sequence": 2, "timestamp": 1514517078157, "hash": "sha256", "content": { "type": "post", "text": "Second post!" }, "signature": "z7W1ERg9UYZjNfE72ZwEuJF79khG+eOHWFp6iF+KLuSrw8Lqa6 IousK4cCn9T5qFa8E14GVek4cAMmMbjqDnAg==.sig.ed25519" }, "timestamp": 1514517078160}</code></pre> </div> </figure> <aside style="align-self: start; position: relative; top: 29px;"> <p>Because the responses are part of a stream, their RPC headers have the stream flag set.</p> <p>All responses use the same request number as the original request but negative.</p> </aside>
<p>Each message posted by the feed is sent back in its own response. This feed only contains two messages.</p> <p>To close the stream the responder sends an RPC message with both the stream and end/err flags set and a JSON body of <code>true</code>. When the requester sees that the stream is being closed they send a final message to close their own end of it (source type requests must always be closed by both ends).</p>
<figure class="request-response"> <img class="response-arrow" src="img/arrow.png" alt=""/> <div class="response"> <div class="header"> <span class="key">Request number</span><span class="value">-1</span> <span class="key">Body type</span><span class="value">JSON</span> <span class="key">Stream</span><span class="value">Yes</span> <span class="key">End/err</span><span class="value">Yes</span> </div> <pre><code>true</code></pre> </div> <div class="request"> <div class="header"> <span class="key">Request number</span><span class="value">1</span> <span class="key">Body type</span><span class="value">JSON</span> <span class="key">Stream</span><span class="value">Yes</span> <span class="key">End/err</span><span class="value">Yes</span> </div> <pre><code>true</code></pre> </div> <img class="request-arrow" src="img/arrow.png" alt=""/> </figure>
<p>Alternatively, to abort a stream before it is finished the requester can send their closing message early, at which point the responder closes their own end.</p>
<figure class="request-response"> <div class="request"> <div class="header"> <span class="key">Request number</span><span class="value">1</span> <span class="key">Body type</span><span class="value">JSON</span> <span class="key">Stream</span><span class="value">Yes</span> <span class="key">End/err</span><span class="value">Yes</span> </div> <pre><code>true</code></pre> </div> <img class="request-arrow" src="img/arrow.png" alt=""/> <img class="response-arrow" src="img/arrow.png" alt=""/> <div class="response"> <div class="header"> <span class="key">Request number</span><span class="value">-1</span> <span class="key">Body type</span><span class="value">JSON</span> <span class="key">Stream</span><span class="value">Yes</span> <span class="key">End/err</span><span class="value">Yes</span> </div> <pre><code>true</code></pre> </div> </figure>
<h4 id="async-example">Async example</h4> <p>One of the few public async procedures is <em>blobs.has</em>, which peers use to ask each other whether they have a particular blob.</p> <p>In this example the requester is asking the responder if they have blob <code>&WWw4tQJ6…</code>:</p>
<figure class="request-response"> <div class="request"> <div class="header"> <span class="key">Request number</span><span class="value">2</span> <span class="key">Body type</span><span class="value">JSON</span> <span class="key">Stream</span><span class="value">No</span> <span class="key">End/err</span><span class="value">No</span> </div> <pre><code>{ "name": ["blobs", "has"], "type": "async", "args": ["&WWw4tQJ6ZrM7o3gA8lOEAcO4zmyqXqb/3bmIKTLQepo=.sha256"]}</code></pre> </div> <img class="request-arrow" src="img/arrow.png" alt=""/> </figure>
<p>The responder does in fact have this blob so they respond with <code>true</code>. Because this is an async procedure and not a stream, there is only one response and no need to close the stream afterwards:</p>
<figure class="request-response"> <img class="response-arrow" src="img/arrow.png" alt=""/> <div class="response"> <div class="header"> <span class="key">Request number</span><span class="value">-2</span> <span class="key">Body type</span><span class="value">JSON</span> <span class="key">Stream</span><span class="value">No</span> <span class="key">End/err</span><span class="value">No</span> </div> <pre><code>true</code></pre> </div> </figure>
<h4 id="error-example">Error example</h4>
<p>Let’s take the previous example and introduce a programming mistake to see how the RPC protocol handles errors:</p>
<figure class="request-response"> <div class="request"> <div class="header"> <span class="key">Request number</span><span class="value">3</span> <span class="key">Body type</span><span class="value">JSON</span> <span class="key">Stream</span><span class="value">No</span> <span class="key">End/err</span><span class="value">No</span> </div> <pre><code>{ "name": ["blobs", "has"], "type": "async", "args": ["<strong>this was a mistake</strong>"]}</code></pre> </div> <img class="request-arrow" src="img/arrow.png" alt=""/> <img class="response-arrow" src="img/arrow.png" alt=""/> <div class="response"> <div class="header"> <span class="key">Request number</span><span class="value">-3</span> <span class="key">Body type</span><span class="value">JSON</span> <span class="key">Stream</span><span class="value">No</span> <span class="key">End/err</span><span class="value">Yes</span> </div> <pre><code>{ "name": "Error", "message": "invalid hash:this was a mistake", "stack": "…"}</code></pre> </div> </figure>
<p>Most importantly, the response has the end/err flag set to indicate that an error occurred. The reference Scuttlebot implementation also includes an error message and a JavaScript stack trace.</p> <p>For source type procedures an error will also end the stream because the end/err flag has the dual purpose of ending streams and indicating that an error occurred.</p>
<h2 id="feeds">Feeds</h2> <p>A Scuttlebutt feed is a list of all the messages posted by a particular identity. When a user writes a message in a Scuttlebutt client and posts it, that message is put onto the end of their feed.</p>
<h3 id="structure">Structure</h3> <p>The messages in a feed form an append-only log, meaning that once a message is posted it cannot be modified. Each message (except the first one) references the ID of the previous message, allowing a chain to be constructed back to the first message in the feed.</p> <img src="img/sigchain.png" class="pagewidth" style="height: 855px;" alt=""/>
<h3 id="message-format">Message format</h3> <p>To create a message to post in a feed, start by filling out these fields:</p> <pre><code>{ "previous": "%XphMUkWQtomKjXQvFGfsGYpt69sgEY7Y4Vou9cEuJho=.sha256", "author": "@FCX/tsDLpubCPKKfIrw4gc+SQkHcaD17s7GI6i/ziWY=.ed25519", "sequence": 2, "timestamp": 1514517078157, "hash": "sha256", "content": { "type": "post", "text": "Second post!" }}</code></pre> <table class="defs"> <tr> <td>previous</td> <td>Message ID of the latest message posted in the feed. If this is the very first message then use <code>null</code>. See below for how to compute a message’s ID.</td> </tr> <tr> <td>author</td> <td>Public key of the feed that the message will be posted in.</td> </tr> <tr> <td>sequence</td> <td>1 for the first message in a feed, 2 for the second and so on.</td> </tr> <tr> <td>timestamp</td> <td>Time the message was created. Number of milliseconds since 1 January 1970 00:00 UTC.</td> </tr> <tr> <td>hash</td> <td>The fixed string <code>sha256</code>, which is the hash function used to compute the message ID.</td> </tr> <tr> <td>content</td> <td>If the message is not encrypted, This is a dictionary containing free-form data for applications to interpret, plus a mandatory <em>type</em> field. The <em>type</em> field allows applications to filter out message types they don’t understand and must be a Unicode string between 3 and 52 code units long (inclusive). If the message is encrypted, then this is a base64 encoded string, followed by a suffix of <code>.box</code>; we will describe private messages later in this document.</td> </tr> </table> <aside style="align-self: start; position: relative; top: 19px;"> <p>Only these fields are allowed and they all must be present.</p> <p>Fields must appear in this order (although <em>author</em> and <em>sequence</em> can be swapped for legacy reasons, but please don’t do this for new messages).</p> <p>Fields within <em>content</em> can appear in any order but the order must be remembered for later.</p> </aside>
<h4 id="signature">Signature</h4> <div> <p>All messages in a feed are signed by that feed’s long-term secret key. This enables recipients to verify that a message was really posted by a particular identity and not tampered with as it gets gossiped and replicated throughout the Scuttlebutt network.</p> <p>Before signing a message it must be serialized according to a specific canonical JSON format. This means for any given message there is exactly one way to serialize it as a sequence of bytes, which is necessary for signature verification to work. The reference implementation verifies that all messages it receives are in the canonical format and rejects messages that aren’t.</p> </div> <aside class="impl"> <img class="icon" src="img/impl.png"/> <h5>Implementations</h5> <div class="lang">JS</div> <div class="vs"><a href="https://github.com/ssbc/ssb-keys/blob/master/index.js">ssb-keys</a></div> <div class="vs"><a href="https://github.com/ssbc/ssb-feed/blob/master/validator.js">ssb-feed/validator.js</a></div> <div class="lang">Py</div> <div class="vs"><a href="https://github.com/pferreir/pyssb/blob/master/ssb/feed/models.py">models.py</a></div> </aside> <p>The canonical format is defined by the <em>ECMA-262 6<sup>th</sup> Edition</em> section <em><a href="https://www.ecma-international.org/ecma-262/6.0/#sec-json.stringify">JSON.stringify</a></em>. For an example, see how the above message is formatted.</p> <p>In brief, the rules are:</p> <ul> <li>Two spaces for indentation.</li> <li>Dictionary entries and list elements each on their own line.</li> <li>Newlines use the line feed character <code>\n</code>.</li> <li>No trailing newline.</li> <li>One space after the colon <code>:</code> for dictionary keys.</li> <li>Empty dictionaries appear as <code>{}</code> and empty lists appear as <code>[]</code>.</li> <li>Strings and numbers formatted according to the sections <em><a href="https://www.ecma-international.org/ecma-262/6.0/#sec-quotejsonstring">QuoteJSONString</a></em> and <em><a href="https://www.ecma-international.org/ecma-262/6.0/#sec-tostring-applied-to-the-number-type">ToString Applied to the Number Type</a></em>. </ul> <p>Then sign the message by computing:</p> <pre><code>signature = nacl_sign_detached( msg: formatted_json_message, key: authors_longterm_sk)</code></pre> <p>Base64 encode the signature and put <code>.sig.ed25519</code> on the end. Finally, add the signature to the message itself. It must be the last entry in the dictionary:</p> <pre><code>{ "previous": "%XphMUkWQtomKjXQvFGfsGYpt69sgEY7Y4Vou9cEuJho=.sha256", "author": "@FCX/tsDLpubCPKKfIrw4gc+SQkHcaD17s7GI6i/ziWY=.ed25519", "sequence": 2, "timestamp": 1514517078157, "hash": "sha256", "content": { "type": "post", "text": "Second post!" }, "<strong>signature</strong>": "z7W1ERg9UYZjNfE72ZwEuJF79khG+eOHWFp6iF+KLuSrw8Lqa6 IousK4cCn9T5qFa8E14GVek4cAMmMbjqDnAg==.sig.ed25519"}</code></pre>
<p>To verify the signature, first remove the signature field from the message. Make sure the remaining message is formatted as JSON using the same formatting rules. Here the order of dictionary entries within <em>content</em> matters; they must be in the same order that you received them.</p> <p>Then remove the <code>.sig.ed25519</code> suffix from the signature, base64 decode it and verify with:</p> <pre><code>nacl_sign_verify_detached( sig: signature, msg: formatted_json_message, key: authors_longterm_pk)</code></pre>
<h4 id="message-id">Message ID</h4> <p>A message ID is a hash of the message including signature. Messages can refer to other messages by their ID.</p> <p>To compute a message’s ID, first format it as JSON using the formatting rules. Like with signatures, dictionary keys must appear in the same order that you received them. The <em>signature</em> key must appear last as shown above (but without wrapping the line).</p> <p>The ID of the above message is:</p> <img src="img/format_message_id.png" style="height: 80px;" alt="%R7lJEkz27lNijPhYNDzYoPjM0Fp+bFWzwX0SmNJB/ZE=.sha256 where everything between % and .sha256 is the base64-encoding of the sha256 hash of the formatted message including signature" /> <p>Currently all IDs are computed using sha256. However the protocol includes the name of the hash function in message IDs and the messages themselves to allow migrating to a different one if needed in the future.</p>
<h3 id="createHistoryStream">createHistoryStream</h3> <p>Scuttlebutt clients maintain a set of feeds that they are interested in. These could be feeds that the user has followed, bookmarked or subscribed to. When peers connect, one of the first things they do is ask each other whether the feeds they are interested in have any new messages.</p> <aside class="impl"> <img class="icon" src="img/impl.png" alt=""/> <h5>Implementations</h5> <div class="lang">JS</div> <div class="vs"><a href="https://github.com/ssbc/ssb-db/blob/master/indexes/clock.js">clock.js</a></div> </aside> <p>The RPC procedure <em>createHistoryStream</em> is how peers ask each other for a list of messages in a particular feed.</p>
<figure class="request-response"> <div class="request"> <div class="header"> <span class="key">Request number</span><span class="value">1</span> <span class="key">Body type</span><span class="value">JSON</span> <span class="key">Stream</span><span class="value">Yes</span> <span class="key">End/err</span><span class="value">No</span> </div> <pre><code>{ "name": ["createHistoryStream"], "type": "source", "args": [ { "id": "@FCX/tsDLpubCPKKfIrw4gc+SQkHcaD17s7GI6i/ziWY=.ed25519", "sequence": 2, "limit": 1, "live": false, "old": true, "keys": true } ]}</code></pre> </div> <img class="request-arrow" src="img/arrow.png" alt=""/> </figure>
<p><em>createHistoryStream</em> takes one argument which is a dictionary of additional options that controls how the feed is returned. Valid options are:</p> <table class="defs"> <tr> <td>id</td> <td>Public key of the feed to return messages from. Required.</td> </tr> <tr> <td>sequence</td> <td>Only return messages later than this sequence number. If not specified then start from the very beginning of the feed.</td> </tr> <tr> <td>limit</td> <td>Maximum number of messages to return. If the limit is exceeded only the earliest messages are returned. Default: unlimited.</td> </tr> <tr> <td>live</td> <td>If true, keep the stream alive and send new messages as they are posted. If false, end the stream after messages are sent and don’t wait for new ones. Default: false.</td> </tr> <tr> <td>old</td> <td>Used together with <em>live</em>. If true, start by sending existing messages already posted by this feed. If false, don’t send existing messages posted by this feed, only send new messages as they arrive. Default: true.</td> </tr> <tr> <td>keys</td> <td>If true, also include message IDs and timestamps of when each message was received by this peer. If false, just send the messages themselves. Default: true.</td> </tr> </table> <aside style="align-self: start; position: relative; top: 60px;"> <p>Clients must use the name <em>sequence</em> when producing requests. However for legacy reasons both <em>sequence</em> and <em>seq</em> should be accepted as the name of this argument when receiving requests. If that behaviour is supported then receiving both <em>sequence</em> and <em>seq</em> arguments with conflicting values must be considered an error.</p> </aside>
<p>Here is a comparison of <em>createHistoryStream</em> responses with <em>keys</em> set to true and false:</p>
<figure class="request-response"> <div class="request"> <div class="header"> <span class="key">Request number</span><span class="value">1</span> <span class="key">Body type</span><span class="value">JSON</span> <span class="key">Stream</span><span class="value">Yes</span> <span class="key">End/err</span><span class="value">No</span> </div> <pre><code>{ "name": ["createHistoryStream"], "type": "source", "args": [ { "id": "@FCX/tsDLpubCPKKfIrw4gc+SQkHcaD17s7GI6i/ziWY=.ed25519", "keys": <strong>true</strong> } ]}</code></pre> </div> <img class="request-arrow" src="img/arrow.png" alt=""/> <img class="response-arrow" src="img/arrow.png" alt=""/> <div class="response"> <div class="header"> <span class="key">Request number</span><span class="value">-1</span> <span class="key">Body type</span><span class="value">JSON</span> <span class="key">Stream</span><span class="value">Yes</span> <span class="key">End/err</span><span class="value">No</span> </div> <pre><code>{ "<strong>key</strong>": "%XphMUkWQtomKjXQvFGfsGYpt69sgEY7Y4Vou9cEuJho=.sha256", "value": { "previous": null, "author": "@FCX/tsDLpubCPKKfIrw4gc+SQkHcaD17s7GI6i/ziWY=.ed25519", "sequence": 1, "timestamp": 1514517067954, "hash": "sha256", "content": { "type": "post", "text": "This is the first post!" }, "signature": "QYOR/zU9dxE1aKBaxc3C0DJ4gRyZtlMfPLt+CGJcY73sv5abKK Kxr1SqhOvnm8TY784VHE8kZHCD8RdzFl1tBA==.sig.ed25519" }, "<strong>timestamp</strong>": 1514517067956}</code></pre> </div> <hr class="mini"/> <div class="request"> <div class="header"> <span class="key">Request number</span><span class="value">2</span> <span class="key">Body type</span><span class="value">JSON</span> <span class="key">Stream</span><span class="value">Yes</span> <span class="key">End/err</span><span class="value">No</span> </div> <pre><code>{ "name": ["createHistoryStream"], "type": "source", "args": [ { "id": "@FCX/tsDLpubCPKKfIrw4gc+SQkHcaD17s7GI6i/ziWY=.ed25519", "keys": <strong>false</strong> } ]}</code></pre> </div> <img class="request-arrow" src="img/arrow.png" alt=""/> <img class="response-arrow" src="img/arrow.png" alt=""/> <div class="response"> <div class="header"> <span class="key">Request number</span><span class="value">-2</span> <span class="key">Body type</span><span class="value">JSON</span> <span class="key">Stream</span><span class="value">Yes</span> <span class="key">End/err</span><span class="value">No</span> </div> <pre><code>{ "previous": null, "author": "@FCX/tsDLpubCPKKfIrw4gc+SQkHcaD17s7GI6i/ziWY=.ed25519", "sequence": 1, "timestamp": 1514517067954, "hash": "sha256", "content": { "type": "post", "text": "This is the first post!" }, "signature": "QYOR/zU9dxE1aKBaxc3C0DJ4gRyZtlMfPLt+CGJcY73sv5abKK Kxr1SqhOvnm8TY784VHE8kZHCD8RdzFl1tBA==.sig.ed25519"}</code></pre> </div> </figure> <!-- ebt documentation --> <h3 id="ebt-replication">Epidemic Broadcast Tree (EBT) Replication</h3> <aside class="impl kicker"> <img class="icon" src="img/impl.png" alt=""/> <h5>Implementations</h5> <div class="lang">JS</div> <div><a href="https://github.com/ssbc/epidemic-broadcast-trees/blob/master/index.js">epidemic-broadcast-trees</a></div> <div class="vs"><a href="https://github.com/ssbc/ssb-ebt/blob/master/index.js">ssb-ebt</a></div> <div class="lang">Go</div> <div class="vs"><a href="https://github.com/planetary-social/scuttlego/tree/main/service/domain/replication/ebt">ebt</a></div> </aside> <p>In addition to classic replication using <i>createHistoryStream</i>, some Scuttlebutt clients implement a more efficient form of replication known as <i>Epidemic broadcast tree replication</i>. This is often referred to by the abbreviation <i>EBT</i>. The implementation of <i>EBT</i> used in Scuttlebutt is loosely based on the push-lazy-push multicast tree protocol, more commonly known as the <i>Plumtree</i> protocol [1].</p> <h4 id="session-initiation">Session Initiation</h4> <p>An <i>EBT</i> session may be initiated once two peers have completed the secret handshake and have established their respective box streams. The peer who acted as the client during the secret handshake takes on the role of the requester, sending an <i>["ebt", "replicate"]</i> request to the connected peer.</p> <figure class="request-response"> <div class="request"> <div class="header"> <span class="key">Request number</span><span class="value">1</span> <span class="key">Body type</span><span class="value">JSON</span> <span class="key">Stream</span><span class="value">Yes</span> <span class="key">End/err</span><span class="value">No</span> </div> <pre><code>{ "name": ["ebt", "replicate"], "type": "duplex", "args": [ { "version": 3, "format": "classic" } ]}</code></pre> </div> <img class="request-arrow" src="img/arrow.png" alt=""/> </figure> <p>The peer who acted as the server during the secret handshake takes on the role of the responder. After having received the replicate request, the responder first validates the arguments to ensure that the version is 3 and the format is "classic". If either of those values are incorrect, the responder terminates the stream with an error.</p> <h4 id="vector-clocks">Vector Clocks</h4> <p>The responder then sends a vector clock (also known as a "note" or "control message") to the requester. The vector clock takes the form of a JSON object with one or more key-value pairs. The key of each pair specifies a Scuttlebutt feed identified by the @-prefixed public key of the author. The value of each pair is a signed integer encoding a replicate flag, a receive flag and a feed sequence number. <figure class="request-response"> <img class="response-arrow" src="img/arrow.png" alt=""/> <div class="response"> <div class="header"> <span class="key">Request number</span><span class="value">-1</span> <span class="key">Body type</span><span class="value">JSON</span> <span class="key">Stream</span><span class="value">Yes</span> <span class="key">End/err</span><span class="value">No</span> </div> <pre><code>{ "@qK93G/R9R5J2fiqK+kxV72HqqPUcss+rth8rACcYr4s=.ed25519": 450, "@L/g6qZQE/2FdO2UhSJ0uyDiZb5LjJLatM/d8MN+INSM=.ed25519": 12, "@fGHFd8rUgoznX/qS/1U7HPF3vnirbSyfaaWlS8cCWR0=.ed25519": 1, "@HEqy940T6uB+T+d9Jaa58aNfRzLx9eRWqkZljBmnkmk=.ed25519": -1}</code></pre> </div> </figure> <p>The requester terminates the stream with an error if any of the received feed identifiers or encoded values are malformed. If the received vector clock is valid, the requester can proceed with decoding the values.</p> <p>The value in each key-value pair of a vector clock encodes a maximum of three data points: a replicate flag, a receive flag and a sequence number. A negative value (usually -1) signals that the responder does not wish to replicate the associated feed, neither sending nor receiving messages. In this scenario, the replicate flag is set to false and both the receive flag and sequence number are irrelevant.</p> <p>A positive value signals that the responder wishes to replicate the associated feed. If the value is positive it should be decoded as follows. First, the JSON number is parsed and converted to a signed integer. Then, the rightmost (lowest order) bit of the number is interpreted as a binary flag with 0 equal to true and 1 equal to false. This flag is referred to as the receive flag. Next, a sign-extending right shift (also called arithmetic right shift) by 1 bit is performed on the binary number, therefore discarding the rightmost (lowest order) bit. The remaining number is then interpreted as a sequence number for the associated feed.</p> <p>If the receive flag is set to true, the peer who sent the vector clock wishes to receive messages for the associated feed. The decoded sequence number defines the latest message held by the peer for that feed.</p> <p>Encoding of a vector clock value involves reversing the steps outlined above. If the peer does not wish to replicate a feed, the value is simply set to -1. Otherwise, the latest sequence number of the associated feed is stored as a signed integer and an arithmetic left shift is performed. The rightmost (lowest order) bit is then set according to the replicate flag as described previously.</p> <table class="clock-values"> <thead> <tr> <th rowspan="2">Encoded</th> <th colspan="3">Decoded</th> </tr> <tr> <th>Replicate flag</th> <th>Receive flag</th> <th>Sequence</th> </tr> </thead> <tbody> <tr> <td>-1</td> <td>False</td> <td>Irrelevant</td> <td>Irrelevant</td> </tr> <tr> <td>0</td> <td>True</td> <td>True</td> <td>0</td> </tr> <tr> <td>1</td> <td>True</td> <td>False</td> <td>0</td> </tr> <tr> <td>2</td> <td>True</td> <td>True</td> <td>1</td> </tr> <tr> <td>3</td> <td>True</td> <td>False</td> <td>1</td> </tr> <tr> <td>12</td> <td>True</td> <td>True</td> <td>6</td> </tr> <tr> <td>450</td> <td>True</td> <td>True</td> <td>225</td> </tr> </tbody> </table> <p>The requester then sends their own vector clock to the responder. At this point, the initial exchange of vector clocks is complete and both peers may begin sending messages at will. Updated vector clocks may continue to be sent by both peers at any point during the session. These updated clocks may reference a subset of the feeds represented in the initial vector clock, or they may reference different feeds entirely. This provides a means for responding to state changes in the local database and follow graph.</p> <aside class="kicker" style="align-self: start; position: relative; top: 58px;"> <p>The same request number is used for all requests occurring between a pair of peers over the course of a single EBT session.</p> </aside> <figure class="request-response"> <div class="request"> <div class="header"> <span class="key">Request number</span><span class="value">1</span> <span class="key">Body type</span><span class="value">JSON</span> <span class="key">Stream</span><span class="value">Yes</span> <span class="key">End/err</span><span class="value">No</span> </div> <pre><code>{ "@fGHFd8rUgoznX/qS/1U7HPF3vnirbSyfaaWlS8cCWR0=.ed25519": 62, "@qK93G/R9R5J2fiqK+kxV72HqqPUcss+rth8rACcYr4s=.ed25519": -1, "@L/g6qZQE/2FdO2UhSJ0uyDiZb5LjJLatM/d8MN+INSM=.ed25519": 10}</code></pre> </div> <img class="request-arrow" src="img/arrow.png" alt=""/> </figure> <p>Messages are sent in exactly the same way as when responding to a <i>createHistoryStream</i> request. <figure class="request-response"> <img class="response-arrow" src="img/arrow.png" alt=""/> <div class="response"> <div class="header"> <span class="key">Request number</span><span class="value">-1</span> <span class="key">Body type</span><span class="value">JSON</span> <span class="key">Stream</span><span class="value">Yes</span> <span class="key">End/err</span><span class="value">No</span> </div> <pre><code>{ "previous": "%GvRqbiZFY0cNyGRD4QwjeMQvnrjz9vMPBsT5JdWrcW0=.sha256", "author": "@L/g6qZQE/2FdO2UhSJ0uyDiZb5LjJLatM/d8MN+INSM=.ed25519", "sequence": 5, "timestamp": 1698824093970, "hash": "sha256", "content": { "type": "contact", "contact": "@fGHFd8rUgoznX/qS/1U7HPF3vnirbSyfaaWlS8cCWR0=.ed25519", "blocking": false, "following": true }, "signature": "8+VQ7sbv0fnD8ZnbunJ19fyvtcwSpHvhlWUakj32nU4woFNNpI qpvkAJ4GMGJdYHoqc8C7asPXPa+wMzbPR1Cw==.sig.ed25519"}</code></pre> </div> </figure> <h4 id="session-initiation">Session Termination</h4> <p>An <i>EBT</i> session may be terminated by either peer at any point, either by sending an error response or by closing the stream. If no error has occurred, the stream is closed when a peer wishes to conclude the session (as described in the <b>Source example</b> of the <b>RPC protocol</b> section above).</p> <h4 id="request-skipping">Request Skipping</h4> <p><i>EBT</i> implementations rely on a mechanism known as <i>request skipping</i> to lower bandwidth overhead and increase replication efficiency. Each peer stores the vector clocks they receive from remote peers; these may be held in memory and persisted to disk to allow later retrieval. When a subsequent <i>EBT</i> session is initiated between peers, each peer first checks the stored vector clock of their remote peer before calculating an updated vector clock to be sent. If the latest locally-available sequence of a feed from the remote peer's vector clock is the same as the sequence in the saved vector clock for that peer, that feed is left out of the new vector clock in the outgoing request (hence the name <i>request skipping</i>). This provides a mechanism for limiting the total number of bytes to be sent over the wire.</p> <aside> <p>Approaches to tracking EBT session state may be gleaned from the JS and Go implementations.</p> </aside> <p>The stored vector clock for the remote peer may differ from their current vector clock. In that case, the remote peer will include the updated feed in their request and the local peer will respond by sending an additional partial vector clock including their sequence for that feed. Once both sides have exchanged their sequence for a particular feed, replication of messages in that feed may occur.</p> <h4 id="vector-clock-partioning">Vector Clock Partitioning</h4> <p>In order to further increase efficiency when connecting to multiple peers, feeds for which the local peer would like to receive updates are only sent to one peer at a time (in the outbound vector clock). A timeout may be used to request the feed from an alternate peer if no updates are available from the initial peer. In this way, the total set of requested feeds is spread across multiple peers.</p> <h4 id="fallback-mechanisms">Fallback Mechanisms</h4> <p><i>EBT</i> is the preferred means for peers to exchange messages. However, not all Scuttlebutt clients support <i>EBT</i> replication. In the case that only one of two connected peers support <i>EBT</i>, both peers may instead fallback to using <i>createHistoryStream</i> to exchange messages. There are several scenarios which may trigger initiation of <i>createHistoryStream</i> replication:</p> <ul> <li>If the requester attempts to initiate an <i>EBT</i> session but the session is terminated with an error by the responder</li> <li>If a <i>createHistoryStream</i> request is immediately sent by the client upon successful connection</li> <li>If the client doesn’t attempt to initiate an <i>EBT</i> session for a certain amount of time</li> </ul> <p>[1] Joao Leitao, Jose Pereira and Luis Rodrigues. 2007. Epidemic Broadcast Trees. In <i>2007 26th IEEE International Symposium on Reliable Distributed Systems (SRDS 2007)</i>, 301-310. https://doi.org/10.1109/SRDS.2007.27</p> <h2 id="blobs">Blobs</h2> <p>A blob is binary data. Blobs are referred to from Scuttlebutt messages and stored by peers in the network. When a user attaches an image to their post, that image is stored as a blob and the message contains a link to it.</p> <p>Being just a sequence of bytes, blobs can contain any data, unlike feeds and messages which have a specific structure. Blobs can handle much larger amounts of data than feeds or messages, which is why they are stored separately. However, clients may decide to not replicate blobs larger than a chosen size, which defaults to 5 megabytes in current implementations.</p> <p>Blobs are referred to by their blob ID:</p> <img src="img/format_blob_id.png" style="height: 80px;" alt="&uaGieSQDJcHfUp6hjIcIq55GoZh4Ug7tNmgaohoxrpw=.sha256 where everything between & and .sha256 is the base64-encoding of the sha256 hash of the blob"/> <p>Blobs are immutable because their ID contains a hash of their contents. This allows peers to fetch blobs from anyone in the network and know they haven’t been tampered with.</p> <p>When a Scuttlebutt implementation receives a new message and sees that it references a blob, it might be a good time to fetch that blob so that the complete message will be available to show the user.</p>
<h3 id="fetching">Fetching</h3> <p>There are two RPC methods for fetching blobs. To get an entire blob use <em>blobs.get</em>, or for only part of the blob use <em>blobs.getSlice</em>.</p> <aside class="impl"> <img class="icon" src="img/impl.png" alt=""/> <h5>Implementations</h5> <div class="lang">JS</div> <div class="vs"><a href="https://github.com/ssbc/multiblob/blob/master/index.js">multiblob</a></div> </aside>
<h4 id="blobs.get">blobs.get</h4> <p><em>blobs.get</em> asks a peer to send you the contents of a blob. There is one argument which is the blob ID:</p> <figure class="request-response"> <div class="request"> <div class="header"> <span class="key">Request number</span><span class="value">1</span> <span class="key">Body type</span><span class="value">JSON</span> <span class="key">Stream</span><span class="value">Yes</span> <span class="key">End/err</span><span class="value">No</span> </div> <pre><code>{ "name": ["blobs", "get"], "type": "source", "args": ["&uaGieSQDJcHfUp6hjIcIq55GoZh4Ug7tNmgaohoxrpw=.sha256"]}</code></pre> </div> <img class="request-arrow" src="img/arrow.png" alt=""/> </figure>
<p>Now the responder begins streaming back the contents of the blob. Each response has a body type of binary and contains a piece of the blob, starting from the beginning.</p> <p>This particular blob is 161,699 bytes long and the responder has decided to send it in pieces of 65,536 bytes each. That means two full pieces of 65,536 bytes will be sent and a final piece with the remaining 30,627 bytes.</p>
<figure class="request-response"> <img class="response-arrow" src="img/arrow.png" alt=""/> <div class="response"> <div class="header"> <span class="key">Request number</span><span class="value">-1</span> <span class="key">Body type</span><span class="value">Binary</span> <span class="key">Stream</span><span class="value">Yes</span> <span class="key">End/err</span><span class="value">No</span> </div> <div class="binary"> <span class="d3">ff</span><span class="d3">d8</span><span class="d3">ff</span><span class="d3">e0</span><span class="d0">00</span><span class="d0">10</span><span class="d1">4a</span><span class="d1">46</span><span class="d1">49</span><span class="d1">46</span><span class="d0">00</span><span class="d0">01</span><span class="d0">02</span><span class="d0">00</span><span class="d0">00</span><span class="d0">01</span><span class="d0">00</span><span class="d0">01</span><span class="d0">00</span><span class="d0">00</span><span class="d3">ff</span><span class="d3">db</span><span class="d0">00</span><span class="d1">43</span> <span class="d0">00</span><span class="d0">08</span><span class="d0">06</span><span class="d0">06</span><span class="d0">07</span><span class="d0">06</span><span class="d0">05</span><span class="d0">08</span><span class="d0">07</span><span class="d0">07</span><span class="d0">07</span><span class="d0">09</span><span class="d0">09</span><span class="d0">08</span><span class="d0">0a</span><span class="d0">0c</span><span class="d0">14</span><span class="d0">0d</span><span class="d0">0c</span><span class="d0">0b</span><span class="d0">0b</span><span class="d0">0c</span><span class="d0">19</span><span class="d0">12</span> <span class="d0">13</span><span class="d0">0f</span><span class="d0">14</span><span class="d0">1d</span><span class="d0">1a</span><span class="d0">1f</span><span class="d0">1e</span><span class="d0">1d</span><span class="d0">1a</span><span class="d0">1c</span><span class="d0">1c</span><span class="d0">20</span><span class="d0">24</span><span class="d0">2e</span><span class="d0">27</span><span class="d0">20</span><span class="d0">22</span><span class="d0">2c</span><span class="d0">23</span><span class="d0">1c</span><span class="d0">1c</span><span class="d0">28</span><span class="d0">37</span><span class="d0">29</span> <span class="d0">2c</span><span class="d0">30</span><span class="d0">31</span><span class="d0">34</span><span class="d0">34</span><span class="d0">34</span><span class="d0">1f</span><span class="d0">27</span><span class="d0">39</span><span class="d0">3d</span><span class="d0">38</span><span class="d0">32</span><span class="d0">3c</span><span class="d0">2e</span><span class="d0">33</span><span class="d0">34</span><span class="d0">32</span><span class="d3">ff</span><span class="d3">db</span> <div class="fadeout">…65536 bytes total</div> </div> </div> <img class="response-arrow" src="img/arrow.png" alt=""/> <div class="response"> <div class="header"> <span class="key">Request number</span><span class="value">-1</span> <span class="key">Body type</span><span class="value">Binary</span> <span class="key">Stream</span><span class="value">Yes</span> <span class="key">End/err</span><span class="value">No</span> </div> <div class="binary"> <span class="d2">ba</span><span class="d3">e4</span><span class="d2">92</span><span class="d0">31</span><span class="d3">ce</span><span class="d1">68</span><span class="d1">74</span><span class="d3">d0</span><span class="d3">d4</span><span class="d2">99</span><span class="d2">9e</span><span class="d1">66</span><span class="d2">b9</span><span class="d3">f3</span><span class="d0">17</span><span class="d3">ef</span><span class="d0">2b</span><span class="d0">20</span><span class="d3">dc</span><span class="d0">07</span><span class="d2">a8</span><span class="d3">ef</span><span class="d2">8a</span><span class="d3">d3</span> <span class="d2">b5</span><span class="d2">b9</span><span class="d0">33</span><span class="d0">0f</span><span class="d0">34</span><span class="d2">90</span><span class="d1">57</span><span class="d2">a7</span><span class="d0">23</span><span class="d2">a9</span><span class="d3">f6</span><span class="d0">35</span><span class="d2">93</span><span class="d0">29</span><span class="d2">b8</span><span class="d2">92</span><span class="d1">52</span><span class="d3">d2</span><span class="d1">61</span><span class="d1">49</span><span class="d0">18</span><span class="d1">55</span><span class="d0">1d</span><span class="d0">3e</span> <span class="d2">b4</span><span class="d2">91</span><span class="d1">5b</span><span class="d2">b2</span><span class="d2">a1</span><span class="d0">2c</span><span class="d3">e1</span><span class="d1">46</span><span class="d1">7b</span><span class="d1">77</span><span class="d0">35</span><span class="d0">1e</span><span class="d3">ca</span><span class="d3">c5</span><span class="d3">f3</span><span class="d3">dc</span><span class="d3">e8</span><span class="d0">16</span><span class="d1">5b</span><span class="d1">69</span><span class="d0">07</span><span class="d3">cf</span><span class="d0">31</span><span class="d0">0d</span> <span class="d3">e8</span><span class="d3">d8</span><span class="d0">35</span><span class="d0">19</span><span class="d2">b5</span><span class="d2">8d</span><span class="d0">1c</span><span class="d3">c8</span><span class="d0">1f</span><span class="d1">73</span><span class="d0">11</span><span class="d2">83</span><span class="d2">8c</span><span class="d1">67</span><span class="d3">f2</span><span class="d2">ac</span><span class="d0">29</span><span class="d0">1d</span><span class="d3">c6</span> <div class="fadeout">…65536 bytes total</div> </div> </div> <img class="response-arrow" src="img/arrow.png" alt=""/> <div class="response"> <div class="header"> <span class="key">Request number</span><span class="value">-1</span> <span class="key">Body type</span><span class="value">Binary</span> <span class="key">Stream</span><span class="value">Yes</span> <span class="key">End/err</span><span class="value">No</span> </div> <div class="binary"> <span class="d3">ca</span><span class="d3">f5</span><span class="d0">07</span><span class="d2">ad</span><span class="d0">37</span><span class="d1">63</span><span class="d1">64</span><span class="d2">af</span><span class="d3">ca</span><span class="d1">7d</span><span class="d3">f3</span><span class="d3">c5</span><span class="d1">58</span><span class="d3">e4</span><span class="d2">8e</span><span class="d0">07</span><span class="d3">cb</span><span class="d3">e9</span><span class="d3">d6</span><span class="d2">80</span><span class="d1">47</span><span class="d3">f7</span><span class="d1">78</span><span class="d3">f6</span> <span class="d2">a0</span><span class="d1">57</span><span class="d0">22</span><span class="d0">0a</span><span class="d0">36</span><span class="d3">e0</span><span class="d2">8c</span><span class="d0">1a</span><span class="d1">79</span><span class="d2">88</span><span class="d0">14</span><span class="d0">38</span><span class="d3">ed</span><span class="d3">e9</span><span class="d1">4e</span><span class="d3">c8</span><span class="d0">3d</span><span class="d0">07</span><span class="d1">4a</span><span class="d1">70</span><span class="d1">51</span><span class="d2">9c</span><span class="d2">83</span><span class="d1">45</span> <span class="d2">83</span><span class="d2">98</span><span class="d2">85</span><span class="d2">a2</span><span class="d1">5e</span><span class="d0">31</span><span class="d3">cd</span><span class="d0">39</span><span class="d1">53</span><span class="d1">6a</span><span class="d2">ae</span><span class="d1">57</span><span class="d0">3d</span><span class="d3">c7</span><span class="d2">a5</span><span class="d1">4b</span><span class="d2">9c</span><span class="d0">1e</span><span class="d0">33</span><span class="d2">9a</span><span class="d1">72</span><span class="d1">4a</span><span class="d3">c8</span><span class="d2">b2</span> <span class="d0">05</span><span class="d0">1c</span><span class="d1">48</span><span class="d0">30</span><span class="d1">72</span><span class="d0">07</span><span class="d1">4c</span><span class="d3">fe</span><span class="d2">94</span><span class="d1">58</span><span class="d1">7c</span><span class="d3">c3</span><span class="d0">14</span><span class="d2">8e</span><span class="d2">ad</span><span class="d3">d7</span><span class="d0">3e</span><span class="d2">94</span><span class="d0">38</span> <div class="fadeout">…30627 bytes total</div> </div> </div> </figure>
<p>Finally, the stream is closed with a JSON message of <code>true</code> with the end/err flag set. The requester sees that the stream is being closed and closes their end of the stream too:</p>
<figure class="request-response"> <img class="response-arrow" src="img/arrow.png" alt=""/> <div class="response"> <div class="header"> <span class="key">Request number</span><span class="value">-1</span> <span class="key">Body type</span><span class="value">JSON</span> <span class="key">Stream</span><span class="value">Yes</span> <span class="key">End/err</span><span class="value">Yes</span> </div> <pre><code>true</code></pre> </div> <div class="request"> <div class="header"> <span class="key">Request number</span><span class="value">1</span> <span class="key">Body type</span><span class="value">JSON</span> <span class="key">Stream</span><span class="value">Yes</span> <span class="key">End/err</span><span class="value">Yes</span> </div> <pre><code>true</code></pre> </div> <img class="request-arrow" src="img/arrow.png" alt=""/> </figure>
<p>Upon receiving the blob the requester should check that what they received matches the sha256 checksum they asked for.</p>
<p><em>blobs.get</em> has an alternate form where the first argument is a JSON dictionary of options:</p> <figure class="request-response"> <div class="request"> <div class="header"> <span class="key">Request number</span><span class="value">2</span> <span class="key">Body type</span><span class="value">JSON</span> <span class="key">Stream</span><span class="value">Yes</span> <span class="key">End/err</span><span class="value">No</span> </div> <pre><code>{ "name": ["blobs", "get"], "type": "source", "args": [ { "hash": "&uaGieSQDJcHfUp6hjIcIq55GoZh4Ug7tNmgaohoxrpw=.sha256", "size": 161699, "max": 200000 } ]}</code></pre> </div> <img class="request-arrow" src="img/arrow.png" alt=""/> </figure>
<p>Valid options are:</p> <table class="defs"> <tr> <td>hash</td> <td>ID of the blob. Required.</td> </tr> <tr> <td>size</td> <td>Expected size of the blob in bytes. If the blob is not exactly this size then reject the request. Optional.</td> </tr> <tr> <td>max</td> <td>Maximum size of the blob in bytes. If the blob is larger then reject the request. Only makes sense to specify <em>max</em> if you don’t already know <em>size</em>. Optional.</td> </tr> </table> <p>This form allows the responder to reject the request if the blob is not the expected size or too large. When a request is rejected the responder sends back a single message with the end/err flag set containing a JSON error message.</p>
<h4 id="blobs.getSlice">blobs.getSlice</h4> <p><em>blobs.getSlice</em> works like <em>blobs.get</em> but rather than receiving the entire blob, the requester specifies a start and end point and the responder only sends back bytes within that range.</p> <p>Reasons for using <em>blobs.getSlice</em> could be to download a blob in pieces from many peers at once, to resume a partially-completed download or to fetch the first part of a video to show a preview. Though beware that if you only fetch part of a blob you cannot be sure that it hasn’t been tampered with. You need the whole blob to verify its integrity.</p>
<figure class="request-response"> <div class="request"> <div class="header"> <span class="key">Request number</span><span class="value">1</span> <span class="key">Body type</span><span class="value">JSON</span> <span class="key">Stream</span><span class="value">Yes</span> <span class="key">End/err</span><span class="value">No</span> </div> <pre><code>{ "name": ["blobs", "getSlice"], "type": "source", "args": [ { "hash": "&uaGieSQDJcHfUp6hjIcIq55GoZh4Ug7tNmgaohoxrpw=.sha256", "start": 65536, "end": 65584 } ]}</code></pre> </div> <img class="request-arrow" src="img/arrow.png" alt=""/> <img class="response-arrow" src="img/arrow.png" alt=""/> <div class="response"> <div class="header"> <span class="key">Request number</span><span class="value">-1</span> <span class="key">Body type</span><span class="value">Binary</span> <span class="key">Stream</span><span class="value">Yes</span> <span class="key">End/err</span><span class="value">No</span> </div> <div class="binary"> <span class="d2">ba</span><span class="d3">e4</span><span class="d2">92</span><span class="d0">31</span><span class="d3">ce</span><span class="d1">68</span><span class="d1">74</span><span class="d3">d0</span><span class="d3">d4</span><span class="d2">99</span><span class="d2">9e</span><span class="d1">66</span><span class="d2">b9</span><span class="d3">f3</span><span class="d0">17</span><span class="d3">ef</span><span class="d0">2b</span><span class="d0">20</span><span class="d3">dc</span><span class="d0">07</span><span class="d2">a8</span><span class="d3">ef</span><span class="d2">8a</span><span class="d3">d3</span> <span class="d2">b5</span><span class="d2">b9</span><span class="d0">33</span><span class="d0">0f</span><span class="d0">34</span><span class="d2">90</span><span class="d1">57</span><span class="d2">a7</span><span class="d0">23</span><span class="d2">a9</span><span class="d3">f6</span><span class="d0">35</span><span class="d2">93</span><span class="d0">29</span><span class="d2">b8</span><span class="d2">92</span><span class="d1">52</span><span class="d3">d2</span><span class="d1">61</span><span class="d1">49</span><span class="d0">18</span><span class="d1">55</span><span class="d0">1d</span><span class="d0">3e</span> </div> </div> <img class="response-arrow" src="img/arrow.png" alt=""/> <div class="response"> <div class="header"> <span class="key">Request number</span><span class="value">-1</span> <span class="key">Body type</span><span class="value">JSON</span> <span class="key">Stream</span><span class="value">Yes</span> <span class="key">End/err</span><span class="value">Yes</span> </div> <pre><code>true</code></pre> </div> <div class="request"> <div class="header"> <span class="key">Request number</span><span class="value">1</span> <span class="key">Body type</span><span class="value">JSON</span> <span class="key">Stream</span><span class="value">Yes</span> <span class="key">End/err</span><span class="value">Yes</span> </div> <pre><code>true</code></pre> </div> <img class="request-arrow" src="img/arrow.png" alt=""/> </figure>
<p><em>Start</em> and <em>end</em> are byte offsets from the start of the blob and form a half-open interval, meaning the end byte is not included.</p> <img src="img/blobs_get_slice.png" style="height: 110px;" alt="For example, start=4 and end=10 returns six bytes: bytes 4, 5, 6, 7, 8, and 9; but not byte 10. (Bytes are numbered so that the first byte is byte 0)"/> <p>Specifying <em>start</em> = 0 means start from the beginning of the blob. Specifying <em>end</em> = <em>size of the blob</em> means keep going all the way to the end of the blob.</p> <p><em>blobs.getSlice</em> also accepts the <em>size</em> and <em>max</em> options, which work the same as in <em>blobs.get</em>. These two options refer to the size of the entire blob, not the range requested.</p>
<h3 id="want-and-have">Want and have</h3> <p>Peers signal to each other which blobs they want and have. This lets peers know who they can fetch a particular blob from. It also allows requests for blobs to propagate through the network until a peer that has the blob is found.</p> <aside class="impl"> <img class="icon" src="img/impl.png" alt=""/> <h5>Implementations</h5> <div class="lang">JS</div> <div class="vs"><a href="https://github.com/ssbc/ssb-blobs/blob/master/inject.js">inject.js</a></div> </aside> <p>Upon connecting, peers typically invoke the RPC method <em>blobs.createWants</em> on each other. <em>blobs.createWants</em> asks a peer to tell you which blobs they want, and also allows the peer to tell you which blobs they have.</p>
<figure class="request-response"> <div class="request"> <div class="header"> <span class="key">Request number</span><span class="value">1</span> <span class="key">Body type</span><span class="value">JSON</span> <span class="key">Stream</span><span class="value">Yes</span> <span class="key">End/err</span><span class="value">No</span> </div> <pre><code>{ "name": ["blobs", "createWants"], "type": "source", "args": []}</code></pre> </div> <img class="request-arrow" src="img/arrow.png" alt=""/> <img class="response-arrow" src="img/arrow.png" alt=""/> <div class="response"> <div class="header"> <span class="key">Request number</span><span class="value">-1</span> <span class="key">Body type</span><span class="value">JSON</span> <span class="key">Stream</span><span class="value">Yes</span> <span class="key">End/err</span><span class="value">No</span> </div> <pre><code>{}</code></pre> </div> </figure>
<p>Responding with an empty dictionary <code>{}</code> means they don’t currently want any blobs. Later, when they want a particular blob the responder will send another response back:</p>
<figure class="request-response"> <img class="response-arrow" src="img/arrow.png" alt=""/> <div class="response"> <div class="header"> <span class="key">Request number</span><span class="value">-1</span> <span class="key">Body type</span><span class="value">JSON</span> <span class="key">Stream</span><span class="value">Yes</span> <span class="key">End/err</span><span class="value">No</span> </div> <pre><code>{"&uaGieSQDJcHfUp6hjIcIq55GoZh4Ug7tNmgaohoxrpw=.sha256": -1}</code></pre> </div> </figure> <aside style="align-self: start; position: relative; top: 19px;"> <p>Only one blob is allowed per response. Send multiple responses if you want more than one blob.</p> </aside>
<p>The <code>-1</code> means the responder wants this blob themselves. <code>-2</code> means they know someone who wants the blob. <code>-3</code> means they know someone who knows someone who wants the blob, and so on.</p> <p>If a peer tells you that they want a blob and you don’t have it yourself then you can forward the want to other peers you are connected to. When forwarding a request, remember to subtract one from the number so that the request does not keep propagating through the network in an infinite loop.</p> <p>Implementations can pick a limit and ignore blob requests that come from further out than their threshold. The reference implementation forwards requests with <code>-1</code> or <code>-2</code>, but does not forward requests with <code>-3</code> or further out.</p> <p>If the responder does in fact have the blob, they can respond with its size in bytes:</p>
<figure class="request-response"> <img class="response-arrow" src="img/arrow.png" alt=""/> <div class="response"> <div class="header"> <span class="key">Request number</span><span class="value">-1</span> <span class="key">Body type</span><span class="value">JSON</span> <span class="key">Stream</span><span class="value">Yes</span> <span class="key">End/err</span><span class="value">No</span> </div> <pre><code>{"&uaGieSQDJcHfUp6hjIcIq55GoZh4Ug7tNmgaohoxrpw=.sha256": 161699}</code></pre> </div> </figure> <aside style="align-self: start; position: relative; top: 18px;"> <p><em>Have</em> and <em>want</em> messages can be distinguished because <em>have</em> messages use a positive number whereas <em>want</em> messages use a negative number.</p> </aside>
<p>Now the requester knows they can retrieve the blob from this peer with <em>blobs.get</em> or <em>blobs.getSlice</em>. <aside class="kicker" style="position: relative; top: 48px;"> <p>By fetching blobs on behalf of one another, Scuttlebutt peers can reach blobs beyond who they are immediately connected to.</p> </aside> <img src="img/blobs_fetch.png" style="height: 385px;" alt="Peer A tells Peer B: 'Want: &uaGie: -1'. Peer B tells Peer C: 'Want: &uaGie: -2'. Peer C has it, so it answers: 'Have: &uaGie: 161699'. Peer B then requests 'Get: &uaGie' and Peer C returns the content of the blob. Peer B then tells Peer A: 'Have: &uaGie'. Peer A hen requests 'Get: &uaGie' from B, and B answers with the blob."/>
<h2 id="following">Following</h2> <p>Feeds can follow other feeds. Following is a way of saying “I am interested in the messages posted by this feed”.</p> <aside class="impl"> <img class="icon" src="img/impl.png" alt=""/> <h5>Implementations</h5> <div class="lang">JS</div> <div class="vs"><a href="https://github.com/ssbc/ssb-friends/blob/master/index.js">ssb-friends</a></div> </aside> <p>Following is an optional part of Scuttlebutt, however current clients do implement it because it enables interesting social dynamics, such as introducing people that otherwise wouldn’t have found each other and allowing trust to be built within human networks. Following is what enables Scuttlebutt to be a social network rather than a group messenger.</p>
<p>When a user wants to follow another feed their client will post a message to their own feed that looks like:</p> <aside class="kicker"> <p>In this message <code>@FCX/ts…</code> is saying that they are following <code>@sxlUkN…</code>.</p> </aside> <pre><code>{ "author": "@FCX/tsDLpubCPKKfIrw4gc+SQkHcaD17s7GI6i/ziWY=.ed25519", "content": { "type": "contact", "contact": "@sxlUkN7dW/qZ23Wid6J1IAnqWEJ3V13dT6TaFtn5LTc=.ed25519", "following": true }, …}</code></pre> <p>Later, if the user decides to unfollow this feed, their client can post another message with <em>following</em> set to <code>false</code>.</p> <p>When two feeds follow each other current user interfaces will call them “friends”, however this is just a cosmetic touch and for everything else they are still treated as individual follows in each direction.</p>
<h3 id="follow-graph">Follow graph</h3> <p>Because each feed publicly announces which feeds it is following, clients can arrange feeds into a graph of who follows who:</p> <img src="img/follow_graph.png" class="pagewidth" style="height: 700px;" alt="The follow graph seen by a user is a directed graph, 'rooted' in the user. It can be seen as concentric circles, with the user in the center, and feeds arranged by proximity to the user according to the number of hops. First, feeds directly followed by the user. Then feeds not directly followed by visible in the user interface (2 hops or more). Then feeds not visible, whose content is fetched by the client nontheless. Then feeds the client has need mentioned but chosen not to fetch."/> <p>Clients may want to use the follow graph to decide which feeds to display to the user and which to replicate. If the user explicitly follows a feed that is a strong signal that the user wants to see messages posted by that feed.</p> <p>One implementation, Patchwork, shows messages up to 2 hops out by default. Messages from feeds 3 hops out are replicated to help keep them available for others, but not directly shown in the user interface. A benefit of replicating at a greater range is that for discussion threads with lots of participants it’s more likely that the client will be able to fill in gaps where someone posted who was further than 2 hops out.</p> <p>Clients could also let users configure how far the visibility and replication ranges are, or a client could use an entirely different method of deciding which feeds to show and replicate that is not based on follow range.</p>
<h2 id="pubs">Pubs</h2> <p>A pub is a publicly-accessible Scuttlebutt peer. Pubs have a social and a technical purpose:</p> <ul> <li>Pubs serve as a gathering point for new users to find other people and for existing users to welcome people who have just joined.</li> <li>Pubs have a stable IP address and allow incoming TCP connections, which enables users to connect even if their internet service provider lacks dedicated IP addresses or refuses incoming connections.</li> </ul> <p>Pubs speak the same protocol as regular peers and behave like regular peers except that they are normally run on servers so that they are always online.</p> <p>Joining a pub means following it and having it follow you back. After a new user joins a pub they will be able to see posts by the pub’s other members, and crucially the other members will be able to see the new member. This works because everyone is now within 2 hops of each other:</p> <img src="img/follow_perspective.png" class="pagewidth" style="height: 365px;" alt="From a new user perspective, the pub will be 1 hop from them, and existing users will be 2 hops from them. From the pub's perspective, all users who are members of the pub are 1 hop from it. From existing users' perspective, the pub is 1 hop from them, and both the new user and other existing users are 2 hops away."/> <p>After joining a pub, a user may decide to follow some of its members directly, in which case the pub is no longer needed for feed visibility but is still useful for accepting TCP connections and replicating messages.</p>
<h3 id="invites">Invites</h3> <p><i>This section describes the original pub invitations, also referred to as the followbot system or legacy invites, not to be confused with <a href="#rooms-joining">room invites</a> or <a href="https://github.com/ssbc/ssb-peer-invites">peer invites</a></i>. <p>Any user can follow a pub by posting a follow message, but to get the pub to follow you back you need an invite.</p> <div> <p>How the pub hands out invites is for its operator to decide. Some pubs have a web page that anybody can visit to generate an invite code. Some pubs have a pre-generated list of invite codes that the operator distributes. Invite codes can be single-use-only, have a maximum number of uses before they expire, or keep working indefinitely. A single user can use several invites if they wish to join several different pubs.</p> <p>Invite codes use this format:</p> </div> <aside> <p>If you want to run a pub that caters to an existing community, you could generate invite codes and give them out to current members. That way everyone who joins will be able to see the other members.</p> </aside> <img src="img/format_invite.png" style="height: 139px;" alt="one.butt.nz:8008:@VJM7w1W19ZsKmG2KnfaoKIM66BRoreEkzaVm/J//wl8=.ed25519~r4hIBk7KC7a9Gknj6Qiuuo4+Et/TS2rjgl6gYgw3OIM=. the parts are: a domain name or IP address of the pub, a colon, the port, a colon, the pub's public key (prefixed with an @ and suffixed with .ed25519 as usual), a tilde, then the base64-encoded invite secret key."/> <p>To make a new invite code, a pub generates a new Ed25519 key pair. The seed for the key pair (derived from the secret key) goes into the invite code, which is given to either a new user or the pub operator to distribute. The public key is remembered by the pub along with any extra conditions such as the number of remaining uses or expiry date.</p>
<h4 id="redeeming-invites">Redeeming invites</h4> <p>To redeem an invite, a user enters the invite code into their Scuttlebutt client. The client opens a new TCP connection to the domain and port contained in the invite and proceeds to set up a peer connection, starting with the handshake.</p> <aside class="impl"> <img class="icon" src="img/impl.png" alt=""/> <h5>Implementations</h5> <div class="lang">JS</div> <div class="vs"><a href="https://github.com/ssbc/scuttlebot/blob/master/plugins/invite.js">invite.js</a></div> <div class="lang">Java</div> <div class="vs"><a href="https://github.com/apache/incubator-tuweni/blob/master/scuttlebutt/src/main/java/org/apache/tuweni/scuttlebutt/Invite.java">Invite</a></div> </aside> <p>However, unlike a regular handshake, the client uses the secret key in the invite as their <img class="inline-key" src="img/key_big_a_secret.png" alt=""/> long term secret key. Because the invite only contains the secret key but not the public key, the client will need to calculate the public key from the secret key. Some cryptographic libraries call this “seeding” an Ed25519 key pair.</p> <p>Handshaking with the invite key allows the client to call a special RPC method <em>invite.use</em>:</p>
<aside class="kicker" style="align-self: start; position: relative; top: 58px;"> <p><em>invite.use</em> has a single argument which is the feed ID of the user who wants to join the pub.</p> </aside> <figure class="request-response"> <div class="request"> <div class="header"> <span class="key">Request number</span><span class="value">1</span> <span class="key">Body type</span><span class="value">JSON</span> <span class="key">Stream</span><span class="value">No</span> <span class="key">End/err</span><span class="value">No</span> </div> <pre><code>{ "name": ["invite", "use"], "type": "async", "args": [{"feed": "@FCX/tsDLpubCPKKfIrw4gc+SQkHcaD17s7GI6i/ziWY=.ed25519"}]}</code></pre> </div> <img class="request-arrow" src="img/arrow.png" alt=""/> </figure> <p>The pub checks the key used in the handshake against its list of unredeemed invites. If the invite code is valid and has not expired, the pub follows the feed ID specified by the requester and responds with its new follow message:</p> <figure class="request-response"> <img class="response-arrow" src="img/arrow.png" alt=""/> <div class="response"> <div class="header"> <span class="key">Request number</span><span class="value">-1</span> <span class="key">Body type</span><span class="value">JSON</span> <span class="key">Stream</span><span class="value">No</span> <span class="key">End/err</span><span class="value">No</span> </div> <pre><code>{ "key": "%+7u6Fa0s1cE6tS9BtKUijDV3QBYQEINH7gLSIkDqRMM=.sha256", "value": { "previous": "%S1v8a0gQkaNNzNOrWuQUomZvFrDO+XARfd6z35OmCb4=.sha256", "author": "@VJM7w1W19ZsKmG2KnfaoKIM66BRoreEkzaVm/J//wl8=.ed25519", "sequence": 14, "timestamp": 1516094092985, "hash": "sha256", "content": { "type": "contact", "contact": "@FCX/tsDLpubCPKKfIrw4gc+SQkHcaD17s7GI6i/ziWY=.ed25519", "following": true, <strong>"pub": true</strong> }, "signature":"zckFCb4XAJxNVy4tcm38o2GoFIWj25ZKa6EOaBYQA7DH/zNeWp hHuJBCCEWK6XRl4VENZWXHxPNHEBCBXDq6CA==.sig.ed25519" }}</code></pre> </div> </figure> <aside style="align-self: start; position: relative; top: 215px;"> <p>This message contains <code>"pub": true</code> to distinguish it from a normal follow.</p> </aside>
<p>Because this a pub, it is probably connected to a few other peers already. The pub will notify its peers of its new follow message using the normal <em>createHistoryStream</em> mechanism, at which point other users of the pub may decide that they are now interested in this new user’s feed and fetch it.</p> <p>Now that the pub has followed the new user, the new user should reciprocate by following the pub back. Clients automatically handle this on behalf of the user when they enter an invite code. The client posts a message to the user’s feed that follows the pub back:</p> <pre><code>{ "author": "@FCX/tsDLpubCPKKfIrw4gc+SQkHcaD17s7GI6i/ziWY=.ed25519", "content": { "type": "contact", "contact": "@VJM7w1W19ZsKmG2KnfaoKIM66BRoreEkzaVm/J//wl8=.ed25519", "following": true }, …}</code></pre> <aside> <p>This message doesn’t contain <code>"pub": true</code>.</p> </aside> <p>The client also posts a message of type <em>pub</em> to the user’s feed:</p> <pre><code>{ "author": "@FCX/tsDLpubCPKKfIrw4gc+SQkHcaD17s7GI6i/ziWY=.ed25519", "content": { "type": "pub", "address": { "host": "one.butt.nz", "port": 8008, "key": "@VJM7w1W19ZsKmG2KnfaoKIM66BRoreEkzaVm/J//wl8=.ed25519" } }, …}</code></pre> <p>These messages help peers discover new pubs where they might be able to fetch new messages from. Clients can maintain a list of pubs they have seen so that they know of alternative places to fetch messages from in case a pub goes offline.</p>
<!-- rooms documentation start -->
<h2 id="rooms">Rooms</h2>
<p>Rooms are an publicly accessible Scuttlebutt peer, but unlike <a href="#pubs">pubs</a> they don't host feed data from other peers. The main feature rooms provide is a way for peers to find each other and establish tunnelled connections among themselves as if they were on the same network. They also provide more moderation tools than a pub with multiple user-levels and different privacy modes.</p> <aside> <p>There are two rooms specifications. There is a legacy <a href="https://github.com/staltz/ssb-room" target="_blank">room 1.0</a> implementation but most users and room operators have moved to the new <a href="https://ssbc.github.io/rooms2/" target="_blank">rooms 2.0</a> specification, which is documented here.</p> </aside> <p>Rooms differ from pubs because even though they're a Scuttlebutt peer, they are not meant to aid replication by <i>diminishing the number of hops between peers</i>. Instead, they provide a simple way for peers to find and connect to each other even though these peers might not have publicly accessible connections, and thus no <i>follow</i> messages are used in the rooms workflow.</p>
<h3 id="rooms-tunnel">Tunnel</h3>
<p>A tunnel is an indirect connection between two Scuttlebutt peers that is facilitated by another peer with a stable and publicly available Internet connection. This facilitator peer, also known as the "portal" or the room, wraps the indirect connection between the other peers thus providing a tunnel for them to meet with one another as if they were on the same network.</p> <aside class="impl"> <img class="icon" src="img/impl.png"/> <h5>Implementations</h5> <div class="lang">JS</div> <div class="vs"><a href="https://github.com/ssbc/ssb-tunnel/blob/master/index.js">ssb-tunnel</a></div> <div class="lang">Go</div> <div class="vs"><a href="https://github.com/ssb-ngi-pointer/go-ssb-room/tree/master/muxrpc/handlers/tunnel/server/plugin.go">plugin.go</a></div> </aside> <img src="img/room_perspective.png" />
<p>To establish a tunnel, one of the peers (the "origin") calls the RPC <em>tunnel.connect</em> on the portal peer, specifying the "target" peer's ID:</p> <figure class="request-response"> <div class="request"> <div class="header"> <span class="key">Request number</span><span class="value">1</span> <span class="key">Body type</span><span class="value">JSON</span> <span class="key">Stream</span><span class="value">Yes</span> <span class="key">End/err</span><span class="value">No</span> </div> <pre><code>{ "name": ["tunnel", "connect"], "type": "duplex", "args": [ { "portal":"@uMYDVPuEKftL4SzpRGVyQxLdyPkOiX7njit7+qT/7IQ=.ed25519", "target":"@sxlUkN7dW/qZ23Wid6J1IAnqWEJ3V13dT6TaFtn5LTc=.ed25519" } ],}</code></pre> </div> <img class="request-arrow" src="img/arrow.png" alt=""/> </figure>
<p>The portal receives this and recognizes it as an intention to connect with the target peer. If the portal recognizes the target peer as being online, then it also calls the RPC <em>tunnel.connect</em> on the target peer, this time also specifying the "origin" peer's ID:</p> <figure class="request-response"> <div class="request"> <div class="header"> <span class="key">Request number</span><span class="value">1</span> <span class="key">Body type</span><span class="value">JSON</span> <span class="key">Stream</span><span class="value">Yes</span> <span class="key">End/err</span><span class="value">No</span> </div> <pre><code>{ "name": ["tunnel", "connect"], "type": "duplex", "args": [ { "origin":"@FCX/tsDLpubCPKKfIrw4gc+SQkHcaD17s7GI6i/ziWY=.ed25519", "portal":"@uMYDVPuEKftL4SzpRGVyQxLdyPkOiX7njit7+qT/7IQ=.ed25519", "target":"@sxlUkN7dW/qZ23Wid6J1IAnqWEJ3V13dT6TaFtn5LTc=.ed25519" } ],}</code></pre> </div> <img class="request-arrow" src="img/arrow.png" alt=""/> </figure>
<p>From this point onward, the portal should know to forward any messages on this stream from the origin to the target, and vice-versa. In the example below, the target sends this to the portal</p> <figure class="request-response"> <img class="response-arrow" src="img/arrow.png" alt=""/> <div class="response"> <div class="header"> <span class="key">Request number</span><span class="value">-1</span> <span class="key">Body type</span><span class="value">Binary</span> <span class="key">Stream</span><span class="value">Yes</span> <span class="key">End/err</span><span class="value">No</span> </div> <div class="binary"> <span class="d3">b4</span><span class="d1">6d</span><span class="d0">1d</span><span class="d3">ee</span><span class="d1">75</span><span class="d1">53</span><span class="d1">7a</span><span class="d1">61</span><span class="d1">74</span><span class="d3">fe</span><span class="d0">01</span><span class="d0">28</span><span class="d2">82</span><span class="d1">45</span><span class="d3">e9</span><span class="d3">ea</span><span class="d1">6c</span><span class="d2">a1</span><span class="d0">3e</span><span class="d1">44</span><span class="d1">49</span><span class="d1">7f</span><span class="d3">b2</span><span class="d0">3d</span> <span class="d3">cd</span><span class="d1">7e</span><span class="d0">14</span><span class="d0">18</span><span class="d1">77</span><span class="d1">73</span><span class="d1">57</span><span class="d2">aa</span><span class="d0">33</span><span class="d3">f3</span><span class="d0">2a</span><span class="d3">f0</span><span class="d3">cd</span><span class="d1">57</span><span class="d3">d6</span><span class="d2">9a</span><span class="d2">89</span><span class="d1">55</span><span class="d3">b5</span><span class="d2">9b</span><span class="d3">c4</span><span class="d2">91</span> <span class="d0">33</span><span class="d0">0d</span><span class="d1">76</span><span class="d1">69</span><span class="d1">5f</span><span class="d0">26</span><span class="d1">7b</span><span class="d1">5e</span><span class="d1">7c</span><span class="d3">b4</span><span class="d1">63</span><span class="d0">20</span><span class="d0">0e</span><span class="d0">3c</span><span class="d0">29</span><span class="d0">16</span><span class="d3">c2</span><span class="d0">08</span> </div> </div> </figure>
<p>And then the portal forwards it to the origin, unaltered:</p> <figure class="request-response"> <img class="response-arrow" src="img/arrow.png" alt=""/> <div class="response"> <div class="header"> <span class="key">Request number</span><span class="value">-1</span> <span class="key">Body type</span><span class="value">Binary</span> <span class="key">Stream</span><span class="value">Yes</span> <span class="key">End/err</span><span class="value">No</span> </div> <div class="binary"> <span class="d3">b4</span><span class="d1">6d</span><span class="d0">1d</span><span class="d3">ee</span><span class="d1">75</span><span class="d1">53</span><span class="d1">7a</span><span class="d1">61</span><span class="d1">74</span><span class="d3">fe</span><span class="d0">01</span><span class="d0">28</span><span class="d2">82</span><span class="d1">45</span><span class="d3">e9</span><span class="d3">ea</span><span class="d1">6c</span><span class="d2">a1</span><span class="d0">3e</span><span class="d1">44</span><span class="d1">49</span><span class="d1">7f</span><span class="d3">b2</span><span class="d0">3d</span> <span class="d3">cd</span><span class="d1">7e</span><span class="d0">14</span><span class="d0">18</span><span class="d1">77</span><span class="d1">73</span><span class="d1">57</span><span class="d2">aa</span><span class="d0">33</span><span class="d3">f3</span><span class="d0">2a</span><span class="d3">f0</span><span class="d3">cd</span><span class="d1">57</span><span class="d3">d6</span><span class="d2">9a</span><span class="d2">89</span><span class="d1">55</span><span class="d3">b5</span><span class="d2">9b</span><span class="d3">c4</span><span class="d2">91</span> <span class="d0">33</span><span class="d0">0d</span><span class="d1">76</span><span class="d1">69</span><span class="d1">5f</span><span class="d0">26</span><span class="d1">7b</span><span class="d1">5e</span><span class="d1">7c</span><span class="d3">b4</span><span class="d1">63</span><span class="d0">20</span><span class="d0">0e</span><span class="d0">3c</span><span class="d0">29</span><span class="d0">16</span><span class="d3">c2</span><span class="d0">08</span> </div> </div> </figure>
<p>Notice that the payload is in binary, because it's encrypted with secret handshake and box stream between the origin peer and the target peer. This means the portal cannot know what the two peers are communicating, and it cannot tamper with the binary payload in a way that the origin and target could not detect.</p>
<p>When the origin or the target close the encrypted duplex stream, they also notify the portal with a standard stream "end" flag.</p>
<h3 id="rooms-joining">Joining</h3>
<p>Before a peer can connect to a room and see other peers, they need to become an <i>authorized user</i> of that room by joining it with an invite code.</p> <p>Rooms can operate in three different modes:</p> <ul> <li><b>Open:</b> anyone can use an <i>open invite code</i> to join the room.</li> <li><b>Community:</b> other members of the room can invite new members by generating <i>individual invite codes</i> for them.</li> <li><b>Restricted:</b> only room administrators can generate <i>individual invite codes</i> and allow new users in.</li> </ul> <aside> <p>Check out the <a href="https://ssb-ngi-pointer.github.io/ssb-http-invite-spec/" target="_blank">full SSB HTTP Invite</a> specification for a deeper understanding of how it works.</p> </aside> <p>Accepting a room invite code is a process that involves both <a href="#rpc-protocol">MUXRPC</a> connections and HTTP requests. The room invite is a URI specified according to the <a href="https://github.com/ssb-ngi-pointer/ssb-uri-spec" target="_blank">SSB URI specification</a>. Once the user navigates of that URI, their operating system or web browser will launch a compatible Scuttlebutt client application that will handle the rest of the workflow.</p> <img src="img/room_invite_uri.png" /> <p>The Scuttlebutt client will extract the invite code and the submission URL from the SSB URI, and use them to claim that invite code to the room. This will involve a series of RPC calls described in the <a href="https://ssb-ngi-pointer.github.io/ssb-http-invite-spec/" target="_blank">SSB HTTP Invites specification</a>.</p> <p>Once a peer is accepted as a member of the room, their client application will see other peers in the same room as if they were on the same network and allow them to gossip with each other.</p>
<!-- end of rooms documentation -->
<h2 id="private-messages">Private messages</h2> <div> <p>Private messages are encrypted so that only certain recipients can read them. They are posted to the sender’s Scuttlebutt feed like normal messages.</p> <p>Anyone can see the fact that a feed posted a private message, who sent it (the feed that posted it), when it was sent and its size. However the list of recipients, number of recipients and message body is concealed.</p> </div> <aside class="impl"> <img class="icon" src="img/impl.png" alt=""/> <h5>Implementations</h5> <div class="lang">JS</div> <div class="vs"><a href="https://github.com/ssbc/ssb-keys/blob/master/index.js">ssb-keys</a></div> <div class="vs"><a href="https://github.com/auditdrivencrypto/private-box/blob/master/index.js">private-box</a></div> </aside> <p>Given the user wants to post these message contents:</p> <pre><code>{ "type": "post", "text": "Don’t tell anybody about this one weird trick…", "recps": [ "@FCX/tsDLpubCPKKfIrw4gc+SQkHcaD17s7GI6i/ziWY=.ed25519", "@VJM7w1W19ZsKmG2KnfaoKIM66BRoreEkzaVm/J//wl8=.ed25519" ]}</code></pre> <aside> <p>Private messages don’t reveal to recipients who the other recipients are. If you want them to know the full list of recipients then include it in the message (<em>recps</em>) as shown here.</p> </aside>
<p>This is how it would appear as a private message when posted to the user’s feed:</p> <pre><code>{ "previous": "%+7u6Fa0s1cE6tS9BtKUijDV3QBYQEINH7gLSIkDqRMM=.sha256", "author": "@FCX/tsDLpubCPKKfIrw4gc+SQkHcaD17s7GI6i/ziWY=.ed25519", "sequence": 15, "timestamp": 1516222868742, "hash": "sha256",
"content": "ilDLCLIPRruIQPuqOq1uKnkMh0VNmxD8q+DXKCbgThoAR4XvotMSbMYnodhEkgUQuEEbxjR/MHTa77 DQKY5QiGbFusUU564iDx1g/tP0qNqwir6eB0LGEna+K5QDj4CtNsHwnmDv7C0p/9n8lq/WtXlKptrO /A6riL+8EfhIWck1KCQGIZNxZz84DtpDXdN1z88rvslDNoPPzQoeGIgkt/RrGsoysuMZoJyN8LZb3X uczoSn+FhS0nWKIYnCy+CtmNiqw+9lATZgXM4+FOY8N3+L+j25hQQI191NNIdFVyMwoxkPL81byqLx ABJDLpMDSOXnWjvyzCJ68UOUwciS16/QdXE647xJ4NSC7u6uMreFIdtHTkQcP556PlZyBFwArJXbwx TUq84f5rTUt3uoG3fOllxFjRs/PPLkIcD1ihxJoSmoTTbFePclRYAV5FptRTJVHg==.box",
"signature": "6EQTBQbBhAxeE3w7kyg/7xWHUR8tXP7jUl7bWnEQVz8RxbCYgb TRUnfX/2v68xfSG5xyLAqDJ1Dh3+d+pmRvAw==.sig.ed25519"}</code></pre>
<p>When a client sees a message where the content is a string ending in <code>.box</code> it will know that it is a private message. Because clients can’t see who the message is addressed to, they will attempt to decrypt every private message they come across in case it is for them.</p>
<h3 id="encrypting">Encrypting</h3> <p>To encrypt a private message, the sender starts with these keys:</p> <img src="img/private_message_starting_keys.png" style="height: 290px;" alt="A generated Curve25519 key pair used to encrypt the message header, call the 'header key'; a generated 32-byte secret key used to encrypt the message body called the 'body key'; and the public long-term key of each recipient (max 7)."/> <p>Next, the sender uses scalar multiplication to derive a shared secret for each recipient:</p> <img src="img/private_message_secret_derivation.png" style="height: 210px;" alt="by computing the product of the secret key they just generated and that recipient's public long-term key."/> <p>The sender constructs the encrypted message by first enclosing the message content in a secret box, then enclosing the key needed to open it into a series of additional secret boxes, one to each recipient:</p> <img src="img/private_message_encrypt.png" class="pagewidth" style="height: 915px;" alt="The plaintext message content is enclosed in a secret box using the public 'body key' and a random 24-byte nonce; this box is 16 bytes (the auth tag) plus the original length of the message. The sender prepares a temporary header, made of the number of recipients (1 byte) and the secret body key (32 bytes). This temporary header is then enclosed as many times as there are recipients, in as many secret boxes (the 'encrypted headers', 16 + 33 bytes each, as they each include their own auth tag). The final message is made of: the 24 bytes nonce (defined above), the public header key (32 bytes), all the encrypted headers (49 bytes each), and finally the encrypted message content (at least 16 + 1 bytes)."/>
<h3 id="decrypting">Decrypting</h3> <p>Decrypting a private message involves scanning through the series of header secret boxes to see if any of them open using your long-term secret key. Then, if one of them opens, taking out the body secret key and using it to open the secret box holding the message content:</p> <img src="img/private_message_decrypt.png" class="pagewidth" style="height: 980px;" alt="base64-decode the message, extract the nonce and the public header key at the beginning. Then use the secret long-term key and the header public key to compute the shared secret. Read the first 49 bytes of the encrypted headers and try to open it as a secret box. If that fails, try the next 49 bytes. Keep trying up to 7 times. If the shared secret opens a secret box, then this message is intended for you. Read the number of recipients and body secret key. Now that you know how many recipients there are, skip forward 49 bytes for each remaining header. Finally open the secret box using the body secret key and read the content."/> <hr/> <h2>Archived material</h2> <div> <aside> <p> The remaining sections contains protocol descriptions that have been archived. These sections should not be regarded as needed for correctly implementing SSB. </p> </aside> <p> Metafeeds was part of a funded and focused research process to fix discovered flaws of SSB. Unfortunately, the proposal never gained adoption. Reasons were many but primarily the SSB development community at the time had already started to scatter. As you will see if you follow the links in the section below, the metafeeds work was thoroughyl specified and saw multiple proof of concept implementations. However, it never became a core part of SSB's family of clients. We leave it in the protocol guide for historical reasons and for all of the interesting ideas it represents. </p> <!-- metafeeds documentation --> <h2 id="metafeeds">Metafeeds</h2> <div> <p>A metafeed is a special kind of feed that only publishes metadata information concerning other feeds, called the <i>subfeeds</i>. A metafeed can also contain other metafeeds as subfeeds, so they form a tree structure. There can only be one <i>root metafeed</i> at the top of this tree. Using metafeeds do not break compatibility with clients that only support the classic feed format. Clients, for compatibility reasons, have a <i>main classic feed</i> alongside the root metafeed. They may support multiple additional feeds beyond that <i>main feed</i>, those additional feeds which will be organised and using a metafeed.</p> <p>Metafeeds lays the necessary groundwork to implement many interesting features such as partial replication, per-application feeds, and even the co-existence of new feed formats with classic ones.</p> </div> <aside> <p>Want to get deeper into metafeeds? Check out the <a href="https://github.com/ssb-ngi-pointer/ssb-meta-feed-spec" target="_blank">Metafeed specification</a>.</p> <p>For some example of new feed formats, check out <a href="https://github.com/AljoschaMeyer/bamboo" target="_blank">Bamboo</a> and <a href="https://github.com/ssbc/ssb-spec-drafts/blob/master/drafts/draft-ssb-core-gabbygrove/00/draft-ssb-core-gabbygrove-00.md" target="_blank">Gabbygrove</a>. </p> </aside> <h3 id="bendy-butt">Structure</h3> <p>Metafeeds uses a different format than the <a href="#structure">classic feed</a>, this new format is called <i>Bendy Butt</i>. It still is an append-only log of cryptographically signed messages, but it is not tied to JSON or V8 engine internals like the old format thus making it easier to implement in other programming languages.</p> <aside> <p>Dig deeper by reading the <a href="https://github.com/ssb-ngi-pointer/bendy-butt-spec" target="_blank">Bendy Butt Specification</a>.</p> <p>Bendy Butt uses <a href="https://github.com/ssb-ngi-pointer/ssb-binary-field-encodings">Binary Field Encodings (BFE)</a> to encode many of its fields.</p> </aside> <p>Bendy Butt encodes its messages using the well-established <a href="https://en.wikipedia.org/wiki/Bencode" target="_blank">Bencode</a> format instead of JSON. While we may display metafeed messages on this documentation as text, the encoding format uses binary delimiters instead of ascii. </p> <p>A Bendy Butt Message is a Bencoded list containing a collection with the following five elements, in this specific order.</p> <table class="defs"> <tr> <td>author</td> <td>BFE encoded feed ID.</td> </tr> <tr> <td>sequence</td> <td>Sequential 32-bit integer starting at 1.</td> </tr> <tr> <td>previous</td> <td>BFE encoded message ID, or BFE <i>nil</i> value for the first message.</td> </tr> <tr> <td>timestamp</td> <td>32-bit integer representing the time the message was created. Number of milliseconds since 1 January 1970 00:00 UTC, also known as UNIX epoch.</td> </tr> <tr> <td>contentSection</td> <td>If the message is not encrypted, This is a Bencoded list containing BFE encoded payload and signature. If it is BFE encrypted data, then upon decryption it becomes a Bencoded list of BFE encoded data and the signature for the payload.</td> </tr> </table> <img src="img/metafeed-structure.png" class="pagewidth" style="height: 855px;" alt=""/> <h3 id="metafeed-contentSection">The content section</h3> <p>Unlike the messages on the <i>classic feed format</i>—with their flexible content section—the messages in a metafeed must conform to very specific types that deal only with metadata information about feeds.</p> <p>The content section can be an encrypted text in which case it is decoded as a BFE-encoded list. If the message is not encrypted, then it is an BFE-encoded list. In both cases, the dictionary inside the list has the same fields.</p> <table class="defs"> <tr> <td>type</td> <td>Is a BFE string. It can only be one of <code>metafeed/add/existing</code>, <code>metafeed/add/derived</code>, or <code>metafeed/tombstone</code>.</td> </tr> <tr> <td>subfeed</td> <td>A BFE Feed ID.</td> </tr> <tr> <td>metafeed</td> <td>A BFE Bendy Butt Feed ID.</td> </tr> <tr> <td>feedpurpose</td> <td>A BFE String. It describes the purpose of the specified feed.</td> </tr> <tr> <td>tangles</td> <td>A BFE dictionary. It conforms to the <a target="_blank" href="https://github.com/ssbc/ssb-tangle">SSB tangles specification</a> and establishes the relationship between the message and other messages on the metafeed.</td> </tr> <tr> <td>nonce</td> <td>BFE bytes. This field is only used for messages of type <code>metafeed/add</code>. The value must be 32 random bytes.</td> </tr> </table> <h3 class="metafeed-message-types">Message Types</h3> <p>Unlike other feed types which have very flexible content sections, metafeeds supports just a small collection of message types. Metafeeds are not supposed to store arbitrary messages. Messages in the metafeed deal only with metadata information about other feeds. The supported types are:</p> <ul> <li><code>metafeed/add/existing</code>: Adds an existing feed to the metafeed.</li> <li><code>metafeed/add/derived</code>: Adds a derived feed to the metafeed.</li> <li><code>metafeed/tombstone</code>: Used to flag that a feed is no longer in use.</li> </ul>
<h3 id="metafeed-announcing">Announcing a metafeed</h3> <p>A classic main feed can let other peers know that it supports metafeeds by announcing its own root metafeed as a message of type <i>metafeed/announce</i>.</p> <pre><code>{ // ... other msg.value field ... content: { type: 'metafeed/announce', metafeed: 'ssb:feed/bendybutt-v1/APaWWDs8g73EZFUMfW37RBULtFEjwKNbDczvdYiRXtA=', tangles: { metafeed: { root: null, previous: null } } }} </code></pre> <p>This announcement message is ignored by existing apps that do not support metafeeds such that adding a metafeed is harmless to existing apps. Clients that support metafeeds will discover new metafeeds and subfeeds by looking for such announcements.</p>
<h3>Example Metafeed with Feeds</h3> <p>The sample diagram below shows a classic main feed, a root metafeed, and the associated subfeeds.</p> <img src="img/metafeed-structure-with-feeds-simplified.png" style="width: 100%;" alt=""/>
<h2 id="metafeed-use-cases">Use cases</h2> <p>The usage of metafeeds unlock a series of potential use cases and features that are much harder to implement using the classic feed format. The list below shows some of these features at a glance, use the links to the side of them to dive deeper into each specification to learn more about them and how they are implemented.</p>
<h3 id="metafeed-partial-replication">Partial Replication</h3> <p>Traditionally, Secure Scuttlebutt replicates feeds by fetching the whole feed starting from their initial message. This is a major friction point for onboarding new users into the platform due to the huge amount of data the client needs to download before a feed is up to date, and the associated computing cost for indexing the new data.</p> <aside> <p>To learn more about how partial replication works, check out the <a href="https://github.com/ssb-ngi-pointer/ssb-secure-partial-replication-spec" target="_blank">Partial Replication specification</a>. </p> </aside> <p>Partial replication allows a client to selectively fetch slices of data starting from the most recent message instead. This allows the user to be able to see recent messages — without their client freezing up — as a result of smaller data transfers and with minimal indexing time.</p>
<h3 id="metafeed-network-identity">Network Identity</h3> <p>Network identities are a way to decouple the key used for <a href="#handshake">Secret Handshake</a> from the <a href="#keys-and-identities">Identity key</a> being used by the user.</p> <aside> <p>Dive deeper by reading the <a href="https://github.com/ssb-ngi-pointer/ssb-network-identity-spec" target="_blank">Network Identity specification</a>.</p> </aside> <p>A user may have multiple network identities. They are stored in their own feed which is listed on the user's root metafeed. By having the identities listed in their own feed, a user can have private identities that are known only to them and those they trust.</p>
<h3 id="metafeed-fusion-identity">Fusion Identity</h3> <p>Fusion Identities solve the challenge of using multiple devices with Secure Scuttlebutt. Each device still uses it's own identity, there is no avoiding that, but they can now share fusion identities with each other. These new fusion identities can be used to compute mentions and notifications taking into account all the keys that are part of the fusion. They also make it possible to send private messages to the fusion identity which are readable by all the members of the fusion identity. Replicating multiple feeds from the same user can be done by following the fusion identity.</p> <aside> <p>Check out the <a href="https://github.com/ssb-ngi-pointer/fusion-identity-spec" target="_blank">Fusion Identity specification</a> for more information.</p> </aside>
<!-- end of metafeeds documentation --> </main> <footer> <p>Thanks for reading the protocol guide!</p> <p class="small"><a href="#top">Table of contents</a></p> </footer></body></html>