Before org.tangled #

Since Tangled is pretty close to do a stable release and claim to be completely decentralized, it's good chance to redesign our overall lexicons. I'll list out what is missing and what needs an attention.

Motivation #

There are bunch of big and small changes I'd like to make from current lexicon schema.

I'm not trying to build a perfect lexicon schema here. We cannot make perfect lexicon schema that we never change and I completely agree with that. But, there are several known problems around current lexicons and migration to org.tangled is perfect opertunity to handle them.

After we do org.tangled migration and full decentralization, the migration cost become much higher than today because

1. In many cases, we can't just modify published lexicons. In most cases can only publish new lexicons in most cases like org.tangled.graph.vouchV2. To avoid doing that, we should prepare for extensibility today.
2. Even if we do breaking changes, we should keep consider about backwards compatibility because we won't have full control over the distributed data. We can't just silently run migration in background like today. The application should always care about legacy support. And this will massively slow down the development speed.

Again, we can't make perfect lexicons, if we should change it, we will change it. But that doesn't justify postponing the known design tasks and plan a breaking change. Breaking chagne is bad and it's always better to avoid it. We can at least avoid future breaking changes for known problems. It's good for applications, self-hosters and users. Like, see current Knot hosting nightmare. We constantly ask self-hosters to upgrade their knots. This is not good. Imagine asking for PDS upgrade for every single feature release... I'm not saying bluesky never does that, but they at least tried. While we can't make it perfect, we can take some time, think deeply and try to make it future-proof. We should not ignore existing issues and say "We can do them later". Inevitable change that we didn't expected and problem we knew their existence but tossed to future are completely different cases. Planning a breaking change is really a bad idea, and it doesn't help us in any sense. The amount of delay we will suffer later will be way massive then what we pay today by designing carefully.

Migration to org.tangled is already a breaking change and the cost of migration is relatively low right now because we are holding the source of truth today. In contrast, doing breaking change in future will be far more complex and take longer. They will definitely come back to haunt us. By "they", I'm talking about problems we can predict today. Not everything but things that we are aware of and can fix by taking some time about them. This more time is far less then the time cost we should pay if we handle them later.

So if we can pack more breaking changes in this upgrade, it will be better for future development speed. It's better to fix today than later.

Alright, enough explanation about the modivation. Now I'll list the things we should change and problems we should think about. I'll list all nit-picks about current lexicons. Every single bits.

PDS records #

No change needed #

• sh.tangled.actor.profile
• sh.tangled.feed.reaction
• sh.tangled.graph.follow

These records are good as-is. And can be simply renamed to org.tangled.*. Maybe we can choose something else than feed.*? While Tangled does have a feed, that's clearly not what people think from Tangled.

I'll omit simple sh.tangled.* -> org.tangled.* renames below.

sh.tangled.publicKey -> org.tangled.key.publicKey #

The original NSID is not clear about what public key we are referring to. Also, I think it's pretty good practice to have at least one group (org.tangled.key) since it allows us to create related lexicons in same category. For example, we can have org.tangled.key.queryPublicKeys xrpc method in bobbin. This "at least one group" rule will be used to several other NSID changes.

sh.tangled.feed.star #

• Change subject to open union
• Prefer com.atproto.repo.strongRef over raw AT-URI

sh.tangled.graph.vouch #

• Change rkey format to TID so users can vouch same user multiple time. more good reasons from this bluesky thread
• Make item type of evidences field to open union of com.atproto.repo.strongRef and org.tangled.repo.strongRef. The exact CID of a record is super important for an evidence.

sh.tangled.repo -> org.tangled.repo.declaration #

• Remove most fields except did. All metadata should be owned by the git repository itself, not it's current owner.
• Raname to org.tangled.repo.declaration.

Knot records #

Following records will be stored in Knot, owned by the git repository itself.

sh.tangled.repo.collaborator -> org.tangled.graph.collaborator #

• COB record. Hosted by #tngl_knot service rather than #atproto_pds
• Rename from repo.* to graph.*. I think graph.* fits better for this case. Or maybe collab.*..?
• Remove repo field as we don't need it anymore.

sh.tangled.repo.artifact -> org.tangled.git.artifact #

related to how COB is stored

• COB record. Hosted by #tngl_knot service rather than #atproto_pds
• Let's keep repo.* sub-namespace for generic "Tangled repository" like how com.atproto.repo is used. I don't really like git.* either, but let's use anything other than repo.*. That's too generic.
• We should figure out some way to host large blobs from Knot. Obvious first thought is to use raw git blob, but then we will suffer with the file size and general sync. Also, if we use git object store, we should figure out how to keep that object during garbage collection.

sh.tangled.label.definition #

• COB record. Hosted by #tngl_knot service rather than #atproto_pds

sh.tangled.feed.comment #

• COB record. Hosted by #tngl_knot service rather than #atproto_pds
• Change subject field to open union of com.atproto.repo.strongRef and org.tangled.repo.strongRef.

sh.tangled.repo.issue -> org.tangled.project.ticket #

Few reasons about NSID change.

• See issue/533 about issue -> ticket rename.
• I removed repo.* sub-namespace because:
  1. It's useless because generally all records are owned by a repository; either atproto repo or tangled git repo.
  2. repo.* should not be used to application-specific collection. Same reason to org.tangled.git.artifact NSID change above.
• I introduced project.* sub-namespace to avoid root level collection. Same reason to org.tangled.key.publicKey NSID change above. So we can later introduce new lexicons under same group, say org.tangled.project.milestone for exaple. I considered tracker.*, task.* and backlog.* too, but project.* seems best fit here. It's not too generic like repo.* while not being too specific like tracker.*.
• The rkey is ticket-id. So ticket #123 will be org.tangled.project.ticket/123. Note that this is any format and not expected to be an integer while integer is prefered. So something like org.tangled.project.ticket/TAN-123 is allowed and AppView is expected to handle them as valid ticket.

sh.tangled.repo.pull -> org.tangled.project.ticket + org.tangled.review.patch #

See issue/533 for many background reasons.

• PR should also be considered as a "ticket"
• Instead of embedding PR rounds to the PR record, we can create individual records. It's good practice to have dedicated collection for a long list rather than embedding them. We define this "PR round collection" as org.tangled.review.patch.
• If we think about it, a patch(=round) is expected to be immutable. So let's do that! We can use CID (or OID as we will store them as git object) as rkey.

So.. a PR #123 with two rounds would look like this:

// at+git://did:example:target_repo_did/org.tangled.project.ticket/123
{
  "title": "ref-based PR system",
  "targetBranch": "master"
}

// at+git://did:example:target_repo_did/org.tangled.review.patch/0e06a3a076211dfdd6f2bc186f5e890f20de3f87
{
  "ticket": "123", // we don't need full at-uri here. record-key format is enough since we know the target authority & collection.
  "source": {
    "source": "did:example:source_repo_did",
    "branch": "sl/ref-based-pr" // optional source branch
  },
  "base": "61e3e4ca7c4c3c5f4e7dfadabd84696503956e45",
  "head": "cb341ba107020731310317e048914f31663a5065"
}

// at+git://did:example:target_repo_did/org.tangled.review.patch/ece99f3f2e3735543d365cc00c1e01ee2fb49404
{
  "ticket": "123", // we don't need full at-uri here. record-key format is enough since we know the target authority & collection.
  "source": {
    "source": "did:example:source_repo_did",
    "branch": "op/ref-based-pr" // optional source branch
  },
  "base": "61e3e4ca7c4c3c5f4e7dfadabd84696503956e45",
  "head": "f11b8c824c208c88d7a0c2155acfd53fc30bde2a"
}

sh.tangled.string -> org.tangled.snippet.string #

But how can org.tangled.snippet.* expand other than strings? I cannot imagine one.

Well we already have one in bobbin: sh.tangled.string.listStrings. Having both sh.tangled.string (as a record) and sh.tangled.string.listStrings (with sh.tangled.string as a group) is an anti-pattern so let's group them with snippet.*.

Deprecated collections #

• sh.tangled.repo.issue.state
• sh.tangled.repo.pull.status
• sh.tangled.label.op

We can embed them to org.tangled.project.ticket record.

• sh.tangled.knot
• sh.tangled.knot.member
• sh.tangled.spindle
• sh.tangled.spindle.member

See issue/335. We don't need these.

Summary #

old collection name	new collection name	host	note
sh.tangled.actor.profile	org.tangled.actor.profile	atproto_pds
sh.tangled.graph.follow	org.tangled.graph.follow	atproto_pds
sh.tangled.publicKey	org.tangled.key.publicKey	atproto_pds
sh.tangled.feed.reaction	org.tangled.feed.reaction	atproto_pds
sh.tangled.feed.star	org.tangled.feed.star	atproto_pds	change subject to open union
sh.tangled.string	org.tangled.snippet.string	atproto_pds	see pr/1568
sh.tangled.graph.vouch	org.tangled.graph.vouch	atproto_pds	change rkey format to TID
sh.tangled.repo	org.tangled.repo.declaration	atproto_pds	remove most fields except did
sh.tangled.repo.collaborator	org.tangled.graph.collaborator	tngl_knot
sh.tangled.repo.artifact	org.tangled.git.artifact	tngl_knot
sh.tangled.label.definition	org.tangled.label.definition	tngl_knot
sh.tangled.feed.comment	org.tangled.feed.comment	tngl_knot	change subject to open union
sh.tangled.repo.issue	org.tangled.project.ticket	tngl_knot	see issue/533
sh.tangled.repo.pull	org.tangled.review.patch + project.ticket	tngl_knot	represents a PR submission, OID as rkey so immutable
sh.tangled.repo.issue.state	deprecated		embed into .ticket
sh.tangled.repo.pull.status	deprecated		embed into .ticket
sh.tangled.label.op	deprecated		embed into .ticket
sh.tangled.knot	deprecated		see issue/335
sh.tangled.knot.member	deprecated		see issue/335
sh.tangled.spindle	deprecated		see issue/335
sh.tangled.spindle.member	deprecated		see issue/335
sh.tangled.repo.issue.comment	deprecated		already deprecated
sh.tangled.repo.pull.comment	deprecated		already deprecated

Spindle xrpc lexicons #

For xrpc lexicons, let me start from spindle first because it's already pretty stabilized and relatively simple.

host	old method name	new method name	auth	note
spindle	sh.tangled.repo.addSecret	org.tangled.secret.putSecret	O
spindle	sh.tangled.repo.removeSecret	org.tangled.secret.deleteSecret	O
spindle	sh.tangled.repo.listSecrets	org.tangled.secret.listSecrets	O
spindle	sh.tangled.ci.triggerPipeline	org.tangled.ci.triggerPipeline	O
spindle	sh.tangled.ci.cancelPipeline	org.tangled.ci.cancelPipeline	O
spindle	sh.tangled.ci.subscribePipelineLogs	org.tangled.ci.subscribePipelineLogs
spindle	sh.tangled.ci.queryPipelines	org.tangled.ci.queryPipelines
spindle	sh.tangled.ci.getPipeline	org.tangled.ci.getPipeline
spindle	sh.tangled.owner	deprecated		see issue/335

We just need few renaming and that's all. Also, I think it's relatively ok to do breaking changes to spindle spec since it's not core service for our infra. While we absolutely don't want to hide git repositories due to Knot compatibility, broken pipeline status view is less terrific.

Knot xrpc lexicons #

There are so many old methods to deprecate and new methods to introduce, so I'll list changes by category in this case.

1. Remove query methods for git data #

This includes cob records since they are also stored in git object store. As we have sync, it doesn't make sense to have query methods directly builtin to Knot. Things like sh.tangled.repo.mergeCheck can be performed from appview (more specifically the knotmirror) after the sync. Knots can have native support of git operation methods, but that won't be a requirement. Imagine a PDS with lots of bsky record query builtin (yes, they do have some, but you get the point.) We should build a stable sync rather than forcing self hosters to upgrade their Knot once a month. This way, we can be more flexible with what we can do from synced git repos.

For example, things like the whole recent git.mergeCommits design discussion will be unecessary if we just do raw git operations from appview.

For similar reason, we should not have record-specific methods like sh.tangled.knot.addCollaborator and rather have generic "record" method such as org.tangled.repo.createRecord.

2. Become more PDS-like service #

• repo/record/blob lifecycle methods
• sync methods (e.g. org.tangled.sync.listRepos, org.tangled.sync.subscribeRepos)

We don't need to implement EVERY methods atproto_pds has because raw git can replace many sync-related methods. For example, we don't need com.atproto.sync.importRepo because it can be done by git clone in our case.

2-a. How to sync git repos? #

We should fix our spec on this because changing sync protocol means breaking change to org.tangled.sync.subscribeRepos which is critical part for individual Knots & entire AppView to operate. Currenlty, git sync is super dumb. Knot emits event about "something" is updated, but not exact list, nor new objects like com.atproto.sync.subscribeRepos#commit.blocks.

I'm not going to say much about sync here because I haven't fully settled my mind and want to actively discuss with others. But with current model, ideally knot should emit:

• exact list of modified refs
• repository digest : so syncer can easily determine if their copy is up-to-date
• packfile : for some performance optimization. sycncer can directly use this payload without running whole git-fetch flow.

Also, since permissioned data proposal is a thing now and our pull-based model is pretty close to them, there are many things we can learn from them. We can start with permissioned data in mind. Private repo is the most requested feature for Tangled.

2-b. How to store blob? #

These records live in Knot and hold one or many blobs:

• git.artifact - for artifact data usually few GB
• feed.comment - for attached media files
• project.ticket - for attached media files

Storing blobs as git objects might not be a good idea since they can be super large. Maybe LFS? I'm not sure about this, but I do want to solve this before the stable release. Yes, this is more close to internal implementation and can be changed later relatively easily, but let's don't be lazy and postpone known tasks. It will only get harder later. For example, if we change the blob hash from OID to something else (e.g. if we use LFS) that can break existing, published records limiting our choices. So let's think about this more carefully while we are basically unlimitted and can choose whatever method we want.

Protocol-level vs Application-level lexicons #

Before continuing, let me share my PoV about Knot spec first.

While we are dumping all records into org.tangled.*, I still want to divide protocol-level generic lexicons and application-specific lexicons. Similar to how bluesky split between com.atproto.* lexicons (underlying generic protocol) and app.bsky.* lexicons (specific application). Let's don't think that Tangled data will be only used by Tangled itself and alternative Tangled appviews. People are already extending existing app.bsky.* records, mixing them with custom lexicons like net.anisota.* and import them as reference like app.bsky.actor.profile. It will be so nice if we can allow similar extensibility in Tangled.

I kinda want to name our custom COB-related protcol to something else and register separate domain for this protocol. It will be less confusing for users when distinguishing which part is related to application (git forge) and which part is generic (collaborative objects). We invented COB records that works with atproto identities to figure out the contributed data problem. It's hard to believe Tangled will be only platform that faces similar problem.

But Knot (git forge) is an application specific service, so what's problem with embedding application logic into it?

1. Unlike spindle which has only 9 xrpc methods and relatively not a core part of our infra, "a Knot" is pretty complicated service. It is the core piece of Tangled infra so spec isn't expected to change that often. Imagine atproto PDS spec changing every month due to new feature introduced in Bluesky. That's horrible user experience.
2. Are you sure we will be only service that face collaborative data problem?

Say someone tries to build a wiki service in atproto. They will immediately face the limitation of PDS architecture with "contributed data problem" like we did. All contributed wiki pages should live under one bucket rather than being partitioned across individual contributors PDS. We already solved this problem with COB records. It's internal implementation is git, but that doesn't always matter for all applications. If we split the COB spec as generic thing that isn't necessarily related to Tangled application, many completely different, but similar products can just use our solution. We've already seen how well this approach can evolve. Users can just do things! ...if previous solver shares their solution.

Well turns out there is already an atproto-wiki project looking for contributive data store: lichen.wiki

OAuth scope problem #

TBD, but basically what I've been discussing with lewis. We need more granular permissions than "this application can write/sign whatever COB records on your behalf." This will be problem for both PDS and Knot. Since atproto team is now understanding our "contribution bucket" need, we can start from AppPassword-style or xrpc-hack and propose upstream spec changes matching our need.

bobbin / apiserver xrpc lexicons #

bobbin is pretty young software and all of its lexicons are for rendering. So, we let's just dump everything under org.tangled.unspecced.* and think about these later. This will allow some flexibility even after the stable release.

More thoughts I couldn't finish writing about #

• should repo identifier a DID or a space?
• what do we expect from COB model? it's pretty critical since oauth/identifiers will change.
• if we use our own uri scheme for COB records, should we avoid using ://?
• exactly how COB will be implemented? I think we at least want multi-author and record versioning as first class.