# Before org.tangled

Since Tangled is pretty close to do a stable release and claim to be completely decentralized, it's good chance to
redesign our overall lexicons. I'll list out what is missing and what needs an attention.

# Motivation

There are bunch of big and small changes I'd like to make from current lexicon schema.

I'm not trying to build a _perfect_ lexicon schema here. We cannot make perfect lexicon schema that we never change and
I completely agree with that. But, there are several known problems around current lexicons and migration to
`org.tangled` is perfect opertunity to handle them.

After we do `org.tangled` migration and full decentralization, the migration cost become much higher than today because

1. In many cases, we can't just modify published lexicons. In most cases can only publish new lexicons in most cases
   like `org.tangled.graph.vouchV2`. To avoid doing that, we should prepare for extensibility **today**.
2. Even if we do breaking changes, we should keep consider about backwards compatibility because we won't have full
   control over the distributed data. We can't just silently run migration in background like today. The application
   should always care about legacy support. And this will massively slow down the development speed.

Again, we can't make perfect lexicons, if we should change it, we will change it. But that doesn't justify postponing
the known design tasks and _plan_ a breaking change. Breaking chagne is bad and it's always better to avoid it. We can
at least avoid future breaking changes for _known_ problems. It's good for applications, self-hosters and users. Like,
see current Knot hosting nightmare. We constantly ask self-hosters to upgrade their knots. This is not good. Imagine
asking for PDS upgrade for every single feature release... I'm not saying bluesky never does that, but they at least
tried. While we can't make it perfect, we can take some time, think deeply and try to make it future-proof. We should
not ignore existing issues and say "We can do them later". Inevitable change that we didn't expected and problem we knew
their existence but tossed to future are completely different cases. Planning a breaking change is really a bad idea,
and it doesn't help us in any sense. The amount of delay we will suffer later will be way massive then what we pay today
by designing carefully.

Migration to `org.tangled` is already a breaking change and the cost of migration is relatively low right now because we
are holding the source of truth today. In contrast, doing breaking change in future will be far more complex and take
longer. They will definitely come back to haunt us. By "they", I'm talking about problems we can predict today. Not
everything but things that we are aware of and can fix by taking some time about them. This more time is far less then
the time cost we should pay if we handle them later.

So if we can pack more breaking changes in this upgrade, it will be better for future development speed. It's better to
fix today than later.

Alright, enough explanation about the modivation. Now I'll list the things we should change and problems we should think
about. I'll list all nit-picks about current lexicons. Every single bits.

## PDS records

#### No change needed

- `sh.tangled.actor.profile`
- `sh.tangled.feed.reaction`
- `sh.tangled.graph.follow`

These records are good as-is. And can be simply renamed to `org.tangled.*`. Maybe we can choose something else than
`feed.*`? While Tangled does have a feed, that's clearly not what people think from Tangled.

I'll omit simple `sh.tangled.*` -> `org.tangled.*` renames below.

#### `sh.tangled.publicKey` -> `org.tangled.key.publicKey`

The original NSID is not clear about what public key we are referring to. Also, I think it's pretty good practice to
have at least one group (`org.tangled.key`) since it allows us to create related lexicons in same category. For example,
we can have `org.tangled.key.queryPublicKeys` xrpc method in bobbin. This "at least one group" rule will be used to
several other NSID changes.

#### `sh.tangled.feed.star`

- Change `subject` to open union
- Prefer `com.atproto.repo.strongRef` over raw AT-URI

#### `sh.tangled.graph.vouch`

- Change rkey format to TID so users can vouch same user multiple time. [more good reasons from this bluesky
  thread](https://bsky.app/profile/did:plc:6ayddqghxhciedbaofoxkcbs/post/3mkstu3uv622o)
- Make item type of `evidences` field to open union of `com.atproto.repo.strongRef` and `org.tangled.repo.strongRef`.
  The exact CID of a record is super important for an evidence.

#### `sh.tangled.repo` -> `org.tangled.repo.declaration`

- Remove most fields except `did`. All metadata should be owned by the git repository itself, not it's current owner.
- Raname to `org.tangled.repo.declaration`.

## Knot records

Following records will be stored in Knot, owned by the git repository itself.

#### `sh.tangled.repo.collaborator` -> `org.tangled.graph.collaborator`

- COB record. Hosted by `#tngl_knot` service rather than `#atproto_pds`
- Rename from `repo.*` to `graph.*`. I think `graph.*` fits better for this case. Or maybe `collab.*`..?
- Remove `repo` field as we don't need it anymore.

#### `sh.tangled.repo.artifact` -> `org.tangled.git.artifact`

> related to how COB is stored

- COB record. Hosted by `#tngl_knot` service rather than `#atproto_pds`
- Let's keep `repo.*` sub-namespace for generic "Tangled repository" like how `com.atproto.repo` is used. I don't really
  like `git.*` either, but let's use anything other than `repo.*`. That's too generic.
- We should figure out some way to host large blobs from Knot. Obvious first thought is to use raw git blob, but then we
  will suffer with the file size and general sync. Also, if we use git object store, we should figure out how to keep
  that object during garbage collection.

#### `sh.tangled.label.definition`

- COB record. Hosted by `#tngl_knot` service rather than `#atproto_pds`

#### `sh.tangled.feed.comment`

- COB record. Hosted by `#tngl_knot` service rather than `#atproto_pds`
- Change `subject` field to open union of `com.atproto.repo.strongRef` and `org.tangled.repo.strongRef`.

#### `sh.tangled.repo.issue` -> `org.tangled.project.ticket`

Few reasons about NSID change.
- See [issue/533] about issue -> ticket rename.
- I removed `repo.*` sub-namespace because:
  1. It's useless because generally all records are owned by a repository; either atproto repo or tangled git repo.
  2. `repo.*` should not be used to application-specific collection. Same reason to `org.tangled.git.artifact` NSID
     change above.
- I introduced `project.*` sub-namespace to avoid root level collection. Same reason to `org.tangled.key.publicKey` NSID
  change above. So we can later introduce new lexicons under same group, say `org.tangled.project.milestone` for exaple.
  I considered `tracker.*`, `task.*` and `backlog.*` too, but `project.*` seems best fit here. It's not too generic like
  `repo.*` while not being too specific like `tracker.*`.
- The rkey is ticket-id. So ticket #123 will be `org.tangled.project.ticket/123`. Note that this is `any` format and not
  expected to be an integer while integer is prefered. So something like `org.tangled.project.ticket/TAN-123` is allowed
  and AppView is expected to handle them as valid ticket.

#### `sh.tangled.repo.pull` -> `org.tangled.project.ticket` + `org.tangled.review.patch`

See [issue/533] for many background reasons.
- PR should also be considered as a "ticket"
- Instead of embedding PR rounds to the PR record, we can create individual records. It's good practice to have
  dedicated collection for a long list rather than embedding them. We define this "PR round collection" as
  `org.tangled.review.patch`.
- If we think about it, a patch(=round) is expected to be immutable. So let's do that! We can use CID (or OID as we will
  store them as git object) as rkey.

So.. a PR #123 with two rounds would look like this:

```json
// at+git://did:example:target_repo_did/org.tangled.project.ticket/123
{
  "title": "ref-based PR system",
  "targetBranch": "master"
}

// at+git://did:example:target_repo_did/org.tangled.review.patch/0e06a3a076211dfdd6f2bc186f5e890f20de3f87
{
  "ticket": "123", // we don't need full at-uri here. record-key format is enough since we know the target authority & collection.
  "source": {
    "source": "did:example:source_repo_did",
    "branch": "sl/ref-based-pr" // optional source branch
  },
  "base": "61e3e4ca7c4c3c5f4e7dfadabd84696503956e45",
  "head": "cb341ba107020731310317e048914f31663a5065"
}

// at+git://did:example:target_repo_did/org.tangled.review.patch/ece99f3f2e3735543d365cc00c1e01ee2fb49404
{
  "ticket": "123", // we don't need full at-uri here. record-key format is enough since we know the target authority & collection.
  "source": {
    "source": "did:example:source_repo_did",
    "branch": "op/ref-based-pr" // optional source branch
  },
  "base": "61e3e4ca7c4c3c5f4e7dfadabd84696503956e45",
  "head": "f11b8c824c208c88d7a0c2155acfd53fc30bde2a"
}
```

#### `sh.tangled.string` -> `org.tangled.snippet.string`

> But how can `org.tangled.snippet.*` expand other than strings? I cannot imagine one.

Well we already have one in bobbin: `sh.tangled.string.listStrings`. Having both `sh.tangled.string` (as a record) and
`sh.tangled.string.listStrings` (with `sh.tangled.string` as a group) is an anti-pattern so let's group them with
`snippet.*`.

## Deprecated collections

- `sh.tangled.repo.issue.state`
- `sh.tangled.repo.pull.status`
- `sh.tangled.label.op`

We can embed them to `org.tangled.project.ticket` record.

- `sh.tangled.knot`
- `sh.tangled.knot.member`
- `sh.tangled.spindle`
- `sh.tangled.spindle.member`

See [issue/335]. We don't need these.

## Summary

| old collection name             | new collection name                           | host        | note                                                 |
| -                               | -                                             | -           | -                                                    |
| `sh.tangled.actor.profile`      | `org.tangled.actor.profile`                   | atproto_pds |                                                      |
| `sh.tangled.graph.follow`       | `org.tangled.graph.follow`                    | atproto_pds |                                                      |
| `sh.tangled.publicKey`          | `org.tangled.key.publicKey`                   | atproto_pds |                                                      |
| `sh.tangled.feed.reaction`      | `org.tangled.feed.reaction`                   | atproto_pds |                                                      |
| `sh.tangled.feed.star`          | `org.tangled.feed.star`                       | atproto_pds | change `subject` to open union                       |
| `sh.tangled.string`             | `org.tangled.snippet.string`                  | atproto_pds | see pr/1568                                          |
| `sh.tangled.graph.vouch`        | `org.tangled.graph.vouch`                     | atproto_pds | change rkey format to TID                            |
| `sh.tangled.repo`               | `org.tangled.repo.declaration`                | atproto_pds | remove most fields except `did`                      |
| `sh.tangled.repo.collaborator`  | `org.tangled.graph.collaborator`              | tngl_knot   |                                                      |
| `sh.tangled.repo.artifact`      | `org.tangled.git.artifact`                    | tngl_knot   |                                                      |
| `sh.tangled.label.definition`   | `org.tangled.label.definition`                | tngl_knot   |                                                      |
| `sh.tangled.feed.comment`       | `org.tangled.feed.comment`                    | tngl_knot   | change `subject` to open union                       |
| `sh.tangled.repo.issue`         | `org.tangled.project.ticket`                  | tngl_knot   | see issue/533                                        |
| `sh.tangled.repo.pull`          | `org.tangled.review.patch` + `project.ticket` | tngl_knot   | represents a PR submission, OID as rkey so immutable |
| `sh.tangled.repo.issue.state`   | deprecated                                    |             | embed into `.ticket`                                 |
| `sh.tangled.repo.pull.status`   | deprecated                                    |             | embed into `.ticket`                                 |
| `sh.tangled.label.op`           | deprecated                                    |             | embed into `.ticket`                                 |
| `sh.tangled.knot`               | deprecated                                    |             | see issue/335                                        |
| `sh.tangled.knot.member`        | deprecated                                    |             | see issue/335                                        |
| `sh.tangled.spindle`            | deprecated                                    |             | see issue/335                                        |
| `sh.tangled.spindle.member`     | deprecated                                    |             | see issue/335                                        |
| `sh.tangled.repo.issue.comment` | deprecated                                    |             | already deprecated                                   |
| `sh.tangled.repo.pull.comment`  | deprecated                                    |             | already deprecated                                   |


## Spindle xrpc lexicons

For xrpc lexicons, let me start from spindle first because it's already pretty stabilized and relatively simple.

| host    | old method name                       | new method name                        | auth | note          |
| -       | -                                     | -                                      | -    | -             |
| spindle | `sh.tangled.repo.addSecret`           | `org.tangled.secret.putSecret`         | O    |               |
| spindle | `sh.tangled.repo.removeSecret`        | `org.tangled.secret.deleteSecret`      | O    |               |
| spindle | `sh.tangled.repo.listSecrets`         | `org.tangled.secret.listSecrets`       | O    |               |
| spindle | `sh.tangled.ci.triggerPipeline`       | `org.tangled.ci.triggerPipeline`       | O    |               |
| spindle | `sh.tangled.ci.cancelPipeline`        | `org.tangled.ci.cancelPipeline`        | O    |               |
| spindle | `sh.tangled.ci.subscribePipelineLogs` | `org.tangled.ci.subscribePipelineLogs` |      |               |
| spindle | `sh.tangled.ci.queryPipelines`        | `org.tangled.ci.queryPipelines`        |      |               |
| spindle | `sh.tangled.ci.getPipeline`           | `org.tangled.ci.getPipeline`           |      |               |
| spindle | `sh.tangled.owner`                    | deprecated                             |      | see issue/335 |

We just need few renaming and that's all. Also, I think it's relatively ok to do breaking changes to spindle spec since
it's not core service for our infra. While we absolutely don't want to hide git repositories due to Knot compatibility,
broken pipeline status view is less terrific.

## Knot xrpc lexicons

There are so many old methods to deprecate and new methods to introduce, so I'll list changes by category in this case.

### 1. Remove query methods for git data

This includes cob records since they are also stored in git object store. As we have sync, it doesn't make sense to have
query methods directly builtin to Knot. Things like `sh.tangled.repo.mergeCheck` can be performed from appview (more
specifically the knotmirror) after the sync. Knots _can_ have native support of git operation methods, but that won't be
a requirement. Imagine a PDS with lots of bsky record query builtin (yes, they do have some, but you get the point.) We
should build a stable sync rather than forcing self hosters to upgrade their Knot once a month. This way, we can be more
flexible with what we can do from synced git repos.

For example, things like [the whole recent `git.mergeCommits` design
discussion](https://discord.com/channels/1361963801993285692/1361963802848919676/1524491946763096135) will be unecessary
if we just do raw git operations from appview.

For similar reason, we should not have record-specific methods like `sh.tangled.knot.addCollaborator` and rather have
generic "record" method such as `org.tangled.repo.createRecord`.

### 2. Become more PDS-like service

- repo/record/blob lifecycle methods
- sync methods (e.g. `org.tangled.sync.listRepos`, `org.tangled.sync.subscribeRepos`)

We don't need to implement EVERY methods `atproto_pds` has because raw git can replace many sync-related methods. For
example, we don't need `com.atproto.sync.importRepo` because it can be done by `git clone` in our case.

### 2-a. How to sync git repos?

We should fix our spec on this because changing sync protocol means breaking change to
`org.tangled.sync.subscribeRepos` which is critical part for individual Knots & entire AppView to operate. Currenlty,
git sync is super dumb. Knot emits event about "something" is updated, but not exact list, nor new objects like
`com.atproto.sync.subscribeRepos#commit.blocks`.

I'm not going to say much about sync here because I haven't fully settled my mind and want to actively discuss with
others. But with current model, ideally knot should emit:

- exact list of modified refs
- repository digest : so syncer can easily determine if their copy is up-to-date
- packfile : for some performance optimization. sycncer can directly use this payload without running whole `git-fetch`
  flow.

Also, since permissioned data proposal is a thing now and our pull-based model is pretty close to them, there are many
things we can learn from them. We can start with permissioned data in mind. Private repo is **the** most requested
feature for Tangled.

### 2-b. How to store blob?

These records live in Knot and hold one or many blobs:

- `git.artifact` - for artifact data usually few GB
- `feed.comment` - for attached media files
- `project.ticket` - for attached media files

Storing blobs as git objects might not be a good idea since they can be super large. Maybe LFS? I'm not sure about this,
but I do want to solve this before the stable release. Yes, this is more close to internal implementation and can be
changed later relatively easily, but let's don't be lazy and postpone known tasks. It will only get harder later. For
example, if we change the blob hash from OID to something else (e.g. if we use LFS) that can break existing, published
records limiting our choices. So let's think about this more carefully while we are basically unlimitted and can choose
whatever method we want.

### Protocol-level vs Application-level lexicons

Before continuing, let me share my PoV about Knot spec first.

While we are dumping all records into `org.tangled.*`, I still want to divide protocol-level generic lexicons and
application-specific lexicons. Similar to how bluesky split between `com.atproto.*` lexicons (underlying generic
protocol) and `app.bsky.*` lexicons (specific application). Let's don't think that Tangled data will be only used by
Tangled itself and alternative Tangled appviews. People are already extending existing `app.bsky.*` records, mixing them
with custom lexicons like `net.anisota.*` and import them as reference like `app.bsky.actor.profile`. It will be so nice
if we can allow similar extensibility in Tangled.

I kinda want to name our custom COB-related protcol to something else and register separate domain for this protocol.
It will be less confusing for users when distinguishing which part is related to application (git forge) and which part
is generic (collaborative objects). We invented COB records that works with atproto identities to figure out the
contributed data problem. It's hard to believe Tangled will be only platform that faces similar problem.

> But Knot (git forge) is an application specific service, so what's problem with embedding application logic into it?

1. Unlike spindle which has only 9 xrpc methods and relatively not a core part of our infra, "a Knot" is pretty
   complicated service. It is **the** core piece of Tangled infra so spec isn't expected to change that often. Imagine
   atproto PDS spec changing every month due to new feature introduced in Bluesky. That's horrible user experience.
2. Are you sure we will be only service that face collaborative data problem?

Say someone tries to build a wiki service in atproto. They will immediately face the limitation of PDS architecture with
"contributed data problem" like we did. All contributed wiki pages should live under one bucket rather than being
partitioned across individual contributors PDS. We already solved this problem with COB records. It's internal
implementation is git, but that doesn't always matter for all applications. If we split the COB spec as generic thing
that isn't necessarily related to Tangled application, many completely different, but similar products can just use our
solution. We've already seen how well this approach can evolve. Users can just do things! ...if previous solver shares
their solution.

> Well turns out there is already an atproto-wiki project looking for contributive data store:
> [lichen.wiki](https://bsky.app/profile/lichen.wiki/post/3mqrh62d3jc2s)

### OAuth scope problem

TBD, but basically what I've been discussing with lewis. We need more granular permissions than "this application can
write/sign whatever COB records on your behalf." This will be problem for both PDS and Knot. Since atproto team is now
understanding our "contribution bucket" need, we can start from AppPassword-style or xrpc-hack and propose upstream spec
changes matching our need.

## bobbin / apiserver xrpc lexicons

bobbin is pretty young software and all of its lexicons are for rendering. So, we let's just dump everything under
`org.tangled.unspecced.*` and think about these later. This will allow some flexibility even after the stable release.

# More thoughts I couldn't finish writing about

- should repo identifier a DID or a space?
- what do we expect from COB model? it's pretty critical since oauth/identifiers will change.
- if we use our own uri scheme for COB records, should we avoid using `://`?
- exactly how COB will be implemented? I think we at least want multi-author and record versioning as first class.

[issue/335]: https://tangled.org/tangled.org/core/issues/335
[issue/533]: https://tangled.org/tangled.org/core/issues/533