# Data source fetches

An automation can declare a list of **fetches** that run after the trigger
event matches but before actions fire. Each fetch resolves to a named entry in
the `fetchContext`, which action templates can reference via
`{{fetchName.record.*}}` and per-fetch conditions can gate on.

There are two kinds today, discriminated by the `kind` field on the stored
`FetchStep` ([lib/db/schema.ts](../lib/db/schema.ts)):

- **`kind: "record"`** — resolve a specific AT URI to its record. The default
  if `kind` is absent (legacy rows).
- **`kind: "search"`** — find a record in a repo by field equality. Used to
  answer "does a record already exist that matches X?" before acting.

Both kinds return a `FetchContextEntry` with a `found` flag; neither throws on
"not found" — that signal is observed via `exists` / `not-exists` conditions
on the fetch.

```ts
type FetchContextEntry = {
  found: boolean;
  uri: string;
  cid: string;
  did?: string;
  collection?: string;
  rkey?: string;
  record: Record<string, unknown>;
};
```

## Record fetches

The simple shape. One HTTP request, O(1), no pagination.

```json
{
  "kind": "record",
  "name": "parentPost",
  "uri": "at://{{event.commit.record.reply.parent.uri}}"
}
```

### Resolution

[lib/actions/fetcher.ts](../lib/actions/fetcher.ts):

1. The `uri` template is rendered against the event + upstream context + owner
   DID. `{{self}}`, `{{event.*}}`, and `{{otherFetch.*}}` all work.
2. The rendered string is validated against the AT URI shape
   (`at://did/collection/rkey`). Non-AT-URI → the fetch errors with a log entry.
3. The URI is fetched via `fetchRecord` ([lib/pds/resolver.ts](../lib/pds/resolver.ts)),
   which resolves the DID to its PDS and calls `com.atproto.repo.getRecord`.
4. A 404, _or_ a 400 with XRPC error body `{"error":"RecordNotFound"}`
   (which is what the `com.atproto.repo.getRecord` lexicon actually returns
   for missing records on most PDSes), writes `found: false`. Any other
   failure is treated as an error.

Record fetches are **independent of each other** and run in parallel via
`Promise.all`. Their `conditions` are evaluated once all record fetches have
resolved (and before any search fetches begin).

### When to use

- Enriching the event with the parent post, the quoted record, the followed
  subject's profile, etc.
- Any case where you already know the exact AT URI to look up.

## Search fetches

The richer shape. Answers "is there a record in this repo/collection whose
field X equals Y?" Used primarily to prevent duplicates in mirror-style
automations — e.g. "only create a Sifa follow if no Sifa follow for this
subject already exists."

```json
{
  "kind": "search",
  "name": "existingMirror",
  "repo": "{{self}}",
  "collection": "id.sifa.graph.follow",
  "where": [{ "field": "subject", "operator": "eq", "value": "{{event.commit.record.subject}}" }],
  "limit": 1,
  "conditions": [{ "field": "found", "operator": "not-exists", "value": "" }]
}
```

### Inputs

- `repo` — template that must resolve to a DID. Typically `{{self}}` but may
  reference an event field or an upstream fetch.
- `collection` — literal NSID (not a template). The collection to scan.
- `where` — list of equality clauses. Currently only `operator: "eq"` is
  supported. Multiple clauses are ANDed.
- `limit` — max number of matches to accept. Defaults to 1. The current
  executor always returns the _first_ match as the context entry; `limit` just
  controls how many matches the executor is willing to find before stopping.
- `conditions` — per-fetch conditions evaluated after the search resolves.
  Typically `found` + `exists` / `not-exists` to gate on presence.

### Execution strategy

[lib/actions/searcher.ts](../lib/actions/searcher.ts):

Search doesn't have a single primitive in AT Proto; there's no server-side
equivalent of "get a record by field equality." The executor picks one of two
strategies.

#### 1. Appview fast-path (Bluesky follows only)

The specific case of "does `actor` follow `subject` on Bluesky" is answered in
O(1) by the Bluesky appview's `app.bsky.graph.getRelationships` endpoint. The
executor detects this shape:

```ts
if (step.collection === "app.bsky.graph.follow") {
  const subjectClause = hasOnlyClause(step, "subject", "eq");
  if (subjectClause) {
    // → single appview request, parses the `following` AT URI out of the response
  }
}
```

Trigger conditions:

- `collection === "app.bsky.graph.follow"`
- Exactly one `where` clause
- That clause is `subject eq <DID>`

Any other shape on `app.bsky.graph.follow` falls through to the generic path.

If the appview returns no `following` URI for the subject, the entry is
`notFoundEntry()` (i.e. `found: false`). This is the correct answer — the user
doesn't follow the subject — and it's distinct from an appview transport
failure, which returns `null` and falls through to `listRecords`.

#### 2. Generic `listRecords` scan

The fallback. Resolves the repo's PDS endpoint, paginates
`com.atproto.repo.listRecords`, and filters results client-side against the
`where` clauses.

```
LIST_RECORDS_PAGE_SIZE  = 100   // capped by the listRecords lexicon
MAX_LIST_RECORDS_PAGES  = 100   // → 10k records scanned at most
HTTP_TIMEOUT_MS         = 10s   // per page
```

The page size is a hard ceiling: `com.atproto.repo.listRecords` declares
`limit` as `minimum: 1, maximum: 100, default: 50`, so spec-compliant PDSs
will reject or clamp anything higher. To scan further we can only add pages.

Per page:

1. Fetch up to 100 records via `listRecords`.
2. For each record, check every `where` clause via dotted-path read (same
   machinery the condition layer uses, `readPath` on the record value).
3. Collect matches until `limit` is reached; the first match is what ends up
   in the context entry.
4. If the PDS returns a `cursor`, continue; otherwise stop.

If the scan completes without finding a match, the entry is `notFoundEntry()`.
If the 100-page cap is hit _and_ the cursor would continue, a warning is
logged and the entry is still `notFoundEntry()` — we intentionally treat an
exhausted scan as "not found" rather than erroring, because the usual caller
is a `not-exists` gate and false negatives are preferable to hard failures.
This is a tradeoff worth understanding: for collections large enough to
exceed 10,000 records, the "does X exist?" answer may be incorrect. For the
motivating use case — follow lists — this covers the vast majority of
accounts; heavy-follower accounts (>10k followees) may see stale-not-found
results until the cap is raised or replaced with an indexed data source.

### Cost

- **Appview path**: one HTTP request, a few KB, returned in tens of ms.
- **listRecords path**: up to 100 HTTP requests (each up to 100 records). A
  small repo resolves in one page; a repo on the order of 10k records is
  typically a few seconds when the PDS is healthy. Each page is 10s-capped
  individually, so the theoretical worst case is 1000s — and because
  searches run sequentially in the fetcher, that serializes with any other
  searches in the same automation. In practice that worst case only shows up
  when a PDS is degraded or throttling; budget for a few seconds, plan for
  a few more.

Searches are **sequential** in the fetcher — unlike record fetches which run in
parallel — because a search's `repo` or `where` clauses can template against
upstream fetch results. Running them serially keeps that dependency model
straightforward for the MVP.

### The `where` clause model is deliberately narrow

The current shape (equality only, always AND) is intentional. AT Proto has no
server-side query language for record fields, so every operator added to
`where` must be evaluated in `listRecords`-scan code. Equality covers the
anti-duplicate use case that motivated search; richer operators would encourage
queries that are quietly expensive on large repos.

## Per-fetch conditions

Both fetch kinds support a `conditions` array. These run _after_ the fetch
resolves and are evaluated against the fetch's own entry — paths are
entry-scoped, not event-scoped:

- `field: "found"` + `exists` / `not-exists` tests the boolean flag directly
  (special-cased, because stringifying `false` would otherwise read as
  non-empty).
- `field: "record.subject"` walks into the fetched record.
- `field: "uri"` / `field: "cid"` test the top-level entry fields.

If any condition on any fetch fails, `resolveFetches` returns `skip: true` and
the handler short-circuits before actions. This is treated as normal filtering
— no delivery log entry, no error. In dry-run mode the handler does write a
"skipped by <fetchName>" log so authors can debug why the automation isn't
firing.

A fetch that errors while carrying conditions is also treated as a skip;
see [Interaction with fetch errors](#interaction-with-fetch-errors).

## Retries on transient failures

Every outbound request made by the fetchers (PDS `getRecord`, `listRecords`,
appview `getRelationships`) goes through `fetchWithRetry` in
[lib/pds/fetch-with-retry.ts](../lib/pds/fetch-with-retry.ts). A single request
is transparently retried on:

- Thrown errors from `fetch` (DNS failure, connection reset, TLS error).
- `AbortSignal.timeout` firing (per-attempt timeout).
- Any `5xx` response from the server.

Retry schedule: `1s + 3s` (two retries). 4xx responses are stable answers and
are never retried; in particular, 404 and the XRPC `RecordNotFound` body on
HTTP 400 still resolve to `found: false` on the first attempt.

Retries run inline in the handler, so a slow retry does delay the event's
processing. Two retries with this schedule add at most ~4s on top of the
per-attempt timeouts (currently 10s each). After retries are exhausted, the
error surfaces to the caller as described below.

## Interaction with fetch errors

A fetch **error** (bad URI, PDS unreachable after retries, search throws) is
distinct from a fetch **not-finding** anything. Errors are collected into the
`errors` array on the `FetchResolution` and the entry is not added to the
context.

Whether the automation fires when a fetch errors depends on whether that
fetch carries **conditions**:

- **Errored fetch with conditions** → `resolveFetches` returns
  `skip: true`, `skippedBy: <step.name>`, `skipCausedByError: true`. The
  automation does not fire. Rationale: a condition is the user declaring
  "only fire if this holds." A transport error means we cannot prove it
  holds, so the safe default for a gated fetch is to not fire. Dry-run
  writes `Skipped: data source "<name>" errored and had conditions` so the
  author can see why.
- **Errored fetch without conditions** (enrichment-only) → the error is
  logged but resolution continues. Subsequent fetches that template against
  the missing entry will see `undefined`. Dry-run still surfaces the error
  per action as `Fetch failed: <name>`.

Real runs always log the underlying error to the console regardless of the
skip path.

The typical pattern for "only act if X doesn't exist yet" therefore looks like:

```json
{
  "kind": "search",
  "name": "existingMirror",
  "repo": "{{self}}",
  "collection": "id.sifa.graph.follow",
  "where": [{ "field": "subject", "operator": "eq", "value": "{{event.commit.record.subject}}" }],
  "limit": 1,
  "conditions": [{ "field": "found", "operator": "not-exists", "value": "" }]
}
```

- Search resolves → `found: false` (nothing matched).
- Condition `found not-exists` passes.
- Handler proceeds to actions.

If the same subject already exists:

- Search resolves → `found: true` with the existing record.
- Condition `found not-exists` fails.
- `skip: true` bubbles up, actions never run, no log in production.