Data source fetches #

An automation can declare a list of fetches that run after the trigger event matches but before actions fire. Each fetch resolves to a named entry in the fetchContext, which action templates can reference via {{fetchName.record.*}} and per-fetch conditions can gate on.

There are two kinds today, discriminated by the kind field on the stored FetchStep (lib/db/schema.ts):

• kind: "record" — resolve a specific AT URI to its record. The default if kind is absent (legacy rows).
• kind: "search" — find a record in a repo by field equality. Used to answer "does a record already exist that matches X?" before acting.

Both kinds return a FetchContextEntry with a found flag; neither throws on "not found" — that signal is observed via exists / not-exists conditions on the fetch.

type FetchContextEntry = {
  found: boolean;
  uri: string;
  cid: string;
  did?: string;
  collection?: string;
  rkey?: string;
  record: Record<string, unknown>;
};

Record fetches #

The simple shape. One HTTP request, O(1), no pagination.

{
  "kind": "record",
  "name": "parentPost",
  "uri": "at://{{event.commit.record.reply.parent.uri}}"
}

Resolution #

lib/actions/fetcher.ts:

1. The uri template is rendered against the event + upstream context + owner DID. {{self}}, {{event.*}}, and {{otherFetch.*}} all work.
2. The rendered string is validated against the AT URI shape (at://did/collection/rkey). Non-AT-URI → the fetch errors with a log entry.
3. The URI is fetched via fetchRecord (lib/pds/resolver.ts), which resolves the DID to its PDS and calls com.atproto.repo.getRecord.
4. A 404, or a 400 with XRPC error body {"error":"RecordNotFound"} (which is what the com.atproto.repo.getRecord lexicon actually returns for missing records on most PDSes), writes found: false. Any other failure is treated as an error.

Record fetches are independent of each other and run in parallel via Promise.all. Their conditions are evaluated once all record fetches have resolved (and before any search fetches begin).

When to use #

• Enriching the event with the parent post, the quoted record, the followed subject's profile, etc.
• Any case where you already know the exact AT URI to look up.

Search fetches #

The richer shape. Answers "is there a record in this repo/collection whose field X equals Y?" Used primarily to prevent duplicates in mirror-style automations — e.g. "only create a Sifa follow if no Sifa follow for this subject already exists."

{
  "kind": "search",
  "name": "existingMirror",
  "repo": "{{self}}",
  "collection": "id.sifa.graph.follow",
  "where": [{ "field": "subject", "operator": "eq", "value": "{{event.commit.record.subject}}" }],
  "limit": 1,
  "conditions": [{ "field": "found", "operator": "not-exists", "value": "" }]
}

Inputs #

• repo — template that must resolve to a DID. Typically {{self}} but may reference an event field or an upstream fetch.
• collection — literal NSID (not a template). The collection to scan.
• where — list of equality clauses. Currently only operator: "eq" is supported. Multiple clauses are ANDed.
• limit — max number of matches to accept. Defaults to 1. The current executor always returns the first match as the context entry; limit just controls how many matches the executor is willing to find before stopping.
• conditions — per-fetch conditions evaluated after the search resolves. Typically found + exists / not-exists to gate on presence.

Execution strategy #

lib/actions/searcher.ts:

Search doesn't have a single primitive in AT Proto; there's no server-side equivalent of "get a record by field equality." The executor picks one of two strategies.

1. Appview fast-path (Bluesky follows only) #

The specific case of "does actor follow subject on Bluesky" is answered in O(1) by the Bluesky appview's app.bsky.graph.getRelationships endpoint. The executor detects this shape:

if (step.collection === "app.bsky.graph.follow") {
  const subjectClause = hasOnlyClause(step, "subject", "eq");
  if (subjectClause) {
    // → single appview request, parses the `following` AT URI out of the response
  }
}

Trigger conditions:

• collection === "app.bsky.graph.follow"
• Exactly one where clause
• That clause is subject eq <DID>

Any other shape on app.bsky.graph.follow falls through to the generic path.

If the appview returns no following URI for the subject, the entry is notFoundEntry() (i.e. found: false). This is the correct answer — the user doesn't follow the subject — and it's distinct from an appview transport failure, which returns null and falls through to listRecords.

2. Generic listRecords scan #

The fallback. Resolves the repo's PDS endpoint, paginates com.atproto.repo.listRecords, and filters results client-side against the where clauses.

LIST_RECORDS_PAGE_SIZE  = 100   // capped by the listRecords lexicon
MAX_LIST_RECORDS_PAGES  = 100   // → 10k records scanned at most
HTTP_TIMEOUT_MS         = 10s   // per page

The page size is a hard ceiling: com.atproto.repo.listRecords declares limit as minimum: 1, maximum: 100, default: 50, so spec-compliant PDSs will reject or clamp anything higher. To scan further we can only add pages.

Per page:

1. Fetch up to 100 records via listRecords.
2. For each record, check every where clause via dotted-path read (same machinery the condition layer uses, readPath on the record value).
3. Collect matches until limit is reached; the first match is what ends up in the context entry.
4. If the PDS returns a cursor, continue; otherwise stop.

If the scan completes without finding a match, the entry is notFoundEntry(). If the 100-page cap is hit and the cursor would continue, a warning is logged and the entry is still notFoundEntry() — we intentionally treat an exhausted scan as "not found" rather than erroring, because the usual caller is a not-exists gate and false negatives are preferable to hard failures. This is a tradeoff worth understanding: for collections large enough to exceed 10,000 records, the "does X exist?" answer may be incorrect. For the motivating use case — follow lists — this covers the vast majority of accounts; heavy-follower accounts (>10k followees) may see stale-not-found results until the cap is raised or replaced with an indexed data source.

Cost #

• Appview path: one HTTP request, a few KB, returned in tens of ms.
• listRecords path: up to 100 HTTP requests (each up to 100 records). A small repo resolves in one page; a repo on the order of 10k records is typically a few seconds when the PDS is healthy. Each page is 10s-capped individually, so the theoretical worst case is 1000s — and because searches run sequentially in the fetcher, that serializes with any other searches in the same automation. In practice that worst case only shows up when a PDS is degraded or throttling; budget for a few seconds, plan for a few more.

Searches are sequential in the fetcher — unlike record fetches which run in parallel — because a search's repo or where clauses can template against upstream fetch results. Running them serially keeps that dependency model straightforward for the MVP.

The where clause model is deliberately narrow #

The current shape (equality only, always AND) is intentional. AT Proto has no server-side query language for record fields, so every operator added to where must be evaluated in listRecords-scan code. Equality covers the anti-duplicate use case that motivated search; richer operators would encourage queries that are quietly expensive on large repos.

Per-fetch conditions #

Both fetch kinds support a conditions array. These run after the fetch resolves and are evaluated against the fetch's own entry — paths are entry-scoped, not event-scoped:

• field: "found" + exists / not-exists tests the boolean flag directly (special-cased, because stringifying false would otherwise read as non-empty).
• field: "record.subject" walks into the fetched record.
• field: "uri" / field: "cid" test the top-level entry fields.

If any condition on any fetch fails, resolveFetches returns skip: true and the handler short-circuits before actions. This is treated as normal filtering — no delivery log entry, no error. In dry-run mode the handler does write a "skipped by " log so authors can debug why the automation isn't firing.

A fetch that errors while carrying conditions is also treated as a skip; see Interaction with fetch errors.

Retries on transient failures #

Every outbound request made by the fetchers (PDS getRecord, listRecords, appview getRelationships) goes through fetchWithRetry in lib/pds/fetch-with-retry.ts. A single request is transparently retried on:

• Thrown errors from fetch (DNS failure, connection reset, TLS error).
• AbortSignal.timeout firing (per-attempt timeout).
• Any 5xx response from the server.

Retry schedule: 1s + 3s (two retries). 4xx responses are stable answers and are never retried; in particular, 404 and the XRPC RecordNotFound body on HTTP 400 still resolve to found: false on the first attempt.

Retries run inline in the handler, so a slow retry does delay the event's processing. Two retries with this schedule add at most ~4s on top of the per-attempt timeouts (currently 10s each). After retries are exhausted, the error surfaces to the caller as described below.

Interaction with fetch errors #

A fetch error (bad URI, PDS unreachable after retries, search throws) is distinct from a fetch not-finding anything. Errors are collected into the errors array on the FetchResolution and the entry is not added to the context.

Whether the automation fires when a fetch errors depends on whether that fetch carries conditions:

• Errored fetch with conditions → resolveFetches returns skip: true, skippedBy: <step.name>, skipCausedByError: true. The automation does not fire. Rationale: a condition is the user declaring "only fire if this holds." A transport error means we cannot prove it holds, so the safe default for a gated fetch is to not fire. Dry-run writes Skipped: data source "<name>" errored and had conditions so the author can see why.
• Errored fetch without conditions (enrichment-only) → the error is logged but resolution continues. Subsequent fetches that template against the missing entry will see undefined. Dry-run still surfaces the error per action as Fetch failed: <name>.

Real runs always log the underlying error to the console regardless of the skip path.

The typical pattern for "only act if X doesn't exist yet" therefore looks like:

{
  "kind": "search",
  "name": "existingMirror",
  "repo": "{{self}}",
  "collection": "id.sifa.graph.follow",
  "where": [{ "field": "subject", "operator": "eq", "value": "{{event.commit.record.subject}}" }],
  "limit": 1,
  "conditions": [{ "field": "found", "operator": "not-exists", "value": "" }]
}

• Search resolves → found: false (nothing matched).
• Condition found not-exists passes.
• Handler proceeds to actions.

If the same subject already exists:

• Search resolves → found: true with the existing record.
• Condition found not-exists fails.
• skip: true bubbles up, actions never run, no log in production.