site.standard Lexicon Migration Summary #

Overview #

This migration adds support for the new site.standard.* lexicons alongside the existing pub.leaflet.* lexicons. Both record formats are now stored in the same database tables and normalized to a common interface for consumption.

Key Changes #

1. Appview Updates (appview/index.ts) #

The firehose consumer now writes site.standard.* records directly to the main tables:

• site.standard.document → documents table
• site.standard.publication → publications table
• site.standard.graph.subscription → publication_subscriptions table

This means both pub.leaflet.* and site.standard.* records coexist in the same tables, differentiated by their $type field.

2. Normalization Layer (src/utils/normalizeRecords.ts) #

Created a normalization layer that converts both record formats to a common interface:

// Normalized types
type NormalizedDocument = SiteStandardDocument.Record & { theme?: PubLeafletPublication.Theme }
type NormalizedPublication = SiteStandardPublication.Record

// Helper functions
normalizeDocumentRecord(data: Json | unknown): NormalizedDocument | null
normalizePublicationRecord(record: Json | unknown): NormalizedPublication | null
getDocumentPages(doc: NormalizedDocument): Page[]

Field mappings handled by normalization:

pub.leaflet	site.standard	Normalized
content.pages	pages	pages
content.title	title	title
bskyPostRef	postRef	postRef
base_path	url (full URL)	url

3. Data Sources Pre-Normalize Records #

Contexts and data fetchers now return pre-normalized data so consumers don't need to normalize themselves:

Hooks updated:

• useLeafletPublicationData() → returns normalizedDocument and normalizedPublication
• useNormalizedPublicationRecord() → new hook for publication dashboard context

Server-side data fetchers updated:

• getPostPageData.ts → returns normalizedDocument and normalizedPublication
• get_profile_data.ts → normalizes publication records before returning
• getSubscriptions.ts → normalizes publication records
• getPublications.ts → normalizes publication records
• getReaderFeed.ts → uses normalized types in Post type
• src/notifications.ts → hydration functions return normalized fields

4. Removed Type Casts #

Removed 70+ casts to PubLeafletDocument.Record and PubLeafletPublication.Record across 40+ files. Components now use the normalized types instead.

5. Updated Components #

All components that consume document/publication data now:

1. Use pre-normalized data from contexts/hooks when available
2. Call normalization functions only when iterating over collections
3. Handle null returns from normalization (unrecognized formats) by not rendering

Architecture #

┌─────────────────────────────────────────────────────────────┐
│                        Firehose                              │
│  (pub.leaflet.* and site.standard.* records)                │
└─────────────────────┬───────────────────────────────────────┘
                      │
                      ▼
┌─────────────────────────────────────────────────────────────┐
│                    appview/index.ts                          │
│  Writes both formats to same tables (documents,             │
│  publications, publication_subscriptions)                    │
└─────────────────────┬───────────────────────────────────────┘
                      │
                      ▼
┌─────────────────────────────────────────────────────────────┐
│                   Database Tables                            │
│  documents.data, publications.record contain raw JSON       │
│  with $type field indicating format                         │
└─────────────────────┬───────────────────────────────────────┘
                      │
                      ▼
┌─────────────────────────────────────────────────────────────┐
│              Data Fetchers / Contexts                        │
│  Call normalizeDocumentRecord() / normalizePublicationRecord()│
│  Return pre-normalized data to consumers                     │
└─────────────────────┬───────────────────────────────────────┘
                      │
                      ▼
┌─────────────────────────────────────────────────────────────┐
│                    Components                                │
│  Use NormalizedDocument / NormalizedPublication types       │
│  No direct casts to pub.leaflet types                       │
└─────────────────────────────────────────────────────────────┘

Files Modified #

Core Infrastructure #

• appview/index.ts - Firehose consumer writes site.standard to main tables
• src/utils/normalizeRecords.ts - New normalization utilities
• lexicons/src/normalize.ts - Core normalization functions

Data Providers #

• components/PageSWRDataProvider.tsx - useLeafletPublicationData() returns normalized data
• app/lish/[did]/[publication]/dashboard/PublicationSWRProvider.tsx - Added useNormalizedPublicationRecord()
• app/lish/[did]/[publication]/[rkey]/getPostPageData.ts - Returns normalized data

Server-Side Data Fetchers #

• app/api/rpc/[command]/get_profile_data.ts
• app/(home-pages)/reader/getSubscriptions.ts
• app/(home-pages)/discover/getPublications.ts
• app/(home-pages)/reader/getReaderFeed.ts
• app/(home-pages)/p/[didOrHandle]/getProfilePosts.ts
• app/(home-pages)/tag/[tag]/getDocumentsByTag.ts

Notification System #

• src/notifications.ts - All hydration functions return normalized fields
• components/Notifications/*.tsx - Use pre-normalized data from props

Various Components #

• Removed casts to PubLeafletDocument.Record and PubLeafletPublication.Record
• Updated to use NormalizedDocument and NormalizedPublication types

Handling Unrecognized Formats #

When normalizeDocumentRecord() or normalizePublicationRecord() encounters a record that doesn't match either pub.leaflet.* or site.standard.* formats, it returns null. Components should handle this by not rendering:

const normalized = normalizeDocumentRecord(data);
if (!normalized) return null; // Don't render unrecognized formats

Testing #

TypeScript compilation passes without errors. The normalization functions include type guards to safely handle both record formats.