hydrant
very fast at protocol indexer with flexible filtering, xrpc queries, cursor-backed event stream, and more, built on fjall
rust fjall at-protocol atproto indexer
Star
Fork
Watch

Commits on test-13-fix
wip
committed
[tests] use plc.klbr.net instead of plc.wtf
committed
[ingest] change the unbounded channels to be bounded
committed
[backfill] dont hold store / mst tree over db commit, drop early when we can
committed
[lib] expose backlinks properly, add dids field support
committed
[ingest,stream] dont read from db, use live event
committed
[db] compact manually if we pruned enough events
committed
[deps] update fjall and friends
committed
[docs] update verbiage
committed
[db] make count tracking more durable
committed
[backfill] fix count tracking on resync
committed
[docs] point to docs
committed
[flake] remove conflicting rust deps from devshell override
committed
[docs] update verbiage
committed
[docs] move title headers into title frontmatter
committed
[docs] fix rewrites
committed
[db] improve ephemeral task, add test for the relay ttl to the gc test
committed
[db] print pruned events log at info level
committed
[docs] update links in docs readme
committed
[docs] add building and deployment
committed
[docs] add request body tables, restore config descriptions from readme
committed
[docs] convert nested lists to headings in api docs
committed
[flake] set VERBIAGE_TITLE=hydrant, update verbiage input
committed
[ingest] store resync buffer commits with named tags
committed
[docs] update verbiage to latest (redirect fix)
committed
[docs] initial documentation + add verbiage to devshell
committed
[firehose] periodically retry offline hosts (every 30 minutes)
committed
[firehose,lib,api] add is_source_running, shortcircuit requestCrawl if source is already running to prevent task restart
committed
[lib,api] add new host limit for hosts added via requestCrawl
committed
[config] enable ephemeral mode by default in relay mode
committed
[docs] remove gitnexus stuff from agents.md
committed
[all] add indexer_stream feature to decouple the events stream from the indexer
committed
[all] simplify code depending on block storage being enabled
committed
[all] dont thread ephemeral bool through the code, use from app state
committed
[indexer] add config option for only indexing links
committed
[ingest,lib,api] add glob rules for pds tiers, TRUSTED_HOSTS -> TIER_RULES
committed
[api] make sure cids are serialized as text in listRepos
committed
[api] fix index page to show indexer if compiled with indexer
committed
[api] also add _health
committed
[firehose] jitter connection startup when we are spawning persisted / configged firehoses
committed
[firehose] use cancel token instead of aborting task, fix bug where upserted tasks would remove the new task from the map
committed
[ingest] make throttling less strict on firehose errors to better match ref behaviour
committed
[lib,api] fix iterating through repos and getting repo info in relay mode
committed
[docs] remove unnecessary method from safe to proxy endpoints
committed
[api] add index file
committed
[build] add rust toolchain toml and update flake to use nci for devshell
committed
[lib] re-export public api deps in deps
committed
[build] add dockerfile
committed
[docs] better instructions for hosting in the readme
committed
[ingest,lib,api] implement host status tracking for pds' and enforce account limit
committed
[docs] update readme
committed
[api] actually reject banned pds in requestCrawl
committed
[firehose] only reset backoff to 0 if the stream has been healthy for over a minute
committed
[firehose] send pings to hosts
committed
[firehose] add jitter on throttles / backoffs
committed
[firehose] print better stream closed errors
committed
[firehose] treat every upgrade error as throttle reason
committed
[firehose] 4xx is also throttle worthy now, also handle rustls AlertReceived better
committed
[firehose,crawler] treat 1xx,2xx,3xx,5xx status codes as server error (aside from crawler, where 500 and 2xx is allowed)
committed
[firehose] respond to pings, better treat close messages
committed