Our Personal Data Server from scratch! tranquil.farm
pds rust database fun oauth atproto

importRepo fails with a large repo (~1.6GB CAR) on Postgres #807

Open suibari.com
Open opened by suibari.com

I tried importing a large repo with com.atproto.repo.importRepo, and it fails with HTTP 500 after a few seconds. The account has about 2.78M records. The CAR from getRepo is 1.59GB with 3,566,094 blocks.

PDS log:

INFO  tranquil_api::repo::import: Importing repo for user did:plc:…: 3566094 blocks, root bafyrei…
ERROR tranquil_api::repo::import: Failed to store import blocks: RepoError { kind: Storage, source: Some(Io(Os { code: 104, kind: ConnectionReset, message: "Connection reset by peer" })), help: None, context: None }

Postgres log at the same moment:

LOG:  invalid message length

I had already raised import.max_size and import.max_blocks above the size of this repo.

Environment: tranquil-pds v0.7.1 built from the Dockerfile on x86_64, postgres:18-alpine with default settings, and a local PLC. This is a test environment, so skip_verification = true.

I can re-test or share more logs if that helps.

thanks for filing! Is this carfile on an existing account that I could also test with?

oh THE bot-tan lol

thanks! i'm working on some other stuff right now but afterwards I will take a look at this if nobody else gets to it first, I forgot if it's an arbitrary limit I set myself out of precaution or whether it's a proper tripwire that ought to be raised somewhat

Sign up or Sign in to add to the discussion.