Fly.io Deployment Guide #

This guide covers deploying your application with worker processes to Fly.io using Redis for distributed event processing.

Current Setup Status ✅ #

Your application is already configured for multi-process deployment:

• fly.toml: Configured with web and feed-worker processes
• package.json: Has the worker:feeds script
• tsup.config.ts: Builds both web and worker entry points
• Dockerfile: Builds all artifacts needed for both processes

Prerequisites #

1. Redis Setup #

Create and attach a Redis database to your app:

# Create Redis database
fly redis create --name annos-redis

# Attach it to your app (automatically sets REDIS_URL environment variable)
fly redis attach annos-redis

# Verify Redis status
fly redis status annos-redis

2. Verify Dependencies #

Ensure worker dependencies are in production dependencies (not devDependencies):

{
  "dependencies": {
    "bullmq": "^5.56.8",
    "ioredis": "^5.6.1"
  }
}

Deployment Process #

1. Deploy the Application #

# Deploy both web and worker processes
fly deploy

# This creates:
# - At least one Machine for the 'web' process
# - At least one Machine for the 'feed-worker' process

2. Verify Deployment #

# Check that both processes are running
fly status

# Should show something like:
# Machines
# PROCESS  ID            VERSION  REGION  STATE   ROLE  CHECKS  LAST UPDATED
# web      e2865641be97  v1       sea     started       ✓       2m ago
# feed-worker 1781973f03  v1       sea     started       ✓       2m ago

3. Monitor Worker Startup #

# Check worker logs for successful startup
fly logs --process feed-worker

# Look for these success messages:
# "Connected to Redis successfully"
# "Feed worker started and listening for events..."

Scaling Workers #

Horizontal Scaling (More Machines) #

# Scale worker processes independently
fly scale count feed-worker=2

# Scale by region
fly scale count feed-worker=2 --region sea,ord

# Scale both web and workers
fly scale count web=3 feed-worker=2

Vertical Scaling (More Resources) #

# Scale worker memory
fly scale memory 1gb --process-group feed-worker

# Scale worker CPU/RAM preset
fly scale vm performance-2x --process-group feed-worker

Troubleshooting #

1. Worker Not Starting #

# Check worker logs for errors
fly logs --process feed-worker

# Common issues:
# - Missing REDIS_URL environment variable
# - Redis connection failed
# - Missing dependencies

2. Redis Connection Issues #

# Test Redis connection from worker
fly ssh console --process feed-worker
node -e "console.log('Redis URL:', process.env.REDIS_URL)"
node -e "const Redis = require('ioredis'); const r = new Redis(process.env.REDIS_URL); r.ping().then(console.log)"

3. Worker Not Processing Jobs #

# Check if jobs are being queued
fly redis connect annos-redis
> KEYS bull:*
> LLEN bull:feeds:waiting

# Monitor worker processing
fly logs --process feed-worker --follow

Environment Variables #

Both web and worker processes share the same environment variables automatically. Key variables:

• REDIS_URL - Set automatically by fly redis attach
• DATABASE_URL - Your database connection
• NODE_ENV - Set to "dev" in your fly.toml

Process Communication #

• Web Process: Publishes events to Redis queues
• Worker Process: Consumes events from Redis queues
• Redis: Acts as the message broker between processes

Monitoring #

View Process Status #

# List all machines by process
fly ps

# Get detailed app status
fly status

Monitor Logs #

# Web process logs
fly logs --process web

# Worker process logs
fly logs --process feed-worker

# All processes
fly logs

Resource Usage #

# Check resource metrics
fly metrics --process feed-worker
fly metrics --process web

Development vs Production #

Local Development #

For local development with Redis tunnel:

# Create tunnel to Redis
fly redis connect annos-redis

# Use tunnel address in your local environment
# Example: redis://localhost:10000

Production #

In production, processes automatically use the REDIS_URL environment variable set by Fly.io.

Key Points #

1. No Dockerfile Changes Needed - Your current Dockerfile works for both processes
2. Shared Environment - All processes share the same environment variables
3. Independent Scaling - Scale web and worker processes separately
4. Automatic Deployment - fly deploy handles both process types
5. Redis Integration - Use Fly's native Redis integration for best performance

Quick Reference Commands #

# Deploy
fly deploy

# Check status
fly status
fly ps

# Scale workers
fly scale count feed-worker=2

# Monitor
fly logs --process feed-worker --follow

# Debug
fly ssh console --process feed-worker
fly redis connect annos-redis