r2-glacier-migration

R2 → AWS Glacier Deep Archive Migration

Migrate ~42.6 TB from Cloudflare R2 to AWS S3 Glacier Deep Archive.

Cost Savings

• R2: ~$638/month → Glacier: ~$42/month
• Savings: ~$596/month

Buckets

Quick Commands

Check Status

# Is transfer running?
ssh ninon "screen -ls"

# View recent logs
ssh ninon "tail -50 ~/r2-glacier/transfer-text.log"

# Check checkpoint
ssh ninon "cat /tmp/r2-glacier-pollinations-text-checkpoint.json | jq ."

Attach to Screen (Interactive)

ssh -t ninon "screen -r r2-text"
# Detach: Ctrl+A, D

Resume if Crashed

ssh ninon "screen -dmS r2-text bash -c '~/r2-glacier/run-text-transfer.sh; exec bash'"

File Locations

Script Usage

# Test (small batch)
node r2-to-glacier-streaming.js -b pollinations-text --batch-size 100 --max-batches 1

# Full transfer
node r2-to-glacier-streaming.js -b pollinations-text --batch-size 50000 --concurrency 30

# Resume after interruption
node r2-to-glacier-streaming.js -b pollinations-text --resume

# Dry run (count only)
node r2-to-glacier-streaming.js -b pollinations-text --dry-run

AWS Setup

• Bucket: s3://pollinations-archive
• Region: us-east-1
• Storage Class: DEEP_ARCHIVE

Verify Archives

aws s3 ls s3://pollinations-archive/ --recursive --human-readable | head -20

Important Notes

• ⚠️ 180-day minimum on Glacier Deep Archive - don't delete R2 until verified!
• Some objects fail with "Header overflow" (very large responses) - acceptable loss
• Text bucket (1.86B objects) takes several days
• R2 egress is free, AWS ingress is free

Start Images Bucket (After Text Complete)

1. Update ~/r2-glacier/run-text-transfer.sh on ninon:

   # Change bucket name
   node r2-to-glacier-streaming.js -b pollinations-images --batch-size 50000 --concurrency 30
   

2. Start new screen:

   ssh ninon "screen -dmS r2-images bash -c '~/r2-glacier/run-images-transfer.sh; exec bash'"
   

Tracking

• Issue: #5860
• PR: #5861