Disaster Recovery Guide

Recovery Steps

1. 1Assess the nature of the database failure (corruption, data loss, hardware failure)
2. 2Stop all services connected to the database to prevent further writes
3. 3Identify the most recent valid backup or WAL position for recovery
4. 4Restore from the latest backup using pg_restore or point-in-time recovery
5. 5Validate data integrity after restoration
6. 6Restart dependent services and verify functionality

Commands

# Stop services
docker-compose stop api scanner

# Restore PostgreSQL from backup
pg_restore -h localhost -U postgres -d mcpsafe \
  --clean --if-exists backup_latest.dump

# For point-in-time recovery (if WAL archiving is enabled)
# Edit postgresql.conf with recovery settings
recovery_target_time = '2024-01-15 10:30:00'

# Verify data integrity
psql -h localhost -U postgres -d mcpsafe \
  -c "SELECT COUNT(*) FROM servers;"

# Restart services
docker-compose up -d api scanner

Recovery Steps

1. 1Identify the Redis failure type (memory, persistence, connectivity)
2. 2Check Redis AOF and RDB persistence files for recovery options
3. 3Restart Redis container or restore from persistence files
4. 4Clear stale cache entries if needed
5. 5Verify job queue functionality
6. 6Monitor for any queued jobs that need reprocessing

Commands

# Check Redis health
docker exec mcpsafe-redis redis-cli ping

# Restart Redis container
docker-compose restart redis

# If data recovery needed from AOF
docker exec mcpsafe-redis redis-cli BGREWRITEAOF

# Clear all cache (if starting fresh)
docker exec mcpsafe-redis redis-cli FLUSHALL

# Verify Redis connectivity
docker exec mcpsafe-redis redis-cli INFO replication

Recovery Steps

1. 1Check scanner container logs for error diagnosis
2. 2Verify Redis connectivity (scanner depends on Redis for job queue)
3. 3Restart the scanner container
4. 4Verify health endpoint is responding
5. 5Test a sample scan to confirm functionality
6. 6Check for any queued scans that need reprocessing

Commands

# Check scanner logs
docker logs mcpsafe-scanner --tail 100

# Verify Redis connection
docker exec mcpsafe-scanner curl -f http://localhost:8001/health

# Restart scanner
docker-compose restart scanner

# Test health endpoint
curl http://localhost:8001/health

# Test a sample scan
curl -X POST http://localhost:8001/scan \
  -H "Content-Type: application/json" \
  -d '{"url": "https://github.com/test/repo"}'

Recovery Steps

1. 1Assess the scope of the failure and affected components
2. 2Provision new infrastructure if hardware failure occurred
3. 3Restore PostgreSQL database from latest backup
4. 4Restore Redis data or accept cache rebuild
5. 5Deploy application containers with verified images
6. 6Run integration tests to verify system functionality
7. 7Restore DNS/load balancer configuration if affected
8. 8Notify stakeholders of recovery completion

Commands

# Pull latest verified images
docker-compose pull

# Restore database first (most critical)
pg_restore -h localhost -U postgres -d mcpsafe \
  --clean --if-exists /backups/latest.dump

# Start all services
docker-compose up -d

# Wait for health checks
sleep 30

# Verify all services are healthy
docker-compose ps

# Run smoke tests
curl http://localhost:3000/health
curl http://localhost:3001/health
curl http://localhost:8001/health

# Check database connectivity
docker exec mcpsafe-postgres pg_isready -U postgres

Symptoms

• !Slow query responses
• !Connection pool exhaustion
• !High CPU usage

Mitigation Actions

• Enable read replicas if available
• Implement connection pooling (PgBouncer)
• Temporarily disable non-critical features
• Scale up database resources

Symptoms

• !Scan requests timing out
• !Health checks failing
• !High error rates

Mitigation Actions

• Route traffic to backup scanner instance
• Queue incoming scans for later processing
• Display maintenance message to users
• Investigate and restart failed containers

Symptoms

• !Cache misses
• !Job queue not processing
• !Session issues

Mitigation Actions

• Fall back to direct database queries
• Queue jobs in memory temporarily
• Restart Redis with persistence recovery
• Clear and rebuild cache if necessary

Symptoms

• !Intermittent connectivity
• !Service-to-service failures
• !DNS issues

Mitigation Actions

• Identify affected network segments
• Fail over to healthy availability zones
• Enable circuit breakers for failing services
• Communicate with cloud provider if infrastructure issue