Production Deployment Best Practices: Zero-Downtime Updates

Deploying code changes to production doesn't have to be scary. With the right workflow, you can update your application with minimal downtime and zero data loss.

The Golden Rule

Never reset the database in production unless absolutely necessary.

Code changes, bug fixes, and even most feature additions can be deployed without touching user data.

Understanding Different Types of Changes

Not all changes are created equal. The deployment strategy depends on what you're changing.

Frontend Changes (UI, Components, Styling)

Examples: Bug fixes in React components, CSS updates, new UI features

git pull origin main
docker compose -f docker-compose.prod.yml build frontend
docker compose -f docker-compose.prod.yml up -d frontend

Downtime: ~5 seconds Data Impact: None

Backend Code Changes (API Logic, Bug Fixes)

Examples: Business logic updates, API modifications, security patches

git pull origin main
docker compose -f docker-compose.prod.yml build backend
docker compose -f docker-compose.prod.yml up -d backend celery_worker celery_beat

Downtime: ~10-30 seconds Data Impact: None

Database Schema Changes (Migrations)

Examples: Adding tables, adding/removing columns, indexes

git pull origin main
docker compose -f docker-compose.prod.yml build backend
docker compose -f docker-compose.prod.yml exec backend flask db upgrade
docker compose -f docker-compose.prod.yml up -d backend

Critical: Always use migrations (flask db upgrade), NEVER use flask db-commands create in production.

The Complete Deployment Workflow

Phase 1: Pre-Deployment

# 1. Backup the database (ALWAYS!)
docker compose -f docker-compose.prod.yml exec postgres \
  pg_dump -U your_user your_db > backup_$(date +%Y%m%d_%H%M%S).sql

# 2. Check current state
docker compose -f docker-compose.prod.yml ps

# 3. Note current commit
git log -1 --oneline

Phase 2: Deployment

# 1. Pull latest code
cd /opt/maxi
git pull origin main

# 2. Rebuild changed services
docker compose -f docker-compose.prod.yml build

# 3. Apply migrations (if database schema changed)
docker compose -f docker-compose.prod.yml exec backend flask db upgrade

# 4. Restart services
docker compose -f docker-compose.prod.yml up -d

Phase 3: Post-Deployment Verification

# 1. Check all containers are running
docker compose -f docker-compose.prod.yml ps

# 2. Monitor logs for errors
docker compose -f docker-compose.prod.yml logs -f backend | grep -i error

# 3. Test critical paths
curl http://your-server-ip:7005/api/v1/health

Phase 4: Rollback (If Needed)

# Code rollback
git log --oneline
git reset --hard abc1234
docker compose -f docker-compose.prod.yml build
docker compose -f docker-compose.prod.yml up -d

# Database rollback
cat backup_20260125_193000.sql | \
  docker compose -f docker-compose.prod.yml exec -T postgres \
  psql -U your_user your_db

Common Mistakes to Avoid

Running db-commands create in Production

# NEVER DO THIS IN PRODUCTION!
docker compose exec backend flask db-commands create

This drops all tables and recreates them = DATA LOSS!

Not Backing Up Before Changes

Always backup first:

pg_dump -U user db > backup.sql
git pull
docker compose up -d --build

Restarting Everything for Small Changes

Better: Rebuild only what changed

docker compose build frontend
docker compose up -d frontend

Not Checking Logs After Deployment

# Monitor for issues
docker compose up -d
docker compose logs -f | grep -i error

Advanced: Zero-Downtime Deployments

Blue-Green Deployment

Run two identical environments. Deploy to the inactive one, then switch traffic.

# docker-compose.blue.yml
services:
  backend:
    container_name: backend_blue
    ports:
      - "5001:5000"

# docker-compose.green.yml
services:
  backend:
    container_name: backend_green
    ports:
      - "5002:5000"

Rolling Updates

services:
  backend:
    deploy:
      replicas: 3
      update_config:
        parallelism: 1
        delay: 10s

Database Migrations Without Downtime

Use backward-compatible migrations:

# Step 1: Add new column (nullable)
def upgrade():
    op.add_column('users', sa.Column('phone', sa.String(20), nullable=True))

# Deploy code that uses phone field optionally
# Step 2: Later, make it required
def upgrade():
    op.alter_column('users', 'phone', nullable=False)

Monitoring and Alerts

Basic Health Checks

#!/bin/bash
HEALTH=$(curl -s http://localhost:5000/api/v1/health | jq -r '.status')
if [ "$HEALTH" != "healthy" ]; then
  echo "Backend unhealthy!" | mail -s "Alert: Backend Down" admin@example.com
fi

Docker Health Checks

services:
  backend:
    healthcheck:
      test: ["CMD", "curl", "-f", "http://localhost:5000/api/v1/health"]
      interval: 30s
      timeout: 10s
      retries: 3

Key Principles

1. Always backup before making changes
2. Use migrations for database schema changes
3. Rebuild only what changed for faster deployments
4. Monitor logs after every deployment
5. Have a rollback plan before you start
6. Test critical paths after deployment
7. Never use db-commands create in production

Quick Reference

Change Type	What to Rebuild	Downtime
Frontend	frontend	~5s
Backend	backend + workers	~30s
Schema	migrations + backend	varies
Env vars	restart affected services	~10s

---

This builds muscle memory for real production deployments. Practice with small changes first, and always have a rollback plan ready.