Docker Compose for Production

Introduction to Docker Compose in Production

In our previous lectures, we've explored how to optimize Docker for production environments and leveraged multi-stage builds for efficient image creation. Now, we'll focus on Docker Compose, a tool you might already be familiar with from development environments, but with a specific focus on production use cases.

While Docker Compose is often associated with development environments, it can be a powerful tool for deploying and managing multi-container applications in production, especially for smaller-scale deployments or environments where Kubernetes might be overkill.

flowchart LR A[Development Compose] --> B[Production Compose] A --> A1[Local volumes] A --> A2[Hot reloading] A --> A3[Debug mode] A --> A4[Development dependencies] B --> B1[Named volumes or external storage] B --> B2[Optimized performance] B --> B3[Health checks & restart policies] B --> B4[Resource constraints]

Think of Docker Compose as the conductor of an orchestra—it coordinates multiple containers to work together seamlessly. In development, the conductor might allow for improvisation and experimentation, but in production, every note must be precisely planned and executed with reliability in mind.

Docker Compose Evolution

From Development to Production

Let's examine how Docker Compose configuration evolves as we move from development to production:

# Development docker-compose.yml
version: '3.8'

services:
  web:
    build: ./frontend
    ports:
      - "3000:3000"
    volumes:
      - ./frontend:/app
      - /app/node_modules
    environment:
      - NODE_ENV=development
    command: npm run dev

  api:
    build: ./backend
    ports:
      - "4000:4000"
    volumes:
      - ./backend:/app
      - /app/node_modules
    environment:
      - NODE_ENV=development
      - DATABASE_URL=postgres://postgres:password@db:5432/devdb
    command: npm run dev
    depends_on:
      - db

  db:
    image: postgres:14
    ports:
      - "5432:5432"
    environment:
      - POSTGRES_PASSWORD=password
      - POSTGRES_DB=devdb
    volumes:
      - pgdata:/var/lib/postgresql/data

volumes:
  pgdata:

This development configuration focuses on convenience and rapid iteration with features like:

Code volumes for hot reloading
Exposed ports for easy debugging
Development-specific commands
Simple environment variables with plain-text passwords

Now let's see how this might evolve for production:

# Production docker-compose.yml
version: '3.8'

services:
  web:
    image: ${REGISTRY}/frontend:${TAG}
    deploy:
      replicas: 2
      resources:
        limits:
          cpus: '0.5'
          memory: 512M
      restart_policy:
        condition: any
        max_attempts: 3
    healthcheck:
      test: ["CMD", "wget", "--spider", "-q", "http://localhost:3000/health"]
      interval: 30s
      timeout: 5s
      retries: 3
      start_period: 10s
    networks:
      - frontend
    ports:
      - "80:3000"
    environment:
      - NODE_ENV=production
      - API_URL=http://api:4000

  api:
    image: ${REGISTRY}/backend:${TAG}
    deploy:
      resources:
        limits:
          cpus: '1.0'
          memory: 1G
      restart_policy:
        condition: any
    healthcheck:
      test: ["CMD", "wget", "--spider", "-q", "http://localhost:4000/health"]
      interval: 30s
      timeout: 5s
      retries: 3
    networks:
      - frontend
      - backend
    environment:
      - NODE_ENV=production
      - DATABASE_URL=postgres://appuser:${DB_PASSWORD}@db:5432/proddb
    depends_on:
      db:
        condition: service_healthy
    secrets:
      - db_password

  db:
    image: postgres:14-alpine
    deploy:
      resources:
        limits:
          cpus: '2.0'
          memory: 2G
      restart_policy:
        condition: any
    healthcheck:
      test: ["CMD-SHELL", "pg_isready -U appuser"]
      interval: 10s
      timeout: 5s
      retries: 5
    volumes:
      - db_data:/var/lib/postgresql/data
    networks:
      - backend
    environment:
      - POSTGRES_USER=appuser
      - POSTGRES_PASSWORD_FILE=/run/secrets/db_password
      - POSTGRES_DB=proddb
    secrets:
      - db_password

networks:
  frontend:
  backend:
    internal: true

volumes:
  db_data:
    driver: local
    driver_opts:
      type: 'none'
      o: 'bind'
      device: '/mnt/data/postgres'

secrets:
  db_password:
    file: ./secrets/db_password.txt

The production configuration prioritizes reliability, security, and performance with features like:

Pre-built images with specific tags
Resource constraints and replica settings
Health checks and restart policies
Network segmentation
Proper secrets management
External volume configuration

Production Composition Patterns

Environment-Specific Compose Files

A common approach is to use multiple compose files for different environments:

# Base configuration common to all environments
docker-compose.yml

# Environment overrides
docker-compose.dev.yml
docker-compose.staging.yml
docker-compose.prod.yml

# To deploy to production
docker compose -f docker-compose.yml -f docker-compose.prod.yml up -d

This pattern allows you to:

Share common configuration across environments
Override specific settings per environment
Keep the base configuration simple and readable

For example, a common base with environment-specific overrides:

# docker-compose.yml (base)
version: '3.8'

services:
  web:
    image: ${REGISTRY}/frontend:${TAG}
    networks:
      - frontend

  api:
    image: ${REGISTRY}/backend:${TAG}
    networks:
      - frontend
      - backend
    depends_on:
      - db

  db:
    image: postgres:14
    networks:
      - backend
    volumes:
      - db_data:/var/lib/postgresql/data

networks:
  frontend:
  backend:
    internal: true

volumes:
  db_data:

# docker-compose.prod.yml (production overrides)
version: '3.8'

services:
  web:
    deploy:
      replicas: 2
      resources:
        limits:
          cpus: '0.5'
          memory: 512M
    healthcheck:
      test: ["CMD", "wget", "--spider", "-q", "http://localhost:3000/health"]
      interval: 30s
      timeout: 5s
      retries: 3
    environment:
      - NODE_ENV=production

  api:
    deploy:
      resources:
        limits:
          cpus: '1.0'
          memory: 1G
    healthcheck:
      test: ["CMD", "wget", "--spider", "-q", "http://localhost:4000/health"]
      interval: 30s
      timeout: 5s
      retries: 3
    environment:
      - NODE_ENV=production
      - DATABASE_URL=postgres://appuser:${DB_PASSWORD}@db:5432/proddb
    secrets:
      - db_password

  db:
    image: postgres:14-alpine
    deploy:
      resources:
        limits:
          cpus: '2.0'
          memory: 2G
    environment:
      - POSTGRES_USER=appuser
      - POSTGRES_PASSWORD_FILE=/run/secrets/db_password
      - POSTGRES_DB=proddb
    secrets:
      - db_password

volumes:
  db_data:
    driver: local
    driver_opts:
      type: 'none'
      o: 'bind'
      device: '/mnt/data/postgres'

secrets:
  db_password:
    file: ./secrets/db_password.txt

Service-Specific Composition

For larger applications, you might break down your compose files by service or component:

flowchart LR A[docker-compose.yml] --> B[base services] C[docker-compose.frontend.yml] --> D[frontend services] E[docker-compose.backend.yml] --> F[backend services] G[docker-compose.db.yml] --> H[database services] I[docker-compose.monitoring.yml] --> J[monitoring services]

# Deploy the complete stack
docker compose -f docker-compose.yml \
  -f docker-compose.frontend.yml \
  -f docker-compose.backend.yml \
  -f docker-compose.db.yml \
  -f docker-compose.monitoring.yml \
  up -d

# Deploy only the backend and database
docker compose -f docker-compose.yml \
  -f docker-compose.backend.yml \
  -f docker-compose.db.yml \
  up -d

Compose Profiles

Newer versions of Docker Compose support profiles for conditionally enabling services:

# Using profiles in docker-compose.yml
services:
  web:
    image: ${REGISTRY}/frontend:${TAG}
    # Always starts

  api:
    image: ${REGISTRY}/backend:${TAG}
    # Always starts

  db:
    image: postgres:14-alpine
    # Always starts

  prometheus:
    image: prom/prometheus:latest
    profiles:
      - monitoring

  grafana:
    image: grafana/grafana:latest
    profiles:
      - monitoring

  elasticsearch:
    image: elasticsearch:7.17.0
    profiles:
      - logging

  kibana:
    image: kibana:7.17.0
    profiles:
      - logging

# Start the core application
docker compose up -d

# Start with monitoring
docker compose --profile monitoring up -d

# Start with logging
docker compose --profile logging up -d

# Start with both monitoring and logging
docker compose --profile monitoring --profile logging up -d

Production-Critical Features

Health Checks

Health checks verify that your services are functioning correctly:

services:
  api:
    image: api-service:latest
    healthcheck:
      test: ["CMD", "curl", "-f", "http://localhost:4000/health"]
      interval: 30s
      timeout: 5s
      retries: 3
      start_period: 40s
    depends_on:
      db:
        condition: service_healthy

  db:
    image: postgres:14-alpine
    healthcheck:
      test: ["CMD-SHELL", "pg_isready -U postgres"]
      interval: 10s
      timeout: 5s
      retries: 5

Benefits of proper health checks:

Dependency resolution: Services only start when dependencies are healthy
Automatic recovery: Unhealthy containers can be restarted
Load balancer integration: Unhealthy instances can be removed from rotation
Monitoring integration: Health status can be tracked and alerted on

Resource Constraints

Setting appropriate resource limits prevents resource contention:

services:
  web:
    image: web-app:latest
    deploy:
      resources:
        limits:
          cpus: '0.5'
          memory: 512M
        reservations:
          cpus: '0.25'
          memory: 256M

Best practices for resource management:

Set both limits and reservations: Guarantees minimum resources while preventing overuse
Monitor actual usage: Adjust based on observed patterns
Consider container density: Balance resource allocation across hosts
Allow for spikes: Don't set limits too close to average usage

Restart Policies

Properly configured restart policies ensure service reliability:

services:
  worker:
    image: worker-service:latest
    deploy:
      restart_policy:
        condition: any  # always, on-failure, none
        delay: 5s
        max_attempts: 3
        window: 120s

Options to consider:

condition: When to restart (any, on-failure, none)
delay: Time between restart attempts
max_attempts: Maximum number of restarts before giving up
window: Time to wait before considering the restart successful

Secrets Management

Proper secrets handling in production is critical for security:

services:
  api:
    image: api-service:latest
    environment:
      - DB_USER=appuser
      - DB_PASSWORD_FILE=/run/secrets/db_password
    secrets:
      - db_password

secrets:
  db_password:
    file: ./secrets/db_password.txt
    # or in swarm mode:
    # external: true

Secrets management approaches:

File-based secrets: Simple but requires secure file management
External secrets: Managed by Docker Swarm or a secrets management service
Environment variables file: Using .env files (less secure but simple)
Third-party solutions: HashiCorp Vault, AWS Secrets Manager, etc.

Network Configuration

Proper network segmentation enhances security:

services:
  web:
    networks:
      - frontend

  api:
    networks:
      - frontend
      - backend

  db:
    networks:
      - backend

networks:
  frontend:
    # Public-facing network
  backend:
    # Internal network for services
    internal: true
    driver: overlay
    driver_opts:
      encrypted: "true"

graph TD A[Internet] --- B[Frontend Network] B --- C[Web Service] B --- D[API Service] D --- E[Backend Network] E --- F[Database Service] E --- G[Cache Service] style B fill:#f9f,stroke:#333,stroke-width:2px style E fill:#bbf,stroke:#333,stroke-width:2px

Network security principles:

Segmentation: Separate networks for different security zones
Least privilege: Connect services only to networks they need
Internal networks: Hide backend services from external access
Encryption: Encrypt traffic between services when necessary

Scaling and High Availability

Service Replication

Docker Compose with Swarm mode supports service replication:

services:
  web:
    image: web-app:latest
    deploy:
      mode: replicated
      replicas: 3
      placement:
        constraints:
          - node.role == worker
        preferences:
          - spread: node.labels.zone

graph TD A[Load Balancer] --> B[Web Replica 1] A --> C[Web Replica 2] A --> D[Web Replica 3] B & C & D --> E[API Service] E --> F[Database]

Load Balancing

Compose with Swarm provides built-in load balancing:

services:
  web:
    image: web-app:latest
    deploy:
      replicas: 3
    ports:
      - "80:80"  # Swarm automatically load balances across replicas

For more advanced load balancing, you might add a dedicated load balancer:

services:
  traefik:
    image: traefik:v2.9
    command:
      - "--providers.docker=true"
      - "--providers.docker.swarmMode=true"
      - "--entrypoints.web.address=:80"
    ports:
      - "80:80"
      - "8080:8080"
    volumes:
      - /var/run/docker.sock:/var/run/docker.sock:ro
    deploy:
      placement:
        constraints:
          - node.role == manager
      
  web:
    image: web-app:latest
    deploy:
      replicas: 3
    labels:
      - "traefik.enable=true"
      - "traefik.http.routers.web.rule=Host(`example.com`)"
      - "traefik.http.services.web.loadbalancer.server.port=80"

Data Persistence

Managing persistent data in a distributed environment:

services:
  db:
    image: postgres:14
    volumes:
      - db_data:/var/lib/postgresql/data
    deploy:
      placement:
        constraints:
          - node.labels.db == true

volumes:
  db_data:
    driver: local
    driver_opts:
      type: 'nfs'
      o: 'addr=10.10.10.10,nolock,soft,rw'
      device: ':/path/to/nfs/share'

Approaches to data persistence in production:

Local volumes: Simple but ties containers to specific hosts
Network file systems: NFS, SMB for shared storage
Volume plugins: Cloud providers (EBS, Azure Disk) or specialized solutions
Database clusters: Use clustered databases instead of single instances

Zero-Downtime Updates

Updating services without disrupting users:

services:
  web:
    image: web-app:${VERSION}
    deploy:
      replicas: 3
      update_config:
        parallelism: 1
        delay: 10s
        order: start-first
        failure_action: rollback
        monitor: 60s

# Update the service
VERSION=1.2.0 docker compose up -d

# For Swarm mode:
VERSION=1.2.0 docker stack deploy -c docker-compose.yml myapp

Key update configuration options:

parallelism: Number of containers to update simultaneously
delay: Time between updates
order: Update order (stop-first or start-first)
failure_action: What to do if an update fails
monitor: Time to monitor for update success

Monitoring and Logging

Container Monitoring

Integrating monitoring solutions with Compose:

# docker-compose.monitoring.yml
services:
  prometheus:
    image: prom/prometheus:latest
    volumes:
      - ./prometheus.yml:/etc/prometheus/prometheus.yml
      - prometheus_data:/prometheus
    ports:
      - "9090:9090"
    command:
      - '--config.file=/etc/prometheus/prometheus.yml'
      - '--storage.tsdb.path=/prometheus'
      - '--web.console.libraries=/etc/prometheus/console_libraries'
      - '--web.console.templates=/etc/prometheus/consoles'
      - '--web.enable-lifecycle'
    
  node-exporter:
    image: prom/node-exporter:latest
    volumes:
      - /proc:/host/proc:ro
      - /sys:/host/sys:ro
      - /:/rootfs:ro
    command:
      - '--path.procfs=/host/proc'
      - '--path.sysfs=/host/sys'
      - '--collector.filesystem.ignored-mount-points=^/(sys|proc|dev|host|etc)($$|/)'
    deploy:
      mode: global
    
  cadvisor:
    image: gcr.io/cadvisor/cadvisor:latest
    volumes:
      - /:/rootfs:ro
      - /var/run:/var/run:ro
      - /sys:/sys:ro
      - /var/lib/docker/:/var/lib/docker:ro
    ports:
      - "8080:8080"
    
  grafana:
    image: grafana/grafana:latest
    volumes:
      - grafana_data:/var/lib/grafana
    environment:
      - GF_SECURITY_ADMIN_PASSWORD=${GRAFANA_PASSWORD}
      - GF_USERS_ALLOW_SIGN_UP=false
    ports:
      - "3000:3000"
    depends_on:
      - prometheus

volumes:
  prometheus_data:
  grafana_data:

graph TD A[Application Containers] --> B[cAdvisor] C[Node] --> D[Node Exporter] B & D --> E[Prometheus] E --> F[Grafana] F --> G[Alerts] F --> H[Dashboards]

Centralized Logging

Setting up a log collection stack:

# docker-compose.logging.yml
services:
  elasticsearch:
    image: docker.elastic.co/elasticsearch/elasticsearch:7.17.0
    environment:
      - discovery.type=single-node
      - ES_JAVA_OPTS=-Xms512m -Xmx512m
    volumes:
      - elasticsearch_data:/usr/share/elasticsearch/data
    ports:
      - "9200:9200"
    deploy:
      resources:
        limits:
          memory: 1G
    
  logstash:
    image: docker.elastic.co/logstash/logstash:7.17.0
    volumes:
      - ./logstash/pipeline:/usr/share/logstash/pipeline
    ports:
      - "5000:5000"
    depends_on:
      - elasticsearch
    
  kibana:
    image: docker.elastic.co/kibana/kibana:7.17.0
    environment:
      - ELASTICSEARCH_URL=http://elasticsearch:9200
    ports:
      - "5601:5601"
    depends_on:
      - elasticsearch

  filebeat:
    image: docker.elastic.co/beats/filebeat:7.17.0
    volumes:
      - ./filebeat.yml:/usr/share/filebeat/filebeat.yml:ro
      - /var/lib/docker/containers:/var/lib/docker/containers:ro
      - /var/run/docker.sock:/var/run/docker.sock:ro
    user: root
    deploy:
      mode: global

volumes:
  elasticsearch_data:

graph TD A[Application Containers] --> B[Docker Log Driver] B --> C[Filebeat] C --> D[Logstash] D --> E[Elasticsearch] E --> F[Kibana]

Log Driver Configuration

Configuring Docker's logging behavior:

services:
  api:
    image: api-service:latest
    logging:
      driver: "json-file"
      options:
        max-size: "10m"
        max-file: "3"
        
  worker:
    image: worker-service:latest
    logging:
      driver: "fluentd"
      options:
        fluentd-address: "localhost:24224"
        tag: "docker.{{.Name}}"

Available log drivers include:

json-file: Default Docker logging driver
syslog: Send logs to syslog server
fluentd: Forward logs to a Fluentd collector
gelf: Graylog Extended Log Format
splunk: Send logs to Splunk Enterprise
awslogs: Amazon CloudWatch Logs

Beyond Compose: Orchestration Evolution

Docker Swarm Mode

Docker Compose files can be used directly with Docker Swarm for cluster deployment:

# Initialize a swarm
docker swarm init

# Deploy a stack from a compose file
docker stack deploy -c docker-compose.yml -c docker-compose.prod.yml myapp

# Scale a service
docker service scale myapp_web=5

# Update a service
docker service update --image web:v2 myapp_web

Benefits of Swarm mode:

Built-in orchestration with minimal additional complexity
Uses the same Compose file format with some extensions
Integrated service discovery and load balancing
Rolling updates and health checks
Secret management

Kubernetes Transition

For larger deployments, you might transition from Compose to Kubernetes:

# Convert compose file to Kubernetes manifests
kompose convert -f docker-compose.yml -o k8s-manifests/

# Apply Kubernetes manifests
kubectl apply -f k8s-manifests/

When to consider Kubernetes:

Complex microservice architectures
Large-scale deployments
Multi-region or multi-cloud deployments
Advanced auto-scaling requirements
Complex networking or security needs

Comparison of Orchestration Options

Feature	Docker Compose	Docker Swarm	Kubernetes
Complexity	Low	Medium	High
Scalability	Limited	Good	Excellent
Learning Curve	Shallow	Moderate	Steep
Auto-healing	Minimal	Yes	Yes
Load Balancing	External only	Built-in	Built-in
Rolling Updates	Manual	Yes	Yes
Health Checks	Yes	Yes	Yes
Resource Constraints	Yes	Yes	Yes
Service Discovery	Basic	Built-in	Built-in
Secrets Management	Basic	Yes	Yes
Community/Ecosystem	Large	Medium	Very Large

Practical Exercise: Production Compose Configuration

Exercise Brief

In this exercise, you'll convert a development-focused Compose configuration to a production-ready setup for a typical web application stack.

Starting Point: Development Compose

# Development docker-compose.yml
version: '3.8'

services:
  web:
    build: ./frontend
    ports:
      - "3000:3000"
    volumes:
      - ./frontend:/app
      - /app/node_modules
    environment:
      - NODE_ENV=development
      - REACT_APP_API_URL=http://localhost:4000
    command: npm start

  api:
    build: ./backend
    ports:
      - "4000:4000"
    volumes:
      - ./backend:/app
      - /app/node_modules
    environment:
      - NODE_ENV=development
      - PORT=4000
      - DATABASE_URL=mongodb://db:27017/devdb
    depends_on:
      - db
    command: npm run dev

  db:
    image: mongo:latest
    ports:
      - "27017:27017"
    volumes:
      - mongodb_data:/data/db

volumes:
  mongodb_data:

Your Task

Create a production version of this compose configuration that includes:

Proper image versioning strategy
Resource constraints for all services
Health checks for all services
Restart policies for reliability
Secrets management for sensitive information
Network segregation for security
Proper volume configuration for data persistence
Monitoring and logging integration

Solution Outline

Here's a sample solution you can use as a reference:

# Production docker-compose.yml
version: '3.8'

services:
  web:
    image: ${REGISTRY}/frontend:${TAG}
    deploy:
      replicas: 2
      resources:
        limits:
          cpus: '0.5'
          memory: 512M
        reservations:
          cpus: '0.1'
          memory: 128M
      restart_policy:
        condition: on-failure
        max_attempts: 3
        window: 120s
    healthcheck:
      test: ["CMD", "wget", "--spider", "-q", "http://localhost:3000/health"]
      interval: 30s
      timeout: 5s
      retries: 3
      start_period: 10s
    networks:
      - frontend
    ports:
      - "80:3000"
    environment:
      - NODE_ENV=production
      - REACT_APP_API_URL=http://api:4000

  api:
    image: ${REGISTRY}/backend:${TAG}
    deploy:
      resources:
        limits:
          cpus: '1.0'
          memory: 1G
        reservations:
          cpus: '0.2'
          memory: 256M
      restart_policy:
        condition: any
    healthcheck:
      test: ["CMD", "wget", "--spider", "-q", "http://localhost:4000/health"]
      interval: 30s
      timeout: 5s
      retries: 3
    networks:
      - frontend
      - backend
    environment:
      - NODE_ENV=production
      - PORT=4000
      - DATABASE_URL=mongodb://db:27017/proddb
      - DATABASE_USER=appuser
      - DATABASE_PASSWORD_FILE=/run/secrets/db_password
    depends_on:
      db:
        condition: service_healthy
    secrets:
      - db_password

  db:
    image: mongo:5.0-focal
    deploy:
      resources:
        limits:
          cpus: '2.0'
          memory: 2G
        reservations:
          cpus: '0.5'
          memory: 512M
      restart_policy:
        condition: any
    healthcheck:
      test: echo 'db.runCommand("ping").ok' | mongo localhost:27017/test --quiet
      interval: 10s
      timeout: 5s
      retries: 5
      start_period: 40s
    volumes:
      - mongodb_data:/data/db
    networks:
      - backend
    environment:
      - MONGO_INITDB_ROOT_USERNAME=appuser
      - MONGO_INITDB_ROOT_PASSWORD_FILE=/run/secrets/db_password
    secrets:
      - db_password

  prometheus:
    image: prom/prometheus:latest
    volumes:
      - ./prometheus.yml:/etc/prometheus/prometheus.yml
      - prometheus_data:/prometheus
    networks:
      - monitoring
      - frontend
      - backend
    deploy:
      resources:
        limits:
          cpus: '0.5'
          memory: 512M
    profiles:
      - monitoring

  grafana:
    image: grafana/grafana:latest
    volumes:
      - grafana_data:/var/lib/grafana
    environment:
      - GF_SECURITY_ADMIN_PASSWORD_FILE=/run/secrets/grafana_password
    ports:
      - "3001:3000"
    networks:
      - monitoring
    deploy:
      resources:
        limits:
          cpus: '0.5'
          memory: 512M
    depends_on:
      - prometheus
    secrets:
      - grafana_password
    profiles:
      - monitoring

networks:
  frontend:
  backend:
    internal: true
  monitoring:

volumes:
  mongodb_data:
    driver: local
    driver_opts:
      type: 'none'
      o: 'bind'
      device: '/mnt/data/mongodb'
  prometheus_data:
  grafana_data:

secrets:
  db_password:
    file: ./secrets/db_password.txt
  grafana_password:
    file: ./secrets/grafana_password.txt

Deployment Script Example

#!/bin/bash
# deploy.sh - Production deployment script

# Set environment variables
export REGISTRY="registry.example.com"
export TAG="$(git describe --tags --always)"

# Create secrets directory if it doesn't exist
mkdir -p secrets

# Generate random passwords if they don't exist
if [ ! -f secrets/db_password.txt ]; then
  openssl rand -base64 16 > secrets/db_password.txt
fi

if [ ! -f secrets/grafana_password.txt ]; then
  openssl rand -base64 16 > secrets/grafana_password.txt
fi

# Deploy the application
docker compose -f docker-compose.yml up -d

# Deploy monitoring stack if requested
if [ "$1" == "--with-monitoring" ]; then
  docker compose -f docker-compose.yml --profile monitoring up -d
fi

# Display deployed services
docker compose ps

Challenge Extension

Once you've completed the basic task, extend your solution to:

Create separate environment-specific compose files (staging, production)
Add a reverse proxy/load balancer (Traefik, Nginx)
Configure container log rotation and forwarding
Implement backup and restore procedures for databases
Create a deployment pipeline for automated updates

Docker Compose Production Best Practices

Configuration Management

Use environment variables for configuration
Separate secrets from configuration
Version control your compose files but not secrets
Document variables with a .env.example file
Use override files for environment-specific settings

Performance Optimization

Set appropriate resource limits for containers
Optimize image sizes with multi-stage builds
Use production-optimized base images (Alpine, distroless)
Implement caching strategies for applications
Monitor and tune performance regularly

Security Considerations

Never store secrets in compose files or images
Use network segmentation to isolate services
Run containers as non-root users
Limit container capabilities to minimum required
Regularly update base images and dependencies
Scan images for vulnerabilities before deployment

Operational Readiness

Implement comprehensive monitoring and alerting
Set up centralized logging for all services
Create backup and restore procedures for data
Document deployment procedures and runbooks
Test failover and recovery scenarios regularly
Implement CI/CD pipelines for automated deployment

Real-World Production Compose Example

Multi-Environment E-Commerce Platform

Let's look at how a medium-sized e-commerce company might structure their Docker Compose configuration for different environments:

graph TD A[Project Structure] --> B[docker-compose.yml] A --> C[docker-compose.dev.yml] A --> D[docker-compose.staging.yml] A --> E[docker-compose.prod.yml] A --> F[docker-compose.monitoring.yml] A --> G[environments/] G --> H[.env.dev] G --> I[.env.staging] G --> J[.env.prod] A --> K[secrets/] K --> L[dev/] K --> M[staging/] K --> N[prod/] A --> O[config/] O --> P[nginx/] O --> Q[prometheus/] O --> R[app/]

Base configuration (docker-compose.yml):

# Base configuration for all environments
version: '3.8'

services:
  nginx:
    image: ${REGISTRY}/nginx:${TAG}
    networks:
      - frontend
    depends_on:
      - web

  web:
    image: ${REGISTRY}/web:${TAG}
    networks:
      - frontend
      - backend
    depends_on:
      - api

  api:
    image: ${REGISTRY}/api:${TAG}
    networks:
      - backend
    depends_on:
      - db
      - redis

  worker:
    image: ${REGISTRY}/worker:${TAG}
    networks:
      - backend
    depends_on:
      - db
      - redis

  db:
    image: postgres:14-alpine
    networks:
      - backend
    volumes:
      - db_data:/var/lib/postgresql/data

  redis:
    image: redis:alpine
    networks:
      - backend
    volumes:
      - redis_data:/data

networks:
  frontend:
  backend:
    internal: true

volumes:
  db_data:
  redis_data:

Production overrides (docker-compose.prod.yml):

# Production-specific configurations
version: '3.8'

services:
  nginx:
    ports:
      - "80:80"
      - "443:443"
    volumes:
      - ./config/nginx/prod.conf:/etc/nginx/conf.d/default.conf
      - ./config/nginx/ssl:/etc/nginx/ssl
    deploy:
      replicas: 2
      resources:
        limits:
          cpus: '0.5'
          memory: 512M
      restart_policy:
        condition: any
    healthcheck:
      test: ["CMD", "curl", "-f", "https://localhost/health"]
      interval: 30s
      timeout: 5s
      retries: 3

  web:
    deploy:
      replicas: 4
      resources:
        limits:
          cpus: '1.0'
          memory: 1G
      restart_policy:
        condition: any
      update_config:
        parallelism: 1
        delay: 10s
        order: start-first
    healthcheck:
      test: ["CMD", "curl", "-f", "http://localhost:3000/health"]
      interval: 30s
      timeout: 5s
      retries: 3
    environment:
      - NODE_ENV=production
      - API_URL=http://api:4000

  api:
    deploy:
      replicas: 6
      resources:
        limits:
          cpus: '2.0'
          memory: 2G
      restart_policy:
        condition: any
      update_config:
        parallelism: 2
        delay: 10s
        order: start-first
    healthcheck:
      test: ["CMD", "curl", "-f", "http://localhost:4000/health"]
      interval: 30s
      timeout: 5s
      retries: 3
    environment:
      - NODE_ENV=production
      - DB_HOST=db
      - DB_USER=appuser
      - DB_PASSWORD_FILE=/run/secrets/db_password
      - REDIS_URL=redis://redis:6379
    secrets:
      - db_password

  worker:
    deploy:
      replicas: 3
      resources:
        limits:
          cpus: '1.0'
          memory: 1G
      restart_policy:
        condition: any
    environment:
      - NODE_ENV=production
      - DB_HOST=db
      - DB_USER=appuser
      - DB_PASSWORD_FILE=/run/secrets/db_password
      - REDIS_URL=redis://redis:6379
    secrets:
      - db_password

  db:
    deploy:
      resources:
        limits:
          cpus: '4.0'
          memory: 8G
      restart_policy:
        condition: any
      placement:
        constraints:
          - node.labels.db == true
    volumes:
      - db_data:/var/lib/postgresql/data
    environment:
      - POSTGRES_USER=appuser
      - POSTGRES_PASSWORD_FILE=/run/secrets/db_password
      - POSTGRES_DB=app
    secrets:
      - db_password
    healthcheck:
      test: ["CMD-SHELL", "pg_isready -U appuser"]
      interval: 10s
      timeout: 5s
      retries: 5

  redis:
    deploy:
      resources:
        limits:
          cpus: '2.0'
          memory: 4G
      restart_policy:
        condition: any
    healthcheck:
      test: ["CMD", "redis-cli", "ping"]
      interval: 10s
      timeout: 5s
      retries: 5

volumes:
  db_data:
    driver: local
    driver_opts:
      type: 'nfs'
      o: 'addr=10.10.10.10,nolock,soft,rw'
      device: ':/mnt/data/prod/postgres'
  redis_data:
    driver: local
    driver_opts:
      type: 'nfs'
      o: 'addr=10.10.10.10,nolock,soft,rw'
      device: ':/mnt/data/prod/redis'

secrets:
  db_password:
    file: ./secrets/prod/db_password.txt

Monitoring configuration (docker-compose.monitoring.yml):

# Monitoring stack
version: '3.8'

services:
  prometheus:
    image: prom/prometheus:latest
    volumes:
      - ./config/prometheus/prometheus.yml:/etc/prometheus/prometheus.yml
      - prometheus_data:/prometheus
    ports:
      - "9090:9090"
    deploy:
      resources:
        limits:
          cpus: '1.0'
          memory: 1G
    networks:
      - monitoring
      - frontend
      - backend

  grafana:
    image: grafana/grafana:latest
    volumes:
      - ./config/grafana/provisioning:/etc/grafana/provisioning
      - grafana_data:/var/lib/grafana
    environment:
      - GF_SECURITY_ADMIN_PASSWORD_FILE=/run/secrets/grafana_password
    ports:
      - "3000:3000"
    deploy:
      resources:
        limits:
          cpus: '1.0'
          memory: 1G
    networks:
      - monitoring
    depends_on:
      - prometheus
    secrets:
      - grafana_password

  node-exporter:
    image: prom/node-exporter:latest
    volumes:
      - /proc:/host/proc:ro
      - /sys:/host/sys:ro
      - /:/rootfs:ro
    command:
      - '--path.procfs=/host/proc'
      - '--path.sysfs=/host/sys'
      - '--collector.filesystem.ignored-mount-points=^/(sys|proc|dev|host|etc)($$|/)'
    deploy:
      mode: global
    networks:
      - monitoring

  cadvisor:
    image: gcr.io/cadvisor/cadvisor:latest
    volumes:
      - /:/rootfs:ro
      - /var/run:/var/run:ro
      - /sys:/sys:ro
      - /var/lib/docker/:/var/lib/docker:ro
    deploy:
      mode: global
    networks:
      - monitoring

networks:
  monitoring:

volumes:
  prometheus_data:
  grafana_data:

secrets:
  grafana_password:
    file: ./secrets/prod/grafana_password.txt

Deployment script example:

#!/bin/bash
# deploy-prod.sh

# Load environment variables
set -a
source environments/.env.prod
set +a

# Set image tag from Git
export TAG=$(git describe --tags --always)

# Deploy the application
docker stack deploy -c docker-compose.yml -c docker-compose.prod.yml app

# Deploy monitoring if requested
if [ "$1" == "--with-monitoring" ]; then
  docker stack deploy -c docker-compose.monitoring.yml monitoring
fi

# Display deployed services
docker stack services app

Conclusion

Docker Compose provides a powerful yet accessible way to deploy multi-container applications in production environments. While it may not offer all the features of more complex orchestration platforms like Kubernetes, its simplicity and flexibility make it an excellent choice for many deployment scenarios.

Key takeaways from this lecture include:

Environment-Specific Configuration: Use different compose files for different environments
Production Features: Leverage health checks, resource constraints, and restart policies
Security Best Practices: Implement proper secrets management and network segmentation
Scaling Options: Use Docker Swarm mode for multi-host deployment and scaling
Monitoring and Logging: Integrate observability solutions for production reliability
Operational Considerations: Plan for data persistence, backups, and high availability

As your applications grow in complexity and scale, you may eventually need to consider more advanced orchestration solutions like Kubernetes. However, Docker Compose provides an excellent foundation for production deployments and can serve many organizations well, especially when combined with Docker Swarm for clustering capabilities.