Docker Compose for Multi-Container Applications

Introduction to Docker Compose

Real-world applications are rarely single containers. Most modern applications consist of multiple interconnected services—a web server, a database, a cache, background workers, etc. Managing these services individually with Docker CLI commands quickly becomes complicated and error-prone.

Docker Compose solves this problem by allowing you to define a multi-container application in a single YAML file. With just one command, you can create and start all the services defined in your configuration.

Think of Docker Compose as a conductor for an orchestra. Just as a conductor coordinates different musical instruments to create a harmonious performance, Docker Compose coordinates multiple containers to work together as a cohesive application.

Key Benefits of Docker Compose

Single configuration file: Define your entire application stack in one place
Simple commands: Start, stop, and rebuild services with a single command
Environment consistency: Ensure the same environment across development, testing, and CI/CD
Parallel execution: Services are created and started in parallel
Isolated environments: Create isolated development environments for each project
Service dependencies: Define the order in which services should start

Docker Compose File Structure

Docker Compose uses a YAML file (typically named docker-compose.yml) to define the services, networks, and volumes required for your application. Let's explore the structure of this file:

Basic Structure

version: '3'  # Compose file format version

services:      # Definition of all application services
  webapp:      # First service name
    image: nginx:alpine
    ports:
      - "8080:80"
    volumes:
      - ./website:/usr/share/nginx/html
    depends_on:
      - database

  database:    # Second service name
    image: mysql:8.0
    environment:
      - MYSQL_ROOT_PASSWORD=secret
      - MYSQL_DATABASE=myapp
    volumes:
      - db_data:/var/lib/mysql

networks:      # Custom networks for service isolation
  backend:
    driver: bridge

volumes:       # Persistent volume definitions
  db_data:     # Named volume for database data

Docker Compose File Versions

Docker Compose has evolved over time, with different file format versions offering different features:

Version 1: Original format, now legacy
Version 2: Added named networks and improved volume management
Version 3: Designed for both Docker Compose and Docker Swarm, current standard
Version 3.x: Various incremental improvements and new features

For most modern use cases, version 3.x is recommended.

Key Components of a Compose File

version: The Compose file format version
services: The containers that make up your application
networks: The networking configuration for your application
volumes: Persistent data storage for your services
configs: External configuration files (available in newer versions)
secrets: Sensitive data management (available in newer versions)

Defining Services in Docker Compose

The services section is the heart of your Docker Compose file, defining each container that makes up your application. Let's explore the configuration options for services:

Basic Service Definition

services:
  webapp:
    image: nginx:alpine       # Use existing image
    # OR
    build: ./webapp           # Build from Dockerfile in ./webapp
    # OR
    build:                    # Build with detailed options
      context: ./webapp       # Build context
      dockerfile: Dockerfile.dev  # Custom Dockerfile name
      args:                   # Build arguments
        ENV_NAME: development

Container Configuration

services:
  webapp:
    image: nginx:alpine
    container_name: my-webapp  # Custom container name
    hostname: webapp           # Container hostname
    restart: always           # Restart policy (no, always, on-failure, unless-stopped)
    
    ports:                    # Port mapping
      - "8080:80"             # HOST:CONTAINER
      - "443:443"
    
    environment:              # Environment variables
      - NODE_ENV=development
      - DEBUG=true
    # OR
    env_file: .env            # Load from .env file
    
    volumes:                  # Volume mounts
      - ./code:/code          # Bind mount
      - logs:/var/log         # Named volume
    
    command: nginx -g 'daemon off;'  # Override default command
    entrypoint: /docker-entrypoint.sh  # Override entrypoint

Dependencies and Healthchecks

services:
  webapp:
    image: my-app
    depends_on:               # Start after these services
      - database
      - redis
    
    healthcheck:              # Container health check
      test: ["CMD", "curl", "-f", "http://localhost/health"]
      interval: 30s
      timeout: 10s
      retries: 3
      start_period: 40s

Network Configuration

services:
  webapp:
    image: my-app
    networks:                 # Connect to these networks
      - frontend
      - backend
    
    network_mode: "host"      # Use host network (alternative)
    # OR
    network_mode: "service:database"  # Use another container's network

Resource Limits

services:
  webapp:
    image: my-app
    deploy:                   # Resource constraints
      resources:
        limits:
          cpus: '0.5'         # Use at most 50% of CPU
          memory: 512M        # Use at most 512MB RAM
        reservations:
          cpus: '0.25'        # Reserve at least 25% of CPU
          memory: 256M        # Reserve at least 256MB RAM

graph TD A[Service Definition] --> B[Image Source
image or build] A --> C[Container Configuration
name, hostname, restart] A --> D[Network Configuration
networks, ports] A --> E[Storage
volumes, bind mounts] A --> F[Environment
env vars, env files] A --> G[Resource Limits
CPU, memory] A --> H[Dependencies
depends_on] A --> I[Health Monitoring
healthcheck] A --> J[Runtime
command, entrypoint]

Networking in Docker Compose

Docker Compose automatically creates a default network for your application, allowing services to communicate using their service names as hostnames. However, you can also define custom networks for more complex scenarios.

Default Networking

By default, Docker Compose creates a bridge network for your application. Services can reach each other using their service names as hostnames:

services:
  webapp:
    image: nginx
    depends_on:
      - api
  
  api:
    image: my-api
    # The webapp service can reach this service at hostname "api"

In this example, the webapp container can communicate with the api container using the hostname "api" (e.g., http://api:8000).

Custom Networks

For more control, you can define custom networks:

services:
  webapp:
    image: nginx
    networks:
      - frontend
      - backend
  
  api:
    image: my-api
    networks:
      - backend
  
  database:
    image: postgres
    networks:
      - backend

networks:
  frontend:
    driver: bridge
  backend:
    driver: bridge
    internal: true  # No external access

In this example:

The webapp service is connected to both networks
The api and database services are only on the backend network
The backend network is internal (not accessible from outside Docker)

Network Drivers

Docker Compose supports various network drivers for different use cases:

bridge: The default driver, creates a private network on a single host
host: Uses the host's networking directly (removes network isolation)
overlay: For multi-host networking, primarily used with Docker Swarm
ipvlan: Gives control over IP addresses at Layer 2 or Layer 3
macvlan: Makes containers appear as physical devices on the network
none: Disables networking for a container

Network Configuration Options

networks:
  frontend:
    driver: bridge
    driver_opts:      # Driver-specific options
      com.docker.network.bridge.name: frontend_bridge
    ipam:             # IP Address Management
      driver: default
      config:
        - subnet: 172.16.238.0/24
          gateway: 172.16.238.1
    enable_ipv6: true
    internal: false   # Allow external connections
    attachable: true  # Allow standalone containers to connect

Volumes and Data Management

Docker containers are ephemeral—when a container stops, any data stored within it is lost. To persist data or share it between containers, Docker Compose provides volume management.

Types of Data Storage

Named volumes: Persistent storage managed by Docker
Bind mounts: Map host directories into containers
tmpfs mounts: Store data in host memory only

Named Volumes

services:
  database:
    image: postgres:13
    volumes:
      - db_data:/var/lib/postgresql/data  # Named volume

volumes:  # Volume declaration
  db_data:  # Will persist even if containers are removed

Bind Mounts for Development

services:
  webapp:
    image: node:14
    volumes:
      - ./src:/app/src  # Maps host ./src to container /app/src
      - ./package.json:/app/package.json

Advanced Volume Configuration

volumes:
  db_data:
    driver: local
    driver_opts:
      type: nfs
      o: addr=192.168.1.1,rw
      device: ":/path/to/data"
      
  encrypted_data:
    driver: local
    driver_opts:
      type: tmpfs
      device: tmpfs
      o: "size=100m,encryption"

Volume Access Modes

services:
  webapp:
    image: node:14
    volumes:
      - db_data:/data:ro  # Read-only access
      
  database:
    image: postgres:13
    volumes:
      - db_data:/var/lib/postgresql/data:rw  # Read-write access

Properly configured volumes ensure your application data persists across container restarts and rebuilds, which is especially important for databases and other stateful services.

Environment Variables and Configuration

Environment variables are a key mechanism for configuring Docker containers. Docker Compose provides several ways to set environment variables for your services.

Inline Environment Variables

services:
  webapp:
    image: my-app
    environment:
      - NODE_ENV=production
      - API_URL=http://api:8000
      - DEBUG=false

Using .env Files

For a cleaner separation of configuration:

services:
  webapp:
    image: my-app
    env_file:
      - ./common.env  # Load variables from this file
      - ./app.env     # Variables from this file override common.env

Example .env file:

# app.env
NODE_ENV=production
API_URL=http://api:8000
DEBUG=false
SECRET_KEY=your-secret-key-here

Project-level .env File

You can also place a .env file in the same directory as your docker-compose.yml file to set default environment variables for the docker-compose command itself:

# .env file in project root
COMPOSE_PROJECT_NAME=myproject
COMPOSE_FILE=docker-compose.yml:docker-compose.prod.yml
POSTGRES_PASSWORD=secret

These variables can be referenced in your Compose file using variable substitution:

Variable Substitution

services:
  database:
    image: postgres:13
    environment:
      - POSTGRES_PASSWORD=${POSTGRES_PASSWORD}  # From .env file
      - POSTGRES_USER=${POSTGRES_USER:-postgres}  # Default if not set

This feature allows you to keep sensitive information out of your Docker Compose file, making it safer to store in version control.

Running Multi-Container Applications

Now that we understand how to define services in Docker Compose, let's explore how to manage the lifecycle of these services using Docker Compose commands.

Basic Docker Compose Commands

# Start all services defined in docker-compose.yml
docker-compose up

# Start services in detached mode (background)
docker-compose up -d

# View running services
docker-compose ps

# View logs for all services
docker-compose logs

# View logs for specific service
docker-compose logs webapp

# Follow logs (like tail -f)
docker-compose logs -f

# Stop services but keep containers
docker-compose stop

# Stop and remove containers
docker-compose down

# Stop and remove containers, networks, images, and volumes
docker-compose down --rmi all --volumes

Building and Rebuilding Services

# Build or rebuild services
docker-compose build

# Build specific service
docker-compose build webapp

# Build with no cache
docker-compose build --no-cache

# Start services and rebuild if needed
docker-compose up --build

# Force recreation of containers
docker-compose up --force-recreate

Scaling Services

# Start multiple instances of a service
docker-compose up -d --scale webapp=3 --scale worker=2

# Scale a running service
docker-compose scale webapp=5

Executing Commands in Running Services

# Run a one-off command in a service
docker-compose exec webapp npm test

# Run a command in a new container of a service
docker-compose run --rm webapp npm install express

# Open a shell in a service container
docker-compose exec webapp sh

flowchart TD A[Docker Compose Commands] --> B[Service Lifecycle] A --> C[Service Inspection] A --> D[Service Interaction] B --> B1[up - Start services] B --> B2[down - Stop services] B --> B3[start/stop - Control services] B --> B4[restart - Restart services] C --> C1[ps - List containers] C --> C2[logs - View logs] C --> C3[config - Validate config] C --> C4[images - List images] D --> D1[exec - Run command in container] D --> D2[run - Run one-off command] D --> D3[port - Print port mapping] D --> D4[top - List running processes]

Real-World Compose Examples

Web Application with Database and Cache

version: '3'

services:
  webapp:
    build: ./webapp
    ports:
      - "8000:8000"
    environment:
      - DATABASE_URL=postgres://postgres:password@db:5432/app
      - REDIS_URL=redis://cache:6379
    depends_on:
      - db
      - cache
    restart: unless-stopped

  db:
    image: postgres:13
    volumes:
      - postgres_data:/var/lib/postgresql/data
    environment:
      - POSTGRES_PASSWORD=password
      - POSTGRES_DB=app
    healthcheck:
      test: ["CMD-SHELL", "pg_isready -U postgres"]
      interval: 10s
      timeout: 5s
      retries: 5

  cache:
    image: redis:6-alpine
    volumes:
      - redis_data:/data

volumes:
  postgres_data:
  redis_data:

Microservices Architecture

version: '3'

services:
  frontend:
    build: ./frontend
    ports:
      - "80:80"
    depends_on:
      - api-gateway

  api-gateway:
    build: ./api-gateway
    ports:
      - "8000:8000"
    depends_on:
      - auth-service
      - product-service
      - user-service

  auth-service:
    build: ./auth-service
    environment:
      - DB_HOST=auth-db
    depends_on:
      - auth-db

  product-service:
    build: ./product-service
    environment:
      - DB_HOST=product-db
    depends_on:
      - product-db

  user-service:
    build: ./user-service
    environment:
      - DB_HOST=user-db
    depends_on:
      - user-db

  auth-db:
    image: mongo:4
    volumes:
      - auth_data:/data/db

  product-db:
    image: postgres:13
    volumes:
      - product_data:/var/lib/postgresql/data
    environment:
      - POSTGRES_PASSWORD=password
      - POSTGRES_DB=products

  user-db:
    image: mysql:8
    volumes:
      - user_data:/var/lib/mysql
    environment:
      - MYSQL_ROOT_PASSWORD=password
      - MYSQL_DATABASE=users

volumes:
  auth_data:
  product_data:
  user_data:

Development Environment with Hot Reloading

version: '3'

services:
  frontend:
    build:
      context: ./frontend
      dockerfile: Dockerfile.dev
    ports:
      - "3000:3000"
    volumes:
      - ./frontend:/app
      - /app/node_modules
    environment:
      - CHOKIDAR_USEPOLLING=true
      - REACT_APP_API_URL=http://localhost:5000

  backend:
    build:
      context: ./backend
      dockerfile: Dockerfile.dev
    ports:
      - "5000:5000"
    volumes:
      - ./backend:/app
      - /app/node_modules
    environment:
      - DATABASE_URL=postgres://postgres:password@db:5432/dev
    depends_on:
      - db

  db:
    image: postgres:13-alpine
    volumes:
      - postgres_data:/var/lib/postgresql/data
    environment:
      - POSTGRES_PASSWORD=password
      - POSTGRES_DB=dev
    ports:
      - "5432:5432"

volumes:
  postgres_data:

Compose for Different Environments

Applications often need different configurations for development, testing, and production environments. Docker Compose provides several approaches to manage this complexity.

Multiple Compose Files

You can use multiple Compose files that build upon each other:

Base configuration (docker-compose.yml):

version: '3'

services:
  webapp:
    build: ./webapp
    environment:
      - NODE_ENV=production

  db:
    image: postgres:13
    volumes:
      - postgres_data:/var/lib/postgresql/data

volumes:
  postgres_data:

Development overrides (docker-compose.override.yml):

version: '3'

services:
  webapp:
    build:
      context: ./webapp
      dockerfile: Dockerfile.dev
    volumes:
      - ./webapp:/app
    environment:
      - NODE_ENV=development
      - DEBUG=true
    ports:
      - "3000:3000"
      
  db:
    ports:
      - "5432:5432"

Production overrides (docker-compose.prod.yml):

version: '3'

services:
  webapp:
    restart: always
    deploy:
      replicas: 2
    ports:
      - "80:3000"

  db:
    restart: always
    environment:
      - POSTGRES_PASSWORD_FILE=/run/secrets/db_password
    secrets:
      - db_password

secrets:
  db_password:
    file: ./secrets/db_password.txt

Using Multiple Compose Files

# Development (default)
docker-compose up

# Production
docker-compose -f docker-compose.yml -f docker-compose.prod.yml up -d

# Testing
docker-compose -f docker-compose.yml -f docker-compose.test.yml up

Environment-specific .env Files

Another approach is to use different .env files:

# Development
docker-compose --env-file .env.dev up

# Production
docker-compose --env-file .env.prod up

graph TD A[Environment Configuration] --> B[Multiple Compose Files] A --> C[Environment Variables] A --> D[Build Arguments] B --> B1[docker-compose.yml
Base config] B --> B2[docker-compose.override.yml
Dev overrides] B --> B3[docker-compose.prod.yml
Production overrides] C --> C1[.env.dev
Development variables] C --> C2[.env.prod
Production variables] C --> C3[.env.test
Testing variables] D --> D1[ARG in Dockerfile] D --> D2[build.args in Compose]

Docker Compose in CI/CD Pipelines

Docker Compose is not just for development—it's also valuable in continuous integration and continuous deployment (CI/CD) pipelines. Here's how Compose can be integrated into your CI/CD workflow:

Testing with Docker Compose

# docker-compose.test.yml
version: '3'

services:
  webapp:
    build: .
    environment:
      - NODE_ENV=test
      - DATABASE_URL=postgres://postgres:password@db:5432/test
    depends_on:
      - db
    command: npm test

  db:
    image: postgres:13-alpine
    environment:
      - POSTGRES_PASSWORD=password
      - POSTGRES_DB=test

CI Pipeline Example (GitHub Actions)

# .github/workflows/ci.yml
name: CI

on:
  push:
    branches: [ main ]
  pull_request:
    branches: [ main ]

jobs:
  test:
    runs-on: ubuntu-latest

    steps:
    - uses: actions/checkout@v2
    
    - name: Build and test with Docker Compose
      run: |
        docker-compose -f docker-compose.yml -f docker-compose.test.yml up --build --exit-code-from webapp
        
    - name: Clean up
      run: docker-compose down

Docker Compose for Staging Deployments

# Deploy to staging server
stages:
  - build
  - test
  - deploy

deploy_staging:
  stage: deploy
  script:
    - echo "$SSH_PRIVATE_KEY" > id_rsa
    - chmod 600 id_rsa
    - scp -i id_rsa docker-compose.yml docker-compose.prod.yml user@staging-server:/app
    - ssh -i id_rsa user@staging-server "cd /app && docker-compose -f docker-compose.yml -f docker-compose.prod.yml up -d --build"
  only:
    - main

Docker Compose provides a consistent environment across development, testing, and deployment, ensuring that your application behaves the same way in all environments.

Limitations and When to Use Kubernetes

While Docker Compose is excellent for development and simple deployments, it has limitations for complex production environments. Understanding these limitations helps you decide when to transition to more advanced orchestration tools like Kubernetes.

Docker Compose Limitations

Single host only: Compose runs services on a single Docker host
Limited scaling: Basic scaling without advanced load balancing
Simple networking: Limited network configuration options
Basic health checks: Limited self-healing capabilities
Manual updates: No built-in rolling updates or canary deployments
Limited secrets management: Basic secrets handling

When to Consider Kubernetes

Consider transitioning to Kubernetes when you need:

Multi-host deployments: Distribute workloads across multiple servers
Advanced auto-scaling: Scale based on CPU, memory, or custom metrics
Self-healing: Automatic replacement of failed containers
Rolling updates: Zero-downtime deployments
Advanced networking: Network policies, service mesh integration
Resource management: Sophisticated CPU/memory allocation
High availability: Multi-zone and multi-region deployments
Advanced secrets: Secure management of sensitive information
Large-scale applications: Managing hundreds or thousands of containers

Transitioning from Compose to Kubernetes

Tools like Kompose can help convert Docker Compose files to Kubernetes configurations:

# Install Kompose
curl -L https://github.com/kubernetes/kompose/releases/download/v1.26.1/kompose-linux-amd64 -o kompose
chmod +x kompose
sudo mv ./kompose /usr/local/bin/kompose

# Convert Compose file to Kubernetes resources
kompose convert -f docker-compose.yml

# This creates Kubernetes YAML files:
# - deployment.yaml
# - service.yaml
# - persistent-volume-claim.yaml
# etc.

Even when using Kubernetes in production, Docker Compose remains valuable for local development due to its simplicity and ease of use.

Practice Activities

Activity 1: Simple Web Application with Database

Create a Docker Compose file for a simple web application with a database:

Create a directory for your project
Create a simple web application (e.g., using Node.js and Express)
Create a docker-compose.yml file that includes:
- A web service that builds from your application code
- A database service (e.g., MongoDB, PostgreSQL, or MySQL)
- Appropriate volume configuration for database persistence
- Port mapping to access the web application
Start the services with docker-compose up
Access the web application in your browser
Make changes to your application code and observe how they affect the running container

Activity 2: Multi-Environment Configuration

Create a Docker Compose setup with different configurations for development and production:

Create a base docker-compose.yml file with common service configurations
Create a docker-compose.override.yml file with development-specific settings:
- Volume mounts for code
- Development environment variables
- Exposed ports for debugging
Create a docker-compose.prod.yml file with production-specific settings:
- No volume mounts for code
- Production environment variables
- Restart policies
- Resource constraints
Test running your application with both configurations

Activity 3: Service Dependencies and Health Checks

Enhance a Docker Compose configuration with dependencies and health checks:

Create a Docker Compose file with at least three services (e.g., web, api, database)
Configure dependency relationships between services using depends_on
Add health checks to each service
Test how the health checks affect service startup order
Deliberately make a health check fail and observe the behavior

Resources for Further Learning

Summary

In this lecture, we've explored Docker Compose as a powerful tool for defining and running multi-container Docker applications:

Docker Compose uses a YAML file to configure application services, networks, and volumes
Services can be configured with images, build contexts, environment variables, volumes, and more
Networking in Docker Compose allows services to communicate using service names as hostnames
Volume management provides persistent storage solutions for containers
Environment variables and configuration can be managed in various ways
Docker Compose commands allow for easy management of application lifecycle
Multiple configuration approaches support different environments (development, testing, production)
Docker Compose can be integrated into CI/CD pipelines
While powerful, Docker Compose has limitations for complex production environments

Docker Compose bridges the gap between single-container Docker usage and more complex orchestration solutions like Kubernetes, making it an essential tool in a developer's toolkit. In our next lecture, we'll explore how to use development containers in VS Code to further enhance your development workflow.