Jenkins Pipeline Fundamentals

Introduction to Jenkins

In our previous lectures, we explored CI/CD concepts and GitHub Actions. Now, we turn to Jenkins, one of the most established and versatile CI/CD platforms in the industry. Think of Jenkins as the Swiss Army knife of automation tools—highly configurable, with a vast ecosystem of plugins and extensions.

While GitHub Actions is tightly integrated with GitHub repositories, Jenkins is a standalone application that can connect to almost any source control system and can be deployed on your own infrastructure, giving you complete control over your CI/CD environment.

flowchart TD A[Source Code Repositories] --> B[Jenkins Server] B --> C[Build Agents] C --> D[Test Environments] D --> E[Deployment Targets] B --> F[Email Notifications] B --> G[Chat Integrations] B --> H[Monitoring Systems]

Jenkins vs. GitHub Actions

Feature	Jenkins	GitHub Actions
Hosting	Self-hosted	Cloud-hosted (GitHub)
Setup Complexity	Higher (installation, configuration)	Lower (built into GitHub)
Extensibility	1,800+ plugins	Marketplace actions
SCM Integration	Multiple SCMs	GitHub-focused
Customization	Highly customizable	Limited to YAML configuration
Infrastructure Control	Complete control	Limited to GitHub-provided runners or self-hosted runners

Jenkins Architecture

Before diving into Jenkins Pipelines, it's important to understand the overall architecture of Jenkins.

flowchart TD A[Controller] --> B[Agent 1] A --> C[Agent 2] A --> D[Agent 3] A --> E[Web UI] A --> F[REST API] A --> G[Pipeline Engine] A --> H[Plugin Manager] B --> I[Build Jobs] C --> J[Test Jobs] D --> K[Deploy Jobs]

Key Components

Controller (Master): The main Jenkins server that orchestrates the entire CI/CD process
Agents (Nodes): Workers that execute the actual build, test, and deployment tasks
Jobs: Individual tasks configured in Jenkins (traditional approach)
Pipelines: Sequences of stages and steps that define complete workflows
Plugins: Extensions that add functionality to Jenkins
Workspace: Directory where builds are executed on agents

Installation and Setup

While we won't go through the installation process in detail, here are the basic steps:

Install Java Development Kit (JDK) on your server
Download and install Jenkins (via package manager, Docker, or war file)
Access the web interface (typically http://localhost:8080)
Complete the initial setup wizard
Install recommended plugins
Create admin user and configure security
Configure system settings (JDK, Maven, etc.)
Set up build agents if needed

# Docker quick start example
docker run -d -p 8080:8080 -p 50000:50000 \
  -v jenkins_home:/var/jenkins_home \
  -v /var/run/docker.sock:/var/run/docker.sock \
  --restart unless-stopped \
  --name jenkins jenkins/jenkins:lts

Jenkins Pipeline Introduction

Jenkins Pipeline is a suite of plugins that supports implementing and integrating continuous delivery pipelines into Jenkins. Instead of creating multiple individual jobs, a pipeline defines the entire delivery process as code.

Pipeline Types

Jenkins offers two syntax options for defining pipelines:

graph TD A[Jenkins Pipeline] --> B[Declarative Pipeline] A --> C[Scripted Pipeline] B --> B1[YAML syntax] B --> B2[Structured sections] B --> B3[Limited flexibility] B --> B4[Easier to learn] C --> C1[Groovy syntax] C --> C2[Programmatic flow] C --> C3[High flexibility] C --> C4[Steeper learning curve]

Pipeline Concepts

Pipeline: The overall workflow definition
Node: An agent where the pipeline executes
Stage: A logical division of the pipeline (build, test, deploy)
Step: An individual task within a stage
Jenkinsfile: A text file containing the pipeline definition, typically saved in your source repository

Basic Declarative Pipeline Syntax

pipeline {
    agent any
    
    stages {
        stage('Build') {
            steps {
                echo 'Building..'
                sh 'npm install'
                sh 'npm run build'
            }
        }
        stage('Test') {
            steps {
                echo 'Testing..'
                sh 'npm test'
            }
        }
        stage('Deploy') {
            steps {
                echo 'Deploying....'
                sh 'npm run deploy'
            }
        }
    }
    
    post {
        always {
            echo 'This will always run'
        }
        success {
            echo 'Build successful!'
        }
        failure {
            echo 'Build failed!'
        }
    }
}

Creating Your First Pipeline

To create a pipeline in Jenkins:

Navigate to Jenkins dashboard and click "New Item"
Enter a name and select "Pipeline" as the item type
Configure the pipeline definition (either directly or by pointing to a Jenkinsfile in SCM)
Save and run the pipeline

Declarative Pipeline in Depth

Declarative Pipeline provides a more structured and opinionated syntax, making it easier to get started and maintain consistent pipelines.

Pipeline Structure

pipeline {
    // Where to execute the pipeline
    agent { ... }
    
    // Pipeline-wide options
    options { ... }
    
    // Environment variables 
    environment { ... }
    
    // Pipeline parameters
    parameters { ... }
    
    // Tools to auto-install and put on PATH
    tools { ... }
    
    // Triggers for running the pipeline
    triggers { ... }
    
    // Stages of the pipeline
    stages {
        stage('Name') { ... }
        stage('Another') { ... }
    }
    
    // Post-execution actions
    post { ... }
}

Agent Configuration

The agent directive specifies where the pipeline, or a specific stage, will execute:

// Execute on any available agent
agent any

// Execute on a specific agent
agent { label 'my-agent' }

// Execute in a Docker container
agent {
    docker {
        image 'node:18-alpine'
        args '-v /tmp:/tmp'
    }
}

// Execute with a Docker image for each stage
agent {
    dockerfile {
        filename 'Dockerfile.build'
        additionalBuildArgs '--build-arg VERSION=1.0.0'
    }
}

// No global agent, specify at stage level
agent none

Environment Variables

Define environment variables for the pipeline:

environment {
    // Simple variable
    DEBUG_MODE = 'true'
    
    // Credentials from Jenkins credentials store
    AWS_CREDS = credentials('aws-key')
    
    // Dynamic variable using script
    BUILD_NUMBER_COMBINED = "${env.BUILD_NUMBER}-${Math.round(Math.random() * 1000)}"
}

Stages and Steps

Stages organize the pipeline into discrete sections, while steps define the actual work:

stages {
    stage('Build') {
        steps {
            // Shell command (platform-specific)
            sh 'npm install'
            
            // Windows batch command
            bat 'npm run build'
            
            // Print message
            echo 'Building application...'
            
            // Multi-line script
            sh '''
                echo Starting build process
                mkdir -p build
                npm run build
                echo Build completed
            '''
        }
    }
    
    stage('Test') {
        steps {
            // Run script from workspace
            sh './run_tests.sh'
            
            // Archive test results
            junit 'test-results/*.xml'
        }
    }
}

Conditional Execution

Use conditionals to control when stages or steps execute:

stage('Deploy to Production') {
    when {
        // Only execute on main branch
        branch 'main'
        
        // Only when parameter is true
        expression { params.DEPLOY_TO_PROD == true }
        
        // All conditions must be true
        allOf {
            branch 'main'
            environment name: 'DEPLOY_TO_PROD', value: 'true'
        }
        
        // Any condition must be true
        anyOf {
            branch 'main'
            branch 'release'
        }
    }
    
    steps {
        // Deploy steps here
    }
}

Post-Execution Actions

The post section defines actions to run at the end of the pipeline or stage:

post {
    // Always execute, regardless of build status
    always {
        echo 'This will always run'
        cleanWs() // Clean workspace
    }
    
    // Execute on successful build
    success {
        echo 'Build successful!'
        emailext subject: 'Build Successful',
                 body: 'Your build has completed successfully.',
                 to: 'team@example.com'
    }
    
    // Execute on failed build
    failure {
        echo 'Build failed!'
        slackSend channel: '#builds',
                  color: 'danger',
                  message: "Build failed: ${env.JOB_NAME} ${env.BUILD_NUMBER}"
    }
    
    // Execute when build status changes from failure to success
    fixed {
        echo 'Build fixed!'
    }
    
    // More post conditions: unstable, regression, aborted, changed
}

Advanced Pipeline Features

Parallel Execution

Run stages or steps in parallel to speed up the pipeline:

stage('Parallel Tests') {
    parallel {
        stage('Unit Tests') {
            steps {
                sh 'npm run test:unit'
            }
        }
        stage('Integration Tests') {
            steps {
                sh 'npm run test:integration'
            }
        }
        stage('E2E Tests') {
            steps {
                sh 'npm run test:e2e'
            }
        }
    }
}

graph TD A[Parallel Tests] --> B[Unit Tests] A --> C[Integration Tests] A --> D[E2E Tests] B --> E[Next Stage] C --> E D --> E

Shared Libraries

Reuse code across multiple pipelines with shared libraries:

// In Jenkinsfile
@Library('my-shared-library') _

pipeline {
    agent any
    
    stages {
        stage('Build') {
            steps {
                // Call a function from the shared library
                buildNodeApp()
            }
        }
    }
}

In your shared library (vars/buildNodeApp.groovy):

def call(Map config = [:]) {
    def nodeVersion = config.nodeVersion ?: '18'
    
    sh "nvm use ${nodeVersion}"
    sh 'npm ci'
    sh 'npm run build'
    
    if (config.runTests) {
        sh 'npm test'
    }
}

Matrix Builds

Test across multiple configurations simultaneously:

stage('Matrix Tests') {
    matrix {
        axes {
            axis {
                name 'NODE_VERSION'
                values '16', '18', '20'
            }
            axis {
                name 'BROWSER'
                values 'chrome', 'firefox', 'safari'
            }
        }
        
        excludes {
            exclude {
                axis {
                    name 'NODE_VERSION'
                    values '16'
                }
                axis {
                    name 'BROWSER'
                    values 'safari'
                }
            }
        }
        
        stages {
            stage('Test') {
                steps {
                    sh "npm test -- --browser=${BROWSER} --node=${NODE_VERSION}"
                }
            }
        }
    }
}

Input Steps

Pause the pipeline for human interaction:

stage('Approve Deployment') {
    steps {
        // Pause for manual approval
        input message: 'Deploy to production?',
              ok: 'Approve',
              submitter: 'admin,devops',
              parameters: [
                  choice(name: 'ENVIRONMENT', choices: ['staging', 'production'], description: 'Select deployment environment')
              ]
    }
}

Stashing and Unstashing

Pass files between stages or agents:

stage('Build') {
    steps {
        sh 'npm run build'
        // Save build artifacts for later stages
        stash includes: 'dist/**', name: 'app-build'
    }
}

stage('Deploy') {
    agent { label 'deploy-agent' }
    steps {
        // Retrieve previously stashed files
        unstash 'app-build'
        sh 'rsync -avz dist/ server:/var/www/html/'
    }
}

Scripted Pipeline

While Declarative Pipeline is recommended for most use cases, Scripted Pipeline provides more flexibility for complex workflows.

Basic Scripted Pipeline Syntax

node('my-agent') {
    // SCM checkout
    checkout scm
    
    // Environment variables
    def nodeVersion = '18'
    
    // Build stage
    stage('Build') {
        try {
            sh "nvm use ${nodeVersion}"
            sh 'npm ci'
            sh 'npm run build'
        } catch (e) {
            currentBuild.result = 'FAILURE'
            throw e
        }
    }
    
    // Test stage
    stage('Test') {
        if (currentBuild.result != 'FAILURE') {
            sh 'npm test'
            
            // Conditional logic
            def testsPassed = sh(script: 'cat test-results.xml | grep failures="0"', returnStatus: true) == 0
            if (testsPassed) {
                echo 'All tests passed!'
            } else {
                echo 'Some tests failed'
                currentBuild.result = 'UNSTABLE'
            }
        }
    }
    
    // Deploy stage 
    stage('Deploy') {
        if (env.BRANCH_NAME == 'main' && currentBuild.result != 'FAILURE') {
            sh 'npm run deploy'
        }
    }
    
    // Post-build actions
    stage('Notify') {
        if (currentBuild.result == 'FAILURE') {
            emailext subject: 'Build Failed',
                     body: 'Check the logs for details',
                     to: 'team@example.com'
        }
    }
}

Key Differences from Declarative Pipeline

Feature	Scripted Pipeline	Declarative Pipeline
Syntax	Groovy script	Structured DSL
Flow Control	Groovy programming constructs (if/else, loops, functions)	Limited, defined sections (when, parallel)
Error Handling	try/catch blocks	post sections
Structure	Flexible	Rigid, predefined
Learning Curve	Steeper (requires Groovy knowledge)	Simpler

When to Use Scripted Pipeline

Complex workflows with dynamic logic
Legacy Jenkins environments
When you need fine-grained control over pipeline execution
When you're comfortable with Groovy programming

Practical Jenkins Pipeline Examples

Example 1: Complete CI/CD Pipeline for a Node.js Application

pipeline {
    agent {
        docker {
            image 'node:18-alpine'
            args '-v /tmp:/tmp'
        }
    }
    
    environment {
        CI = 'true'
        STAGING_SERVER = 'staging.example.com'
        PRODUCTION_SERVER = 'production.example.com'
        DEPLOY_CREDS = credentials('deploy-ssh-key')
    }
    
    stages {
        stage('Checkout') {
            steps {
                checkout scm
            }
        }
        
        stage('Install') {
            steps {
                sh 'npm ci'
            }
        }
        
        stage('Lint') {
            steps {
                sh 'npm run lint'
            }
        }
        
        stage('Test') {
            steps {
                sh 'npm test'
                junit 'test-results/*.xml'
            }
        }
        
        stage('Build') {
            steps {
                sh 'npm run build'
                archiveArtifacts artifacts: 'dist/**/*', fingerprint: true
            }
        }
        
        stage('Deploy to Staging') {
            when {
                branch 'develop'
            }
            steps {
                sh 'npm run deploy -- --server=$STAGING_SERVER'
            }
        }
        
        stage('Deploy to Production') {
            when {
                branch 'main'
            }
            steps {
                input message: 'Deploy to production?', ok: 'Approve'
                sh 'npm run deploy -- --server=$PRODUCTION_SERVER'
            }
        }
    }
    
    post {
        always {
            cleanWs()
        }
        success {
            slackSend channel: '#builds', color: 'good', message: "Build succeeded: ${env.JOB_NAME} ${env.BUILD_NUMBER}"
        }
        failure {
            slackSend channel: '#builds', color: 'danger', message: "Build failed: ${env.JOB_NAME} ${env.BUILD_NUMBER}"
        }
    }
}

Example 2: Multi-Branch Pipeline with Parallel Testing

pipeline {
    agent none
    
    stages {
        stage('Build') {
            agent {
                docker {
                    image 'maven:3.8.5-openjdk-17'
                }
            }
            steps {
                sh 'mvn -B -DskipTests clean package'
                stash includes: 'target/*.jar', name: 'app'
            }
        }
        
        stage('Test') {
            parallel {
                stage('Unit Tests') {
                    agent {
                        docker {
                            image 'maven:3.8.5-openjdk-17'
                        }
                    }
                    steps {
                        sh 'mvn test'
                        junit 'target/surefire-reports/*.xml'
                    }
                }
                
                stage('Integration Tests') {
                    agent {
                        docker {
                            image 'maven:3.8.5-openjdk-17'
                        }
                    }
                    steps {
                        sh 'mvn verify -DskipUnitTests'
                        junit 'target/failsafe-reports/*.xml'
                    }
                }
                
                stage('Code Analysis') {
                    agent {
                        docker {
                            image 'maven:3.8.5-openjdk-17'
                        }
                    }
                    steps {
                        sh 'mvn sonar:sonar'
                    }
                }
            }
        }
        
        stage('Deploy') {
            when {
                anyOf {
                    branch 'main'
                    branch 'develop'
                }
            }
            agent {
                label 'deploy'
            }
            steps {
                unstash 'app'
                script {
                    if (env.BRANCH_NAME == 'main') {
                        sh './deploy.sh production'
                    } else if (env.BRANCH_NAME == 'develop') {
                        sh './deploy.sh staging'
                    }
                }
            }
        }
    }
}

Example 3: Cross-Platform Pipeline with Matrix Builds

pipeline {
    agent none
    
    stages {
        stage('Build and Test') {
            matrix {
                axes {
                    axis {
                        name 'PLATFORM'
                        values 'linux', 'windows', 'mac'
                    }
                    axis {
                        name 'BROWSER'
                        values 'chrome', 'firefox', 'edge'
                    }
                }
                
                excludes {
                    exclude {
                        axis {
                            name 'PLATFORM'
                            values 'mac'
                        }
                        axis {
                            name 'BROWSER'
                            values 'edge'
                        }
                    }
                }
                
                agent {
                    label "${PLATFORM}"
                }
                
                stages {
                    stage('Checkout') {
                        steps {
                            checkout scm
                        }
                    }
                    
                    stage('Install') {
                        steps {
                            script {
                                if (PLATFORM == 'windows') {
                                    bat 'npm ci'
                                } else {
                                    sh 'npm ci'
                                }
                            }
                        }
                    }
                    
                    stage('Test') {
                        steps {
                            script {
                                if (PLATFORM == 'windows') {
                                    bat "npm run test:${BROWSER}"
                                } else {
                                    sh "npm run test:${BROWSER}"
                                }
                            }
                        }
                    }
                }
            }
        }
    }
}

Jenkins Pipeline Best Practices

Code Organization

Store Jenkinsfile in SCM: Keep your pipeline definition alongside your code
Use Declarative Pipeline: Unless you need the advanced features of Scripted Pipeline
Extract Complex Logic: Move complex code to shared libraries
Keep Stages Focused: Each stage should have a single responsibility
Use Meaningful Names: Give descriptive names to stages and steps

Performance Optimization

Use Parallel Execution: Run independent tasks simultaneously
Optimize Build Steps: Cache dependencies, minimize unnecessary steps
Use Appropriate Agents: Match job requirements to agent capabilities
Clean Up Workspaces: Prevent disk space issues with cleanWs()
Archive Only What's Needed: Be selective with archiveArtifacts

Security Considerations

Use Credentials Management: Never hardcode sensitive information
Limit Pipeline Permissions: Apply the principle of least privilege
Review Pipeline Scripts: Especially when using dynamic script execution
Validate Input Parameters: Sanitize user inputs to prevent injection
Secure Agent Communications: Encrypt controller-agent communication

Maintenance and Debugging

Add Comments: Document complex or non-obvious sections
Use Pipeline Visualization: Leverage the Blue Ocean plugin for better visibility
Implement Error Handling: Gracefully handle and report failures
Include Diagnostic Information: Log environment details for debugging
Set Appropriate Timeouts: Prevent hung pipelines with timeout options

Practical Exercise: Creating a Jenkins Pipeline

Exercise Brief

Create a Jenkins Pipeline for a Java web application that:

Builds the application with Maven
Runs unit and integration tests in parallel
Performs static code analysis
Builds a Docker image
Deploys to a staging environment
Requests approval before deployment to production

Step-by-Step Approach

Create a new Jenkinsfile in your project repository
Define the pipeline structure with appropriate stages
Configure build and test steps
Add parallel execution for tests
Include Docker image building
Configure deployment stages with approval

Solution Outline

pipeline {
    agent none
    
    environment {
        DOCKER_REGISTRY = 'registry.example.com'
        IMAGE_NAME = 'java-webapp'
        IMAGE_TAG = "${env.BUILD_NUMBER}"
        DOCKER_CREDS = credentials('docker-registry-credentials')
    }
    
    stages {
        stage('Build') {
            agent {
                docker {
                    image 'maven:3.8.5-openjdk-17'
                    args '-v $HOME/.m2:/root/.m2'
                }
            }
            steps {
                sh 'mvn -B -DskipTests clean package'
                stash includes: 'target/*.jar', name: 'app'
                stash includes: 'Dockerfile', name: 'dockerfile'
            }
        }
        
        stage('Test') {
            parallel {
                stage('Unit Tests') {
                    agent {
                        docker {
                            image 'maven:3.8.5-openjdk-17'
                            args '-v $HOME/.m2:/root/.m2'
                        }
                    }
                    steps {
                        sh 'mvn test'
                        junit 'target/surefire-reports/*.xml'
                    }
                }
                
                stage('Integration Tests') {
                    agent {
                        docker {
                            image 'maven:3.8.5-openjdk-17'
                            args '-v $HOME/.m2:/root/.m2'
                        }
                    }
                    steps {
                        sh 'mvn verify -DskipUnitTests'
                        junit 'target/failsafe-reports/*.xml'
                    }
                }
                
                stage('Code Analysis') {
                    agent {
                        docker {
                            image 'maven:3.8.5-openjdk-17'
                            args '-v $HOME/.m2:/root/.m2'
                        }
                    }
                    steps {
                        sh 'mvn sonar:sonar'
                    }
                }
            }
        }
        
        stage('Build Docker Image') {
            agent {
                label 'docker'
            }
            steps {
                unstash 'app'
                unstash 'dockerfile'
                sh '''
                    docker build -t ${DOCKER_REGISTRY}/${IMAGE_NAME}:${IMAGE_TAG} .
                    echo ${DOCKER_CREDS_PSW} | docker login ${DOCKER_REGISTRY} -u ${DOCKER_CREDS_USR} --password-stdin
                    docker push ${DOCKER_REGISTRY}/${IMAGE_NAME}:${IMAGE_TAG}
                '''
            }
        }
        
        stage('Deploy to Staging') {
            agent {
                label 'deploy'
            }
            steps {
                sh '''
                    echo "Deploying to staging environment..."
                    kubectl set image deployment/webapp webapp=${DOCKER_REGISTRY}/${IMAGE_NAME}:${IMAGE_TAG} --namespace=staging
                    kubectl rollout status deployment/webapp --namespace=staging
                '''
            }
        }
        
        stage('Approve Production Deployment') {
            steps {
                input message: 'Deploy to production?', ok: 'Approve'
            }
        }
        
        stage('Deploy to Production') {
            agent {
                label 'deploy'
            }
            steps {
                sh '''
                    echo "Deploying to production environment..."
                    kubectl set image deployment/webapp webapp=${DOCKER_REGISTRY}/${IMAGE_NAME}:${IMAGE_TAG} --namespace=production
                    kubectl rollout status deployment/webapp --namespace=production
                '''
            }
        }
    }
    
    post {
        success {
            echo 'Pipeline completed successfully'
            slackSend channel: '#deployments', color: 'good', message: "Deployment successful: ${env.JOB_NAME} ${env.BUILD_NUMBER}"
        }
        failure {
            echo 'Pipeline failed'
            slackSend channel: '#deployments', color: 'danger', message: "Deployment failed: ${env.JOB_NAME} ${env.BUILD_NUMBER}"
        }
    }
}

Challenge Extension

Once you have the basic pipeline working, try extending it with these features:

Add automated security scanning
Implement version tagging based on Git tags
Add performance testing before production deployment
Configure automated rollback if production health checks fail
Add notification to relevant teams based on which part of the pipeline failed

Real-World Jenkins Pipeline Use Cases

Example 1: Financial Institution

A major bank uses Jenkins Pipeline for:

Strict compliance checks for financial regulations
Multi-environment deployments with approval gates
Integration with legacy mainframe systems
Comprehensive audit logging for every deployment
Security scanning at multiple stages

Example 2: E-commerce Platform

A large e-commerce company implements Jenkins Pipeline for:

Microservices deployments across hundreds of services
Feature flag integration for gradual rollouts
Load testing before peak shopping season
Automated database schema migrations
Multi-region deployments with traffic shifting

Example 3: Game Development Studio

A game development company uses Jenkins Pipeline for:

Cross-platform builds (Windows, macOS, Linux, consoles)
Asset compilation and optimization
Automated playtesting
Distribution to multiple app stores
Beta testing deployment with analytics integration

Conclusion

Jenkins Pipeline provides a powerful, flexible framework for implementing CI/CD workflows. Key takeaways from this lecture include:

Pipeline as Code: Define your entire delivery process as code in a Jenkinsfile
Declarative vs. Scripted: Choose based on your needs for structure vs. flexibility
Advanced Features: Leverage parallel execution, shared libraries, and matrix builds
Best Practices: Follow established patterns for maintainable pipelines
Real-World Applications: Adapt these concepts to your specific deployment needs

Jenkins remains one of the most powerful and customizable CI/CD tools available, and mastering Jenkins Pipeline will give you the skills to implement sophisticated automation workflows for projects of any size or complexity.

In tomorrow's lectures, we'll explore more advanced topics in DevOps, including production Docker environments and multi-stage builds.