Mastering Jenkins Pipelines with Parallel & Scripted Stages

Optimizing Jenkins Pipelines: Parallel and Scripted Stages

When working with Jenkins pipelines, developers often face the challenge of efficiently building and testing across multiple environments. While a basic declarative pipeline might serve in many cases, adding parallel stages or leveraging scripted pipelines can significantly enhance your workflow. This article explains how you can combine declarative and scripted approaches in Jenkins to streamline the process, especially when dealing with multiple architectures.

A Basic Jenkins Pipeline Setup

A declarative Jenkins pipeline usually starts with a straightforward setup. Here’s an example:

pipeline {
    agent {
        label "linux&&x86_64"
    }
    stages {
        stage("Building / Testing") {
            stage("Testing") {
                steps {
                    sh("make test")
                }
            }
            stage("Building") {
                steps {
                    sh("make clean && make")
                }
            }
        }
    }
}

In this example, Jenkins runs both the building and testing processes sequentially. However, as your pipeline becomes more complex, the need for a more flexible approach arises.

Introducing Parallel Stages for Enhanced Efficiency

Jenkins allows you to run tasks in parallel, which can significantly speed up the process when testing and building across multiple platforms. Here’s how you can implement parallel stages within your pipeline:

pipeline {
    agent none
    stages {
        parallel {
            stage("x86_64 Building / Testing") {
                agent { label "x86_64&&linux" }
                stage("Testing") {
                    steps {
                        sh("make test")
                    }
                }
                stage("Building") {
                    steps {
                        sh("make clean && make")
                    }
                }
            }
            stage("s390x Building / Testing") {
                agent { label "s390x&&linux" }
                stage("Testing") {
                    steps {
                        sh("make test")
                    }
                }
                stage("Building") {
                    steps {
                        sh("make clean && make")
                    }
                }
            }
            stage("ppc64le Building / Testing") {
                agent { label "ppc64le&&linux" }
                stage("Testing") {
                    steps {
                        sh("make test")
                    }
                }
                stage("Building") {
                    steps {
                        sh("make clean && make")
                    }
                }
            }
        }
    }
}

By using the parallel block, Jenkins runs the building and testing tasks for multiple architectures simultaneously, improving efficiency. This method is particularly useful when you need to support different platforms, such as x86_64, s390x, and ppc64le.

Automating Multiple Architectures with Scripted Pipelines

Another approach to streamline your Jenkins pipeline is using scripted stages. This method allows you to define your stages dynamically, reducing the amount of repetitive code in your pipeline. Here’s an example:

def arches = ['x86_64', 's390x', 'ppc64le', 'armv8', 'armv7', 'i386']
archStages = arches.collectEntries {
    "${it} Building / Testing": -> {
        node("${it}&&linux") {
            stage("Testing") {
                checkout scm
                sh("make test")
            }
            stage("Building") {
                sh("make clean && make")
            }
        }
    }
}
pipeline {
    agent none
    stages {
        stage("Building / Testing") {
            script {
                parallel(archStages)
            }
        }
    }
}

This approach uses a scripted pipeline to define a dynamic set of stages for each architecture. However, it does come with its drawbacks. This method isn’t fully declarative, which might go against the principle of using declarative pipelines. Additionally, defining archStages outside the pipeline block can lead to issues if you add more stages before the archStages definition.

The Challenge of Balancing Declarative and Scripted Pipelines

While the scripted approach offers flexibility, it also makes your pipeline less declarative. As a result, you lose some of the benefits of declarative pipelines, such as clearer syntax and the ability to manage stages more effectively. Balancing these two approaches is key to building efficient, maintainable Jenkins pipelines.

At ZippyOPS, we specialize in providing consulting, implementation, and managed services for DevOps, DevSecOps, and DataOps, among others. Our experts can help you implement best practices for Jenkins pipeline optimization, ensuring you achieve the right balance between declarative and scripted approaches. Additionally, we offer solutions across cloud environments, microservices, infrastructure, and security. Learn more about our services here.

Conclusion

Optimizing Jenkins pipelines with parallel and scripted stages can significantly improve your CI/CD process, especially when dealing with multiple platforms. By running tasks in parallel or automating stages dynamically, you can reduce build times and improve overall efficiency. However, it’s important to find the right balance between declarative and scripted pipelines to maintain readability and manageability. If you’re looking to streamline your Jenkins workflows, consider consulting experts who can guide you through these optimizations. At ZippyOPS, we are ready to assist you with your DevOps journey.

For more information, reach out to us at sales@zippyops.com.