Beginner's Introduction to Jenkins - CI/CD

Introduction

The advent of fast-paced development with many methodologies like Scrum, Agile and Kanban brought along some key issues: developers working in small increments spent a lot of time waiting for the new version to build, reach the testers and eventually get deployed. The development process would be much faster if this cycle eliminated human intervention wherever possible.

The idea for automation servers arose. Throughout the years, many solutions came and went - but Jenkins managed to come out on top and become the standard when it comes to automation. While perfect for simple scheduling and executing shell or batch scripts, being open-source and having large community support, Jenkins provides easy integration with many tools, platforms and frameworks with over 1,500 plugins making the entire process effortless.

Key Jenkins Concepts

To understand why someone should use Jenkins, we need to understand what are the problems Jenkins aims to solve.

The CI/CD Pipeline

From start to finish, there are multiple stages in the software life cycle. Automation helps us bridge the gaps between them, making the entire process seamless. Taking a plain, run-of-the-mill workflow as an example - an automation server will listen for new development versions, fetch them and run the appropriate commands to build and test the new version, and finally put the new version into production if everything was successful.

All of these stages, which are made out of smaller steps, their timing and order need to be defined as a single pipeline.

The Controller-Agent Architecture

In order to spread the load of concurrent builds and tasks, Jenkins introduces the Controller-Agent architecture. The "Controller" - the Jenkins server is responsible for administering projects, configurations, users and data. "Agents" can be called upon to execute certain stages of a particular pipeline. This brings many benefits such as easy scaling, optimal hardware (large-scale math or data-heavy processes), test servers and platform specific builds.

Plugins

Plugins are the core of Jenkins' success. Any Java developer can write their own plugin and share it with the community. Jenkins stresses this very much and has written some very detailed guides on the subject. Whatever tool you want to use in the pipeline, Jenkins probably has you covered with a plugin that makes the entire process of configuration and integration with that tool a breeze.

Besides the workflow aspect, there are many plugins written directly for managing Jenkins itself - from a prettier UI to easier management of multiple users and their access privileges.

Naming Conventions

Throughout Jenkins' life cycle, the nomenclature changed a bit, amongst other things due to the fact that certain terminology can be viewed as offensive.

Even though certain naming conventions were introduced years ago, the community still uses them interchangeably. In order to prevent confusion, here are some synonymous terms:

• Master == Controller
• Slave == Agent
• Job == Project
• Workflow == Pipeline

The older Master/Slave architecture was renamed to the Controller/Agent architecture, due to negative connotations of the terminology.

Installation

There are many ways to install Jenkins. Along platform specific installations, many cloud hosting platforms offer pre-configured Jenkins packages. There is also an official Docker image, as well as a generic Java war file.

This guide will cover the installation process on Ubuntu 20.04, though the process doesn't differ much for Windows, Mac or other distributions. Check out Jenkins' download page for your specific machine (or cloud service).

Installing a JDK/JRE

Being written in Java, Jenkins requires a Java Runtime Environment in order to run. Note that for OpenJDK, only versions 8 and 11 are supported. If you want to build Java applications with Jenkins, you'll need to install a JDK instead of only the JRE.

Let's go ahead and install openjdk-11-jdk:

$ sudo apt install openjdk-11-jdk

Updating Sources List and Installing with apt

Jenkins is not available in the official Linux repository, so we'll have to add its own repository. We are going to install the LTS (Long-Term Support) version which is updated every 12 weeks, according to the Jenkins website.

For the bleeding edge, if for example you want to develop plugins, remove the -stable part from the first and second lines below:

$ wget -q -O - https://pkg.jenkins.io/debian-stable/jenkins.io.key | sudo apt-key add -
$ sudo sh -c 'echo deb https://pkg.jenkins.io/debian-stable binary/ > \
        /etc/apt/sources.list.d/jenkins.list'
$ sudo apt update
$ sudo apt install jenkins

Finally, we can go ahead and register Jenkins as a service and run it via the terminal:

$ sudo systemctl daemon-reload
$ sudo systemctl start jenkins

Accessing Jenkins from the Browser

By default, Jenkins is hosted on port 8080.

If you have installed Jenkins on your local machine, it can be accessed from the browser by navigating to localhost:8080. If you installed it under a virtual machine, running the ifconfig command (part of net-tools) will reveal the VM's IP address on your local network.

Firing up Jenkins for the first time, a hidden password is required. It can be found in /var/lib/jenkins/secrets/initialAdminPassword, written on the server. Retrieve and enter it to continue:

On the second page, choose the "Install suggested plugins" option. Plugin installation is covered later on in this guide. Wait for the plugins to install, set up an administrator account, configure the Jenkins URL (in our case, we will leave it at localhost:8080) for others to access, and you will be presented with the Dashboard:

Using Jenkins

Jenkins is a big and complex project, and we'll cover most of the most important features.

To do so, we'll cover three workflow examples:

• Using the browser to build a Maven project with email notifications.
• Connecting to a GitHub repository and building our Maven app with the repo's Jenkinsfile.
• Using the jenkins-cli.jar to perform everyday tasks such as manipulating jobs, starting builds, checking logs etc. from the command line.

We will be using a dummy Maven project, made for this guide.

Simple Local Build

To easily set up Maven - we will install a Maven plugin for Jenkins.

Installing plugins

Navigate to the "Manage Plugins" option within the "Manage Jenkins" section:

With the "Available" tab selected, search for "Maven" and check the adjacent box. Select "Install without restart":

Wait for the plugin to install before continuing.

Configuring Maven

From the "Manage Jenkins" section, go to "Global Tool Configuration". Scroll to the bottom, and add a Maven installation. Save the new changes.

Configuring Email

Go to "Configure System" within "Manage Jenkins". Navigate down to the "E-Mail Notification" and set up an email address Jenkins will be using. Note that Google and many other services require setting up application-specific passwords, due to security concerns.

Creating a Project

Select the "New Item" from the sidebar, give your project a name and mark it as a "Maven Project" before hitting "OK":

At the bottom, configure the POM file location and configure the additional post step for sending email notifications. Note that under Unix-like systems, Jenkins creates a separate jenkins user, so access privileges may be required.

The default "Trigger" for sending emails is a failed build, but that can be changed in the "Advanced settings".

Running the Build

With the steps configured, we can start the build. The sidebar will inform us of the progress, as well as show us the build history. Failed builds (#1) will show a different color.

Navigate to the "Console Output" option for more detail:

Tracking a GitHub Repo and reading it's Jenkinsfile

We had to go over the simpler example above to get the feel for how Jenkins works. The following example is by far the most frequent way any pipeline will be set up. Having the pipeline configuration placed inside a text file and within a CVS allows for greater portability and adjustability.

In order to push our changes upstream, or if working on a private repository, we will have to authenticate Jenkins within GitHub. We'll also be pointing Jenkins in the direction of the Jenkinsfile, which is usually contained within the repository's root.

The Jenkinsfile is just a file that contains the definition of a pipeline. It's supposed to be checked into source control. Using the Jenkinsfile, Jenkins can execute pipelines.

GitHub Credentials

To securely access GitHub, we will generate a key-pair, register the private key within Jenkins, and write the public key to the GitHub repo's deploy list. That way, we can have open-source, public code (or simply code shared between colleagues) while still maintaining security.

To generate the key pair, we will use the ssh-keygen command:

$ ssh-keygen -t rsa -b 4096

Choose where to store the keys and set a password if you’d like to.

Next, copy the private key and go to the Jenkins Dashboard. From there, under "Manage Jenkins", go to "Manage Credentials -> Jenkins -> Global credentials -> Add credentials".

Select "SSH Username with private key", paste in the key along with its passphrase and hit OK.

Copy the public key and paste it into the deploy key list for your repository, located at https://github.com/user/repo/setting/keys/new. Give it a name, check the box to allow write access if you want pushing changes to be a part of your pipeline and click "Add key".

Now, it's time to create the project and set that up.

Creating the Project

From the dashboard, go to "New Item -> Multibranch Pipeline". Give it a name and continue. Under "Branch Sources" select "git", give the source a name, type the address of the repository and select the appropriate credentials we just added. Save the changes and continue.

We'll make our project listen for changes in the repository by scrolling down to "Scan Multibranch Pipeline Triggers" and checking the little box, selecting "1 minute" as our desired interval. Hit "Save".

The Git route is also the more generic, since the procedure is the same for any remote Git repository - it doesn't have to be hosted on GitHub, but maybe on a company's server, or a different hosting service such as BitBucket and GitLab.

So far our project will only scan for changes on our Git repository. Let's write the Jenkinsfile so that Jenkins knows what it's supposed to do.

Writing a Jenkinsfile

To define a set of procedures to run - a pipeline - we use Jenkinsfiles. Similar to how Docker relies of Dockerfiles as a list of commands to run, so does Jenkins. Jenkinsfiles use a Groovy-like syntax.

Let's go ahead and write a Groovy-like script, called "Jenkinsfile" (no extension) and commit/push it to our repository's root:

pipeline {
    agent any
    stages{
        stage('build'){
            steps {
                sh 'mvn clean install'
            }
        }
    }
    post {
        always {
            mail to :"[email protected]",
                subject: "Build Finished: ${currentBuild.fullDisplayName}",
                body:"Check out status at ${env.BUILD_URL}"
        }
    }
}

This simple syntax is very readable and fairly self-explanatory. We've defined a pipeline, which can be executed by any available agent. There is only one stage ('build'), which will run a shell command to build our code.

After the stage finishes, an email will be sent out notifying that the pipeline finished, providing some information and a link to the build's URL for more information.

This process, or any other if we were to make changes to the Jenkinsfile, will be executed every time a new commit gets pushed to the repository.

Menial Tasks on The Command-line

In order to avoid going the browser route for everyday tasks such as starting/stopping jobs, checking the console output of a build or importing/exporting, we will need to download jenkins-cli.jar.

It is included with every Jenkins installation:

$ wget localhost:8080/jnlpJars/jenkins-cli.jar

To see if we're able to connect to Jenkins, lets type in the following command:

$ java -jar jenkins-cli.jar http://localhost:8080 who-am-i
Authenticated as: anonymous
Authorities:
  anonymous

The connection was successful. Let's connect with a user with greater permissions to be able to manage projects, plugins, builds etc. To avoid typing our password in the console, an API token can be set up.

Generating an API Token

Let's create a token for our admin user, follow the steps on the image and copy the text we received:

In order to specify a user, we will be using the -auth option which receives the [USER:SECRET | @FILE] parameters. Let's put our username:token in a file called credentials which we will be referencing when connecting.

$ java -jar jenkins-cli.jar -s http://localhost:8080 -auth @credentials who-am-i
Authenticated as: admin
Authorities:
    authenticated

Listing Jobs and Running Builds

To list and run our jobs we will use the list-jobs and build commands, and use the console command to get the output. Note that GitHub Maven App, being a multi-branch project requires specifying the branch with project/branch syntax, in our case `GitHub Maven/Maven':

$ java -jar jenkins-cli.jar -s http://localhost:8080 -auth @credentials list-jobs
GitHub Maven App
Maven App
$ java -jar jenkins-cli.jar -s http://localhost:8080 -auth @credentials build 'GitHub Maven App/Maven'
$ $ java -jar jenkins-cli.jar -s http://localhost:8080 -auth @credentials console 'GitHub Maven App/Maven'
Started from command line by
                            admin
Running as SYSTEM
Building in workspace /var/lib/jenkins/workspace/Maven App
Parsing POMs
.
.
.

Importing/Exporting Existing Jobs

All configurations in Jenkins are defined with XML. This makes it easy to reuse existing configurations or import new ones. Besides project definitions, global configurations like plugins and credentials are all written in XML.

jenkins-cli provides support for importing and exporting with get-job and create-job commands which take XML as their parameter. The following code duplicates our Maven project into a new one:

$ java -jar jenkins-cli.jar -s http://localhost:8080 -auth @credentials get-job 'Maven App' > myMavenProject.xml
$ java -jar jenkins-cli.jar -s http://localhost:8080 -auth @credentials create-job 'new Maven App' < myMavenProject.xml

Conclusion

The key takeaway here is that of the power of automation. It is well worth it to put in some time and effort in a powerful tool, carefully set up everything and never again waste time doing manual work. The return on investment is invaluable.

This guide is meant to be an introduction to Jenkins' power and extensibility. With the basic principles well understood, your Jenkins knowledge will skyrocket once you start experimenting with different projects, mixing and matching different build tools, languages and build environments.

If you're eager to continue your Jenkins journey - you can check our guide on Scheduling Jobs in Jenkins or Setting up Gated Check-ins for Spring Boot Projects on GitHub.