Skip to main content

CI, CD with Docker, Jenkins and Tutum

In this blog, I will give an overview of Continuous Integration (CI) and Continuous Deployment (CD) and cover few CI, CD Use cases with Docker, Jenkins and Tutum. Docker provides Container runtime and tools around Containers to create a Container platform. Jenkins is a CI/CD application to build, test and deploy applications. Tutum is a SaaS Container platform that can be used to build, deploy and manage Docker Containers. I have covered overview of Tutum in previous blog. There are Use cases where these applications work well with each other and the Use cases in this blog will illustrate that.

CI/CD Overview

Traditional approach of releasing software has the following problems:
  • Software release cycles were spaced apart which caused new features taking longer time to reach the customers.
  • Majority of the process of getting software from development stage to production was manual.
  • Considering the different deployment scenarios, it was difficult to guarantee that software worked in all environments and configurations.
Containers have tried to mitigate some of the problems above. By using microservices and Container approach, it is guaranteed that the application will behave similar in development and production stages. Process automation and appropriate testing is still needed for Container based environment.
Continuous Integration refers to the process of making an executable or a Container image automatically after developer has done the UT and commit.
Continuous delivery refers to the process of taking the developer built image, setting up the staging environment for testing the image and deploying it successfully for production.
Following diagram show the different stages of CI, CD cycle.
cicd1
Following are some notes on the above diagram:
  • The first row in the above diagram captures the steps for CI and the second row are the steps for CD.
  • It is possible that some of the blocks above could be split into multiple items and the ordering could be customized.
  • The ultimate goal is to create an automated workflow of the steps above and handling failures gracefully.

Docker, Jenkins and Tutum for CI/CD

Why Docker is suited for CI/CD?

  • Ability to create consistent build environments
  • Build environments can be created and removed with ease
  • Ability to run multiple isolated build jobs
  • Create consistent staging and production deployment environment
  • Assists with different upgrade patterns(like Rolling upgrade with Canary pattern)

Docker, Jenkins Integration

Jenkins has been doing CI/CD for close to 10 years much before Containers became prevalent. With Docker and Container adoption increasing, Jenkins saw the value in integrating Jenkins with Docker. Following are some integration possibilities.
  • Jenkins application is available as Docker Container.
  • Jenkins can be used to automate creation of Container images.
  • Jenkins can be used to run build jobs inside a Container. This allows for parallel jobs to be run on a single Jenkins slave host with full isolation between jobs.
Following are the Docker plugins available within Jenkins that allows for close integration with Docker as described here.
  • Docker Workflow– provides first-class support for Jenkins Workflow to build real world CD pipelines for containerized applications using Jenkins and Docker
  • Build and Publish– builds projects from a Dockerfile and pushes a tested and proven gold image to Docker Hub
  • Docker Hub Notification – triggers downstream jobs when a tagged container is pushed to Docker Hub
  • Docker Traceability– identifies which build pushed a particular container that is running in production and displays that on the Jenkins builds page
  • Docker Slaves– uses Docker containers as standardized build environments shared between Jenkins masters to improve isolation and elasticity
  • Docker Custom Build Environment– specifies customized build environments as Docker containers

Comparing Jenkins with Tutum

Both Jenkins and Tutum can be used for CI, CD applications. Following are some comparisons between the two.
  • Jenkins provides better integration with third-party testing and code analysis tools.
  • Tutum is available only as SaaS. Jenkins can be run on-premise.
  • Tutum provides its own registry, inherent load balancing and service discovery capabilities and rolling upgrade facilities.
  • For pure Container based CI/CD, Tutum would be better. If there is a need to integrate legacy and Container based applications and also use third-party tools, Jenkins is the only option.

Use cases

My environment:
  • Ubuntu 14.04 VM running on a Windows host.
  • Docker engine running on Ubuntu exposing the Docker service on TCP port 4243(Docker service can be exposed by:
Edit  “/etc/default/docker”:
DOCKER_OPTS="-D -H tcp://0.0.0.0:4243 -H unix:///var/run/docker.sock"
Restart docker
  • Jenkins is run as a Docker Container on Ubuntu host. (docker run -p 8080:8080 -p 50000:50000 jenkins)
  • I have used the same Ubuntu VM also as my Jenkins slave host. This is typically not preferred. For testing purposes, this is fine.

Jenkins and Tutum for creating and deploying Docker Containers

Following diagram describes a sample application:
cicd3
Following are some notes on the above application:
  • The application is written in Python as a Flask application. The application starts a web service and counts the number of web hits.  The application source code includes Dockerfile to build the Container and docker-compose.yml file to link the application Container to database.
  • Source code for the application is stored in Github.
  • There are 3 integrations done from Github for this Use case. Any commit to Github will trigger all the 3 integrations simultaneously.
  • The first integration is to Docker hub where a job is created inside Docker hub to automatically create new Container image and push to Docker hub repository.
  • The second integration is to Jenkins where Jenkins creates a Docker slave Container to build new Container image and push it to Docker hub. Containers with the new Container image also get deployed with the docker-compose file.
  • The third integration is to Tutum where Tutum builds the new Container image and automatically deploys the Container using the Stackfile specified in Tutum.
Integration with Docker hub:
The step involves creating Automated build from Docker hub. Docker hub has integrations with Github and Bitbucket.
Integration with Jenkins:
Following are the steps involved:
  • Run Jenkins Container
  • Install Github and Docker plugins in Jenkins
  • Create Docker slave with a specific label using Docker plugin. Docker slave will be used to build Docker image. Docker image can be built directly on Jenkins host. I prefer doing the build inside a Container as it gives other isolation advantages associated with Containers. To use Docker inside slave Container and build Containers, we need to mount Docker.sock from host machine so that Containers are created in the slave machine directly. This can be done by mounting the following volumes:
/var/run/docker.sock:/var/run/docker.sock
/usr/bin/docker:/usr/bin/docker
/usr/bin/docker-compose:/usr/bin/docker-compose
  • The slave Container needs to have the necessary libraries and packages needed for building the particular application that we are targeting. It also needs to have Git installed to download the source code. Following is the Dockerfile for my “myjenkinsslaveprojectinJava” Container. This is not a Java project use case, I have used this Container since it had some packages I needed.
FROM evarga/jenkins-slave
# if we want to install via apt
USER root
RUN apt-get update && apt-get -yy install sudo
RUN apt-get -yy install git
RUN apt-get install -yy software-properties-common
RUN add-apt-repository "deb http://ppa.launchpad.net/natecarlson/maven3/ubuntu precise main"
RUN apt-get update
RUN apt-get -yy --force-yes install maven3
RUN ln -s /usr/share/maven3/bin/mvn /usr/bin/mvn
  • Create Jenkins freestyle project. In the project configuration, we specify that the build project needs to be run as a slave Container with the label that we used above when creating the Docker slave. We also specify the Github project from where the source code needs to be taken. We can setup the trigger based on the commit or based on periodic polling.
  • Once the Container image is built, Jenkins uses Docker build and publish plugin to publish the Container image to Docker hub.
  • Once the Container image is pushed to Docker hub, Jenkins starts the application using docker-compose.
Integration with Tutum
Following are the steps involved with Tutum integration:
  • Create public cloud node or your own node inside Tutum
  • Create a repository linking to Github where the source code is present. Tutum provides a private repository for Container images.
  • Create Stack using the Container image.
  • Using Github integration, Tutum automatically pushes a new Container image when there is any change to source code.
  • Since Tutum stackfile uses the Container image, any new updates to Container image will automatically trigger Tutum to redeploy the Containers.
Following video walks through the application detail:

Jenkins slave with multiple Docker executors

In this Use case, we will use Jenkins to create 2 Docker slave Containers, one for building a C programming project(https://github.com/smakam/primes)  and another for building a Java programming project(https://github.com/smakam/game-of-life). These are public projects that I have forked for this Use case.
Following are the steps involved:
  • Create Docker image for Java project. I have used Dockerfile mentioned in the previous use case.
  • Create Docker image for C project. I used the following Dockerfile where I installed gcc and make.
FROM evarga/jenkins-slave
# if we want to install via apt
USER root
RUN apt-get update && apt-get -yy install sudo
RUN apt-get -yy install git
RUN apt-get -yy install gcc automake autoconf make
  • Create Docker slave for Java project using Java Docker image and identify this by a unique label.
  • Create Docker slave for C project using Java Docker image and identify this by a unique label.
  • Create C build project using the corresponding Docker slave label. Github project source code path is specified here along with the build trigger mechanism.
  • Create Java build project using the corresponding Docker slave label. Github project source code path is specified here along with the build trigger mechanism.
When project is built, Docker Container gets spawned where build is executed. Once build is executed, binaries are copied to source volumes and Container exits. This allows for builds to always run in a clean environment giving predictable results.
Following video gives a demo of this Use case:  Jenkins slaves with multiple docker

References



Popular posts from this blog

How to read or extract text data from passport using python utility.

Hi ,  Lets get start with some utility which can be really helpful in extracting the text data from passport documents which can be images, pdf.  So instead of jumping to code directly lets understand the MRZ, & how it works basically. MRZ Parser :                 A machine-readable passport (MRP) is a machine-readable travel document (MRTD) with the data on the identity page encoded in optical character recognition format Most travel passports worldwide are MRPs.  It can have 2 lines or 3 lines of machine-readable data. This method allows to process MRZ written in accordance with ICAO Document 9303 (endorsed by the International Organization for Standardization and the International Electrotechnical Commission as ISO/IEC 7501-1)). Some applications will need to be able to scan such data of someway, so one of the easiest methods is to recognize it from an image file. I 'll show you how to retrieve the MRZ information from a picture of a passport using the PassportE
Read more

How to generate class diagrams pictures in a Django/Open-edX project from console

A class diagram in the Unified Modeling Language ( UML ) is a type of static structure diagram that describes the structure of a system by showing the system’s classes, their attributes, operations (or methods), and the relationships among objects. https://github.com/django-extensions/django-extensions Step 1:   Install django extensions Command:  pip install django-extensions Step 2:  Add to installed apps INSTALLED_APPS = ( ... 'django_extensions' , ... ) Step 3:  Install diagrams generators You have to choose between two diagram generators: Graphviz or Dotplus before using the command or you will get: python manage.py graph_models -a -o myapp_models.png Note:  I prefer to use   pydotplus   as it easier to install than Graphviz and its dependencies so we use   pip install pydotplus . Command:  pip install pydotplus Step 4:  Generate diagrams Now we have everything installed and ready to generate diagrams using the comm
Read more

How to Remove course from Open-edX

Go to vagrant  => 1. In the edx-platform directory:  - cd /edx/app/edxapp/edx-platform 2. Run the following Django management command:   - sudo -u www-data /edx/bin/python.edxapp /edx/bin/manage.edxapp lms dump_course_ids --settings aws    - sudo -u www-data /edx/bin/python.edxapp /edx/bin/manage.edxapp lms dump_course_ids --settings=devstack 3. Find the course ID which you'd like to delete in the resulting list of course IDs. 4. Copy the course ID into the following command and run it:  - sudo -u www-data /edx/bin/python.edxapp /edx/bin/manage.edxapp cms delete_course <COURSE_ID> --settings aws  -   sudo -u www-data /edx/bin/python.edxapp /edx/bin/manage.edxapp cms delete_course <COURSE_ID> --settings=devstack  - You'll be asked to verify the deletion . To verify the deletion, run the command from step 2 above and ensure that the course ID is not in the list. Help reference : https://openedx.atlassian.net/wiki/spa
Read more