Scaling Jenkins with Docker and Apache Mesos
Continuous integration and continuous delivery at scale
Join Carlos Sanchez for a study in running Jenkins at scale. He’ll share his experience using Docker and Apache Mesos to create one of the biggest Jenkins clusters to date. You’ll drill down into the details with Carlos to get a better understanding of how Apache Mesos works. Together you’ll explore the challenges of running containerized and distributed applications (particularly JVM ones) through a real-world use case. By the end of this course, you’ll have a solid grounding in using these popular open source technologies for continuous integration and continuous delivery at scale.
This two-hour course uses a mix of lectures, exercises, and Q&A.
What you'll learn-and how you can apply it
By the end of this live, online course, you’ll understand:
- How Apache Mesos works and how Docker containers are executed in a Mesos cluster
- How Jenkins can use a Mesos cluster as a provider to provision build agents on demand
- How Java applications behave inside a Docker container
And you’ll be able to:
- Create a Apache Mesos cluster for local development using Docker Compose
- Create Jenkins jobs that are executed dynamically based on demand
- Use Jenkins Pipelines to execute jobs in one or more Docker containers
This training course is for you because...
- You’re a build/release engineer or are interested in deploying Docker at scale
- You work with Jenkins or other Java applications
- You want to become a Docker expert!
You will need to have one of the following versions of Docker and Docker Compose installed on your computer:
- Linux: at least Docker 1.11 and Docker Compose 1.7.1
- Mac: Docker for Mac beta 1.12, which should already include Docker Compose
- Windows: Docker for Windows beta 1.12, which should already include Docker Compose
Note: Be sure to use Docker for Mac and Docker for Windows beta versions (not Docker Toolbox) as they expose the Docker containers under localhost as the Linux version does. All the examples in this course will use localhost for a consistent experience.
Materials and downloads needed
Source from https://github.com/carlossg/scaling-jenkins-with-docker-and-mesos and its git submodules ( git submodule init and git submodule update after git clone ) After installation of Docker, pull all the images using: docker-compose -f mesos-jenkins/docker-compose.yml pull
Also pull any images you may want to use as Jenkins build agents, at least:
docker pull csanchez/java-with-docker-client:8-jdk-1.12.0
docker pull java:8-jdk
docker pull golang:1.6
docker pull maven:3.3.9-jdk-8
docker pull maven:3-onbuild
About your instructor
Carlos Sanchez specializes in software automation, from build tools to continuous delivery, following the DevOps philosophy. Involved in open source for more than 10 years, he is a member of the ASF as well as other open source groups and has contributed to several projects including Jenkins, Apache Maven, and Puppet. Carlos has spoken at several conferences around the world such as ApacheCon, JavaOne, and FOSDEM. He currently works at CloudBees scaling the Jenkins platform.
The timeframes are only estimates and may vary according to how the class is progressing
- Scaling Jenkins
- A Docker-based architecture
- Cluster scheduling with Docker containers
- Containers’ use of memory, CPU, and networking
- Exercise: Effects of memory limits in containers
- Jenkins Mesos Plugin
- Exercise: Setting up a local Mesos cluster with Docker Compose, Jenkins, and the Mesos plugin
- Exercise: Using Jenkins pipeline to build with any Docker image, for example golang
- Scaling Jenkins to the next level: Description of architecture and issues faced building one of the largest Jenkins clusters to date