You are previewing Mastering CoreOS.
O'Reilly logo
Mastering CoreOS

Book Description

Create production CoreOS clusters and master the art of deploying Container-based microservices

About This Book

  • Confidently deploy distributed applications and effectively manage distributed infrastructure using Containers and CoreOS

  • Build secure, scalable CoreOS clusters to deploy distributed applications using open source technologies and industry best practices

  • Build secure, scalable CoreOS clusters to deploy distributed applications using open source technologies and industry best practices

  • Who This Book Is For

    This book is intended for Cloud application developers and Cloud infrastructure administrators. If you are looking to deploy a CoreOS cluster or you already have a CoreOS cluster that you want to manage better in terms of performance, security, and scaling, then this book is perfect for you. To follow the hands-on stuff, you need to have a Google and an AWS Cloud account and be able to run CoreOS VMs on your machine. A basic understanding of public and private clouds, Containers, Docker, Linux, and CoreOS is required.

    What You Will Learn

  • Install CoreOS on a VM, on the Cloud, and bare metal, and find out how to keep your cluster secure and up to date

  • Configure and troubleshoot key CoreOS services, such as etcd, systemd, and fleet, for distributed application deployment

  • Study container networking using CoreOS Flannel and other solutions, such as Docker libnetwork, Weave, and Calico

  • Explore the container filesystem and container volume management using Docker volume, NFS, GlusterFS, and Flocker

  • Get to know the internals of container technologies such as Docker, Rkt, and Container orchestration using Openstack, Kubernetes and Docker native solutions

  • Troubleshoot CoreOS cluster and Containers using monitoring and logging tools and master production techniques such as staging, security, and automation

  • In Detail

    CoreOS makes Google and Amazon-style Cloud infrastructure available for anyone building their own private Cloud. This book covers the CoreOS internals and the technologies used in the deployment of container-based distributed applications. It starts with an overview of CoreOS and distributed application development while sharing knowledge on related technologies. Critical CoreOS services and networking and storage considerations for CoreOS are covered next.

    In latter half of the book, you will learn about Container runtime systems such as Docker and Rkt and Container Orchestration using Kubernetes. You will also find out about the integration of popular orchestration solutions such as OpenStack, the AWS Container service, and the Google Container Engine with CoreOS and Docker. Lastly, we cover troubleshooting as well as production considerations.

    Style and approach

    This is an easy-to–follow, comprehensive guide that covers both basic and advanced concepts. All topics are illustrated with practical examples that can be used in both simulation and production environments.

    Downloading the example code for this book. You can download the example code files for all Packt books you have purchased from your account at http://www.PacktPub.com. If you purchased this book elsewhere, you can visit http://www.PacktPub.com/support and register to have the code file.

    Table of Contents

    1. Mastering CoreOS
      1. Table of Contents
      2. Mastering CoreOS
      3. Credits
      4. About the Author
      5. Acknowledgments
      6. About the Reviewers
      7. www.PacktPub.com
        1. Support files, eBooks, discount offers, and more
          1. Why subscribe?
          2. Free access for Packt account holders
      8. Preface
        1. What this book covers
        2. What you need for this book
        3. Who this book is for
        4. Conventions
        5. Reader feedback
        6. Customer support
          1. Downloading the example code
          2. Errata
          3. Piracy
          4. Questions
      9. 1. CoreOS Overview
        1. Distributed application development
          1. Components of distributed application development
          2. Advantages and disadvantages
        2. A minimalist Container-optimized OS
        3. Containers
          1. Technology
          2. Advantages
          3. An overview of Docker architecture
          4. Advantages of Docker
        4. CoreOS
          1. Properties
          2. Advantages
          3. Supported platforms
          4. CoreOS components
            1. Kernel
            2. Systemd
              1. Systemd units
                1. Etcd2.service
                2. Fleet.service
                3. Docker.service
              2. The procedure to start a simple systemd service
              3. Demonstrating systemd HA
            3. Etcd
              1. Etcd discovery
              2. Cluster size
              3. Etcd cluster details
              4. Simple set and get operations using etcd
            4. Fleet
              1. The Fleet architecture
              2. A Fleet scheduling example
                1. A global unit example
                2. Scheduling based on metadata
              3. Fleet HA
            5. Flannel
              1. A Flannel service unit
            6. Rkt
          5. The CoreOS cluster architecture
            1. The development cluster
            2. The production cluster
        5. Docker versus Rkt
          1. History
          2. APPC versus OCI
          3. The current status
          4. Differences between Docker and Rkt
        6. A workflow for distributed application development with Docker and CoreOS
        7. Summary
        8. References
        9. Further reading and tutorials
      10. 2. Setting up the CoreOS Lab
        1. Cloud-config
          1. The CoreOS cloud-config file format
            1. The main sections of cloud-config
            2. A sample CoreOS cloud-config
          2. The cloud-config validator
            1. A hosted validator
              1. Valid cloud-config
              2. Invalid cloud-config
            2. The cloudinit validator
          3. Executing cloud-config
        2. The CoreOS cluster with Vagrant
          1. Steps to start the Vagrant environment
          2. Important files to be modified
            1. Vagrantfile
            2. User-data
            3. Config.rb
          3. Vagrant – a three-node cluster with dynamic discovery
            1. Generating a discovery token
            2. Steps for cluster creation
          4. Vagrant – a three-node cluster with static discovery
          5. Vagrant – a production cluster with three master nodes and three worker nodes
        3. A CoreOS cluster with AWS
          1. AWS – a three-node cluster using Cloudformation
          2. AWS – a three-node cluster using AWS CLI
        4. A CoreOS cluster with GCE
          1. GCE – a three-node cluster using GCE CLI
        5. CoreOS installation on Bare Metal
        6. Basic debugging
          1. journalctl
          2. systemctl
          3. Cloud-config
          4. Logging from one CoreOS node to another
          5. Important files and directories
          6. Common mistakes and possible solutions
        7. Summary
        8. References
        9. Further reading and tutorials
      11. 3. CoreOS Autoupdate
        1. The CoreOS release cycle
        2. The partition table on CoreOS
        3. CoreOS automatic update
        4. Update and reboot services
          1. Update-engine.service
            1. Debugging update-engine.service
          2. Locksmithd.service
            1. Locksmith strategy
              1. The etcd-lock scheme
              2. Reboot
              3. Best-effort
              4. Off
            2. Groups
            3. Locksmithctl
            4. Debugging locksmithd.service
        5. Setting update options
          1. Using cloud-config
          2. Manual configuration
        6. Update examples
          1. Updating within the same release channel
          2. Updating from one release channel to another
        7. CoreUpdate
        8. Vagrant CoreOS update
        9. Summary
        10. References
        11. Further reading and tutorials
      12. 4. CoreOS Primary Services – Etcd, Systemd, and Fleet
        1. Etcd
          1. Versions
          2. Installation
            1. Standalone installation
          3. Accessing etcd
            1. REST
            2. Etcdctl
          4. Etcd configuration
          5. Etcd operations
          6. Etcd tuning
          7. Etcd proxy
          8. Adding and removing nodes from a cluster
          9. Node migration and backup
          10. Etcd security
            1. Certificate authority – etcd-ca
            2. Installing etcd-ca
            3. Etcd secure client-to-server communication using a server certificate
            4. Etcd secure client-to-server communication using server certificate and client certificate
            5. A secure cloud-config
          11. Authentication
          12. Etcd debugging
        2. Systemd
          1. Unit types
          2. Unit specifiers
          3. Unit templates
          4. Drop-in units
            1. Default cloud-config drop-in units
            2. Cloud-config custom drop-in units
            3. Runtime drop-in unit – specific parameters
            4. Runtime drop-in unit – full service
          5. Network units
        3. Fleet
          1. Installation
          2. Accessing Fleet
            1. Local fleetctl
            2. Remote fleetctl
            3. Remote fleetctl with an SSH tunnel
            4. Remote HTTP
            5. Using etcd security
          3. Templates, scheduling, and HA
          4. Debugging
          5. Service discovery
            1. Simple etcd-based discovery
            2. Sidekick discovery
            3. ELB service discovery
        4. Summary
        5. References
        6. Further reading and tutorials
      13. 5. CoreOS Networking and Flannel Internals
        1. Container networking basics
        2. Flannel
          1. Manual installation
          2. Installation using flanneld.service
          3. Control path
          4. Data path
          5. Flannel as a CNI plugin
          6. Setting up a three-node Vagrant CoreOS cluster with Flannel and Docker
          7. Setting up a three-node CoreOS cluster with Flannel and RKT
          8. An AWS cluster using Flannel
            1. An AWS cluster using VXLAN networking
            2. An AWS cluster using AWS-VPC
          9. A GCE cluster using Flannel
            1. GCE cluster using VXLAN networking
            2. A GCE cluster using GCE networking
          10. Experimental multitenant networking
          11. Experimental client-server networking
            1. Setting up client-server Flannel networking
        3. Docker networking
          1. Docker experimental networking
            1. A multinetwork use case
            2. The Docker overlay driver
            3. The external networking calico plugin
          2. The Docker 1.9 update
        4. Other Container networking technologies
          1. Weave networking
          2. Calico networking
          3. Setting up Calico with CoreOS
          4. Kubernetes networking
        5. Summary
        6. References
        7. Further reading and tutorials
      14. 6. CoreOS Storage Management
        1. Storage concepts
        2. The CoreOS filesystem
          1. Mounting the AWS EBS volume
          2. Mounting NFS storage
            1. Setting up NFS server
            2. Setting up the CoreOS node as a client for the NFS
        3. The container filesystem
          1. Storage drivers
          2. Docker and the Union filesystem
        4. Container data
          1. Docker volumes
            1. Container volume
            2. Volumes with the host mount directory
            3. A data-only container
            4. Removing volumes
          2. The Docker Volume plugin
            1. Flocker
              1. Flocker volume migration using AWS EBS as a backend
              2. Flocker volume migration using the ZFS backend
              3. Flocker on CoreOS with an AWS EBS backend
              4. Flocker recent additions
            2. GlusterFS
              1. Setting up a GlusterFS cluster
              2. Setting up GlusterFS for a CoreOS cluster
              3. Accessing GlusterFS using the Docker Volume plugin
          3. Ceph
          4. NFS
            1. Container data persistence using NFS
        5. The Docker 1.9 update
        6. Summary
        7. References
        8. Further reading and tutorials
      15. 7. Container Integration with CoreOS – Docker and Rkt
        1. Container standards
          1. App container specification
            1. The Container image format
              1. Container image discovery
                1. Simple discovery
                2. Meta discovery
              2. The app container executor
              3. App container pods
              4. The app container metadata service
            2. APPC tools
              1. Actool
              2. Acbuild
              3. Docker2aci
            3. Open Container Initiative
              1. Runc
              2. The relationship of OCI with APPC
                1. OCI and APPC latest updates
            4. Libnetwork
            5. CNI
            6. The relationship between Libnetwork and CNI
            7. Cloud Native Computing Foundation
        2. Docker
          1. The Docker daemon and an external connection
          2. Dockerfile
          3. The Docker Image repository
            1. Creating your own Docker registry
            2. Continuous integration
          4. The Docker content trust
            1. Pushing secure image
            2. Pulling secure image
            3. Pulling same image with no security
          5. Container debugging
            1. Logs
            2. Login inside Container
            3. Container properties
            4. Container processes
            5. The Container's CPU and memory usage
        3. Rkt
          1. Basic commands
            1. Fetch image
            2. List images
            3. Run image
            4. List pods
            5. Garbage collection
            6. Delete image
            7. Export image
            8. The nginx container with volume mounting and port forwarding
            9. Pod status
          2. Rkt image signing
          3. Rkt with systemd
          4. Rkt with Flannel
        4. Summary
        5. References
        6. Further reading and tutorials
      16. 8. Container Orchestration
        1. Modern application deployment
        2. Container Orchestration
          1. Kubernetes
            1. Concepts of Kubernetes
              1. Pods
              2. Networking
              3. Services
            2. Kubernetes architecture
            3. Kubernetes installation
              1. Non-Coreos Kubernetes installation
              2. Kubectl installation
              3. Vagrant installation
              4. GCE installation
              5. AWS installation
            4. An example of a Kubernetes application
            5. Kubernetes with Rkt
            6. Kubernetes 1.1 update
          2. Docker Swarm
            1. The Docker Swarm installation
            2. An example of Docker Swarm
          3. Mesos
          4. Comparing Kubernetes, Docker Swarm, and Mesos
        3. Application definition
          1. Docker-compose
            1. A single-node application
            2. A multinode application
        4. Packaged Container Orchestration solutions
          1. The AWS Container service
            1. Installing ECS and an example
          2. Google Container Engine
            1. Installing GCE and an example
          3. CoreOS Tectonic
        5. Summary
        6. References
        7. Further reading and tutorials
      17. 9. OpenStack Integration with Containers and CoreOS
        1. An overview of OpenStack
        2. CoreOS on OpenStack
          1. Get OpenStack Kilo running in Devstack
          2. Setting up keys and a security group
          3. Setting up external network access
          4. Download the CoreOS image and upload to Glance
          5. Updating the user data to be used for CoreOS
        3. OpenStack and Containers
          1. The Nova Docker driver
            1. Installing the Nova Driver
          2. Installing Docker
            1. Install the Nova Docker plugin
          3. The Devstack installation
          4. The Heat Docker plugin
            1. Installing the Heat plugin
          5. Magnum
            1. The Magnum architecture
            2. Installing Magnum
        4. Container networking using OpenStack Kuryr
          1. OpenStack Neutron
          2. Containers and networking
          3. OpenStack Kuryr
          4. The current state and roadmap of Kuryr
        5. Summary
        6. References
        7. Further reading and tutorials
      18. 10. CoreOS and Containers – Troubleshooting and Debugging
        1. CoreOS Toolbox
        2. Other CoreOS debugging tools
        3. Container monitoring
          1. Sysdig
            1. Examples of Sysdig
            2. Csysdig
            3. The Sysdig cloud
            4. Kubernetes integration
          2. Cadvisor
        4. The Docker remote API
        5. Container logging
          1. Docker logging drivers
            1. The JSON-file driver
            2. The Syslog driver
            3. The journald driver
          2. Logentries
            1. Exporting CoreOS journal logs
            2. Container logs
        6. Summary
        7. References
        8. Further reading and tutorials
      19. 11. CoreOS and Containers – Production Considerations
        1. CoreOS cluster design considerations
          1. The update strategy
          2. Cluster considerations
        2. Distributed infrastructure design considerations
          1. Service discovery
            1. Service discovery using Registrator and Consul
            2. Dynamic load balancing
              1. Load balancing with confd and nginx
              2. Load balancing with HAdiscover and HAproxy
          2. Deployment patterns
            1. The Sidecar pattern
            2. The Ambassador pattern
            3. The Adapter pattern
            4. Rolling updates with the Canary pattern
          3. Containers and PaaS
          4. Stateful and Stateless Containers
        3. Security
          1. Secure the external daemons
          2. SELinux
          3. Container image signing
        4. Deployment and automation
          1. Continuous Integration and Continuous Delivery
          2. Ansible integration with CoreOS and Docker
            1. Using Ansible to manage CoreOS
            2. Using Ansible to manage Docker Containers
            3. Ansible as a Container
            4. Using Ansible to install Docker
        5. The CoreOS roadmap
          1. Ignition
          2. DEX
          3. Clair
        6. The Docker roadmap
          1. Tutum
          2. UCP
          3. Nautilus
        7. Microservices infrastructure
          1. Platform choices
          2. Solution providers
        8. Summary
        9. References
        10. Further reading and tutorials
      20. Index