Apache Kafka Series - Kafka Cluster Setup and Administration

Apache Kafka Series - Kafka Cluster Setup and Administration

by Stéphane Maarek
Released May 2018
Publisher(s): Packt Publishing
ISBN: 9781789346534

Read it now on the O’Reilly learning platform with a 10-day free trial.

O’Reilly members get unlimited access to books, live events, courses curated by job role, and more from O’Reilly and nearly 200 top publishers.

Start your free trial

Video description

This course is designed for system administrators, architects and developers. It includes a step-by-step guide for installing a ZooKeeper quorum and a Kafka cluster. Hands-on training on Kafka administration is also provided.This course has Amazon Web Services for our cloud environment, which is used by many companies around the world. AWS knowledge is a pre-requisite. This course reveals exactly how your Kafka cluster on multiple machines should be set up and configured. We will start with understanding the Kafka basics, cluster size and the configuration. We will also have hands-on learning on the AWS set up, single broker set up, multi broker set up, testing the cluster, Kafka manager (cluster management), demonstrating Kafka resiliency and so on. We will enlighten you on Kafka performance with respect to I/Os, network, RAM, CPU, OS (Operating System) and so on. Includes hands-on learning on how to run Kafka in production on AWS, how to change a Kafka broker configuration and we will also cover the advanced Kafka configurations.

What You Will Learn

  • Setup a Zookeeper and Kafka cluster on three machines in AWS
  • Learn how to deploy Kafka in Production and understand the target architecture for clusters in AWS
  • Setup ZooKeeper Cluster, learn its role for Kafka and usage
  • Setup Kafka in Cluster-Mode with 3 brokers, including configuration, usage and maintenance
  • Shutdown and Recover Kafka brokers, to overcome the common Kafka broker problems
  • Configure Kafka Cluster with production settings and optimizations for better performances based on your workload
  • Setup web administration tools using Docker: ZooNavigator, Kafka Manager, Confluent Schema Registry, Confluent REST Proxy, Landoop Kafka Topics UI

Administer Kafka using Kafka Manager

Audience

System administrators or architects, who would like to learn or setup a Kafka cluster on multiple servers. Developers who want to understand the inner depth of a Kafka setup. This course is designed for expert students and should not be taken until all the pre-requisites are met

About The Author

Stéphane Maarek: Stéphane Maarek is a solutions architect, consultant, and software developer who has a particular interest in all things related to big data and analytics. He is also a bestseller instructor on Udemy for his courses on Apache Kafka, Apache NiFi, and AWS Lambda. He loves Apache Kafka and regularly contributes to the Apache Kafka project.

Stéphane has also written a guest blog post that was featured on the Confluent website, the company behind Apache Kafka. He is also an AWS Certified Solutions Architect and has many years of experience with technologies such as Apache Kafka, Apache NiFi, Apache Spark, Hadoop, PostgreSQL, Tableau, Spotfire, Docker, Ansible, and more.

Table of contents

  1. Chapter 1 : Course Introduction and Pre Requisites
    1. Course Objectives
    2. What is not covered in this course?
    3. Pre-Requisites: This is a Challenging course!
    4. Who is this course for?
    5. About me!
    6. 5 Golden Course Rules
  2. Chapter 2 : Target Architecture
    1. Cloud Provider: AWS Alternatives
    2. Zookeeper Quorum Architecture
    3. Kafka Cluster Architecture
    4. Final Architecture (with Web Tools)
    5. Note on IP addresses and DNS
  3. Chapter 3 : Zookeeper Quorum Setup
    1. What is Zookeeper?
    2. Zookeeper Role in a Kafka Cluster
    3. Zookeeper Quorum Sizing
    4. Zookeeper Configuration
    5. Hands-On: Zookeeper AWS Setup
    6. How to SSH Into our Servers?
    7. Hands-On: Single Zookeeper Machine Setup
    8. Hands-On: Zookeeper Service Setup
    9. Using the Zookeeper Command Line Interface (CLI)
    10. Hands-On: Zookeeper Quorum setup - Part 1 AWS
    11. Hands-On: Zookeeper Quorum Setup - Part 2 Zookeeper
    12. Zookeeper Four Letter Words
    13. Zookeeper Internal File System
    14. Factors impacting Zookeeper Performance
    15. Note: Zookeeper in AWS
    16. Hands-On: Web tools AWS machine Docker setup
    17. Management Tools for Zookeeper Zoonavigator demo
  4. Chapter 4 : Kafka Cluster Setup
    1. Kafka Basics
    2. Kafka Cluster Size Discussions
    3. Kafka Configuration
    4. Hands-On: Kafka AWS Setup
    5. Hands-On: Single Kafka Broker Setup
    6. Hands-On: Running Kafka Commands
    7. Hands-On: Kafka Multi Broker (Cluster) Setup
    8. Hands-On: Testing the Kafka Cluster
    9. Can I connect to my Kafka cluster?
    10. advertised.listeners setting - most important setting
    11. Hands-On: Kafka Manager (Cluster Management)
    12. Hands-On: Demonstrating Kafka Resiliency
    13. Kafka Performance: I/O
    14. Kafka Performance: Network
    15. Kafka Performance: RAM
    16. Kafka Performance: CPU
    17. Kafka Performance: OS (Operating System)
    18. Kafka Performance: Other
    19. Running Kafka in Production on AWS
    20. Hands-On: How to change a Kafka Broker Configuration
    21. Advanced Kafka Configuration
    22. Landoop Kafka Topics UI, Confluent REST Proxy, Confluent Schema Registry Setup
  5. Chapter 5 : Next steps!
    1. What's Next? Learning further

Product information

  • Title: Apache Kafka Series - Kafka Cluster Setup and Administration
  • Author(s): Stéphane Maarek
  • Release date: May 2018
  • Publisher(s): Packt Publishing
  • ISBN: 9781789346534