You are previewing Apache Kafka Cookbook.
O'Reilly logo
Apache Kafka Cookbook

Book Description

Over 50 hands-on recipes to efficiently administer, maintain, and use your Apache Kafka installation

About This Book

  • Quickly configure and manage your Kafka cluster

  • Learn how to use the Apache Kafka cluster and connect it with tools for big data processing

  • A practical guide to monitor your Apache Kafka installation

  • Who This Book Is For

    If you are a programmer or big data engineer using or planning to use Apache Kafka, then this book is for you. This book has several recipes which will teach you how to effectively use Apache Kafka. You need to have some basic knowledge of Java. If you don’t know big data tools, this would be your stepping stone for learning how to consume the data in these kind of systems.

    What You Will Learn

  • Learn how to configure Kafka brokers for better efficiency

  • Explore how to configure producers and consumers for optimal performance

  • Set up tools for maintaining and operating Apache Kafka

  • Create producers and consumers for Apache Kafka in Java

  • Understand how Apache Kafka can be used by several third party system for big data processing, such as Apache Storm, Apache Spark, Hadoop, and more

  • Monitor Apache Kafka using tools like graphite and Ganglia

  • In Detail

    This book will give you details about how to manage and administer your Apache Kafka Cluster.

    We will cover topics like how to configure your broker, producer, and consumer for maximum efficiency for your situation. Also, you will learn how to maintain and administer your cluster for fault tolerance. We will also explore tools provided with Apache Kafka to do regular maintenance operations. We shall also look at how to easily integrate Apache Kafka with big data tools like Hadoop, Apache Spark, Apache Storm, and Elasticsearch.

    Style and approach

    Easy-to-follow, step-by-step recipes explaining from start to finish how to accomplish real-world tasks.

    Downloading the example code for this book. You can download the example code files for all Packt books you have purchased from your account at http://www.PacktPub.com. If you purchased this book elsewhere, you can visit http://www.PacktPub.com/support and register to have the code file.

    Table of Contents

    1. Apache Kafka Cookbook
      1. Table of Contents
      2. Apache Kafka Cookbook
      3. Credits
      4. About the Author
      5. About the Reviewers
      6. www.PacktPub.com
        1. Support files, eBooks, discount offers, and more
          1. Why Subscribe?
          2. Free Access for Packt account holders
      7. Preface
        1. What this book covers
        2. What you need for this book
        3. Who this book is for
        4. Sections
          1. Getting ready
          2. How to do it…
          3. How it works…
          4. There's more…
          5. See also
        5. Conventions
        6. Reader feedback
        7. Customer support
          1. Downloading the example code
          2. Errata
          3. Piracy
          4. Questions
      8. 1. Initiating Kafka
        1. Introduction
        2. Setting up multiple Kafka brokers
          1. Getting ready
          2. How to do it...
          3. How it works…
          4. There's more…
          5. See also
        3. Creating topics
          1. Getting ready
          2. How to do it...
          3. How it works...
          4. There's more…
        4. Sending some messages from the console
          1. Getting ready
          2. How to do it...
          3. How it works...
          4. There's more…
        5. Consuming from the console
          1. Getting ready
          2. How to do it...
          3. How it works...
          4. There's more...
      9. 2. Configuring Brokers
        1. Introduction
        2. Configuring the basic settings
          1. Getting ready
          2. How to do it...
          3. How it works…
          4. There's more…
        3. Configuring threads and performance
          1. Getting ready
          2. How to do it...
          3. How it works…
          4. There's more...
        4. Configuring the log settings
          1. Getting ready
          2. How to do it...
          3. How it works…
          4. There's more…
          5. See also
        5. Configuring the replica settings
          1. Getting ready
          2. How to do it...
          3. How it works…
          4. There's more...
        6. Configuring the ZooKeeper settings
          1. Getting ready
          2. How to do it…
          3. How it works…
          4. See also
        7. Configuring other miscellaneous parameters
          1. Getting ready
          2. How to do it...
          3. How it works…
          4. See also
      10. 3. Configuring a Producer and Consumer
        1. Introduction
        2. Configuring the basic settings for producer
          1. Getting ready
          2. How to do it...
          3. How it works…
        3. Configuring the thread and performance for producer
          1. Getting ready
          2. How to do it...
          3. How it works…
          4. See also
        4. Configuring the basic settings for consumer
          1. Getting ready
          2. How to do it...
          3. How it works…
        5. Configuring the thread and performance for consumer
          1. Getting ready
          2. How to do it...
          3. How it works…
        6. Configuring the log settings for consumer
          1. Getting ready
          2. How to do it...
          3. How it works…
        7. Configuring the ZooKeeper settings for consumer
          1. Getting ready
          2. How to do it...
          3. How it works…
        8. Other configurations for consumer
          1. Getting ready
          2. How to do it...
          3. How it works…
          4. See also
      11. 4. Managing Kafka
        1. Introduction
        2. Consumer offset checker
          1. Getting ready
          2. How to do it...
          3. How it works…
        3. Understanding dump log segments
          1. Getting ready
          2. How to do it…
          3. How it works…
        4. Exporting the ZooKeeper offsets
          1. Getting ready
          2. How to do it...
          3. How it works…
        5. Importing the ZooKeeper offsets
          1. Getting ready
          2. How to do it...
          3. How it works…
        6. Using GetOffsetShell
          1. Getting ready
          2. How to do it…
          3. How it works…
        7. Using the JMX tool
          1. Getting ready
          2. How to do it...
          3. How it works…
          4. There's more…
        8. Using the Kafka migration tool
          1. Getting ready
          2. How to do it...
          3. How it works…
        9. The MirrorMaker tool
          1. Getting ready
          2. How to do it...
          3. How it works…
          4. See also
        10. Replay Log Producer
          1. Getting ready
          2. How to do it...
          3. How it works…
        11. Simple Consumer Shell
          1. Getting ready
          2. How to do it...
          3. How it works…
        12. State Change Log Merger
          1. Getting ready
          2. How to do it...
          3. How it works…
        13. Updating offsets in Zookeeper
          1. Getting ready
          2. How to do it...
          3. How it works…
        14. Verifying consumer rebalance
          1. Getting ready
          2. How to do it...
          3. How it works…
      12. 5. Integrating Kafka with Java
        1. Introduction
        2. Writing a simple producer
          1. Getting ready
          2. How to do it...
          3. How it works…
          4. See also
        3. Writing a simple consumer
          1. Getting ready
          2. How to do it...
          3. How it works…
          4. See also
        4. Writing a high-level consumer
          1. Getting ready
          2. How to do it...
          3. How it works…
          4. See also
        5. Writing a producer with message partitioning
          1. Getting ready
          2. How to do it...
          3. How it works…
          4. There's more…
        6. Multithreaded consumers in Kafka
          1. Getting ready
          2. How to do it...
          3. How it works…
      13. 6. Operating Kafka
        1. Introduction
        2. Adding and removing topics
          1. Getting ready
          2. How to do it...
          3. How it works…
          4. There's more…
          5. See also
        3. Modifying topics
          1. Getting ready
          2. How to do it...
          3. How it works…
          4. There's more…
        4. Implementing a graceful shutdown
          1. Getting ready
          2. How to do it...
          3. How it works…
        5. Balancing leadership
          1. Getting ready
          2. How to do it...
          3. How it works…
          4. There's more…
        6. Mirroring data between Kafka clusters
          1. Getting ready
          2. How to do it...
          3. How it works…
          4. There's more…
        7. Expanding clusters
          1. Getting ready
          2. How to do it...
          3. How it works…
          4. There's more…
        8. Increasing the replication factor
          1. Getting ready
          2. How to do it...
          3. How it works…
          4. There's more…
        9. Checking the consumer position
          1. Getting ready
          2. How to do it...
          3. How it works…
        10. Decommissioning brokers
          1. Getting ready
          2. How to do it...
          3. How it works…
      14. 7. Integrating Kafka with Third-Party Platforms
        1. Introduction
        2. Using Flume
          1. Getting ready
          2. How to do it...
          3. How it works…
          4. See also
        3. Using Gobblin
          1. Getting ready
          2. How to do it...
          3. How it works…
          4. See also
        4. Using Logstash
          1. Getting ready
          2. How to do it...
          3. How it works…
          4. There's more…
          5. See also
        5. Configuring Kafka for real-time
          1. Getting ready
          2. How to do it...
          3. How it works…
        6. Integrating Spark with Kafka
          1. Getting ready
          2. How to do it...
          3. How it works…
          4. There's more…
        7. Integrating Storm with Kafka
          1. Getting ready
          2. How to do it...
          3. How it works…
          4. There's more…
          5. See also
        8. Integrating Elasticsearch with Kafka
          1. Getting ready
          2. How to do it...
          3. How it works…
          4. There's more…
          5. See also
        9. Integrating SolrCloud with Kafka
          1. Getting ready
          2. How to do it...
          3. How it works…
          4. See also
      15. 8. Monitoring Kafka
        1. Introduction
        2. Monitoring server stats
          1. Getting ready
          2. How to do it…
          3. How it works…
          4. See also
        3. Monitoring producer stats
          1. Getting ready
          2. How to do it...
            1. How it works…
            2. See also
          3. Monitoring consumer stats
            1. Getting ready
            2. How to do it...
            3. How it works…
            4. See also
          4. Connecting to Graphite
            1. Getting ready
            2. How to do it...
            3. How it works…
            4. See also
          5. Monitoring with Ganglia
            1. Getting ready
            2. How to do it...
            3. How it works…
            4. See also
      16. Index