Streaming Architecture

Preface

The ability to handle and process continuous streams of data provides a considerable competitive edge. As a result, being able to take advantage of streaming data is beginning to be seen as an essential part of building a data-driven organization.

The expanding use of streaming data raises the question of how best to design systems to handle it effectively, from the ingestion from multiple sources, through a variety of uses, including streaming analytics and the question of persistence.

Emerging best practices for the design of streaming architectures may surprise you—the scope of powerful design for streaming systems extends far beyond specific real-time or near–real time applications. New approaches to streaming designs can greatly improve the efficiency of your overall organization.

Who Should Use This Book

If you already use streaming data and want to design an architecture for best performance, or if you are just starting to explore the value of streaming data, this book should be helpful. You’ll also find real-world use cases that help you see how to put these approaches to work in several different settings. For developers, you’ll also find links to sample programs.

This book is designed for both nontechnical and technical audiences, including business analysts, architects, team leaders, data scientists, and developers.

What Is Covered

In this book, we:

• Explain how to recognize opportunities where streaming data may be useful

• Show how to design streaming architecture for best results in a multiuser system

• Describe why particular capabilities should be present in the message-passing layer to take advantage of this type of design

• Explain why stream-based architectures are helpful to support microservices

• Describe particular tools for messaging and streaming analytics that best fit the requirements of a strong stream-based design.

Chapters 1–3 explain the basic aspects of strong architecture for streaming and microservices. If you are already familiar with many business goals for streaming data, you may want to start with Chapter 2, in which we describe the type of architecture that we recommend for streaming systems.

In addition to explaining the capabilities needed to support this emerging best practice, we also describe some of the currently available technologies that meet these requirements well. Chapter 4 goes into some detail on Apache Kafka, including links to sample programs provided by the authors. Chapter 5 describes another preferred technology for effective message passing known as MapR Streams, which uses the Apache Kafka API but with some additional capabilities.

Later chapters provide a deeper dive into real-world use cases that employ streaming data as well as a look forward to how this exciting field is likely to evolve.

Conventions Used in This Book

The following typographical conventions are used in this book:

Italic

Indicates new terms, URLs, email addresses, filenames, and file extensions.

Constant width

Used for program listings, as well as within paragraphs to refer to program elements such as variable or function names, databases, data types, environment variables, statements, and keywords.

Note

This icon indicates a general note.

Tip

This icon signifies a tip or suggestion.

Warning

This icon indicates a warning or caution.

Supplemental material (code examples, exercises, etc.) is available for download at: https://www.mapr.com/blog/getting-started-sample-programs-apache-kafka-09 and https://www.mapr.com/blog/getting-started-sample-programs-mapr-streams

This book is here to help you get your job done. In general, if example code is offered with this book, you may use it in your programs and documentation. You do not need to contact us for permission unless you’re reproducing a significant portion of the code. For example, writing a program that uses several chunks of code from this book does not require permission. Selling or distributing a CD-ROM of examples from O’Reilly books does require permission. Answering a question by citing this book and quoting example code does not require permission. Incorporating a significant amount of example code from this book into your product’s documentation does require permission.

We appreciate, but do not require, attribution. An attribution usually includes the title, author, publisher, and ISBN. For example: “Streaming Architecture by Ted Dunning and Ellen Friedman (O’Reilly). Copyright 2016 Ted Dunning and Ellen Friedman, 978-1-491-95392-1.”

If you feel your use of code examples falls outside fair use or the permission given above, feel free to contact us at [e-mail].

Safari® Books Online

Note

Safari Books Online is an on-demand digital library that delivers expert content in both book and video form from the world’s leading authors in technology and business.

Technology professionals, software developers, web designers, and business and creative professionals use Safari Books Online as their primary resource for research, problem solving, learning, and certification training.

Safari Books Online offers a range of plans and pricing for enterprise, government, education, and individuals.

Members have access to thousands of books, training videos, and prepublication manuscripts in one fully searchable database from publishers like O’Reilly Media, Prentice Hall Professional, Addison-Wesley Professional, Microsoft Press, Sams, Que, Peachpit Press, Focal Press, Cisco Press, John Wiley & Sons, Syngress, Morgan Kaufmann, IBM Redbooks, Packt, Adobe Press, FT Press, Apress, Manning, New Riders, McGraw-Hill, Jones & Bartlett, Course Technology, and hundreds more. For more information about Safari Books Online, please visit us online.

How to Contact Us

Please address comments and questions concerning this book to the publisher:

• O’Reilly Media, Inc.
• 1005 Gravenstein Highway North
• Sebastopol, CA 95472
• 800-998-9938 (in the United States or Canada)
• 707-829-0515 (international or local)
• 707-829-0104 (fax)

We have a web page for this book, where we list errata, examples, and any additional information. You can access this page at http://bit.ly/streaming-architecture.

To comment or ask technical questions about this book, send email to [e-mail].

For more information about our books, courses, conferences, and news, see our website at http://www.oreilly.com.

Find us on Facebook: http://facebook.com/oreilly

Follow us on Twitter: http://twitter.com/oreillymedia

Watch us on YouTube: http://www.youtube.com/oreillymedia