Apache Oozie

Preface

Hadoop is fast becoming the de facto big data platform across all industries. An entire ecosystem of tools, products, and services targeting every functionality and requirement have sprung up around Hadoop. Apache Oozie occupies an important space in this ever-expanding ecosystem. Since Hadoop’s early days at Yahoo!, it has been a natural platform for Extract, Transform, and Load (ETL) and other forms of data pipelines. Without a mature workflow management and scheduling system, implementing such pipelines can be a challenge. Oozie satisfies these requirements and provides a viable tool to implement complex, real-world data pipelines. In this book, we have tried our best to introduce readers to all the facets of Oozie and walk them through the intricacies of this rather powerful and flexible platform.

Software workflow systems are ubiquitous and each system has its own idiosyncrasies. But Oozie is a lot more than just another workflow system. One of Oozie’s strengths is that it was custom built from the ground up for Hadoop. This not only means that Oozie works well on Hadoop, but that the authors of Oozie had an opportunity to build a new system incorporating much of their knowledge about other legacy workflow systems. Although some users view Oozie as just a workflow system, it has evolved into something more than that. The ability to use data availability and time-based triggers to schedule workflows via the Oozie coordinator is as important to today’s users as the workflow. The higher-level concept of bundles, which enable users to package multiple coordinators into complex data pipelines, is also gaining a lot of traction as applications and pipelines moving to Hadoop are getting more complicated.

We are both very lucky to have been involved in Oozie’s journey from its early days. We have played several roles in its evolution, ranging from developer, architect, open source committer, Project Management Committee (PMC) member, product manager, and even demanding customer. We have tried to leverage all of that perspective to present a comprehensive view of Oozie in this book. We strongly believe in the vision of Oozie and its potential to make Hadoop a more powerful platform. Hadoop’s use is expanding and we notice that users want to use it in smarter and more interesting ways. We have seen many projects in the past getting bogged down with writing, operating, and debugging the workflow system meant to manage the business application. By delegating all of the workflow and scheduling complexities to Oozie, you can focus on developing your core business application.

This book attempts to explain all the technical details of Oozie and its various features with specific, real-world examples. The target audience for this book is Oozie users and administrators at all levels of expertise. Our only requirement for the reader is a working knowledge of Hadoop and the ecosystem tools. We are also very aware of the challenges of operating a Hadoop cluster in general and Oozie in particular, and have tried our best to cover operational issues and debugging techniques in depth. Last but not the least, Oozie is designed to be very flexible and extensible and we want to encourage users to get comfortable with the idea of becoming an Oozie developer if they so desire. We would love to grow the Oozie community and continue the innovation in this part of the Hadoop ecosystem. While it would be nice to achieve all of these goals with this book, the most fundamental hope is that readers find it helpful in using Oozie and Hadoop more effectively every day in their jobs.

Contents of This Book

We start the book off with a brief introduction to Oozie in Chapter 1, Introduction to Oozie, and an overview of the important concepts in Chapter 2, Oozie Concepts. Chapter 3, Setting Up Oozie, gets your hands dirty right away with detailed instructions on installing and configuring Oozie. We want this book to be a hands-on experience for our readers, so deployment must be mastered early.

Oozie is primarily a workflow system in most users’ worlds. Chapters 4, and 5, take you on an in-depth journey through the world of writing and configuring workflows. These chapters also explain parameterization and variable substitution in detail. This will establish a very good basis for the rest of the book, as the other major Oozie features are built on top of the workflow system.

Chapter 6, Oozie Coordinator, covers the concepts of the coordinator and helps you to start writing coordinator apps. We then look at the data dependency mechanism in Chapter 7, Data Trigger Coordinator. Data triggers are a powerful and distinguishing feature of Oozie and this chapter explains all the intricacies of managing data dependencies.

Bundles are the higher-level pipeline abstraction and Chapter 8, Oozie Bundles, delves deep into the world of bundles with specific examples and use cases to clarify some of the advanced concepts. It also introduces concepts and challenges like reprocessing, which production pipelines routinely deal with.

In Chapter 9, Advanced Topics, we cover the powerful security features in Oozie, including Kerberos support and impersonation. This chapter also explains the management of shared libraries in Oozie and cron-based scheduling, which comes in handy for a certain class of use cases.

We cover the developer aspects regarding extending Oozie in Chapter 10, Developer Topics. Readers can learn how to implement custom extensions to their Oozie systems. It teaches them how to write their own Expression Language (EL) functions and custom actions.

Last, but not the least, we realize that debugging Oozie workflows and managing the operational details of Oozie are an important part of mastering Oozie. Thus, Chapter 11, Oozie Operations, focuses exclusively on these topics. We start by explaining the command-line interface (CLI) tool and the REST API and then discuss monitoring and debugging. We also cover the purge service, reprocessing, and other operational aspects in this chapter.

Conventions Used in This Book

The following typographical conventions are used in this book:

Italic

Indicates new terms, URLs, email addresses, filenames, and file extensions.

Constant width

Used for program listings, as well as within paragraphs to refer to program elements such as variable or function names, databases, data types, environment variables, statements, and keywords.

Constant width bold

Shows commands or other text that should be typed literally by the user.

Constant width italic

Shows text that should be replaced with user-supplied values or by values determined by context.

Note

This element signifies a general note.

Tip

This element signifies a tip or suggestion.

Caution

This element indicates a warning or caution.

Using Code Examples

The source code for all the examples in the book is available on GitHub.

This book is here to help you get your job done. In general, you may use the code in your programs and documentation. You do not need to contact us for permission unless you’re reproducing a significant portion of the code. For example, writing a program that uses several chunks of code from this book does not require permission. Selling or distributing a CD-ROM of examples from O’Reilly books does require permission. Answering a question by citing this book and quoting example code does not require permission. Incorporating a significant amount of example code from this book into your product’s documentation does require permission.

We appreciate, but do not require, attribution. An attribution usually includes the title, author, publisher, and ISBN. For example: “Apache Oozie by Mohammad Kamrul Islam and Aravind Srinivasan (O’Reilly). Copyright 2015 Mohammad Islam and Aravindakshan Srinivasan, 978-1-449-36992-7.”

If you feel your use of code examples falls outside fair use or the permission given above, feel free to contact us at [e-mail].

Safari® Books Online

Note

Safari Books Online (www.safaribooksonline.com) is an on-demand digital library that delivers expert content in both book and video form from the world’s leading authors in technology and business.

Technology professionals, software developers, web designers, and business and creative professionals use Safari Books Online as their primary resource for research, problem solving, learning, and certification training.

Safari Books Online offers a range of product mixes and pricing programs for organizations, government agencies, and individuals. Subscribers have access to thousands of books, training videos, and prepublication manuscripts in one fully searchable database from publishers like O’Reilly Media, Prentice Hall Professional, Addison-Wesley Professional, Microsoft Press, Sams, Que, Peachpit Press, Focal Press, Cisco Press, John Wiley & Sons, Syngress, Morgan Kaufmann, IBM Redbooks, Packt, Adobe Press, FT Press, Apress, Manning, New Riders, McGraw-Hill, Jones & Bartlett, Course Technology, and dozens more. For more information about Safari Books Online, please visit us online.

How to Contact Us

Please address comments and questions concerning this book to the publisher:

• O’Reilly Media, Inc.
• 1005 Gravenstein Highway North
• Sebastopol, CA 95472
• 800-998-9938 (in the United States or Canada)
• 707-829-0515 (international or local)
• 707-829-0104 (fax)

We have a web page for this book, where we list errata, examples, and any additional information. You can access this page at http://bit.ly/apache-oozie.

To comment or ask technical questions about this book, send email to [e-mail].

For more information about our books, courses, conferences, and news, see our website at http://www.oreilly.com.

Find us on Facebook: http://facebook.com/oreilly

Follow us on Twitter: http://twitter.com/oreillymedia

Watch us on YouTube: http://www.youtube.com/oreillymedia

Acknowledgments

As the saying goes, it takes a village to raise a child. After working on this book, we now realize it takes an even bigger crowd to finish a book! We would like to take this opportunity to thank everybody who helped us with this book. There are a lot of people we would like to thank and we apologize if we have missed a name or two (it’s certainly not our intention to forget anybody here). We will start with our family and personal friends because without their understanding, support, encouragement, and patience, this book would not have been possible.

At the top of our list is Robert Kanter from Cloudera. We thank him for his unwavering support. His in-depth knowledge and contributions to the Oozie code base and the community were a major source of information for us both directly and indirectly. He was our “go to” reviewer and sounding board throughout the process. We are very thankful for his incredible attention to detail and for his commitment to this project. We are convinced that without Robert’s involvement, this book would have been a lesser product.

A sincere vote of thanks goes out to Mona Chitnis and Virag Kothari from Yahoo! for all the detailed review comments and also for being there to answer any and all of our questions about various areas of the Oozie code. In addition, we also received a lot of comments and suggestions from a few other key reviewers. Their extensive and insightful thoughts definitely enhanced both the technical depth and the readability of this book. Hien Luu (LinkedIn), Jakob Homan (Microsoft), Denis Sheahan (Facebook), and William Kang (LinkedIn) deserve special mention in this regard. Special thanks to Raymie Stata (Altiscale) for his encouragement and support for this book. We also thank David Chaiken (Altiscale), Barbara Lewis (Altiscale), and Ann McCown (Altiscale) for their support.

We would also like to thank Sumeet Singh from Yahoo!, who initially encouraged us to write a book on Oozie and Santhosh Srinivasan from Cloudera for helping the two of us come together to work on this book. Santhosh has spent some time in the past as a manager of Yahoo!’s Oozie team and his perspective and understanding of this area was a major help to us.

None of this would have been possible without Alejandro Abdelnur, the cocreator of Oozie. Alejandro was personally involved with the contents of the early chapters and without his involvement, this project would have been a much harder endeavor. We sincerely thank him for his direct and indirect help and for serving as a sounding board and inspiration for us.

Finally, we thank all the O’Reilly folks for their support and resources. There are too many to thank individually, but they are the true owners of this project and deserve all the credit for making this happen. They were there every step of the way and helped us realize the vision of a book on Oozie.