Foundations for Architecting Data Solutions

Preface

If you’re reading this book, you already know that there have been dramatic shifts in the data management landscape in recent years. We’ve seen a shift from third-party, proprietary solutions to new, open source distributed data systems. Of course, the common term used to refer to these newer solutions is “big data” (a term we find to be less and less useful), but it’s important to note that many of the earlier proprietary systems utilize distributed architectures that can store and process large volumes of data. Although we can apply these proprietary solutions and the newer open source solutions to solve many of the same problems, there are some distinct differences that have contributed to the growth of the newer systems. This includes not just the economies of the open source approach, but also technology approaches that facilitate the implementation of many applications that are challenging with previous solutions.

Along with the growth of these systems, we’ve seen a corresponding growth in books, articles, training, conferences, and so on dedicated to help you, the practitioner, use these systems, so it’s reasonable to ask why yet another book on this “big data” stuff? To quote a cliché, we think the answer is that it becomes easy to miss the forest for the trees. Most of these materials focus on low-level details such as implementing applications using distributed processing engines like MapReduce or Spark or applying advanced algorithms to perform data analysis. Others focus on higher-level architectural considerations; for example, Hadoop Application Architectures (O’Reilly), coauthored by the authors of this book.

The gap that we see is a perspective that takes an even wider view; in other words, what steps need to be taken to ensure successful data projects in this new landscape, from planning to execution? While developing the expertise in the architectures and component systems is critical, there are larger considerations that are equally important to your data projects, and these considerations can often be lost in the excitement of exploring new technologies.

These considerations include things like the following:

• Ensuring that you understand your problem

• Selecting software solutions that fit your use case

• Addressing project risk

• Building teams to successfully deliver projects

• Ensuring the implementation of robust, maintainable architectures and solutions as your project progresses

If you’re an experienced software development practitioner, these considerations might sound very familiar, as they should. It’s absolutely true that managing successful modern data projects requires many of the same processes as other software development projects. However, these new software systems and architectures require a new set of knowledge and considerations when developing projects. For example, evaluating software in an open source world can be very different from selecting proprietary solutions. Our intention is not to provide another book on managing software projects, but rather provide guidelines for applying sound project management and development practices to modern data solutions.

Who This Book Is For

This book is targeted at folks in an organization who are making decisions about data management projects and implementing those projects, such as the following:

• CxOs such as chief information officers or chief technology officers responsible for high-level decision making in an organization

• Project and product managers responsible for delivering data projects

• Lead architects, technical leads, and developers tasked with developing data projects

Again, we’re not trying to provide you with the detailed knowledge to implement applications using specific components; instead, we provide a framework for understanding the basis of successful modern data projects. We want you to come away with the knowledge required to successfully navigate projects and make the decisions required to deliver data projects that provide real value to users.

Navigating This Book

Each chapter in this book addresses a different topic in data project management. It’s not necessary to read through the entire book, because most chapters can stand on their own. However, before undertaking a new project, it would be helpful to read Chapter 1 through Chapter 3.

The following describes all of the chapters in this book:

• Chapter 1, Key Data Project Types and Considerations begins with an overview of the three major data project use cases and provides a list of items for each use case type that are important to consider before undertaking a project. This is a good place to start before undertaking a new data project.

• Chapter 2, Evaluating and Selecting Data Management Solutions provides guidelines for selecting technology solutions in this new world of distributed, open source software. This will also be a very useful chapter if you’re starting on a data project, or just getting started in this space.

• Chapter 3, Managing Risk in Data Projects discusses project risk and how to manage it. Managing risk is an important activity in any software project, and implementing large data projects adds some unique aspects that need to be managed to ensure success.

• Chapter 4, Interface Design discusses the design and implementation of interfaces in your systems. Defining effective abstractions and contracts is crucial to creating maintainable and scalable systems, so this chapter provides some guidelines based on our experience implementing large data projects.

• Chapter 5, Distributed Storage Systems covers distributed storage systems. Data storage is a core component of any data system, and this chapter provides a breakdown of some commonly available distributed storage systems. More important, it provides a framework for evaluating systems.

• Chapter 6, The Meta of Enterprise Data covers metadata management. This is another critically important but often overlooked aspect of building systems that manage and process data.

• Chapter 7, Ensuring Data Integrity discusses data integrity. This is another important aspect of building data systems that needs to be planned for from project inception. Ensuring the integrity and lineage of data becomes more challenging and critical when building systems that store large volumes of data with multiple formats.

• Chapter 8, Data Processing discusses frameworks for performing distributed processing of data. The ability to process and analyze the data in your system is of course another crucial aspect to building systems that provide value. Similar to the storage chapter, this chapter provides a framework for understanding available systems and performing evaluations to determine which ones are suitable for your use cases.

Conventions Used in This Book

The following typographical conventions are used in this book:

Italic

Indicates new terms, URLs, email addresses, filenames, and file extensions.

Constant width

Used for program listings, as well as within paragraphs to refer to program elements such as variable or function names, databases, data types, environment variables, statements, and keywords.

Note

This element signifies a general note.

Using Code Examples

This book is here to help you get your job done. In general, if example code is offered with this book, you may use it in your programs and documentation. You do not need to contact us for permission unless you’re reproducing a significant portion of the code. For example, writing a program that uses several chunks of code from this book does not require permission. Selling or distributing a CD-ROM of examples from O’Reilly books does require permission. Answering a question by citing this book and quoting example code does not require permission. Incorporating a significant amount of example code from this book into your product’s documentation does require permission.

We appreciate, but do not require, attribution. An attribution usually includes the title, author, publisher, and ISBN. For example: “Foundations for Architecting Data Solutions by Ted Malaska and Jonathan Seidman (O’Reilly). Copyright 2018 Ted Malaska and Jonathan Seidman, 978-1-492-03874-0.”

If you feel your use of code examples falls outside fair use or the permission given above, feel free to contact us at [e-mail].

O’Reilly Safari

Note

Safari (formerly Safari Books Online) is a membership-based training and reference platform for enterprise, government, educators, and individuals.

Members have access to thousands of books, training videos, Learning Paths, interactive tutorials, and curated playlists from over 250 publishers, including O’Reilly Media, Harvard Business Review, Prentice Hall Professional, Addison-Wesley Professional, Microsoft Press, Sams, Que, Peachpit Press, Adobe, Focal Press, Cisco Press, John Wiley & Sons, Syngress, Morgan Kaufmann, IBM Redbooks, Packt, Adobe Press, FT Press, Apress, Manning, New Riders, McGraw-Hill, Jones & Bartlett, and Course Technology, among others.

For more information, please visit http://oreilly.com/safari.

How to Contact Us

Please address comments and questions concerning this book to the publisher:

• O’Reilly Media, Inc.
• 1005 Gravenstein Highway North
• Sebastopol, CA 95472
• 800-998-9938 (in the United States or Canada)
• 707-829-0515 (international or local)
• 707-829-0104 (fax)

We have a web page for this book, where we list errata, examples, and any additional information. You can access this page at http://bit.ly/foundations-for-architecting-data-solutions.

To comment or ask technical questions about this book, send email to [e-mail].

For more information about our books, courses, conferences, and news, see our website at http://www.oreilly.com.

Find us on Facebook: http://facebook.com/oreilly

Follow us on Twitter: http://twitter.com/oreillymedia

Watch us on YouTube: http://www.youtube.com/oreillymedia

Acknowledgments

Many people provided invaluable feedback and support while we wrote this book, especially Mark Grover, Kevin O’Dell, and Steven Totman, who provided their time and expertise to review content. These reviewers helped us out and greatly improved the quality of this book; any mistakes that remain are our own.

We’d like to thank our O’Reilly editors Nicole Tache and Michele Cronin for helping shepherd this book to completion. We also want to thank the folks at O’Reilly Media who provided help and support throughout the development of this book.

Our apologies to those whom we may have mistakenly omitted from this list.