Preface
Modern object storage began when applications started to store and serve up assets for web and mobile users. More and more applications needed to deliver content directly to users regardless of whether they were using a laptop, tablet, or smartphone. Object storage is powerful because it can speak the language of the web: HTTP. Additionally, because of its adeptness at storing non-transactional data, it is perfect for archives, backups, file sharing, scientific and research data, and digital media.
This book covers both the Swift object storage project (referred to simply as Swift) and SwiftStack, an object storage software company and platform. SwiftStack offers user-friendly ways of accessing, deploying, and managing Swift clusters. Our goal is to show you how to use Swift on your own, present SwiftStack’s main features, and help you decide whether SwiftStack makes sense for you. We believe SwiftStack is a terrific product suitable for many uses, but rest assured that the Swift ecosystem is rich and growing, so there are plenty of options for service providers if SwiftStack turns out not to be a good fit. In the end we want to sell you on Swift object storage, not necessarily on SwiftStack.
Now, six years out from having first used object storage for my applications, I want to share my experiences dealing with the dramatic growth of data. I was fortunate to have a front-row seat as part of the team building one of the first widely used Platform-as-a Service (PaaS) environments running on top of Amazon Web Services. This helped the industry see how powerful an HTTP-based object storage system could be. Personally, I have seen how object storage delivers more flexibility, less lock-in, better control, and lower costs than what traditional storage systems could provide. I have also heard similar things from other organizations. With data (but not IT budgets) growing exponentially, there is absolute certainty that data storage challenges will persist and grow, affecting pretty much everyone who stores and serves data at scale.
The catalyst for this book was the pain I experienced firsthand deploying and managing object storage systems at scale. As a deployer, I often found myself wanting additional practical information. I also wanted to understand more fully the fundamentals of how these systems work, and why and how they should be used. Now it’s my hope that this book can deliver both practical and theoretical insights about an important and rapidly evolving topic: object storage with Swift.
Before proceeding further, I’d like to share John Dickinson’s memories of working on the original Swift development team. John explains why and how Swift was developed and captures the collaborative and innovative spirit of OpenStack and the development of Swift.
Why This Book?
At SwiftStack, a company that I cofounded in 2011, our big idea was to provide Swift object storage with a management system so our customers could achieve (1) flexibility in how (and where) they deployed their storage, (2) control of their data without being locked in to a vendor, and (3) extremely economical private cloud storage.
These features are the essence of software-defined storage (SDS), a new term that is still evolving. We think the term perfectly illustrates the fundamental change that Swift represents. Key aspects of SDS are scalability, adaptability, and the ability to use most any hardware. Operators can now make choices concerning how their storage is scaled and managed, and how users can store and access data—all driven programmatically for the entire storage tier, regardless of where the storage resources are deployed. SwiftStack was very early in this trend and was one of the first to offer an SDS storage controller to manage commodity storage hardware.
SwiftStack is dedicated to delivering and improving Swift solutions because we believe that data storage should scale globally, without technical constraints, limits, or lock-in. When data-centric organizations and data-centric applications can grow unconstrained, they will be able to discover and take advantage of new opportunities and innovations in their industries.
This book was started as a way to voice those beliefs. We are highlighting SDS, which is a fundamental change that is happening in the storage sector. Of course, we will shine the spotlight brightly on understanding the Swift object storage project and we will introduce SwiftStack. We bring not just our ideas—but the experience of the entire SwiftStack team. SwiftStack is fortunate to include some of the founders of Swift and top contributing Swift developers. Together, we have learned much from real-world deployments, large and small, which we will share with you throughout this book.
Who Should Read This Book?
OpenStack Swift is written for deployers, operators, and developers interested in using Swift. Although it’s not necessary to read this book cover to cover, we make an attempt to provide the topics in a logical order.
This book attempts to explain Swift in clear and simple terms so that anyone can understand the basic concepts and principles, though a few skills will be useful to get the most out of this book. We assume some familiarity with Linux administration, networking, and storage. Additionally, basic programming skills will come in handy for the chapters covering the Swift API. We offer programming overviews on Python, Ruby, PHP, and Java.
This book aims to help newcomers learn from our experiences. Our goal is to get you up and running more quickly and easily, making your successful conversion to Swift, well, swifter!
What’s in This Book?
This book has three major components. Part I (Chapters 1 through 4) covers the fundamentals of Swift in a way we hope is accessible to all our readers. Think of it as a Swift boot camp. We aim to familiarize you with the architecture, history, use cases, and basic operation of Swift.
Part II (Chapters 5 through 8) is geared to developers who want to get the most out of their Swift cluster. A big shift has happened with the transition from filesystems to objects, and these chapters will provide a foundation on how to use an HTTP-addressable object storage system. How to use Swift—and how not to use Swift—will be covered in gruesome detail.
The next major component of the book is for storage operators, architects, and systems administrators. We break this section into three parts—installation, planning a deployment, and ongoing operations. Part III (Chapters 9 and 10) covers Swift installation options. Part IV (Chapters 11 through 14) offers guidance on deployment considerations: hardware, networking, and Swift features of interest to operators. Finally, Part V (Chapters 15 through 17) covers what to do when (not if) something goes wrong.
Depending on your interests, here are a few possible itineraries through the book:
- If you’re responsible for designing and architecting a Swift cluster, read Part I, then check out Part IV.
- If you’re a developer, after reading Part I, continue with Part II to learn about Swift and its API.
- If you’re a systems administrator or in operations, read Part I, then head straight to Part III, then Part IV.
Conventions Used in This Book
The following typographical conventions are used in this book:
- Italic
- Indicates new terms, special emphasis, URLs, email addresses, filenames, and extensions.
-
Constant width - Used for program listings, as well as within paragraphs to refer to program elements such as variable or function names, databases, data types, environment variables, statements, and keywords.
-
Constant width bold - Highlights key parts of examples.
- Angle brackets (<>)
- Shows text that should be replaced with user-supplied values or values determined by context.
The following terms are used in this book:
- Drive, disk, device
- A drive refers to a storage medium, generally a hard disk drive (HDD) or solid-state drive (SSD). The term disk can also be used. A device is a drive as it is seen by the system.
- OpenStack Object Storage, OpenStack Swift, Swift Object Storage, Swift
- OpenStack Object Storage is commonly called OpenStack Swift or Swift Object Storage, or simply referred to as Swift within the OpenStack community. All these terms are used in the book and all references to Swift should be understood as referring to the OpenStack Swift project.
- Multi-region cluster
- Sometimes referred to as global cluster or geo-cluster. We prefer the term multi-region cluster (MRC) as it best fits with the actual tiering name (regions) and indicates that regions need not be separated by vast geographic distances.
- Swift node
- A machine running Swift processes is referred to as a node.
Using Code Examples
This book aims to help you work effectively with Swift. In general, you may use the code in this book in your programs and documentation. You do not need to contact us for permission unless you’re reproducing a significant portion of the code. For example, writing a program that uses several chunks of code from this book does not require permission. Selling or distributing a CD-ROM of examples does require permission. Answering a question by citing this book and quoting example code does not require permission. Incorporating a significant amount of example code from this book into your product’s documentation does require permission.
We appreciate, but do not require, attribution. An attribution usually includes the title, author, and publication year. For example: “OpenStack Swift, by Joe Arnold and members of the SwiftStack team (O’Reilly). Copyright 2015 SwiftStack, Inc., 978-1-491-90082-6.”
If you feel your use of code examples falls outside fair use or the permission given here, feel free to contact us via email (contact@swiftstack.com).
Examples will be maintained at http://swiftstack.com/book/.
Safari® Books Online
Note
Safari Books Online is an on-demand digital library that delivers expert content in both book and video form from the world’s leading authors in technology and business.
Technology professionals, software developers, web designers, and business and creative professionals use Safari Books Online as their primary resource for research, problem solving, learning, and certification training.
Safari Books Online offers a range of plans and pricing for enterprise, government, education, and individuals.
Members have access to thousands of books, training videos, and prepublication manuscripts in one fully searchable database from publishers like O’Reilly Media, Prentice Hall Professional, Addison-Wesley Professional, Microsoft Press, Sams, Que, Peachpit Press, Focal Press, Cisco Press, John Wiley & Sons, Syngress, Morgan Kaufmann, IBM Redbooks, Packt, Adobe Press, FT Press, Apress, Manning, New Riders, McGraw-Hill, Jones & Bartlett, Course Technology, and hundreds more. For more information about Safari Books Online, please visit us online.
How to Contact Us
SwiftStack provides an SDS system for object storage. Our solution combines a unique, decoupled storage controller with Swift, which gives our customers cost-effective, scale-out storage running on standard server hardware.
The company was founded in 2011 to help operations teams implement and manage an easy-to-use, multitenant, and highly scalable cloud storage platform. Our product is compatible with Red Hat Enterprise Linux, CentOS, and Ubuntu, and can run on a wide range of hardware.
SwiftStack is headquartered in San Francisco, CA, with support operations covering Asian and European time zones. To contact us, you can email contact@swiftstack.com. If you have specific questions for me, you can email me at joe@swiftstack.com.
As you use this book and work with Swift and SwiftStack, we invite your comments and feedback. From the very start, Swift has benefited from the contributions of hundreds of developers and users. And we hope that doesn’t stop.
This book was a collaborative effort. We’re proud of what we’ve put together. But we also know there will be future editions and we invite your comments, feedback, and suggestions. Please let us know what we need to correct or add; share your insights; and help us create a resource that will serve you and the broader community better. You can do so by visiting http://swiftstack.com/book/.
Please address comments and questions concerning this book to the publisher:
| O’Reilly Media, Inc. |
| 1005 Gravenstein Highway North |
| Sebastopol, CA 95472 |
| 800-998-9938 (in the United States or Canada) |
| 707-829-0515 (international or local) |
| 707-829-0104 (fax) |
We have a web page for this book, where we list errata, examples, and any additional information. You can access this page at http://bit.ly/openstack_swift.
To comment or ask technical questions about this book, send email to bookquestions@oreilly.com.
For more information about our books, courses, conferences, and news, see our website at http://www.oreilly.com.
Find us on Facebook: http://facebook.com/oreilly
Follow us on Twitter: http://twitter.com/oreillymedia
Watch us on YouTube: http://www.youtube.com/oreillymedia
Acknowledgments
Over a hundred developers have contributed to Swift. I’d like to thank each one of them for their ongoing contributions. There are also hundreds of operators of Swift who count on a stable, reliable storage system. Without them, Swift would just be an interesting idea. So, I would like to thank all of the Swift operators for trusting us with your data.
This book is the collaborative work of several authors. Amanda Plimpton served as a linchpin for this project. In addition to the chapters she wrote, she was able to dream up, assemble, and pull out of us the many pieces of content needed to form a cohesive whole. Jon Solera authored five crucial chapters on how to architect and develop Swift applications, how to use and develop Swift middleware, and options for authentication and authorization. Martin Lanner contributed several chapters on building, managing, and benchmarking Swift clusters.
This project couldn’t have come together without the support of a large cast. Many at SwiftStack have contributed to writing this book. Thanks to Hugo Kuo for his expert knowledge of configuring Swift, and to Alexander Corwin, Darrell Bishop, John Dickinson, Clay Gerrard, Sam Merritt, Zack M. Davis, and everyone else on the team who provided feedback. Thanks to Anders Tjernlund for thinking we should write this book in the first place.
Thanks to our technical reviewers, Paul Luse, Peter Portante, and Rodney Peck, who took time from their busy schedules to help us make a better book. Thanks to Christina Marcet for technical editing. Finally, many thanks to Mark Feldman for providing his craft of writing to this project and for advocating for the reader.
Get OpenStack Swift now with the O’Reilly learning platform.
O’Reilly members experience books, live events, courses curated by job role, and more from O’Reilly and nearly 200 top publishers.
