System Performance Tuning, 2nd Edition

Preface

This book is about the art of tuning systems for optimal application performance. Within these pages, I’ll discuss how to perform the actual tuning itself and get the most out of your applications. I’ll explain the underlying algorithms behind these tunable parameters, so you can make intelligent decisions based on your own environment. I’ll touch upon capacity planning as well, which describes how to appropriately size systems for different uses.

Who Should Buy This Book?

This book is primarily for people who are interested in optimizing their computer systems for performance. Those faced with purchasing decisions will find a clear description of how various system components interact with each other, as well as pointers on what to watch for. People writing or tuning code will find a concise explanation of modern compiler options, as well as an appreciation for how the underlying operating system manages their runtime applications. Finally, those who are simply interested in learning more about how their computers work may find the explanations within these pages to be interesting.

A Note on Coverage

This book is heavily oriented to the Solaris operating environment (up to and including Solaris 8) and Linux. However, this book emphasizes Solaris. There are a handful of reasons for this, which I would like to elucidate:

• The majority of datacenter environments run Solaris; these environments tend to be the ones that require the most effort towards performance tuning.

• Solaris machines tend to be more focused on performance. I suspect this is because Sun systems are more expensive than their Linux counterparts, on average. As a result, people tend to be a lot more picky about performance, so more work has been done in that area on Solaris. If your Linux box doesn’t perform well enough, you can just buy another one and split up the workload -- it’s cheap. If your several-million-dollar Ultra Enterprise 10000 doesn’t perform well and your company is losing nontrivial sums of money every minute because of it, you call Sun Service and start demanding answers.

• Finally, the tools available for performance analysis on Solaris are much stronger than those available on Linux. Part of this is that it’s not glamorous to implement performance analysis frameworks; part is that sometimes you need to effect hardware changes to get the best performance data; part is that Linux is a relatively new operating system, developed by relatively few, whereas Solaris (and its predecessor SunOS, which provided a lot of feedback into the Solaris development process) is a rather old operating system, and Sun is a huge company. The end result is that because our ability to analyze performance problems in depth and to make intelligent tuning decisions is largely based on the data provided by our performance data-gathering tools (otherwise, every time we tune, we’re just taking a shot in the dark) it is extremely difficult to tune Linux systems. This goes directly against one major principle of performance tuning (see Section 1.2.1).

I don’t mean to indicate any bias for or against any one operating system.^[1] The simple truth is that serious performance tuning on Linux is a headache right now, because the tools to facilitate understanding just aren’t available.

If you are running Linux, you absolutely must download and install the sysstat package, which provides (very limited) versions of iostat, mpstat, and sar. While far from being as complete as the Solaris versions, they are at least a way to get some information. As of this writing, they can be downloaded from the author’s web site (Sebastien Godard) at http://perso.wanadoo.fr/sebastien.godard/.

How to Read This Book

This book is part reference and part story. Readers with some background in performance tuning and analysis may be tempted to immediately flip to a specific section (perhaps one concerning where they suspect a problem to be) and start using the book as a reference. I strongly discourage that.

This Book as a Story

To get the most out of this book, read it first as a story. Habits, ways of thinking, and approaches to problems are some of the most important things you can gain from this text, and these are to be soaked up over the course of hours, not gained by memorizing items from a list.

Many of the topics we’ll discuss here may appear quite simple and straightforward. The truth is that they go to the foundations of modern computer architecture -- why computers work at all -- and are often are deceivingly complex. Don’t be alarmed if you have to reread parts of the book; it takes time to absorb some of this material.

This Book as a Reference

Once you’ve read the story and understand the overall picture of how the pieces of the performance puzzle fit together, then this book will prove to be a useful reference. If you haven’t read the story, you might look at one of the charts (such as the description of kernel memory tunables in Section 4.2.5.1 in Chapter 4) and think that you’ve understood the whole picture. If you believe that, there’s a lot of complex background that you have just missed out on. After you’ve read this book as a story, then you’ll have an appreciation for the issues surrounding complex problems.

Organization

This book consists of nine chapters:

Chapter 1, sets the stage for the rest of the book through a discussion of fundamental principles.

Chapter 2, describes performance management through understanding and limiting system workloads, as well as describing some common benchmarks.

Chapter 3, briefly discusses modern processor architecture, caching, multiprocessor systems, and how the operating system schedules tasks.

Chapter 4, explains how computer memory systems are organized, and how that management interacts with other parts of the system.

Chapter 5, describes disks in the simplest state: how hard drives function and what limitations they have that can be worked around.

Chapter 6, is in some senses a continuation of Chapter 5; it is a discussion of the technique of connecting multiple disks into a single logical unit.

Chapter 7, addresses issues surrounding network performance, from the physical layer through communications protocols like TCP/IP and file sharing systems like Samba and NFS.

Chapter 8, is a brief introduction to how to write fast code, as well as a summary of modern compiler usage.

Chapter 9, is a summary of some of the key points discussed throughout the book, presented as a reference for solving problems and configuring systems quickly.

Typographic Conventions

The following conventions are used in this book:

Constant Width

Used for source code samples and for quotations from source code within the text, including variable and function names. Constant width is also used for output printed by a computer and for the contents of files.

Constant Bold

Used for commands typed verbatim by the user.

Italic

Used for command names, directory names, and filenames. It is also used for new terms when they are defined.

Bold

Used for vectors (in the mathematical sense) and for command options.

Comments and Questions

We have tested and verified all of the information in this book to the best of our ability, but you may find that features have changed, that typos have crept in, or that we have made a mistake. Please let us know about what you find, as well as your suggestions for future editions, by contacting:

O’Reilly & Associates, Inc.

1005 Gravenstein Highway North

Sebastopol, CA 95472

[phone] (in the U.S. or Canada)

[phone] (international/local)

[phone] (fax)

You can also send us messages electronically. To be put on the mailing list or request a catalog, send email to:

[e-mail]

To ask technical questions or comment on the book, send email to:

[e-mail]

We have a web site for the book, where we’ll list examples, errata, and any plans for future editions. You can access this page at:

http://www.oreilly.com/catalog/spt2/

For more information about this book and others, see the O’Reilly web site:

http://www.oreilly.com

Personal Comments and Acknowledgments

I am indebted to many people for their help and assistance as I wrote this book.

First and foremost, I would like to thank my editor, Mike Loukides. This book slipped by more than a few deadlines, and his patience, wonderful advice, and upbeat attitude were absolutely vital to this book being finished. I also owe a huge debt to my management teams, both at the University of Illinois (particularly Mona Heath and Ed Krol) and at Sun Microsystems (Dr. Keng-Tai Ko and Miroslav Klivansky). Their support, patience, and understanding proved invaluable.

My friends were a constant source of support during this affair, both in providing technical advice and being understanding of the “No, I can’t go out, I have to work on the book” phrase that suddenly took over my vocabulary for two years. I would like to especially thank Keith Wessel, Kris Wehner, Majdi Abbas, Deb Fligor, Jay Kreibich, Adrian Cockcroft, Richard McDougall, Elizabeth Purcell, Tayfun Kocaglu, Paul Strong, Phil Stracchino, Christof Harper, and Ken MacInnis (in no particular order). Dr. Sanjay Patel was kind enough to admit a student with none of the prerequisites into his computer architectures course at the University of Illinois; his encouragement is very much appreciated. Rose Platt, Nancy Weber, and Dr. Robert Cahn opened their home to me and, in doing so, provided me with a safe retreat to sit and write for long breaks. Kate Secor kept me sane as I wrote the last half of this book: she jokes that she was my “good writing totem” as whenever she was around, I got a lot more written than otherwise.

It’s hard for me to overstate the contributions of my family to this book, particularly my parents, Drs. Diane and Antonino Musumeci. It’s somehow very reassuring to look at the people who raised you and realize that they have both written books -- how hard can it be, right?^[2] My brothers Domenico and Walter kept me grounded in the real world, my grandfather Walter reminded me that there was a time before computers existed, and the family dog Gizmo was a constant source of amusement (it’s hard to keep focused when a 15-pound Border Terrier is tearing around the house as fast as he can, and sometimes you need to be not focused).

I would like to dedicate this book to my grandmother, Kathleen Mikolaitis.

Acknowledgments from Mike Loukides

First and foremost, I want to thank Gian-Paolo D. Musumeci for doing what I could never have done: write the second edition of this book. It’s really his work. I’m proud of the first edition, but I have to confess that it is primarily a piece of journalism. I had access to a number of system experts (Doug Gilmore, Chris Ryland, Tan Bronson, and the rest of the Multiflow diaspora, among others), and reported their expertise. (Many people aren’t aware that the book started as part of the Multiflow TRACE/UNIX documentation set, although Multiflow went out of business before it could be published.) Gian-Paolo is the real thing, where I’m just a journalist: he brings a depth of knowledge and experience to the book that I never had. As proud as I am of the first edition, the second edition is a much better book.

---

^[1] In the interest of full disclosure, the author was employed by Sun Microsystems for about half the time it took to write this book.

^[2] The answer is: very hard.