Linux Server Hacks

Preface

A hacker does for love what others would not do for money.

—/usr/games/fortune

The word hack has many connotations. A “good hack” makes the best of the situation of the moment, using whatever resources are at hand. An “ugly hack” approaches the situation in the most obscure and least understandable way, although many “good hacks” may also appear unintelligible to the uninitiated.

The effectiveness of a hack is generally measured by its ability to solve a particular technical problem, inversely proportional to the amount of human effort involved in getting the hack running. Some hacks are scalable and some are even sustainable. The longest running and most generally accepted hacks become standards and cause many more hacks to be invented. A good hack lasts until a better hack comes along.

A hack reveals the interface between the abstract and wonderfully complex mind of the designer, and the indisputable and vulgar experience of human needs. Sometimes, hacks may be ugly and only exist because someone had an itch that needed scratching. To the engineer, a hack is the ultimate expression of the Do-It-Yourself sentiment: no one understands how a hack came to be better than the person who felt compelled to solve the problem in the first place. If a person with a bent for problem solving thinks a given hack is ugly, then they are almost always irresistibly motivated to go one better — and hack the hack, something that we encourage the readers of this book to do.

In the end, even the most capable server, with the most RAM and running the fastest (and most free) operating system on the planet, is still just a fancy back-scratcher fixing the itch of the moment, until a better, faster and cheaper back-scratcher is required.

Where does all of this pseudo-philosophical rambling get you? Hopefully, this background will give you some idea of the mindset that prompted the compiling of this collection of solutions that we call Linux Server Hacks. Some are short and simple, while some are quite complex. All of these hacks are designed to solve a particular technical problem that the designer simply couldn’t let go without “scratching.” I hope that some of them will be directly applicable to an “itch” or two that you may have felt yourself as a new or experienced administrator of Linux servers.

How This Book is Organized

A competent sysadmin must be a jack-of-all-trades. To be truly effective, you’ll need to be able to handle every problem the system throws at you, from power on to halt. To assist you in the time in between, I present this collection of time-saving and novel approaches to daily administrative tasks.

• Server Basics begins by looking at some of the most common sorts of tasks that admins encounter: manipulating the boot process, effectively working with the command line, automating common tasks, watching (and regulating) how system resources are used, and tuning various pieces of the Linux kernel to make everything run more efficiently. This isn’t an introduction to system administration but a look at some very effective and non-obvious techniques that even seasoned sysadmins may have overlooked.

• Revision Control gives a crash-course in using two fundamental revision control systems, RCS and CVS. Being able to recall arbitrary previous revisions of configuration files, source code, and documentation is a critical ability that can save your job. Too many professional admins are lacking in revision control basics (preferring instead to make the inevitable, but unsupportable .old or .orig backup). This section will get you up and running quickly, giving you commands and instructions that are succinct and to the point.

• The next section, Backups, looks at quick and easy methods for keeping spare copies of your data. I pay particular attention to network backups, rsync, and working with ISOs. I’ll demonstrate some of the enormous flexibility of standard system backup tools and even present one way of implementing regular “snapshot” revisions of a filesystem (without requiring huge amounts of storage).

• Networking is my favorite section of this entire book. The focus isn’t on basic functionality and routing, but instead looks at some obscure but insanely useful techniques for making networks behave in unexpected ways. I’ll set up various kinds of IP tunnels (both encrypted and otherwise), work with NAT, and show some advanced features that allow for interesting behavior based on all kinds of parameters. Did you ever want to decide what to do with a packet based on its data contents? Take a look at this section.

• Monitoring is an eclectic mix of tips and tools for finding out exactly what your server is up to. It looks at some standard (and some absolutely required “optional”) packages that will tell you volumes about who is using what, when, and how on your network. It also looks at a couple of ways to mitigate inevitable service failures and even help detect when naughty people attempt to do not-so-nice things to your network.

• Truly a font of hackery unto itself, the SSH section describes all sorts of nifty uses for ssh, the cryptographically strong (and wonderfully flexible) networking tool. There are a couple of versions of ssh available for Linux, and while many of the examples will work in all versions, they are all tested and known to work with OpenSSH v3.4p1.

• Scripting provides a short digression by looking at a couple of odds and ends that simply couldn’t fit on a single command line. These hacks will save you time and will hopefully serve as examples of how to do some nifty things in shell and Perl.

• Information Services presents three major applications for Linux: BIND 9, MySQL, and Apache. This section assumes that you’re well beyond basic installation of these packages, and are looking for ways to make them deliver their services faster and more efficiently, without having to do a lot of work yourself. You will see methods for getting your server running quickly, helping it scale to very large installations and behave in all sorts of slick ways that save a lot of configuration and maintenance time.

How to Use This Book

You may find it useful to read this book from cover to cover, as the hacks do build on each other a bit from beginning to end. However, each hack is designed to stand on its own as a particular example of one way to accomplish a particular task. To that end, I have grouped together hacks that fit a particular theme into sections, but I do cross-reference quite a bit between hacks from different sections (and also to more definitive resources on the subject). Don’t consider a given section as a cut-and-dried chapter with rigidly defined subject boundaries but more as a convenient way of collecting similar (and yet independent) hacks. You may want to read this book much like the way most people browse web pages online: follow whatever interests you, and if you get lost, follow the links within the piece to find more information.

Conventions Used in This Book

The following is a list of the typographical conventions used in this book:

Italic

Used to indicate new terms, URLs, filenames, file extensions, directories, commands and options, and program names.

Constant Width

Used to show code examples, the contents of files, or the output from commands.

Constant Width Bold

Used in examples and tables to show commands or other text that should be typed literally.

Constant Width Italic

Used in examples and tables to show text that should be replaced with user-supplied values.

The thermometer icons, found next to each hack, indicate the relative complexity of the hack:

beginner

moderate

expert

How to Contact Us

We have tested and verified the information in this book to the best of our ability, but you may find that features have changed (or even that we have made mistakes!). Please let us know about any errors, inaccuracies, bugs, misleading or confusing statements, and typos that you find in this book.

You can write to us at:

O’Reilly & Associates, Inc.

1005 Gravenstein Hwy N.

Sebastopol, CA 95472

[phone] (in the U.S. or Canada)

[phone] (international/local)

[phone] (fax)

To ask technical questions or to comment on the book, send email to:

[e-mail]

Visit the web page for Linux Server Hacks to find additional support information, including examples and errata. You can find this page at:

http://www.oreilly.com/catalog/linuxsvrhack

For more information about this book and others, see the O’Reilly web site:

http://www.oreilly.com

Gotta Hack?

Got a good hack you’d like to share with others? Go to the O’Reilly Hacks web site at:

http://hacks.oreilly.com

You’ll find book-related resources, sample hacks and new hacks contributed by users. You’ll find information about additional books in the Hacks series.

Credits

• Rob Flickenger authored the majority of hacks in this book.

  Rob has worked with Linux since Slackware 3.5. He was previously the system administrator of the O’Reilly Network (an all-Linux shop, naturally) and is the author of Building Wireless Community Networks, also by O’Reilly.

• Rael Dornfest ([Hack #87])

  Rael is a Researcher at the O’Reilly & Associates focusing on technologies just beyond the pale. He assesses, experiments, programs, and writes for the O’Reilly network and O’Reilly publications.

• Seann Herdejurgen ([Hack #47], [Hack #62])

  Seann has been working with UNIX since 1987 and now architects high availability solutions as a senior systems engineer with D-Tech corporation in Dallas, Texas. He holds a Master’s degree in Computer Science from Texas A&M University. He may be reached at: http://seann.herdejurgen.com/.

• Dru Lavigne ([Hack #39])

  Dru is an instructor at a private technical college in Kingston, ON where she teaches the fundamentals of TCP/IP networking, routing, and security. Her current hobbies include matching every TCP and UDP port number to its associated application(s) and reading her way through all of the RFCs.

• Cricket Liu ([Hack #77])

  Cricket matriculated at the University of California’s Berkeley campus, that great bastion of free speech, unencumbered Unix, and cheap pizza. He worked for a year as Director of DNS Product Management for VeriSign Global Registry Services, and is a co-author of DNS and BIND, also published by O’Reilly & Associates.

• Mike Rubel ([Hack #42])

  Mike (http://www.mikerubel.org) studied Mechanical Engineering at Rutgers University (B.S. 1998) and Aeronautics at Caltech (M.S. 1999), where he is now a graduate student. He has enjoyed using Linux and GNU software for several years in the course of his numerical methods research.

• Jennifer Vesperman (all of the CVS pieces except [Hack #36] were adapted from her online CVS pieces for O’ReillyNet)

  Jenn contributes to open source, as a user, writer and occasional programmer. Her coding experience ranges from the hardware interface to an HDLC card to the human interface of Java GUIs. Jenn is the current coordinator and co-sysadmin for Linuxchix.org.

• Schuyler Erle (contributed code for httptop, mysql-table-restore, balance-push, find-whois, and vtundgen)

  By day, Schuyler is a mild-mannered Internet Systems Developer for O’Reilly & Associates. By night, he crusades for justice and freedom as a free software hacker and community networking activist.

• Morbus Iff ([Hack #89], [Hack #90], [Hack #91])

  Kevin Hemenway, better known as Morbus Iff, is the creator of disobey.com, which bills itself as “content for the discontented.” Publisher and developer of more home cooking than you could ever imagine, he’d love to give you a Fry Pan of Intellect upside the head. Politely, of course. And with love.

Acknowledgments

I would like to thank my family and friends for their support and encouragement. Thanks especially to my dad for showing me “proper troubleshooting technique” at such an early age and inevitably setting me on the path of the Hacker before I had even seen a computer.

Of course, this book would be nothing without the excellent contributions of all the phenomenally talented hackers contained herein. But of course, our hacks are built by hacking the shoulders of giants (to horribly mix a metaphor), and it is my sincere hope that you will in turn take what you learn here and go one better, and most importantly, tell everyone just how you did it.