You are previewing bash Cookbook.

bash Cookbook

Cover of bash Cookbook by Carl Albing... Published by O'Reilly Media, Inc.
  1. bash Cookbook
    1. SPECIAL OFFER: Upgrade this ebook with O’Reilly
    2. A Note Regarding Supplemental Files
    3. Preface
      1. Who Should Read This Book
      2. About This Book
      3. Conventions Used in This Book
      4. Using Code Examples
      5. We'd Like to Hear from You
      6. Safari® Enabled
      7. Acknowledgments
    4. 1. Beginning bash
      1. 0. Why bash?
      2. 0. The bash Shell
      3. 1.1. Decoding the Prompt
      4. 1.2. Showing Where You Are
      5. 1.3. Finding and Running Commands
      6. 1.4. Getting Information About Files
      7. 1.5. Showing All Hidden (dot) Files in the Current Directory
      8. 1.6. Using Shell Quoting
      9. 1.7. Using or Replacing Built-ins and External Commands
      10. 1.8. Determining If You Are Running Interactively
      11. 1.9. Setting bash As Your Default Shell
      12. 1.10. Getting bash for Linux
      13. 1.11. Getting bash for xBSD
      14. 1.12. Getting bash for Mac OS X
      15. 1.13. Getting bash for Unix
      16. 1.14. Getting bash for Windows
      17. 1.15. Getting bash Without Getting bash
      18. 1.16. Learning More About bash Documentation
    5. 2. Standard Output
      1. 2.1. Writing Output to the Terminal/Window
      2. 2.2. Writing Output but Preserving Spacing
      3. 2.3. Writing Output with More Formatting Control
      4. 2.4. Writing Output Without the Newline
      5. 2.5. Saving Output from a Command
      6. 2.6. Saving Output to Other Files
      7. 2.7. Saving Output from the ls Command
      8. 2.8. Sending Both Output and Error Messages to Different Files
      9. 2.9. Sending Both Output and Error Messages to the Same File
      10. 2.10. Appending Rather Than Clobbering Output
      11. 2.11. Using Just the Beginning or End of a File
      12. 2.12. Skipping a Header in a File
      13. 2.13. Throwing Output Away
      14. 2.14. Saving or Grouping Output from Several Commands
      15. 2.15. Connecting Two Programs by Using Output As Input
      16. 2.16. Saving a Copy of Output Even While Using It As Input
      17. 2.17. Connecting Two Programs by Using Output As Arguments
      18. 2.18. Using Multiple Redirects on One Line
      19. 2.19. Saving Output When Redirect Doesn't Seem to Work
      20. 2.20. Swapping STDERR and STDOUT
      21. 2.21. Keeping Files Safe from Accidental Overwriting
      22. 2.22. Clobbering a File on Purpose
    6. 3. Standard Input
      1. 3.1. Getting Input from a File
      2. 3.2. Keeping Your Data with Your Script
      3. 3.3. Preventing Weird Behavior in a Here-Document
      4. 3.4. Indenting Here-Documents
      5. 3.5. Getting User Input
      6. 3.6. Getting Yes or No Input
      7. 3.7. Selecting from a List of Options
      8. 3.8. Prompting for a Password
    7. 4. Executing Commands
      1. 4.1. Running Any Executable
      2. 4.2. Telling If a Command Succeeded or Not
      3. 4.3. Running Several Commands in Sequence
      4. 4.4. Running Several Commands All at Once
      5. 4.5. Deciding Whether a Command Succeeds
      6. 4.6. Using Fewer if Statements
      7. 4.7. Running Long Jobs Unattended
      8. 4.8. Displaying Error Messages When Failures Occur
      9. 4.9. Running Commands from a Variable
      10. 4.10. Running All Scripts in a Directory
    8. 5. Basic Scripting: Shell Variables
      1. 5.1. Documenting Your Script
      2. 5.2. Embedding Documentation in Shell Scripts
      3. 5.3. Promoting Script Readability
      4. 5.4. Separating Variable Names from Surrounding Text
      5. 5.5. Exporting Variables
      6. 5.6. Seeing All Variable Values
      7. 5.7. Using Parameters in a Shell Script
      8. 5.8. Looping Over Arguments Passed to a Script
      9. 5.9. Handling Parameters with Blanks
      10. 5.10. Handling Lists of Parameters with Blanks
      11. 5.11. Counting Arguments
      12. 5.12. Consuming Arguments
      13. 5.13. Getting Default Values
      14. 5.14. Setting Default Values
      15. 5.15. Using null As a Valid Default Value
      16. 5.16. Using More Than Just a Constant String for Default
      17. 5.17. Giving an Error Message for Unset Parameters
      18. 5.18. Changing Pieces of a String
      19. 5.19. Using Array Variables
    9. 6. Shell Logic and Arithmetic
      1. 6.1. Doing Arithmetic in Your Shell Script
      2. 6.2. Branching on Conditions
      3. 6.3. Testing for File Characteristics
      4. 6.4. Testing for More Than One Thing
      5. 6.5. Testing for String Characteristics
      6. 6.6. Testing for Equal
      7. 6.7. Testing with Pattern Matches
      8. 6.8. Testing with Regular Expressions
      9. 6.9. Changing Behavior with Redirections
      10. 6.10. Looping for a While
      11. 6.11. Looping with a read
      12. 6.12. Looping with a Count
      13. 6.13. Looping with Floating-Point Values
      14. 6.14. Branching Many Ways
      15. 6.15. Parsing Command-Line Arguments
      16. 6.16. Creating Simple Menus
      17. 6.17. Changing the Prompt on Simple Menus
      18. 6.18. Creating a Simple RPN Calculator
      19. 6.19. Creating a Command-Line Calculator
    10. 7. Intermediate Shell Tools I
      1. 7.1. Sifting Through Files for a String
      2. 7.2. Getting Just the Filename from a Search
      3. 7.3. Getting a Simple True/False from a Search
      4. 7.4. Searching for Text While Ignoring Case
      5. 7.5. Doing a Search in a Pipeline
      6. 7.6. Paring Down What the Search Finds
      7. 7.7. Searching with More Complex Patterns
      8. 7.8. Searching for an SSN
      9. 7.9. Grepping Compressed Files
      10. 7.10. Keeping Some Output, Discarding the Rest
      11. 7.11. Keeping Only a Portion of a Line of Output
      12. 7.12. Reversing the Words on Each Line
      13. 7.13. Summing a List of Numbers
      14. 7.14. Counting String Values
      15. 7.15. Showing Data As a Quick and Easy Histogram
      16. 7.16. Showing a Paragraph of Text After a Found Phrase
    11. 8. Intermediate Shell Tools II
      1. 8.1. Sorting Your Output
      2. 8.2. Sorting Numbers
      3. 8.3. Sorting IP Addresses
      4. 8.4. Cutting Out Parts of Your Output
      5. 8.5. Removing Duplicate Lines
      6. 8.6. Compressing Files
      7. 8.7. Uncompressing Files
      8. 8.8. Checking a tar Archive for Unique Directories
      9. 8.9. Translating Characters
      10. 8.10. Converting Uppercase to Lowercase
      11. 8.11. Converting DOS Files to Linux Format
      12. 8.12. Removing Smart Quotes
      13. 8.13. Counting Lines, Words, or Characters in a File
      14. 8.14. Rewrapping Paragraphs
      15. 8.15. Doing More with less
    12. 9. Finding Files: find, locate, slocate
      1. 9.1. Finding All Your MP3 Files
      2. 9.2. Handling Filenames Containing Odd Characters
      3. 9.3. Speeding Up Operations on Found Files
      4. 9.4. Finding Files Across Symbolic Links
      5. 9.5. Finding Files Irrespective of Case
      6. 9.6. Finding Files by Date
      7. 9.7. Finding Files by Type
      8. 9.8. Finding Files by Size
      9. 9.9. Finding Files by Content
      10. 9.10. Finding Existing Files and Content Fast
      11. 9.11. Finding a File Using a List of Possible Locations
    13. 10. Additional Features for Scripting
      1. 10.1. "Daemon-izing" Your Script
      2. 10.2. Reusing Code with Includes and Sourcing
      3. 10.3. Using Configuration Files in a Script
      4. 10.4. Defining Functions
      5. 10.5. Using Functions: Parameters and Return Values
      6. 10.6. Trapping Interrupts
      7. 10.7. Redefining Commands with alias
      8. 10.8. Avoiding Aliases, Functions
    14. 11. Working with Dates and Times
      1. 11.1. Formatting Dates for Display
      2. 11.2. Supplying a Default Date
      3. 11.3. Automating Date Ranges
      4. 11.4. Converting Dates and Times to Epoch Seconds
      5. 11.5. Converting Epoch Seconds to Dates and Times
      6. 11.6. Getting Yesterday or Tomorrow with Perl
      7. 11.7. Figuring Out Date and Time Arithmetic
      8. 11.8. Handling Time Zones, Daylight Saving Time, and Leap Years
      9. 11.9. Using date and cron to Run a Script on the Nth Day
    15. 12. End-User Tasks As Shell Scripts
      1. 12.1. Starting Simple by Printing Dashes
      2. 12.2. Viewing Photos in an Album
      3. 12.3. Loading Your MP3 Player
      4. 12.4. Burning a CD
      5. 12.5. Comparing Two Documents
    16. 13. Parsing and Similar Tasks
      1. 13.1. Parsing Arguments for Your Shell Script
      2. 13.2. Parsing Arguments with Your Own Error Messages
      3. 13.3. Parsing Some HTML
      4. 13.4. Parsing Output into an Array
      5. 13.5. Parsing Output with a Function Call
      6. 13.6. Parsing Text with a read Statement
      7. 13.7. Parsing with read into an Array
      8. 13.8. Getting Your Plurals Right
      9. 13.9. Taking It One Character at a Time
      10. 13.10. Cleaning Up an SVN Source Tree
      11. 13.11. Setting Up a Database with MySQL
      12. 13.12. Isolating Specific Fields in Data
      13. 13.13. Updating Specific Fields in Data Files
      14. 13.14. Trimming Whitespace
      15. 13.15. Compressing Whitespace
      16. 13.16. Processing Fixed-Length Records
      17. 13.17. Processing Files with No Line Breaks
      18. 13.18. Converting a Data File to CSV
      19. 13.19. Parsing a CSV Data File
    17. 14. Writing Secure Shell Scripts
      1. 14.1. Avoiding Common Security Problems
      2. 14.2. Avoiding Interpreter Spoofing
      3. 14.3. Setting a Secure $PATH
      4. 14.4. Clearing All Aliases
      5. 14.5. Clearing the Command Hash
      6. 14.6. Preventing Core Dumps
      7. 14.7. Setting a Secure $IFS
      8. 14.8. Setting a Secure umask
      9. 14.9. Finding World-Writable Directories in Your $PATH
      10. 14.10. Adding the Current Directory to the $PATH
      11. 14.11. Using Secure Temporary Files
      12. 14.12. Validating Input
      13. 14.13. Setting Permissions
      14. 14.14. Leaking Passwords into the Process List
      15. 14.15. Writing setuid or setgid Scripts
      16. 14.16. Restricting Guest Users
      17. 14.17. Using chroot Jails
      18. 14.18. Running As a Non-root User
      19. 14.19. Using sudo More Securely
      20. 14.20. Using Passwords in Scripts
      21. 14.21. Using SSH Without a Password
      22. 14.22. Restricting SSH Commands
      23. 14.23. Disconnecting Inactive Sessions
    18. 15. Advanced Scripting
      1. 15.1. Finding bash Portably for #!
      2. 15.2. Setting a POSIX $PATH
      3. 15.3. Developing Portable Shell Scripts
      4. 15.4. Testing Scripts in VMware
      5. 15.5. Using for Loops Portably
      6. 15.6. Using echo Portably
      7. 15.7. Splitting Output Only When Necessary
      8. 15.8. Viewing Output in Hex
      9. 15.9. Using bash Net-Redirection
      10. 15.10. Finding My IP Address
      11. 15.11. Getting Input from Another Machine
      12. 15.12. Redirecting Output for the Life of a Script
      13. 15.13. Working Around "argument list too long" Errors
      14. 15.14. Logging to syslog from Your Script
      15. 15.15. Sending Email from Your Script
      16. 15.16. Automating a Process Using Phases
    19. 16. Configuring and Customizing bash
      1. 16.1. bash Startup Options
      2. 16.2. Customizing Your Prompt
      3. 16.3. Change Your $PATH Permanently
      4. 16.4. Change Your $PATH Temporarily
      5. 16.5. Setting Your $CDPATH
      6. 16.6. Shortening or Changing Command Names
      7. 16.7. Adjusting Shell Behavior and Environment
      8. 16.8. Adjusting readline Behavior Using .inputrc
      9. 16.9. Keeping a Private Stash of Utilities by Adding ~/bin
      10. 16.10. Using Secondary Prompts: $PS2, $PS3, $PS4
      11. 16.11. Synchronizing Shell History Between Sessions
      12. 16.12. Setting Shell History Options
      13. 13. Creating a Better cd Command
      14. 16.14. Creating and Changing into a New Directory in One Step
      15. 16.15. Getting to the Bottom of Things
      16. 16.16. Adding New Features to bash Using Loadable Built-ins
      17. 16.17. Improving Programmable Completion
      18. 16.18. Using Initialization Files Correctly
      19. 16.19. Creating Self-Contained, Portable RC Files
      20. 16.20. Getting Started with a Custom Configuration
    20. 17. Housekeeping and Administrative Tasks
      1. 17.1. Renaming Many Files
      2. 17.2. Using GNU Texinfo and Info on Linux
      3. 17.3. Unzipping Many ZIP Files
      4. 17.4. Recovering Disconnected Sessions Using screen
      5. 17.5. Sharing a Single bash Session
      6. 17.6. Logging an Entire Session or Batch Job
      7. 17.7. Clearing the Screen When You Log Out
      8. 17.8. Capturing File Metadata for Recovery
      9. 17.9. Creating an Index of Many Files
      10. 17.10. Using diff and patch
      11. 17.11. Counting Differences in Files
      12. 17.12. Removing or Renaming Files Named with Special Characters
      13. 17.13. Prepending Data to a File
      14. 17.14. Editing a File in Place
      15. 17.15. Using sudo on a Group of Commands
      16. 17.16. Finding Lines in One File But Not in the Other
      17. 17.17. Keeping the Most Recent N Objects
      18. 17.18. Grepping ps Output Without Also Getting the grep Process Itself
      19. 17.19. Finding Out Whether a Process Is Running
      20. 17.20. Adding a Prefix or Suffix to Output
      21. 17.21. Numbering Lines
      22. 17.22. Writing Sequences
      23. 17.23. Emulating the DOS Pause Command
      24. 17.24. Commifying Numbers
    21. 18. Working Faster by Typing Less
      1. 18.1. Moving Quickly Among Arbitrary Directories
      2. 18.2. Repeating the Last Command
      3. 18.3. Running Almost the Same Command
      4. 18.4. Substituting Across Word Boundaries
      5. 18.5. Reusing Arguments
      6. 18.6. Finishing Names for You
      7. 18.7. Playing It Safe
    22. 19. Tips and Traps: Common Goofs for Novices
      1. 19.1. Forgetting to Set Execute Permissions
      2. 19.2. Fixing "No such file or directory" Errors
      3. 19.3. Forgetting That the Current Directory Is Not in the $PATH
      4. 19.4. Naming Your Script Test
      5. 19.5. Expecting to Change Exported Variables
      6. 19.6. Forgetting Quotes Leads to "command not found" on Assignments
      7. 19.7. Forgetting That Pattern Matching Alphabetizes
      8. 19.8. Forgetting That Pipelines Make Subshells
      9. 19.9. Making Your Terminal Sane Again
      10. 19.10. Deleting Files Using an Empty Variable
      11. 19.11. Seeing Odd Behavior from printf
      12. 19.12. Testing bash Script Syntax
      13. 19.13. Debugging Scripts
      14. 19.14. Avoiding "command not found" When Using Functions
      15. 19.15. Confusing Shell Wildcards and Regular Expressions
    23. A. Reference Lists
      1. A.1. bash Invocation
      2. A.2. Prompt String Customizations
      3. A.3. ANSI Color Escape Sequences
      4. A.4. Built-in Commands and Reserved Words
      5. A.5. Built-in Shell Variables
      6. A.6. set Options
      7. A.7. shopt options
      8. A.8. Adjusting Shell Behavior Using set, shopt, and Environment Variables
      9. A.9. Test Operators
      10. A.10. I/O Redirection
      11. A.11. echo Options and Escape Sequences
      12. A.12. printf
      13. A.13. Date and Time String Formatting with strftime
      14. A.14. Pattern-Matching Characters
      15. A.15. extglob Extended Pattern-Matching Operators
      16. A.16. tr Escape Sequences
      17. A.17. Readline Init File Syntax
      18. A.18. emacs Mode Commands
      19. A.19. vi Control Mode Commands
      20. A.20. Table of ASCII Values
    24. B. Examples Included with bash
      1. B.1. Startup-Files Directory Examples
    25. C. Command-Line Processing
      1. C.1. Command-Line Processing Steps
    26. D. Revision Control
      1. D.1. CVS
      2. D.2. Subversion
      3. D.3. RCS
      4. D.4. Other
    27. E. Building bash from Source
      1. E.1. Obtaining bash
      2. E.2. Unpacking the Archive
      3. E.3. What's in the Archive
      4. E.4. Who Do I Turn To?
    28. Index
    29. About the Authors
    30. Colophon
    31. SPECIAL OFFER: Upgrade this ebook with O’Reilly

Chapter 1. Beginning bash

What's a shell, and why should you care about it?

Any recent computer operating system (by recent, we mean since about 1970) has some sort of user interface—some way of specifying commands for the operating system to execute. But in lots of operating systems, that command interface was really built in and there was only one way to talk to the computer. Furthermore, an operating system's command interface would let you execute commands, but that was about all. After all, what else was there to do?

The Unix operating system popularized the notion of separating the shell (the part of the system that lets you type commands) from everything else: the input/output system, the scheduler, memory management, and all of the other things the operating system takes care of for you (and that most users don't want to care about). The shell was just one more program; it was a program whose job was executing other programs on behalf of users.

But that was the beginning of a revolution. The shell was just another program that ran on Unix, if you didn't like the standard one, you could create your own. So by the end of Unix's first decade, there were at least two competing shells: the Bourne Shell, sh (which was a descendant of the original Thomson shell), plus the C Shell, csh. By the end of Unix's second decade, there were a few more alternatives: the Korn shell, (ksh), and the first versions of the bash shell (bash). By the end of Unix's third decade, there were probably a dozen different shells.

You probably don't sit around saying "should I use csh or bash or ksh today?" You're probably happy with the standard shell that came with your Linux (or BSD or Mac OS X or Solaris or HP/UX) system. But disentangling the shell from the operating system itself made it much easier for software developers (such as Brian Fox, the creator of bash, and Chet Ramey, the current developer and maintainer of bash), to write better shells—you could create a new shell without modifying the operating system itself. It was much easier to get a new shell accepted, since you didn't have to talk some operating vendor into building the shell into their system; all you had to do was package the shell so that it could be installed just like any other program.

Still, that sounds like a lot of fuss for something that just takes commands and executes them. And you would be right—a shell that just let you type commands wouldn't be very interesting. However, two factors drove the evolution of the Unix shell: user convenience and programming. And the result is a modern shell that does much more than just accept commands.

Modern shells are very convenient. For example, they remember commands that you've typed, and let you re-use those commands. Modern shells also let you edit those commands, so they don't have to be the same each time. And modern shells let you define your own command abbreviations, shortcuts, and other features. For an experienced user, typing commands (e.g., with shorthand, shortcuts, command completion) is a lot more efficient and effective than dragging things around in a fancy windowed interface.

But beyond simple convenience, shells are programmable. There are many sequences of commands that you type again and again. Whenever you do anything a second time, you should ask "Can't I write a program to do this for me?" You can. A shell is also a programming language that's specially designed to work with your computer system's commands. So, if you want to generate a thousand MP3 files from WAV files, you write a shell program (or a shell script). If you want to compress all of your system's logfiles, you can write a shell script to do it. Whenever you find yourself doing a task repeatedly, you should try to automate it by writing a shell script. There are more powerful scripting languages, like Perl, Python, and Ruby, but the Unix shell (whatever flavor of shell you're using) is a great place to start. After all, you already know how to type commands; why make things more complex?

Why bash?

Why is this book about bash, and not some other shell? Because bash is everywhere. It may not be the newest, and it's arguably not the fanciest or the most powerful (though if not, it comes close), nor is it the only shell that's distributed as open source software, but it is ubiquitous.

The reason has to do with history. The first shells were fairly good programing tools, but not very convenient for users. The C shell added a lot of user conveniences (like the ability to repeat a command you just typed), but as a programming language it was quirky. The Korn shell, which came along next (in the early 80s), added a lot of user conveniences, and improved the programming language, and looked like it was on the path to widespread adoption. But ksh wasn't open source software at first; it was a proprietary software product, and was therefore difficult to ship with a free operating system like Linux. (The Korn shell's license was changed in 2000, and again in 2005.)

In the late 1980s, the Unix community decided standardization was a good thing, and the POSIX working groups (organized by the IEEE) were formed. POSIX standardized the Unix libraries and utilities, including the shell. The standard shell was primarily based on the 1988 version of the Korn Shell, with some C shell features and a bit of invention to fill in the gaps. bash was begun as part of the GNU project's effort to produce a complete POSIX system, which naturally needed a POSIX shell.

bash provided the programming features that shell programmers needed, plus the conveniences that command-line users liked. It was originally conceived as an alternative to the Korn shell, but as the free software movement became more important, and as Linux became more popular, bash quickly overshadowed ksh.

As a result, bash is the default user shell on every Linux distribution we know about (there are a few hundred Linux distros, so there are probably a few with some oddball default shell), as well as Mac OS X. It's also available for just about every other Unix operating system, including BSD Unix and Solaris. In the rare cases where bash doesn't ship with the operating system, it's easy to install. It's even available for Windows (via Cygwin). It's both a powerful programming language and a good user interface and you won't find yourself sacrificing keyboard shortcuts to get elaborate programming features.

You can't possibly go wrong by learning bash. The most common default shells are the old Bourne shell and bash, which is mostly Bourne shell compatible. One of these shells is certainly present on any modern, major Unix or Unix-like operating system. And as noted, if bash isn't present you can always install it. But there are other shells. In the spirit of free software, the authors and maintainers of all of these shells share ideas. If you read the bash change logs, you'll see many places where a feature was introduced or tweaked to match behavior on another shell. But most people won't care. They'll use whatever is already there and be happy with it. So if you are interested, by all means investigate other shells. There are many good alternatives and you may find one you like better—though it probably won't be as ubiquitous as bash.

The best content for your career. Discover unlimited learning on demand for around $1/day.