You are previewing bash Cookbook.

bash Cookbook

Cover of bash Cookbook by Carl Albing... Published by O'Reilly Media, Inc.
  1. bash Cookbook
    1. SPECIAL OFFER: Upgrade this ebook with O’Reilly
    2. A Note Regarding Supplemental Files
    3. Preface
      1. Who Should Read This Book
      2. About This Book
      3. Conventions Used in This Book
      4. Using Code Examples
      5. We'd Like to Hear from You
      6. Safari® Enabled
      7. Acknowledgments
    4. 1. Beginning bash
      1. 0. Why bash?
      2. 0. The bash Shell
      3. 1.1. Decoding the Prompt
      4. 1.2. Showing Where You Are
      5. 1.3. Finding and Running Commands
      6. 1.4. Getting Information About Files
      7. 1.5. Showing All Hidden (dot) Files in the Current Directory
      8. 1.6. Using Shell Quoting
      9. 1.7. Using or Replacing Built-ins and External Commands
      10. 1.8. Determining If You Are Running Interactively
      11. 1.9. Setting bash As Your Default Shell
      12. 1.10. Getting bash for Linux
      13. 1.11. Getting bash for xBSD
      14. 1.12. Getting bash for Mac OS X
      15. 1.13. Getting bash for Unix
      16. 1.14. Getting bash for Windows
      17. 1.15. Getting bash Without Getting bash
      18. 1.16. Learning More About bash Documentation
    5. 2. Standard Output
      1. 2.1. Writing Output to the Terminal/Window
      2. 2.2. Writing Output but Preserving Spacing
      3. 2.3. Writing Output with More Formatting Control
      4. 2.4. Writing Output Without the Newline
      5. 2.5. Saving Output from a Command
      6. 2.6. Saving Output to Other Files
      7. 2.7. Saving Output from the ls Command
      8. 2.8. Sending Both Output and Error Messages to Different Files
      9. 2.9. Sending Both Output and Error Messages to the Same File
      10. 2.10. Appending Rather Than Clobbering Output
      11. 2.11. Using Just the Beginning or End of a File
      12. 2.12. Skipping a Header in a File
      13. 2.13. Throwing Output Away
      14. 2.14. Saving or Grouping Output from Several Commands
      15. 2.15. Connecting Two Programs by Using Output As Input
      16. 2.16. Saving a Copy of Output Even While Using It As Input
      17. 2.17. Connecting Two Programs by Using Output As Arguments
      18. 2.18. Using Multiple Redirects on One Line
      19. 2.19. Saving Output When Redirect Doesn't Seem to Work
      20. 2.20. Swapping STDERR and STDOUT
      21. 2.21. Keeping Files Safe from Accidental Overwriting
      22. 2.22. Clobbering a File on Purpose
    6. 3. Standard Input
      1. 3.1. Getting Input from a File
      2. 3.2. Keeping Your Data with Your Script
      3. 3.3. Preventing Weird Behavior in a Here-Document
      4. 3.4. Indenting Here-Documents
      5. 3.5. Getting User Input
      6. 3.6. Getting Yes or No Input
      7. 3.7. Selecting from a List of Options
      8. 3.8. Prompting for a Password
    7. 4. Executing Commands
      1. 4.1. Running Any Executable
      2. 4.2. Telling If a Command Succeeded or Not
      3. 4.3. Running Several Commands in Sequence
      4. 4.4. Running Several Commands All at Once
      5. 4.5. Deciding Whether a Command Succeeds
      6. 4.6. Using Fewer if Statements
      7. 4.7. Running Long Jobs Unattended
      8. 4.8. Displaying Error Messages When Failures Occur
      9. 4.9. Running Commands from a Variable
      10. 4.10. Running All Scripts in a Directory
    8. 5. Basic Scripting: Shell Variables
      1. 5.1. Documenting Your Script
      2. 5.2. Embedding Documentation in Shell Scripts
      3. 5.3. Promoting Script Readability
      4. 5.4. Separating Variable Names from Surrounding Text
      5. 5.5. Exporting Variables
      6. 5.6. Seeing All Variable Values
      7. 5.7. Using Parameters in a Shell Script
      8. 5.8. Looping Over Arguments Passed to a Script
      9. 5.9. Handling Parameters with Blanks
      10. 5.10. Handling Lists of Parameters with Blanks
      11. 5.11. Counting Arguments
      12. 5.12. Consuming Arguments
      13. 5.13. Getting Default Values
      14. 5.14. Setting Default Values
      15. 5.15. Using null As a Valid Default Value
      16. 5.16. Using More Than Just a Constant String for Default
      17. 5.17. Giving an Error Message for Unset Parameters
      18. 5.18. Changing Pieces of a String
      19. 5.19. Using Array Variables
    9. 6. Shell Logic and Arithmetic
      1. 6.1. Doing Arithmetic in Your Shell Script
      2. 6.2. Branching on Conditions
      3. 6.3. Testing for File Characteristics
      4. 6.4. Testing for More Than One Thing
      5. 6.5. Testing for String Characteristics
      6. 6.6. Testing for Equal
      7. 6.7. Testing with Pattern Matches
      8. 6.8. Testing with Regular Expressions
      9. 6.9. Changing Behavior with Redirections
      10. 6.10. Looping for a While
      11. 6.11. Looping with a read
      12. 6.12. Looping with a Count
      13. 6.13. Looping with Floating-Point Values
      14. 6.14. Branching Many Ways
      15. 6.15. Parsing Command-Line Arguments
      16. 6.16. Creating Simple Menus
      17. 6.17. Changing the Prompt on Simple Menus
      18. 6.18. Creating a Simple RPN Calculator
      19. 6.19. Creating a Command-Line Calculator
    10. 7. Intermediate Shell Tools I
      1. 7.1. Sifting Through Files for a String
      2. 7.2. Getting Just the Filename from a Search
      3. 7.3. Getting a Simple True/False from a Search
      4. 7.4. Searching for Text While Ignoring Case
      5. 7.5. Doing a Search in a Pipeline
      6. 7.6. Paring Down What the Search Finds
      7. 7.7. Searching with More Complex Patterns
      8. 7.8. Searching for an SSN
      9. 7.9. Grepping Compressed Files
      10. 7.10. Keeping Some Output, Discarding the Rest
      11. 7.11. Keeping Only a Portion of a Line of Output
      12. 7.12. Reversing the Words on Each Line
      13. 7.13. Summing a List of Numbers
      14. 7.14. Counting String Values
      15. 7.15. Showing Data As a Quick and Easy Histogram
      16. 7.16. Showing a Paragraph of Text After a Found Phrase
    11. 8. Intermediate Shell Tools II
      1. 8.1. Sorting Your Output
      2. 8.2. Sorting Numbers
      3. 8.3. Sorting IP Addresses
      4. 8.4. Cutting Out Parts of Your Output
      5. 8.5. Removing Duplicate Lines
      6. 8.6. Compressing Files
      7. 8.7. Uncompressing Files
      8. 8.8. Checking a tar Archive for Unique Directories
      9. 8.9. Translating Characters
      10. 8.10. Converting Uppercase to Lowercase
      11. 8.11. Converting DOS Files to Linux Format
      12. 8.12. Removing Smart Quotes
      13. 8.13. Counting Lines, Words, or Characters in a File
      14. 8.14. Rewrapping Paragraphs
      15. 8.15. Doing More with less
    12. 9. Finding Files: find, locate, slocate
      1. 9.1. Finding All Your MP3 Files
      2. 9.2. Handling Filenames Containing Odd Characters
      3. 9.3. Speeding Up Operations on Found Files
      4. 9.4. Finding Files Across Symbolic Links
      5. 9.5. Finding Files Irrespective of Case
      6. 9.6. Finding Files by Date
      7. 9.7. Finding Files by Type
      8. 9.8. Finding Files by Size
      9. 9.9. Finding Files by Content
      10. 9.10. Finding Existing Files and Content Fast
      11. 9.11. Finding a File Using a List of Possible Locations
    13. 10. Additional Features for Scripting
      1. 10.1. "Daemon-izing" Your Script
      2. 10.2. Reusing Code with Includes and Sourcing
      3. 10.3. Using Configuration Files in a Script
      4. 10.4. Defining Functions
      5. 10.5. Using Functions: Parameters and Return Values
      6. 10.6. Trapping Interrupts
      7. 10.7. Redefining Commands with alias
      8. 10.8. Avoiding Aliases, Functions
    14. 11. Working with Dates and Times
      1. 11.1. Formatting Dates for Display
      2. 11.2. Supplying a Default Date
      3. 11.3. Automating Date Ranges
      4. 11.4. Converting Dates and Times to Epoch Seconds
      5. 11.5. Converting Epoch Seconds to Dates and Times
      6. 11.6. Getting Yesterday or Tomorrow with Perl
      7. 11.7. Figuring Out Date and Time Arithmetic
      8. 11.8. Handling Time Zones, Daylight Saving Time, and Leap Years
      9. 11.9. Using date and cron to Run a Script on the Nth Day
    15. 12. End-User Tasks As Shell Scripts
      1. 12.1. Starting Simple by Printing Dashes
      2. 12.2. Viewing Photos in an Album
      3. 12.3. Loading Your MP3 Player
      4. 12.4. Burning a CD
      5. 12.5. Comparing Two Documents
    16. 13. Parsing and Similar Tasks
      1. 13.1. Parsing Arguments for Your Shell Script
      2. 13.2. Parsing Arguments with Your Own Error Messages
      3. 13.3. Parsing Some HTML
      4. 13.4. Parsing Output into an Array
      5. 13.5. Parsing Output with a Function Call
      6. 13.6. Parsing Text with a read Statement
      7. 13.7. Parsing with read into an Array
      8. 13.8. Getting Your Plurals Right
      9. 13.9. Taking It One Character at a Time
      10. 13.10. Cleaning Up an SVN Source Tree
      11. 13.11. Setting Up a Database with MySQL
      12. 13.12. Isolating Specific Fields in Data
      13. 13.13. Updating Specific Fields in Data Files
      14. 13.14. Trimming Whitespace
      15. 13.15. Compressing Whitespace
      16. 13.16. Processing Fixed-Length Records
      17. 13.17. Processing Files with No Line Breaks
      18. 13.18. Converting a Data File to CSV
      19. 13.19. Parsing a CSV Data File
    17. 14. Writing Secure Shell Scripts
      1. 14.1. Avoiding Common Security Problems
      2. 14.2. Avoiding Interpreter Spoofing
      3. 14.3. Setting a Secure $PATH
      4. 14.4. Clearing All Aliases
      5. 14.5. Clearing the Command Hash
      6. 14.6. Preventing Core Dumps
      7. 14.7. Setting a Secure $IFS
      8. 14.8. Setting a Secure umask
      9. 14.9. Finding World-Writable Directories in Your $PATH
      10. 14.10. Adding the Current Directory to the $PATH
      11. 14.11. Using Secure Temporary Files
      12. 14.12. Validating Input
      13. 14.13. Setting Permissions
      14. 14.14. Leaking Passwords into the Process List
      15. 14.15. Writing setuid or setgid Scripts
      16. 14.16. Restricting Guest Users
      17. 14.17. Using chroot Jails
      18. 14.18. Running As a Non-root User
      19. 14.19. Using sudo More Securely
      20. 14.20. Using Passwords in Scripts
      21. 14.21. Using SSH Without a Password
      22. 14.22. Restricting SSH Commands
      23. 14.23. Disconnecting Inactive Sessions
    18. 15. Advanced Scripting
      1. 15.1. Finding bash Portably for #!
      2. 15.2. Setting a POSIX $PATH
      3. 15.3. Developing Portable Shell Scripts
      4. 15.4. Testing Scripts in VMware
      5. 15.5. Using for Loops Portably
      6. 15.6. Using echo Portably
      7. 15.7. Splitting Output Only When Necessary
      8. 15.8. Viewing Output in Hex
      9. 15.9. Using bash Net-Redirection
      10. 15.10. Finding My IP Address
      11. 15.11. Getting Input from Another Machine
      12. 15.12. Redirecting Output for the Life of a Script
      13. 15.13. Working Around "argument list too long" Errors
      14. 15.14. Logging to syslog from Your Script
      15. 15.15. Sending Email from Your Script
      16. 15.16. Automating a Process Using Phases
    19. 16. Configuring and Customizing bash
      1. 16.1. bash Startup Options
      2. 16.2. Customizing Your Prompt
      3. 16.3. Change Your $PATH Permanently
      4. 16.4. Change Your $PATH Temporarily
      5. 16.5. Setting Your $CDPATH
      6. 16.6. Shortening or Changing Command Names
      7. 16.7. Adjusting Shell Behavior and Environment
      8. 16.8. Adjusting readline Behavior Using .inputrc
      9. 16.9. Keeping a Private Stash of Utilities by Adding ~/bin
      10. 16.10. Using Secondary Prompts: $PS2, $PS3, $PS4
      11. 16.11. Synchronizing Shell History Between Sessions
      12. 16.12. Setting Shell History Options
      13. 13. Creating a Better cd Command
      14. 16.14. Creating and Changing into a New Directory in One Step
      15. 16.15. Getting to the Bottom of Things
      16. 16.16. Adding New Features to bash Using Loadable Built-ins
      17. 16.17. Improving Programmable Completion
      18. 16.18. Using Initialization Files Correctly
      19. 16.19. Creating Self-Contained, Portable RC Files
      20. 16.20. Getting Started with a Custom Configuration
    20. 17. Housekeeping and Administrative Tasks
      1. 17.1. Renaming Many Files
      2. 17.2. Using GNU Texinfo and Info on Linux
      3. 17.3. Unzipping Many ZIP Files
      4. 17.4. Recovering Disconnected Sessions Using screen
      5. 17.5. Sharing a Single bash Session
      6. 17.6. Logging an Entire Session or Batch Job
      7. 17.7. Clearing the Screen When You Log Out
      8. 17.8. Capturing File Metadata for Recovery
      9. 17.9. Creating an Index of Many Files
      10. 17.10. Using diff and patch
      11. 17.11. Counting Differences in Files
      12. 17.12. Removing or Renaming Files Named with Special Characters
      13. 17.13. Prepending Data to a File
      14. 17.14. Editing a File in Place
      15. 17.15. Using sudo on a Group of Commands
      16. 17.16. Finding Lines in One File But Not in the Other
      17. 17.17. Keeping the Most Recent N Objects
      18. 17.18. Grepping ps Output Without Also Getting the grep Process Itself
      19. 17.19. Finding Out Whether a Process Is Running
      20. 17.20. Adding a Prefix or Suffix to Output
      21. 17.21. Numbering Lines
      22. 17.22. Writing Sequences
      23. 17.23. Emulating the DOS Pause Command
      24. 17.24. Commifying Numbers
    21. 18. Working Faster by Typing Less
      1. 18.1. Moving Quickly Among Arbitrary Directories
      2. 18.2. Repeating the Last Command
      3. 18.3. Running Almost the Same Command
      4. 18.4. Substituting Across Word Boundaries
      5. 18.5. Reusing Arguments
      6. 18.6. Finishing Names for You
      7. 18.7. Playing It Safe
    22. 19. Tips and Traps: Common Goofs for Novices
      1. 19.1. Forgetting to Set Execute Permissions
      2. 19.2. Fixing "No such file or directory" Errors
      3. 19.3. Forgetting That the Current Directory Is Not in the $PATH
      4. 19.4. Naming Your Script Test
      5. 19.5. Expecting to Change Exported Variables
      6. 19.6. Forgetting Quotes Leads to "command not found" on Assignments
      7. 19.7. Forgetting That Pattern Matching Alphabetizes
      8. 19.8. Forgetting That Pipelines Make Subshells
      9. 19.9. Making Your Terminal Sane Again
      10. 19.10. Deleting Files Using an Empty Variable
      11. 19.11. Seeing Odd Behavior from printf
      12. 19.12. Testing bash Script Syntax
      13. 19.13. Debugging Scripts
      14. 19.14. Avoiding "command not found" When Using Functions
      15. 19.15. Confusing Shell Wildcards and Regular Expressions
    23. A. Reference Lists
      1. A.1. bash Invocation
      2. A.2. Prompt String Customizations
      3. A.3. ANSI Color Escape Sequences
      4. A.4. Built-in Commands and Reserved Words
      5. A.5. Built-in Shell Variables
      6. A.6. set Options
      7. A.7. shopt options
      8. A.8. Adjusting Shell Behavior Using set, shopt, and Environment Variables
      9. A.9. Test Operators
      10. A.10. I/O Redirection
      11. A.11. echo Options and Escape Sequences
      12. A.12. printf
      13. A.13. Date and Time String Formatting with strftime
      14. A.14. Pattern-Matching Characters
      15. A.15. extglob Extended Pattern-Matching Operators
      16. A.16. tr Escape Sequences
      17. A.17. Readline Init File Syntax
      18. A.18. emacs Mode Commands
      19. A.19. vi Control Mode Commands
      20. A.20. Table of ASCII Values
    24. B. Examples Included with bash
      1. B.1. Startup-Files Directory Examples
    25. C. Command-Line Processing
      1. C.1. Command-Line Processing Steps
    26. D. Revision Control
      1. D.1. CVS
      2. D.2. Subversion
      3. D.3. RCS
      4. D.4. Other
    27. E. Building bash from Source
      1. E.1. Obtaining bash
      2. E.2. Unpacking the Archive
      3. E.3. What's in the Archive
      4. E.4. Who Do I Turn To?
    28. Index
    29. About the Authors
    30. Colophon
    31. SPECIAL OFFER: Upgrade this ebook with O’Reilly
O'Reilly logo

Appendix C. Command-Line Processing

Throughout the book we've seen a variety of ways in which the shell processes input lines, especially using read. We can think of this process as a subset of the things the shell does when processing command lines. This appendix provides a more detailed description of the steps involved in processing the command line and how you can get bash to make a second pass with eval. The material in this Appendix Also appears in Learning the bash Shell by Cameron Newham (O'Reilly).

Command-Line Processing Steps

We've touched upon command-line processing throughout this book; we've mentioned how bash deals with single quotes (''), double quotes (";"), and backslashes (\); how it separates characters on a line into words, even allowing you to specify the delimiter it uses via the environment variable $IFS; how it assigns the words to shell variables (e.g., $1, $2, etc); and how it can redirect input and output to/from files or to other processes (pipeline). In order to be a real expert at shell scripting (or to debug some gnarly problems), you might need to understand the various steps involved in command-line processing—especially the order in which they occur.

Each line that the shell reads from STDIN or from a script is called a pipeline because it contains one or more commands separated by zero or more pipe characters (|). Figure C-1 shows the steps in command-line processing. For each pipeline it reads, the shell breaks it up into commands, sets up the ...

The best content for your career. Discover unlimited learning on demand for around $1/day.