You are previewing Shell Scripting Recipes: A Problem-Solution Approach.
O'Reilly logo
Shell Scripting Recipes: A Problem-Solution Approach

Book Description

This book is geared towards any Unix user who doesn't want to spend time creating or testing shell scripts. Instead, Shell Scripting Recipes dissects and explains over 150 much-needed and practical real-world examples, and then shows the reader how and when to appropriately use them.

Because most scripts found in this book are POSIX (Portable Operating System Interface)-compliant, they are supported by many of the major shell variants, including Bash, ksh and sh, among others. File conversion, system administration, and resource monitoring are just a few of the topics covered in this highly practical shell scripting reference.

Table of Contents

  1. Copyright
  2. About the Author
  3. About the Technical Reviewer
  4. Acknowledgments
  5. Introduction
    1. Why Shell Scripts?
    2. Which Shells Will Run These Scripts?
    3. Looking Through the Book
    4. Thinking Outside the Book
  6. 1. The POSIX Shell and Command-Line Utilities
    1. 1.1. Shell Commands
      1. 1.1.1. echo
      2. 1.1.2. printf
      3. 1.1.3. set
      4. 1.1.4. shift
      5. 1.1.5. type
      6. 1.1.6. getopts
      7. 1.1.7. case
      8. 1.1.8. eval
      9. 1.1.9. local
    2. 1.2. Parameters and Variables
      1. 1.2.1. Positional Parameters
      2. 1.2.2. Special Parameters
      3. 1.2.3. standard-vars—A Collection of Useful Variables
    3. 1.3. Patterns
      1. 1.3.1. Pathname Expansion
      2. 1.3.2. Regular Expressions
    4. 1.4. Parameter Expansion
      1. 1.4.1. The Bourne Shell Expansions
      2. 1.4.2. POSIX Parameter Expansions
      3. 1.4.3. Shell-Specific Expansions, bash2, and ksh93
    5. 1.5. Shell Arithmetic
    6. 1.6. Aliases
    7. 1.7. Sourcing a File
    8. 1.8. Functions
      1. 1.8.1. Functions Are Fast
      2. 1.8.2. Command Substitution Is Slow
      3. 1.8.3. Using the Functions in This Book
    9. 1.9. standard-funcs: A Collection of Useful Commands
      1. 1.9.1. get_key—Get a Single Keystroke from the User
      2. 1.9.2. getline—Prompt User to Enter a Line
      3. 1.9.3. press_any_key—Prompt for a Single Keypress
      4. 1.9.4. menu1—Print a Menu and Execute a Selected Command
      5. 1.9.5. arg—Prompt for Required Argument If None Supplied
      6. 1.9.6. die—Print Error Message and Exit with Error Status
      7. 1.9.7. show_date—Display Date in D[D] MMM YYYY Format
      8. 1.9.8. date_vars—Set Date and Time Variables
      9. 1.9.9. is_num—Is This a Positive Integer?
      10. 1.9.10. abbrev_num—Abbreviate Large Numbers
      11. 1.9.11. commas—Add Thousands Separators to a Number
      12. 1.9.12. pr1—Print Arguments, One to a Line
      13. 1.9.13. checkdirs—Check for Directories; Create If Necessary
      14. 1.9.14. checkfiles—Check That a Directory Contains Certain Files
      15. 1.9.15. zpad—Pad a Number with Leading Zeroes
      16. 1.9.16. cleanup—Remove Temporary Files and Reset Terminal on Exit
    10. 1.10. The Unix Utilities
      1. 1.10.1. cat: Concatenate Files to the Standard Output
      2. 1.10.2. sed: A Text Stream Editor
      3. 1.10.3. awk: Pattern Scanning and Processing Language
      4. 1.10.4. grep: Print Lines Matching a Regular Expression
      5. 1.10.5. date: Show or Set the System Date
      6. 1.10.6. tr: A Character Translation Utility
      7. 1.10.7. wc: Count Characters, Words, and Lines in a File
      8. 1.10.8. file: Determine the File Type
      9. 1.10.9. ls: Sort and Provide Details About Files
      10. 1.10.10. uniq: Remove Consecutive Duplicate Lines
      11. 1.10.11. sudo: Execute Commands as the Superuser
      12. 1.10.12. split: Divide a File into Equal-Sized Pieces
      13. 1.10.13. which: Show the Full Path to a Command
      14. 1.10.14. gs, gv: Render, Convert, or View PostScript and PDF Files
    11. 1.11. Summary
  7. 2. Playing with Files: Viewing, Manipulating, and Editing Text Files
    1. 2.1. End of the Line for OS Differences: Converting Text Files
      1. 2.1.1. dos2unix—Convert Windows Text Files to Unix
      2. 2.1.2. unix2dos—Convert a Unix File to Windows
      3. 2.1.3. mac2unix—Convert Macintosh Files to Unix
      4. 2.1.4. unix2mac—Convert Unix Files to Mac Format
      5. 2.1.5. dos2mac—Convert Windows Files to Macintosh
      6. 2.1.6. mac2dos—Convert Macintosh Files to Windows
    2. 2.2. Displaying Files
      1. 2.2.1. prn—Print File with Line Numbers
      2. 2.2.2. prw—Print One Word per Line
      3. 2.2.3. wbl—Sort Words by Length
    3. 2.3. Formatting File Facts
      1. 2.3.1. finfo—File Information Display
      2. 2.3.2. wfreq—Word Frequency
      3. 2.3.3. lfreq—Letter Frequency
      4. 2.3.4. fed—A Simple Batch File Editor
    4. 2.4. Summary
  8. 3. String Briefs
    1. 3.1. Character Actions: The char-funcs Library
      1. 3.1.1. chr—Convert a Decimal Number to an ASCII Character
      2. 3.1.2. asc—Convert a Character to Its Decimal Equivalent
      3. 3.1.3. nxt—Get the Next Character in ASCII Sequence
      4. 3.1.4. upr—Convert Character(s) to Uppercase
      5. 3.1.5. lwr—Convert Character(s) to Lowercase
    2. 3.2. String Cleaning: The string-funcs Library
      1. 3.2.1. sub—Replace First Occurrence of a Pattern
      2. 3.2.2. gsub—Globally Replace a Pattern in a String
      3. 3.2.3. repeat—Build a String of a Specified Length
      4. 3.2.4. index, rindex—Find Position of One String Within Another
      5. 3.2.5. substr—Extract a Portion of a String
      6. 3.2.6. insert_str—Place One String Inside Another
    3. 3.3. Summary
  9. 4. What's in a Word?
    1. 4.1. Finding and Massaging Word Lists
    2. 4.2. wf-funcs: WordFinder Function Library
      1. 4.2.1. write_config—Write User's Information to the Configuration File
      2. 4.2.2. do_config—Check For and Source Default Configuration File
      3. 4.2.3. set_sysdict—Select the Dictionary Directory
      4. 4.2.4. mkwsig—Sort Letters in a Word
      5. 4.2.5. wf-clean—Remove Carriage Returns and Accents
      6. 4.2.6. wf-compounds—Squish Compound Words and Save with Lengths
      7. 4.2.7. wf-setup—Prepare Word and Anagram Lists
    3. 4.3. Playing with Matches
      1. 4.3.1. wf—Find Words That Match a Pattern
      2. 4.3.2. wfb—Find Words That Begin with a Given Pattern
      3. 4.3.3. wfe—Find Words That End with a Given Pattern
      4. 4.3.4. wfc—Find Words That Contain a Given Pattern
      5. 4.3.5. wfit—Find Words That Fit Together in a Grid
      6. 4.3.6. anagram—Find Words That Are Anagrams of a Given Word
      7. 4.3.7. aplus—Find Anagrams of a Word with a Letter Added
      8. 4.3.8. aminus—Remove Each Letter in Turn and Anagram What's Left
    4. 4.4. Summary
  10. 5. Scripting by Numbers
    1. 5.1. The math-funcs Library
      1. 5.1.1. fpmul—Multiply Decimal Fractions
      2. 5.1.2. int—Return the Integer Portion of a Decimal Fraction
      3. 5.1.3. round—Round the Argument to the Nearest Integer
      4. 5.1.4. pow—Raise a Number to Any Given Power
      5. 5.1.5. square—Raise a Number to the Second Power
      6. 5.1.6. cube—Raise a Number to the Third Power
      7. 5.1.7. calc—A Simple Command-Line Calculator
    2. 5.2. Adding and Averaging
      1. 5.2.1. total—Add a List of Numbers
      2. 5.2.2. mean—Find the Arithmetic Mean of a List of Numbers
      3. 5.2.3. median—Find the Median of a List of Numbers
      4. 5.2.4. mode—Find the Number That Appears Most in a List
      5. 5.2.5. range—Find the Range of a Set of Numbers
      6. 5.2.6. stdev—Finding the Standard Deviation
    3. 5.3. Converting Between Unit Systems
      1. 5.3.1. conversion-funcs—Converting Metric Units
      2. 5.3.2. conversion—A Menu System for Metric Conversion
    4. 5.4. Summary
  11. 6. Loose Names Sink Scripts: Bringing Sanity to Filenames
    1. 6.1. What's in a Name?
      1. 6.1.1. POSIX Portable Filenames
      2. 6.1.2. OK Filenames
    2. 6.2. Functioning Filenames: The filename-funcs Library
      1. 6.2.1. basename—Extract the Last Element of a Pathname
      2. 6.2.2. dirname—Return All but the Last Element of a Pathname
      3. 6.2.3. is_pfname—Check for POSIX Portable Filename
      4. 6.2.4. is_OKfname—Check Whether a Filename Is Acceptable
      5. 6.2.5. pfname—Convert Nonportable Characters in Filename
      6. 6.2.6. OKfname—Make a Filename Acceptable
      7. 6.2.7. is_whitespc—Does the Filename Contain Whitespace Characters?
      8. 6.2.8. whitespc—Fix Filenames Containing Whitespace Characters
      9. 6.2.9. is_dir—Is This a Directory I See Before Me?
      10. 6.2.10. nodoublechar—Remove Duplicate Characters from a String
      11. 6.2.11. new_filename—Change Filename to Desired Character Set
      12. 6.2.12. fix_pwd—Fix All the Filenames in the Current Directory
      13. 6.2.13. fixfname—Convert Filenames to Sensible Names
    3. 6.3. Summary
  12. 7. Treading a Righteous PATH
    1. 7.1. The path-funcs Library
      1. 7.1.1. path—Display a User's PATH
      2. 7.1.2. unslash—Remove Extraneous Slashes
      3. 7.1.3. checkpath—Clean Up the PATH Variable
      4. 7.1.4. addpath—Add Directories to the PATH Variable
      5. 7.1.5. rmpath—Remove One or More Directories from $PATH
    2. 7.2. Summary
  13. 8. The Dating Game
    1. 8.1. The date-funcs Library
      1. 8.1.1. split_date—Divide a Date into Day, Month, and Year
      2. 8.1.2. is_leap_year—Is There an Extra Day This Year?
      3. 8.1.3. days_in_month—How Many Days Hath September?
      4. 8.1.4. date2julian—Calculate the Julian Day Number
      5. 8.1.5. julian2date—Convert Julian Back to Year, Month, and Day
      6. 8.1.6. dateshift—Add or Subtract a Number of Days
      7. 8.1.7. diffdate—Find the Number of Days Between Two Dates
      8. 8.1.8. day_of_week—Find the Day of the Week for Any Date
      9. 8.1.9. display_date—Show a Date in Text Format
      10. 8.1.10. parse_date—Decipher Various Forms of Date String
      11. 8.1.11. valid_date—Where Was I on November 31st?
    2. 8.2. Summary
  14. 9. Good Housekeeping: Monitoring and Tidying Up File Systems
    1. 9.1.
      1. 9.1.1. dfcmp—Notify User of Major Changes in Disk Usage
      2. 9.1.2. symfix—Remove Broken Symbolic Links
      3. 9.1.3. sym2file—Converts Symbolic Links to Regular Files
      4. 9.1.4. zrm—Remove Empty Files
      5. 9.1.5. undup—Remove Duplicate Files
      6. 9.1.6. lsr—List the Most Recent (or Oldest) Files in a Directory
    2. 9.2. Summary
  15. 10. POP Goes the E-Mail
    1. 10.1.
      1. 10.1.1. popcheck—Display Number of Messages in POP3 Mailbox
      2. 10.1.2. pop3-funcs—Functions for Managing a POP3 Mailbox
      3. 10.1.3. pop3list—Examine a POP3 Mailbox
      4. 10.1.4. pop3filter—Delete and Retrieve Messages from a POP3 Server
      5. 10.1.5. pflog—Print a Summary of a Day's Activity
      6. 10.1.6. viewlog—Continuously View the pop3filter Log File
    2. 10.2. Summary
  16. 11. PostScript: More Than an Afterthought
    1. 11.1.
      1. 11.1.1. ps-labels—Producing Mailing Labels from a List
      2. 11.1.2. ps-envelopes—Print Envelopes from List
      3. 11.1.3. ps-grid—Convert a Crossword Grid to PostScript
    2. 11.2. Summary
  17. 12. Screenplay: The screen—funcs Library
    1. 12.1.
      1. 12.1.1. screen-vars—Variables for Screen Manipulation
      2. 12.1.2. set_attr—Set Screen-Printing Attributes
      3. 12.1.3. set_fg, set_bg, set_fgbg-—Set Colors forPrinting to the Screen
      4. 12.1.4. cls—Clear the Screen
      5. 12.1.5. printat—Position Cursor by Row and Column
      6. 12.1.6. put_block_at—Print Lines in a Column Anywhere on the Screen
      7. 12.1.7. get_size—Set LINES and COLUMNS Variables
      8. 12.1.8. max_length—Find the Length of the Longest Argument
      9. 12.1.9. print_block_atxy—Print a Block of Lines Anywhere on the Screen
      10. 12.1.10. vbar, hbar—Print a Vertical or Horizontal Bar
      11. 12.1.11. center—Center a String on N Columns
      12. 12.1.12. flush_right—Align String with the Right Margin
      13. 12.1.13. ruler—Draw a Ruler Across the Width and Height of the Window
      14. 12.1.14. box_block, box_block_at—Print Text Surrounded by a Box
      15. 12.1.15. clear_area, clear_area_at—Clear an Area of the Screen
      16. 12.1.16. box_area, box_area_at—Draw a Box Around an Area
      17. 12.1.17. screen-demo—Saving and Redisplaying Areas of the Screen
    2. 12.2. Summary
  18. 13. Backing Up the Drive
    1. 13.1.
      1. 13.1.1. bu—Backup Directories
      2. 13.1.2. bin-pack—Fill Fixed-Capacity Directories with Files
      3. 13.1.3. unpack—Extract All Files from One or More Archives
    2. 13.2. Summary
  19. 14. Aging, Archiving, and Deleting Files
    1. 14.1.
      1. 14.1.1. date-file—Add a Datestamp to a Filename
      2. 14.1.2. rmold—Remove Old Files
      3. 14.1.3. keepnewest—Remove All but the Newest or Oldest Files
    2. 14.2. Summary
  20. 15. Covering All Your Databases
    1. 15.1.
      1. 15.1.1. lookup—Find the Corresponding Value for a Key
    2. 15.2. shdb-funcs: Shell Database Function Library
      1. 15.2.1. load_db—Import Database into Shell Array
      2. 15.2.2. split_record—Split a Record into Fields
      3. 15.2.3. csv_split—Extract Fields from CSV Records
      4. 15.2.4. put_record—Assemble a Database Record from an Array
      5. 15.2.5. put_csv—Assemble Fields into a CSV Record
      6. 15.2.6. db-demo—View and Edit a Password File
    3. 15.3. PhoneBase: A Simple Phone Number Database
      1. 15.3.1. ph—Look Up a Phone Number
      2. 15.3.2. phadd—Add an Entry to PhoneBase
      3. 15.3.3. phdel—Delete an Entry from PhoneBase
      4. 15.3.4. phx—Show ph Search Results in an X Window
    4. 15.4. Summary
  21. 16. Home on the Web
    1. 16.1. Playing with Hypertext: The html-funcs Library
      1. 16.1.1. get_element—Extract the First Occurrence of an Element
      2. 16.1.2. split_tags—Put Each Tag on Its Own Line
      3. 16.1.3. html-title—Get the Title from an HTML File
    2. 16.2. HTML on the Fly: The cgi-funcs Library
      1. 16.2.1. x2d2—Convert a Two-Digit Hexadecimal Number to Decimal
      2. 16.2.2. dehex—Convert Hex Strings (%XX) to Characters
      3. 16.2.3. filedate—Find and Format the Modification Date of a File
    3. 16.3. Creating HTML Files
      1. 16.3.1. mk-htmlindex—Create an HTML Index
      2. 16.3.2. pretext—Create a Wrapper Around a Text File
      3. 16.3.3. text2html—Convert a Text File to HTML
      4. 16.3.4. demo.cgi—A CGI Script
    4. 16.4. Summary
  22. 17. Taking Care of Business
    1. 17.1.
      1. 17.1.1. prcalc—A Printing Calculator
      2. 17.1.2. gle—Keeping Records Without a Shoebox
    2. 17.2. Summary
  23. 18. Random Acts of Scripting
    1. 18.1. The rand-funcs Library
      1. 18.1.1. random—Return One or More Random Integers in a Given Range
      2. 18.1.2. toss—Simulate Tossing a Coin
      3. 18.1.3. randstr—Select a String at Random
    2. 18.2. A Random Sampling of Scripts
      1. 18.2.1. rand-date—Generate Random Dates in ISO Format
      2. 18.2.2. randsort—Print Lines in Random Order
      3. 18.2.3. randomword—Generate Random Words According to Format Specifications
      4. 18.2.4. dice—Roll a Set of Dice
      5. 18.2.5. throw—Throw a Pair of Dice
    3. 18.3. Summary
  24. 19. A Smorgasbord of Scripts
    1. 19.1.
      1. 19.1.1. topntail—Remove Top and Bottom Lines from a File
      2. 19.1.2. flocate—Locate Files by Filename Alone
      3. 19.1.3. sus—Display a POSIX Man Page
      4. 19.1.4. cwbw—Count Words Beginning With
      5. 19.1.5. cci—Configure, Compile, and Install from Tarball
      6. 19.1.6. ipaddr—Find a Computer's Network Address
      7. 19.1.7. ipaddr.cgi—Print the Remote Address of an HTTP Connection
      8. 19.1.8. iprev—Reverse the Order of Digits in an IP Address
      9. 19.1.9. intersperse—Insert a String Between Doubled Characters
      10. 19.1.10. ll—Use a Pager for a Directory Listing Only If Necessary
      11. 19.1.11. name-split—Divide a Person's Full Name into First, Last, and Middle Names
      12. 19.1.12. rot13—Encode or Decode Text
      13. 19.1.13. showfstab—Show Information from /etc/fstab
      14. 19.1.14. unique—Remove All Duplicate Lines from a File
    2. 19.2. Summary
  25. 20. Script Development Management
    1. 20.1.
      1. 20.1.1. script-setup—Prepare the Scripting Environment
      2. 20.1.2. cpsh—Install Script and Make Backup Copy
      3. 20.1.3. shgrep—Search Scripts for String or Regular Expression
      4. 20.1.4. shcat—Display a Shell Script
    2. 20.2. Summary
  26. A. Internet Scripting Resources
    1. A.1. Introductions to Shell Scripting
    2. A.2. Intermediate and Advanced Scripting
    3. A.3. Collections of Scripts
    4. A.4. Home Pages for Shells
    5. A.5. Regular Expressions, sed, and awk
    6. A.6. Miscellaneous Pages
    7. A.7. History of the Shell