You are previewing Linux Shell Scripting Cookbook - Second Edition.
O'Reilly logo
Linux Shell Scripting Cookbook - Second Edition

Book Description

Don’t neglect the shell – this book will empower you to use simple commands to perform complex tasks. Whether you’re a casual or advanced Linux user, the cookbook approach makes it all so brilliantly accessible and, above all, useful.

  • Master the art of crafting one-liner command sequence to perform text processing, digging data from files, backups to sysadmin tools, and a lot more

  • And if powerful text processing isn't enough, see how to make your scripts interact with the web-services like Twitter, Gmail

  • Explores the possibilities with the shell in a simple and elegant way - you will see how to effectively solve problems in your day to day life

  • In Detail

    The shell remains one of the most powerful tools on a computer system — yet a large number of users are unaware of how much one can accomplish with it. Using a combination of simple commands, we will see how to solve complex problems in day to day computer usage.

    Linux Shell Scripting Cookbook, Second Edition will take you through useful real-world recipes designed to make your daily life easy when working with the shell. The book shows the reader how to effectively use the shell to accomplish complex tasks with ease.

    The book discusses basics of using the shell, general commands and proceeds to show the reader how to use them to perform complex tasks with ease.

    Starting with the basics of the shell, we will learn simple commands with their usages allowing us to perform operations on files of different kind. The book then proceeds to explain text processing, web interaction and concludes with backups, monitoring and other sysadmin tasks.

    Linux Shell Scripting Cookbook, Second Edition serves as an excellent guide to solving day to day problems using the shell and few powerful commands together to create solutions.

    Table of Contents

    1. Linux Shell Scripting Cookbook
      1. Table of Contents
      2. Linux Shell Scripting Cookbook
      3. Credits
      4. About the Authors
      5. About the Reviewers
      6. www.PacktPub.com
        1. Support files, eBooks, discount offers and more
          1. Why Subscribe?
          2. Free Access for Packt account holders
      7. Preface
        1. What this book covers
        2. What you need for this book
        3. Who this book is for
        4. Conventions
        5. Reader feedback
        6. Customer support
          1. Downloading the example code
          2. Errata
          3. Piracy
          4. Questions
      8. 1. Shell Something Out
        1. Introduction
        2. Printing in the terminal
          1. How to do it...
          2. How it works...
          3. There's more...
            1. Escaping newline in echo
            2. Printing a colored output
        3. Playing with variables and environment variables
          1. Getting ready
          2. How to do it...
          3. There's more...
            1. Finding the length of a string
            2. Identifying the current shell
            3. Checking for super user
            4. Modifying the Bash prompt string (username@hostname:~$)
        4. Function to prepend to environment variables
          1. How to do it...
          2. How it works...
        5. Math with the shell
          1. Getting ready
          2. How to do it...
        6. Playing with file descriptors and redirection
          1. Getting ready
          2. How to do it...
          3. How it works...
          4. There's more...
            1. Redirection from a file to a command
            2. Redirecting from a text block enclosed within a script
            3. Custom file descriptors
        7. Arrays and associative arrays
          1. Getting ready
          2. How to do it...
          3. There's more...
            1. Defining associative arrays
            2. Listing of array indexes
        8. Visiting aliases
          1. How to do it...
          2. There's more...
            1. Escaping aliases
        9. Grabbing information about the terminal
          1. Getting ready
          2. How to do it...
        10. Getting and setting dates and delays
          1. Getting ready
          2. How to do it...
          3. How it works...
          4. There's more...
            1. Producing delays in a script
        11. Debugging the script
          1. How to do it...
          2. How it works...
          3. There's more...
            1. Shebang hack
        12. Functions and arguments
          1. How to do it...
          2. There's more...
            1. The recursive function
            2. Exporting functions
            3. Reading the return value (status) of a command
            4. Passing arguments to commands
        13. Reading the output of a sequence of commands in a variable
          1. Getting ready
          2. How to do it...
          3. There's more...
            1. Spawning a separate process with subshell
            2. Subshell quoting to preserve spacing and the newline character
        14. Reading n characters without pressing the return key
          1. How to do it...
        15. Running a command until it succeeds
          1. How to do it...
          2. How it works...
          3. There's more...
            1. A faster approach
            2. Adding a delay
        16. Field separators and iterators
          1. Getting ready
          2. How to do it...
        17. Comparisons and tests
          1. How to do it...
      9. 2. Have a Good Command
        1. Introduction
        2. Concatenating with cat
          1. How to do it...
          2. How it works…
          3. There's more...
            1. Getting rid of extra blank lines
            2. Displaying tabs as ^I
            3. Line numbers
        3. Recording and playing back of terminal sessions
          1. Getting ready
          2. How to do it...
          3. How it works...
        4. Finding files and file listing
          1. Getting ready
          2. How to do it...
          3. There's more...
            1. Search based on filename or regular expression match
            2. Negating arguments
            3. Search based on the directory depth
            4. Search based on file type
            5. Search on file times
            6. Search based on file size
            7. Deleting based on the file matches
            8. Match based on the file permissions and ownership
            9. Executing commands or actions with find
            10. Skipping specified directories when using the find command
        5. Playing with xargs
          1. Getting ready
          2. How to do it...
          3. How it works…
          4. There's more...
            1. Passing formatted arguments to a command by reading stdin
            2. Using xargs with find
            3. Counting the number of lines of C code in a source code directory
            4. While and subshell trick with stdin
        6. Translating with tr
          1. Getting ready
          2. How to do it...
          3. How it works…
          4. There's more...
            1. Deleting characters using tr
            2. Complementing character set
            3. Squeezing characters with tr
            4. Character classes
        7. Checksum and verification
          1. Getting ready
          2. How to do it...
          3. How it works...
          4. There's more...
            1. Checksum for directories
        8. Cryptographic tools and hashes
          1. How to do it...
        9. Sorting unique and duplicates
          1. Getting ready
          2. How to do it...
          3. How it works…
          4. There's more...
            1. Sorting according to the keys or columns
            2. uniq
        10. Temporary file naming and random numbers
          1. How to do it...
          2. How it works…
        11. Splitting files and data
          1. How to do it...
          2. There's more…
            1. Specifying a filename prefix for the split files
        12. Slicing filenames based on extension
          1. How to do it…
          2. How it works…
        13. Renaming and moving files in bulk
          1. Getting ready
          2. How to do it...
          3. How it works…
        14. Spell checking and dictionary manipulation
          1. How to do it...
          2. How it works...
        15. Automating interactive input
          1. Getting ready
          2. How to do it...
          3. How it works…
          4. There's more...
            1. Automating with expect
        16. Making commands quicker by running parallel processes
          1. How to do it...
          2. How it works...
      10. 3. File In, File Out
        1. Introduction
        2. Generating files of any size
          1. How to do it...
        3. The intersection and set difference (A-B) on text files
          1. Getting ready
          2. How to do it...
          3. How it works...
        4. Finding and deleting duplicate files
          1. Getting ready
          2. How to do it...
          3. How it works...
        5. Working with file permissions, ownership, and the sticky bit
          1. How to do it...
          2. There's more...
            1. Changing ownership
            2. Setting sticky bit
            3. Applying permissions recursively to files
            4. Applying ownership recursively
            5. Running an executable as a different user (setuid)
        6. Making files immutable
          1. Getting ready
          2. How to do it...
        7. Generating blank files in bulk
          1. Getting ready
          2. How to do it...
        8. Finding symbolic links and their targets
          1. How to do it...
          2. How it works...
        9. Enumerating file type statistics
          1. Getting ready
          2. How to do it...
          3. How it works...
        10. Using loopback files
          1. How to do it...
          2. How it works...
          3. There's more...
            1. Creating partitions inside loopback images
            2. Quicker way to mount loopback disk images with partitions
            3. Mounting ISO files as loopback
            4. Flush changing immediately with sync
        11. Creating ISO files and hybrid ISO
          1. Getting ready
          2. How to do it...
          3. There's more...
            1. Hybrid ISO that boots off a flash drive or hard disk
            2. Burning an ISO from the command line
            3. Playing with the CD-ROM tray
        12. Finding the difference between files, patching
          1. How to do it...
          2. There's more...
            1. Generating difference against directories
        13. Using head and tail for printing the last or first 10 lines
          1. How to do it...
        14. Listing only directories – alternative methods
          1. Getting ready
          2. How to do it...
          3. How it works...
        15. Fast command-line navigation using pushd and popd
          1. Getting ready
          2. How to do it...
          3. There's more...
            1. Most frequently used directory switching
        16. Counting the number of lines, words, and characters in a file
          1. How to do it...
        17. Printing the directory tree
          1. Getting ready
          2. How to do it...
          3. There's more...
            1. HTML output for tree
      11. 4. Texting and Driving
        1. Introduction
        2. Using regular expressions
          1. How to do it...
          2. How it works...
          3. There's more...
            1. Treatment of special characters
            2. Visualizing regular expressions
        3. Searching and mining a text inside a file with grep
          1. How to do it...
          2. There's more...
            1. Recursively search many files
            2. Ignoring case of pattern
            3. grep by matching multiple patterns
            4. Including and excluding files in a grep search
            5. Using grep with xargs with zero-byte suffix
            6. Silent output for grep
            7. Printing lines before and after text matches
        4. Cutting a file column-wise with cut
          1. How to do it...
          2. There's more
            1. Specifying the range of characters or bytes as fields
        5. Using sed to perform text replacement
          1. How to do it…
          2. There's more...
            1. Removing blank lines
            2. Performing replacement directly in the file
            3. Matched string notation (&)
            4. Substring match notation (\1)
            5. Combination of multiple expressions
            6. Quoting
        6. Using awk for advanced text processing
          1. Getting ready...
          2. How to do it…
          3. How it works…
          4. There's more…
            1. Special variables
            2. Passing an external variable to awk
            3. Reading a line explicitly using getline
            4. Filtering lines processed by awk with filter patterns
            5. Setting delimiter for fields
            6. Reading the command output from awk
            7. Using loop inside awk
            8. String manipulation functions in awk
        7. Finding the frequency of words used in a given file
          1. Getting ready
          2. How to do it...
          3. How it works...
          4. See also
        8. Compressing or decompressing JavaScript
          1. Getting ready
          2. How to do it...
          3. How it works...
          4. See also
        9. Merging multiple files as columns
          1. How to do it...
          2. See also
        10. Printing the nth word or column in a file or line
          1. How to do it...
          2. See also
        11. Printing text between line numbers or patterns
          1. Getting ready
          2. How to do it...
          3. See also
        12. Printing lines in the reverse order
          1. Getting ready
          2. How to do it...
          3. How it works...
        13. Parsing e-mail addresses and URLs from text
          1. How to do it...
          2. How it works...
          3. See also
        14. Removing a sentence in a file containing a word
          1. Getting ready
          2. How to do it...
          3. How it works...
          4. See also
        15. Replacing a pattern with text in all the files in a directory
          1. How to do it...
          2. How it works...
          3. There's more...
        16. Text slicing and parameter operations
          1. How to do it...
          2. See also
      12. 5. Tangled Web? Not At All!
        1. Introduction
        2. Downloading from a web page
          1. Getting ready
          2. How to do it...
          3. How it works...
          4. There's more...
            1. Restricting the download speed
            2. Resume downloading and continue
            3. Copying a complete website (mirroring)
            4. Accessing pages with HTTP or FTP authentication
        3. Downloading a web page as plain text
          1. How to do it...
        4. A primer on cURL
          1. Getting ready
          2. How to do it…
          3. How it works...
          4. There's more...
            1. Continuing and resuming downloads
            2. Setting the referer string with cURL
            3. Cookies with cURL
            4. Setting a user agent string with cURL
            5. Specifying a bandwidth limit on cURL
            6. Specifying the maximum download size
            7. Authenticating with cURL
            8. Printing response headers excluding the data
          5. See also
        5. Accessing Gmail e-mails from the command line
          1. How to do it...
          2. How it works...
          3. See also
        6. Parsing data from a website
          1. How to do it...
          2. How it works...
          3. See also
        7. Image crawler and downloader
          1. How to do it...
          2. How it works...
          3. See also
        8. Web photo album generator
          1. Getting ready
          2. How to do it...
          3. How it works...
          4. See also
        9. Twitter command-line client
          1. Getting ready
          2. How to do it...
          3. How it works...
          4. See also
        10. Creating a "define" utility by using the Web backend
          1. Getting ready
          2. How to do it...
          3. How it works...
          4. See also
        11. Finding broken links in a website
          1. Getting ready
          2. How to do it...
          3. How it works...
          4. See also
        12. Tracking changes to a website
          1. Getting ready
          2. How to do it...
          3. How it works...
          4. See also
        13. Posting to a web page and reading the response
          1. Getting ready
          2. How to do it...
          3. How it works...
          4. See also
      13. 6. The Backup Plan
        1. Introduction
        2. Archiving with tar
          1. Getting ready
          2. How to do it...
          3. How it works...
          4. There's more...
            1. Appending files to an archive
            2. Extracting files and folders from an archive
            3. stdin and stdout with tar
            4. Concatenating two archives
            5. Updating files in an archive with a timestamp check
            6. Comparing files in the archive and file system
            7. Deleting files from the archive
            8. Compression with the tar archive
            9. Excluding a set of files from archiving
            10. Excluding version control directories
            11. Printing total bytes
          5. See also
        3. Archiving with cpio
          1. How to do it...
          2. How it works...
        4. Compressing data with gzip
          1. How to do it...
          2. There's more...
            1. Gzip with tarball
            2. zcat - reading gzipped files without extracting
            3. Compression ratio
            4. Using bzip2
            5. Using lzma
          3. See also
        5. Archiving and compressing with zip
          1. How to do it...
          2. How it works...
        6. Faster archiving with pbzip2
          1. Getting ready
          2. How to do it...
          3. How it works...
          4. There's more...
            1. Manually specifying the number of CPUs
            2. Specifying the compression ratio
        7. Creating filesystems with compression
          1. Getting ready
          2. How to do it...
          3. There's more...
            1. Excluding files while creating a squashfs file
        8. Backup snapshots with rsync
          1. How to do it...
          2. How it works...
          3. There's more...
            1. Excluding files while archiving with rsync
            2. Deleting non-existent files while updating rsync backup
            3. Scheduling backups at intervals
        9. Version control-based backup with Git
          1. Getting ready
          2. How to do it...
        10. Creating entire disk images using fsarchiver
          1. Getting ready
          2. How to do it...
          3. How it works...
      14. 7. The Old-boy Network
        1. Introduction
        2. Setting up the network
          1. Getting ready
          2. How to do it...
          3. There's more...
            1. Printing the list of network interfaces
            2. Displaying IP addresses
            3. Spoofing the hardware address (MAC address)
            4. Name server and DNS (Domain Name Service)
              1. DNS lookup
            5. Showing routing table information
          4. See also
        3. Let us ping!
          1. How to do it...
          2. There's more
            1. Round trip time
            2. Limiting the number of packets to be sent
            3. Return status of the ping command
            4. Traceroute
        4. Listing all the machines alive on a network
          1. Getting ready
          2. How to do it...
          3. How it works...
          4. There's more...
            1. Parallel pings
              1. Using fping
          5. See also
        5. Running commands on a remote host with SSH
          1. Getting ready
          2. How to do it...
          3. There's more...
            1. SSH with compression
            2. Redirecting data into stdin of remote host shell commands
            3. Running graphical commands on a remote machine
          4. See also
        6. Transferring files through the network
          1. Getting ready
          2. How to do it...
          3. There's more...
            1. Automated FTP transfer
            2. SFTP (Secure FTP)
            3. The rsync command
            4. SCP (secure copy program)
            5. Recursive copying with SCP
          4. See also
        7. Connecting to a wireless network
          1. Getting ready
          2. How to do it...
          3. How it works...
          4. See also
        8. Password-less auto-login with SSH
          1. Getting ready
          2. How to do it...
        9. Port forwarding using SSH
          1. How to do it...
          2. There's more...
            1. Non-interactive port forward
            2. Reverse port forwarding
        10. Mounting a remote drive at a local mount point
          1. Getting ready
          2. How to do it...
          3. See also
        11. Network traffic and port analysis
          1. Getting ready
          2. How to do it...
          3. How it works...
          4. There's more...
            1. Opened port and services using netstat
        12. Creating arbitrary sockets
          1. Getting ready
          2. How to do it...
          3. There's more...
            1. Quickly copying files over the network
        13. Sharing an Internet connection
          1. Getting ready
          2. How to do it...
        14. Basic firewall using iptables
          1. How to do it...
          2. How it works...
          3. There's more...
      15. 8. Put on the Monitor's Cap
        1. Introduction
        2. Monitoring disk usage
          1. Getting ready
          2. How to do it...
          3. There's more...
            1. Displaying disk usage in KB, MB, or Blocks
            2. Displaying the grand total sum of disk usage
            3. Printing files in specified units
            4. Excluding files from the disk usage calculation
            5. Finding the 10 largest size files from a given directory
            6. Disk free information
        3. Calculating the execution time for a command
          1. How to do it...
          2. How it works...
        4. Collecting information about logged in users, boot logs, and boot failures
          1. Getting ready
          2. How to do it...
        5. Listing the top 10 CPU consuming processes in an hour
          1. Getting ready
          2. How to do it...
          3. How it works...
          4. See also
        6. Monitoring command outputs with watch
          1. How to do it...
          2. There's more
            1. Highlighting the differences in the watch output
        7. Logging access to files and directories
          1. Getting ready
          2. How to do it...
          3. How it works...
        8. Logfile management with logrotate
          1. Getting ready
          2. How to do it...
          3. How it works...
        9. Logging with syslog
          1. Getting ready
          2. How to do it...
          3. See also
        10. Monitoring user logins to find intruders
          1. Getting ready
          2. How to do it…
          3. How it works…
        11. Remote disk usage health monitor
          1. Getting ready
          2. How to do it…
          3. How it works…
          4. See also
        12. Finding out active user hours on a system
          1. Getting ready
          2. How to do it…
          3. How it works…
        13. Measuring and optimizing power usage
          1. Getting ready
          2. How to do it...
        14. Monitoring disk activity
          1. Getting ready
          2. How to do it...
        15. Checking disks and filesystems for errors
          1. Getting ready
          2. How to do it...
          3. How it works...
      16. 9. Administration Calls
        1. Introduction
        2. Gathering information about processes
          1. Getting ready
          2. How to do it...
          3. How it works...
          4. There's more...
            1. top
            2. Sorting the ps output with respect to a parameter
            3. Finding the process ID when given command names
            4. Filters with ps for real user or ID, effective user or ID
            5. TTY filter for ps
            6. Information about process threads
            7. Specifying output width and columns to be displayed
            8. Showing environment variables for a process
            9. About which, whereis, file, whatis, and load average
          5. See also
        3. Killing processes and send or respond to signals
          1. Getting ready
          2. How to do it...
          3. There's more...
            1. The kill family of commands
            2. Capturing and responding to signals
        4. Sending messages to user terminals
          1. Getting ready
          2. How to do it...
          3. How it works...
        5. Gathering system information
          1. How to do it...
        6. Using /proc for gathering information
          1. How to do it...
        7. Scheduling with cron
          1. Getting ready
          2. How to do it…
          3. How it works...
          4. There's more…
            1. Specifying environment variables
            2. Running commands at system start up/boot
            3. Viewing the cron table
            4. Removing the cron table
        8. Writing and reading the MySQL database from Bash
          1. Getting ready
          2. How to do it…
          3. How it works…
        9. User administration script
          1. How to do it…
          2. How it works…
        10. Bulk image resizing and format conversion
          1. Getting ready
          2. How to do it..
          3. How it works…
          4. See also
        11. Taking screenshots from the terminal
          1. Getting ready
          2. How to do it...
        12. Managing multiple terminals from one
          1. Getting ready
          2. How to do it...
      17. Index