You are previewing Linux Shell Scripting Cookbook.
O'Reilly logo
Linux Shell Scripting Cookbook

Book Description

Get straight into writing scripts with this recipe-packed cookbook. Ideal as an end-to-end beginner’s guide or as a "dip-in" reference for experts, it helps you gain control over Linux through the powerful shell scripting language.

  • Master the art of crafting one-liner command sequence to perform tasks such as text processing, digging data from files, and lot more

  • Practical problem solving techniques adherent to the latest Linux platform

  • Packed with easy-to-follow examples to exercise all the features of the Linux shell scripting language

  • Part of Packt's Cookbook series: Each recipe is a carefully organized sequence of instructions to complete the task as efficiently as possible

  • In Detail

    GNU/Linux is a remarkable operating system that comes with a complete development environment that is stable, reliable, and extremely powerful. The shell being the native interface to communicate with the operating system is capable of controlling the entire operating system. There are numerous commands on Linux shell which are documented but hard to understand. The man pages are helpful but they are very lengthy and it does not give any clues on key areas where commands can be used. Proper usage of shell commands can easily solve many complex tasks with a few lines of code, but most linux users don't have the right know-how to use the Linux shell to its full potential.

    Linux Shell Scripting Cookbook is a collection of essential command-line recipes along with detailed descriptions tuned with practical applications.It covers most of the commands on Linux with a variety of usecases accompanied by plenty of examples. This book helps you to perform complex data manipulations involving tasks such as text processing, file management,backups and more with the combination of few commands.

    Linux Shell Scripting Cookbook shows you how to capitalize on all the aspects of Linux using the shell scripting language. This book teaches you how to use commands to perform simple tasks all the way to scripting complex tasks such as managing large amounts of data on a network.

    It guides you on implementing some of the most common commands in Linux with recipes that handle any of the operations or properties related with files like searching and mining inside a file with grep.It also shows you how utilities such as sed, awk, grep, cut can be combined to solve text processing related problems.The focus is on saving time by automating many activities that we perform interactively through as browser with a few lines of script.

    This book will take you from a clear problem description to a fully functional program.The recipes contained within the chapter will introduce the reader to specific problems and provide hands-on solutions.

    This practical book will help you gain control over all the aspects of Linux using the powerful shell scripting language

    Table of Contents

    1. Linux Shell Scripting Cookbook
      1. Linux Shell Scripting Cookbook
      2. Credits
      3. About the Author
      4. About the Reviewers
      5. www.PacktPub.com
        1. Support files, eBooks, discount offers, and more
          1. Why Subscribe?
          2. Free Access for Packt account holders
      6. Preface
        1. What this book covers
        2. What you need for this book
        3. Who this book is for
        4. Conventions
        5. Reader feedback
        6. Customer support
          1. Errata
          2. Piracy
          3. Questions
      7. 1. Shell Something Out
        1. Introduction
        2. Printing in the terminal
          1. How to do it...
          2. There's more...
            1. Escaping newline in echo
            2. Printing colored output
        3. Playing with variables and environment variables
          1. Getting ready
          2. How to do it...
          3. There's more...
            1. Finding length of string
            2. Identifying the current shell
            3. Check for super user
            4. Modifying the Bash prompt string (username@hostname:~$)
        4. Doing math calculations with the shell
          1. Getting ready
          2. How to do it...
        5. Playing with file descriptors and redirection
          1. Getting ready
          2. How to do it...
          3. There's more...
            1. Redirection from file to command
            2. Redirecting from a text block enclosed within a script
            3. Custom file descriptors
        6. Arrays and associative arrays
          1. Getting ready
          2. How to do it...
          3. There's more...
            1. Defining associative arrays
            2. Listing of array indexes
        7. Visiting aliases
          1. Getting ready
          2. How to do it...
          3. There's more...
            1. Escaping aliases
        8. Grabbing information about terminal
          1. Getting ready
          2. How to do it...
        9. Getting, setting dates, and delays
          1. Getting ready
          2. How to do it...
          3. There's more...
            1. Producing delays in a script
        10. Debugging the script
          1. Getting ready
          2. How to do it...
          3. There's more...
            1. Shebang hack
        11. Functions and arguments
          1. How to do it...
          2. There's more...
            1. Recursive function
            2. Exporting functions
            3. Reading command return value (status)
            4. Passing arguments to commands
        12. Reading the output of a sequence of commands
          1. Getting ready
          2. How to do it...
          3. There's more...
            1. Spawning a separate process with subshell
            2. Subshell quoting to preserve spacing and newline character
        13. Reading "n" characters without pressing Return
          1. Getting ready
          2. How to do it...
        14. Field separators and iterators
          1. Getting ready
          2. How to do it...
        15. Comparisons and tests
          1. Getting ready
          2. How to do it...
      8. 2. Have a Good Command
        1. Introduction
        2. Concatenating with cat
          1. Getting ready
          2. How to do it...
          3. How it works…
          4. There's more...
            1. Squeezing blank lines
            2. Displaying tabs as ^I
            3. Line numbers
        3. Recording and playback of terminal sessions
          1. Getting ready
          2. How to do it...
          3. How it works...
        4. Finding files and file listing
          1. Getting ready
          2. How to do it...
          3. There's more...
            1. Search based on file name or regular expression match
            2. Negating arguments
            3. Search based on the directory depth
            4. Search based on file type
            5. Search on up file times
            6. Search based on file size
            7. Deleting based on the file matches
            8. Match based on the file permissions and ownership
            9. Executing commands or actions with find
            10. Skip specified directories from the find
        5. Playing with xargs
          1. Getting ready
          2. How to do it...
          3. How it works…
          4. There's more...
            1. Passing formatted arguments to a command by reading stdin
            2. Using xargs with find
            3. Counting number of lines of C code in a source code directory over many C files.
            4. While and subshell trick with stdin
        6. Translating with tr
          1. Getting ready
          2. How to do it...
          3. How it works…
          4. There's more...
            1. Deleting characters using tr
            2. Complementing character set
            3. Squeezing characters with tr
            4. Character classes
        7. Checksum and verification
          1. Getting ready
          2. How to do it...
          3. How it works...
          4. There's more...
            1. Checksum for directories
        8. Sorting, unique and duplicates
          1. Getting ready
          2. How to do it...
          3. How it works…
          4. There's more...
            1. Sort according to the keys or columns
            2. uniq
            3. String pattern generation with uniq
        9. Temporary file naming and random numbers
          1. How to do it...
          2. How it works…
        10. Splitting files and data
          1. How to do it...
          2. There's more…
            1. Specifying filename prefix for the split files
        11. Slicing filenames based on extension
          1. How to do it…
          2. How it works..
        12. Renaming and moving files in bulk
          1. Getting ready
          2. How to do it...
          3. How it works…
        13. Spell checking and dictionary manipulation
          1. How to do it...
          2. How it works...
        14. Automating interactive input
          1. Getting ready
          2. How to do it...
          3. How it works…
          4. There's more...
            1. Automating with expect
      9. 3. File In, File Out
        1. Introduction
        2. Generating files of any size
          1. How to do it...
        3. Intersection and set difference (A-B) on text files
          1. Getting ready
          2. How to do it...
        4. Finding and deleting duplicate files
          1. Getting ready
          2. How to do it...
          3. How it works...
          4. See also
        5. Making directories for a long path
          1. Getting ready
          2. How to do it...
        6. File permissions, ownership, and sticky bit
          1. Getting ready
          2. How to do it...
          3. There's more...
            1. Changing ownership
            2. Setting the sticky bit
            3. Applying permissions recursively to files
            4. Applying ownership recursively
            5. Running an executable as a different user (setuid)
        7. Making files immutable
          1. Getting ready
          2. How to do it...
        8. Generating blank files in bulk
          1. Getting ready
          2. How to do it...
        9. Finding a symbolic link and its target
          1. Getting ready
          2. How to do it...
        10. Enumerating file type statistics
          1. Getting ready
          2. How to do it...
          3. How it works...
        11. Loopback files and mounting
          1. Getting ready
          2. How to do it...
          3. There's more...
            1. Mounting ISO files as loopback
            2. Flush changes immediately with sync
        12. Creating ISO files, Hybrid ISO
          1. Getting ready
          2. How to do it...
          3. There's more...
            1. Hybrid ISO that boots off flash drive or hard disk
            2. Burning an ISO from command line
            3. Playing with CD Rom tray
        13. Finding difference between files, patching
          1. How to do it...
          2. There's more...
            1. Generating diff against directories
        14. head and tail – printing the last or first 10 lines
          1. How to do it...
        15. Listing only directories – alternative methods
          1. Getting ready
          2. How to do it...
        16. Fast command-line navigation using pushd and popd
          1. Getting ready
          2. How to do it...
          3. There's more...
            1. Most frequently used directory switching
        17. Counting number of lines, words, and characters in a file
          1. Getting ready
          2. How to do it...
          3. There's more...
            1. Print length of longest length line
        18. Printing directory tree
          1. Getting ready
          2. How to do it...
          3. There's more...
            1. HTML output for tree
      10. 4. Texting and Driving
        1. Introduction
        2. Basic regular expression primer
          1. Getting ready
          2. How to do it...
          3. How it works...
          4. There's more...
            1. Treatment of special characters
        3. Searching and mining "text" inside a file with grep
          1. Getting ready
          2. How to do it...
          3. There's more...
            1. Recursively search many files
            2. Ignoring case of pattern
            3. grep by matching multiple patterns
            4. Include and exclude files (wild card pattern) in grep search
            5. Using grep with xargs with zero-byte suffix
            6. Silent output for grep
            7. Print lines before and after text matches
        4. Column-wise cutting of a file with cut
          1. Getting ready
          2. How to do it...
          3. There's more...
            1. Specifying range of characters or bytes as fields
        5. Frequency of words used in a given file
          1. Getting ready
          2. How to do it...
          3. How it works...
          4. See also
        6. Basic sed primer
          1. How to do it…
          2. There's more...
            1. Removing blank lines
            2. Matched string notation (&)
            3. Substring match notation (\1)
            4. Combination of multiple expressions
            5. Quoting
        7. Basic awk primer
          1. How to do it…
          2. How it works…
          3. There's more…
            1. Special variables
            2. Passing a variable value from outside to awk
            3. Reading a line explicitly using getline
            4. Filtering lines processed by awk with filter patterns
            5. Setting delimiter for fields
            6. Reading command output from awk
            7. Using loop inside awk
        8. Replacing strings from a text or file
          1. Getting ready
          2. How to do it...
          3. There's more...
            1. Making replacement saved in the file
          4. See also
        9. Compressing or decompressing JavaScript
          1. Getting ready
          2. How to do it...
          3. How it works...
          4. See also
        10. Iterating through lines, words, and characters in a file
          1. Getting ready
          2. How to do it...
          3. How it works...
          4. See also
        11. Merging multiple files as columns
          1. How to do it...
          2. See also
        12. Printing the nth word or column in a file or line
          1. Getting ready
          2. How to do it...
          3. See also
        13. Printing text between line numbers or patterns
          1. Getting ready
          2. How to do it...
          3. See also
        14. Checking palindrome strings with a script
          1. Getting ready
          2. How to do it...
          3. How it works...
          4. There's more...
            1. Simplest and direct method
          5. See also
        15. Printing lines in the reverse order
          1. Getting ready
          2. How to do it...
          3. How it works...
          4. See also
        16. Parsing e-mail addresses and URLs from text
          1. Getting ready
          2. How to do it...
          3. How it works...
          4. See also
        17. Printing n lines before or after a pattern in a file
          1. Getting ready
          2. How to do it...
          3. See also
        18. Removing a sentence in a file containing a word
          1. Getting ready
          2. How to do it...
          3. How it works...
          4. See also
        19. Implementing head, tail, and tac with awk
          1. Getting ready
          2. How to do it...
          3. How it works...
          4. See also
        20. Text slicing and parameter operations
          1. How to do it...
          2. See also
      11. 5. Tangled Web? Not At All!
        1. Introduction
        2. Downloading from a web page
          1. Getting ready
          2. How to do it...
          3. There's more...
            1. Restricted with speed downloads
            2. Resume downloading and continue
            3. Using cURL for download
            4. Copying a complete website (mirroring)
            5. Accessing pages with HTTP or FTP authentication
        3. Downloading a web page as formatted plain text
          1. How to do it...
        4. A primer on cURL
          1. Getting ready
          2. How to do it…
          3. There's more...
            1. Continue/Resume downloading
            2. Set referer string with cURL
            3. Cookies with cURL
            4. Setting a user agent string with cURL
            5. Specifying bandwidth limit on cURL
            6. Specifying the maximum download size
            7. Authenticating with cURL
            8. Printing response headers excluding data
          4. See also
        5. Accessing Gmail from the command line
          1. How to do it...
          2. How it works...
          3. See also
        6. Parsing data from a website
          1. How to do it...
          2. How it works...
          3. See also
        7. Image crawler and downloader
          1. How to do it...
          2. How it works...
          3. See also
        8. Web photo album generator
          1. Getting ready
          2. How to do it...
          3. How it works...
          4. See also
        9. Twitter command-line client
          1. Getting ready
          2. How to do it...
          3. How it works...
          4. See also
        10. define utility with Web backend
          1. Getting ready
          2. How to do it...
          3. How it works...
          4. See also
        11. Finding broken links in a website
          1. Getting ready
          2. How to do it...
          3. How it works...
          4. See also
        12. Tracking changes to a website
          1. Getting ready
          2. How to do it...
          3. How it works...
          4. See also
        13. Posting to a web page and reading response
          1. Getting ready
          2. How to do it...
          3. There's more
            1. POST in curl
            2. POST data using wget
          4. See also
      12. 6. The Backup Plan
        1. Introduction
        2. Archiving with tar
          1. Getting ready
          2. How to do it...
          3. There's more...
            1. Appending files to an archive
            2. Extracting files and folders from an archive
            3. stdin and stdout with tar
            4. Concatenating two archives
            5. Updating files in an archive with timestamp check
            6. Comparing files in archive and file system
            7. Deleting files from archive
            8. Compression with tar archive
            9. Excluding a set of files from archiving
            10. Excluding version control directories
            11. Printing total bytes
          4. See also
        3. Archiving with cpio
          1. How to do it...
        4. Compressing with gunzip (gzip)
          1. How to do it...
          2. There's more...
            1. Gzip with tarball
            2. zcat – reading gzipped files without extracting
            3. Compression ratio
          3. See also
        5. Compressing with bunzip (bzip)
          1. How to do it...
          2. There's more...
            1. Keeping input files without removing them
            2. Compression ratio
          3. See also
        6. Compressing with lzma
          1. How to do it...
          2. There's more...
            1. Extracting an lzma tarball
            2. Keeping input files without removing them
            3. Compression ratio
          3. See also
        7. Archiving and compressing with zip
          1. How to do it...
        8. squashfs – the heavy compression filesystem
          1. Getting ready
          2. How to do it...
          3. There's more...
            1. Excluding files while creating a squashfs file
        9. Cryptographic tools and hashes
          1. How to do it...
        10. Backup snapshots with rsync
          1. How to do it...
          2. There's more...
            1. Excluding files while archiving with rsync
            2. Deleting non-existent files while updating rsync backup
            3. Scheduling backups at intervals
        11. Version control based backup with Git
          1. Getting ready
          2. How to do it...
        12. Cloning hard drive and disks with dd
          1. Getting ready
          2. How to do it...
          3. There's more...
            1. Mounting image files
          4. See also
      13. 7. The Old-boy Network
        1. Introduction
        2. Basic networking primer
          1. Getting ready
          2. How to do it...
          3. There's more...
            1. Printing the list of network interfaces
            2. Assigning and displaying IP addresses
            3. Spoofing Hardware Address (MAC Address)
            4. Name server and DNS (Domain Name Service)
            5. DNS lookup
            6. Setting default gateway, showing routing table information
            7. Traceroute
          4. See also
        3. Let's ping!
          1. How to do it...
          2. There's more
            1. Round trip time
            2. Limiting number of packets to be sent
            3. Return status of ping command
        4. Listing all the machines alive on a network
          1. Getting ready
          2. How to do it...
          3. How it works...
          4. There's more...
            1. DNS lookup with fping
          5. See also
        5. Transferring files
          1. Getting ready
          2. How to do it...
          3. There's more...
            1. Automated FTP transfer
            2. SFTP (Secure FTP)
            3. RSYNC
            4. SCP (Secure Copy)
            5. Recursive copying with SCP
          4. See also
        6. Setting up an Ethernet and wireless LAN with script
          1. Getting ready
          2. How to do it...
          3. How it works...
          4. See also
        7. Password-less auto-login with SSH
          1. How to do it...
        8. Running commands on remote host with SSH
          1. Getting ready
          2. How to do it...
          3. There's more...
            1. SSH with compression
            2. Redirecting data into stdin of remote host shell commands
          4. See also
        9. Mounting a remote drive at a local mount point
          1. Getting ready
          2. How to do it...
          3. See also
        10. Multi-casting window messages on a network
          1. Getting ready
          2. How to do it...
          3. How it works...
          4. See also
        11. Network traffic and port analysis
          1. Getting ready
          2. How to do it...
          3. There's more...
            1. Opened port and services using netstat
      14. 8. Put on the Monitor's Cap
        1. Introduction
        2. Disk usage hacks
          1. Getting ready
          2. How to do it...
          3. There's more...
            1. Displaying disk usage in KB, MB, or Blocks
            2. Displaying the grand total sum of disk usage
            3. Printing files in specified units
            4. Excluding files from disk usage calculation
            5. Finding the 10 largest size files from a given directory
            6. Disk free information
        3. Calculating execution time for a command
          1. How to do it...
        4. Information about logged users, boot logs, and failure boot
          1. Getting ready
          2. How to do it...
        5. Printing the 10 most frequently-used commands
          1. Getting ready
          2. How to do it...
          3. How it works...
        6. Listing the top 10 CPU consuming process in a hour
          1. Getting ready
          2. How to do it...
          3. How it works...
          4. See also
        7. Monitoring command outputs with watch
          1. How to do it...
          2. There's more
            1. Highlighting the differences in watch output
        8. Logging access to files and directories
          1. Getting ready
          2. How to do it...
          3. How it works...
        9. Logfile management with logrotate
          1. Getting ready
          2. How to do it...
        10. Logging with syslog
          1. Getting ready
          2. How to do it...
          3. See also
        11. Monitoring user logins to find intruders
          1. Getting started
          2. How to do it…
          3. How it works…
        12. Remote disk usage health monitor
          1. Getting ready
          2. How to do it…
          3. How it works…
          4. See also
        13. Finding out active user hours on a system
          1. Getting ready
          2. How to do it…
          3. How it works…
      15. 9. Administration Calls
        1. Introduction
        2. Gathering information about processes
          1. Getting ready
          2. How to do it...
          3. There's more...
            1. top
            2. Sorting ps output with respect to a parameter
            3. Finding process ID when given command names
            4. Filters with ps for real user or ID, effective user or ID
            5. TTY filter for ps
            6. Information about process threads
            7. Specifying output width and columns to be displayed
            8. Showing environment variables for a process
          4. See also
        3. Killing processes and send or respond to signals
          1. Getting ready
          2. How to do it...
          3. There's more...
            1. kill family of commands
            2. Capturing and responding to signals
        4. which, whereis, file, whatis, and loadavg explained
          1. How to do it...
        5. Sending messages to user terminals
          1. Getting ready
          2. How to do it...
          3. How it works...
        6. Gathering system information
          1. How to do it...
        7. Using /proc – gathering information
          1. How to do it...
        8. Scheduling with cron
          1. Getting ready
          2. How to do it…
          3. There's more…
            1. Specifying environment variables
            2. Viewing the cron table
            3. Removing the cron table
        9. Writing and reading MySQL database from Bash
          1. Getting ready
          2. How to do it…
          3. How it works…
        10. User administration script
          1. How to do it…
          2. How it works…
        11. Bulk image resizing and format conversion
          1. Getting ready
          2. How to do it..
          3. How it works…
          4. See also