You are previewing Exploring Expect.

Exploring Expect

Cover of Exploring Expect by Don Libes Published by O'Reilly Media, Inc.
  1. Exploring Expect
    1. SPECIAL OFFER: Upgrade this ebook with O’Reilly
    2. A Note Regarding Supplemental Files
    3. Preface
      1. Expect—Why another tool?
      2. Tcl—A Little History
      3. Acknowledgments
      4. We’d Like to Hear From You
    4. How To Read This Book
      1. Notational Conventions
      2. Exercises
    5. 1. Intro—What Is Expect?
      1. Ouch, Those Programs Are Painful!
      2. A Very Brief Overview
      3. A First Script—dialback
      4. Total Automation
      5. Differing Behavior When Running Non-Interactively
      6. Partial Automation
      7. Dangerous, Unfriendly, Or Otherwise Unlikable User Interfaces
      8. Graphical Applications
      9. A Little More About Tcl
      10. Job Control
      11. Background Processes
      12. Using Expect With Other Programs
      13. Using Expect On UNIX
      14. Using Expect On Other Operating Systems
      15. Using Expect In Real Applications
      16. Using Expect In Commercial Applications—Legalese
      17. Obtaining Expect and the Examples
      18. Expect And Tcl Resources
      19. Exercises
    6. 2. Tcl—Introduction And Overview
      1. Everything Is A String
      2. Quoting Conventions
      3. Expressions
      4. Braces—Deferring Evaluation
      5. Control Structures
      6. More On Expressions
      7. Lists
      8. More Ways To Manipulate Strings
      9. Arrays
      10. Indirect References
      11. Handling Errors
      12. Evaluating Lists As Commands
      13. Passing By Reference
      14. Working With Files
      15. File I/O
      16. Executing UNIX Commands
      17. Environment Variables
      18. Handling Unknown Commands
      19. Libraries
      20. Is There More To Tcl?
      21. Exercises
    7. 3. Getting Started With Expect
      1. The send Command
      2. The expect Command
      3. Anchoring
      4. What Happens When Input Does Not Match
      5. Pattern-Action Pairs
      6. Example—Timed Reads In The Shell
      7. The spawn Command
      8. The interact Command
      9. Example—Anonymous ftp
      10. Exercises
    8. 4. Glob Patterns And Other Basics
      1. The * Wildcard
      2. More Glob Patterns
      3. Backslashes
      4. Handling Timeout
      5. Handling End Of File (eof)
      6. Hints On The spawn Command
      7. Back To Eof
      8. The close Command
      9. Programs That Ignore Eof
      10. The wait Command
      11. Exercises
    9. 5. Regular Expressions
      1. Regular Expressions—A Quick Start
      2. Identifying Regular Expressions And Glob Patterns
      3. Using Parentheses To Override Precedence
      4. Using Parentheses For Feedback
      5. More On The timed–read Script
      6. Pattern Matching Strategy
      7. Nested Parentheses
      8. Always Count Parentheses Even Inside Of Alternatives
      9. Example—The Return Value From A Remote Shell
      10. Matching Customized Prompts
      11. Example—A Smart Remote Login Script
      12. What Else Gets Stored In expect_out
      13. More On Anchoring
      14. Exercises
    10. 6. Patterns, Actions, And Limits
      1. Matching Anything But
      2. Really Complex Patterns
      3. Really Simple Patterns
      4. Matching One Line And Only One Line
      5. Tcl’s string match Command
      6. Tcl’s regexp Command
      7. Tcl’s regsub Command
      8. Ignoring Case
      9. All Those Other String Functions Are Handy, Too
      10. Actions That Affect Control Flow
      11. Example—rogue
      12. Character Graphics
      13. More Actions That Affect Control Flow
      14. Matching Multiple Times
      15. Recognizing Prompts (Yet Again)
      16. Speed Is On Your Side
      17. Controlling The Limits Of Pattern Matching Input
      18. The full_buffer Keyword
      19. Double Buffering
      20. Perpetual Buffering
      21. The Politics Of Patterns
      22. Expecting A Null Character
      23. Parity
      24. Length Limits
      25. Comments In expect Commands
      26. Restrictions On expect Arguments
      27. eval—Good, Bad, And Ugly
      28. Exercises
    11. 7. Debugging Patterns And Controlling Output
      1. Pattern Debugging
      2. Enabling Internal Diagnostics
      3. Logging Internal Diagnostics
      4. Disabling Normal Program Output
      5. The log_user Command
      6. Example—su2
      7. Recording All Expect Output
      8. Sending Messages To The Log
      9. About File Names
      10. Log And Diagnostic State
      11. Exercises
    12. 8. Handling A Process And A User
      1. The send_user Command
      2. The send_error Command
      3. The expect_user Command
      4. Dealing With Programs That Reprompt
      5. Dealing With Programs That Miss Input
      6. Sleeping
      7. Line Versus Character-Oriented And Other Terminal Modes
      8. Echoing
      9. Prompting For A Password On Behalf Of A Program
      10. Security And Insecurity
      11. Resetting The Terminal Upon Exit
      12. More On The stty Command
      13. The system Command
      14. Redirecting The Standard Input Or Output
      15. The expect_tty Command
      16. The send_tty Command
      17. Exercises
    13. 9. The Expect Program
      1. Expect—Just Another Program
      2. Invoking Scripts Without Saying “expect”
      3. Rewriting The #! Line
      4. The .exp Extension
      5. The—And Other Flags
      6. The —c Flag
      7. The -f Flag
      8. Writing The #! Line
      9. The −i Flag
      10. The -n And -N Flags
      11. The -d Flag
      12. The -D Flag
      13. The -b Flag
      14. The - Flag
      15. The interpreter Command
      16. Exercises
    14. 10. Handling Multiple Processes
      1. The spawn_id Variable
      2. Example—chess Versus chess
      3. Example—Automating The write Command
      4. How exp_continue Affects spawn_id
      5. The Value Of spawn_id Affects Many Commands
      6. Symbolic Spawn Ids
      7. Job Control
      8. Procedures Introduce New Scopes
      9. How Expect Writes Variables In Different Scopes
      10. Predefined Spawn Ids
      11. Exercises
    15. 11. Handling Multiple Processes Simultaneously
      1. Implicit Versus Explicit Spawn Ids
      2. Waiting From Multiple Processes Simultaneously
      3. Example—Answerback
      4. Which Pattern Goes With Which Spawn Id
      5. Which Spawn Id Matched
      6. Spawn Id Lists
      7. Example—Connecting Together Two Users To An Application
      8. Example—Timing All Commands
      9. Matching Any Spawn Id Already Listed
      10. The expect_before And expect_after Commands
      11. Indirect Spawn Ids
      12. Exercises
    16. 12. Send
      1. Implicit Versus Explicit Spawn Ids
      2. Sending To Multiple Processes
      3. Sending Without Echoing
      4. Sending To Programs In Cooked Mode
      5. Sending Slowly
      6. Sending Humanly
      7. Sending Nulls
      8. Sending Breaks
      9. Sending Strings That Look Like Flags
      10. Sending Character Graphics
      11. Comparing send To puts
      12. Exercises
    17. 13. Spawn
      1. The Search Path
      2. Philosophy--Processes Are Smart
      3. Treating Files As Spawned Processes
      4. Opening Ttys
      5. Bugs And Workarounds
      6. Process Pipelines And Ptys
      7. Automating xterm
      8. Checking For Errors From spawn
      9. spawn -noecho
      10. Example—unbuffer
      11. Obtaining Console Output
      12. Setting Pty Modes From spawn
      13. Hung Ptys
      14. Restrictions On Spawning Multiple Processes
      15. Getting The Process Id From A Spawn Id
      16. Using File I/O Commands On Spawned Processes
      17. Exercises
    18. 14. Signals
      1. Signals
      2. Signals In Spawned Processes
      3. Notes On Specific Signals
      4. When And Where Signals Are Evaluated
      5. Overriding The Original Return Value
      6. Using A Different Interpreter To Process Signals
      7. Exit Handling
      8. Exercises
    19. 15. Interact
      1. The interact Command
      2. Simple Patterns
      3. Exact Matching
      4. Matching Patterns From The Spawned Process
      5. Regular Expressions
      6. What Happens To Things That Do Not Match
      7. More Detail On Matching
      8. Echoing
      9. Avoiding Echoing
      10. Giving Feedback Without -echo
      11. Telling The User About New Features
      12. Sending Characters While Pattern Matching
      13. The continue And break Actions
      14. The return Action
      15. The Default Action
      16. Detecting End-Of-File
      17. Matching A Null Character
      18. Timing Out
      19. More On Terminal Modes (Or The -reset Flag)
      20. Example—Preventing Bad Commands
      21. Exercises
    20. 16. Interacting With Multiple Processes
      1. Connecting To A Process Other Than The Currently Spawned Process
      2. Connecting To A Process Instead Of The User
      3. Example—rz And sz Over rlogin
      4. Redirecting Input And Output
      5. Default Input And Output
      6. Controlling Multiple Processes—kibitz
      7. Combining Spawn Ids In A Single -input Or -output
      8. Which Spawn Id Matched
      9. Indirect Spawn Ids
      10. An Extended Example—xkibitz
      11. Exercises
    21. 17. Background Processing
      1. Putting Expect In The Background
      2. Running Expect Without A Controlling Terminal
      3. Disconnecting The Controlling Terminal
      4. The fork Command
      5. The disconnect Command
      6. Reconnecting
      7. Using kibitz From Other Expect Scripts
      8. Mailing From Expect
      9. A Manager For Disconnected Processes—dislocate
      10. Expect As A Daemon
      11. Example—Automating Gopher and Mosaic telnet Connections
      12. Exercises
    22. 18. Debugging Scripts
      1. Tracing
      2. Logging
      3. Command Tracing
      4. Variable Tracing
      5. Example—Logging By Tracing
      6. UNIX System Call Tracing
      7. Tk And tkinspect
      8. Traditional Debugging
      9. Debugger Command Overview And Philosophy
      10. Stepping Over Procedure Calls
      11. Stepping Into Procedure Calls
      12. Where Am I
      13. The Current Scope
      14. Moving Up And Down The Stack
      15. Returning From A Procedure
      16. Continuing Execution
      17. Defining Breakpoints
      18. Help
      19. Changing Program Behavior
      20. Changing Debugger Behavior
      21. Exercises
    23. 19. Expect + Tk = Expectk
      1. Tk—A Brief Technical Overview
      2. Expectk
      3. The send Command
      4. An Extended Example—tkpasswd
      5. The expect Command And The Tk Event Loop
      6. The expect_background Command
      7. Multiple Spawn Ids In expect_background
      8. Background Actions
      9. Example—A Dumb Terminal Emulator
      10. Example—A Smarter Terminal Emulator
      11. Using The Terminal Emulator For Testing And Automation
      12. Exercises
    24. 20. Extended Examples
      1. Encrypting A Directory
      2. File Transfer Over telnet
      3. You Have Unread News—tknewsbiff
      4. Exercises
    25. 21. Expect, C, And C++
      1. Overview
      2. Linking
      3. Include Files
      4. Ptys And Processes
      5. Allocating Your Own Pty
      6. Closing The Connection To The Spawned Process
      7. Expect Commands
      8. Regular Expression Patterns
      9. Exact Matching
      10. Matching A Null
      11. What Characters Matched
      12. When The Number Of Patterns Is Not Known In Advance
      13. Expecting From Streams
      14. Running In The Background
      15. Handling Multiple Inputs And More On Timeouts
      16. Output And Debugging Miscellany
      17. Pty Trapping
      18. Exercises
    26. 22. Expect As Just Another Tcl Extension
      1. Adding Expect To Another Tcl-based Program
      2. Differences Between Expect And The Expect Extension In Another Program
      3. Adding Extensions To Expect
      4. Adding Extensions To Expectk
      5. Creating Script-less Expect Programs
      6. Functions And Variables In The Expect Extension
      7. Exercises
    27. 23. Miscellaneous
      1. Random Numbers
      2. Example—Generating Random Passwords
      3. The Expect Library
      4. Expect Versions
      5. Timestamps
      6. The time Command
      7. Exercises
    28. A. Appendix—Commands and Variables
      1. Commands And Flags
      2. Variables
    29. Index
    30. About the Author
    31. SPECIAL OFFER: Upgrade this ebook with O’Reilly

Handling Timeout

Much of the time, expect commands have only one argument—a pattern with no action—similar to the very first one in this chapter:

expect "hi"

All this does is wait for hi before continuing. You could also write this as:

expect "hi" {}

to show the empty action, but expect does not require it. Only the last action in an expect command can be omitted:

expect  {
    "hi"            {send "You said hi\n"}
    "hello"            {send "Hello yourself\n"}

As a natural consequence of this, it is typical to write expect commands with the exception strings at the top and the likely string at the bottom. For example, you could add some error checking to the beginning of the anonymous ftp script from the previous chapter:

spawn ftp $argv
set timeout 10
expect {
    "connection refused" exit
    "unknown host" exit
send "anonymous\r"

If the script sees Name it will go on and send anonymous\r. But if it sees "unknown host" or "connection refused“, the script will exit. Scripts written this way flow gracefully from top to bottom.

If, after 10 seconds, none of these patterns have been seen, expect will timeout and the next command in the script will be executed. I used this behavior in constructing the timed_read script in the previous chapter. Here, however, I only want to go to the next command if Name is successfully matched.

You can distinguish the successful case from the timeout by associating an action with the timeout. This is done by using the special pattern timeout. It looks like this:

expect {
    timeout {puts "timed out"; exit}
    "connection refused" exit
    "unknown host" exit

If none of the patterns match after ten seconds, the script will print "timed out" and exit. The result is that the script is more robust. It will only go on if it has been prompted to. And it cannot hang forever. You control how long it waits.

Although the timeout pattern is invaluable, it is not a replacement for all error handling. It is tempting to remove the patterns "connection refused" and "unknown host“:

expect {
    timeout exit

Now suppose "unknown host" is seen. It does not match Name and nothing else arrives within the ten seconds. At the end of ten seconds, the command times out. While the script still works, it fails very slowly.

This is a common dilemma. By explicitly specifying all the possible errors, a script can handle them more quickly. But that takes work on your part while writing the script. And sometimes it is impossible to find out all the error messages that a program could produce.

In practice, it suffices to catch the common errors, and let timeout handle the obscure conditions. It is often possible to find a pattern with appropriate wildcards that match many errors. For example, once ftp is connected, it is always possible to distinguish errors. ftp prefaces all output with a three-digit number. If it begins with a 4 or 5, it is an error. Assuming ftp’s line is the only thing in expect’s input buffer, you can match errors using the range construct described on page 91:

expect {
    timeout {unexpected ...}
    "^\[45]" {error ...}

As I described in Chapter 3 (p. 73), the ^ serves to anchor the 4 or 5 to the beginning of the buffer. If there are previous lines in the buffer—as is more likely—you can use the pattern "\n\[45]“. The linefeed (\n) matches the end of the carriage-return linefeed combination that appears at the end of any line intended to be output on a terminal.

When the timeout pattern is matched, the data that has arrived is not moved to expect_out(buffer). (In Chapter 11 (p. 248), I will describe the rationale for this behavior.) If you need the data, you must match it with a pattern. You can use the * wildcard to do so:

expect *

As I noted earlier, this command is guaranteed to return immediately, and expect_out(buffer) will contain what had arrived when the previous timeout occurred.

By convention, the timeout pattern itself is not quoted. This serves as a reminder to the reader that expect is not waiting for the literal string "timeout“. Putting quotes around it does not change expect’s treatment of it. It is still be interpreted as a special pattern. Quotes only protect strings from being broken up, such as by spaces. For that reason, you can actually write a subset of expect patterns without any quotes. Look at the following intentionally obfuscated examples:

expect "hi" there
expect  hi  there
expect "hi  there"

In the first and second commands, hi is the pattern, while "hi there" is the pattern in the third command. For consistency, use quotes around all textual patterns, and leave them off the special pattern timeout. In Chapter 5 (p. 109), I will show how to wait for the literal string timeout.

Here is another example of the timeout pattern. You can use the ping command to test whether a host is up or not. Assume that host elvis is up and houdini is down. Not all versions of ping produce the same output, but here is how it looks when I run it:

% ping elvis
elvis is alive
% ping houdini
no answer from houdini

What ping actually does is to send a message to the host which the host should acknowledge. ping usually reports very quickly that the host is up, but it only says "no answer" after waiting quite a while—20 seconds is common.

If the host is on your local network, chances are that if the host does not respond within a second or two, it is not going to respond. If you are only looking for a host to farm out some background task, this heuristic works well. Realistically, it is exactly the same heuristic that ping uses—just a little less tolerant. Here is an Expect script that forces ping to respond after 2 seconds.

spawn ping $host
set timeout 2
expect "alive" {exit 0} timeout {exit 1}

If the expect sees alive within two seconds, it returns 0 to the caller; otherwise it returns 1. When called from a /bin/sh script, you find the result by inspecting the status. This is stored in the shell variable $? (or $status in csh).

$ echo $?

Strictly speaking, the status must be an integer. This is good in many cases—integers are easier than strings to check anyway. However, it is possible to get the effect of returning a string simply by printing it out. Consider the following commands which print out the same messages as ping:

spawn ping $host
set timeout 2
expect "alive" {exit 0} timeout {
    puts "no answer from $host"
    exit 1

The timeout action prints the string "no answer from ..." because the script will abort ping before it gets a chance to print its own error message. The alive action does not have to do anything extra because ping already prints the string. Both strings are sent to the standard output. In Chapter 7 (p. 171), you will see how to prevent printing strings from the underlying process, and even substitute your own if desired.

Some versions of ping have a user-settable timeout. But the technique I have shown is still useful. Many other programs are completely inflexible, having long fixed timeouts or none at all.

rsh is a program for executing shell commands remotely.[19]rsh is an example of a program that is very inflexible when it comes to timeouts. rsh waits for 75 seconds before deciding that a machine is down. And there is no way to change this time period. If rsh finds the machine is up, it will then execute the command but without any ability to timeout at all. It would be nice if rsh and other commands all had the ability to timeout, but it is not necessary since you can achieve the same result with an Expect script.

Rather than writing separate scripts to control rsh and every other problem utility, you can write a parameterized script to timeout any program. The two parameters of interest are the program name and the timeout period. These can be passed as the first and second arguments. Assuming the script is called maxtime, it could be used from the shell to run a program prog for at most 20 seconds with the following:

% maxtime 20 prog

Here is the script:

#!/usr/local/bin/expect --
set timeout [lindex $argv 0]
spawn [lindex $argv 1]

The script starts by setting the timeout from the first argument. Then the program named by the second argument is spawned. Finally, expect waits for output. Since there are no patterns specified, expect never matches using any of the output. And because there are no patterns to match, after enough time, expect times out. Because there is no timeout action, expect simply returns, and the script ends. Alternatively, if the program ends before the timeout, expect notices this and returns immediately. Again, the script ends.

[19] Some systems call it remsh.

The best content for your career. Discover unlimited learning on demand for around $1/day.