You are previewing Exploring Expect.

Exploring Expect

Cover of Exploring Expect by Don Libes Published by O'Reilly Media, Inc.
  1. Exploring Expect
    1. SPECIAL OFFER: Upgrade this ebook with O’Reilly
    2. A Note Regarding Supplemental Files
    3. Preface
      1. Expect—Why another tool?
      2. Tcl—A Little History
      3. Acknowledgments
      4. We’d Like to Hear From You
    4. How To Read This Book
      1. Notational Conventions
      2. Exercises
    5. 1. Intro—What Is Expect?
      1. Ouch, Those Programs Are Painful!
      2. A Very Brief Overview
      3. A First Script—dialback
      4. Total Automation
      5. Differing Behavior When Running Non-Interactively
      6. Partial Automation
      7. Dangerous, Unfriendly, Or Otherwise Unlikable User Interfaces
      8. Graphical Applications
      9. A Little More About Tcl
      10. Job Control
      11. Background Processes
      12. Using Expect With Other Programs
      13. Using Expect On UNIX
      14. Using Expect On Other Operating Systems
      15. Using Expect In Real Applications
      16. Using Expect In Commercial Applications—Legalese
      17. Obtaining Expect and the Examples
      18. Expect And Tcl Resources
      19. Exercises
    6. 2. Tcl—Introduction And Overview
      1. Everything Is A String
      2. Quoting Conventions
      3. Expressions
      4. Braces—Deferring Evaluation
      5. Control Structures
      6. More On Expressions
      7. Lists
      8. More Ways To Manipulate Strings
      9. Arrays
      10. Indirect References
      11. Handling Errors
      12. Evaluating Lists As Commands
      13. Passing By Reference
      14. Working With Files
      15. File I/O
      16. Executing UNIX Commands
      17. Environment Variables
      18. Handling Unknown Commands
      19. Libraries
      20. Is There More To Tcl?
      21. Exercises
    7. 3. Getting Started With Expect
      1. The send Command
      2. The expect Command
      3. Anchoring
      4. What Happens When Input Does Not Match
      5. Pattern-Action Pairs
      6. Example—Timed Reads In The Shell
      7. The spawn Command
      8. The interact Command
      9. Example—Anonymous ftp
      10. Exercises
    8. 4. Glob Patterns And Other Basics
      1. The * Wildcard
      2. More Glob Patterns
      3. Backslashes
      4. Handling Timeout
      5. Handling End Of File (eof)
      6. Hints On The spawn Command
      7. Back To Eof
      8. The close Command
      9. Programs That Ignore Eof
      10. The wait Command
      11. Exercises
    9. 5. Regular Expressions
      1. Regular Expressions—A Quick Start
      2. Identifying Regular Expressions And Glob Patterns
      3. Using Parentheses To Override Precedence
      4. Using Parentheses For Feedback
      5. More On The timed–read Script
      6. Pattern Matching Strategy
      7. Nested Parentheses
      8. Always Count Parentheses Even Inside Of Alternatives
      9. Example—The Return Value From A Remote Shell
      10. Matching Customized Prompts
      11. Example—A Smart Remote Login Script
      12. What Else Gets Stored In expect_out
      13. More On Anchoring
      14. Exercises
    10. 6. Patterns, Actions, And Limits
      1. Matching Anything But
      2. Really Complex Patterns
      3. Really Simple Patterns
      4. Matching One Line And Only One Line
      5. Tcl’s string match Command
      6. Tcl’s regexp Command
      7. Tcl’s regsub Command
      8. Ignoring Case
      9. All Those Other String Functions Are Handy, Too
      10. Actions That Affect Control Flow
      11. Example—rogue
      12. Character Graphics
      13. More Actions That Affect Control Flow
      14. Matching Multiple Times
      15. Recognizing Prompts (Yet Again)
      16. Speed Is On Your Side
      17. Controlling The Limits Of Pattern Matching Input
      18. The full_buffer Keyword
      19. Double Buffering
      20. Perpetual Buffering
      21. The Politics Of Patterns
      22. Expecting A Null Character
      23. Parity
      24. Length Limits
      25. Comments In expect Commands
      26. Restrictions On expect Arguments
      27. eval—Good, Bad, And Ugly
      28. Exercises
    11. 7. Debugging Patterns And Controlling Output
      1. Pattern Debugging
      2. Enabling Internal Diagnostics
      3. Logging Internal Diagnostics
      4. Disabling Normal Program Output
      5. The log_user Command
      6. Example—su2
      7. Recording All Expect Output
      8. Sending Messages To The Log
      9. About File Names
      10. Log And Diagnostic State
      11. Exercises
    12. 8. Handling A Process And A User
      1. The send_user Command
      2. The send_error Command
      3. The expect_user Command
      4. Dealing With Programs That Reprompt
      5. Dealing With Programs That Miss Input
      6. Sleeping
      7. Line Versus Character-Oriented And Other Terminal Modes
      8. Echoing
      9. Prompting For A Password On Behalf Of A Program
      10. Security And Insecurity
      11. Resetting The Terminal Upon Exit
      12. More On The stty Command
      13. The system Command
      14. Redirecting The Standard Input Or Output
      15. The expect_tty Command
      16. The send_tty Command
      17. Exercises
    13. 9. The Expect Program
      1. Expect—Just Another Program
      2. Invoking Scripts Without Saying “expect”
      3. Rewriting The #! Line
      4. The .exp Extension
      5. The—And Other Flags
      6. The —c Flag
      7. The -f Flag
      8. Writing The #! Line
      9. The −i Flag
      10. The -n And -N Flags
      11. The -d Flag
      12. The -D Flag
      13. The -b Flag
      14. The - Flag
      15. The interpreter Command
      16. Exercises
    14. 10. Handling Multiple Processes
      1. The spawn_id Variable
      2. Example—chess Versus chess
      3. Example—Automating The write Command
      4. How exp_continue Affects spawn_id
      5. The Value Of spawn_id Affects Many Commands
      6. Symbolic Spawn Ids
      7. Job Control
      8. Procedures Introduce New Scopes
      9. How Expect Writes Variables In Different Scopes
      10. Predefined Spawn Ids
      11. Exercises
    15. 11. Handling Multiple Processes Simultaneously
      1. Implicit Versus Explicit Spawn Ids
      2. Waiting From Multiple Processes Simultaneously
      3. Example—Answerback
      4. Which Pattern Goes With Which Spawn Id
      5. Which Spawn Id Matched
      6. Spawn Id Lists
      7. Example—Connecting Together Two Users To An Application
      8. Example—Timing All Commands
      9. Matching Any Spawn Id Already Listed
      10. The expect_before And expect_after Commands
      11. Indirect Spawn Ids
      12. Exercises
    16. 12. Send
      1. Implicit Versus Explicit Spawn Ids
      2. Sending To Multiple Processes
      3. Sending Without Echoing
      4. Sending To Programs In Cooked Mode
      5. Sending Slowly
      6. Sending Humanly
      7. Sending Nulls
      8. Sending Breaks
      9. Sending Strings That Look Like Flags
      10. Sending Character Graphics
      11. Comparing send To puts
      12. Exercises
    17. 13. Spawn
      1. The Search Path
      2. Philosophy--Processes Are Smart
      3. Treating Files As Spawned Processes
      4. Opening Ttys
      5. Bugs And Workarounds
      6. Process Pipelines And Ptys
      7. Automating xterm
      8. Checking For Errors From spawn
      9. spawn -noecho
      10. Example—unbuffer
      11. Obtaining Console Output
      12. Setting Pty Modes From spawn
      13. Hung Ptys
      14. Restrictions On Spawning Multiple Processes
      15. Getting The Process Id From A Spawn Id
      16. Using File I/O Commands On Spawned Processes
      17. Exercises
    18. 14. Signals
      1. Signals
      2. Signals In Spawned Processes
      3. Notes On Specific Signals
      4. When And Where Signals Are Evaluated
      5. Overriding The Original Return Value
      6. Using A Different Interpreter To Process Signals
      7. Exit Handling
      8. Exercises
    19. 15. Interact
      1. The interact Command
      2. Simple Patterns
      3. Exact Matching
      4. Matching Patterns From The Spawned Process
      5. Regular Expressions
      6. What Happens To Things That Do Not Match
      7. More Detail On Matching
      8. Echoing
      9. Avoiding Echoing
      10. Giving Feedback Without -echo
      11. Telling The User About New Features
      12. Sending Characters While Pattern Matching
      13. The continue And break Actions
      14. The return Action
      15. The Default Action
      16. Detecting End-Of-File
      17. Matching A Null Character
      18. Timing Out
      19. More On Terminal Modes (Or The -reset Flag)
      20. Example—Preventing Bad Commands
      21. Exercises
    20. 16. Interacting With Multiple Processes
      1. Connecting To A Process Other Than The Currently Spawned Process
      2. Connecting To A Process Instead Of The User
      3. Example—rz And sz Over rlogin
      4. Redirecting Input And Output
      5. Default Input And Output
      6. Controlling Multiple Processes—kibitz
      7. Combining Spawn Ids In A Single -input Or -output
      8. Which Spawn Id Matched
      9. Indirect Spawn Ids
      10. An Extended Example—xkibitz
      11. Exercises
    21. 17. Background Processing
      1. Putting Expect In The Background
      2. Running Expect Without A Controlling Terminal
      3. Disconnecting The Controlling Terminal
      4. The fork Command
      5. The disconnect Command
      6. Reconnecting
      7. Using kibitz From Other Expect Scripts
      8. Mailing From Expect
      9. A Manager For Disconnected Processes—dislocate
      10. Expect As A Daemon
      11. Example—Automating Gopher and Mosaic telnet Connections
      12. Exercises
    22. 18. Debugging Scripts
      1. Tracing
      2. Logging
      3. Command Tracing
      4. Variable Tracing
      5. Example—Logging By Tracing
      6. UNIX System Call Tracing
      7. Tk And tkinspect
      8. Traditional Debugging
      9. Debugger Command Overview And Philosophy
      10. Stepping Over Procedure Calls
      11. Stepping Into Procedure Calls
      12. Where Am I
      13. The Current Scope
      14. Moving Up And Down The Stack
      15. Returning From A Procedure
      16. Continuing Execution
      17. Defining Breakpoints
      18. Help
      19. Changing Program Behavior
      20. Changing Debugger Behavior
      21. Exercises
    23. 19. Expect + Tk = Expectk
      1. Tk—A Brief Technical Overview
      2. Expectk
      3. The send Command
      4. An Extended Example—tkpasswd
      5. The expect Command And The Tk Event Loop
      6. The expect_background Command
      7. Multiple Spawn Ids In expect_background
      8. Background Actions
      9. Example—A Dumb Terminal Emulator
      10. Example—A Smarter Terminal Emulator
      11. Using The Terminal Emulator For Testing And Automation
      12. Exercises
    24. 20. Extended Examples
      1. Encrypting A Directory
      2. File Transfer Over telnet
      3. You Have Unread News—tknewsbiff
      4. Exercises
    25. 21. Expect, C, And C++
      1. Overview
      2. Linking
      3. Include Files
      4. Ptys And Processes
      5. Allocating Your Own Pty
      6. Closing The Connection To The Spawned Process
      7. Expect Commands
      8. Regular Expression Patterns
      9. Exact Matching
      10. Matching A Null
      11. What Characters Matched
      12. When The Number Of Patterns Is Not Known In Advance
      13. Expecting From Streams
      14. Running In The Background
      15. Handling Multiple Inputs And More On Timeouts
      16. Output And Debugging Miscellany
      17. Pty Trapping
      18. Exercises
    26. 22. Expect As Just Another Tcl Extension
      1. Adding Expect To Another Tcl-based Program
      2. Differences Between Expect And The Expect Extension In Another Program
      3. Adding Extensions To Expect
      4. Adding Extensions To Expectk
      5. Creating Script-less Expect Programs
      6. Functions And Variables In The Expect Extension
      7. Exercises
    27. 23. Miscellaneous
      1. Random Numbers
      2. Example—Generating Random Passwords
      3. The Expect Library
      4. Expect Versions
      5. Timestamps
      6. The time Command
      7. Exercises
    28. A. Appendix—Commands and Variables
      1. Commands And Flags
      2. Variables
    29. Index
    30. About the Author
    31. SPECIAL OFFER: Upgrade this ebook with O’Reilly

Chapter 4. Glob Patterns And Other Basics

In the last chapter, I showed some simple patterns that allow you to avoid having to specify exactly what you want to wait for. In this chapter, I will describe how to use patterns that you are already probably familiar with from the shell—glob patterns. I will also describe what happens when patterns do not match. I will go over some other basic situations such as how to handle timeouts. Finally I will describe what to do at theends of scripts and processes.

The * Wildcard

Suppose you want to match all of the input and the only thing you know about it is that hi occurs within it. You are not sure if there is more to it, or even if another hi might appear. You just want to get it all. To do this, use the asterisk (*). The asterisk is a wildcard that matches any number of characters. You can write:

expect "hi*"
send "$expect_out(0,string) $expect_out(buffer)"

If the input buffer contained "philosophic\n“, expect would match the entire buffer. Here is the output from the previous commands:


The pattern hi matched the literal hi while the * matched the string "losophic\n“. The first p was not matched by anything in the pattern so it shows up in expect_out(buffer) but not in expect_out(0,string).

Earlier I said that * matches any number of characters. More precisely, it tries to match the longest string possible while still allowing the pattern itself to match. With the input buffer of "philosophic\n“, compare the effects of the following two commands:

expect "hi*"
expect "hi*hi"

In the first one, the * matches losophic\n. This is the longest possible string that the * can match while still allowing the hi to match hi. In the second expect, the * only matches losop, thereby allowing the second hi to match. If the * matched anything else, the entire pattern would fail to match.

What happens with the following command in which there are two asterisks?

expect "*hi*"

This could conceivably match in two ways corresponding to the two occurrences of “hi” in the string.


* matches

hi matches

* matches

possibility (1)




possibility (2)




What actually happens is possibility (1). The first * matches philosop. As before, each * tries to match the longest string possible allowing the total pattern to match, but the *’s are matched from left to right. The leftmost *’s match strings before the rightmost *’s have a chance. While the outcome is the same in this case (that is, the whole pattern matches), I will show cases later where it is necessary to realize that pattern matching proceeds from left to right.

* At The Beginning Of A Pattern Is Rarely Useful

Patterns match at the earliest possible character in a string. In Chapter 3 (p. 74), I showed how the pattern hi matched the first hi in philosophic. However, in the example above, the subpattern hi matched the second hi. Why the difference?

The difference is that hi was preceded by "*“. Since the * is capable of matching anything, the leading * causes the match to start at the beginning of the string. In contrast, the earliest point that the bare hi can match is the first hi. Once that hi has matched, it cannot match anything else—including the second hi.

In practice, a leading * is usually redundant. Most patterns have enough literal letters that there is no choice in how the match occurs. The only remaining difference is that the leading * forces the otherwise unmatched leading characters to be stored in expect_out(0,string). However, the characters will already be stored in expect_out(buffer) so there is little merit on this point alone.[18]

* At The End Of A Pattern Can Be Tricky

When a * appears at the right end of a pattern, it matches everything left in the input buffer (assuming the rest of the pattern matches). This is a useful way of clearing out the entire buffer so that the next expect does not return a mishmash of things that were received previously and things that are brand new.

Sometimes it is even useful to say:

expect *

Here the * matches anything. This is like saying, “I don’t care what’s in the input. Throw it away.” This pattern always matches, even if nothing is there. Remember that * matches anything, and the empty string is anything! As a corollary of this behavior, this command always returns immediately. It never waits for new data to arrive. It does not have to since it matches everything.

In the examples demonstrating * so far, each string was entered by a person who pressed return afterwards. This is typical of most programs, because they run in what is called cooked mode. Cooked mode includes the usual line-editing features such as backspace and delete-previous-word. This is provided by the terminal driver, not the program. This simplifies most programs. They see the line only after you have edited it and pressed return.

Unfortunately, output from processes is not nearly so well behaved. When you watch the output of a program such as ftp or telnet (or cat for that matter), it may seem as if lines appear on your screen as atomic units. But this is not guaranteed. For example, in the previous chapter, I showed that when ftp starts up it looks like this:

% ftp
Connected to
220 ftp.UU.NET FTP server (Version 6.34 Thu Oct 22 14:32:01 EDT 1992) ready.
Name (

Even though the program may have printed "Connected to\n" all at once—perhaps by a single printf in a C program—the UNIX kernel can break this into small chunks, spitting out a few characters eachtime to the terminal. For example, it might print out "Conn" and then”ecte" and then "d to" and so on. Fortunately, computers are so fast that humans do not notice the brief pauses in the middle of output. The reason the system breaks up output like this is that programs usually produce characters faster than the terminal driver can display them. The operating system will obligingly wait for the terminal driver to effectively say, “Okay, I’ve displayed that last bunch of characters. Send me a couple more.” In reality, the system does not just sit there and wait. Since it is running many other programs at the same time, the system switches its attention frequently to other programs. Expect itself is one such “other program” in this sense.

When Expect runs, it will immediately ask for all the characters that a program produced only to find something like "Conn“. If told to wait for a string that matches "Name*:“, Expect will keep asking the computer if there is any more output, and it will eventually find the output it is looking for.

As I said, humans are slow and do not notice this chunking effect. In contrast, Expect is so fast that it is almost always waiting. Thus, it sees most output come as chunks rather than whole lines. With this in mind, suppose you wanted to find out the version of ftp that a host is using. By looking back at the output, you can see that it is contained in the greeting line that begins "220" and ends with "ready.“. Naively, you could wait for that line as:

expect "220*"                     ;# dangerous

If you are lucky, you might get the entire line stored in $expect_out(0,string). You might even get the next line in there as well. But more likely, you will only get a fragment, such as "220 f" or "220 ftp.UU.NE“. Since the pattern 220* matches either of these, expect has no reason to wait further and will return. As I stated earlier, expect returns with whatever is the longest string that matches the pattern. The problem here is that the remainder of the line may not have shown up yet!

If you want to get the entire line, you must be more specific. The following pattern works:


By specifying the text that ends the line, you force expect to wait for the entire line to arrive. The "." is not actually needed just to find the version identifier. You could just make the pattern:


Leaving off the e would be too short. This would allow the pattern to match the r in server rather than ready. It is possible to make the overall pattern even shorter by looking for more unusual patterns. But quite often you trade off readability. There is an art to choosing patterns that are correct, yet not too long but still readable. A good guideline is to give more priority to readability. The pattern matching performed by Expect is very inexpensive.

[18] The more likely reason to see scripts that begin many patterns with "*" is that prior to Expect version 4, all patterns were anchored, with the consequence that most patterns required a leading "*“.

The best content for your career. Discover unlimited learning on demand for around $1/day.