You are previewing Linux® Debugging and Performance Tuning: Tips and Techniques.
O'Reilly logo
Linux® Debugging and Performance Tuning: Tips and Techniques

Book Description

Learn Linux debugging and optimization—at kernel and application levels—hands-on!

This is the definitive guide to Linux software debugging and performance optimization at both the kernel and application levels. Using extensive Linux code examples, Steve Best systematically introduces open source tools and best-practice techniques for delivering bug-free, well-tuned code.

Drawing on his exceptional experience optimizing Linux systems for IBM, Best covers issues ranging from memory management and I/O to system processes and kernel bug messages. You'll walk through real debugging sessions, discovering the strategies experts use to debug even the most complex application- and kernel-related problems. You'll master sophisticated profiling techniques for identifying and resolving bottlenecks more quickly and learn how to capture the right data in the event of trouble. Coverage includes

  • Bottleneck identification

  • Code coverage

  • Debuggers: gdb, kgdb, and KDB

  • Memory management

  • /proc kernel data analysis

  • System process monitoring

  • Network performance

  • Oops bug messages

  • Syslog and event logging

  • Execution traces

  • Profiling kernel behavior

  • Cache misses

  • User-Mode Linux

  • Dynamic probes

  • Crash dump analysis

  • And more...

Linux® Debugging and Performance Tuning will be indispensable for every developer who needs to supercharge the Linux kernel and applications, and for every administrator and support specialist who must resolve Linux reliability or performance issues.

© Copyright Pearson Education. All rights reserved.

Table of Contents

  1. Copyright
    1. Dedication
  2. Prentice Hall Open Source Software Development Series
  3. Foreword
  4. Acknowledgments
  5. About the Author
  6. Introduction
    1. This Book’s Audience
    2. Chapter Descriptions
  7. 1. Profiling
    1. stopwatch
    2. date
    3. time
    4. clock
    5. gettimeofday
    6. Performance Tuning Using GNU gprof
    7. gcc Option Needed for gprof
      1. Building the sample1 Program and Using gprof
    8. kprof
      1. Installation
      2. Building Graphviz, the Graph Feature
    9. Summary
    10. Web Resources for Profiling
  8. 2. Code Coverage
    1. What Is Code Coverage Analysis?
    2. gcov Can’t Find Logic Errors
    3. Types of Code Coverage
    4. gcc Options Needed for gcov
    5. Summary
    6. Web Resource for gcov
  9. 3. GNU Debugger (gdb)
    1. Installing gdb
    2. gdb Commands
      1. Some Useful gdb Commands
    3. Compiling a Program That Is to Be Debugged with gdb
    4. A Typical gdb Session
      1. Invoking the gdb Debugger
    5. Debugging Using a Core File
    6. Running the Program and Getting the Core Dump
    7. Graphical gdb Interfaces
    8. Data Display Debugger
      1. ddd Features
      2. Installing ddd
      3. ddd Ways to View a Program
    9. Insight
      1. Insight Features
      2. Installing Insight
    10. Debugging Symbols Increase the Executable’s Size
    11. Debugging Techniques
    12. Summary
    13. Web Resources for GNU Debugger
  10. 4. Memory Management Debugging
    1. Dynamic Memory Functions
    2. MEMWATCH
    3. YAMD
    4. Electric Fence
    5. Valgrind
      1. Installing Valgrind
      2. Losing Your Memory
      3. Cache Profiling
      4. Some Limitations of Valgrind
    6. Summary
    7. Web Resources for Memory Checkers
  11. 5. System Information (/proc)
    1. What Is /proc?
    2. Ways an Administrator Can Use /proc
    3. A Map of /proc
      1. Process-Specific Subdirectories
        1. buddyinfo
        2. bus
          1. lsusb
          2. lspci
        3. cmdline
        4. cpuinfo
        5. diskstats
          1. iostat
          2. sar
        6. interrupts
        7. lsdev
        8. kallsyms
        9. kcore
        10. kmsg
        11. loadavg
        12. meminfo
        13. vmstat
        14. free
        15. mpstat
        16. net
        17. slabinfo
        18. stat
        19. sys
        20. dev/fs
        21. /sys/kernel
        22. /sys/vm
        23. sysrq-trigger
        24. /proc/vmstat
    4. Summary
    5. Web Resources for /proc
  12. 6. System Tools
    1. Processes
    2. Task States
    3. Tools for Working with Processes
      1. ps Reports the Process Status
        1. Building and Running ps-test2
        2. The ps Option to Show the Syscall Currently Being Executed
        3. How to Start a New Process
      2. pgrep Lists the Process IDs That Match the Input Criteria
      3. pstree Displays a Tree of Processes
      4. top Displays Tasks
    4. strace Traces System Calls
    5. The Magic Key Sequence Gets a Back Trace
    6. lsof Lists Open Files
    7. Network Debugging Tools
      1. ifconfig Configures the Network Interface
      2. arp Manipulates the System’s ARP Cache
      3. tcpdump Dumps Traffic on a Network
      4. ethereal Interactively Browses Network Traffic
      5. netstat Shows the Network Status
    8. Summary
    9. Web Resources for Systems Tools
  13. 7. System Error Messages
    1. Kernel Files
      1. Typical Distribution /boot Layout of the Kernel Files
        1. Config File
        2. System.map
          1. Programs That Use System.map
    2. Oops Analysis
      1. Building the Kernel
    3. Processing the Oops Using ksymoops
    4. Using gdb to Display jfs_mount
    5. The gcc -S Option Generates Assembly Code
      1. Assembler Output Generated by GNU C
    6. Kernel Mailing List Oops
      1. An Oops in the 3c59x Network Driver 2.4.7ac11
      2. A Linux-2.5.1-pre5 Oops
    7. perror Describes a System or MySQL Error Code
    8. Summary
    9. Web Resources for Oops
  14. 8. Event Logging
    1. Error Logging Subsystem for syslogd
      1. syslogd Key Options
      2. Remote Logging
      3. syslog.conf
        1. Understanding Facilities and Priorities
        2. Actions
      4. Changing syslogd Messages
      5. klogd
      6. Logger Command
      7. Provide Sufficient Disk Space
      8. Log File Management
      9. Debugging Using Syslog Messages
    2. Error Logging Subsystem for Event Logging
      1. Event Logging Interfaces
      2. Benefits of Event-Type Logging
      3. Log Management
      4. Different Types of Event Log Records
      5. Managing the Event Log
      6. Event Logging Utilities
      7. Remote Logging
    3. Setting up the Event Consolidator
      1. Forwarding Syslog Messages to the Event Log
      2. evlog Packages
    4. Summary
    5. Credits
    6. Web Resource for Event Logging
  15. 9. Linux Trace Toolkit
    1. Architecture Component Tasks
    2. Package and Installation
    3. Building the Kernel
    4. Building LTT User-Level Tools
    5. Data Acquisition
    6. Recording the Data
    7. Stopping Data Recording
    8. Ways to Analyze Tracing Processes
    9. Data Interpretation
      1. tracevisualizer
        1. Event Graph
        2. Process Analysis
        3. Raw Trace
        4. Key tracevisualizer Options
          1. -o (Omit Event Types)
          2. -t (Event Types Tracing)
          3. -c (CPU ID Tracing)
          4. -p (PID Tracing)
          5. -a (Account Time Tracing)
    10. Tracing Example for Tarring a Subdirectory
    11. Data Reviewing Text Tools
      1. tracedcore filename
      2. traceanalyze filename
      3. tracedump filename
    12. Summary
    13. Credits
    14. Web Resource for the Linux Trace Toolkit
  16. 10. oprofile: a Profiler Supported by the Kernel
    1. Instrumentation
    2. Sampling
    3. oprofile: a System-Wide Profiler
      1. Building the Kernel
    4. Utilities for oprofile
    5. General Profile Steps
    6. Examining a Single Executable’s Profile
    7. Report Examples
      1. System-Wide Binary Image Summary
      2. System-Wide Symbol Summary, Including Per-Application Libraries
    8. Saving Profiling Data
    9. Hardware Counters
      1. Minimizing Cache Misses
      2. Padding and Aligning Structures
      3. Packing
      4. Loop Grouping
      5. Blocking
        1. Data Cache Misses
    10. The Prospect Profiler
    11. Summary
    12. Web Resources for Profiling
  17. 11. User-Mode Linux
    1. UML: Introduction to the Kernel and Root File System
    2. Patching and Building the Kernel
      1. UML Kernel Options
        1. gcov support
        2. gprof support
    3. Building the Kernel
    4. Root Image
    5. File Systems
      1. Swap Partition
    6. Setting up a gdb Session
    7. Booting UML
    8. A Typical gdb Session
    9. GDB Thread Analysis
    10. Tips
      1. Consoles and Serial Lines
      2. UML Networking Support
      3. Root Image
      4. Adding File Systems
      5. Shutting Down
    11. UML Utilities
    12. Summary
    13. Credits
    14. Web Resources for User-Mode Linux
  18. 12. Dynamic Probes
    1. Unique Aspects of Dynamic Probes
    2. General Steps for Using Probes
    3. Kprobes: Kernel Dynamic Probes
      1. Building the Kernel
      2. Kprobe Interfaces
      3. Registering and Unregistering Kprobes
        1. register_probe
          1. addr
          2. pre_handler
          3. post_handler
          4. fault_handler
        2. unregister_probe
    4. Probe Example for sys_open
    5. Makefile for a Basic Kernel Module
    6. Finding Kprobes That Are Active on the System
    7. Finding an Offset in sys_open
    8. Jumper Probes
    9. Uses of Kprobes
    10. Successful Employment of Dprobes
    11. Summary
    12. Credits
    13. Web Resource for Dynamic Probes
  19. 13. Kernel-Level Debuggers (kgdb and kdb)
    1. kgdb
      1. kgdb Kernel Patching
        1. Building the Kernel
      2. Checking the Null Modem Serial Cable Setup
        1. Booting the Kernel
        2. Connecting gdb
      3. A Typical gdb Session
      4. Using kgdb to Stop and View the Kernel Source
      5. gdb Thread Analysis
    2. kdb
      1. kdb Kernel Patching
      2. Building the Kernel
      3. kdb Activation
        1. Using kdb Via a Serial Console Port
      4. kdb Commands
      5. kdb Debug Sessions
        1. kdb Debug Session 1
        2. kdb Debug Session 2
        3. kdb Debug Session 3
          1. List All Processes (ps)
          2. Back Trace for All Processes (bta)
    3. Summary
    4. Credits
    5. Web Resources for kgdb and kdb
  20. 14. Crash Dump
    1. Kernel Configuration
      1. Crash Dump Kernel Options
    2. Patching and Building the Kernel
      1. Building the Kernel
    3. General Crash Dump Steps
    4. LKCD Commands
    5. System Panic and Crash Dump Taken
    6. Netdump: The Network Crash Dump Facility from Both the Client and Server
      1. Server
      2. Client
    7. diskdump: a Crash Dump Facility
      1. General Steps to Configure diskdump
    8. Viewing an mcore Crash Dump
    9. Summary
    10. Credits
    11. Web Resources for Crash Dump