Pro Python System Administration

Pro Python System Administration

by Rytis Sileika
Released July 2010
Publisher(s): Apress
ISBN: 9781430226055

Read it now on the O’Reilly learning platform with a 10-day free trial.

O’Reilly members get unlimited access to books, live events, courses curated by job role, and more from O’Reilly and nearly 200 top publishers.

Start your free trial

Book description

As time goes on, system administrators are presented with increasingly complicated challenges. In the early days, a team of engineers might have had to look after one or two systems. These days, one engineer can administer hundreds or thousands of systems.

System administrators are gradually replacing their tools with more advanced and flexible ones. One of the choices is Python. Structurally, Python is a modern, high-level language with a very clean syntax. Python comes with many built-in libraries that can make automation tasks easier. It also has extensive set of third-party libraries and a very active development community. This flexibility makes Python a good choice for a wide variety of tasks, from prototyping science applications to system maintenance and administration jobs.

  • This book explains and shows how to apply Python scripting in practice. Unlike the majority of the Python books, it will show you how to approach and resolve real-world issues that most system administrators will come across in their careers.

  • In this book, you will find several projects in the categories of network administration, web server administration, and monitoring and database management. In each project, we will define the problem, design the solution, and go through the more interesting implementation steps.

  • Each project is accompanied with the source code of a fully working prototype, which you'll be able to use immediately or adapt to your requirements and environment.

Table of contents

  1. Copyright
  2. About the Author
  3. About the Technical Reviewer
  4. Acknowledgments
  5. Introduction
    1. Prerequisites for This Book
    2. Structure of This Book
      1. Chapter 1: Reading and Collecting Performance Data Using SNMP
      2. Chapter 2: Managing Devices Using the SOAP API
      3. Chapter 3: Creating a Web Application for IP Address Accountancy
      4. Chapter 4: Integrating the IP Address Application with DHCP
      5. Chapter 5: Maintaining a List of Virtual Hosts in an Apache Configuration File
      6. Chapter 6: Gathering and Presenting Statistical Data from Apache Log Files
      7. Chapter 7: Performing Complex Searches and Reporting on Application Log Files
      8. Chapter 8: A Web Site Availability Check Script for Nagios
      9. Chapter 9: Management and Monitoring Subsystem
      10. Chapter 10: Remote Monitoring Agents
      11. Chapter 11: Statistics Gathering and Reporting
      12. Chapter 12: Automatic MySQL Database Performance Tuning
      13. Chapter 13: Amazon EC2/S3 as a Data Warehouse Solution
    3. The Example Source Code
  6. 1. Reading and Collecting Performance Data Using SNMP
    1. 1.1. Application Requirements and Design
      1. 1.1.1. Specifying the Requirements
      2. 1.1.2. High-Level Design Specification
    2. 1.2. Introduction to SNMP
      1. 1.2.1. The System SNMP Variables Node
      2. 1.2.2. The Interfaces SNMP Variables Node
      3. 1.2.3. Authentication in SNMP
      4. 1.2.4. Querying SNMP from the Command Line
    3. 1.3. Querying SNMP Devices from Python
      1. 1.3.1. Configuring the Application
      2. 1.3.2. Using the PySNMP Library
        1. 1.3.2.1. The SNMP GET Command
        2. 1.3.2.2. The SNMP SET Command
        3. 1.3.2.3. The SNMP GETNEXT Command
      3. 1.3.3. Implementing the SNMP Read Functionality
    4. 1.4. Storing Data with RRDTool
      1. 1.4.1. Introduction to RRDTool
      2. 1.4.2. Using RRDTool from a Python Program
      3. 1.4.3. Creating a RoundRobin Database
      4. 1.4.4. Writing and Reading Data from the RoundRobin Database
      5. 1.4.5. Plotting Graphs with RRDTool
      6. 1.4.6. Integrating RRDTool with the Monitoring Solution
    5. 1.5. Creating Web Pages with the Jinja2 Templating System
      1. 1.5.1. Loading Template Files with Jinja2
      2. 1.5.2. The Jinja2 Template Language
        1. 1.5.2.1. Accessing Variables
        2. 1.5.2.2. Flow Control Statements
      3. 1.5.3. Generating Web Site Pages
    6. 1.6. Summary
  7. 2. Managing Devices Using the SOAP API
    1. 2.1. What Is the SOAP API?
      1. 2.1.1. The Structure of a SOAP Message
      2. 2.1.2. Requesting Services with SOAP
      3. 2.1.3. Finding Information about Available Services with WSDL
    2. 2.2. SOAP Support in Python
    3. 2.3. Converting WSDL Schema to Python Helper Module
    4. 2.4. Defining Requirements for Our Load Balancer Tool
      1. 2.4.1. Basic Requirements
      2. 2.4.2. Code Structure
      3. 2.4.3. Configuration
    5. 2.5. Accessing Citrix Netscaler Load Balancer with the SOAP API
      1. 2.5.1. Fixing Issues with Citrix Netscaler WSDL
      2. 2.5.2. Creating a Connection Object
      3. 2.5.3. Logging In: Our First SOAP Call
        1. 2.5.3.1. Finding What Is Being Returned in the Response from a Web Service
        2. 2.5.3.2. How Is the Session Maintained After We Have Logged In?
    6. 2.6. Gathering Performance Statistics Data
      1. 2.6.1. SOAP Methods for Reading Statistical Data and Their Return Values
      2. 2.6.2. Reading System Health Data
      3. 2.6.3. Reading Service Status Data
    7. 2.7. Automating Administration Tasks
      1. 2.7.1. Device Configuration SOAP Methods
      2. 2.7.2. Setting a Service State
    8. 2.8. A Word About Logging and Error Handling
      1. 2.8.1. Using the Python logging Module
        1. 2.8.1.1. Logging Levels and Scope
        2. 2.8.1.2. Configuring and Using the Logger
      2. 2.8.2. Handling Exceptions
    9. 2.9. Summary
  8. 3. Creating a Web Application for IP Address Accountancy
    1. 3.1. Designing the Application
      1. 3.1.1. Setting Out the Requirements
      2. 3.1.2. Making Design Decisions
      3. 3.1.3. Defining the Database Schema
      4. 3.1.4. Creating the Application Workflow
        1. 3.1.4.1. The Search and Display Functions
        2. 3.1.4.2. The Add Function
        3. 3.1.4.3. The Delete Function
        4. 3.1.4.4. The Modify Function
        5. 3.1.4.5. The System Health Check Function
        6. 3.1.4.6. The Name Resolution Function
    2. 3.2. The Basic Concepts of the Django Framework
      1. 3.2.1. What Is Django?
        1. 3.2.1.1. The Object-to-Relation Database Mapper
        2. 3.2.1.2. The Administration Interface
        3. 3.2.1.3. A Flexible Template System
        4. 3.2.1.4. Open Source Community Support
      2. 3.2.2. The Model/View/Controller Pattern
        1. 3.2.2.1. The Data Model Component
        2. 3.2.2.2. The View Component
        3. 3.2.2.3. The Controller Component
      3. 3.2.3. Installing the Django Framework
      4. 3.2.4. The Structure of a Django Application
      5. 3.2.5. Using Django with Apache Web Server
    3. 3.3. Implementing Basic Functionality
      1. 3.3.1. Defining the Database Model
      2. 3.3.2. URL Configuration
      3. 3.3.3. Using the Management Interface
        1. 3.3.3.1. Enabling the Management Interface
        2. 3.3.3.2. Allowing the Administration Plug-in to Manage New Models
      4. 3.3.4. Viewing Records
      5. 3.3.5. Using Templates
      6. 3.3.6. Deleting Records
      7. 3.3.7. Adding New Records
      8. 3.3.8. Modifying Existing Records
    4. 3.4. Summary
  9. 4. Integrating the IP Address Application with DHCP
    1. 4.1. Extending the Design and Requirements
      1. 4.1.1. Extending the Database Schema
      2. 4.1.2. Additions to the Workflow
    2. 4.2. Adding DHCP Network Data
      1. 4.2.1. Defining Data Models
      2. 4.2.2. Additional Workflows
      3. 4.2.3. The Add Function
      4. 4.2.4. The Modify Function
      5. 4.2.5. The Delete Function
    3. 4.3. Extending DHCP Configuration with Address Pools
      1. 4.3.1. The Address Pool Data Model
      2. 4.3.2. Displaying DHCP Network Details
      3. 4.3.3. The Add and Delete Functions
    4. 4.4. Reworking the URL Structure
      1. 4.4.1. Generating URLs in the Model Class
      2. 4.4.2. Reverse Resolution of URLs
      3. 4.4.3. Assigning Names to URL Patterns
      4. 4.4.4. Using URL References in the Templates
    5. 4.5. Adding Client Classification
      1. 4.5.1. Additions to the Data Model
      2. 4.5.2. Using Template Inheritance
      3. 4.5.3. Class Rules Management
        1. 4.5.3.1. Generic Views
        2. 4.5.3.2. Displaying a List of Objects
        3. 4.5.3.3. A Detailed View of the Object
        4. 4.5.3.4. Adding and Modifying New Objects
        5. 4.5.3.5. Deleting Objects
    6. 4.6. Generating the DHCP Configuration File
    7. 4.7. Other Modifications
      1. 4.7.1. Resolving IPs to Hostnames
      2. 4.7.2. Checking Whether the Address Is In Use
    8. 4.8. Summary
  10. 5. Maintaining a List of Virtual Hosts in an Apache Configuration File
    1. 5.1. Specifying the Design and Requirements for the Application
      1. 5.1.1. Functional Requirements
      2. 5.1.2. High-Level Design
    2. 5.2. Setting Up the Environment
      1. 5.2.1. Apache Configuration
      2. 5.2.2. Creating a Django Project and Application
      3. 5.2.3. Configuring the Application
      4. 5.2.4. Defining the URL Structure
    3. 5.3. The Data Model
      1. 5.3.1. The Basic Model Structure
    4. 5.4. Modifying the Administration Interface
      1. 5.4.1. Improving the Class and Object Lists
        1. 5.4.1.1. Customizing the Class Names
        2. 5.4.1.2. Adding New Fields to the Object List
        3. 5.4.1.3. Reorganizing the Form Fields
      2. 5.4.2. Adding Custom Object Actions
    5. 5.5. Generating the Configuration File
    6. 5.6. Summary
  11. 6. Gathering and Presenting Statistical Data from Apache Log Files
    1. 6.1. Application Structure and Functionality
      1. 6.1.1. Application Requirements
      2. 6.1.2. Application Design
    2. 6.2. Plug-in Framework Implementation in Python
      1. 6.2.1. The Mechanics of a Plug-in Framework
        1. 6.2.1.1. Interface Model
        2. 6.2.1.2. Plug-in Registration and Discovery
      2. 6.2.2. Creating the Plug-in Framework
        1. 6.2.2.1. Discovery and Registration
        2. 6.2.2.2. Defining the Plug-in Modules
    3. 6.3. Log-Parsing Application
      1. 6.3.1. Format of Apache Log Files
      2. 6.3.2. Log File Reader
      3. 6.3.3. Calling the Plug-in Methods
        1. 6.3.3.1. Tagging the Plug-in Classes
        2. 6.3.3.2. Plug-in Methods and the Call Mechanism
    4. 6.4. Plug-in Modules
      1. 6.4.1. Installing the Required Libraries
        1. 6.4.1.1. Using the GeoIP Python Bindings
      2. 6.4.2. Writing the Plug-in Code
    5. 6.5. Summary
  12. 7. Performing Complex Searches and Reporting on Application Log Files
    1. 7.1. Defining the Problem
      1. 7.1.1. Why We Use Exceptions
      2. 7.1.2. Are Exceptions Always a Bad Sign?
      3. 7.1.3. Why We Should Analyze Exceptions
    2. 7.2. Parsing Complex Log Files
      1. 7.2.1. What Can We Find in a Typical Log File?
      2. 7.2.2. The Structure of an Exception Stack Trace Log
    3. 7.3. Handling Multiple Files
      1. 7.3.1. Handling Multiple Files
      2. 7.3.2. Using the Built In Bzip2 Library
      3. 7.3.3. Traversing Through Large Data Files
      4. 7.3.4. What Are Generators, And How Do We Use Them?
    4. 7.4. Detecting Exceptions
      1. 7.4.1. Detecting Potential Candidates
      2. 7.4.2. Filtering Legitimate Exception Traces
    5. 7.5. Storing Data in Data Structures
      1. 7.5.1. The Structure of Exception Stack Trace Data
      2. 7.5.2. Generating an Exception Fingerprint for Unknown Exceptions
      3. 7.5.3. Detecting Known Exceptions
        1. 7.5.3.1. The Configuration File
        2. 7.5.3.2. Parsing XML Files with Python
        3. 7.5.3.3. Storing and Applying Filters
        4. 7.5.3.4. The Benefits of a Precompiled Search Over a Plain-Text Search
    6. 7.6. Producing Reports
      1. 7.6.1. Grouping Exceptions
      2. 7.6.2. Producing Differently Formatted Output for the Same Dataset
      3. 7.6.3. Calculating Group Statistics
    7. 7.7. Summary
  13. 8. A Web Site Availability Check Script for Nagios
    1. 8.1. Requirements for the Check System
    2. 8.2. The Nagios Monitoring System
      1. 8.2.1. Nagios Plug-In Architecture
    3. 8.3. The Site Navigation Check
      1. 8.3.1. Installing the Beautiful Soup HTML Parsing Library
      2. 8.3.2. Retrieving a Web Page
      3. 8.3.3. Parsing the HTML Pages with Beautiful Soup
      4. 8.3.4. Adding the New Check to the Nagios System
      5. 8.3.5. Emulating the User Login Process
    4. 8.4. Summary
  14. 9. Management and Monitoring Subsystem
    1. 9.1. Design
      1. 9.1.1. The Components
        1. 9.1.1.1. The Monitoring Server
        2. 9.1.1.2. The Monitoring Agent
        3. 9.1.1.3. The Sensors
      2. 9.1.2. The Data Objects
        1. 9.1.2.1. Configuration
        2. 9.1.2.2. Performance Readings
        3. 9.1.2.3. Site Configuration
        4. 9.1.2.4. Scheduling
    2. 9.2. The Data Structures
      1. 9.2.1. Introduction to Data Normalization
      2. 9.2.2. Configuration Data
      3. 9.2.3. Performance Data
      4. 9.2.4. Scheduling
      5. 9.2.5. Site Configuration
        1. 9.2.5.1. Representing the Information in an ER Diagram
    3. 9.3. Communication Flows
      1. 9.3.1. XML-RPC for Information Exchange
        1. 9.3.1.1. Structure
        2. 9.3.1.2. Python Support
      2. 9.3.2. CherryPy
    4. 9.4. The Server Process
      1. 9.4.1. Storing Data in a SQLite3 Database
        1. 9.4.1.1. Initializing the Database File
      2. 9.4.2. Actions
        1. 9.4.2.1. Accepting Sensor Readings
        2. 9.4.2.2. Supplying a New Configuration
        3. 9.4.2.3. Providing New Sensor Code
        4. 9.4.2.4. The Server Health Check
    5. 9.5. The Scheduler
      1. 9.5.1. Actions
      2. 9.5.2. Running Multiple Processes
        1. 9.5.2.1. Multithreading, Multiprocessing, and GIL
        2. 9.5.2.2. Basic Usage Patterns and Examples
      3. 9.5.3. Running Methods at Equal Intervals
        1. 9.5.3.1. A Simple Clock Implementation
      4. 9.5.4. A Cron-Like Scheduler
      5. 9.5.5. Ticket Dispatcher
    6. 9.6. Summary
  15. 10. Remote Monitoring Agents
    1. 10.1. Design
      1. 10.1.1. The Passive Component
      2. 10.1.2. Architecture
      3. 10.1.3. Actions
        1. 10.1.3.1. Accepting a New Configuration
        2. 10.1.3.2. Upgrading the Sensors
        3. 10.1.3.3. Submitting Sensor Readings
    2. 10.2. The Security Model
    3. 10.3. Configuration
      1. 10.3.1. The ConfigParser Library
        1. 10.3.1.1. The File Format
        2. 10.3.1.2. Using the ConfigParser Class Methods
      2. 10.3.2. The Configuration Class Wrapper
    4. 10.4. The Sensor Design
    5. 10.5. Running External Processes
      1. 10.5.1. Using the subprocess Library
      2. 10.5.2. Controlling the Running Processes
      3. 10.5.3. Communicating with External Processes
        1. 10.5.3.1. Using File Descriptors
        2. 10.5.3.2. Using File Objects
        3. 10.5.3.3. Using the Pipe Objects
        4. 10.5.3.4. Redirecting Standard Error
    6. 10.6. Automatically Updating Sensor Code
      1. 10.6.1. Sending and Receiving Binary Data with XML-RPC
      2. 10.6.2. Working with Files and Archives (TAR and BZip2)
    7. 10.7. Summary
  16. 11. Statistics Gathering and Reporting
    1. 11.1. Application Requirements and Design
    2. 11.2. Using the NumPy Library
      1. 11.2.1. Installing NumPy
      2. 11.2.2. NumPy Examples
        1. 11.2.2.1. Working with Arrays
        2. 11.2.2.2. Basic Mathematical and Statistical Operations
        3. 11.2.2.3. Calculating the Mean and Standard Deviation
        4. 11.2.2.4. Finding the Trend Line of a Dataset
        5. 11.2.2.5. Reading and Writing Data to Files
    3. 11.3. Representing Data with matplotlib
      1. 11.3.1. Installing matplotlib
      2. 11.3.2. Understanding the Library Structure
      3. 11.3.3. Plotting Graphs
        1. 11.3.3.1. Changing the Appearance of the Plot Primitives
        2. 11.3.3.2. Drawing Bars and Using Multiple Axes
        3. 11.3.3.3. Working with Text Strings
      4. 11.3.4. Saving Plots to a File
    4. 11.4. Graphing Statistical Data
      1. 11.4.1. Collating Data from the Database
        1. 11.4.1.1. Displaying Available Hosts
      2. 11.4.2. Drawing Timescale Graphs
        1. 11.4.2.1. The Index Page
        2. 11.4.2.2. Host Details Page
        3. 11.4.2.3. Graph Collection Pages
        4. 11.4.2.4. Plotting Performance Graphs
    5. 11.5. Summary
  17. 12. Automatic MySQL Database Performance Tuning
    1. 12.1. Requirements Specification and Design
      1. 12.1.1. Basic Application Requirements
      2. 12.1.2. System Design
    2. 12.2. Modifying the Plug-in Framework
      1. 12.2.1. Changes to the Host Application
      2. 12.2.2. Modifying the Plug-in Manager
    3. 12.3. Writing the Producer Plug-ins
      1. 12.3.1. Accessing the MySQL Database from Python Applications
      2. 12.3.2. Querying the Configuration Variables
      3. 12.3.3. Querying the Server Status Variables
      4. 12.3.4. Collecting the Host Configuration Data
    4. 12.4. Writing the Consumer Plug-ins
      1. 12.4.1. Checking the MySQL Version
      2. 12.4.2. Checking the Key Buffer Size Setting
      3. 12.4.3. Checking the Slow Queries Counter
    5. 12.5. Summary
  18. 13. Using Amazon EC2/S3 as a Data Warehouse Solution
    1. 13.1. Specifying the Problem and the Solution
      1. 13.1.1. The Problem
      2. 13.1.2. Our Solution
      3. 13.1.3. Design Specifications
    2. 13.2. The Amazon EC2 and S3 Crash Course
      1. 13.2.1. Authentication and Security
        1. 13.2.1.1. Account Identifier
        2. 13.2.1.2. Access Credentials
        3. 13.2.1.3. X.509 Certificates
        4. 13.2.1.4. EC2 Key Pair
      2. 13.2.2. The Simple Storage System Concepts
      3. 13.2.3. The Elastic Computing Cloud Concepts
        1. 13.2.3.1. Amazon Machine Images and Instances
        2. 13.2.3.2. Elastic Block Store
        3. 13.2.3.3. Security Groups
        4. 13.2.3.4. Elastic IPs and Load Balancers
      4. 13.2.4. User Interfaces
    3. 13.3. Creating a Custom EC2 Image
      1. 13.3.1. Reusing Existing Images
      2. 13.3.2. Making Modifications
        1. 13.3.2.1. Install the Additional Packages
        2. 13.3.2.2. Create and Setup an Elastic Block Store Volume
        3. 13.3.2.3. Configure the MySQL Instance
      3. 13.3.3. Bundling the New AMI
    4. 13.4. Controlling the EC2 Using the Boto Python Module
      1. 13.4.1. Setting Up the Configuration Variables
      2. 13.4.2. Initializing the EC2 Instance Programmatically
        1. 13.4.2.1. Launching the EC2 Instance
        2. 13.4.2.2. Attaching the EBS Volume
        3. 13.4.2.3. Mounting the EBS Device
        4. 13.4.2.4. Starting the MySQL Instance
      3. 13.4.3. Transferring the Data
      4. 13.4.4. Destroying the EC2 Instance Programmatically
        1. 13.4.4.1. Shutting Down the MySQL Instance
        2. 13.4.4.2. Unmounting the File System
        3. 13.4.4.3. Detaching the EBS Volume
        4. 13.4.4.4. Taking a Snapshot of the Volume
        5. 13.4.4.5. Shutting Down the Instance
        6. 13.4.4.6. The Control Sequence
    5. 13.5. Summary

Product information

  • Title: Pro Python System Administration
  • Author(s): Rytis Sileika
  • Release date: July 2010
  • Publisher(s): Apress
  • ISBN: 9781430226055