You are previewing Foundations of Software and System Performance Engineering: Process, Performance Modeling, Requirements, Testing, Scalability, and Practice.
O'Reilly logo
Foundations of Software and System Performance Engineering: Process, Performance Modeling, Requirements, Testing, Scalability, and Practice

Book Description

“If this book had been available to Healthcare.gov’s contractors, and they read and followed its life cycle performance processes, there would not have been the enormous problems apparent in that application. In my 40+ years of experience in building leading-edge products, poor performance is the single most frequent cause of the failure or cancellation of software-intensive projects. This book provides techniques and skills necessary to implement performance engineering at the beginning of a project and manage it throughout the product’s life cycle. I cannot recommend it highly enough.”

Don Shafer, CSDP, Technical Fellow, Athens Group, LLC

Poor performance is a frequent cause of software project failure. Performance engineering can be extremely challenging. In Foundations of Software and System Performance Engineering, leading software performance expert Dr. André Bondi helps you create effective performance requirements up front, and then architect, develop, test, and deliver systems that meet them.

Drawing on many years of experience at Siemens, AT&T Labs, Bell Laboratories, and two startups, Bondi offers practical guidance for every software stakeholder and development team participant. He shows you how to define and use metrics; plan for diverse workloads; evaluate scalability, capacity, and responsiveness; and test both individual components and entire systems. Throughout, Bondi helps you link performance engineering with everything else you do in the software life cycle, so you can achieve the right performance–now and in the future–at lower cost and with less pain.

This guide will help you

• Mitigate the business and engineering risk associated with poor system performance

• Specify system performance requirements in business and engineering terms

• Identify metrics for comparing performance requirements with actual performance

• Verify the accuracy of measurements

• Use simple mathematical models to make predictions, plan performance tests, and anticipate the impact of changes to the system or the load placed upon it

• Avoid common performance and scalability mistakes

• Clarify business and engineering needs to be satisfied by given levels of throughput and response time

• Incorporate performance engineering into agile processes

• Help stakeholders of a system make better performance-related decisions

• Manage stakeholders’ expectations about system performance throughout the software life cycle, and deliver a software product with quality performance

André B. Bondi is a senior staff engineer at Siemens Corp., Corporate Technologies in Princeton, New Jersey. His specialties include performance requirements, performance analysis, modeling, simulation, and testing. Bondi has applied his industrial and academic experience to the solution of performance issues in many problem domains. In addition to holding a doctorate in computer science and a master’s in statistics, he is a Certified Scrum Master.

Table of Contents

  1. Title Page
  2. Copyright Page
  3. Contents
  4. Praise for Foundations of Software and System Performance Engineering
  5. Preface
    1. Scope and Purpose
    2. Audience
  6. Acknowledgments
  7. About the Author
  8. Chapter 1. Why Performance Engineering? Why Performance Engineers?
    1. 1.1. Overview
    2. 1.2. The Role of Performance Requirements in Performance Engineering
    3. 1.3. Examples of Issues Addressed by Performance Engineering Methods
    4. 1.4. Business and Process Aspects of Performance Engineering
    5. 1.5. Disciplines and Techniques Used in Performance Engineering
    6. 1.6. Performance Modeling, Measurement, and Testing
    7. 1.7. Roles and Activities of a Performance Engineer
    8. 1.8. Interactions and Dependencies Between Performance Engineering and Other Activities
    9. 1.9. A Roadmap Through the Book
    10. 1.10. Summary
    11. References
  9. Chapter 2. Performance Metrics
    1. To Do
    2. 2.1. General
    3. 2.2. Examples of Performance Metrics
    4. 2.3. Useful Properties of Performance Metrics
    5. 2.4. Performance Metrics in Different Domains
    6. 2.5. Examples of Explicit and Implicit Metrics
    7. 2.6. Time Scale Granularity of Metrics
    8. 2.7. Performance Metrics for Systems with Transient, Bounded Loads
    9. 2.8. Summary
    10. 2.9. Exercises
    11. References
  10. Chapter 3. Basic Performance Analysis
    1. 3.1. How Performance Models Inform Us About Systems
    2. 3.2. Queues in Computer Systems and in Daily Life
    3. 3.3. Causes of Queueing
    4. 3.4. Characterizing the Performance of a Queue
    5. 3.5. Basic Performance Laws: Utilization Law, Little’s Law
    6. 3.6. A Single Server Queue
    7. 3.7. Networks of Queues: Introduction and Elementary Performance Properties
    8. 3.8. Open and Closed Queueing Network Models
    9. 3.9. Bottleneck Analysis for Single Class Closed Queueing Networks
    10. 3.10. Regularity Conditions for Computationally Tractable Queueing Network Models
    11. 3.11. Mean Value Analysis of Single Class Closed Queueing Network Models
    12. 3.12. Multiple Class Queueing Networks
    13. 3.13. Finite Pool Sizes, Lost Calls, and Other Lost Work
    14. 3.14. Using Models for Performance Prediction
    15. 3.15. Limitations and Applicability of Simple Queueing Network Models
    16. 3.16. Linkage Between Performance Models, Performance Requirements, and Performance Test Results
    17. 3.17. Applications of Basic Performance Laws to Capacity Planning and Performance Testing
    18. 3.18. Summary
    19. Exercises
    20. References
  11. Chapter 4. Workload Identification and Characterization
    1. 4.1. Workload Identification
    2. 4.2. Reference WorkLoads for a system in different environments
    3. 4.3. Time-Varying Behavior
    4. 4.4. Mapping Application Domains to Computer System Workloads
    5. 4.5. Numerical Specification of the Workloads
    6. 4.6. Numerical Illustrations
    7. 4.7. Summary
    8. Exercises
    9. References
  12. Chapter 5. From Workloads to Business Aspects of Performance Requirements
    1. 5.1. Overview
    2. 5.2. Performance Requirements and Product Management
    3. 5.3. Performance Requirements and the Software Life Cycle
    4. 5.4. Performance Requurements and the Mitigation of Business Risk
    5. 5.5. Commercial Considerations and Performance Requirements
    6. 5.6. Guidelines for Specifying Performance Requirements
    7. 5.7. Summary
    8. Exercises
    9. References
  13. Chapter 6. Qualitative and Quantitative Types of Performance Requirements
    1. 6.1. Qualitative Attributes Related to System Performance
    2. 6.2. The Concept of Sustainable Load
    3. 6.3. Formulation of Response Time Requirements
    4. 6.4. Formulation of Throughput Requirements
    5. 6.5. Derived and Implicit Performance Requirements
    6. 6.6. Performance Requirements Related to Transaction Failure Rates, Lost Calls, and Lost Packets
    7. 6.7. Performance Requirements Concerning Peak and Transient Loads
    8. 6.8. Summary
    9. Exercises
    10. References
  14. Chapter 7. Eliciting, Writing, and Managing Performance Requirements
    1. 7.1. Elicitation and Gathering of Performance Requirements
    2. 7.2. Ensuring That Performance Requirements are Enforceable
    3. 7.3. Common Patterns and Anti-Patterns for Performance Requirements
    4. 7.4. The Need for Mathematically Consistent Requirements: Ensuring That Requirements Conform to Basic Performance Laws
    5. 7.5. Expressing Performance Requirements in Terms of Parameters with Unknown Values
    6. 7.6. Avoidance of Circular Dependencies
    7. 7.7. External Performance Requirements and Their Implications for the Performance Requirements of Subsystems
    8. 7.8. Structuring Performance Requirements Documents
    9. 7.9. Layout of a Performance Requirement
    10. 7.10. Managing Performance Requirements: Responsibilities of the Performance Requirements Owner
    11. 7.11. Performance Requirements Pitfall: Transition from a Legacy System to a New System
    12. 7.12. Formulating Performance Requirements to Facilitate Performance Testing
    13. 7.13. Storage and Reporting of Performance Requirements
    14. 7.14. Summary
    15. References
  15. Chapter 8. System Measurement Techniques and Instrumentation
    1. 8.1. General
    2. 8.2. Distinguishing Between Measurement and Testing
    3. 8.3. Validate, Validate, Validate; Scrutinise, Scrutinise, Scrutinise
    4. 8.4. Resource Usage Measurements
    5. 8.5. Utilizations and the Averaging Time Window
    6. 8.6. Measurement of Multicore or Multiprocessor Utilizations Systems
    7. 8.7. Measuring Memory-Related Activity
    8. 8.8. Measurement in Production Vs. Measurement for Performance Testing and Scalability
    9. 8.9. Measuring Systems With One Host and with Multiple Hosts
    10. 8.10. Measurements from Within the Application
    11. 8.11. Measurements in Middleware
    12. 8.12. Measurements of Commercial Databases
    13. 8.13. Response Time Measurements
    14. 8.14. Code Profiling
    15. 8.15. Validation of Measurements Using Basic Properties of Performance Metrics
    16. 8.16. Measurement Procedures and Data Organization
    17. 8.17. Organization of Performance Data, Data Reduction, and Presentation
    18. 8.18. Interpreting Measurements in a Virtualized Environment
    19. 8.19. Summary
    20. 8.20. Exercises
    21. References
  16. Chapter 9. Performance Testing
    1. 9.1. Overview of Performance Testing
    2. 9.2. Special Challenges
    3. 9.3. Performance Test Planning and Performance Models
    4. 9.4. A Wrong Way to Evaluate System Throughput
    5. 9.5. Provocative Performance Testing
    6. 9.6. Preparing a Performance Test
    7. 9.7. Lab Discipline in Performance Testing
    8. 9.8. Performance Testing Challenges Posed by Systems with Multiple Hosts
    9. 9.9. Performance Testing Scripts and Checklists
    10. 9.10. Best Practices for Documenting Test Plans and Test Results
    11. 9.11. Linking the Performance Test Plan to Performance Requirements
    12. 9.12. The Role of Performance Tests in Detecting and Debugging Concurrency Issues
    13. 9.13. Planning Tests for System Stability
    14. 9.14. Prospective Testing When Requirements are Unspecified
    15. 9.15. Structuring the Test Environment to Reflect the Scalability of the Architecture
    16. 9.16. Data Collection
    17. 9.17. Data Reduction and Presentation
    18. 9.18. Interpreting the Test Results
    19. 9.19. Automating Performance Tests and the Analysis of the Outputs
    20. 9.20. Summary
    21. Exercises
    22. References
  17. Chapter 10. System Understanding, Model Choice, and Validation
    1. 10.1. Overview
    2. 10.2. Phases of a Modeling Study
    3. 10.3. Example: A Conveyor System
    4. 10.4. Example: Modeling Asynchronous I/O
    5. 10.4. Systems with Load Dependent or Time-Varying Behaviour
    6. 10.5. Summary
    7. Exercises
    8. References
  18. Chapter 11. Scalability and Performance
    1. 11.1. What is Scalability?
    2. 11.2. Scaling Methods
    3. 11.3. Types of Scalability
    4. 11.4. Interactions Between Types of Scalability
    5. 11.5. Qualitative Analysis of Load Scalability and Examples
    6. 11.6. Scalability Limitations in a Development Environment
    7. 11.7. Improving Load Scalability
    8. 11.8. Some Mathematical Analyses
    9. 11.9. Avoiding Scalability Ptifalls
    10. 11.10. Performance Testing and Scalability
    11. 11.11. Summary
    12. Exercises
    13. References
  19. Chapter 12. Performance Engineering Pitfalls
    1. 12.1. Overview
    2. 12.2. Pitfalls in Priority Scheduling
    3. 12.3. Transient CPU Saturation is not Always a Bad Thing
    4. 12.4. Diminishing Returns with Multiprocessors or Multiple Cores
    5. 12.5. Garbage Collection Can Degrade Performance
    6. 12.6. Virtual Machines: Panacea or Complication?
    7. 12.7. Measurement PitFall: Delayed Time Stamping and Monitoring in Real Time Systems
    8. 12.8. Pitfalls in Performance Measurement
    9. 12.9. Eliminating a Bottlneck Could UnMask a New One
    10. 12.10. Pitfalls in Performance Requirements Engineering
    11. 12.10. Eliminating a Bottlneck Could UnMask a New One
    12. 12.11. Organizational Pitfalls in Performance Engineering
    13. 12.12. Summary
    14. Exercises
    15. References
  20. Chapter 13. Agile Processes and Performance Engineering
    1. 13.1. Overview
    2. 13.2. Performance Engineering Under an Agile Development Process
    3. 13.3. Agile Methods in the Implementation and Execution of Performance Tests
    4. 13.4. The Value of Play Time in an Agile Performance Testing Process
    5. 13.5. Summary
    6. Exercises
    7. References
  21. Chapter 14. Working with Stakeholders to Learn, Influence, and Tell the Performance Engineering Story
    1. 14.1. Determining What Aspect of Performance Matters to Whom
    2. 14.2. Where does the Performance Story Begin?
    3. 14.3. Identification of Performance Concerns, Drivers, and Stakeholders
    4. 14.4. Influencing the Performance Story
    5. 14.5. Reporting on Performance Status to Different Stakeholders
    6. 14.6. Examples
    7. 14.7. The Role of a Capacity Management Engineer
    8. 14.8. Example: Explaing the Role of Measurement Intervals when Interpreting Measurements
    9. 14.9. Ensuring Ownership of Performance Concerns and Explanations By Diverse Stakeholders
    10. 14.10. Negotiating Choices for Design Changes and Recommendations for System Improvement Among Stakeholders
    11. 14.11. Summary
    12. Exercises
    13. References
  22. Chapter 15. Where to Learn More
    1. 15.1. Overview
    2. 15.2. Conferences and Journals
    3. 15.3. Texts on Performance Analysis
    4. 15.4. Queueing Theory
    5. 15.5. Discrete Event Simulation
    6. 15.6. Performance Evaluation of Specific Types of Systems
    7. 15.7. Statistical Methods
    8. 15.8. Performance Tuning
    9. 15.9. Summary
    10. References