You are previewing Performance Testing Guidance for Web Applications.
O'Reilly logo
Performance Testing Guidance for Web Applications

Book Description

Performance Testing Guidance for Web Applications provides an end-to-end approach for implementing performance testing. Whether you are new to performance testing or looking for ways to improve your current performance-testing approach, you will gain insights that you can tailor to your specific scenarios.

Table of Contents

  1. Performance Testing Guidance for Web Applications: Patterns & Practices
    1. SPECIAL OFFER: Upgrade this ebook with O’Reilly
    2. Foreword by Alberto Savoia
      1. Foreword
    3. Foreword by Rico Mariani
      1. Foreword
    4. Introduction
      1. Scope of This Guide
      2. Why We Wrote This Guide
      3. Features of This Guide
      4. Who Should Read This Guide
      5. How to Use This Guide
        1. Ways to Use the Guide
      6. Organization of This Guide
        1. Parts
          1. Part I, Introduction to Performance Testing
          2. Part II, Exemplar Performance Testing Approaches
          3. Part III, Identify the Test Environment
          4. Part IV, Identify Performance Acceptance Criteria
          5. Part V, Plan and Design Tests
          6. Part VI, Execute Tests
          7. Part VII, Analyze Results and Report
          8. Part VIII, Performance-Testing Techniques
      7. Approach Used in This Guide
      8. Feedback on the Guide
      9. The Team Who Brought You This Guide
      10. Contributors and Reviewers
      11. External Contributors and Reviewers
      12. Tell Us About Your Success
    5. I. Introduction to Performance Testing
      1. 1. Fundamentals of Web Application Performance Testing
        1. Objectives
        2. Overview
        3. How to Use This Chapter
        4. Core Activities of Performance Testing
        5. Why Do Performance Testing?
        6. Project Context
        7. The Relationship Between Performance Testing and Tuning
          1. Cooperative Effort
          2. Tuning Process Overview
        8. Performance, Load, and Stress Testing
        9. Baselines
        10. Benchmarking
        11. Terminology
        12. Summary
      2. 2. Types of Performance Testing
        1. Objectives
        2. Overview
        3. How to Use This Chapter
        4. Performance Testing
        5. Key Types of Performance Testing
        6. Summary Matrix of Benefits by Key Performance Test Types
        7. Additional Concepts / Terms
        8. Summary
      3. 3. Risks Addressed Through Performance Testing
        1. Objectives
        2. Overview
        3. How to Use This Chapter
        4. Summary Matrix of Risks Addressed by Performance Testing Types
        5. Speed-Related Risks
          1. Speed-Related Risk-Mitigation Strategies
        6. Scalability-Related Risks
          1. Scalability-Related Risk-Mitigation Strategies
        7. Stability-Related Risks
          1. Stability-Related Risk-Mitigation Strategies
        8. Summary
    6. II. Exemplar Performance Testing Approaches
      1. 4. Web Application Performance Testing Core Activities
        1. Objectives
        2. Overview
        3. How to Use This Chapter
        4. Overview of Activities
        5. Summary Table of Core Performance-Testing Activities
        6. Core Performance-Testing Activities Walkthrough
        7. Activity 1. Identify the Test Environment
          1. Considerations
        8. Activity 2. Identify Performance Acceptance Criteria
          1. Considerations
        9. Activity 3. Plan and Design Tests
          1. Considerations
        10. Activity 4. Configure the Test Environment
          1. Considerations
        11. Activity 5. Implement the Test Design
          1. Considerations
        12. Activity 6. Execute the Test
          1. Considerations
        13. Activity 7. Analyze Results, Report, and Retest
        14. Summary
      2. 5. Coordinating Performance Testing with an Iteration-Based Process
        1. Objectives
        2. Overview
        3. How to Use This Chapter
        4. Introduction to the Approach
        5. Iterative Performance Testing Activities
        6. Relationship to Core Performance Testing Activities
        7. Activity 1. Understand the Project Vision and Context
          1. Checklist
        8. Activity 2. Identify Reasons for Testing Performance
          1. Checklist
        9. Activity 3. Identify the Value Performance Testing Adds to the Project
          1. Checklist
        10. Activity 4. Configure the Test Environment
          1. Checklist
        11. Activity 5. Identify and Coordinate Tasks
          1. Checklist
        12. Activity 6. Execute Task(s)
          1. Checklist
        13. Activity 7. Analyze Results and Report
          1. Checklist
        14. Activity 8. Revisit Activities 1-3 and Consider Performance Acceptance Criteria
          1. Checklist
        15. Activity 9. Reprioritize Tasks
          1. Checklist
        16. Summary
      3. 6. Managing an Agile Performance Test Cycle
        1. Objectives
        2. Overview
        3. How to Use This Chapter
        4. Introduction to the Approach
        5. Agile Performance-Testing Activities
        6. Relationship to Core Performance-Testing Activities
        7. Activity 1. Understand the Project Vision and Context
          1. Project Vision
          2. Project Context
          3. Understand the System
          4. Understand the Project Environment
          5. Understand the Performance Build Schedule
        8. Activity 2. Identify Reasons for Testing Performance
          1. Success Criteria
        9. Activity 3. Identify the Value Performance Testing Adds to the Project
          1. Discussion Points
        10. Activity 4. Configure the Test Environment
        11. Activity 5. Identify and Coordinate Tasks
          1. Discussion Points
        12. Activity 6. Execute Task(s)
          1. Keys to Conducting a Performance-Testing Task
        13. Activity 7. Analyze Results and Report
        14. Activity 8. Revisit Activities 1-3 and Consider Performance Acceptance Criteria
        15. Activity 9. Reprioritize Tasks
        16. Additional Considerations
        17. Summary
      4. 7. Managing the Performance Test Cycle in a Regulated (CMMI) Environment
        1. Objectives
        2. Overview
        3. How to Use This Chapter
        4. Introduction to the Approach
        5. CMMI Performance-Testing Activities
        6. Relationship to Core Performance-Testing Activities
        7. CMMI Performance Testing Activity Flow
        8. Activity 1. Understand the Process and Compliance Criteria
          1. Determine the Process
          2. Determine Compliance Criteria
        9. Activity 2. Understand the System and the Project Plan
          1. Understand the System
          2. Review the Project Plan
        10. Activity 3. Identify Performance Acceptance Criteria
          1. Performance Requirements
          2. Performance Goals
          3. Performance-Testing Objectives
        11. Activity 4. Plan Performance-Testing Activities
          1. Map Work Items to Project Plan
          2. Add Durations
          3. Prioritize Work Items by Phase
          4. Add Detail to the Plan
        12. Activity 5. Design Tests
          1. Determine Navigation Paths for Key Scenarios
          2. Determine Individual User Data and Variances
          3. Determine the Relative Distribution of Scenarios
          4. Identify Target Load Levels
          5. Identify Metrics to Be Captured During Test Execution
          6. Considerations
        13. Activity 6. Configure the Test Environment
        14. Activity 7. Implement the Test Design
        15. Activity 8. Execute Work Items
        16. Activity 9. Report Results and Archive Data
        17. Activity 10. Modify the Plan and Gain Approval for Modifications
        18. Activity 11. Return to Activity 5
        19. Activity 12. Prepare the Final Report
        20. Summary
    7. III. Identify the Test Environment
      1. 8. Evaluating Systems to Increase Performance Testing Effectiveness
        1. Objectives
        2. Overview
        3. How to Use This Chapter
        4. Approach for Evaluating the System
        5. Capture System Functions and/or Business Processes
          1. Considerations
        6. Capture User Activities
          1. Considerations
        7. Capture the Logical and Physical Architecture
          1. Logical Architecture
          2. Physical Architecture
          3. System Architecture
          4. Considerations
        8. Summary
    8. IV. Identify Performance Acceptance Criteria
      1. 9. Determining Performance Testing Objectives
        1. Objectives
        2. Overview
        3. How to Use This Chapter
        4. Terminology
        5. Approach for Determining Performance Testing Objectives
        6. Determine the Objectives of Performance Testing
          1. Determine Overall Objectives
          2. Review the Project Plan
          3. Review the Architecture
          4. Ask Team Members
        7. Capture or Estimate Resource Usage Targets and Thresholds
        8. Capture or Estimate Resource Budgets
        9. Identify Metrics
        10. Communicate Results
        11. Stay Aware of Changing Objectives, Targets, and Budgets
        12. Case Studies—Identifying Performance-testing Objectives
          1. Case Study 1
            1. Scenario
            2. Performance Objectives
            3. Performance Budget/Constraints
            4. Performance-Testing Objectives
            5. Questions
          2. Case Study 2
            1. Scenario
            2. Performance Objectives
            3. Performance Budget/Constraints
            4. Performance-Testing Objectives
            5. Questions
          3. Case Study 3
            1. Scenario
            2. Performance Objectives
            3. Performance Budget/Constraints
            4. Performance-Testing Objectives
            5. Questions
        13. Summary
      2. 10. Quantifying End-User Response Time Goals
        1. Objectives
        2. Overview
        3. How to Use This Chapter
        4. Terminology
        5. Approach for Quantifying End-User Response Time
        6. Determine Application Functionality and Usage
        7. Verbalize and Capture Performance Requirements and Goals
          1. Review Project Documentation and Related Contracts
          2. Interview Stakeholders Who Will Influence the "Go Live" Decision
          3. Determine If There Are Relevant Standards and/or Competitive Baselines Related to the Application
        8. Quantify Performance Requirements and Goals
          1. Separate Requirements from Goals
          2. Quantify Captured Performance Goals
          3. Quantify Captured Performance Requirements
        9. Record Performance Requirements and Goals
        10. Summary
      3. 11. Consolidating Various Types of Performance Acceptance Criteria
        1. Objectives
        2. Overview
        3. How to Use This Chapter
        4. Terminology
        5. Approach for Consolidating Acceptance Criteria
        6. Investigate End-User Requirements
        7. Collect Business Requirements
        8. Determine Technical Requirements
        9. Research Standards, Compliance, and Contracts
        10. Establish Performance-Testing Objectives
        11. Compare and Consolidate Performance Characteristics
        12. Review and Update the Performance Plan
        13. Summary
    9. V. Plan and Design Tests
      1. 12. Modeling Application Usage
        1. Objectives
        2. Overview
        3. How to Use This Chapter
        4. Approach for Modeling Application Usage
        5. Identify the Objectives
          1. Considerations
        6. Determine Key Usage Scenarios
          1. Considerations
        7. Determine Navigation Paths for Key Scenarios
          1. Considerations
            1. Example Visual Representation
        8. Determine Individual User Data and Variances
          1. Web Site Metrics in Web Logs
        9. Determine the Relative Distribution of Scenarios
          1. Considerations
        10. Identify Target Load Levels
          1. Quantifying the Volume of Application Usage: Theory
          2. Quantifying the Volume of Application Usage
            1. Inputs
            2. Output
          3. Integrating Model Variance
          4. Considerations
        11. Prepare to Implement the Model
          1. Considerations
        12. Summary
      2. 13. Determining Individual User Data and Variances
        1. Objectives
        2. Overview
        3. How to Use This Chapter
        4. User Delays
        5. Consequences of Improperly Modeling User Delays
          1. Step 1—Determine User Delays
          2. Step 2—Apply Delay Ranges
          3. Step 3—Apply Distributions
            1. Linear or Uniform Distribution
            2. Normal Distribution
            3. Negative Exponential Distribution
            4. Double Hump Normal Distribution
        6. Determining Individual User Data
          1. Considerations
        7. User Abandonment
          1. Considerations
        8. Summary
    10. VI. Execute Tests
      1. 14. Test Execution
        1. Objectives
        2. Overview
        3. How to Use this Chapter
        4. Approach for Test Execution
        5. Validate the Test Environment
          1. Additional Considerations
        6. Validate Tests
          1. Dynamic Data
          2. Test Validation
          3. Additional Considerations
          4. Questions to Ask
        7. Run Tests
          1. Keys to Efficiently and Effectively Running Tests
          2. Additional Considerations
          3. Questions to Ask
        8. Baseline and Benchmark
          1. Baselines
          2. Benchmarking
        9. Archive Tests
          1. Additional Considerations
        10. Summary
    11. VII. Analyze Results and Report
      1. 15. Key Mathematic Principles for Performance Testers
        1. Objectives
        2. Overview
        3. How to Use This Chapter
        4. Exemplar Data Sets
          1. Data Sets Summary
          2. Data Set A
          3. Data Set B
          4. Data Set C
        5. Averages
        6. Percentiles
        7. Medians
        8. Normal Values
        9. Standard Deviations
        10. Uniform Distributions
        11. Normal Distributions
        12. Statistical Significance
        13. Statistical Equivalence
        14. Statistical Outliers
        15. Confidence Intervals
        16. Summary
      2. 16. Performance Test Reporting Fundamentals
        1. Objectives
        2. Overview
        3. How to Use This Chapter
        4. Principles of Effective Reporting
          1. Report Early, Report Often
          2. Report Visually
          3. Report Intuitively
          4. Use the Right Statistics
          5. Consolidate Data Correctly
            1. Additional Considerations
          6. Summarize Data Effectively
            1. Additional Considerations
          7. Customize Reports for the Intended Audience
          8. Use Concise Verbal Summaries
          9. Make the Data Available
        5. Frequently Reported Performance Data
          1. End-user Response Times
            1. Exemplar1
            2. Exemplar2
            3. Considerations
          2. Resource Utilizations
            1. Exemplar for Stakeholders
            2. Exemplar for Technical Team Members
            3. Additional Considerations
          3. Volumes, Capacities, and Rates
            1. Exemplar
            2. Additional Considerations
          4. Component Response Times
            1. Exemplar
            2. Additional Considerations
          5. Trends
            1. Exemplar
            2. Additional Considerations
        6. Questions to Be Answered By Reporting
          1. All Roles
          2. Executive Stakeholders
          3. Project-Level Managers
          4. Technical Team Members
        7. Types of Results Sharing
          1. Raw Data Display
          2. Technical Reports
          3. Stakeholder Reports
        8. Creating a Technical Report
          1. Exemplar Results Graph
          2. Exemplar Tables for Single-Instance Measurements
          3. Exemplar Workload Model Graphic
          4. Exemplar Test Environment Graphic
          5. Exemplar Summary Statement
          6. Exemplar References Section
        9. Creating a Stakeholder Report
          1. Exemplar Criteria Statement
          2. Exemplar Results Graph
          3. Exemplar Tables for Single-Instance Measurements
          4. Exemplar Criteria-Based Results Summary
          5. Exemplar Workload Model Graphic
          6. Exemplar Test Environment Graphic
          7. Exemplar Observations and Recommendations Statement
          8. Exemplar References Section
        10. Summary
    12. VIII. Performance Testing Techniques
      1. 17. Load-Testing Web Applications
        1. Objectives
        2. Overview
        3. How to Use This Chapter
        4. Input
        5. Output
        6. Approach for Load Testing
        7. Step 1—Identify Performance Acceptance Criteria
        8. Step 2—Identify Key Scenarios
        9. Step 3—Create a Workload Model
        10. Step 4—Identify Target Load Levels
        11. Step 5—Identify Metrics
        12. Step 6—Design Specific Tests
        13. Step 7—Run Tests
        14. Step 8—Analyze the Results
        15. Summary
      2. 18. Stress Testing Web Applications
        1. Objectives
        2. Overview
          1. Examples of Stress Conditions
          2. Examples of Stress-Related Symptoms
        3. How to Use This Chapter
        4. Input
        5. Output
        6. Approach for Stress Testing
        7. Step 1—Identify Test Objectives
        8. Step 2—Identify Key Scenario(s)
        9. Step 3—Identify the Workload
        10. Step 4—Identify Metrics
        11. Step 5—Create Test Cases
          1. Test 1—Place Order Scenario
        12. Step 6—Simulate Load
        13. Step 7—Analyze Results
        14. Usage Scenarios for Stress Testing
        15. Exploratory Stress Testing
        16. Summary
    13. Index
    14. About the Author
    15. SPECIAL OFFER: Upgrade this ebook with O’Reilly