DW 2.0: The Architecture for the Next Generation of Data Warehousing

DW 2.0: The Architecture for the Next Generation of Data Warehousing

by W.H. Inmon, Derek Strauss, Genia Neushloss
Released July 2010
Publisher(s): Morgan Kaufmann
ISBN: 9780080558332

Read it now on the O’Reilly learning platform with a 10-day free trial.

O’Reilly members get unlimited access to books, live events, courses curated by job role, and more from O’Reilly and nearly 200 top publishers.

Start your free trial

Book description

DW 2.0: The Architecture for the Next Generation of Data Warehousing is the first book on the new generation of data warehouse architecture, DW 2.0, by the father of the data warehouse. The book describes the future of data warehousing that is technologically possible today, at both an architectural level and technology level.

The perspective of the book is from the top down: looking at the overall architecture and then delving into the issues underlying the components. This allows people who are building or using a data warehouse to see what lies ahead and determine what new technology to buy, how to plan extensions to the data warehouse, what can be salvaged from the current system, and how to justify the expense at the most practical level. This book gives experienced data warehouse professionals everything they need in order to implement the new generation DW 2.0.

It is designed for professionals in the IT organization, including data architects, DBAs, systems design and development professionals, as well as data warehouse and knowledge management professionals.

  • First book on the new generation of data warehouse architecture, DW 2.0
  • Written by the "father of the data warehouse", Bill Inmon, a columnist and newsletter editor of The Bill Inmon Channel on the Business Intelligence Network
  • Long overdue comprehensive coverage of the implementation of technology and tools that enable the new generation of the DW: metadata, temporal data, ETL, unstructured data, and data quality control

Table of contents

  1. Cover
  2. Title Page
  3. Copyright
  4. Dedication
  5. Preface
  6. Acknowledgments
  7. About the Authors
  8. Table of Contents
  9. Chapter 1: A brief history of data warehousing and first-generation data warehouses
    1. Data Base Management Systems
    2. Online Applications
    3. Personal Computers and 4GL Technology
    4. The Spider Web Environment
    5. Evolution from the Business Perspective
    6. The Data Ware House Environment
    7. What Is a Data Warehouse?
    8. Integrating Data—a Painful Experience
    9. Volumes of Data
    10. A Different Development Approach
    11. Evolution to the DW 2.0 Environment
    12. The Business Impact of the Data Warehouse
    13. Various Components of the Data Warehouse Environment
    14. The Evolution of Data Warehousing from the Business Perspective
    15. Other Notions About a Data Warehouse
    16. The Active Data Ware House
    17. The Federated Data Warehouse Approach
    18. The Star Schema Approach
    19. The Data Mart Data Warehouse
    20. Building a “Real” Data Warehouse
    21. Summary
  10. Chapter 2: An introduction to DW 2.0
    1. DW 2.0—a New Paradigm
    2. DW 2.0—from the Business Perspective
    3. The Life Cycle of Data
    4. Reasons for the Different Sectors
    5. Metadata
    6. Access of Data
    7. Structured Data/Unstructured Data
    8. Textual Analytics
    9. Blather
    10. The Issue of Terminology
    11. Specific Text/General Text
    12. Metadata—a Major Component
    13. Local Metadata
    14. A Foundation of Technology
    15. Changing Business Requirements
    16. The Flow of Data Within DW 2.0
    17. Volumes of Data
    18. Useful Applications
    19. DW 2.0 and Referential Integrity
    20. Reporting In DW 2.0
    21. Summary
  11. Chapter 3: DW 2.0 components—about the different sectors
    1. The Interactive Sector
    2. The Integrated Sector
    3. The Near Line Sector
    4. The Archival Sector
    5. Unstructured Processing
    6. From the Business Perspective
    7. Summary
  12. Chapter 4: Metadata in DW 2.0
    1. Reusability of Data and Analysis
    2. Metadata In DW 2.0
    3. Active Repository/Passive Repository
    4. The Active Repository
    5. Enterprise Metadata
    6. Metadata and the System of Record
    7. Taxonomy
    8. Internal Taxonomies/External Taxonomies
    9. Metadata In the Archival Sector
    10. Maintaining Metadata
    11. Using Metadata—an Example
    12. From the End-User Perspective
    13. Summary
  13. Chapter 5: Fluidity of the DW 2.0 technology infrastructure
    1. The Technology Infrastructure
    2. Rapid Business Changes
    3. The Treadmill of Change
    4. Getting Off the Treadmill
    5. Reducing the Length of Time For It to Respond
    6. Semantically Temporal, Semantically Static Data
    7. Semantically Temporal Data
    8. Semantically Stable Data
    9. Mixing Semantically Stable and Unstable Data
    10. Separating Semantically Stable and Unstable Data
    11. Mitigating Business Change
    12. Creating Snapshots of Data
    13. A Historical Record
    14. Dividing Data
    15. From the End-User Perspective
    16. Summary
  14. Chapter 6: Methodology and approach for DW 2.0
    1. Spiral Methodology—a Summary of Key Features
    2. The Seven Streams Approach—an Overview
    3. Enterprise Reference Model Stream
    4. Enterprise Knowledge Coordination Stream
    5. Information Factory Development Stream
    6. Data Profiling and Mapping Stream
    7. Data Correction Stream (Previously Called the Data Cleansing Stream)
    8. Infrastructure Stream
    9. Total Information Quality Management Stream
    10. Summary
  15. Chapter 7: Statistical processing and DW 2.0
    1. Two Types of Transactions
    2. Using Statistical Analysis
    3. The Integrity of the Comparison
    4. Heuristic Analysis
    5. Freezing Data
    6. Exploration Processing
    7. The Frequency of Analysis
    8. The Exploration Facility
    9. The Sources for Exploration Processing
    10. Refreshing Exploration Data
    11. Project-Based Data
    12. Data Marts and the Exploration Facility
    13. A Backflow of Data
    14. Using Exploration Data Internally
    15. From the Perspective of the Business Analyst
    16. Summary
  16. Chapter 8: Data models and DW 2.0
    1. An Intellectual Road Map
    2. The Data Model and Business
    3. The Scope of Integration
    4. Making the Distinction Between Granular and Summarized Data
    5. Levels of the Data Model
    6. Data Models and the Interactive Sector
    7. The Corporate Data Model
    8. A Transformation of Models
    9. Data Models and Unstructured Data
    10. From the Perspective of the Business User
    11. Summary
  17. Chapter 9: Monitoring the DW 2.0 environment
    1. Monitoring the DW 2.0 Environment
    2. The Transaction Monitor
    3. Monitoring Data Quality
    4. A Data Warehouse Monitor
    5. The Transaction Monitor—Response Time
    6. Peak-Period Processing
    7. The Etl Data Quality Monitor
    8. The Data Warehouse Monitor
    9. Dormant Data
    10. From the Perspective of the Business User
    11. Summary
  18. Chapter 10: DW 2.0 and security
    1. Protecting Access to Data
    2. Encryption
    3. Drawbacks
    4. The Firewall
    5. Moving Data Offline
    6. Limiting Encryption
    7. A Direct Dump
    8. The Data Warehouse Monitor
    9. Sensing an Attack
    10. Security For Near Line Data
    11. From the Perspective of the Business User
    12. Summary
  19. Chapter 11: Time-variant data
    1. All Data In DW 2.0—Relative To Time
    2. Time Relativity In the Interactive Sector
    3. Data Relativity Elsewhere In DW 2.0
    4. Transactions In the Integrated Sector
    5. Discrete Data
    6. Continuous Time Span Data
    7. A Sequence of Records
    8. Nonoverlapping Records
    9. Beginning and Ending a Sequence of Records
    10. Continuity of Data
    11. Time-Collapsed Data
    12. Time Variance In the Archival Sector
    13. From the Perspective of the End User
    14. Summary
  20. Chapter 12: The flow of data in DW 2.0
    1. The Flow of Data Throughout the Architecture
    2. Entering the Interactive Sector
    3. The Role of ETL
    4. Data Flow Into the Integrated Sector
    5. Data Flow Into the Near Line Sector
    6. Data Flow Into the Archival Sector
    7. The Falling Probability of Data Access
    8. Exception-Based Flow of Data
    9. From the Perspective of the Business User
    10. Summary
  21. Chapter 13: ETL processing and DW 2.0
    1. Changing States of Data
    2. Where ETL Fits
    3. From Application Data to Corporate Data
    4. ETL In Online Mode
    5. ETL In Batch Mode
    6. Source and Target
    7. An ETL Mapping
    8. Changing States—an Example
    9. More Complex Transformations
    10. ETL and Throughput
    11. ETL and Metadata
    12. ETL and An Audit Trail
    13. ETL and Data Quality
    14. Creating Etl
    15. Code Creation or Parametrically Driven ETL
    16. ETL and Rejects
    17. Changed Data Capture
    18. ELT
    19. From the Perspective of the Business User
    20. Summary
  22. Chapter 14: DW 2.0 and the granularity manager
    1. The Granularity Manager
    2. Raising the Level of Granularity
    3. Filtering Data
    4. The Functions of the Granularity Manager
    5. Home-Grown Versus Third-Party Granularity Managers
    6. Parallelizing the Granularity Manager
    7. Metadata as a By-Product
    8. From the Perspective of the Business User
    9. Summary
  23. Chapter 15: DW 2.0 and performance
    1. Good Performance—a Cornerstone For DW 2.0
    2. Online Response Time
    3. Analytical Response Time
    4. The Flow of Data
    5. Queues
    6. Heuristic Processing
    7. Analytical Productivity and Response Time
    8. Many Facets to Performance
    9. Indexing
    10. Removing Dormant Data
    11. End-User Education
    12. Monitoring the Environment
    13. Capacity Planning
    14. Metadata
    15. Batch Parallelization
    16. Parallelization for Transaction Processing
    17. Workload Management
    18. Data Marts
    19. Exploration Facilities
    20. Separation of Transactions Into Classes
    21. Service Level Agreements
    22. Protecting the Interactive Sector
    23. Partitioning Data
    24. Choosing the Proper Hardware
    25. Separating Farmers and Explorers
    26. Physically Group Data Together
    27. Check Automatically Generated Code
    28. From the Perspective of the Business User
    29. Summary
  24. Chapter 16: Migration
    1. Houses and Cities
    2. Migration In a Perfect World
    3. The Perfect World Almost Never Happens
    4. Adding Components Incrementally
    5. Adding the Archival Sector
    6. Creating Enterprise Metadata
    7. Building the Metadata Infrastructure
    8. “Swallowing” Source Systems
    9. ETL as a Shock Absorber
    10. Migration to the Unstructured Environment
    11. From the Perspective of the Business User
    12. Summary
  25. Chapter 17: Cost justification and DW 2.0
    1. Is DW 2.0 Worth It?
    2. Macro-Level Justification
    3. A Micro-Level Cost Justification
    4. Company B Has DW 2.0
    5. Creating New Analysis
    6. Executing the Steps
    7. So How Much Does all of This Cost?
    8. Consider Company B
    9. Factoring the Cost of DW 2.0
    10. Reality of Information
    11. The Real Economics of DW 2.0
    12. The Time Value of Information
    13. The Value of Integration
    14. Historical Information
    15. First-Generation DW and DW 2.0—The Economics
    16. From the Perspective of the Business User
    17. Summary
  26. Chapter 18: Data quality in DW 2.0
    1. The DW 2.0 Data Quality Tool Set
    2. Data Profiling Tools and the Reverse-Engineered Data Model
    3. Data Model Types
    4. Data Profiling Inconsistencies Challenge Top-Down Modeling
    5. Summary
  27. Chapter 19: DW 2.0 and unstructured data
    1. DW 2.0 and Unstructured Data
    2. Reading Text
    3. Whereto Do Textual Analytical Processing
    4. Integrating Text
    5. Simple Editing
    6. Stop Words
    7. Synonym Replacement
    8. Synonym Concatenation
    9. Homographic Resolution
    10. Creating Themes
    11. External Glossaries/Taxonomies
    12. Stemming
    13. Alternate Spellings
    14. Text Across Languages
    15. Direct Searches
    16. Indirect Searches
    17. Terminology
    18. Semistructured Data/Value = Name Data
    19. The Technology Needed to Prepare the Data
    20. The Relational Data Base
    21. Structured/Unstructured Linkage
    22. From the Perspective of the Business User
    23. Summary
  28. Chapter 20: DW 2.0 and the system of record
    1. Other Systems of Record
    2. From the Perspective of the Business User
    3. Summary
  29. Chapter 21: Miscellaneous topics
    1. Data Marts
    2. The Convenience of a Data Mart
    3. Transforming Data Mart Data
    4. Monitoring DW 2.0
    5. Moving Data from One Data Mart to Another
    6. Bad Data
    7. A Balancing Entry
    8. Resetting a Value
    9. Making Corrections
    10. The Speed of Movement of Data
    11. Data Warehouse Utilities
    12. Summary
  30. Chapter 22: Processing in the DW 2.0 environment
    1. Summary
  31. Chapter 23: Administering the DW 2.0 environment
    1. The Data Model
    2. Architectural Administration
    3. Defining the Moment When an Archival Sector Will Be Needed
    4. Determining Whether the Near Line Sector Is Needed
    5. Metadata Administration
    6. Data Base Administration
    7. Stewardship
    8. Systems and Technology Administration
    9. Management Administration of the DW 2.0 Environment
    10. Prioritization and Prioritization Conflicts
    11. Budget
    12. Scheduling and Determination of Milestones
    13. Allocation of Resources
    14. Managing Consultants
    15. Summary
  32. Index
  33. Instructions for online access

Product information

  • Title: DW 2.0: The Architecture for the Next Generation of Data Warehousing
  • Author(s): W.H. Inmon, Derek Strauss, Genia Neushloss
  • Release date: July 2010
  • Publisher(s): Morgan Kaufmann
  • ISBN: 9780080558332