Solving Operational Business Intelligence with InfoSphere Warehouse Advanced Edition

268 Solving Operational Business Intelligence with InfoSphere Warehouse Advanced Edition

7.1 Understand your data latency requirements

The initial planning and architecting steps of a data ingest project must quickly

identify the key characteristics in the ingest process for this project.

When that is done, you can then proceed to work on details (or seek guidance)

applicable to your situation. This section covers:

򐂰 Quantifying your service level objectives: quantifying the key elements of the

business requirements for the ingest process.

򐂰 Analyzing your ETL scenario: identifying what general parameters of the data

ingest process you are dealing with.

򐂰 Calculating the ingest rate: help determine how challenging the performance

requirements are.

7.1.1 Quantify your service level objectives

This section describes how to quantify key aspects of the business requirements

for your ETL application design. These requirements are expressed as service

level objectives (SLOs).

To calculate SLOs, establish the ingest schedule and also the volume of data

presented and the processing window available during each ingest schedule.

Use these figures to calculate the data ingest rate that your ETL application must

contain. Collectively, these figures comprise your service level objectives for the

ETL application. All design and development decisions must be anchored around

meeting these objectives in concert with all other workloads.

A data warehouse often has multiple SLOs for different target tables or data

sources. Defining SLOs requires quantifying the following elements:

򐂰 Data latency

Data latency refers to the gap, in time, between when data enters source

systems and when it must be available for query in the data warehouse.

Typical values are: “start of next business day,” “30 minutes,” or “5 minutes”.

Note: The term ETL is used to describe an application used to ingest data into

a data warehouse. The SQL Warehousing (SQW) tools discussed in this

chapter are more closely aligned with an ELT process where data is extracted

from source, loaded into a staging area of the database, and transformed and

moved to the data warehouse tables.