DATA WAREHOUSING FUNDAMENTALS: A Comprehensive Guide for IT Professionals

13.3. DATA QUALITY CHALLENGES

There is an interesting but strange aspect of the whole data cleansing initiative for the data warehouse. We are striving toward having clean data in the data warehouse. We want to ascertain the extent of the pollution. Based on the condition of the data, we plan data cleansing activities. What is strange about this whole set of circumstances is that the pollution of data occurs outside the data warehouse. As part of the data warehouse project team, you are taking measures to eliminate the corruption that arises in a place outside your control.

All data warehouses need historical data. A substantial part of the historical data comes from antiquated legacy systems. Frequently, the end-users use the historical data in the data warehouse for strategic decision making without knowing exactly what the data really means. In most cases, detailed metadata hardly exists for the old legacy systems. You are expected to fix the data pollution problems that emanate from the old operational systems without the assistance of adequate information about the data there.

13.3.1. Sources of Data Pollution

In order to come up with a good strategy for cleansing the data, it will be worthwhile to review a list of common sources of data pollution. Why does data get corrupted in the source systems? Study the following list of data pollution sources against the background of what data quality really is.

System conversions. Trace the evolution of order processing in any company. ...