Building a Scalable Data Warehouse with Data Vault 2.0

Preface

When I was asked by the Department of Defense to build a scalable data warehouse, I was confronted with a problem. Back then, before the term Big Data was invented, there was no approach for building such systems – systems that could accommodate large data sets, delivered at high frequencies, and in multiple structures.

I started intensive research to come up with a viable solution for this challenge. The analysis was based on patterns from nature, because I expected that a partial solution would already exist somewhere. Over more than 10 years, from 1990 to early 2000, I tested the applicability of these natural patterns in data warehousing. By doing so, I reduced the initial list of 50 potential entities down to three. These remaining ...