In this chapter we will introduce the schema for the rest of the book: the Data Value Pyramid. Throughout the rest of our lessons, we will use the Data Value Pyramid to iteratively build value from very simple records up to interactive predictions. We begin with theory, then dive into practice in the next chapter using the framework we previously introduced.
Building Agile Big Data products means staging an environment where reproducible insights occur, are reinforced, and are extended up the value stack. It starts simply with displaying records. It ends with driving actions that create value and capture some of it. Along the way is a voyage of discovery.
This voyage has structure. It is called the data-value stack.
The data value stack mirrors Maslow’s hierarchy of needs, in the sense that lower levels must precede higher levels. The higher levels like predictions depend on the lower levels like reports, so we can’t skip steps. If we do so, we will lack sufficient structure and understanding of our data to easily build features and value at the higher levels.
The data value stack begins with the simple display of records, where the focus is on connecting or ‘plumbing’ our data pipeline all the way through from the raw data to the users’ screen. We then move on to charts, where ...