Principles of Big Data

Chapter 10

Special Considerations in Big Data Analysis

Outline

Background

Theory in Search of Data

Data in Search of a Theory

Overfitting

Bigness Bias

Too Much Data

Fixing Data

Data Subsets in Big Data: Neither Additive nor Transitive

Additional Big Data Pitfalls

Those who ignore Statistics are condemned to reinvent it.

Brad Efron

Background

Big Data statistics are plagued by several intrinsic flaws. When the amount of data is sufficiently large, you can find almost anything you seek lurking somewhere within; the found observations may have statistical significance without having any significance in reality. Also, whenever you select a subset of data from an enormous collection, you may have no way of knowing the relevance of the data that ...