O'Reilly logo

Learning Hadoop 2 by Garry Turkington, Gabriele Modena

Stay ahead with the world's most comprehensive technology and business learning platform.

With Safari, you learn the way you learn best. Get unlimited access to videos, live online training, learning paths, books, tutorials, and more.

Start Free Trial

No credit card required

Challenges of external data

When we rely on external data to drive our application, we are implicitly dependent on the quality and stability of that data. This is, of course, true for any data, but when the data is generated by an external source over which we do not have control, the risks are most likely higher. Regardless, when building what we expect to be reliable applications on top of such data feeds, and especially when our data volumes grow, we need to think about how to mitigate these risks.

Data validation

We use the general term data validation to refer to the act of ensuring that incoming data complies with our expectations and potentially applying normalization to modify it accordingly or to even delete malformed or corrupt input. ...

With Safari, you learn the way you learn best. Get unlimited access to videos, live online training, learning paths, books, interactive tutorials, and more.

Start Free Trial

No credit card required