Apache Flume: Distributed Log Collection for Hadoop

Compliance and data expiry

Remember that the data your company is collecting on your customers may contain sensitive information. You may be bound by other regulatory limitations on access to data such as Payment Card Industry (PCI—http://en.wikipedia.org/wiki/PCI_DSS) or Sarbanes Oxley (SOX—http://en.wikipedia.org/wiki/Sarbanes%E2%80%93Oxley_Act). If you aren't properly handling access to this data in your cluster, the government will lean on you or worse, you won't have customers anymore if they feel you aren't protecting their rights and identities. Consider scrambling, trimming, or obfuscating your data of personal information. Chances are the business insight you are looking for falls more into the category of "how many people who search ...