Persistent data storage layer

The persistent data storage layer is responsible for consuming and persisting the raw source data provided by the ingestion layer. Little or no transformation of the raw source data takes place before it is persisted to ensure that the raw data remains in its original format. A data lake is a class of persistent data store that is often implemented to store raw data in this manner. Example technologies used to implement the persistent data storage layer include the following:

  • Traditional network-based stores, such as Storage Area Networks (SAN) and Network Attached Storage (NAS)
  • Open source technologies, such as the HDFS
  • Cloud-based technologies, such as AWS S3 and Azure BLOBs

Get Machine Learning with Apache Spark Quick Start Guide now with the O’Reilly learning platform.

O’Reilly members experience books, live events, courses curated by job role, and more from O’Reilly and nearly 200 top publishers.