ETL patterns is another area of computing that lends itself very well to serverless platforms. At a high level, ETL jobs comprise the following three steps:
- Extracting data from one data source
- Transforming that data appropriately
- Loading the processed data into another data source
Often used in analytics and/or data warehousing, ETL jobs are hard to escape. Since this problem is again ephemeral and because users would probably like their ETL jobs to execute as quickly as possible, serverless systems are a great platform in this problem space. While serverless computation is typically short-lived, we will see how ETL processes can be designed to be long-running in order to work through large amounts of data.
In the fan-out ...
