Serverless Design Patterns and Best Practices

Time limits

The final limit to keep in mind is the execution limit. Even if your MapReduce implementation can stay within the storage and memory limits of your FaaS provider, you will still have to ensure your functions complete their work within a given time limit. As of this writing, AWS Lambda functions have an upper limit of 300 seconds. If any part of your MapReduce system takes longer than 300 seconds, you're out of luck and will have to find a workaround. With mappers, it's relatively simple to break the work into smaller pieces and execute more concurrent mappers. However, when the reducer runs, it must load all of the mapper data to compile it down to the final result set. If this takes longer than 300 seconds, it will be impossible ...