Video description
Translating a Spark application from running in a local environment to running on a production cluster in the cloud requires several critical steps, including publishing artifacts, installing dependencies, and defining the steps in a pipeline. This video is a hands-on guide through the process of deploying your Spark ML pipelines in production. You’ll learn how to create a pipeline that supports model reproducibility—making your machine learning models more reliable—and how to update your pipeline incrementally as the underlying data change. Learners should have basic familiarity with the following: Scala or Python; Hadoop, Spark, or Pandas; SBT or Maven; Amazon Web Services such as S3, EMR, and EC2; Bash, Docker, and REST.
- Understand how various cloud ecosystem components interact (i.e., Amazon S3, EMR, EC2, and so on)
- Learn how to architect the components of a cloud ecosystem into an end-to-end model pipeline
- Explore the capabilities and limitations of Spark in building an end-to-end model pipeline
- Learn to write, publish, deploy, and schedule an ETL process using Spark on AWS using EMR
- Understand how to create a pipeline that supports model reproducibility and reliability
Jason Slepicka is a senior data engineer with Los Angeles based DataScience, where he builds pipelines and data science platform infrastructure. He has a decade of experience integrating data to support efforts like fighting human trafficking for DARPA, exploring the evolution of evolvability in yeast, and tracking intruders in computer networks. Jason has both a Bachelor's and Master’s in Computer Science from the University of Arizona and is working on his PhD in Computer Science at the University of Southern California Information Sciences Institute.
Table of contents
Product information
- Title: Deploying Spark ML Pipelines in Production on AWS
- Author(s):
- Release date: December 2017
- Publisher(s): O'Reilly Media, Inc.
- ISBN: 9781491988862
You might also like
video
MLOps for Containers with AWS and GCP
Learn to build MLOps predictions with containers * AWS App Runner * AWS ECR (EC2 Container …
video
Learn to Containerize Algorithms Deploy to AWS with App Runner
Learn to build greedy algorithms, then convert them to microservices and finally containerized them and deploy …
video
Container continuous delivery for AWS with App Runner
Learn to perform Continuous Delivery of Container Based Projects using ECR, Cloud9 and App Runner. 00:00 …
video
Cloud-Native C# WEB SERVICES with AWS, CLOUD9, ECR and APP RUNNER
Learn to use pure Cloud-Native tools to developer AWS Microservices in .NET 6.0. These services include: …