Summary
In this chapter, we walked you through the steps on how to submit applications written in Python to Spark from the command line. The selection of the spark-submit parameters has been discussed. We also showed you how you can package your Python code and submit it alongside your PySpark script. Furthermore, we showed you how you can track the execution of your job.
In addition, we also provided a quick overview of how to run Databricks notebooks using the Databricks Jobs feature. This feature simplifies the transition from development to production, allowing you to take your notebook and execute it as an end-to-end workflow.
This brings us to the end of this book. We hope you enjoyed the journey, and that the material contained herein will ...
Get Learning PySpark now with the O’Reilly learning platform.
O’Reilly members experience books, live events, courses curated by job role, and more from O’Reilly and nearly 200 top publishers.
