Spark submit

We are now ready to run our image recognition application! Since it is a Spark application, we can execute it via spark-submit on the Linux command line. To do this, navigate to the directory where we installed Apache Spark (see Chapter 2, Setting Up a Local Development Environment). Then, we can execute the spark-submit program by passing it the following command-line arguments:

  • --master: The Spark Master URL.
  • --packages: The third-party libraries and dependencies required for the Spark application to work. In our case, our image-recognition application is dependent on the availability of the sparkdl third-party library.
  • --py-files: Since our image-recognition application is a PySpark application, we pass the filesystem paths ...

Get Machine Learning with Apache Spark Quick Start Guide now with the O’Reilly learning platform.

O’Reilly members experience books, live events, courses curated by job role, and more from O’Reilly and nearly 200 top publishers.