Fast Data Processing with Spark 2 - Third Edition

Building your Spark job with something else

If neither sbt nor Maven suits your needs, you may decide to use another build system. Thankfully, Spark supports building a fat JAR file with all its dependencies, which makes it easy to include in the build system of your choice. Simply, run the sbt/sbt assembly in the Spark directory and copy the resulting assembly JAR file at core/target/spark-core-assembly-1.5.2.jar to your build dependencies, and you are good to go. It is more common to use the spark-assembly-1.5.2-hadoop2.6.0.jar file. These files exist in $SPARK_HOME$/lib (if users use a prebuilt version) or in $SPARK_HOME$/ assembly/target/scala-2.10/ (if users build the source code with Maven or sbt).

Tip

No matter what your build system is, ...