Configuration can be applied to Spark in the following ways:
- Spark properties control application-level settings, including execution behavior, memory management, dynamic allocation, scheduling, and security, which can be defined in the following order of precedence:
- Via a Spark configuration programmatic object called SparkConf defined in your driver program
- Via command-line arguments passed to spark-submit or spark-shell
- Via default options set in conf/spark-defaults.conf
- Environmental variables control per-machine settings, such as the local IP address of the local worker node, and which can be defined in conf/spark-env.sh.
