Modern Big Data Processing with Hadoop

Configuration of YARN

You can perform the following steps for the configuration of YARN:

1. Start Hadoop NameNode, secondary NameNode, and DataNode
2. Alter yarn-env.sh.

Find corresponding XML files based on your Hadoop installation.

3. Add the following under the definition of YARN_CONF_DIR:

export HADOOP_CONF_DIR="${HADOOP_CONF_DIR:-$YARN_HOME/etc/hadoop}"
export HADOOP_COMMON_HOME="${HADOOP_COMMON_HOME:-$YARN_HOME}"
export HADOOP_HDFS_HOME="${HADOOP_HDFS_HOME:-$YARN_HOME}"  

4. Alter yarn-site.xml:

<?xml version="1.0"?> <configuration> <property> <name>yarn.nodemanager.aux-services</name> <value>mapreduce.shuffle</value> </property> <property> <name>yarn.nodemanager.aux-services.mapreduce.shuffle.class</name> <value>org.apache.hadoop.mapred.ShuffleHandler</value> ...