O'Reilly logo

Agile Data Science 2.0 by Russell Jurney

Stay ahead with the world's most comprehensive technology and business learning platform.

With Safari, you learn the way you learn best. Get unlimited access to videos, live online training, learning paths, books, tutorials, and more.

Start Free Trial

No credit card required

Appendix A. Appendix A: Manual Installation

In this chapter, we cover the details of installing the tools for the stack used in this book.

Installing Hadoop

You can download the latest Hadoop from the Apache Hadoop downloads page at http://hadoop.apache.org/releases.html. At the time of writing, the latest Hadoop release is 2.6.4, and I downloaded hadoop 2.6.4 from http://apache.osuosl.org/hadoop/common/hadoop-2.6.4/hadoop-2.6.4.tar.gz.

A recipe for a headless install of Hadoop is available at manual_install.sh. In addition to downloading and unpackaging hadoop, we also need to setup our Hadoop environment variables: HADOOP_HOME, HADOOP_CLASSPATH, HADOOP_CONF_DIR and we need to put hadoop’s executables in our PATH. First lets setup a PROJECT_HOME variable to help find the right paths. You will need to set this yourself by editing your .bash_profile file.

export PROJECT_HOME=/Users/rjurney/Software/Agile_Data_Code_2

Now we can setup our environment directly.

# May need to update this link... see http://hadoop.apache.org/releases.html
curl -Lko /tmp/hadoop-2.7.3.tar.gz http://apache.osuosl.org/hadoop/common/hadoop-2.7.3/hadoop-2.7.3.tar.gz

mkdir hadoop
tar -xvf /tmp/hadoop-2.7.3.tar.gz -C hadoop --strip-components=1
echo '# Hadoop environment setup' >> ~/.bash_profile
export HADOOP_HOME=$PROJECT_HOME/hadoop
echo 'export HADOOP_HOME=$PROJECT_HOME/hadoop' >> ~/.bash_profile
export PATH=$PATH:$HADOOP_HOME/bin
echo 'export PATH=$PATH:$HADOOP_HOME/bin' >> ~/.bash_profile
export HADOOP_CLASSPATH ...

With Safari, you learn the way you learn best. Get unlimited access to videos, live online training, learning paths, books, interactive tutorials, and more.

Start Free Trial

No credit card required