R: Recipes for Analysis, Visualization and Machine Learning

Operating HDFS with rhdfs

The rhdfs package is an interface between Hadoop and R, which can call an HDFS API in the backend to operate HDFS. As a result, you can easily operate HDFS from the R console through the use of the rhdfs package. In the following recipe, we will demonstrate how to use the rhdfs function to manipulate HDFS.

Getting ready

To proceed with this recipe, you need to have completed the previous recipe by installing rhdfs into R, and validate that you can initial HDFS via the hdfs.init function.

How to do it...

Perform the following steps to operate files stored on HDFS:

1. Initialize the rhdfs package:

   > Sys.setenv(HADOOP_CMD="/usr/bin/hadoop")
   > Sys.setenv(HADOOP_STREAMING="/usr/lib/hadoop-mapreduce/hadoop-streaming-2.5.0-cdh5.2.0.jar") ...