Fast Data Processing with Spark 2 - Third Edition

iPython

Finally, let's fire up iPython and interact with the SparkContext object. As mentioned in Chapter 3, Building and Running a Spark Application, refer to the iPython site (http://jupyter.readthedocs.org/en/latest/install.html) for installing the Jupyter and iPython system.

First, change the directory to fdps-v3, where you would have downloaded the code and data for this book:

cd ~/fdps-v3

The command to start iPython is as follows:

PYSPARK_DRIVER_PYTHON=ipython PYSPARK_DRIVER_PYTHON_OPTS="notebook" ~/Downloads/spark-2.0.0/bin/pyspark

The iPython notebook will be launched in the web browser, as shown in the following screenshot, and you will see a list of iPython notebooks:

Click on the 000-PreFlightCheck.ipynb notebook:

Run the first cell ...