Installing GraphFrames
If you are running your job from a Spark CLI (for example, spark-shell, pyspark, spark-sql, spark-submit), you can use the –-packages command, which will extract, compile, and execute the necessary code for you to use the GraphFrames package.
For example, to use the latest GraphFrames package (version 0.3) with Spark 2.0 and Scala 2.11 with spark-shell, the command is:
> $SPARK_HOME/bin/spark-shell --packages graphframes:graphframes:0.3.0-spark2.0-s_2.11
If you are using a notebook service, you may need to install the package first. For example, the following section shows the steps to install the GraphFrames library within the free Databricks Community Edition (http://databricks.com/try-databricks).
Creating a library
Within ...
Get Learning PySpark now with the O’Reilly learning platform.
O’Reilly members experience books, live events, courses curated by job role, and more from O’Reilly and nearly 200 top publishers.
