Sparkling Pandas
Sparkling Pandas aims to make use of the distributed computing power of PySpark to scale the data analysis with Pandas. This is built on Spark's DataFrame and provides API similar to Pandas. This recipe shows how to use Sparkling Pandas.
Getting ready
To step through this recipe, you will need a running Spark Cluster either in pseudo distributed mode or in one of the distributed modes, that is, standalone, YARN, or Mesos. Also, have Python and IPython installed on the Linux machine, that is, Ubuntu 14.04.
How to do it…
- Install
blas
,scipy
, andsparklingpandas
as follows:apt-get install libblas-dev liblapack-dev libatlas-base-dev gfortran pip install scipy pip install sparklingpandas
- Now import the package as follows:
import sparklingpandas ...
Get Apache Spark for Data Science Cookbook now with the O’Reilly learning platform.
O’Reilly members experience books, live events, courses curated by job role, and more from O’Reilly and nearly 200 top publishers.