Spark for Data Science

References

DataFrame reference on the SQL programming guide of Apache Spark official resource:

• https://spark.apache.org/docs/latest/sql-programming-guide.html#creating-dataframes

Databricks: Introducing DataFrames in Apache Spark for Large Scale Data Science:

• https://databricks.com/blog/2015/02/17/introducing-dataframes-in-spark-for-large-scale-data-science.html

Databricks: From Pandas to Apache Spark's DataFrame:

• https://databricks.com/blog/2015/08/12/from-pandas-to-apache-sparks-dataframe.html

API reference guide on Scala for Spark DataFrames:

• https://spark.apache.org/docs/1.5.1/api/java/org/apache/spark/sql/DataFrame.html

A Cloudera blogpost on Parquet - an efficient general-purpose columnar file format for Apache Hadoop:

• http://blog.cloudera.com/blog/2013/03/introducing-parquet-columnar-storage-for-apache-hadoop/ ...