O'Reilly logo

Real-Time Big Data Analytics by Shilpi Saxena, Sumit Gupta

Stay ahead with the world's most comprehensive technology and business learning platform.

With Safari, you learn the way you learn best. Get unlimited access to videos, live online training, learning paths, books, tutorials, and more.

Start Free Trial

No credit card required

Converting RDDs to DataFrames

In this section, we will discuss the strategies exposed by Spark SQL for transforming existing RDDs into DataFrames.

In today's enterprise world, data analysis requires the usage of more than one tool or technology. There could be scenarios where we want the Spark batch to initially load and process the data for a few insights and at the same we also want Spark SQL to process the same data to get the rest of the insights. In these kinds of scenarios, data would be loaded only once, either by a Spark batch or Spark SQL, and then it will be further processed by other Spark extensions. We need to consider that loading the data twice will be a waste of memory and time.

In order to solve this problem, Spark SQL (DataFrames) ...

With Safari, you learn the way you learn best. Get unlimited access to videos, live online training, learning paths, books, interactive tutorials, and more.

Start Free Trial

No credit card required