Data exploration
Now that we have a clean data frame with relevant data and the initial hypothesis, it is time to really explore what we have. The DataFrames abstraction provides functions such as group by out of the box for you to look around. You may register the cleaned data frame as a table and run the time-tested SQL statements to do just the same.
This is also the time to plot a few graphs. This phase of visualization is the exploratory analysis mentioned in the data visualization chapter. The objectives of this exploration are greatly influenced by the initial information you garner from the business stakeholders and the hypothesis. In other words, your discussions with the stakeholders help you know what to look for.
There are some general ...
Get Spark for Data Science now with the O’Reilly learning platform.
O’Reilly members experience books, live events, courses curated by job role, and more from O’Reilly and nearly 200 top publishers.
