We read a text file A Tale of Two Cities by Charles Dickens from http://www.gutenberg.org/ into an RDD and then proceed to tokenize the words by using whitespace as the separator in a lambda expression using .split() and .flatmap() of RDD itself. We then proceed to use the .count() method of RDDs to output the total number of words. While this is simple, you have to bear in mind that the operation takes place using the distributed parallel framework of Spark with only a couple of lines.
How it works...
Get Apache Spark 2.x Machine Learning Cookbook now with the O’Reilly learning platform.
O’Reilly members experience books, live events, courses curated by job role, and more from O’Reilly and nearly 200 top publishers.
