O'Reilly logo
  • Jinyong Yang thinks this is interesting:

import org.apache.spark.SparkContext import org.apache.spark.SparkContext._ import org.apache.spark.SparkFiles; import au.com.bytecode.opencsv.CSVReader import java.io.StringReader object LoadCsvWithCountersExample { def main(args: Array[String]) { if (args.length != 2) { System.err.println("Usage: LoadCsvExample <master> <inputfile>") System.exit(1) } val master = args(0) val inputFile = args(1) val sc = new SparkContext(master, "Load CSV With Counters Example", System.getenv("SPARK_HOME"), Seq(System.getenv("JARS"))) val invalidLineCounter = sc.accumulator(0) val invalidNumericLineCounter = sc.accumulator(0) sc.addFile(inputFile) val inFile = sc.text...

From

Cover of Fast Data Processing with Spark - Second Edition

Note

q q