Understanding the future of optimization – project Tungsten
Project Tungsten, starting with Spark Version 1.4, is the initiative to bring Spark closer to bare metal. The goal of this project is to substantially improve the memory and CPU efficiency of the Spark applications and push the limits of underlying hardware.
In distributed systems, conventional wisdom has been to always optimize network I/O as that has been the most scarce and bottlenecked resource. This trend has changed in the last few years. Network bandwidth, in the last 5 years, has changed from 1 gigabit per second to 10 gigabit per second.
On similar lines, the disk bandwidth has increased from 50 MB/s to 500 MB/s and SSDs are being deployed more and more. CPU clock speed, on the ...
Get Spark Cookbook now with the O’Reilly learning platform.
O’Reilly members experience books, live events, courses curated by job role, and more from O’Reilly and nearly 200 top publishers.
