The dangers of benchmarking
It is sometimes all too easy to focus too much on the results of a particular benchmark and let this "tunnel vision" take over all performance work.
It is very important that a wide selection of standard industry benchmarks exists, as they will be used by anyone from hardware vendors to undergraduate researchers, for ultimately increasing performance for various runtimes and applications. Conclusions drawn, given a benchmark setup, can have wide-ranging implications.
One danger is, of course, if a mainstream benchmark gets too well adopted. Classic examples here are the SPECjvm benchmark suite and later the SPECjbb benchmark.
To put it bluntly, if a graduate student can run a benchmark from his workstation with a simple ...
Get Oracle JRockit now with the O’Reilly learning platform.
O’Reilly members experience books, live events, courses curated by job role, and more from O’Reilly and nearly 200 top publishers.
