People write toy Java benchmarks all the time. Nearly always they “get it wrong” – wrong in the sense that the code they write doesn’t measure what they think it does. Oh, it measures something all right – just not what they want.
This session presents some common benchmarking pitfalls, demonstrating pieces of real, bad (and usually really bad) benchmarks such as the following: SpecJVM98 209_db isn’t a DB test; it’s a bad string-sort test and indirectly a measure of the size of your TLBs and caches. SpecJAppServer2004 is a test of your DB and network speed, not your JVM. SpecJBB2000 isn’t a middleware test; it’s a perfect young-gen-only garbage collection test. The session goes through some of the steps any programmer would go through to make a canned program run fast – that is, it shows you how benchmarks get “spamed.” The session is for any programmer who has tried to benchmark anything.