What do you recommend for benchmarking HotSpot, or any virtual machine?
We like to use the SPEC JVM98 benchmark. We use it for tracking our own progress over time, and we use it for comparing ourselves to other virtual machines. The SPEC JVM98 benchmark was developed by a consortium of interested vendors under the auspices of the Standard Performance Evaluation Corporation (SPEC). It is the only industry-standard benchmark for Java platforms. The benchmark is a collection of kernels from several types of programs, most of them based on real applications. The benchmark seems to have a good mix of operations and realistic behaviors (method invocations, storage allocation and lifetimes, input and output). We find that the benchmark is predictive of the performance we see across a number of real applications. It comes with an easy-to-use harness that ensures that it is run the same way on all platforms, so fair comparisons can be made between platforms. The SPEC JVM98 benchmark is available from http://www.spec.org/osg/jvm98/. Other than that, we like benchmar
