On Benchmarketing

TPM is bragging about the SPECjbb2005 performance of the p7 system. Interestingly he is talking about the nonsensical benchmark again: Multiple JVMs per system with 32 bit. As i wrote before: When you want a cluster, buy a cluster and don't build one for an embarrassingly parallel problem in a box. I think, that the only relevant test today is 1 JVM/64-bit today. By the way: Those multitude of JVMs/32-bit doesn't use the real advantage of the big machines (large heaps for you application), however it's the only chance those SGI systems or extremly-NUMA systems can compete with larger SMP systems, because of the vastly reduced interconnect usage (which is quite useful to avoid the long arm of Amdahl's law)

This blog contains only two entries until today but both have a very interesting view to different parts of the x86-world: Scientias Blog.

Interesting article about new TPC benchmark suites. It´s good to see, that the media support Sun opinion, that TPC-C is absolutly useless this days. It should be subsituted by something more realistic sooner than later:

One problem with TPC-C was the ease with which certain servers could generate unrealistically high scores using hardware and software configurations that are highly improbable in the real world. For example, high TPC-C scores come from servers with colossal numbers of hard drives--6,548 in the case of IBM's top score.

and

Another problem hinged on the fact that TPC-C's test was easily distributed among relatively independent servers linked in a cluster. That gave the impression that a number of inexpensive machines were as good as a single multiprocessor behemoth, leading the consortium to list results separately for clustered and non-clustered systems.

I looked for a project into the SPEC website today and saw that the T2000 still leads the SPECweb2005 benchmark by roughly 4000 points. And this benchmark was made on the T2000 in November last year. Even the new Woodcrests stopped short of 10000 points and that were already fully blown 4 core systems (maximum for Woodcrest right now). Now keep in consideration that Niagara II already runs in our labs. Interesting times . Well ... Rolf, we should do the power meter gag again next cebit

Okay, very impressive number: 1.000.000 operation per second. But look at the bill of material: 24 Filerheads, not the small one, the NetApp 6040 is one of the bigbadass machines, 768 Gigabytes of Cache, 48 GB NVRAM, 1152 Harddisks. Does anybody uses really a environment like this? It´s seems to me like using an SF25K for fileserving.
This is the nature of benchmarketing. It´s not a matter of being realistic of realistic workloads, it´s a expression of the fine art of understanding and outsmart a benchmark to make a marketing statement.