[Phoronix] Ubuntu 9.04 vs. Fedora 11 Performance

[Gilboa Davara]

On Fri, 2009-06-12 at 19:05 -0700, Adam Williamson wrote:
> On Sat, 2009-06-13 at 04:33 +0300, Gilboa Davara wrote:
> 
> > You don't like Phoronix' benchmark? Why? What should they have done
> > differently? Have you ever contacted Phoronix (E.g. Using their forums)
> > and tried to resolve these issues? Did they refuse?
> 
> They should use distribution-compiled binaries - or at least record and
> explain the fact that they don't, and check whether there are
> significant differences between the compiled binaries they use on each
> distro.

Up until 30 minutes ago, I was unaware of the fact that they use
test-suite compiled binaries.
Though I'd imagine that in Phoronix' view, having (far) different
compile options in the distribution supplied binaries might generate
invalid results. (Due to missing features, non-standard optimization,
etc)

Of-cause, the best solution would have been to test -both- versions -
read: Phoronix-compiled binaries next to distribution supplied binaries
This should generate far cleaner (and far more interesting) results.

> And when they observe anomalies, they should try and do some
> kind of research to confirm the result and figure out _why_, not just
> note the fact of the anomaly.

I fear that you're expecting far too much from popular website.
I'd rather see an open dialog between Phoronix and the different
distributions an in effort to gain usable test-data out of their
benchmarks.

> Multiple people have pointed this out to them in the past, but they
> haven't really made a concerted effort to change.

Has anyone attempted to start an open dialog with them using their
forums? [1].
At least the past, Micheal (Phoronix founder) was very responsive.

> I have a kind of
> love/hate relationship with Phoronix - they're a popular site and do
> some really good stuff, but they also make a lot of frustratingly lazy
> mistakes and shorthand contractions in many articles.

I believe we should praise Phoronix for their work, even if we do not
agree with their methodology.
As I said in my previous post, Phoronix completely changed the landscape
of OSS websites and OSS benchmarking.

- Gilboa
[1] http://www.phoronix.com/forums/forumdisplay.php?f=49