|
|
Point 417 coming to a WU near you...or
don't shoot me, I'm just the new benchmark...
After some inertia on my part and a desire to maintain consistent benchmarking
between the various incarnations of SETI software the benchmark WU has been updated
to reflect current thought and requirements. To keep the TLC benchmarking
numbers relevant a new WU was needed that allowed better comparisons between
various systems. A very detailed and extensive
thread on the relevance of the current (now ex) 'fast' benchmark WU has been
developing. In response to this and other considerations the benchmark has been
changed to one with an angle range (AR) of 0.417 which as many of
you will know is a very common and 'typical' (dangerous word) AR for a work
unit. It also will take considerably longer to process as unlike the high angle
range WU's it does not skip a shedful of gaussian analysis. Also the unleashing
of v3.03 has seen completion times up by about 50% so you are going to be on the
end of a double whammy and perhaps less likely to benchmark your system.
Part of the reason for the timing of the change was to introduce it at the
beginning of a new SETI CLI/GUI version to avoid the use of the old benchmark on
the v3.03. Also according to the Berkeley Boys'n'Girls the upgrade will soon
(read weeks to months) become mandatory with the intention of killing off all
clients below v3.01 - so again this is a good time to implement any benchmarking
change. I think I understand some of your misgivings and mumblings of discontent
about this and I also expect that fewer of you will give your valuable time to
benchmarking. Thank you for what you have done so far and thanks in advance for
what you might do in the future. As always any thoughts and comments either to
me or better yet add them to the thread
which I've been following fairly closely. Any rants, flames, trollings or rude
words will be ignored (mainly) unless in close proximity to a completed
benchmark! In any case please send benchmark subs to Max
or RB
as the page is still down. In this transition period it would be a good
verification if you added the result.sah to your email. Any old
unmentioned benchmarks will be honoured at my discretion. Thanks.
On the good news front SETI
Spy has crept to 3.03 (delightful coincidence) and includes a debug feature
to help you calibrate completion times specifically for your system. Read the Version
History page for instructions. The debugger writes out information to a
'dbg' file every 2 minutes, allowing you to compare the actual time of
processing with the 'reported' % completed. At the end of the WU you will know
the precise time taken and can compare the actual % (a little arithmetic needed)
with the 'reported' %. You need to see it to really follow what I'm on about.
Have fun.
Max out.
Beyond the relevance of
benchmarks...
The submissions page is still down so keep sending to Max
or RB,
it is being worked on by RB
who is very busy (goes without saying almost). No sooner do I take time off and
complacently imagine that all is done and dusted with the v3 Client when the
Berkeley Bunch come up with v3.03. This is a much slower beast (50% or more by
some accounts) and I guess the next round of benchmarking is upon us! I have
been following the beta (3.01, 2 & 3) progress with the GUI and feared the
worst as I noticed completion times increase drastically. But if the problematic
connections to the WU servers ease and the basic stats are more accessible then
there is a definite trade off. Though the recent angle range glut of 11's and
12's have been flushed it would seem that with their present hardware set-up the
SETI@home team was getting
very stretched.
As a community we have turned this into a serious competitive enterprise but the
longer completion times do represent more science (I believe everything I'm
told, no conspiracies please) for the SETI aims. A date for making v3 obsolete
must be haunting administrators and home based farmers alike! Thoughts of
quitting altogether have permeated many threads and discussions and those that
baulk at having to upgrade hundreds (or even thousands) of clients have my
sympathies. This could be seen as a good test of crisis management with
non-essential software and the next couple of months could see quite a sort out
in the big player ranks.
Onto the numbers and a time that Roelof
gave me ages ago for his 486 that lumbered in at 307 hours (don't think the
minutes are really that relevant). He was hoping to top and tail the results
table but that goal seems to have eluded him. Cadeus
stumped up a 13:02 with a PenPro at 200MHz with Win98 while same hardware but
Win2000 for gordon
gave 12:05. I think there is more to that hour difference than OS. JoseM
has been a busy bunny by benching all 4 Linux clients on his 300MHz PII Katmai
with Red Hat 7. Both static clients (i386, 10:29 and i686, 10:28) came in
considerably slower than the dynamic i386 at 9:54 and i686 at 9:45. That's a
very useful difference and if you want to know more about these set-ups you
could try SETIStats4Linux
which considering the general lack of SETI info for Linux is a useful addition
to the fold. Big jump now to joe
and faithful Celeron in at 6:15 which probably has a little leeway for dipping
under the six but it's a good start! Anyone for Sun? Well an Ultra Sparc IIi
that benched a 6:08 for DW
at 400Mhz (doesn't matter if you have a 2MB cache if it's half clock speed).
Another Cadeus
time now of 5:44 on a naturally-aspirated PIII 600 which is seriously slow as
you should be well into the 4's. Already in the 4's is Chboss
with a Celery II (900/100) at 4:16 which is respectable but not eye-catching as
can be seen by comparison with tomwyatt.
His Celeron II (909/107) reached 3:47, that extra FSB is still helpful for a low
time. But before you start crowing remember there are CPU's of this ilk 30
minutes ahead of you! From almost 3 weeks back (sorry) a benchmark from Poof
for his PIII 800 running Red Hat 6.2. Using
WINE to emulate a Windows environment (now you know how the rest of us feel) and
Win Client produced a 3:42. Ross
is next up at 3:28 for a PII 900/150 (!), he sent me a screenshot of the subs
page with his numbers which merits a mention for initiative! In at 3:27 is Gyro77
and PIII at 944/118MHz. A while ago Action
benched a 700Mhz Xeon (1MB cache) and now has come up with the numbers for
smaller brother running at 600MHz of 3:25. As before this is running two
benchmark clients simultaneously, pleasant. To round off there's EvilOwl's
1GHz PIII stumping up 3:17 on Millennium.
Beyond has
submitted many times over the past ten days or so in an attempt to clear up some
nagging thoughts. He noticed that the differences in processing times with
various OS and hardware configurations didn't translate very accurately to what
he was seeing in his real world results. Others have mentioned this but no one
has pushed the comparisons quite so far as Beyond.
This naturally led him to the belief that the high angle range of the benchmark
WU was culpable for the majority of the difference. Others have pointed this out
before but not in quite such arobust style. He's used these times as the basis
of a very interesting forum
thread concerning the validity of the benchmark WU. So rather than just
repeat the times I suggest you read the thread and contribute if you feel so
inclined.
"As you know I've been benchmarking like crazy. I found a disturbing
problem: Standard vs. VHAR WU times and the invalidity of the Ars benchmark WU.
On my way to the Forum one day I noticed an oddity that was emerging in the
pattern of WU times on different machines. Upon investigation it seems that WU
times for "normal" angle range WUs are not well predicted by the Ars
benchmark WU time." There is much more...
It's really good to see such a structured discussion and thoughtful analysis of
its relevance or lack of. If the present benchmark has slipped in relevance then
it's probably time to change it for something more representative, especially as
3.03 does even more science. A different benchmark would also impact Roelof
considerably as he uses the results page information as the basis for some of
his SETI Spy CpF
comparisons. For the 2 to 3 client upgrade I was loath to change the benchmark
WU but I can see that it is slipping badly in applicability. In effect our
machines benchmarks are just comparing their ability to complete a certain task
(the bench WU) and not giving a very good comparison in how they measure up to
real WUs. For your information most of the 'real world'
WU's crunched (80% or so) have an angle range between 0.4 and 1 (check your SETI
Spy log and you'll see this is true) with 0.417 appearing strikingly often.
The angle range of a unit also is given on line 13 of every WU if you open one
up in a text editor. The bench WU is 6.718, an area where amongst other things
no gaussian searches occur. Any angle range over 1 completes in considerably
less time, hence the recent joy at the glut of 11's and 12's. To refresh
your memory about distributions of WU's by angle range and the type of
processing done have a look at zAmboni's
article.
Max out.
Benchmark submission angst...
If you are trying to submit benchmarks email them to Max
or RB
(there's a preference poll in itself) as there are 'issues' with the submissions
page. Apologies for your wasted time and inconvenience. Is that contrite enough?
Should be some new entries into the results page this weekend for those of you
slobbering for numbers and more speed but no promises! While I'm at it if anyone
can get this bit of software
to work I'd be interested. First saw it mentioned at BP6.com
( short article: BP6 -
Just Got Better! 23/11/00) and from the screenshot
it looks like a less extensive version of AGPinfo
(which doesn't run on Win2000).Since my Japanese is not what it used to be the
details are a little sketchy. It claims to be a Northbridge BX chipset interface
for setting registers to optimum values through a simple GUI. Anyway my
BP6/Win2k won't run it so if yours does let me know. This could be slightly
easier to use than main man Oda's WPCRedit
et al.
Thanks.
Max out.
|
