Team Ars Technica Lamb Chop - Statistics and Benchmarking

However, there is no better proof point than actual benchmarks and actual running times. The 3.03 client has been long lived and we have many executions of our standard benchmark unit against a wide variety of hardware. To get an idea how our "standard" benchmark unit runs, see bench/303results.htm .

A key source of variability in SETI@home run times is a property known as angle range. Angle range controls a great deal of activity in the client, including "how hard" the code will look for certain patterns. This is because the angle range controls how finely the telescope obtains its signal in terms of how it travels over the sky, because the telescope's scan varies by the angle range.

After much consideration, we picked for our standard 3.03 benchmark unit a particular unit with an angle range of 0.417. There are several good reasons:

It is the most statistically frequent angle range.
It is near the predicted theoretical average in run time.
The units themselves do not vary greatly in run time.

I myself have seen that the benchmark unit time is a very useful stand-in for expected average time, especially if one is not looking for three decimal points of accuracy but only "is this box running about right."

However, as you might expect, particular properties of the data may affect run time. In SETI@home, the known player in this that matters is a property called angle range.

What remains, then, is to show how the work unit run time varies by angle range so that one can explain unit-to-unit variation. If that can be done, then one can find a proper equivalent machine on the benchmark page and use that to predict what a given work unit will achieve.

VLAR, "normal" range, and VHAR

We have adopted the VLAR and VHAR bit of jargon for certain ranges. VLAR (very low angle range) is normally considered to be any work unit below 0.1 and especially those below 0.01. VHAR is any angle range above 1.0 or so, especially above 2. These cutoffs are not strictly observed, but about right. The remaining ones (often between 0.2 and 0.6) are "normal" (actually, we don't really have a formal name for these).

Since the 0.417 is reasonably stable and common, I used it for my basic comparison point. I picked some work units that have been completed, and arranged and sorted them by angle range in a scatter diagram. I then divided their run time by the average run time of the 0.417 WUs in the corresponding sample. In addition to the raw scatter, I had Lotus 123 plot a "curve fitting" that showed the best fit to the data. Here is the result:

Several things are shown here. First, under Windows, VLAR units cost more than they do under Linux (strictly speaking, the data does not show that because I had no sufficiently low angle ranges in this particular Linux sample, but this is widely known from other samples to be true). We have long speculated about the causes (probably some invocation of a Windows API), but it has never been fully settled. It is, howver, an empirical fact.

Second, it shows that until the angle range gets small, the variance is not terribly great under either Windows or Linux, nor under AMD versus P III. Since we don't have all four combinations, that hasn't been definitively shown here, but it is consistent with other results and there is no reason to expect otherwise.

To keep the diagram reasonable in terms of viewing, the VHAR were kept off the above chart. A chart focussing on the higher angle ranges is shown here:

As can be seen, the gradual improvement is nearly linear. Given the relatively small sample sizes involved, the actual slope is probably identical with perhaps some variance for differences in data flow between the chips. In fact, they are bunched together so well at the left (between 0.6 or so and 1.0) that the P III results entirely cover the AMD results so that it appears only P III exist. But, that is an artifact of the plotting. As the green line shows, there are also green units present. Once the angle range much exceeds 0.6, it rapidly converges on 0.8 or so of the 0.417 angle range run time costs. It can even get slightly better as "very" VHARs occur. As shown, one sees some very large ranges sometimes (around 15 versus 0.417 as a more typical value). These are not common, though when they occur, they occur in bunches (as is true of VLARs). This is because the incoming tapes from Aricebo are not random and so the angle ranges aren't.

As a rule of thumb, then, one can expect VLARs to come in at around 1.2 to 1.5 of "normal" under Windows and VHAR to come in about 20 per cent faster in all cases. We have seen some really bad cases for VLAR of 0.01, far above 1.5 times worse, but not in this particular sample.

Note that some queuing software (SETI Queue in particular) has the ability to route work units of a specific angle range to specific machines. This is mostly used to steer VLARs to Linux machines. This one a good way of dealing with the VLAR problem. Second best in an all Windows shop is to steer VLARs to the slowest machine. Though that is "dreadful squared" for that machine, it keeps the faster machines moving and keeps the WU total highest overall.