[Shootout-list] Threads and Threads-Flow Proposal

[Theodora Carson]

I'm about to submit some working examples of the Threads and Threads-Flow
benchmarks written in Python.

They are working, correct but will likely fail the actual benchmark run.
This got me thinking about these two benchmarks and what their real intent
should be...

The reason the benchmarks may fail is that the upper bound of how many
threads are allowed to be created varies with OS, kernel revision, libc
revision, etc. as well as the thread structures of Python.  This is not an
easy thing to fix from within a portable program written in any language.
I've tested the program on Windows XP, and two subrevisions of Linux 2.6.10
on two different architectures.  On one platform the program could allocate
~1000 threads, another didn't get past ~380 and one succeeded with the
benchmark requirement of 3000.

Obviously both threads want to create a "lot" of threads.  I suspect to make
signficant timings for some of the languages, the value was upped to
unlikely-to-be-seen-in-production 3000 threads.  But one could more portably
and usefully wrap the main loop in an encompassing repetition of the test.
This makes particular sense if one agrees with the following
characterization of the benchmarks' intent:

Threads -- test the creation, start, and tear down time of a chain of
threads.  This models systems that dynamically spawn tasks so 10 repetitions
over a chain of 300 threads is just as valid as 1 repetition with 3000
threads -- the same number of creates, starts, and stops get exercised.

Threads-Flow -- test the overhead of moving messages repeatedly through a
group of threads in a worst case scenario where each thread is woken up with
a tiny bit of work to do before passing the message onwards.  Once again,
measurement of the moving overhead is just as valid for 2000 messages over
300 threads as 200 messages over 3000 threads.  The same number of messages
are moved.

We are testing the limits of the languages for a type of application
behaviour; not the limits of a 2.6 Linux environment running on a 1.1Ghz
Athlon.

I hope this gets changed.

Kevin