[Shootout-list] mandelbrot problem

[Stephen Weeks]

The mandelbrot benchmark is flawed because it uses floating point
computations with exact comparisons to yield bits in the output and
then requires an exact match on the output.  On x86, floating point
numbers are kept internally at 80 bits and writes to memory round them
to 64 bits; hence, the specified correct output of the benchmark is an
accident of how the compiler that generated that executable made
certain low-level decisions (register allocation, instruction
selection, ...) that affect reads/writes of doubles to memory.

As evidence of the problem, observe that mandelbrot.mlton produces
different output depending on whether the program is compiled with
"-ieee-fp false" or "-ieee-fp true", where the latter forces writes to
memory after every floating point operation.  The outputs are visually
indistinguishable, and, in the case of N=400, differ in only one byte
(zero-based character position 48).

As you might imagine, "-ieee-fp true" carries a very high performance
cost due to the memory traffic, so it isn't typically a viable
option.  In the case of the mandelbrot benchmark, a simple test on my
machine shows an over 5X slowdown.

In any case, it seems to me that the benchmark should either introduce
some tolerance internally, so that the low-level compiler decisions
are not exposed, or it should allow some tolerance in the output.