[Pkg-openmpi-maintainers] Bug#598553: Bug#598553: r-cran-rmpi: slave processes eat CPU when they have nothing to do

[Zack Weinberg]

On Sat, Oct 2, 2010 at 12:51 PM, Manuel Prinz <manuel at debian.org> wrote:
>
> I did some reading and it seems the Open MPI indeed does support two modes
> of waiting: aggressive and degraded. The default behavior is "aggressive",
> but you can switch them by setting the mpi_yield_when_idle MCA parameter.
> See the following FAQ entries (and links therein):
>
> http://www.open-mpi.org/faq/?category=running#force-aggressive-degraded
> http://www.open-mpi.org/faq/?category=running#oversubscribing
>
> I guess this is basically the behaviour you want. It would be great if you
> could give it a try and report back if it works for you. If it doesn't do
> what you (and I) expect, I'll forward this issue upstream.

I wrote a test MPI program that just calls MPI_Probe() once - this
should block forever, since there are no sends happening.  When run
with

$ mpirun -np 2 ./a.out

MPI_Probe never returns and the processes spin through poll(), which
is what I originally reported.  So far so good.  If I change the
invocation to

$ mpirun -np 2 --mca mpi_yield_when_idle 1 ./a.out

the behavior is the same, except that the processes alternate between
poll() and sched_yield().  This doesn't help anything; the scheduler
is still being thrashed, and the CPU is not allowed to go idle.  [In
fact, my understanding of the Linux scheduler is that a zero-timeout
poll() counts as a yield, so "Aggressive" mode isn't even doing
anything constructive!]

The desired behavior is for an idle cluster's processes to BLOCK in
poll().  So mpi_yield_when_idle does not do what I want.

Also, putting "mpi_yield_when_idle = 1" into
~/.openmpi/mca-params.conf has no effect, contra the documentation --
this perhaps ought to be its own bug.  (I can set MCA parameters for R
with environment variables, but that's not nearly as convenient as the
host file.)

zw