Bug#827322: multipath-tools: Not all paths discovered when large number of paths are present

[Andrew Patterson]

On Thu, 16 Jun 2016 00:15:37 +0530 Ritesh Raj Sarraf <rrs at debian.org> wrote:
> On Wed, 2016-06-15 at 11:12 -0600, Andrew Patterson wrote:
> > 
> > Thanks for reminding me about max_fds. At boot:
> > 
> > # cat /proc/sys/fs/nr_open
> > 1048576
> > 
> > and
> > 
> > # multipath -t | grep max_fds
> > Â Â Â Â Â Â Â  max_fds "max"
> > 
> > So this should already be set to 1048576. I tried the test again after
> > setting max_fd 1048576 in /etc/multipath.conf and got the same
> > behavior.
> 
> This dates way back. 2009 is a memorable year.
> 
> 
> https://www.redhat.com/archives/dm-devel/2009-July/msg00020.html
> 
> Quoting:
> 
> +
> +4. I see a lot of "io_setup failed" message using the directio checker
> +======================================================================
> +
> +The directio path checker makes use of the asynchronous I/O API (aio) provided
> +by modern Linux systems. Asynchronous I/O allows an application to submit I/O
> +requests asynchronously and be notified later of their completion status. To
> +allow this, we must allocate an asynchronous I/O context (an object of type
> +aio_context_t) and this task is handled by the io_setup system call.
> +
> +A system wide limit on the number of AIO contexts that may be active
> +simultaneously is imposed via the aio-max-nr sysctl parameter.
> +
> +Once this limit has been reached further calls to io_setup will fail with the
> +error number EAGAIN leading to the "io_setup failed" messages seen for e.g.
> when
> +running "multipath -ll".
> +
> +To avoid this problem the number of available aio contexts should be increased
> +by setting the aio-max-nr parameter. This can be set on a one-time basis via
> the
> +/proc file system, for e.g.:
> +
> +Â Â Â Â Â Â Â Â # echo 131072 > /proc/sys/fs/aio-max-nr
> +
> +Doubles the number of available contexts from the default value of 65536.
> +
> +To make this setting persistent a line may be added to /etc/sysctl.conf:
> +
> +Â Â Â Â Â Â Â Â fs.aio-max-nr = 131072
> +
> +Consult appropriate application and operating system tuning recommendations for
> +guidance on appropriate values for this parameter.
> 
> 
> -- 
> Ritesh Raj Sarraf | http://people.debian.org/~rrs

For most users this is not an issue. How many systems are going to use
that many LUNs?

One counter-example is using one or more FC LUNs per VM guest on a
host. In such a case, one can just modify /etc/sysctl.conf to increase
the setting. We can run into problems when using
multipath-tools-boot. In this case, we use the default kernel settings
for aio-max-nr since multipath is run from the initramfs. I have
worked around this by modifying the script in
/usr/share/initramfs/scripts/local-top/multipath to temporarilly set
aio-max-nr to 1048576 while doing discovery.


-- 
Andrew Patterson
Hewlett-Packard Enterprise