[Pkg-haproxy-maintainers] Issue with Haproxy Reload

[Rajeev Jaggavarapu]

Hi Joseph,

Thank you, and We figured out the issue with the server. The server is
taking much time to respond.

This template we for HaProxy is:

global
  daemon
  log 127.0.0.1 local0
  log 127.0.0.1 local1 notice

defaults
  option forwardfor
  option http-server-close
  *mode http*
  balance leastconn
  log            global
  retries             3
  maxconn          4096
  timeout connect  10s
  timeout client  50s
  timeout server  50s

On top of haproxy we are using Nginx, we set "keepalive_timeout 15" on
Nginx.

When we do multiple reloads (when HaProxy is serving sessions), the
previous process is persisting, once after the user closes his sessions,
those processes are not getting killed by themselves.

root      9328  0.6  0.4  56204  8608 ?        Ss   08:50   0:00
/usr/sbin/haproxy -D -f /etc/haproxy/haproxy.cfg -p /var/run/haproxy.pid
-sf 8163
root      9918  2.3  0.4  56304  8724 ?        Ss   08:11   0:57
/usr/sbin/haproxy -D -f /etc/haproxy/haproxy.cfg -p /var/run/haproxy.pid
-sf 9878

Meanwhile, if we update the Haproxy with new configs, sometimes, the
request(new configured sites) is trying to serve by the old process, This
case some user is getting 500 gateway errors.

We are using "*HTTP" *mode.

Thank you.


On Fri, Sep 11, 2015 at 1:12 PM, Joseph Lynch <joe.e.lynch at gmail.com> wrote:

> Hi Rajeev,
>
> > We are using Haproxy on top of Mesos cluster, We are doing  dynamic
> reloads for Haproxy based on marathon events (50-100 times in a day). We
> have nearly 300 applications that are running on Mesos (300 virtual hosts
> in Haproxy).
>
> That should be very doable; for context we reload HAProxy thousands of
> times per day and have around the same number of services. We do
> leverage our improvements to https://github.com/airbnb/synapse to
> minimize the number of reloads we have to do, but marathon is good at
> making us reload. Just curious, how do you have HAProxy deployed, is
> it running on a centralized machine somewhere or is it running on
> every host?
>
> > When we do dynamic reloads, Haproxy is taking long time to reload
> Haproxy, we observed that for 50 applications takes 30-40secs to reload
> Haproxy.
>
> This seems very surprising to me unless you're doing something like
> SSL. Can you post a portion of your config?
>
> > We have a single config file for Haproxy, when we do reload all the
> applications are getting reloaded (Front-ends), this causing downtime of
> all applications. Is there anyway to reduce the downtime and impact on
> end-users.
> >
> > We tried this scenario,
> > "
> http://engineeringblog.yelp.com/2015/04/true-zero-downtime-haproxy-reloads.html
> "
> >
> > By this if user requests while reload, the requests are queued and
> serving after reload.
>
> Full disclaimer, I wrote that post, and I'm not sure that it will be
> all that useful to you if your clients are external or your reloads
> take > 30s. "The largest drawback is that this works only for outgoing
> links and not for incoming traffic." It would theoretically not be
> hard to extend to incoming traffic using ifb but I haven't worked on
> actually proving out that solution. If the reload takes > 30s that
> technique simply won't work (you'll be buffering connections for 30s,
> and likely dropping them). If the 30s reloads are unavoidable you will
> likely want to consider one of the alternative strategies mentioned in
> the post. For example you can just drop SYNs since the 1s penalty
> isn't that big of a deal (will still see 30s+ of unavailability), use
> nginx/haproxy to route in front of haproxy (can be a bit confusing and
> hard to work with), or make something similar to
> http://inside.unbounce.com/product-dev/haproxy-reloads/ (be wary you
> pay conntrack with a solution like that).
>
> > But if we do multiple reloads one after another, HaProxy old processes
> persist even after reloading the HaProxy service, this is causing the
> serious issue.
> >
> > root      7816  0.1  0.0  20024  3028 ?        Ss   03:52   0:00
> /usr/sbin/haproxy -f /etc/haproxy/haproxy.cfg -p /var/run/haproxy.pid -D
> -sf 6778
> > root      7817  0.0  0.0  20024  3148 ?        Ss   03:52   0:00
> /usr/sbin/haproxy -f /etc/haproxy/haproxy.cfg -p /var/run/haproxy.pid -D
> -sf 6778
> >
> > Is there any solution stop the previous process once after it serving
> the request.
>
> That is expected behaviour
> ​​
> afaik. Those processes are likely still
> alive because there are still open connections held against them. How
> long is the longest timeout on your backend servers? This is common
> with long lived TCP mode backends, but those apps are often resilient
> to losing the TCP connection so you may just be able to kill the
> haproxy instances (it's what we do).
>
> > Can we separate the configurations based on front-ends like in Nginx, so
> that only those apps will effect if there is any changes in backend.
>
> I mean there is nothing that stops you from running multiple haproxy
> instances that bind to different ports. I think the right place to
> start though is figuring out why reloading takes so long, which can
> probably be figured out by looking at the config.
>
> Good luck,
> -Joey
>



-- 
Best Regards,
Rajeev Jaggavarapu
​,​
Paxcel Labs.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.alioth.debian.org/pipermail/pkg-haproxy-maintainers/attachments/20150915/d9a32354/attachment.html>