Hey all, I've got an odd problem with HAProxy 1.6.3 that I'm not sure where to begin troubleshooting. I set up mailers for alerting when one of my DB cluster nodes changes state. I'm getting the alerts for when a node goes down (so mailing is working), but I'm not getting another one when that node comes back up. I've checked the documentation and it claims the alerts will be sent when there is a STATE change, which sounds to me like it should alert on ANY state change, up or down.
Is anyone else seeing this problem? I ask this list in lieu of having to join another mailing list to ask about it in case someone here knows how to fix it.
On Mon, 2016-05-16 at 15:58 -0400, Mark Haney wrote:
This is quite interesting. I see mails in the mail.log for the DOWN entries (there are two, one DOWN each for two test DB nodes), but none for any UP state changes. The UP state changes are logged in the haproxy.log file, just like the DOWN ones, so I don't see why the UP alerts isn't being sent out.
That /kind/ of thing is typical of a network being said to be up before it's actually usable, and messaging try to make use of the network in its part-way-there state, and never repeating the attempt when the network is actually working.
I used to get that sort of thing with NTP, years ago. I'd always have to restart it after the network went up. It'd try, fail, and give up.
On 13May2016 09:46, Mark Haney mark.haney@vifprogram.com wrote:
Hey all, I've got an odd problem with HAProxy 1.6.3 that I'm not sure where to begin troubleshooting. I set up mailers for alerting when one of my DB cluster nodes changes state. I'm getting the alerts for when a node goes down (so mailing is working), but I'm not getting another one when that node comes back up. I've checked the documentation and it claims the alerts will be sent when there is a STATE change, which sounds to me like it should alert on ANY state change, up or down.
Is anyone else seeing this problem? I ask this list in lieu of having to join another mailing list to ask about it in case someone here knows how to fix it.
I presume you mean this:
http://cbonte.github.io/haproxy-dconv/configuration-1.6.html#3.6
I've just looked at the 1.6.5 code and the email alerts are sent on both UP and DOWN under exactly the same circumstances as the syslog UP and DOWN alerts, etc.
I'd be:
- checking syslog for the matching UP and DOWN alerts
- looking at the mail logs on the relevant SMTP server for messages at the right times
- watching the haproxy stats page to check that the server you're expecting alerts for really is UP or DOWN when you expect it to be
- making a test clause in a small standalone haproxy.cfg file and turning servers on and off by hand to watch the behaviour
If nothing is learnt from this, then building and trying 1.6.5 to see if it is different in behaviour from 1.6.3, but I'd be surprised: the code is straight forward and unconditional (see server.c, search for send_email_alert, it shows up right under calls to send_log).
Superficially, I'm suspecting some mail filing issue, but only after looking at the above items first.
Cheers, Cameron Simpson cs@zip.com.au
This is quite interesting. I see mails in the mail.log for the DOWN entries (there are two, one DOWN each for two test DB nodes), but none for any UP state changes. The UP state changes are logged in the haproxy.log file, just like the DOWN ones, so I don't see why the UP alerts isn't being sent out.
Ah well, I guess I'll hit up the HAPROXY list to see if they know anything.
Thanks for the help.
On Sat, May 14, 2016 at 7:02 PM, Cameron Simpson cs@zip.com.au wrote:
On 13May2016 09:46, Mark Haney mark.haney@vifprogram.com wrote:
Hey all, I've got an odd problem with HAProxy 1.6.3 that I'm not sure where to begin troubleshooting. I set up mailers for alerting when one of my DB cluster nodes changes state. I'm getting the alerts for when a node goes down (so mailing is working), but I'm not getting another one when that node comes back up. I've checked the documentation and it claims the alerts will be sent when there is a STATE change, which sounds to me like it should alert on ANY state change, up or down.
Is anyone else seeing this problem? I ask this list in lieu of having to join another mailing list to ask about it in case someone here knows how to fix it.
I presume you mean this:
http://cbonte.github.io/haproxy-dconv/configuration-1.6.html#3.6
I've just looked at the 1.6.5 code and the email alerts are sent on both UP and DOWN under exactly the same circumstances as the syslog UP and DOWN alerts, etc.
I'd be:
checking syslog for the matching UP and DOWN alerts
looking at the mail logs on the relevant SMTP server for messages at
the right times
- watching the haproxy stats page to check that the server you're
expecting alerts for really is UP or DOWN when you expect it to be
- making a test clause in a small standalone haproxy.cfg file and
turning servers on and off by hand to watch the behaviour
If nothing is learnt from this, then building and trying 1.6.5 to see if it is different in behaviour from 1.6.3, but I'd be surprised: the code is straight forward and unconditional (see server.c, search for send_email_alert, it shows up right under calls to send_log).
Superficially, I'm suspecting some mail filing issue, but only after looking at the above items first.
Cheers, Cameron Simpson cs@zip.com.au -- users mailing list users@lists.fedoraproject.org To unsubscribe or change subscription options: http://lists.fedoraproject.org/admin/lists/users@lists.fedoraproject.org Fedora Code of Conduct: http://fedoraproject.org/code-of-conduct Guidelines: http://fedoraproject.org/wiki/Mailing_list_guidelines Have a question? Ask away: http://ask.fedoraproject.org
-
Cameron Simpson -
Mark Haney -
Tim