[PATCH] Add a nagios check for each rabbitmq server

[Jeremy Cline]

Commit eae92f73e95 installed the nagios scripts[0] that are packaged for
epel7-infra on the RabbitMQ hosts. This is an attempt to use them with
nagios. I don't know anything about nagios though, so I have no idea if
this is even close to right, or if it works.

[0] https://github.com/nagios-plugins-rabbitmq/nagios-plugins-rabbitmq

Signed-off-by: Jeremy Cline <jcline(a)redhat.com&gt;

---
 roles/nagios_client/tasks/main.yml            | 15 +++++++++++++
 .../templates/check_rabbitmq_server.cfg.j2    |  1 +
 roles/nagios_server/tasks/main.yml            | 22 +++++++++++++++++++
 .../nrpe/check_rabbitmq_server.cfg.j2         |  1 +
 4 files changed, 39 insertions(+)
 create mode 100644 roles/nagios_client/templates/check_rabbitmq_server.cfg.j2
 create mode 100644 roles/nagios_server/templates/nrpe/check_rabbitmq_server.cfg.j2

diff --git a/roles/nagios_client/tasks/main.yml b/roles/nagios_client/tasks/main.yml
index 0259a43ca..65a60a7c3 100644
--- a/roles/nagios_client/tasks/main.yml
+++ b/roles/nagios_client/tasks/main.yml
@@ -233,6 +233,21 @@
   tags:
   - nagios_client
 
+- name: install nrpe checks for the RabbitMQ cluster
+  template:
+    src: "{{ item }}.j2"
+    dest: "/etc/nrpe.d/{{ item }}"
+    owner: root
+    group: root
+    mode: 0600
+  with_items:
+    - check_rabbitmq_server.cfg
+  when: inventory_hostname.startswith('rabbitmq')
+  notify:
+  - restart nrpe
+  tags:
+  - nagios_client
+
 - name: nrpe service start
   service: name=nrpe state=started enabled=true
   tags:
diff --git a/roles/nagios_client/templates/check_rabbitmq_server.cfg.j2
b/roles/nagios_client/templates/check_rabbitmq_server.cfg.j2
new file mode 100644
index 000000000..498a8757e
--- /dev/null
+++ b/roles/nagios_client/templates/check_rabbitmq_server.cfg.j2
@@ -0,0 +1 @@
+command[check_nrpe_check_rabbitmq_server]=exec
/usr/lib64/nagios/plugins-rabbitmq/check_rabbitmq_server -H localhost --password "{{
(env == 'production')|ternary(rabbitmq_monitoring_password_production,
rabbitmq_monitoring_password_staging) }}" --user nagios-monitoring
diff --git a/roles/nagios_server/tasks/main.yml b/roles/nagios_server/tasks/main.yml
index 2d6d50c31..f61584ed6 100644
--- a/roles/nagios_server/tasks/main.yml
+++ b/roles/nagios_server/tasks/main.yml
@@ -110,6 +110,28 @@
   - config
   - nagios_server
 
+- name: Ensure the nrpe.d directory exists for nrpe configs
+  file:
+    dest: /etc/nrpe.d
+    state: directory
+  tags:
+  - nagios_server
+
+- name: Add nrpe configurations to /etc/nrpe.d
+  template:
+    src: "/nrpe/{{ item }}.j2"
+    dest: "/etc/nrpe.d/{{ item }}"
+    owner: root
+    group: root
+    mode: 0600
+  with_items:
+    - check_rabbitmq_server.cfg
+  notify:
+  - restart nrpe
+  tags:
+  - config
+  - nagios_server
+
 ## Copy over the nagios configs
 - name: Copy /etc/nagios/conf.d/*cfg files
   copy: src=nagios/configs/{{ item }} dest=/etc/nagios/conf.d/{{ item }}
diff --git a/roles/nagios_server/templates/nrpe/check_rabbitmq_server.cfg.j2
b/roles/nagios_server/templates/nrpe/check_rabbitmq_server.cfg.j2
new file mode 100644
index 000000000..498a8757e
--- /dev/null
+++ b/roles/nagios_server/templates/nrpe/check_rabbitmq_server.cfg.j2
@@ -0,0 +1 @@
+command[check_nrpe_check_rabbitmq_server]=exec
/usr/lib64/nagios/plugins-rabbitmq/check_rabbitmq_server -H localhost --password "{{
(env == 'production')|ternary(rabbitmq_monitoring_password_production,
rabbitmq_monitoring_password_staging) }}" --user nagios-monitoring
-- 
2.21.0

[Aurelien Bompard]

Le jeu. 16 mai 2019 à 16:52, Jeremy Cline <jcline(a)redhat.com&gt; a écrit :

>  Commit eae92f73e95 installed the nagios scripts[0] that are packaged
> for
>  epel7-infra on the RabbitMQ hosts. This is an attempt to use them with
>  nagios. I don't know anything about nagios though, so I have no idea if
>  this is even close to right, or if it works.
>
>  [0] https://github.com/nagios-plugins-rabbitmq/nagios-plugins-rabbitmq
>
>  
I'd like to try to implement this, and possibly add app-specific monitoring
of queues afterwards.
But I'll need someone who knows our Nagios setup to help me. I am familiar
with Nagios concepts (eg NRPE).
Ideally we'd lock a small timeslot and implement it together.  Anybody?

A.

[Aurelien Bompard]

>  I'd like to try to implement this, and possibly add app-specific
>  monitoring of queues afterwards.
>  
I've made a few changes to Jeremy's proposal, because I wanted to make use
of the configuration file that the NRPE plugin already deploys.
Attached is my proposed change to the Ansible repo.

If that works I'll add more checks later on.

A.

[Kevin Fenzi]

On 8/1/19 8:33 AM, Aurelien Bompard wrote:
> >
> > I'd like to try to implement this, and possibly add app-specific
> > monitoring of queues afterwards.
> >
>  
>  I've made a few changes to Jeremy's proposal, because I wanted to make use
>  of the configuration file that the NRPE plugin already deploys.
>  Attached is my proposed change to the Ansible repo.
>  
>  If that works I'll add more checks later on. 
Thanks very much for working on this. I have had it marked to look into
for ages, but obviously I was never getting to it. :)

kevin

[Aurelien Bompard]

>  I've made a few changes to Jeremy's proposal, because I wanted to make use
>  of the configuration file that the NRPE plugin already deploys.
>  Attached is my proposed change to the Ansible repo.
>
>  If that works I'll add more checks later on.
>
>  
OK I deployed that config but now SELinux is preventing NRPE from
connecting to the RabbitMQ management interface:
avc:  denied  { name_connect } for  pid=5182 comm="perl" dest=15672
scontext=system_u:system_r:nrpe_t:s0
tcontext=system_u:object_r:amqp_port_t:s0 tclass=tcp_socket permissive=0

What should I do? Create a SELinux module to allow that connection? Do we
have a policy regarding that sort of module creation?

A.

[Aurelien Bompard]

>  What should I do? Create a SELinux module to allow that connection? Do we
>  have a policy regarding that sort of module creation?
>  
I see that the Copr role has a policy module in Ansible (both source and
binary), copies the binary to the destination and loads it with "semodule
-i".  Can I do this? It seems suboptimal because a binary is stored in
Ansible, and because the policy module should be built on the same distro
version as where it's going to be loaded. I'll try to build the module one
the destination host directly, unless you think it's a bad idea from a
security perspective.

A.

[Aurelien Bompard]

Alright, I now have quite a few checks for the RabbitMQ servers. Those
checks also give out interesting metrics like queue sizes, connections,
message throughput, etc.
Do we have something in place to use and display those metrics?
I'd like to look at what our common usage values and trends are before
setting Nagios thresholds on those metrics.

Thanks!

Aurélien

[Kevin Fenzi]

On 8/2/19 7:12 AM, Aurelien Bompard wrote:
>  Alright, I now have quite a few checks for the RabbitMQ servers.
> Those
>  checks also give out interesting metrics like queue sizes, connections,
>  message throughput, etc.
>  Do we have something in place to use and display those metrics?
>  I'd like to look at what our common usage values and trends are before
>  setting Nagios thresholds on those metrics. 
Perhaps we could use
https://pypi.org/project/collectd-rabbitmq/
with our collectd server?

( https://admin.fedoraproject.org/collectd/ )

kevin