Oh, one more thing.
fmn has been alerting a lot. I am pretty sure from your description this
is because when it's rebuilding it's cache it doesn't process anything.
Apr 19 00:20:39 <zodbot> Aurélien Bompard - ansible.git:ad66644567
[*roles/openshift-apps/fmn/templates/deploymentconfig.yml] FMN: run the IRC sender now
Apr 19 00:40:56 <zodbot> PROBLEM -
rabbitmq01.iad2.fedoraproject.org/Check
queue fmn is CRITICAL: RABBITMQ_QUEUE CRITICAL - messages CRITICAL (830), messages_ready
OK (805) messages_unacknowledged OK (25) consumers OK (1) (noc01)
Apr 19 01:00:57 <zodbot> PROBLEM -
rabbitmq01.iad2.fedoraproject.org/Check
queue fmn is CRITICAL: RABBITMQ_QUEUE CRITICAL - messages CRITICAL (3052), messages_ready
OK (3027) messages_unacknowledged OK (25) consumers OK (1) (noc01)
Apr 19 02:00:57 <zodbot> PROBLEM -
rabbitmq01.iad2.fedoraproject.org/Check
queue fmn is CRITICAL: RABBITMQ_QUEUE CRITICAL - messages CRITICAL (9903), messages_ready
OK (9878) messages_unacknowledged OK (25) consumers OK (1) (noc01)
Apr 19 03:00:57 <zodbot> PROBLEM -
rabbitmq01.iad2.fedoraproject.org/Check
queue fmn is CRITICAL: RABBITMQ_QUEUE CRITICAL - messages CRITICAL (19733), messages_ready
OK (19708) messages_unacknowledged OK (25) consumers OK (1) (noc01)
Apr 19 04:00:57 <zodbot> PROBLEM -
rabbitmq01.iad2.fedoraproject.org/Check
queue fmn is CRITICAL: RABBITMQ_QUEUE CRITICAL - messages CRITICAL (28472), messages_ready
OK (28447) messages_unacknowledged OK (25) consumers OK (1) (noc01)
Apr 19 05:00:57 <zodbot> PROBLEM -
rabbitmq01.iad2.fedoraproject.org/Check
queue fmn is CRITICAL: RABBITMQ_QUEUE CRITICAL - messages CRITICAL (36369), messages_ready
OK (36344) messages_unacknowledged OK (25) consumers OK (1) (noc01)
Apr 19 05:40:57 <zodbot> RECOVERY -
rabbitmq01.iad2.fedoraproject.org/Check
queue fmn is OK: RABBITMQ_QUEUE OK - messages OK (1) messages_ready OK (0)
messages_unacknowledged OK (1) consumers OK (1) All queues under the thresholds (noc01)
Apr 19 13:42:58 <zodbot> PROBLEM -
rabbitmq01.iad2.fedoraproject.org/Check
queue fmn is WARNING: RABBITMQ_QUEUE WARNING - messages WARNING (282), messages_ready OK
(257) messages_unacknowledged OK (25) consumers OK (1) (noc01)
Apr 19 13:52:58 <zodbot> PROBLEM -
rabbitmq01.iad2.fedoraproject.org/Check
queue fmn is CRITICAL: RABBITMQ_QUEUE CRITICAL - messages CRITICAL (1257), messages_ready
OK (1232) messages_unacknowledged OK (25) consumers OK (1) (noc01)
So, we should figure out a way to not do this. ;)
We could just bump the check up to like 40000 or something?
Or you all could revisit just continuing to process while rebuilding the
cache?
Or something. :)
kevin