FMN / notifications status

[Kevin Fenzi]

So, yesterday we managed to get the python3 version of fmn working in
staging seemingly well, so I stupidly decided to try and migrate
production to it. This has been fraught with peril. ;( 

First, only the notifs-backend01 ip is allowed to reach out to IRC, so
after saving off the notifs-backend01 (python2) fmn disk and vm xml I
tried to get notifs-backend02 to use 01's ip. That works, but made
things like IPA mad, since the hostname/ip was mismatched. 
So, I just redeployed a new notifs-backend01 with the python3 version. 
Sadly, the IRC part which we hadn't been able to test in staging, wasn't
working right at all. So, I went back to the old notifs-backend01.

Unfortunately, that vm is now not processing correctly either. 
(Even though it's the same exact VM that was there before). :( 
With help from pingou, we cleaned up some errors, but it's still not
seeming to be sending out things. It may be that it's just loading
caches and such and will start working here soon. 

So, here's my plan:

* Leave the current python2 one alone for a while and hope it starts
processing things again.
* Later in the day, I am going to save off the current python2 backend01
(again) and deploy the python3 one (again). 
Hopefully that gets at least emails flowing again. 
* pingou is going to look at it when he gets in in the morning tomorrow
and fix issues.

Hopefully that will result in a working python3 FMN. ;( 

Since we are going into freeze this morning, I'm going to treat this
work as "it's an outage, fix it" and not worry about the freeze for it.

So, sorry for the lack of notifications last night and today,
I wish things had gone smoother. 

kevin