Day one of the datacenter service migration

[Kevin Fenzi]

Greetings everyone. 

I thought I'd share with everyone how things went today and where we are
at on the datacenter service migration. :) 

We did get everything we planns to migrated today: 

* staging services shutdown and machines with those resources readied to
 be shipped out. 

* fedora-messaging/fedmsg buses/clusters. We ran into a brief hiccup
here as I hadn't properly issued the rabbitmq cluster certs with an
alternative name for 'rabbitmq.fedoraproject.org', but we found it and
fixed it. All consumers and producers should be connecting to the new
rabbitmq cluster in the new datacenter now. 

* notifications service. Since this service is brittle and ancient we
elected to just copy the vm over and adjust its settings for the new dc
after. I am not 100% sure it's functioning properly yet (as it takes a
while to start up), but it seems to be close. 

* pdc. This one was much longer than I expected, and I am sorry about
that as it prevents people from committing. It turns out the pdc
database is pretty gigantic, so it took a few hours to dump it out and
load it in the new db server. ;( It's up and working now. 

* mirrormanager. There's some stats reporting and a few crons that may
need twekaing, but I think the basic service is working. 

* Authenitcation stack (ipa, fas, ipsilon). I ran into a few snags here
with routing and vpns, but everything should be moved over and working
normally.

On the unplanned side it turned out to be more complex than I had though
to just move some of our openshift apps and not the others. Because of
that I made the decision to just move all the (user facing) ones today. 
That includes: fas, ipsilon, bodhi, compose-tracker, elections,
greenwave, waiverdb, mdapi and a few more non user facing ones. 

The openshift apps move caused a outage for the elections app (both a
short one while it was entirely down, and another short time when
authentication wasn't yet working), and additionally when bodhi was
moved it was inadvertently restarted before it's database was synced
over, so if you say a bunch of bodhi actions today where it said it was
pushing things to stable that were already stable, that was the cause.
;( I quicked stopped the app and resynced the db, and hopefully not much
damage was done. 

Tomorrow is the big day for koji and all it's assosicated services. 
This work will start at 15UTC, so if you have any builds to do, make
sure they finish before then. Any in progress builds will be canceled at
around 15UTC, you can resubmit them once things are back up. 

Thanks again for everyones patience with this move and hopefully we will
survive the week. :) 

kevin