Datacenter move day 2

[Kevin Fenzi]

Greetings everyone. 

Just a quick recap of our day 2 datacenter move. 

Today was the buildsystem and all it's assocated things. 

* src.fedoraproject.org took longer than anticipated due to some
repeated syncs to make sure we got all the content bit for bit.
It's up now and should be working mostly normally. There's an issue with
fedora-messaging bus that we hope to fix in the morning. 

* koji took a long time to move. Almost all of it in a database
dump/restore. We actually did this instead of a replication or the like
because we wanted to get out of the rhel7 postgresql 9.2 to a more
recent version. We are now on rhel8 with postgresql 12.2. This gives us
ability moving forward to do nice things like partitioning the large
tables we have and in general I hope it will be faster. koji is also now
on 1.21.0 release and all the builders (and now the hubs too) are 
Fedora 32. 

As for builders in the new datacenter, we have 32 buildvm's, 40
buildvm-ppc64's, the same number of s390x (15) and 15 each of
buildvm-a64 (aarch6) and buildvm-a32 (armv7/armhfp). So, likely arm
builds are going to be the last to finish for many. Luckily this only
needs to last us a few weeks until we can get the rest of our arm
builders shipped and re-added. 

Our signing infrastructure is up and tested, but we have a few tweaks to
make to autosigning tomorrow morning. So, some builds may wait to be
signed for a bit until thats processing away again. 

We have not yet does updates pushes or rawhide compose, it's 11pm now
for me, so I think tomorrow is the better time to try those and clean up
anything we hit. Additionally, adding new packages seems to mostly work,
but it's emitting a error also, so we want to check that in the morning
before processing down the existing queue of requests. 

ODCS, mbs, osbs and resultsdb were all moved over, but there's various
issues bringing them up, so we will be doing that tomorrow. 

Tomorrow is fixing those things above, and moving mailman and
datagrepper. After today we are on the downhill side I think. 

Thanks everyone for being patient while we get everything back to fully
working. 

kevin