koji db replication in staging... working!

[Kevin Fenzi]

Greetings. 

After a bunch of playing around, I managed to get bidirectional
database replication working for koji.stg.fedoraproject.org. 

Basic outline: 

db-koji01.stg and db-koji02.stg both have postgresql-9.4-bdr installed. 
init a koji db on both. 
create a koji user and set password on both.
restore prod db dump on one of them. 
enable replication of the koji db.
wait for a while for them to sync up 
(It took about 4.5 hours to sync ~170GB of database)
run the staging sql script on either node. 

I then added keepalived to keep a 'application ip' between the two of
them (favoring 01). 

Then I pointed koji01.stg at the application ip 
(well, I really just removed it's /etc/hosts entry for db-koji)

I tested switching the ip back and forth. 
I tested rebooting one node or the other. 
I tested disabling keepalived on one and rebooting (so the other one
became and stayed primary). 

Everything seemed to work. ;) 

So, next steps: 

1. Pound on koji.stg and see if anything breaks. ;) I tried to enable
koschei, but I think we have it currently set too not do staging
builds. If we could enable it that might be good. If anyone notices
anything broken on koji.stg, please let me know. If you can think of
some common cases we should test, please let me know that too. ;)

2. I am going to look at setting up another pair and get this all in
ansible and then see about migrating other staging services. I'm
hopeful that if koji works all our less database heavy apps will work
ok too. 

3. If everything keeps looking good, move to production. 

Longer term what does this mean?

It means we can do updates/reboot cycles with pretty close to no
downtime. We may have to be clever about the openvpn hub, but if we can
reboot database servers as desired we may be able to avoid maint
windows entirely, or at least reduce them a great deal. 

kevin