On 10/11/2016 09:07 PM, Kevin Fenzi wrote:

After a bunch of playing around, I managed to get bidirectional
database replication working for koji.stg.fedoraproject.org. 

Basic outline: 

db-koji01.stg and db-koji02.stg both have postgresql-9.4-bdr installed. 
init a koji db on both. 
create a koji user and set password on both.
restore prod db dump on one of them. 
enable replication of the koji db.
wait for a while for them to sync up 
(It took about 4.5 hours to sync ~170GB of database)
run the staging sql script on either node. 

I then added keepalived to keep a 'application ip' between the two of
them (favoring 01). 

Then I pointed koji01.stg at the application ip 
(well, I really just removed it's /etc/hosts entry for db-koji)

I tested switching the ip back and forth. 
I tested rebooting one node or the other. 
I tested disabling keepalived on one and rebooting (so the other one
became and stayed primary). 

Everything seemed to work. ;) 

So, next steps: 

1. Pound on koji.stg and see if anything breaks. ;) I tried to enable
koschei, but I think we have it currently set too not do staging
builds. If we could enable it that might be good. If anyone notices
anything broken on koji.stg, please let me know. If you can think of
some common cases we should test, please let me know that too. ;)

2. I am going to look at setting up another pair and get this all in
ansible and then see about migrating other staging services. I'm
hopeful that if koji works all our less database heavy apps will work
ok too. 

3. If everything keeps looking good, move to production. 

Longer term what does this mean?

It means we can do updates/reboot cycles with pretty close to no
downtime. We may have to be clever about the openvpn hub, but if we can
reboot database servers as desired we may be able to avoid maint
windows entirely, or at least reduce them a great deal. 


infrastructure mailing list -- infrastructure@lists.fedoraproject.org
To unsubscribe send an email to infrastructure-leave@lists.fedoraproject.org

Any tests in particular an apprentice could pound on it to check ?  Not honestly sure I have access to koji.stg but I'd be game for testing common and edge cases fas:linuxmodder
Corey Sheldon
Freelance IT Consultant, FamNa, Hacktivist
+1.310.737.7237 (Text only)
PGP Keys:
0x8B4E89435A88E539 | C006 564F FA67 CDEA E29B  F202 8B4E 8943 5A88 E539
0x59276298d2264944 | 6292 9ABD 6374 6AA7 6D4B  730F 5927 6298 D226 4944

Disclaimer: ALL Correspondence shall be deemed as private and confidential, re-distrubution is discouraged unless requested in the the correspondence in question.