infrastructure September 2010

infrastructure@lists.fedoraproject.org

29 participants
36 discussions

Lessons learned: Initial check/step list for updates
by Stephen John Smoogen 10 Sep '10

10 Sep '10

0. Plan time in infrastructure(a)lists.fedoraprojects.org 1. Open ticket on infrastructure for downtime. Updates will occur during day Reboots will occur during evening 2. Send email to devel-announce, announce, infrastructure 3. Update servers during working hours and work out issues in ticket. ** releng updates the following boxes: cvs01, pkgs01, nfs01, bnfs01, bxen*, x86-*, ppc*, koji*, db03, xb-01, compose-*, sign-vault01 4. Change DNS to turn off proxy on bodhost01 (or similar external proxy server). 5. Reboot bodhost01 6. Confirm proxy is working on bodhost/fix issues. 7. Change proxy dns to only go to bodhost01 8. Turn off nagios for servers. 9. Turn off nagios-external for services. 10. Reboot order counts 11. releng deals with the boxes listed above unless told otherwise. 12. reboots with database servers first xen15: db02 xen12: db01 13. reboot PHX2 boxes xen03: xen04: xen06: xen07: xen09: xen10: xen11: xen13: backup01: 14. reboot Outside boxes (can be in parallel to PHX2) cnode01: cnode02: cnode03: ibiblio01: internetx01: osuosl01: people01: serverbeach1: serverbeach2: serverbeach3: serverbeach4: serverbeach5: telia1: tummy1: 15. reboot bastion.fedoraproject.org log into bastion1 from outside system log into bastion2 from outside world log into xen05 from bastion01 bastion01: sudo su /usr/sbin/puppetd --disable sudo su /sbin/service openvpn start bastion02 sudo su /sbin/service openvpn start xen05 sudo /sbin/shutdown -r now once xen05/bastion2 server is back up, we can bastion01: sudo su /sbin/service openvpn stop sudo su /usr/sbin/puppetd --enable 16. reboot puppet01 log into bastion2 from outside world ssh xen14 sudo /sbin/shutdown -r now 17. re-enable DNS for proxy servers test proxy servers from puppet01 edit dns in git puppet make ns1 18. re-enable nagios on internal/external 19. Setup transifex agent on app servers: app01 app02 app03 app04 app07 sudo -u transifex /var/lib/transifex/ssh-add.sh -f 20. Log and report problems to list. 21. Close ticket. -- Stephen J Smoogen. “The core skill of innovators is error recovery, not failure avoidance.” Randy Nelson, President of Pixar University. "We have a strategic plan. It's called doing things."" — Herb Kelleher, founder Southwest Airlines

1 0

Outage: Updates - 2010-09-10 00:00->02:00 UTC
by Stephen John Smoogen 08 Sep '10

08 Sep '10

There will be an outage starting at 2010-09-10 00:00:00 UTC, which will last approximately 2 hours. To convert UTC to your local time, take a look at http://fedoraproject.org/wiki/Infrastructure/UTCHowto or run: date -d '2010-09-10 00:00:00 UTC' BFO - http://boot.fedoraproject.org/ Bodhi - https://admin.fedoraproject.org/updates/ Buildsystem - http://koji.fedoraproject.org/ CVS / Source Control DNS - ns1.fedoraproject.org, ns2.fedoraproject.org Docs - http://docs.fedoraproject.org/ Email system Fedora Account System - https://admin.fedoraproject.org/accounts/ Fedora Community - https://admin.fedoraproject.org/community/ Fedora Hosted - https://fedorahosted.org/ Fedora People - http://fedorapeople.org/ Fedora Talk - http://talk.fedoraproject.org/ Main Website - http://fedoraproject.org/ Mirror List - https://mirrors.fedoraproject.org/ Mirror Manager - https://admin.fedoraproject.org/mirrormanager/ Package Database - https://admin.fedoraproject.org/pkgdb/ Smolt - http://smolts.org/ Spins - http://spins.fedoraproject.org/ Start - http://start.fedoraproject.org/ Torrent - http://torrent.fedoraproject.org/ Translation Services - http://translate.fedoraproject.org/ Wiki - http://fedoraproject.org/wiki/ Unaffected Services: None Ticket Link: https://fedorahosted.org/fedora-infrastructure/ticket/2379 -- Stephen J Smoogen. “The core skill of innovators is error recovery, not failure avoidance.” Randy Nelson, President of Pixar University. "We have a strategic plan. It's called doing things."" — Herb Kelleher, founder Southwest Airlines

1 0

speeding up the group/dump query in FAS
by Matt Domsch 07 Sep '10

07 Sep '10

Currently this query takes several minutes. HAProxy (or something in the proxy series) times out the request, and returns a 500 to the client. https://admin.fedoraproject.org/accounts/group/dump/ Any chance that could be optimized? FWIW, I use this query in my ftbfs script to convert package owner names to email addresses, so I can send the FTBFS report to the owner's emails directly (on bcc). Thanks, Matt -- Matt Domsch Technology Strategist Dell | Office of the CTO

4 3

Question about using fedorahosted as a git mirror
by Richard W.M. Jones 07 Sep '10

07 Sep '10

There's a legitimate use of git hosting providers as mirrors, where they don't act as the canonical upstream of a project, but they act as backup git mirrors. This is useful if you want to allow an alternate place to pull from (for higher availability) or for enabling transifex support where you don't want to give out ssh access to the primary git host: http://fedoraproject.org/wiki/L10N_FAQ#How_do_I_add_a_module_to_Transifex.3… It'd be great if fedorahosted could be used for this. Is this an acceptable use for fedorahosted? If permitted, would such projects need to be hidden from the front page? Rich. -- Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones virt-df lists disk usage of guests without needing to install any software inside the virtual machine. Supports Linux and Windows. http://et.redhat.com/~rjones/virt-df/

3 5

Upcoming Fedora 14 Tasks
by John Poelstra 07 Sep '10

07 Sep '10

Start End Name Tue 14-Sep Tue 28-Sep Beta Infrastructure Change Freeze Have you found something that is working really well in your schedule? Or maybe you've found something you believe would work better in the future? Update the schedule retrospective page now to capture all of the important details as they happen: https://fedoraproject.org/wiki/Fedora_14_Schedule_Retrospective

1 0

[PATCH/RFC] func: Simplify minion.conf creation
by Todd Zullinger 03 Sep '10

03 Sep '10

Puppet conveniently makes most variables from Puppet.settings available via Puppet[:variable]. Use this to generate minion.conf via a standard ERB template. --- This change is the sort of thing that really begs for a second set of eyes, as it would cause too much trouble if it didn't work as intended. A quick way to test that the template does the right thing is to grab http://fedorapeople.org/~tmz/tmp/func-minion-template-test.pp and run it through puppet, e.g. sudo puppet /tmp/func-minion-template-test.pp. I noticed this while poking the func setup to see what settings were needed to make func use puppet's certificates with the new code from Seth in func's git tree. (I also noticed that the overlord.conf isn't puppet managed, and that's the one I was more curious to see, easily found on puppet01. ;) modules/func/files/make-minion-conf.sh | 24 ------------------------ modules/func/manifests/init.pp | 24 ++++++++++++------------ modules/func/templates/minion.conf.erb | 13 +++++++++++++ 3 files changed, 25 insertions(+), 36 deletions(-) delete mode 100755 modules/func/files/make-minion-conf.sh create mode 100644 modules/func/templates/minion.conf.erb diff --git a/modules/func/files/make-minion-conf.sh b/modules/func/files/make-minion-conf.sh deleted file mode 100755 index ea50c95..0000000 --- a/modules/func/files/make-minion-conf.sh +++ /dev/null @@ -1,24 +0,0 @@ -#!/bin/bash - - -certfile=`ruby -e "require 'puppet'; Puppet.parse_config; puts Puppet.settings['hostcert']"` -keyfile=`ruby -e "require 'puppet'; Puppet.parse_config; puts Puppet.settings['hostprivkey']"` -cacert=`ruby -e "require 'puppet'; Puppet.parse_config; puts Puppet.settings['localcacert']"` - - -cat <<EOM>/etc/func/minion.conf -# configuration for minions - -[main] -log_level = DEBUG -acl_dir = /etc/func/minion-acl.d - -listen_addr = -listen_port = 51234 -use_certmaster = 0 - -cert_file = $certfile -key_file = $keyfile -ca_file = $cacert - -EOM diff --git a/modules/func/manifests/init.pp b/modules/func/manifests/init.pp index 50d6ed8..b9d39fe 100644 --- a/modules/func/manifests/init.pp +++ b/modules/func/manifests/init.pp @@ -23,23 +23,23 @@ class func::funcd { } } + File { + require => Package['func'], + } + file { '/etc/logrotate.d/func_rotate': source => 'puppet:///func/func_rotate', - require => Package['func'] - } - file { '/usr/local/bin/make-minion-conf.sh': - source => 'puppet:///func/make-minion-conf.sh', - require => Package['func'], - mode => 0750, - notify => Exec['makeminionconf'] } - exec { 'makeminionconf': - require => File['/usr/local/bin/make-minion-conf.sh'], - command => '/bin/bash /usr/local/bin/make-minion-conf.sh', - refreshonly => true, - notify => Service['funcd'] + + file { '/etc/func/minion.conf': + content => template('func/minion.conf.erb'), + notify => Service['funcd'], } + # This can be removed once all systems are updated + file { '/usr/local/bin/make-minion-conf.sh': + ensure => absent, + } } class func::certmaster { diff --git a/modules/func/templates/minion.conf.erb b/modules/func/templates/minion.conf.erb new file mode 100644 index 0000000..33395ca --- /dev/null +++ b/modules/func/templates/minion.conf.erb @@ -0,0 +1,13 @@ +# configuration for minions + +[main] +log_level = DEBUG +acl_dir = /etc/func/minion-acl.d + +listen_addr = +listen_port = 51234 +use_certmaster = 0 + +cert_file = <%= Puppet[:hostcert] %> +key_file = <%= Puppet[:hostprivkey] %> +ca_file = <%= Puppet[:localcacert] %> -- 1.7.2.2 -- Todd OpenPGP -> KeyID: 0xBEAF0CE3 | URL: www.pobox.com/~tmz/pgp ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ History teaches that grave threats to liberty often come in times of urgency, when constitutional rights seem too extravagant to endure. -- Justice Thurgood Marshall, 1989

2 6

2025

2024

2023

2022

2021

2020

2019

2018

2017

2016

2015

2014

2013

2012

2011

2010

2009

2008

2007

2006

infrastructure September 2010