Change Request: Race condition syncing static websites to proxies
by Toshio Kuratomi
mmcgrath noted that we have a race condition in our syncStatic.sh script
that is leading to proxies not having copies of the website. Here's a patch
to the syncStatic script to fix that:
diff --git a/modules/fedora-web/files/syncStatic.sh b/modules/fedora-web/files/syncStatic.sh
index 23f5181..9c1b7ac 100644
--- a/modules/fedora-web/files/syncStatic.sh
+++ b/modules/fedora-web/files/syncStatic.sh
@@ -26,7 +26,7 @@ cd fedora-web
pushd fedoraproject.org > /dev/null
make > /dev/null 2>&1
-rsync -qa --delete out/* /srv/web/fedoraproject.org/
+rsync -qa --delete out/ /srv/web/fedoraproject.org/
popd > /dev/null
# Make sure everything else builds from master.
@@ -34,26 +34,22 @@ popd > /dev/null
/usr/bin/git checkout -q master || exit 1
pushd spins.fedoraproject.org > /dev/null
-/bin/rm -rf /srv/web/spins.fedoraproject.org/*
make > /dev/null 2>&1
-rsync -qa --delete out/* /srv/web/spins.fedoraproject.org/
+rsync -qa --delete out/ /srv/web/spins.fedoraproject.org/
popd > /dev/null
pushd talk.fedoraproject.org > /dev/null
-/bin/rm -rf /srv/web/talk.fedoraproject.org/*
make > /dev/null 2>&1
-rsync -qa --delete out/* /srv/web/talk.fedoraproject.org/
+rsync -qa --delete out/ /srv/web/talk.fedoraproject.org/
popd > /dev/null
pushd boot.fedoraproject.org > /dev/null
-/bin/rm -rf /srv/web/boot.fedoraproject.org/*
make > /dev/null 2>&1
-rsync -qa --delete out/* /srv/web/boot.fedoraproject.org/
+rsync -qa --delete out/ /srv/web/boot.fedoraproject.org/
popd > /dev/null
pushd start.fedoraproject.org > /dev/null
-/bin/rm -rf /srv/web/start.fedoraproject.org/*
make > /dev/null 2>&1
-rsync -qa --delete out/* /srv/web/start.fedoraproject.org/
+rsync -qa --delete out/ /srv/web/start.fedoraproject.org/
popd > /dev/null
The patch does two things:
1) Removes the rm -rf step as that is causing the race (when proxies sync
data after the rm -rf and before hte rsync, they get an empty directory)
2) Removes the wildcard from the rsync. The wildcard was preventing the
--delete from functioning as expected. I believe (but ricky can confirm)
that the --delete not removing files that are no longer needed from the
final output directory is why we had the rm -rf in the first place.
Could I get two +1's for this change?
-Toshio
14 years
proxy07 logs sync additions (need review and +1)
by Stephen John Smoogen
Proxy07 was not set up with rsync added. This means the http logs for
a while have not been syncing. Fix is to copy, add, commit and push
rsync conf files that proxy[123456] have already. Please review and +1
or -1 before I push.
commit 347fa93b0bcf3e8b6f7f865177a728079b6d2904
Author: Stephen Smoogen <smooge(a)redhat.com>
Date: Mon May 10 23:47:41 2010 +0000
Add in proxy07 so sync's actually work
diff --git a/modules/rsync/files/rsync.proxy07
b/modules/rsync/files/rsync.proxy07
new file mode 100644
index 0000000..f02e097
--- /dev/null
+++ b/modules/rsync/files/rsync.proxy07
@@ -0,0 +1,14 @@
+# default: off
+# description: The rsync server is a good addition to an ftp server, as it \
+# allows crc checksumming etc.
+service rsync
+{
+ disable = no
+ socket_type = stream
+ wait = no
+ user = root
+ server = /usr/bin/rsync
+ server_args = --daemon
+ log_on_failure += USERID
+}
+
diff --git a/modules/rsync/files/rsyncd.conf.proxy07
b/modules/rsync/files/rsyncd.conf.proxy07
new file mode 100644
index 0000000..eb2c13d
--- /dev/null
+++ b/modules/rsync/files/rsyncd.conf.proxy07
@@ -0,0 +1,16 @@
+pid file = /var/run/rsyncd.pid
+syslog facility = daemon
+dont compress = *.gz *.tgz *.zip *.z *.rpm *.deb *.bz2 *.iso
+use chroot = false
+transfer logging = false
+exclude = .snapshot/ .~tmp~/ /.private/ /.private/** **/.nfs*
+
+# For distributing applications
+[log]
+comment = Server Logs
+path = /var/log
+uid = root
+gid = root
+read only = yes
+hosts allow = 10.5.126.28 192.168.1.35
--
Stephen J Smoogen.
“The core skill of innovators is error recovery, not failure avoidance.”
Randy Nelson, President of Pixar University.
"We have a strategic plan. It's called doing things.""
— Herb Kelleher, founder Southwest Airlines
14 years
[PATCH] add recoreds for all the new hosts
by Dennis Gilmore
From: Dennis Gilmore <ausil(a)puppet01.phx2.fedoraproject.org>
---
modules/bind/files/master/125.5.10.in-addr.arpa | 34 ++++++++++----------
modules/bind/files/master/127.5.10.in-addr.arpa | 36 ++++++++++----------
modules/bind/files/master/phx2.fedoraproject.org | 37 ++++++++++++++++++++--
3 files changed, 69 insertions(+), 38 deletions(-)
diff --git a/modules/bind/files/master/125.5.10.in-addr.arpa b/modules/bind/files/master/125.5.10.in-addr.arpa
index ad1b19b..5f905dc 100644
--- a/modules/bind/files/master/125.5.10.in-addr.arpa
+++ b/modules/bind/files/master/125.5.10.in-addr.arpa
@@ -2,7 +2,7 @@ $TTL 86400
@ IN SOA 125.5.10.in-addr.arpa. hostmaster.fedoraproject.org. (
; Make SURE to update the serial # when changes are made. Format is YYYYMMDDNN
; (NN is just a counter to handle multiple changes in one day).
- 2010031800 ; Serial
+ 2010051000 ; Serial
28800 ; Refresh
14400 ; Retry
1000000 ; Expire
@@ -39,8 +39,8 @@ $TTL 86400
28 IN PTR ppc08.phx2.fedoraproject.org.
29 IN PTR ppc09.phx2.fedoraproject.org.
30 IN PTR ppc10.phx2.fedoraproject.org.
-31 IN PTR unused.
-32 IN PTR unused.
+31 IN PTR ppc11.phx2.fedoraproject.org.
+32 IN PTR ppc12.phx2.fedoraproject.org.
33 IN PTR unused.
34 IN PTR unused.
35 IN PTR unused.
@@ -88,19 +88,19 @@ $TTL 86400
77 IN PTR unused.
78 IN PTR unused.
79 IN PTR unused.
-80 IN PTR unused.
-81 IN PTR unused.
-82 IN PTR unused.
-83 IN PTR unused.
-84 IN PTR unused.
-85 IN PTR unused.
-86 IN PTR unused.
-87 IN PTR unused.
-88 IN PTR unused.
-89 IN PTR unused.
-90 IN PTR unused.
-91 IN PTR unused.
-92 IN PTR unused.
+80 IN PTR x86-10.phx2.fedoraproject.org.
+81 IN PTR x86-11.phx2.fedoraproject.org.
+82 IN PTR x86-12.phx2.fedoraproject.org.
+83 IN PTR x86-13.phx2.fedoraproject.org.
+84 IN PTR x86-14.phx2.fedoraproject.org.
+85 IN PTR x86-15.phx2.fedoraproject.org.
+86 IN PTR x86-16.phx2.fedoraproject.org.
+87 IN PTR x86-17.phx2.fedoraproject.org.
+88 IN PTR x86-18.phx2.fedoraproject.org.
+89 IN PTR x86-19.phx2.fedoraproject.org.
+90 IN PTR x86-20.phx2.fedoraproject.org.
+91 IN PTR x86-21.phx2.fedoraproject.org.
+92 IN PTR x86-22.phx2.fedoraproject.org.
93 IN PTR unused.
94 IN PTR unused.
95 IN PTR unused.
@@ -128,7 +128,7 @@ $TTL 86400
117 IN PTR unused.
118 IN PTR unused.
119 IN PTR unused.
-120 IN PTR unused.
+120 IN PTR bxen04.phx2.fedoraproject.org.
121 IN PTR bxen03.phx2.fedoraproject.org.
122 IN PTR bxen02.phx2.fedoraproject.org.
123 IN PTR bxen01.phx2.fedoraproject.org.
diff --git a/modules/bind/files/master/127.5.10.in-addr.arpa b/modules/bind/files/master/127.5.10.in-addr.arpa
index fd94075..c2650e0 100644
--- a/modules/bind/files/master/127.5.10.in-addr.arpa
+++ b/modules/bind/files/master/127.5.10.in-addr.arpa
@@ -2,7 +2,7 @@ $TTL 86400
@ IN SOA 127.5.10.in-addr.arpa. hostmaster.fedoraproject.org. (
; Make SURE to update the serial # when changes are made. Format is YYYYMMDDNN
; (NN is just a counter to handle multiple changes in one day).
- 2010041601 ; Serial
+ 2010051000 ; Serial
28800 ; Refresh
14400 ; Retry
1000000 ; Expire
@@ -88,7 +88,7 @@ $TTL 86400
77 IN PTR x86-07-nfs.phx2.fedoraproject.org.
78 IN PTR x86-08-nfs.phx2.fedoraproject.org.
79 IN PTR x86-09-nfs.phx2.fedoraproject.org.
-80 IN PTR xenbuilder04-nfs.phx2.fedoraproject.org.
+80 IN PTR xb-01-nfs.phx2.fedoraproject.org.
81 IN PTR xen01-nfs.phx2.fedoraproject.org.
82 IN PTR xen02-nfs.phx2.fedoraproject.org.
83 IN PTR xen03-nfs.phx2.fedoraproject.org.
@@ -132,15 +132,15 @@ $TTL 86400
121 IN PTR bxen01-nfs.phx2.fedoraproject.org.
122 IN PTR bxen02-nfs.phx2.fedoraproject.org.
123 IN PTR bxen03-nfs.phx2.fedoraproject.org.
-124 IN PTR unused.
+124 IN PTR bxen04-nfs.phx2.fedoraproject.org.
125 IN PTR unused.
126 IN PTR unused.
127 IN PTR unused.
128 IN PTR unused.
129 IN PTR unused.
130 IN PTR unused.
-131 IN PTR unused.
-132 IN PTR unused.
+131 IN PTR ppc11-nfs.phx2.fedoraproject.org.
+132 IN PTR ppc12-nfs.phx2.fedoraproject.org.
133 IN PTR unused.
134 IN PTR unused.
135 IN PTR unused.
@@ -148,19 +148,19 @@ $TTL 86400
137 IN PTR unused.
138 IN PTR unused.
139 IN PTR unused.
-140 IN PTR unused.
-141 IN PTR unused.
-142 IN PTR unused.
-143 IN PTR unused.
-144 IN PTR unused.
-145 IN PTR unused.
-146 IN PTR unused.
-147 IN PTR unused.
-148 IN PTR unused.
-149 IN PTR unused.
-150 IN PTR unused.
-151 IN PTR unused.
-152 IN PTR unused.
+140 IN PTR x86-10-nfs.phx2.fedoraproject.org.
+141 IN PTR x86-11-nfs.phx2.fedoraproject.org.
+142 IN PTR x86-12-nfs.phx2.fedoraproject.org.
+143 IN PTR x86-13-nfs.phx2.fedoraproject.org.
+144 IN PTR x86-14-nfs.phx2.fedoraproject.org.
+145 IN PTR x86-15-nfs.phx2.fedoraproject.org.
+146 IN PTR x86-16-nfs.phx2.fedoraproject.org.
+147 IN PTR x86-17-nfs.phx2.fedoraproject.org.
+148 IN PTR x86-18-nfs.phx2.fedoraproject.org.
+149 IN PTR x86-19-nfs.phx2.fedoraproject.org.
+150 IN PTR x86-20-nfs.phx2.fedoraproject.org.
+151 IN PTR x86-21-nfs.phx2.fedoraproject.org.
+152 IN PTR x86-22-nfs.phx2.fedoraproject.org.
153 IN PTR unused.
154 IN PTR unused.
155 IN PTR unused.
diff --git a/modules/bind/files/master/phx2.fedoraproject.org b/modules/bind/files/master/phx2.fedoraproject.org
index 8a7dc9c..003656b 100644
--- a/modules/bind/files/master/phx2.fedoraproject.org
+++ b/modules/bind/files/master/phx2.fedoraproject.org
@@ -3,7 +3,7 @@ $TTL 3600
; Make SURE to update the serial # when changes are made. Format is
; YYYYMMDDNN (NN is just a counter to handle multiple changes in one
; day).
- 2010031701 ; Serial
+ 2010051000 ; Serial
28800 ; Refresh
14400 ; Retry
1000000 ; Expire
@@ -133,6 +133,7 @@ bxen02 IN A 10.5.125.122
bxen2 IN CNAME bxen02
bxen03 IN A 10.5.125.121
bxen3 IN CNAME bxen03
+bxen04 IN A 10.5.125.120
xen01 IN A 10.5.126.101
xen1 IN CNAME xen01
xen02 IN A 10.5.126.102
@@ -181,7 +182,19 @@ x86-7 IN CNAME x86-07
x86-08 IN A 10.5.125.18
x86-8 IN CNAME x86-08
x86-09 IN A 10.5.125.19
-x86-9 IN CNAME x86-09
+x86-10 IN A 10.5.125.80
+x86-11 IN A 10.5.125.81
+x86-12 IN A 10.5.125.82
+x86-13 IN A 10.5.125.83
+x86-14 IN A 10.5.125.84
+x86-15 IN A 10.5.125.85
+x86-16 IN A 10.5.125.86
+x86-17 IN A 10.5.125.87
+x86-18 IN A 10.5.125.88
+x86-19 IN A 10.5.125.89
+x86-20 IN A 10.5.125.90
+x86-21 IN A 10.5.125.91
+x86-22 IN A 10.5.125.92
ppc01 IN A 10.5.125.21
ppc1 IN CNAME ppc01
ppc02 IN A 10.5.125.22
@@ -201,6 +214,8 @@ ppc8 IN CNAME ppc08
ppc09 IN A 10.5.125.29
ppc9 IN CNAME ppc09
ppc10 IN A 10.5.125.30
+ppc11 IN A 10.5.125.31
+ppc12 IN A 10.5.125.32
compose-x86-01 IN A 10.5.125.41
compose-x86-1 IN CNAME compose-x86-01
compose-x86-02 IN A 10.5.126.126
@@ -294,6 +309,8 @@ ppc07-nfs IN A 10.5.127.47
ppc08-nfs IN A 10.5.127.48
ppc09-nfs IN A 10.5.127.49
ppc10-nfs IN A 10.5.127.50
+ppc11-nfs IN A 10.5.127.131
+ppc12-nfs IN A 10.5.127.132
puppet01-nfs IN A 10.5.127.51
puppet02-nfs IN A 10.5.127.52
releng01-nfs IN A 10.5.127.53
@@ -316,7 +333,20 @@ x86-06-nfs IN A 10.5.127.76
x86-07-nfs IN A 10.5.127.77
x86-08-nfs IN A 10.5.127.78
x86-09-nfs IN A 10.5.127.79
-xenbuilder04-nfs IN A 10.5.127.80
+x86-10-nfs IN A 10.5.127.140
+x86-11-nfs IN A 10.5.127.141
+x86-12-nfs IN A 10.5.127.142
+x86-13-nfs IN A 10.5.127.143
+x86-14-nfs IN A 10.5.127.144
+x86-15-nfs IN A 10.5.127.145
+x86-16-nfs IN A 10.5.127.146
+x86-17-nfs IN A 10.5.127.147
+x86-18-nfs IN A 10.5.127.148
+x86-19-nfs IN A 10.5.127.149
+x86-20-nfs IN A 10.5.127.150
+x86-21-nfs IN A 10.5.127.151
+x86-22-nfs IN A 10.5.127.152
+xb-01-nfs IN A 10.5.127.80
xen01-nfs IN A 10.5.127.81
xen02-nfs IN A 10.5.127.82
xen03-nfs IN A 10.5.127.83
@@ -339,6 +369,7 @@ xen19-nfs IN A 10.5.127.99
bxen01-nfs IN A 10.5.127.121
bxen02-nfs IN A 10.5.127.122
bxen03-nfs IN A 10.5.127.123
+bxen04-nfs IN A 10.5.127.124
;;
app01.stg IN A 10.5.126.81
app02.stg IN A 10.5.126.82
--
1.5.5.6
14 years
Re: Hello Everyone
by Abhishek Bose
>> Hi,
>> ?
>> I am Abhishek Bose from India.I have been using Fedora since Fedora 10
(pretty new in the group I guess). I am a Redhat Certified Engineer (RHCE)
and working in Redhat
>> Linux, Solaris, Netbackup since last?2.5 years but my main area of focus
is Redhat Linux. I am working for?a company as a Linux Admin where I have to
build and maintain
>> servers along with several applications running on it. I would like to
participate in?developing and helping in Fedora whenever and wherever
possible.
>> ?
>Welcome Abhishek! Was there a particular area in infrastructure you were
>interested in participating in?
> -Mike
Hi Mike
Right now I want to overlook the proceedings of the community and in the
meantime if there are some backend system administration jobs needs to be
done I will be more than happy to help.
Regards,
/Abhishek
14 years
[PATCH] handle Amazon EC2 client yum misconfiguration
by Matt Domsch
EC2 clients with malformed yum URLs are hitting us a lot. Kill those
requests at the proxy servers to keep them from hitting MM.
diff --git a/modules/mirrormanager/templates/mirrormanager-mirrorlist.conf.erb b/modules/mirrormanager/templates/mirrormanager-mirrorlist.conf.erb
index bd576ce..3b7fdcc 100644
--- a/modules/mirrormanager/templates/mirrormanager-mirrorlist.conf.erb
+++ b/modules/mirrormanager/templates/mirrormanager-mirrorlist.conf.erb
@@ -13,6 +13,9 @@ RequestHeader set CP-Location /mirrormanager
Alias /releases.txt /srv/web/mirrors/releases.txt
RewriteEngine On
+# BEGIN hack for misconfigured Amazon EC2 clients
+RewriteRule ^/mirrorlist?repo=epel-5&arch=\$basea\$ - [F]
+# END hack
RewriteRule ^/publiclist(.*) <%= proxyurl %>/publiclist/$1 [P,L]
RewriteRule ^/mirrorlist(.*) <%= proxyurl %>/mirrorlist$1 [P,L]
RewriteRule ^/metalink(.*) <%= proxyurl %>/metalink$1 [P,L]
--
Matt Domsch
Technology Strategist
Dell | Office of the CTO
14 years
[PATCH] bump mirrorlist processes
by Matt Domsch
We're running into a problem where we're seeing 30 requests/second for
the mirrorlists, but we only have 6 processes available for apache to
send those to.
This bumps up the number of processes, from 6 to 30, and cuts the
number of threads from 25 to 1 (it's a single-threaded requestor app,
and there's the python GIL still to deal with).
We may need to do more on the mirrorlist-server.py process to handle
the load (it fork()s on every connection), but this is a start...
diff --git a/modules/mirrormanager/files/mirrorlist-server.conf b/modules/mirrormanager/files/mirrorlist-server.conf
index a26e4b3..ba8e7ab 100644
--- a/modules/mirrormanager/files/mirrorlist-server.conf
+++ b/modules/mirrormanager/files/mirrorlist-server.conf
@@ -7,7 +7,7 @@ Alias /publiclist /var/lib/mirrormanager/mirrorlists/publiclist/
ExpiresDefault "modification plus 1 hour"
</Directory>
-WSGIDaemonProcess mirrorlist user=apache processes=6 threads=25 display-name=mirrorlist maximum-requests=1000
+WSGIDaemonProcess mirrorlist user=apache processes=30 threads=1 display-name=mirrorlist maximum-requests=1000
WSGIScriptAlias /metalink /usr/share/mirrormanager/mirrorlist-server/mirrorlist_client.wsgi
WSGIScriptAlias /mirrorlist /usr/share/mirrormanager/mirrorlist-server/mirrorlist_client.wsgi
--
Matt Domsch
Technology Strategist
Dell | Office of the CTO
14 years
Hello
by Ed Smith
Hello, All-
My name is Ed and I am interested in getting involved with the Fedora
project. To give you some of my background, I have a Bachelor's degree in
Electrical Engineering but have spent the majority of my professional career
in the realm of scientific application programming, system administration
and network security. I have contributed to open source projects in the
past mostly with code contributions that included random patch submissions
for the FreeBSD project. I am adept at most compiled/scripted languages to
include C, C++, ObjC, Python, Bash, etc with the exclusion of Java/C#.
After several years I am again in a position where I can contribute again to
the F/OSS community. In the past I mostly used the BSD flavor of UNIX OSes,
Debian and Solaris but I recently tried Redhat again after a number of years
and I am quite impressed with the latest iterations of the package. I'll be
lurking on the list and on the IRC channel from time to time if anyone is
interested in a somewhat veteran hand. As a note, in the past I had an
array of hardware and processor architectures that I could test various code
on but at the moment I am limited to one x86 desktop machine for any
development and testing. Thank you for your time and have a nice day.
V/R
Ed
14 years
Hello
by John Stanton
Hello,
I have been following the infrastructure list for sometime now. I am a
system admin and I have some basic scripting knowledge with
Perl/Python. I would like to help out with some of the sysadmin groups,
such as sysadmin-noc. How can I get involved? I also have the RHCE and
I am thinking about trying to get some additional Red Hat certs.
--
John Stanton, RHCE
john(a)stantonnet.org <mailto:john@stantonnet.org>
14 years
Another Hello
by Rob Marti
I've been lurking in #fedora-admin for a few days now, and figured I'd
introduce myself. My name is Rob Marti, I have my RHCE, and after
using Red Hat at my job for the past few years, I decided to get
involved with the Fedora Project. Since I'm not a really strong
coder, development is out, and being partially colorblind means web
design stuff is out. I'm okay with that, however, because I really
like dealing with the backend system administration stuff a lot more.
I think I'd like to be involved in sysadmin-noc at first, and maybe
expand from there into hosted and tools.
14 years
