Outage Notification - 2008-10-15 13:30 UTC
by Mike McGrath
There will be an outage starting at 2008-10-15 13:30 UTC, which will last
approximately 1 hour.
To convert UTC to your local time, take a look at
http://fedoraproject.org/wiki/Infrastructure/UTCHowto
or run:
date -d '2008-10-15 13:30 UTC'
Affected Services:
Buildsystem
Database
Fedora Hosted
Translation Services
Websites
Unaffected Services:
DNS
CVS / Source Control
Fedora People
Fedora Talk
Mail
Mirror System
Torrent
Ticket Link:
https://fedorahosted.org/fedora-infrastructure/ticket/911
Reason for Outage:
Migrating to postgres 8.3. This should require minutes of outage time.
We're going to do a dump, delete, update, import. Doing this one in the
morning because its short, so we can monitor it during the day instead of
have it fail at night, and so we don't interrupt any of the builds or
mashes that would happen in the evening. Actual downtime should be about
5-10 minutes
Contact Information:
Please join #fedora-admin in irc.freenode.net or respond to this email to
track the status of this outage.
15 years, 6 months
Maps
by Mike McGrath
Anyone interested in getting our fedoraproject.org/maps/ infrastructure
back in place? It was up and running in FC6, after the rebuild and no
more FC6 boxes though, we no longer have active maps.
-Mike
15 years, 6 months
pre-release freeze on the 21st
by Mike McGrath
Just a note to everyone, as long as we stay on schedule another
pre-release freeze will happen starting on the 21st ending on the 5th.
The final full infrastructure freeze does not begin until the 11th.
I'd also encourage those with the proper skill set to focus on rawhide
during these times, tackle the final bugs, focus on blockers, QA, etc.
-Mike
15 years, 6 months
Outage Notification - 2008-10-11 03:00 UTC
by Seth Vidal
Outage Notification - 2008-10-11 03:00 UTC
There will be an outage starting at 2008-10-11 03:00 UTC, which will last
approximately 2 hours.
To convert UTC to your local time, take a look at
http://fedoraproject.org/wiki/Infrastructure/UTCHowto
or run:
date -d '2008-10-11 03:00 UTC'
Affected Services:
Fedora People
Unaffected Services:
Buildsystem
CVS / Source Control
Database
DNS
Fedora Hosted
Fedora Talk
Mail
Mirror System
Torrent
Translation Services
Websites
Reason for Outage: Migrating fedorapeople.org from Duke University hosting to
Boston University hosting.
Contact Information:
Please join #fedora-admin in irc.freenode.net or respond to this email to track
the status of this outage.
15 years, 6 months
Meeting Log - 2008-10-09
by Ricky Zhou
20:00 -!- mmcgrath changed the topic of #fedora-meeting to: Infrastructure -- Who's here?
20:00 < G> boo
20:00 < dgilmore> not I
20:00 < jcollie> hello
20:00 < mdomsch> barely
20:00 * ValHolla grabs a seat
20:00 < mmcgrath> G: boo?
20:01 < mmcgrath> its not halloween yet!
20:01 < G> mmcgrath: :)
20:01 * ivazquez is here for once
20:01 < mmcgrath> heh, lets look at the tickets.
20:01 -!- mmcgrath changed the topic of #fedora-meeting to: Infrastructure -- The Dow is way down
20:01 -!- ValHolla [i=0c29727e(a)gateway/web/ajax/mibbit.com/x-d4aae136aecdd915] has quit Client Quit
20:02 -!- mmcgrath changed the topic of #fedora-meeting to: Infrastructure -- Tickets
20:02 < mmcgrath> .tiny https://fedorahosted.org/fedora-infrastructure/query?status=new&status=as...
20:02 < zodbot> mmcgrath: http://tinyurl.com/2hyyz6
20:02 -!- ValHolla [i=0c29727e(a)gateway/web/ajax/mibbit.com/x-8aa88275c5dc8cbb] has joined #fedora-meeting
20:02 < mmcgrath> .ticket 395
20:02 < zodbot> mmcgrath: #395 (Audio Streaming of Fedora Board Conference Calls) - Fedora Infrastructure - Trac - https://fedorahosted.org/fedora-infrastructure/ticket/395
20:02 < mmcgrath> jcollie: ping
20:02 < jcollie> nothin' :)
20:02 < mmcgrath> heheh k
20:02 < mmcgrath> .ticket 446
20:02 < zodbot> mmcgrath: #446 (Possibility to add external links on spins page) - Fedora Infrastructure - Trac - https://fedorahosted.org/fedora-infrastructure/ticket/446
20:02 < mmcgrath> dgilmore: ^^^
20:02 * f13 is here
20:03 * ValHolla sits back down
20:03 < kcadieux> hello
20:04 -!- JSchmitt [n=s4504kr(a)p4FDD213F.dip0.t-ipconnect.de] has joined #fedora-meeting
20:04 * ricky
20:05 < jcollie> i thought we were going to close this one?
20:05 < mmcgrath> we'll skip that for now
20:05 < mmcgrath> I thought so too.
20:05 * mmcgrath also hasn't heard much on
20:05 < mmcgrath> .ticket 740
20:05 < zodbot> mmcgrath: #740 (Loaning out system time to OLPC participants) - Fedora Infrastructure - Trac - https://fedorahosted.org/fedora-infrastructure/ticket/740
20:05 < mmcgrath> ok so
20:06 -!- mmcgrath changed the topic of #fedora-meeting to: Infrastructure -- App farm and new bapp servers
20:06 < mdomsch> yea!
20:06 < mmcgrath> Did anyone have any questions about how our new farm works, bapp, etc?
20:06 < mmcgrath> any comments?
20:06 -!- brothers [n=brothers(a)66-234-43-147.nyc.cable.nyct.net] has joined #fedora-meeting
20:06 -!- finalzone [n=luya@fedora/Finalzone] has joined #fedora-meeting
20:06 < mdomsch> bapp* will be in phx
20:07 < mdomsch> we expect
20:07 < mmcgrath> likely
20:07 < mdomsch> ?
20:07 < mdomsch> good
20:07 < mmcgrath> There's still some jobs to move to bapp1, in particular the damned lies stuff
20:08 < mmcgrath> K, well if there's nothing else there..
20:08 -!- mmcgrath changed the topic of #fedora-meeting to: Infrastructure -- telia1 and ibiblio1
20:08 < mmcgrath> We now have proxy servers in 4 locations.
20:09 < mmcgrath> 1 in germany, 3 in the states.
20:09 < mmcgrath> 5 proxy servers total (two of them are in PHX)
20:09 < mmcgrath> this will give us far more flexability during problems as well as distribute network load more evenly on release day.
20:09 < mmcgrath> it will also let us take a closer look at geoIP and DNS
20:09 < G> mmcgrath: what happens if WWI happens all over again? :)
20:10 < mmcgrath> we'll have to find an oil rig
20:10 < mmcgrath> or one of google's new cruise ships :)
20:10 < mmcgrath> At present, I'm thinking that 5 proxy servers more then suits our needs.
20:10 < G> sounds good, I thought maybe host a server in Iraq or something :)
20:11 < jcollie> G: i nominate anyone but me to go install that one
20:11 < mmcgrath> I'd like to focus on our caching policies and stuff. But I don't see a need to add additional proxy servers at this time. And if a new site comes up in asia, europe or south america, we may move one of our proxy servers there.
20:11 < G> mmcgrath: from an APAC view, I'd like to be bias and say, one somewhere out this way would be nice
20:11 < mmcgrath> G: yeah, that area could certainly use one
20:12 < jcollie> something in africa would be good too, but that'd be almost as fun as the iraq one
20:12 < mmcgrath> heh
20:12 < G> if we can get a site, Australia/New Zealand/Singapore/Japan or something we'd be covering all the regions
20:12 < mmcgrath> if there's nothing else there we'll move on :)
20:12 -!- mmcgrath changed the topic of #fedora-meeting to: Infrastructure -- Duke, planet, buildsys, people, torrent.
20:13 < jcollie> if fedora on olpc takes off there could be a lot if activity from africa
20:13 < G> isn't buildsys at SB now?
20:13 < mmcgrath> At this time the old buildsys and torrent boxes are no longer being used (hosted at duke)
20:13 < G> jcollie: true
20:13 < mmcgrath> G: it is.
20:13 < mmcgrath> seth is also working to move planet and people to a combined machine at Boston University.
20:13 < G> k, I was thinking I was going even more potty :)
20:13 < mmcgrath> this is for a couple of reasins, the primary one being that the hardware that people and planet is on right now are just woefully out of date.
20:13 < jcollie> are the systems at duke worth re-purposing?
20:14 < mmcgrath> jcollie: not that we can use but, IIRC, they've already been taken care of.
20:14 * mmcgrath is almost afraid to know how old that torrent box is.
20:14 < mmcgrath> but new torrent is in ibiblio
20:14 < mmcgrath> buildsys is at SB
20:14 < G> mmcgrath: loan boxes to OLPC :)
20:14 < mmcgrath> heh
20:15 < mmcgrath> so thats just kind of a heads up. I'm hoping to update our global map to reflect that soon as its completed.
20:15 -!- mmcgrath changed the topic of #fedora-meeting to: Infrastructure -- Puppet (sysctl)
20:15 -!- tibbs [n=tibbs@fedora/tibbs] has quit "Konversation terminated!"
20:16 < mmcgrath> For those of you that aren't familiar we added sysctl support to puppet a couple days back. I'm going to be making more change that as we go forward.
20:16 < mmcgrath> I'm also, very slowly, completing our conversion to module
20:16 < mmcgrath> s
20:16 < mmcgrath> Anyone have any questions on that? or anything thats come up since puppet training?
20:17 < ricky> We should push for high quality modules that we can make public
20:17 * jcollie says modules good
20:17 < mmcgrath> ricky: yeah
20:17 < ricky> Some modules, like the haproxy one contain Fedora-specific configs, so we need to think about how to do that the "puppet" way
20:17 < mmcgrath> ricky: well, even that module is useful. Just takes someone to alter that file (which they'd have to do anyway for their environment)
20:18 < mmcgrath> it gets trickier if haproxy required something strange that was specific to Fedora
20:18 < mmcgrath> Ok, anyone have anything else on that?
20:19 < mmcgrath> Ok, well thats really all I had this week.
20:19 -!- mmcgrath changed the topic of #fedora-meeting to: Infrastructure -- Open Floor
20:19 < mmcgrath> anyone have anything they'd like to discuss related to Infrastructure ?
20:19 < abadger1999> publictest machines.
20:19 < mmcgrath> abadger1999: sure, whats up?
20:20 < abadger1999> How many do we have up now? Do we have resources to make more?
20:20 < abadger1999> Is there a SOP? :-)
20:20 < ricky> The new "rule" is that they just stay out of PHX, right?
20:20 -!- JSchmitt [n=s4504kr@fedora/JSchmitt] has quit "Konversation terminated!"
20:20 < mmcgrath> I think we just have one or two up, and we can make more at telia1 and ibiblio1
20:20 < mmcgrath> abadger1999: no SOP really, just the kickstart and dns SOP's are needed.
20:21 -!- che [n=rkastl@redhat/che] has joined #fedora-meeting
20:21 < mmcgrath> abadger1999: do you want one? I can walk you through making one if you have any questions, or I can just throw one together.
20:22 < abadger1999> mmcgrath: You shouuld probably walk me through making one.
20:22 < jcollie> do you have to be in -main to create a new PT system?
20:22 < G> jcollie: yep
20:22 < G> jcollie: shall we say "damn" together ? :P
20:22 < abadger1999> I think dmalcolm's rpmgrok and rnorwood's amber have yet to land somewhere post-incident.
20:22 < abadger1999> ricky: Am I right on that?
20:22 < G> abadger1999: thats my understanding...
20:22 < ricky> Yup.
20:22 -!- fbijlsma [n=fbijlsma(a)92.50.110.191] has quit "Leaving"
20:23 < G> abadger1999: + QA don't have anywhere for testopia/etc\
20:23 < G> to errr test anyway
20:23 < mmcgrath> abadger1999: I'd assume it had ended up on pt15.
20:24 < ricky> Haha, we all logged in at the same minute :-)
20:24 < abadger1999> heh :-)
20:25 < abadger1999> Hmm.... Okay, there's plenty of free memory there right now.
20:25 < abadger1999> I'll get those guys using it and you can walk me through creating a new pt instance if they outgrow it.
20:26 < mmcgrath> sure thing
20:26 < mmcgrath> Ok, well, we're only about a half hour in. Does anyone else have anything to discuss? if not we'll close the meeting early
20:26 < abadger1999> Cool. That's all I have.
20:27 * G thinks
20:27 < ricky> Infrastructure contact info? :-)
20:28 < G> wwoods says they can't use Testopia because of legal
20:28 < mmcgrath> ricky: I actually have that. I'll email you after the meeting
20:28 < ricky> Coool
20:28 < G> wwoods: are you guys actually looking at any other options?
20:29 < ricky> Compose hosts stuff done? Will F10 be made entirely on our infrastructure? :-)
20:29 < wwoods> at the moment, no. dmalcolm was working on a new frontend that used the same database
20:29 < wwoods> but I haven't heard anything about that in a couple of months
20:29 < G> k, so I guess QA doesn't need a PT machine then
20:29 < wwoods> at the moment, no
20:30 < G> wwoods: I guess that means you guys want the QA wiki namespace?
20:30 * G recalls that as the alternative
20:31 < mmcgrath> well, if they do need one wwoods knows who to tell :)
20:31 < wwoods> too busy dealing with F10 release process stuff to mess with that now, but... exactly
20:31 < mmcgrath> Anyone have anything else to discuss? If not I'll close the meeting in 30
20:31 < G> mmcgrath: nope
20:32 < mmcgrath> 15
20:32 < G> except git is a pain...
20:32 < mmcgrath> yes it is :)
20:32 < mmcgrath> 5
20:32 < G> I symlink the wrong update script and it goes haywire :)
20:32 -!- mmcgrath changed the topic of #fedora-meeting to: Infrastructure -- Meeting End
15 years, 6 months
Puppet SELinux support and an introduction
by Sean E. Millichamp
Hi everyone,
I have been working on getting some SELinux support into Puppet, based
on some initial work by Frank Sweetser. I know the Fedora
Infrastructure team uses Puppet and I heard a rumor that there was
interest here in better SELinux support.
ReductiveLabs just cut a 0.24.6rc1 containing this support. If you
would like to test it I would certainly appreciate any feedback and
bugreports. I believe they intend to release 0.24.6 as soon as
possible, so the sooner the better.
Now a brief introduction, since I know that is the thing to do on this
mailing list:
I ran my first Linux-in-production system in 1996, started the switch to
Red Hat Linux 4.2 as soon as it was released, and have been using RHL,
RHEL, CentOS, and Fedora since on both servers and on my desktop (as my
exclusive OS since around 1999). These days I am a senior systems
architect for a web-based firm that did around 1.2 billion page views in
2007 supported by around 300 Linux servers running our various dev,
test, and prod environments. Some of the key technologies in use here
are Apache, Perl/mod_perl, MySQL, MySQL cluster, RH Cluster Suite/GFS,
Nagios, Cobbler/Kickstart for deployment, cfengine (and soon to be
Puppet) for configuration management. My current main task is
reimplementing our cfengine configurations in Puppet so I've been deep
in Puppet recently.
I have this idea that I would like to help on the Fedora Infrastructure
team, to help contribute something back to Fedora, but between job and
family feel that I probably don't have the time needed to dedicate to
it, so I had just been lurking here until now.
Cheers,
Sean
15 years, 6 months
Some architectural changes
by Mike McGrath
So we've finally completed some consolidation we've been working on for
the past couple of months. We've also added a new class of server (bappX
servers). What does this all mean?
Well, app1+ are now all identical, for the most part. This allows us to
think of our app servers as truly being part of a farm. The exception to
this right now is those apps requiring our nfs mount in PHX. I'm still
debating what to do about this but so far haven't really come up with a
good answer.
We'll be distributing load with haproxy. Just because an app is on a
server doesn't mean that it will get hit. This allows us to distribute
load intelligently and hopefully allow us to get local proxies preferring
local app servers so geoIP dns can be deployed.
The bapp servers (there's only 1 right now) will be our job control
servers. Right now we kind of just have cron jobs spread all over the
place and in some instances this has caused problems with production
traffic. For example, while damned lies is checking out and updating all
of its repos to get the latest translation stats for
http://translate.fedoraproject.org/ app1 and app2 would typically see high
load and if you happened to hit one of them while trying to get to pkgdb,
you might experience slow response time.
Now though we'll be running this job on bapp1 and pushing or pulling the
relevant content via rsync. We use this model pretty successfully for
other applications like mirrormanager and smolt, but now we have a
dedicated server for it. Additionally we can add the bapp servers to the
haproxy farms as a backup server, no traffic goes to it unless its needed.
-Mike
15 years, 6 months
New puppet trick - sysctl
by Mike McGrath
Just so you guys know we're using a sysctl plugin now so we can do things
like
sysctl { 'net.ipv4.ip_forward':
value => '1'
}
I'm going to go through tomorrow and convert some of our existing configs.
We don't use sysctl heavily but this will be handy in places like our
database servers where each db server has different settings depending on
the host.
-Mike
15 years, 6 months
netatalk for EPEL?
by Matt Domsch
Jiri, I see you're the owner of netatalk in Fedora. netatalk includes
the 'timeout' command which I'd like to use with some scripts in
Fedora Infrastructure, specifically some of MirrorManager's scripts.
Can I talk you into building and maintaining netatalk in EPEL, on EL-5
branch at least?
Thanks,
Matt
--
Matt Domsch
Linux Technology Strategist, Dell Office of the CTO
linux.dell.com & www.dell.com/linux
15 years, 6 months