[Fedora-infrastructure-list] Admin/infrastructure meeting log for 2006-06-22
Elliot Lee
sopwith at redhat.com
Thu Jun 22 22:30:43 UTC 2006
Hi all,
Attached is the log for today's meeting and the backup discussion that followed...
Please make sure you're on fedora-infrastructure-list (https://www.redhat.com/mailman/listinfo/fedora-infrastructure-list/),
since that's becoming the primary list for infrastructure discussions.
Thanks for everyone who made it to today's meeting! It's really neat to see all the projects that are moving along.
Best,
-- Elliot
-------------- next part --------------
Jun 22 15:56:42 <Sopwith> meeting in 5min :)
Jun 22 16:03:17 <Sopwith> Anyone home?
Jun 22 16:03:22 <lyz> hi
Jun 22 16:03:24 <iWolf> I'm here.
Jun 22 16:03:33 <iWolf> But only for a few.....
Jun 22 16:03:54 * lmacken
Jun 22 16:04:37 * dgilmore is here
Jun 22 16:07:59 <Sopwith> OK
Jun 22 16:08:26 <Sopwith> lyz: You're here two weeks in a row - that makes you a member :)
Jun 22 16:08:35 <lyz> rock on
Jun 22 16:08:40 <Sopwith> iwolf: proxy4 all done?
Jun 22 16:08:45 <iWolf> :(
Jun 22 16:08:59 <iWolf> app1 is done!
Jun 22 16:09:08 <Sopwith> yea!
Jun 22 16:09:09 <iWolf> proxy4 is updated, fstab fixed, kernel reinstalled, apache installed.
Jun 22 16:09:26 <iWolf> I need to reboot it to be sure my fstab stuff worked and to get the new kernel.
Jun 22 16:09:28 <iWolf> and then dist-conf the config.
Jun 22 16:09:31 <iWolf> Then it's done as well.
Jun 22 16:09:41 <iWolf> Oh, cleaned up the old cvs stuff from it as well.
Jun 22 16:09:53 <iWolf> I hope to do that tomorrow night, the weekend at the latest.
Jun 22 16:09:57 <Sopwith> Cool
Jun 22 16:10:11 <tgrman> since I'm new here, any URL that details which servers do what?
Jun 22 16:10:43 <Sopwith> tgrman: We have a web page for that, but it's currently private 'cause we're not sure it's a good idea to tell hackers how the machines are laid out...
Jun 22 16:10:55 <tgrman> yeah, that makes sense :)
Jun 22 16:11:15 <dgilmore> which i think is fair, and i dont have access to that info :)
Jun 22 16:11:33 <Sopwith> Basically, we have 4 proxy servers, 2 app servers, and a db server for the running some web apps, plus a cvs server, build machines, and a couple of misc administrative boxes
Jun 22 16:11:56 <mmcgrath> sorry, I'm here now
Jun 22 16:11:57 <mmcgrath> whats up?
Jun 22 16:12:02 <Sopwith> OK, so upgrades are going very smoothly except for the fact that I have done nothing about upgrading db1, mea culpa.
Jun 22 16:12:29 <Sopwith> mmcgrath: Just getting started. It's a lazy summer afternoon, no rush :)
Jun 22 16:12:41 * Sopwith sticks db1 upgrade back on his todo list
Jun 22 16:13:01 <Sopwith> Oh, we did have good news on the 'being able to access the BIOS' front ;-)
Jun 22 16:13:25 <iWolf> That will be quite nice!
Jun 22 16:13:25 <tgrman> upgrading DB servers is always fun, espicially if you want no downtime on the apps using them
Jun 22 16:13:36 <mmcgrath> warren: thanks for the perl-Net-SNMP push.
Jun 22 16:13:50 <mmcgrath> that we do.
Jun 22 16:13:52 <mmcgrath> still don't know why it matters, but it seems to.
Jun 22 16:13:53 <Sopwith> dgilmore: Where are you and Dan with the backup system? (For the benefit of others here who aren't in on the thread, which should be moved to the list now that it exists)
Jun 22 16:14:18 * dgilmore needs to subscibe to the list
Jun 22 16:14:42 <mmcgrath> Sopwith: Seth and I were talking about backs. His preference is something that supports an ssh tunnel. He doesn't care if backups at each site are local, or if we pull them to the phx colo.
Jun 22 16:14:42 <dgilmore> right now im looking at the options that are available to us
Jun 22 16:14:42 <mmcgrath> He's pushing amanda :-D
Jun 22 16:15:08 <Sopwith> BTW, fedora-infrastructure-list exists now - use it unless the topic really needs to be private
Jun 22 16:15:24 <mmcgrath> it'll take me a bit to get used to that I'm sure :-D
Jun 22 16:15:49 <Sopwith> dgilmore: Cool. Sounds like it's worth syncing up with skvidal & mmcgrath for their opinions.
Jun 22 16:15:49 <dgilmore> im considering bacula, amanda, rsync and scrips, rdiff-backup
Jun 22 16:16:01 <dgilmore> Sopwith: will do
Jun 22 16:16:05 <Sopwith> dgilmore: Maybe e-mail the list to let them know where things stand in more detail, and start to gather opinions.
Jun 22 16:16:33 <dgilmore> Sopwith: will do
Jun 22 16:16:35 <Sopwith> dgilmore: Sounds like you have a good number of solutions to choose from, nifty
Jun 22 16:16:48 <mmcgrath> brb
Jun 22 16:17:09 <Sopwith> firewalling stuff - lmacken has been looking at that
Jun 22 16:18:01 <iWolf> Sorry guys, I have to leave early. I will catch up via the logs....
Jun 22 16:18:22 <Sopwith> iwolf: OK, see ya
Jun 22 16:18:23 <lmacken> yeah
Jun 22 16:18:38 <lmacken> i've been playing around with pyroman (someone mentioned it during the last meeting, but i forget who)
Jun 22 16:19:15 <lmacken> i've defined a profile for all of our machines (in phx so far), and I'm currently working on porting our rc.firewall script over to pyroman configs
Jun 22 16:19:37 <lmacken> i'll drop a status update to infrastrucure-list once i have something we can play with
Jun 22 16:19:45 <Sopwith> Cool
Jun 22 16:20:38 <Sopwith> Documentation - I put the services list up with contact info under InfrastructurePrivate - feel free to fix it if you have access and see something wrong
Jun 22 16:21:50 <Sopwith> I also found that a backup & restore plan of some sort is already in the fedora-config module, in the access subdir
Jun 22 16:22:53 <Sopwith> Haven't documented dist-conf or fedora-config yet
Jun 22 16:23:55 <Sopwith> mmcgrath: OTRS queues - did you set the one up for the voting app? Can we get another one set up for cvs maintenance? (e.g. making branches for extras)
Jun 22 16:24:13 <Sopwith> rordway: ping on monitoring
Jun 22 16:24:41 <abadger1999> mmcgrath setup voting for me. It's working well.
Jun 22 16:25:39 <Sopwith> abadger1999: So there's an e-mail address set up that people can mail if they have problems or questions?
Jun 22 16:26:21 <abadger1999> voting at fedoraproject.org
Jun 22 16:26:32 <Sopwith> Awesome. I should list that on the services page.
Jun 22 16:26:35 <abadger1999> Or use the web interface.
Jun 22 16:26:59 <Sopwith> ja8sun did all the xferlog stuff, I think, except redirecting the alias. I can do that really quickly
Jun 22 16:27:12 <abadger1999> The voting application should be sending a message there if it has errors contacting the database or other stuff that requires manual intervention as well.
Jun 22 16:27:19 <mmcgrath> sorry back.
Jun 22 16:27:24 <mmcgrath> yes, the OTRS voting queues are up.
Jun 22 16:27:42 <mmcgrath> abadger1999 even has his app throwing erros to email voting at fedoraproject.org
Jun 22 16:28:48 <Sopwith> cool
Jun 22 16:31:25 <Sopwith> Other misc items: ja8sun was looking at the CentOS mirroring setup. Damian is "on vacation" (i.e. hacking on the hardware reporting tool :). i18n.redhat.com migration still lurches slowly ahead on internal threads.
Jun 22 16:31:44 <Sopwith> joe^? you here?
Jun 22 16:32:06 <Sopwith> He showed a bit of an interest in the account system last week...
Jun 22 16:33:20 <Sopwith> lyz: Hey, you there?
Jun 22 16:33:24 <lyz> yup
Jun 22 16:33:38 <Sopwith> lyz: Don't want to let you get lost in the shuffle - is there something you're working on or wanting to work on?
Jun 22 16:33:39 --- [lyz] (n=lyz at c-71-57-127-145.hsd1.il.comcast.net) : Tom
Jun 22 16:33:39 --- [lyz] #fedora-admin
Jun 22 16:33:39 --- [lyz] irc.freenode.net :http://freenode.net/
Jun 22 16:33:39 --- [lyz] is identified to services
Jun 22 16:33:39 --- [lyz] End of WHOIS list.
Jun 22 16:34:01 <lyz> Sopwith, not working on anything currently. Willing to work on anything
Jun 22 16:34:25 <Sopwith> :)
Jun 22 16:34:37 <lyz> got something for me?
Jun 22 16:35:23 <Sopwith> The account system and Extras package DB don't seem to have people are clearly on top of them. Do you have an interest in writing python+SQL webapp type stuff for them?
Jun 22 16:35:42 <mjk|wrk> I was going to ask about the extras pkg db
Jun 22 16:35:47 <Sopwith> I think last week I dropped the ball partly by not catching up with joe^ to tell him what was needed.
Jun 22 16:35:51 <Sopwith> mjk: Heh cool :)
Jun 22 16:36:04 <lyz> I've written a little in Python and SQL. If someone is watching me, I should be ok
Jun 22 16:36:28 <Sopwith> lyz: Oh, we're alll here to help :)
Jun 22 16:36:43 <lyz> cool
Jun 22 16:37:21 <Sopwith> I don't know if the Extras team is doing any actual work on the package DB, but someone needs to lead out in figuring out "what are we actually going to do" for both tasks, and doing requires more general e-mailing and leadership skills than any coding.
Jun 22 16:37:23 <mjk|wrk> I might be able to help there also, but I know no python...
Jun 22 16:37:30 <mjk|wrk> willing to learn tho ;)
Jun 22 16:37:52 <Sopwith> mjk|wrk: Are you involved in other aspects of Fedora anyways? I just figured you might have a reason why you perked up at mention of the pkg DB :)
Jun 22 16:38:15 <mjk|wrk> I help with the status report, which resulted in the FE pkg db
Jun 22 16:38:18 <mjk|wrk> threads
Jun 22 16:38:31 <Sopwith> oh, you are that dude! I like those reports
Jun 22 16:38:37 <mjk|wrk> plus other qa type things
Jun 22 16:38:58 <mjk|wrk> there is Christian too, he has been doing for the most part, but now we take turns
Jun 22 16:39:04 <Sopwith> Yea mon... Have you talked with wwoods at all to find out about how he might be able to help you with Extras QA?
Jun 22 16:39:16 <mjk|wrk> nope
Jun 22 16:39:42 <mjk|wrk> but will make a note to poke him on irc next time I see him
Jun 22 16:40:26 <abadger1999> lyz, mjk|wrk: I'm looking into upgrading CVS to a new VCS and I think I'm going to want to tie into the package DB.
Jun 22 16:40:32 <Sopwith> He recently started into a role at Red Hat as Mr. Fedora QA Guy, and I'm sure he'd like to know where things stand.
Jun 22 16:40:59 <lyz> abadger1999, email me the details on it
Jun 22 16:40:59 <dgilmore> back soon
Jun 22 16:41:39 <mjk|wrk> Sopwith: ok, I will make an effort to catch him
Jun 22 16:41:41 <Sopwith> lyz: So for next week, would you be comfortable putting together a wiki page for the Account System version 2.0 task, and starting to collect requirements from people via the mailing list?
Jun 22 16:41:42 <mjk|wrk> abadger1999: cool
Jun 22 16:41:46 <abadger1999> lyx, mjk|wrk: I'm writing up my thoughts on the wiki and I'll mail you a link when I have it written.
Jun 22 16:41:59 <lyz> Sopwith, sure
Jun 22 16:42:07 <lyz> brb
Jun 22 16:43:06 <Sopwith> lyz: If you want to work on the pkg db as well, that's fine, but mjk is interested in pkg db as well, so I figure it'd split things out a bit better if you can focus on account system. Is that OK with you?
Jun 22 16:43:59 <Sopwith> tgrman: While lyz is going to raid the kitchen, would you like to introduce yourself and tell us what you might want to work on? (Assuming that's what you're here for :)
Jun 22 16:44:19 <lyz> back
Jun 22 16:44:21 <tgrman> sure, that's why i'm here :-)
Jun 22 16:44:44 <lyz> Sopwith, I'm not familiar with pkg db, but I'm game
Jun 22 16:45:27 <Sopwith> Cool. I'll look forward to seeing your post on fedora-infrastructure-list :)
Jun 22 16:46:02 * mjk|wrk thinks he should sub to that list
Jun 22 16:46:08 <Sopwith> yup, definitely
Jun 22 16:46:15 --- Topic for #fedora-admin is This is the meeting place of the Fedora Infrastructure team - the system administrators of the Fedora Project | http://fedoraproject.org/wiki/Infrastructure | Regular meetings: http://fedoraproject.org/wiki/Infrastructure/Meetings
Jun 22 16:46:15 --- Topic for #fedora-admin set by nman64 at Thu Jan 5 17:03:19 2006
Jun 22 16:46:26 <Sopwith> Need to get someone to put that in the /topic
Jun 22 16:47:54 <tgrman> Curt Moore, used RH since 6.2, built lots of Beowulf clusters in the past using RH/Fedora, willing to work on whatever needs attention, what all is left outstanding?
Jun 22 16:49:13 <Sopwith> Assuming you've read the wiki pages and familiarized yourself with what's going on, have any of the existing projects popped out as ones that are interesting to you?
Jun 22 16:49:48 --> dferris (n=dferris at epiphany.colorado.edu) has joined #fedora-admin
Jun 22 16:50:48 <Sopwith> One grunt-work task that needs doing is routing tickets in OTRS and being the friendly smiley face that helps give quick responses :)
Jun 22 16:50:51 <mmcgrath> also discussing on the list is good too.
Jun 22 16:50:51 <Sopwith> dferris: heya!
Jun 22 16:51:24 <Sopwith> tgrman: Yea, make sure you're on the list
Jun 22 16:52:01 <dferris> sorry I'm late
Jun 22 16:52:53 <tgrman> Sopwith: yeah I've read through the wiki, any of the systems upgrade stuff would be interesting for starters unless there is something else more pressing
Jun 22 16:53:02 <Sopwith> dferris: we assigned you janitor duty, hope you don't mind :)
Jun 22 16:54:37 <Sopwith> tgrman: Hmm, so my personal preference is to have people do a few smaller things before giving them root on all the boxes, just to make sure they're not trying to rig our ongoing election ;-)
Jun 22 16:55:05 <mmcgrath> :-D
Jun 22 16:55:11 <mjk|wrk> lol
Jun 22 16:55:28 <dferris> I'm not rigging anything
Jun 22 16:55:29 <dferris> yet
Jun 22 16:55:36 <Sopwith> haha
Jun 22 16:56:10 <tgrman> Sopwith: yeah, that's understandable. :-) I manage a farm of servers, it's just something I'm good at.
Jun 22 16:56:32 <Sopwith> tgrman: Cool. So it sounds like you're more interested in the sysadmin side of things than development?
Jun 22 16:57:31 <dferris> I can do grunt work before doing anything imporant, that's no big deal
Jun 22 16:57:54 <Sopwith> dferris: We talked a bit about backup stuff earlier, and you're in on that I assume :)
Jun 22 16:58:21 <Sopwith> mmcgrath: I'm curious, what exactly does it take to give people admin access to OTRS? Sound like a decent first-step to you?
Jun 22 16:59:19 <Sopwith> (First step as far as getting people involved)
Jun 22 17:00:00 <mmcgrath> I'll just have to add them.
Jun 22 17:00:06 <mmcgrath> the admin interface is still fairly manual.
Jun 22 17:00:14 <Sopwith> erk, ok
Jun 22 17:00:20 <tgrman> Sopwith: yeah, for now at least. I also supervise lots of development at work so for me it's enjoyable to do more sysadmin type stuff rather than programming. But it's not all about my enjoyment, I'll do whatever needs to be done. I also have lots of experience devloping PHP/PGSQL/MYSQL apps.
Jun 22 17:00:26 <mmcgrath> I'm sure there's an easy way to fully incorperate it into the accounting system but it sounds like lots of people are going to need access to the admin interface.
Jun 22 17:00:40 <mmcgrath> like the website people for example, they're not in the sysadmin group but will need access.
Jun 22 17:00:57 <Sopwith> mmcgrath: Yea, I was just wondering - is there any way to authorize access via membership in a separate group ('otrsadmin' or something)
Jun 22 17:01:12 <rordway> Sopwith: sorry, had some system problems to attend to
Jun 22 17:01:26 <Sopwith> rordway: Hey man, party's over, go home
Jun 22 17:01:28 <Sopwith> (kidding!)
Jun 22 17:02:17 <Sopwith> We're about out of time, so it's understandable if some people need to wander off, but I'm interested in hearing about rordway's todo items from the past week...
Jun 22 17:03:30 <rordway> talked with mmcgrath about nagios, but from what I understand we're waiting for rhel4 upgrades
Jun 22 17:03:44 <rordway> so not much has happened
Jun 22 17:03:47 <Sopwith> Cool
Jun 22 17:04:07 <Sopwith> Did you get a chance to look at the fedora-metrics code?
Jun 22 17:04:20 <rordway> and I haven't had time to look at the metrics code, it's intersession so I've been busy with server maintenance
Jun 22 17:04:25 <dgilmore> back
Jun 22 17:04:34 <rordway> nope, not yet :-(
Jun 22 17:04:45 <Sopwith> dgilmore: dferris is here if you two wanna make evil plots about backup stuff...
Jun 22 17:05:24 <Sopwith> rordway: OK, cool. RHEL4 upgrades are pretty much my and iwolf's problem at this point - I'll really truly try to make the db1 upgrade happen soon.
Jun 22 17:05:26 <dgilmore> Sopwith: sure, once i work out how to insert 1 million votes for me in the election :D
Jun 22 17:05:28 <rordway> had to bring down our SAN today for microcode updates and put in a temporary firewall so I can migrate our bridging firewall for our *ick* Windows boxes from Debian to RHEL
Jun 22 17:05:32 <Sopwith> dgilmore: Haha
Jun 22 17:06:15 <mmcgrath> rordway: another way we could help is to get nrpe approved. https://bugzilla.redhat.com/bugzilla/show_bug.cgi?id=180092
Jun 22 17:06:44 <lyz> Gotta run, i'm on the fedora-infrastructure-list list now. I will send a message out about the Account System 2.0 tonight
Jun 22 17:06:45 <rordway> mmcgrath: cool, I'll take a look at that after I eat lunch... yeah, it's been that kind of day ;-)
Jun 22 17:06:55 <Sopwith> lyz: Woohoo, thanks & see you.
Jun 22 17:06:55 <-- ozgur has quit ("exitus immortalus...")
Jun 22 17:07:03 <lyz> Let me know if there's anything else I can do
Jun 22 17:07:05 <lyz> c ya
Jun 22 17:07:16 <-- lyz has quit ("Leaving")
Jun 22 17:07:36 <lmacken> Sopwith: any idea what watchsyn is and where it might live ?
Jun 22 17:07:43 <rordway> what is the listserv for fedora-infrastructure-list?
Jun 22 17:08:02 <Sopwith> https://www.redhat.com/mailman/listinfo/fedora-infrastructure-list/
Jun 22 17:08:14 <Sopwith> lmacken: No clue at all
Jun 22 17:08:25 <Sopwith> lmacken: Could it be an iptables module?
Jun 22 17:08:41 <lmacken> Sopwith: maybe ? this is in rc.firewall:
Jun 22 17:08:42 <lmacken> # This chain is updated by the watchsyn script, so by default it is
Jun 22 17:08:42 <lmacken> # just there. New IP addresses will get added and their traffic will
Jun 22 17:08:42 <lmacken> # get dropped.
Jun 22 17:08:56 <Sopwith> Maybe the script watches for SYN flooders.
Jun 22 17:09:04 <Sopwith> We sure don't run any watchsyn script that I know of...
Jun 22 17:09:10 <lmacken> heh
Jun 22 17:09:13 <lmacken> meh, oh well
Jun 22 17:09:36 <mjk|wrk> locate watchsyn ?
Jun 22 17:11:02 <dgilmore> dferris: do you have any backup stuff you like?
Jun 22 17:11:27 <Sopwith> FWIW, I think dferris was the one who first suggested bacula...
Jun 22 17:11:35 <dferris> yeah
Jun 22 17:11:39 <dferris> I use it here at work
Jun 22 17:11:44 <rordway> Sopwith: I'm going to go forage for some lunch, I'll get back in touch with you once I get a chance to look through the metrics code
Jun 22 17:11:49 <dferris> it's complicated to setup, but when it runs it rocks
Jun 22 17:11:53 <Sopwith> rordway: Cool stuff, thanks
Jun 22 17:12:02 <Sopwith> In case it matters to anyone, I think the main meeting is over :)
Jun 22 17:12:05 <rordway> and I'm on the fedora-infrastructure-list now too
Jun 22 17:12:06 <dgilmore> ahh yeah
Jun 22 17:12:11 <mmcgrath> heh
Jun 22 17:12:24 <mmcgrath> Sopwith: do we have the ability to give non sudo access to the machines yet?
Jun 22 17:12:26 <dgilmore> I need to spend some time and finish packaiging up for extras
Jun 22 17:12:30 <dgilmore> Sorry dferris i forgot
Jun 22 17:12:56 <Sopwith> mmcgrath: Would you give an example scenario?
Jun 22 17:13:12 >chanserv< info #fedora-admin
Jun 22 17:13:13 -ChanServ- Channel: #fedora-admin
Jun 22 17:13:13 -ChanServ- Contact: nman64, last seen: 3 days (0h 34m 38s) ago
Jun 22 17:13:13 -ChanServ- Alternate: Sopwith << ONLINE >>
Jun 22 17:13:13 -ChanServ- Registered: 23 weeks 6 days (23h 19m 2s) ago
Jun 22 17:13:13 -ChanServ- Topic: This is the meeting place of the Fedora Infrastructure team - the system administrators of the Fedora Project | http://fedoraproject.org/wiki/Infrastructure | Regular meetings: http://fedoraproject.org/wiki/Infrastructure/Meetings
Jun 22 17:13:13 -ChanServ- Email: sysadmin-members at fedora.redhat.com
Jun 22 17:13:13 -ChanServ- Contact URI: http://fedoraproject.org/wiki/Infrastructure
Jun 22 17:13:13 -ChanServ- Options: Secure, SecureOps, ChanGuard
Jun 22 17:13:13 -ChanServ- Mode Lock: -s+ntc
Jun 22 17:13:23 >chanserv< help
Jun 22 17:13:23 -ChanServ- ChanServ allows you to register and control various
Jun 22 17:13:23 -ChanServ- aspects of channels. ChanServ can often prevent
Jun 22 17:13:23 -ChanServ- malicious users from "taking over" channels by limiting
Jun 22 17:13:23 -ChanServ- who is allowed channel operator priviliges. Any channel
Jun 22 17:13:23 -ChanServ- which is not used for 120 days will be expired and may
Jun 22 17:13:23 -ChanServ- be dropped. ChanServ's commands are listed below.
Jun 22 17:13:23 -ChanServ- For more information on a specific command, type
Jun 22 17:13:23 -ChanServ- /msg ChanServ help <command>.
Jun 22 17:13:23 -ChanServ-
Jun 22 17:13:23 -ChanServ- REGISTER Register a channel
Jun 22 17:13:23 -ChanServ- DROP Cancel the registration of a channel
Jun 22 17:13:23 -ChanServ- IDENTIFY Identify yourself with your password
Jun 22 17:13:23 -ChanServ- SET Set various channel options
Jun 22 17:13:23 -ChanServ- ACCESS Modify the list of privileged users
Jun 22 17:13:23 -ChanServ- AUTOREM Maintain the AutoRemove list
Jun 22 17:13:23 -ChanServ- LEVEL Change the level required for functions
Jun 22 17:13:23 -ChanServ- LIST Display list of channels matching a pattern
Jun 22 17:13:23 -ChanServ- INFO Display information for a channel
Jun 22 17:13:23 -ChanServ- GETKEY Retrieve the key (+k) to a channel
Jun 22 17:13:23 -ChanServ- INVITE Invite yourself to a channel
Jun 22 17:13:23 -ChanServ- OP Op yourself on a channel
Jun 22 17:13:23 -ChanServ- VOICE Voice yourself on a channel
Jun 22 17:13:23 -ChanServ- UNBAN Unban yourself on a channel
Jun 22 17:13:23 -ChanServ- CLEAR Clear various channel modes
Jun 22 17:13:32 >chanserv< op #fedora-admin
Jun 22 17:13:32 --- ChanServ sets modes [#fedora-admin +o Sopwith]
Jun 22 17:13:34 --- Topic for #fedora-admin is This is the meeting place of the Fedora Infrastructure team - the system administrators of the Fedora Project | http://fedoraproject.org/wiki/Infrastructure | Regular meetings: http://fedoraproject.org/wiki/Infrastructure/Meetings
Jun 22 17:13:34 --- Topic for #fedora-admin set by nman64 at Thu Jan 5 17:03:19 2006
Jun 22 17:13:50 <mmcgrath> I was just thinking instead of giving people straight sudo access to the boxes we could implement some non sudo access to get familiar with the boxes
Jun 22 17:14:37 --- Sopwith has changed the topic to: This is the meeting place of the Fedora Infrastructure & Sysadmin team | http://fedoraproject.org/wiki/Infrastructure | Regular meetings: http://fedoraproject.org/wiki/Infrastructure/Meetings | https://www.redhat.com/mailman/listinfo/fedora-infrastructure-list/
Jun 22 17:14:44 <Sopwith> mmcgrath: That makes sense.
Jun 22 17:15:18 <dgilmore> Sopwith: how much storage space do we have on our backup box?
Jun 22 17:16:27 <f13> I use bacula too, and I would be able to help out in setting it up. I set up a bacula system for Pogo Linux and Real Networks
Jun 22 17:16:34 <Sopwith> dgilmore: datadump has 3.0T, of which there is 112G free. (I think datadump will wind up dying someday eventually, since its role is not 100% clear)
Jun 22 17:17:16 <Sopwith> There is also ext-backup, which works a lot like datadump except its disk array gets backed up to tape in turn (by the RH sysadmin team)
Jun 22 17:17:20 <mmcgrath> can bacula tunnel over ssh automatically or should we setup a tunnel?
Jun 22 17:17:31 <dferris> bacula can use SSL
Jun 22 17:17:34 <mmcgrath> which brings me to another question... should we implement OpenVPN for our multiple sites?
Jun 22 17:17:34 <dferris> natively
Jun 22 17:17:42 <f13> dferris: thats not a tunnel
Jun 22 17:17:51 <mmcgrath> SSL as in we'll have to make firewall changes all over the place?
Jun 22 17:18:09 <f13> if you want to tunnel, you have to setup the tunnel yourself. However there is a good facility for pre and post backup commands.
Jun 22 17:18:39 <f13> mmcgrath: what are you looking to tunnel through? Just shove all bacula communication through port 22 to avoid firewall changes?
Jun 22 17:18:50 <tgrman> I use SSH tunnels for my DB backups, they work well
Jun 22 17:18:53 <f13> mmcgrath: or is there concerns about having the backup unit outside the local network to the builders?
Jun 22 17:18:54 <dferris> I think you could
Jun 22 17:19:05 <dferris> it uses TCP ports
Jun 22 17:19:12 <dferris> I think 9102 is the default
Jun 22 17:19:19 <f13> there are three ports
Jun 22 17:19:27 <Sopwith> The network the backup units are on has full access to all the Fedora systems, AFAIK
Jun 22 17:19:28 <f13> or two that the clients have to worry about
Jun 22 17:19:38 <Sopwith> (except as denied by per-system iptables, which is easy to fix)
Jun 22 17:19:38 <f13> one port for the director, and one port for the storage.
Jun 22 17:19:46 <mmcgrath> f13: i'm worried about other backups too, like those from the duke servers.
Jun 22 17:19:53 <f13> (as director box can be completely seperate from the storage box(es))
Jun 22 17:20:34 <dgilmore> mmcgrath: i agress we should be backing up all boxes, duke ones as wel as ones in RH facilities
Jun 22 17:21:04 <mmcgrath> but we can coordinate them seperately. We could have the duke backups onsite at duke and the RH ones on site in PHX.
Jun 22 17:21:17 <f13> yeah
Jun 22 17:21:28 <mmcgrath> we really need to guard the backups closely though.
Jun 22 17:21:32 <f13> the same director can direct the backups, but the duke boxen could just backup to a localized storage unit
Jun 22 17:21:38 <dgilmore> http://darcs.complete.org/debian/bacula/examples/ssh-tunnel-README.txt
Jun 22 17:21:48 <f13> the communication from director to client can be ssh tunneled.
Jun 22 17:22:10 <f13> dgilmore: I used that some, but I had some issues with hit
Jun 22 17:22:14 <dferris> you have to use the pre backup scripts to bring up the tunnel
Jun 22 17:22:14 <mmcgrath> In that case bacula sounds fine to me, I'm just not that familiar with it.
Jun 22 17:22:26 <dgilmore> f13: ok
Jun 22 17:22:27 <f13> with sending large data down the tunnel. THe tunnel would die or falter, and the backup would get stuck.
Jun 22 17:22:44 <mmcgrath> :: cough :: openvpn :-DɎM
Jun 22 17:22:53 <f13> heh
Jun 22 17:22:58 <Sopwith> mmcgrath: What are the security implications of that?
Jun 22 17:23:02 <dgilmore> openswan :D
Jun 22 17:23:31 <Sopwith> My secret fear is someone breaking into the Red Hat machines in PHX via the Fedora boxes, and Fedora losing all its ability to host cool stuff as a result...
Jun 22 17:24:02 <dgilmore> that would not be good
Jun 22 17:24:36 <f13> Sopwith: how many of the machines need to directly contact RH machines? Is there not a reason why the Fedora network could be segregated?
Jun 22 17:24:37 <Sopwith> That's why a VPN connection between Duke and PHX scares me a tiny bit (although if there are real benefits, we can probably afford to take the risk)
Jun 22 17:24:48 <mmcgrath> thats a valid fear
Jun 22 17:24:50 <Sopwith> f13: They are segregated, but not 100%
Jun 22 17:24:59 <f13> Sopwith: whats the blocker on 100%?
Jun 22 17:25:06 <mmcgrath> we could protect against it though.
Jun 22 17:25:28 <mmcgrath> it'd be nice to have an encrypted channel between the two sites.
Jun 22 17:25:43 <Sopwith> f13: For example, the physical netapp that hosts some Fedora stuff in PHX also hosts some stuff for other RH stuff. The Fedora machines can't read/write to the other stuff, but how much do you trust OnTap?
Jun 22 17:26:50 <dgilmore> What do we have at Duke? plague-server, wiki, ?
Jun 22 17:26:50 <Sopwith> dgilmore/dferris: Anyways, sorry about that tangent - you two are the ones I consider as being responsible for getting us all to decide on the best backup solution and then implement it.
Jun 22 17:27:04 <mmcgrath> and transfer of the fedora accounting stuff.
Jun 22 17:27:06 <Sopwith> plague-server, a build machine or two, wiki and a few other web pieces.
Jun 22 17:27:18 <Sopwith> Yea, but the account system transfer is only one-way right now.
Jun 22 17:27:24 <mmcgrath> thats true
Jun 22 17:27:34 <tgrman> IMHO, probably the way to go would be the SSH tunnel, that way the secure connection is opened during the backup and closed once it's done. That way you don't have the constant trusted connection like with the VPN.
Jun 22 17:27:43 <Sopwith> tgrman: Hmm, good point.
Jun 22 17:27:56 <dferris> be right back
Jun 22 17:28:21 <mmcgrath> we can always start with the tunnel and if its not working we can move to something more robust.
Jun 22 17:28:38 <dgilmore> we could rsync over a ssh tunnel data from duke.
Jun 22 17:28:43 <f13> Sopwith: ok, storage is a valid segregation point.
Jun 22 17:29:14 <dgilmore> Sopwith: of that 3TB how much is the current backups?
Jun 22 17:29:21 <tgrman> mmcgrath: sounds like a plan to me. I've used SSH tunnels for the past few years to to backups, rsync, etc and they work very well
Jun 22 17:29:37 <Sopwith> f13: It's not that I and Stacy haven't worked to segregate things as much as possible, it's just that there are things like the netapp and the cyclades that we can't split because we can't really justify buying separate ones for Fedora
Jun 22 17:29:42 <Sopwith> (Although that may change)
Jun 22 17:29:46 <f13> tgrman: on demand ssh tunnels can be fragile though. Just to keep in mind.
Jun 22 17:29:51 <Sopwith> hmm, ext-backup is down
Jun 22 17:29:58 <f13> Sopwith: indeed.
Jun 22 17:30:04 <mmcgrath> I'll donate a cyclades!
Jun 22 17:30:13 <mmcgrath> just kidding.
Jun 22 17:30:47 <Sopwith> I think GIS's intent is to have separate pieces at some point (the cyclades in particular shouldn't be too hard to find the money for), but companies are slower than us :)
Jun 22 17:31:32 <Sopwith> dgilmore: doing a du right now to see how much space the current copies of the Fedora-related data takes.
Jun 22 17:31:38 <dgilmore> Im keeping my T1000 to myself
Jun 22 17:31:43 <Sopwith> hah
Jun 22 17:31:46 <dgilmore> Sopwith: thanks
Jun 22 17:31:48 <dferris> must be nice
Jun 22 17:32:18 <dgilmore> dferris: sun donated it to me for aurora it arrives toomoorow
Jun 22 17:32:33 <Sopwith> dgilmore: Neat, you're involved in aurora?
Jun 22 17:32:50 <dgilmore> Sopwith: yeah
Jun 22 17:33:14 <dgilmore> Sopwith: im responsible for building extras for aurora
Jun 22 17:33:50 <dferris> dgilmore: if you get tired if it, I'll send you my shipping address
Jun 22 17:34:22 <Sopwith> dgilmore: This du might take a while - I'll have to email it to you.
Jun 22 17:34:30 <dgilmore> Sopwith: cool
Jun 22 17:34:34 <dgilmore> dferris: unlikely
Jun 22 17:34:39 <dferris> lol
Jun 22 17:34:40 <dferris> :)
Jun 22 17:40:04 <dferris> Sopwith: I think before we decide on what to do for backups we need to know how much we need to back up, what boxes/files need to be backed up, and where we're going to store it all. The I think we can figure out which software will best suit our needs.
Jun 22 17:40:20 <Sopwith> ok, all good questions
Jun 22 17:40:23 <Sopwith> oh wait
Jun 22 17:40:25 <Sopwith> du came back
Jun 22 17:40:32 <Sopwith> 2.4G cvs.fedora.redhat.com/cvsroot
Jun 22 17:40:32 <Sopwith> 120G cvs.fedora.redhat.com/homeroot
Jun 22 17:40:32 <Sopwith> 28G cvs.fedora.redhat.com/reporoot
Jun 22 17:40:32 <Sopwith> 11M db1.fedora.phx.redhat.com/db
Jun 22 17:40:32 <Sopwith> 1.9M db1.fedora.phx.redhat.com/db_mysql
Jun 22 17:40:43 <Sopwith> what in the world is in homeroot?
Jun 22 17:41:22 <dgilmore> so most of that 3T is not our data?
Jun 22 17:41:30 <Sopwith> Correct.
Jun 22 17:41:37 <Sopwith> There is internal stuff also backed up to that box.
Jun 22 17:42:00 <Sopwith> Oh, and I daresay that all that space used by homeroot will go away once I rm -rf it
Jun 22 17:42:10 <dferris> lol
Jun 22 17:42:26 <Sopwith> It's mainly a bunch of .src.rpm's that gafton imported in September 2004
Jun 22 17:43:10 <tgrman> also things to keep in mind, is there a need for intermediate backups on disk that get archived off to tape at some interval? otherwise recovery takes a bit longer since you have to go find the tape, etc.
Jun 22 17:43:26 <dgilmore> so we probably have 32GBor so
Jun 22 17:43:38 <dferris> do we have a tape drive or library avaliable?
Jun 22 17:43:40 <mmcgrath> Hey guys, I gotta run, be back on later.
Jun 22 17:43:48 <Sopwith> mmcgrath: later
Jun 22 17:44:03 <f13> dferris: tape isn't going to help much
Jun 22 17:44:03 <Sopwith> dferris: Not directly. I *heart* my rotate-with-hardlinks solution though :)
Jun 22 17:44:06 <dgilmore> bye mmcgrath
Jun 22 17:44:11 <dferris> lol
Jun 22 17:44:11 <f13> dferris: there isn't anybody there to rotate tapes.
Jun 22 17:44:33 <f13> Sopwith: Bacula can _really_ solve that. it does disk volume files that can be created on the fly, and recycled logically
Jun 22 17:44:35 <dferris> good because I hate dealing with my tape library here
Jun 22 17:44:51 <f13> dferris: I loved the two tape libraries I've worked with.
Jun 22 17:46:06 <dferris> f13: the library works fine, I just don't like dealing with rotating all the tapes, we have more than will fit in the library so I have to import and export tapes. :)
Jun 22 17:46:56 <dferris> and yeah, bacula works great with disk volumes
Jun 22 17:48:29 <f13> dferris: heh, my weekly tape backups included logic to mark the last used tape as full and prep it for ejection. On monday I'd shuffle a few tapes adn walk away. Another cron job would come along and find the new tapes. I never had to touch the console.
Jun 22 17:50:41 <dferris> sounds like I have more work to do :)
Jun 22 17:52:04 <-- ChanServ has quit (zelazny.freenode.net irc.freenode.net)
Jun 22 17:54:00 --> ChanServ (ChanServ at services.) has joined #fedora-admin
Jun 22 17:54:00 --- irc.freenode.net sets modes [#fedora-admin +o ChanServ]
Jun 22 17:54:00 --- ChanServ sets modes [#fedora-admin -o Sopwith]
Jun 22 17:54:54 <dgilmore> So what else needs discussing?
Jun 22 17:55:59 <f13> dferris: bacula scripting is fun. I did all mine in python.
Jun 22 17:56:33 <Sopwith> dgilmore: Is it going to make our backups easier to manage? How does the configuration and administration side of bacula work?
Jun 22 17:57:02 <dferris> f13: I'll have to read more on it. It's been working well enough for the past 2 months that I have left it alone to make sure all the different periods are set right
Jun 22 17:57:38 <dferris> dgilmore: Everything is configured through the config files
Jun 22 17:57:41 <dferris> there are 3 of them
Jun 22 17:58:02 <dferris> most of the admin work for the backup jobs is done in the bacula director config file, like adding jobs, scheduling, etc
Jun 22 17:59:50 <dgilmore> Sopwith: I dont know if bacula will make things easier
Jun 22 17:59:53 <f13> yeah, client side is fire and forget
Jun 22 18:00:11 <f13> all the heavy lifting happens in teh director.conf and possibly the storage.conf
Jun 22 18:00:29 <dferris> 90% of the config is done in the bacula-dir.conf
Jun 22 18:00:33 <f13> but the client just needs to know how to communicate with the director. The director directs the client in everything it needs to do.
Jun 22 18:00:39 <dferris> there are some things that are a pain
Jun 22 18:00:43 <dferris> like adding new hosts
Jun 22 18:00:45 <f13> there is a nice 900 page manual for it too (:
Jun 22 18:00:58 <f13> dferris: a properly formatted bacula-dir.conf file makes adding hosts a breeze
Jun 22 18:01:07 <f13> I sorted mine in a way that made very logical sense.
Jun 22 18:01:45 <dferris> f13: I organized my file, it takes some copying and pasting
Jun 22 18:01:47 <dferris> that's it
Jun 22 18:02:27 <f13> y5j p cw <newname>
Jun 22 18:02:32 * f13 loves vim (:
Jun 22 18:02:54 * dgilmore loves vim also
Jun 22 18:03:00 <lmacken> mmm.. vim7 tabs..
Jun 22 18:03:06 <lmacken> so hot
Jun 22 18:03:14 <dgilmore> I noticed the other week spot was uning nano
Jun 22 18:03:23 * lmacken pukes all over the place
Jun 22 18:03:48 <f13> spot is special
Jun 22 18:04:26 <dgilmore> it was his box so he can do what he wants
Jun 22 18:05:33 <-- tibbs has quit (Remote closed the connection)
Jun 22 18:07:02 <abadger1999> Sopwith: As I was writing the voting app I wondered if we could have access to some web frameworks to make it easier to produce.
Jun 22 18:09:51 <Sopwith> dferris: I think the most important thing is having a central file where we can look to see what's getting backed up, and change it easily
Jun 22 18:09:58 <Sopwith> abadger1999: TurboGears
Jun 22 18:10:09 <abadger1999> Sopwith: Is it on the app servers?
Jun 22 18:10:14 <Sopwith> No :(
Jun 22 18:10:19 <dferris> yeah, that's no problem with bacula
Jun 22 18:10:38 <abadger1999> Sopwith: Humbug. :-(
Jun 22 18:11:07 <dferris> but first, we need the list of things to back up. Then we can figure out if we want to use bacula :)
Jun 22 18:11:12 <f13> Sopwith: not only can you look at a file, you can run a cli tool and issue commands to see whats coming up.
Jun 22 18:11:22 <f13> bacula has a pretty cool console tool
Jun 22 18:11:33 <f13> and an even more crackrock gnome applet to tell you whats going on.
Jun 22 18:11:45 <Sopwith> hehe, ok
Jun 22 18:12:17 <Sopwith> abadger1999: One of my two complaints with TG is that it's a real pain to install. I don't know if it's packaged in Extras, but that'd seem like an interesting project for someone.
Jun 22 18:12:31 <abadger1999> Sopwith: It's in Extras.
Jun 22 18:13:25 <abadger1999> Sopwith: I've got it installed but haven't had a project I could use it on yet.
Jun 22 18:13:36 <abadger1999> Sopwith: What's the other complaint?
Jun 22 18:13:49 <Sopwith> Learning curve is really steep
Jun 22 18:13:50 --> lyz (n=lyz at dsl081-149-006.chi1.dsl.speakeasy.net) has joined #fedora-admin
Jun 22 18:14:10 <Sopwith> Once you get past those two things, it's great.
Jun 22 18:14:42 <abadger1999> Hmmm... I've used all the pieces except kid before so I think I'd have fun :-)
Jun 22 18:15:08 * f13 goes home
Jun 22 18:15:27 <abadger1999> Does it have to startup its own server and you need to Proxy it with Apache directives?
Jun 22 18:15:48 <Sopwith> That was my impression from the app I wrote with it
Jun 22 18:16:10 <Sopwith> But that'll work fine with our Multitier Web Architecture(tm)
Jun 22 18:16:18 * Sopwith always gets a kick out of saying that.
Jun 22 18:16:19 <mjk|wrk> TG looks nice
Jun 22 18:16:27 <abadger1999> Yeah -- that's what I had to do to get a couple cherrypy apps to work.
Jun 22 18:17:38 <abadger1999> Sopwith: Would it be possible to install it to the app servers or would things like that make it a show stopper for deploying there?
Jun 22 18:18:04 <Sopwith> abadger1999: I think we could install it as long as we have packages that will run on RHEL-4.
Jun 22 18:18:20 <Sopwith> Usually FC-3 packages will run fine on RHEL-4.
Jun 22 18:18:36 <abadger1999> \me thinks of accounts systems rewrites and package db web interfaces that could benefit.
Jun 22 18:19:13 <Sopwith> yup
Jun 22 18:20:36 <abadger1999> Looks like TurboGears was first built for FE4.
Jun 22 18:20:59 <dferris> Sopwith: I have to jet, I'll be in touch over the backups
Jun 22 18:21:23 <Sopwith> dferris: Cool stuff, thanks for making it :)
Jun 22 18:21:55 <dferris> Sopwith: np, I'll try no to be late next week, but I have to take the time to torture the grad students :)
Jun 22 18:22:06 <-- dferris has quit ("Ex-Chat")
Jun 22 18:22:51 <Sopwith> The main problem is that FESCO has this somewhat arbitrary and kooky policy about not making branches < FC-4, so it can be painful
More information about the infrastructure
mailing list