New subject: Firewall (WAS: IRC Log for Fedora Infrastructure Meeting (2007-06-14))

Thursday, 14 June 2007

[15:06] mmcgrath has set the subject to fedora-infrastructure meeting --  Who's here?
[15:06] xDamox: Me
[15:06] mmcgrath: PING ALL: who's here?
[15:06] G: I'm here
[15:06] mbonnet: yo
[15:06] cemc has joined the group chat (n=gimre(a)voy.narancs.net)
[15:06] G: I'm gonna have to disappear in 15-20
[15:06] jcollie: howdy!
[15:06] f13: I'm here.
[15:06] mmcgrath: abadger1999: you around?
[15:06] abadger1999: Yep.
[15:06] mmcgrath: allrighty then
[15:07] * lmacken !
[15:07] * wolfy grabs a viewing seat
[15:07] mmcgrath has set the subject to abadger1999 - Package Database -------------
[15:07] mmcgrath: abadger1999: whats the word?
[15:07] abadger1999: We have progress 
[15:07] abadger1999: Script to import pkgs is written and I'm finding all sorts of
problem with owners.list.
[15:08] mmcgrath: thats good at least 
[15:08] mmcgrath: how bad is it?
[15:08] abadger1999: Mostly lack of EPEL owner information for EPEL branches
[15:08] lmacken: abadger1999: is there an owners API yet?  If not, want some help?
[15:08] abadger1999: lmacken: I'm using nottings owners.py.
[15:09] lmacken: abadger1999: ah, where does that code live ?
[15:09] lmacken: maybe i can stuff that into bodhi in the mean time
[15:09] abadger1999: Since the pkgdb will get rid of owners.list I'm not too worried
about an API.
[15:09] G: abadger1999: in that case dgilmore would be the person to poke I think
[15:09] glezos has joined the group chat (n=glezos@fedora/glezos)
[15:09] lmacken: well, bodhi needs to know who owns what
[15:09] abadger1999: cvs-int:/cvs/extras/CVSROOT/admin/owners.py
[15:09] lmacken: thanks
[15:09] mmcgrath: abadger1999: if you do get together a comprehensive list of stuff
missing from EPEL send it my way.
[15:09] abadger1999: I have a few changes to what's checked in that I'll have to
add later.
[15:10] abadger1999: Will do.
[15:10] abadger1999: warren is going to work on koji syncing.
[15:10] tibbs has left ("Konversation terminated!" (n=tibbs@fedora/tibbs))
[15:10] abadger1999: I'm going to do bugzilla sync and cvs acls sync this week.
[15:10] abadger1999: G (Nigel Jones) has started looking at the code and made some changes
to the way it looks.
[15:11] mmcgrath: cool
[15:11] G: (Minor to start with, just getting to grips with Turbogears and how it works)
[15:11] abadger1999: So we're on track for next week or the week after.
[15:11] mmcgrath: abadger1999: anything else?
[15:11] abadger1999: That's about it.
[15:11] mmcgrath: abadger1999: the package db only contacts the database, correct?  Does
it need any write access to a file system like koji or cvs?
[15:12] MrBawb has joined the group chat (i=abob(a)guppy.drown.org)
[15:12] abadger1999: i think cvs acls can pull from the packagedb for now instead of pkgdb
pushing to cvs-int.
[15:12] abadger1999: koji I'm not sure about.  warren's looking into it.
[15:13] mmcgrath: cool.
[15:13] mmcgrath: pull is better than push for security reasons.
[15:13] mmcgrath: same reason bodhi is deployed on app5 instead of in our cluster.
[15:13] warren: you rather koji pull from packagedb?
[15:13] mbonnet: that's not really possible
[15:13] mbonnet: unless we have a cronjob that does the sync periodically
[15:13] mmcgrath: warren: if its filesystem stuff, yes.  if its db stuff then no.
[15:14] warren: abadger1999, it is all db right?
[15:14] abadger1999: mbonnet: How do you currently sync owners.list?
[15:14] warren: abadger1999, see /cvs/pkgs/CVSROOT/admin/owners-sync.py
[15:14] mbonnet: I believe the script is run by hand right now
[15:14] f13: yes, by hand
[15:15] * dgilmore is here
[15:15] f13: when we make changes to owners.list we run the sync script
[15:15] f13: less than ideal, but functional
[15:15] mmcgrath: <nod>
[15:15] abadger1999: Hmm...  Is it just pushing into koji's db?
[15:15] dgilmore: abadger1999: EPEL owner information is in owners.epel.list
[15:16] abadger1999: if so it can be a cronjob or we can write a callback that pushes the
information from the package db to koji when it's updated.
[15:16] rdieter has left (Remote closed the connection (n=rdieter(a)sting.unl.edu))
[15:17] mmcgrath: abadger1999: we can figure it out.  We should move on for the meeting
though.
[15:17] f13: abadger1999: it uses the koji API to do ownership adds/changes.
[15:17] f13: doesn't talk to the db directly
[15:17] abadger1999: f13: Sounds like it won't be a problem then.
[15:17] abadger1999: mmcgrath: And won't need to write to any filesystems.
[15:17] mmcgrath: abadger1999: excellent
[15:17] mmcgrath: k, moving on
[15:18] mmcgrath has set the subject to Config Management - mmcgrath
[15:18] mmcgrath: nothing majorly new here.  I'm going through and making sure our xen
dom0's are setup properly.
[15:18] mmcgrath: I've also started forcing some packages to uninstall and some
services not to start (cups, gpm, etc)
[15:18] mmcgrath has set the subject to VCS - jcollie
[15:18] mmcgrath: jcollie: ping?
[15:18] jcollie: yo
[15:19] mmcgrath: jcollie: Are you still playing with VCS solutions?
[15:19] jcollie: i think that the discussion last week on -devel and -infra was good
[15:19] f13: abadger1999: it needs to be made smarter, like knowing about different owners
for different tags and all that, but that's just details (:
[15:19] jcollie: i just need to sit down and write up a more concrete proposal
[15:19] mmcgrath: jcollie: me too, its gotten more interest this time around then 6 mo.
ago or so
[15:20] mmcgrath: jcollie: solid, make sure to get some good input from the jeremy's
and jesse's in the world 
[15:20] mmcgrath: k, moving next
[15:20] jcollie: i think it'll be a mix... there'll be a repository that looks a
lot like we have now, but with some meta-language or -tags to pull patches out of a
"exploded tree" repo
[15:20] * mdomsch joins belatedly
[15:20] mmcgrath has set the subject to Firewall System Rewrite - lmacken skvidal
[15:20] mmcgrath: mdomsch: yo
[15:21] mmcgrath: jcollie: excellent, thanks for getting that stuff together.
[15:21] mmcgrath: lmacken: ping
[15:21] mmcgrath: skvidal: ping?
[15:21] lmacken: no updates on this from my end.. have we decided to abandon pyroman ?
[15:21] mmcgrath: lmacken: Not sure, I know xDamox has some opinions on it.
[15:21] mmcgrath: xDamox: ping?
[15:21] xDamox: yo
[15:21] mmcgrath: you had some items to discuss regarding the Firewall System?
[15:21] skvidal: mmcgrath: I think we should just go with simple iptables files in
/etc/sysconfig
[15:21] xDamox: Yea, I updated the template we were using and neaten it up a little
[15:21] mbonnet: question: does our firewall system have some kind of NAT/conntrack
limit?
[15:22] mmcgrath: skvidal: I agree, what about boxes that have different firewall needs
though?
[15:22] xDamox: I can help 100% with the iptables writing.
[15:22] skvidal: mmcgrath: that's what puppet is for
[15:22] lmacken: taking the strict & simple rule approach sounds good to me
[15:22] mbonnet: I'm wondering exceeding that limit might be the cause of the
intermittent connection drops people see connecting to koji.fp.o
[15:22] skvidal: mmcgrath: distribute files out based on host
[15:22] fchiulli has joined the group chat
(i=824c4010(a)gateway/web/cgi-irc/ircatwork.com/x-04ce31ce5397d4ea)
[15:22] mmcgrath: mbonnet: Both the host based and hardware firewalls can do it but only
the proxy servers actually do do it now.
[15:22] mmcgrath: skvidal: ahh, yes.  A puppet template would work well for that I think.
[15:23] mmcgrath: mbonnet: I'll verify that we're not rate limiting in any way on
the hardware firewall.
[15:23] xDamox: mmcgrath, do we have an up to date list of services running on each box
[15:23] xDamox: and their ports?
[15:23] mmcgrath: xDamox: we're pretty close.
[15:23] mmcgrath: skvidal: do you have a link to the iptables rules you'd suggested on
the list?
[15:24] skvidal: yes
[15:24] skvidal: uno momento
[15:24] skvidal: http://linux.duke.edu/~skvidal/misc/iptables-template
[15:24] xDamox: Ok. If you could give me a copy, I could do a sample firewall for some
boxes maybe and have skvidal and lmacken check it over?
[15:24] lmacken: xDamox: sounds good to me
[15:24] xDamox: that good with you too skvidal ?
[15:24] skvidal: xDamox: fine - I already have those on a couple of the boxes due to the
release
[15:25] skvidal: iirc they're on proxy1 and 2
[15:25] mmcgrath: xDamox: remember KISS 
[15:25] xDamox: Ok, yep 
[15:25] xDamox: Ill make it a simple as possible
[15:25] G: Have fun with the rest of meeting, I'm out
[15:25] mmcgrath: G: later, thanks for coming
[15:26] xDamox: also I am sure skvidal and lmacken will be able to simplify it more 
[15:26] mmcgrath: xDamox: cool, take what skvidal has at
http://linux.duke.edu/~skvidal/misc/iptables-template and give it a good lookover.
[15:26] mmcgrath: I'll create an erb (puppet template) out of it and see how it goes.
[15:26] xDamox: yep  will do
[15:27] dgilmore: mbonnet: i dont know what on our firewalls as far as that goes
[15:27] dgilmore: mbonnet: we dont control the nat part of it
[15:28] mmcgrath: k, xDamox when you're done send'er to the list and we can get
this all underway.
[15:28] mmcgrath has set the subject to Server Upgrades - mmcgrath
[15:28] xDamox: OK mmcgrath,
[15:28] mmcgrath: So I'm trying to get some additional RAM in some of our servers.
[15:28] mmcgrath: but we have more pressing issues.. .namely a lot of our newer boxes
don't have warrantys.
[15:29] mmcgrath: so I'm trying to figure out where money should come from to pay for
that.
[15:29] mmcgrath: additionally we have a lot of boxes that are reaching the end of their
natural life and should be replaced.
[15:29] mmcgrath: Fortunately if we stick with high capacity devices, this will allow us
to use our rack more efficiently.
[15:29] mmcgrath: The major limiting factor being cost, heat and power.
[15:29] mmcgrath: just letting everyone know whats going on there.
[15:30] mmcgrath has set the subject to Xen Conversion - mmcgrath
[15:30] mmcgrath: So I've started doing some work with iscsi
[15:30] dgilmore: 
[15:30] mmcgrath: It's actually going quite well.
[15:30] * mmcgrath digs up a bonnie run
[15:30] warren: what will serve iscsi?
[15:30] mmcgrath: warren: the netapp already is.
[15:31] mmcgrath: grr pastebin
[15:31] dgilmore: mbonnet: how much storage do we have?  how much did you use for iscsi?
[15:31] dgilmore: mmcgrath: http://paste.ausil.us
[15:32] dgilmore: mmcgrath: ^^^^^^^^^^^^^^  meant you not mbonnet
[15:32] mmcgrath: dgilmore: already on it 
[15:32] mmcgrath: ok, here's an iscsi run on publictest9
[15:32] mmcgrath: http://paste.ausil.us/161
[15:32] mmcgrath: dgilmore: right now 500G
[15:32] mmcgrath: all in all I've been quite pleased with it.
[15:33] mmcgrath: I've kickstarted a few boxes with iscsi, the package install portion
(about 400 packages) takes about 2 minutes.
[15:33] londo: mmcgrath: random access seems slow to me
[15:33] dgilmore: mmcgrath: live migration is easy to do?
[15:33] mmcgrath: dgilmore: yep, so far its just worked for me.  There's a brief
network blip I need to work on.  The box itself doesn't experience it that bad but I
think there's some arp issues.
[15:34] Karl_le_Rouge has joined the group chat
(n=RedKarl(a)ALyon-257-1-149-122.w81-251.abo.wanadoo.fr)
[15:35] dgilmore: awesome
[15:35] mmcgrath: londo: I've seen random seek as high as 705.2.
[15:35] mmcgrath: The larger the test was on iscsi the slower that got though, always a
good excuse to tweak and test though 
[15:36] mmcgrath: All in all I think iscsi will work very well for us.  We just need to
watch carefully network utilization and overall netapp utilization.
[15:36] londo: mmcgrath: numbers from tiobench will be nice if you can get them
[15:36] mmcgrath: londo: is it in extras?
[15:36] londo: mmcgrath: yeap
[15:36] mmcgrath: londo: cool, I'll run it then.
[15:36] mmcgrath: k, moving on
[15:37] mmcgrath has set the subject to Bacula
[15:37] mmcgrath: So I've been testing out bacula on xen6 and publictest[3-4]
[15:37] mmcgrath: everything's been working great.
[15:37] f13: hurray!
[15:37] dgilmore: mmcgrath: how much total disk do we need to backup?
[15:37] mmcgrath: We're just blocking on https://bugzilla.redhat.com/230344
[15:37] f13: a scary amount
[15:37] f13: (if you count /mnt/koji)
[15:38] mmcgrath: dgilmore: wellllll, depends, do you con't /mnt/koji or not?
[15:38] dgilmore: welll  we really should backup /mnt/koji
[15:38] f13: mmcgrath: btw, did the new disk shelf show up in phx?
[15:38] mmcgrath: dgilmore: the plan right now is to do a backup of everything on
xen6's local storage which is 378G.  I'm working on getting a tape backup for
everything though (including koji)
[15:38] * dgilmore needs a cloning machine 
[15:39] mmcgrath: f13: I've not heard one way or the other but I was under the
impression that it should be there by now.  I'll send an emil.
[15:39] dgilmore: mmcgrath:   ok
[15:39] mmcgrath: dgilmore: I've got the tape drive as a priority2 thing after our
warranty issue with the soc.
[15:39] mmcgrath: all in all though, ixs says he'll have more time in the comming days
for us to do a formal review.
[15:40] dgilmore: mmcgrath:   yeah  we would probably want LT)2 or 3  with at least 10
slots
[15:40] mmcgrath: For those that haven't used it Bacula is really slick.
[15:40] skvidal: is it wicked slick?
[15:40] londo: if you are going to move things on netapp/iscsi is it possible to do the
backup there (if a tape drive is available)
[15:40] mmcgrath: super wicked slick.
[15:40] abadger1999: skvidal: wykd
[15:40] dgilmore: i need to find time to get it reviewed
[15:40] mmcgrath: londo: thats the problem, we had 3 netapps to deal with now we have 1
super netapp and I'm not comfortable with backing up to itself.
[15:41] mmcgrath: londo: sorry, I missed your (if tape drive) comment.
[15:41] dgilmore: mmcgrath: i agree
[15:41] f13: I loved bacula when I was using it.
[15:41] mmcgrath: k, moving on
[15:41] mmcgrath has set the subject to Translators stuff -
[15:42] f13: seriously hot stuff
[15:42] mmcgrath: glezos: has been working on this.  Its now at
http://publictest4.fedora.redhat.com/
[15:42] mmcgrath: this will be a very big deal when we start moving stuff to it.
[15:42] RedKarl has left (Connection timed out
(n=RedKarl(a)ALyon-257-1-39-129.w90-14.abo.wanadoo.fr))
[15:42] JSchmitt has left ("Konversation terminated!"
(n=s4504kr@fedora/JSchmitt))
[15:42] mmcgrath: so all keep your eyes out for it and help out because all parties
involved can use it.
[15:42] mmcgrath has set the subject to account system -
[15:42] mmcgrath: Nothing new here.  If anyone is interested in working on it with me that
would be good.
[15:43] mmcgrath has set the subject to Project Hosted - f13
[15:43] mmcgrath: f13: ?
[15:43] f13: nothing new.  Trac git plugin sucks.
[15:44] mmcgrath: <nod>
[15:44] f13: Oh, I created a script to create trac projects, but havne't put it in scm
any where or documented it
[15:44] abadger1999: f13: Could you give me access to the hosted box?
[15:44] f13: sure.
[15:44] abadger1999: Thanks.
[15:44] f13: at some point it should be FAS'd but...
[15:44] mmcgrath: <nod>
[15:45] mmcgrath: next
[15:45] mmcgrath has set the subject to FedoraPeople.org - skvidal
[15:45] mmcgrath: skvidal: anything new?
[15:45] skvidal: nothing
[15:45] warren: Is that planned for shell and web?
[15:45] skvidal: yes
[15:45] dgilmore: mmcgrath: just thought of something ill switch off plague on June 29 
for FC-5
[15:46] dgilmore: skvidal: anyidea when you will get to rebuild the box?
[15:46] skvidal: dgilmore: not this week and probably not beginning of next since I'll
be in orientation, etc
[15:46] mmcgrath: dgilmore: <nod>
[15:47] skvidal: but I'll be working again come next week
[15:47] skvidal: so it's a start
[15:47] skvidal: and I should be able to spend the time
[15:47] mmcgrath: cool
[15:48] mmcgrath has set the subject to Ibiblio Mirror - On hold
[15:48] mmcgrath: The ibiblio mirror is on hold for probably about a week while we hook
don up with direct I2 access to our mirror in RDU.
[15:48] mdomsch: mmcgrath, pick set up the static route already
[15:48] mmcgrath: mdomsch: hmm, I'll have to check with don, he was under the
impression he needed to wait a bit.
[15:49] mmcgrath: Ok, thats all I've got.
[15:49] mmcgrath has set the subject to Open Floor ----------------
[15:49] lmacken: word
[15:49] lmacken: I was wondering what you guys thought about having some sort of
development environment for our webapps.
[15:49] lmacken: So, there are a handfull of people that are interested in hacking on
bodhi, but due it's dependencies on koji and mash, it's extremely difficult to
develop it anywhere other than PHX.  I've currently been doing all of my development
on publictest2, which has been working out great.
[15:49] lmacken: So a possibility for this is to have some Xen guest with a read-only
mount of /mnt/koji and blocked out from the rest of PHX.
[15:49] mdomsch: lmacken, +1; /me misses publictest7
[15:50] lmacken: yeah, and honestly.. i have no idea how to start hacking on
mirrormanager, smolt, etc
[15:50] lmacken: i think if we opened the doors a bit, our infrastructure could improve
vastly
[15:50] lmacken: mdomsch: feel free to hack on publictest2 for now 
[15:50] mmcgrath: lmacken: the main limiting facter on that is RAM, but we can set
something up.
[15:51] lmacken: mmcgrath: cool
[15:51] mmcgrath: lmacken: we should probably setup more shared xen instances.
[15:51] abadger1999: lmacken: +1
[15:51] dgilmore: im going to start work on enabling secondary archs if anyone wants to
help feel fee to talk to me
[15:52] dgilmore: mmcgrath: can we possibly get another vlan?
[15:52] jcollie: mmcgrath, could i get a xen guest for testing the git/vcs stuff?
[15:52] lmacken: mmcgrath: cool.. so what is the next action to getting this ready? 
creating a group for infrahackers and granting access on a restricted guest ?
[15:52] dgilmore: mmcgrath: so we can seperate the some guestd  for this kind of thing
[15:52] mmcgrath: lmacken: well I'll need to find where we have RAM avaiable for the
instances.  Its item "Server Upgrades" on the wiki.
[15:52] mmcgrath: dgilmore: we should.
[15:53] wolfy has left ("When you are down and out something always turns up-and it
is usually the noses of your friends." (n=lonewolf@fedora/wolfy))
[15:53] lmacken: mmcgrath: ok.. well, publictest2 has been my playground for the past few
months.. any reason not to just start using that ?
[15:53] mmcgrath: jcollie: I think we can setup something, it'll be a bit
[15:53] mmcgrath: lmacken: not sure, I think it only has 512M ram right?
[15:53] lmacken: mmcgrath: i'm not sure
[15:54] dgilmore: mmcgrath: im pretty sure thats all it ahs
[15:54] dgilmore: has
[15:54] lmacken: mmcgrath: also, i noticed that you setup the security guest.. does
bressers know about it yet ?
[15:54] mmcgrath: k, I'll try to find ways to consolidate some of our lesser machines
into a bigger, sort of super machine.
[15:54] lmacken: mmcgrath: cool
[15:54] mmcgrath: lmacken: I think dgilmore did that
[15:55] lmacken: ah
[15:55] lmacken: dgilmore: is the security guest ready to go ?
[15:56] dgilmore: lmacken: not yet
[15:56] lmacken: dgilmore: k, just checking 
[15:56] dgilmore: i need to add the security group to get shell access
[15:58] mmcgrath: solid
[15:58] mmcgrath: so anyone have anything else?  If not I'll close the meeting in 30
seconds?
[15:58] mmcgrath: 10
[15:59] mmcgrath has set the subject to Meeting End -----------------------

2024

2023

2022

2021

2020

2019

2018

2017

2016

2015

2014

2013

2012

2011

2010

2009

2008

2007

2006

IRC Log for Fedora Infrastructure Meeting (2007-06-14)