#fedora-meeting: Infrastructure (2012-08-23)
Meeting started by nirik at 18:00:08 UTC. The full logs are available at
* Greets! (nirik, 18:00:08)
* New folks introductions and Apprentice tasks. (nirik, 18:01:40)
* Applications status / discussion (nirik, 18:04:16)
* LINK: http://status.fedoraproject.org/changes.rss
* ACTION: make sure everything is set with status (ssh key sync and
docs) and then announce/use. (nirik, 18:19:07)
* ACTION: iambryan to help update outage docs. (nirik, 18:19:16)
* testing python-bugzilla update on app01.stg (nirik, 18:21:47)
* Sysadmin status / discussion (nirik, 18:22:22)
* IMM password changes ongoing (nirik, 18:24:02)
* euca cloud has f16 images working now. (nirik, 18:24:16)
* F18 Alpha Release tickets/Freeze (nirik, 18:27:53)
* LINK: https://fedorahosted.org/fedora-infrastructure/ticket/3427
* LINK: https://fedorahosted.org/fedora-infrastructure/ticket/3428
* LINK: https://fedorahosted.org/fedora-infrastructure/ticket/3429
* Upcoming Tasks/Items (nirik, 18:30:17)
* 2012-08-14 to 2012-09-04 F18 Alpha Freeze (nirik, 18:30:34)
* 2012-08-31 end of 2nd quarter (nirik, 18:30:34)
* 2012-09-04 F18 Alpha release. (nirik, 18:30:34)
* 2012-09-10 to 2012-09-24 Skvidal is out (nirik, 18:30:34)
* 2012-09-25 to 2012-10-09 F18 Beta Freeze (nirik, 18:30:34)
* 2012-09-26 to 2012-09-28 Puppetconf (nirik, 18:30:35)
* 2012-10-09 F18 Beta release (nirik, 18:30:37)
* 2012-10-23 to 2012-11-13 F18 Final Freeze (nirik, 18:30:39)
* 2012-11-13 F18 release. (nirik, 18:30:41)
* 2012-11-26 to 2012-11-29 Security FAD (nirik, 18:30:43)
* 2012-11-30 end of 3nd quarter (nirik, 18:30:45)
* Open Floor (nirik, 18:31:42)
Meeting ended at 18:35:34 UTC.
* make sure everything is set with status (ssh key sync and docs) and
* iambryan to help update outage docs.
Action Items, by person
* iambryan to help update outage docs.
* make sure everything is set with status (ssh key sync and docs) and
People Present (lines said)
* nirik (88)
* relrod (14)
* WolfKit (10)
* abadger1999 (10)
* smooge (10)
* iambryan (7)
* threebean (7)
* pingou (5)
* skvidal (5)
* zodbot (4)
* ctria (3)
* lmacken (2)
* rossdylan (2)
* Betapanda (1)
* ricky (0)
* mdomsch (0)
* dgilmore (0)
* CodeBlock (0)
18:00:08 <nirik> #startmeeting Infrastructure (2012-08-23)
18:00:08 <zodbot> Meeting started Thu Aug 23 18:00:08 2012 UTC. The chair is nirik.
Information about MeetBot at http://wiki.debian.org/MeetBot
18:00:08 <zodbot> Useful Commands: #action #agreed #halp #info #idea #link #topic.
18:00:08 <nirik> #meetingname infrastructure
18:00:08 <zodbot> The meeting name has been set to 'infrastructure'
18:00:08 <nirik> #topic Greets!
18:00:08 <nirik> #chair smooge skvidal CodeBlock ricky nirik abadger1999 lmacken
dgilmore mdomsch threebean
18:00:08 <zodbot> Current chairs: CodeBlock abadger1999 dgilmore lmacken mdomsch
nirik ricky skvidal smooge threebean
18:00:15 * abadger1999 here
18:00:16 * skvidal is here
18:00:21 * Betapanda is here
18:00:31 <WolfKit> excuse me. sillyness from another channel
18:00:32 * WolfKit is here
18:00:33 <iambryan> /me is here
18:00:35 * lmacken
18:00:46 * threebean
18:00:50 * relrod here
18:01:25 * pingou is here
18:01:31 <nirik> ok, lets go ahead and dive on in...
18:01:40 <nirik> #topic New folks introductions and Apprentice tasks.
18:01:45 <nirik> If any new folks want to give a quick one line bio or any
apprentices would like to ask general questions, they can do so in this part of the
meeting. Don't be shy
18:02:19 <smooge> here
18:02:40 * rossdylan is here
18:02:53 <nirik> any new folks want to say hi? or shall we move on...
18:03:02 <iambryan> Hey all, I am a fedora user for the past 9 years and have
admin'd RHEL boxes with NOC infra management apps
18:03:36 <iambryan> I would like to help out on the main admin side and learn some
python while I'm at it
18:03:46 <pingou> welcome :)
18:03:49 <ctria> hi all
18:03:54 <ctria> sorry for being late
18:03:55 <iambryan> Thanks
18:03:57 <nirik> welcome iambryan. :) We can point you to some things to look at
after the meeting over in #fedora-admin
18:04:12 <nirik> no worries ctria.
18:04:16 <nirik> #topic Applications status / discussion
18:04:16 <ctria> hm i'm on time! i'm back after some months away
18:04:23 <iambryan> Great, will stick around then :)
18:04:57 <nirik> Any applications news this week or upcoming? abadger1999 / relrod /
threebean / lmacken / pingou / anyone I missed?
18:05:18 <pingou> I'm slowly but surely trying to move FAS's db model to
18:05:39 <pingou> eof for me
18:06:03 <nirik> pingou: cool. so thats in prep for moving it to TG2? or ?
18:06:07 <relrod> Just pushed another Flask build to el6 testing, to fix the broken
deps warnings on the doc subpackage. So please continue testing flask from epel, poke
me/file bugs on it if it does anything insane
18:06:25 <nirik> relrod: any news on the plugins issues tflink was hitting?
18:06:38 <relrod> nirik: Haven't had a chance to talk with him much yet, no :(
18:06:43 <pingou> nirik: I think that will help whatever framework we choose (if it
18:07:07 <relrod> nirik: will try and catch up with him today if he's around
18:07:12 <nirik> cool.
18:07:38 <rossdylan> I need to look over fedbadges once more and make sure evrything
works in EPEL, and the fedora user streams application has a basic outline at this stage
18:07:52 <relrod> nirik: do we want to talk about the status app here or save it for
later in the meeting?
18:07:57 <relrod> we were going to revisit it this week
18:08:05 <nirik> relrod: oh yeah... either way. we can talk about it now...
18:09:01 <nirik> So, background is:
18:09:32 <nirik> we have http://status.fedoraproject.org/
setup. This is intended to
be a place people can see if there are outages or issues...
18:09:50 <nirik> it's running in openshift, and does not depend on our infra for
18:10:00 <relrod> sorry about that, had to answer the door, back now
18:10:09 <nirik> do folks want to announce and start using this? or are their
18:10:27 <smooge> 1) how is it used. 2) where is it documented?
18:10:30 <nirik> I used it the other day for the hosted outage... no idea if people
noticed. (but of course we have not yet announced it)
18:10:47 <skvidal> and it is documented in infradocs, iirc
18:10:50 <nirik> When there is an outage, it's manually updated by someone with
info about the outage.
18:10:54 <relrod> smooge:
18:11:02 <nirik> and yeah, ^
18:11:33 <nirik> we need to make sure ssh keys are syncing, I don't know that
that is enabled yet.
18:12:00 * lmacken noticed, thanks to the favicon :)
18:12:09 <nirik> the rss feed is also nice.
18:12:27 <nirik> you can see when an outage started and ended (or at least when an
admin noted it/removed it)
18:12:52 <smooge> thanks.. the next question.. who can make changes? sysadmin-main,
18:12:59 <nirik> http://status.fedoraproject.org/changes.rss
18:13:18 <nirik> if/when the sync script is working, it should be sysadmin-main +
18:13:31 <nirik> we could change that if folks prefer...
18:13:46 <smooge> no it sounds right.. I just wanted to add it to the documentation
18:14:05 <nirik> yeah, good idea.
18:14:16 <abadger1999> We should "link" together all the SOPs that go into
making an outage.
18:14:25 <nirik> abadger1999: also a good idea.
18:14:46 <abadger1999> there's topic in irc, status app, outage ticket. planned
outages: nagios, mail to list.
18:15:06 <nirik> yep.
18:15:41 <abadger1999> app itself sounds good -- but some way to remember to keep it
and everything else accurate is what I think we currently lack.
18:15:46 <nirik> also for outages, it might be helpfull if the first responder
handles the actual issue and gets others to handle the paperwork stuff... but I guess that
depends on others being there to do so.
18:16:05 <nirik> we have an outage sop
18:16:06 <abadger1999> <nod>
18:16:12 <nirik> it likely needs updating.
18:16:16 <nirik> any takers? ;)
18:16:39 <iambryan> I can take a look at that
18:17:16 <abadger1999> <nod> Yeah -it currently has a busted hyperlink to the
outage template and procedure for ticket and email.
18:17:25 <relrod> abadger1999: yeah, we have a nagios check when a thing is marked
as down - it will alert us so we remember to set it back when the service comes back.
Nothing in place to remind people to actually set things as down though
18:17:41 <nirik> iambryan: that would be great. ;) please do ask lots of questions
and we can get you access to update the doc. ;)
18:18:03 <iambryan> Great, already a member with docs if that helps
18:18:26 <nirik> docs always help. Just keeping them up to date can be a chore. ;)
18:19:07 <nirik> #action make sure everything is set with status (ssh key sync and
docs) and then announce/use.
18:19:16 <nirik> #action iambryan to help update outage docs.
18:19:25 <nirik> Any other applications news?
18:19:27 <threebean> Lots of stuff happening here. The main thing is a new release
of packages and tagger in stg that contains some of the features of pkgdb migrated over.
A more complete list of other stuff is here -> http://fpaste.org/OQPD/
18:19:59 <threebean> Could use an extra pair of eyes over packages and tagger to
make sure stuff's busted.
18:20:10 <threebean> er.. make sure its not busted. ;)
18:20:15 <abadger1999> :-)
18:20:31 <nirik> cool.
18:21:04 <threebean> Oh, question.. (I should know this, but) when is the freeze
18:21:18 <nirik> as soon as alpha goes out... which is sadly now a week later. ;(
18:21:33 <nirik> oh, another application item to note:
18:21:34 <threebean> ah, I saw that. Ok. I'll keep an eye on announce and
18:21:47 <nirik> #info testing python-bugzilla update on app01.stg
18:22:07 <nirik> we should try and test anything else we can think of in stg for the
new python-bugzilla so we can update prod at some point.
18:22:22 <nirik> #topic Sysadmin status / discussion
18:22:36 <nirik> any sysadmin side things this week or pending?
18:22:53 <nirik> skvidal has been working on our cloud... it even had an instance
work on it this morning. ;)
18:23:01 <skvidal> they are working now
18:23:01 <relrod> still slowly making progress on IMM password changes
18:23:02 <threebean> cool
18:23:06 <skvidal> and I have f16 images
18:23:09 <WolfKit> is this the part where I talk about 2fa?
18:23:11 <WolfKit> :)
18:23:17 <nirik> hosted03 is dead. Long live hosted01/02. ;)
18:23:19 <skvidal> so we're not running those awful centos/ubuntu monsters
18:23:30 <abadger1999> nirik: Re: python-bugzilla -- is RH IT tracking whether they
see an increase in queries?
18:23:47 <nirik> abadger1999: not that I know of. we can ping them? would you like
to do that? or would you like me to?
18:24:02 <nirik> #info IMM password changes ongoing
18:24:15 <abadger1999> nirik: I'm not really sure who to contact there now that
dkl has left.
18:24:16 <nirik> #info euca cloud has f16 images working now.
18:24:24 <nirik> WolfKit: sure.
18:24:46 <nirik> abadger1999: kbaker? (who is in -noc actually)
18:24:48 <WolfKit> well. I had some code changes that made pam_url stop segfaulting
on 64-bit hosts
18:25:12 <WolfKit> of course, I went to spin up the VM so I could give a live demo..
and what do you think happened.. it's *gone* :/
18:25:24 <nirik> :(
18:25:26 <WolfKit> awilcox on ind-wkst041 virt-machines: ls -al
18:25:30 <WolfKit> awilcox on ind-wkst041 virt-machines:
18:25:32 <abadger1999> nirik: Cool. I'll ping him to see -- since we also have
things that run in production that don't run in stg probably need to also alert him
when we turn on production.
18:25:44 <nirik> abadger1999: thanks.
18:25:48 <WolfKit> I have no idea how this happened. running a disk recovery
software on it now..
18:25:53 <nirik> WolfKit: so, you get to fix the bug a second time? :(
18:26:04 <WolfKit> nirik: it seems like it. at least I know it's fixable
18:27:05 <nirik> yeah, hopefully it will look familliar.
18:27:44 <nirik> ok, any other sysadmin stuff?
18:27:53 <nirik> #topic F18 Alpha Release tickets/Freeze
18:28:08 <nirik> as noted, alpha has slipped a week, so our freeze continues an
18:28:23 <nirik> not sure we need to go over alpha tickets, since it's a ways
18:28:51 <smooge> what is the ticket that tracks tickets for it?
18:29:26 <nirik> we have:
18:29:28 <nirik> https://fedorahosted.org/fedora-infrastructure/ticket/3427
18:29:30 <nirik> https://fedorahosted.org/fedora-infrastructure/ticket/3428
18:29:32 <nirik> https://fedorahosted.org/fedora-infrastructure/ticket/3429
18:30:17 <nirik> #topic Upcoming Tasks/Items
18:30:30 <nirik> here comes the info dump of upcoming items:
18:30:34 <nirik> #info 2012-08-14 to 2012-09-04 F18 Alpha Freeze
18:30:34 <nirik> #info 2012-08-31 end of 2nd quarter
18:30:34 <nirik> #info 2012-09-04 F18 Alpha release.
18:30:34 <nirik> #info 2012-09-10 to 2012-09-24 Skvidal is out
18:30:34 <nirik> #info 2012-09-25 to 2012-10-09 F18 Beta Freeze
18:30:35 <nirik> #info 2012-09-26 to 2012-09-28 Puppetconf
18:30:37 <nirik> #info 2012-10-09 F18 Beta release
18:30:39 <nirik> #info 2012-10-23 to 2012-11-13 F18 Final Freeze
18:30:41 <nirik> #info 2012-11-13 F18 release.
18:30:43 <nirik> #info 2012-11-26 to 2012-11-29 Security FAD
18:30:45 <nirik> #info 2012-11-30 end of 3nd quarter
18:30:51 <nirik> anything anyone wants to note or schedule or amend?
18:31:42 <nirik> #topic Open Floor
18:31:51 <nirik> any general items for open floor or new topics?
18:33:26 * nirik listens to crickets chirp.
18:33:29 <smooge> not from me
18:33:35 <smooge> working on my talk..
18:33:36 <relrod> I start school next week, not sure what my availability will be
like yet, from here on out until the end of the semester
18:33:46 <relrod> hopefully not too bad though
18:33:47 <smooge> well school comes first
18:33:52 <smooge> then this fedora thing
18:33:57 <relrod> smooge: ;)
18:33:57 <nirik> yeah. :)
18:34:22 <nirik> I need to work on finalizing the FAD stuff... see if we can meet
budget and reserve things, etc... will try and do that more soon.
18:34:43 <relrod> sounds fun, I look forward to that
18:35:29 <nirik> ok, thanks for coming everyone... lets get back to it in
#fedora-admin / noc / apps
18:35:30 <smooge> nirik, let me know what I can do to help
18:35:34 <nirik> #endmeeting