#fedora-meeting: Infrastructure (2014-01-16)
Meeting started by nirik at 19:00:13 UTC. The full logs are available at
* welcome all planets of the solar federation (nirik, 19:00:13)
* New folks introductions and Apprentice tasks (nirik, 19:02:30)
* Applications status / discussion (nirik, 19:15:00)
* LINK: https://apps.fedoraproject.org/datagrepper
* LINK: https://apps.fedoraproject.org/packages
* new packages app ( epel bugs and enhancements) and datagrepper (new
frontpage and fixes) rolled out last week. (nirik, 19:25:26)
* new askbot in stg is almost working. need to fix email sending then
we can schedule prod upgrade (nirik, 19:25:51)
* hyperkitty roll out moving forward. Will make prod instances soon
and start figuring out forwarding setup. (nirik, 19:26:23)
* ACTION: nirik to write up a infra list post about how mailing lists
are setup now, so we can decide how to add in mailman3/hyperkitty.
* new coprs rolled out today (nirik, 19:31:24)
* Sysadmin status / discussion (nirik, 19:34:32)
* Upcoming Tasks/Items (nirik, 19:43:47)
* LINK: https://apps.fedoraproject.org/calendar/list/infrastructure/
* goal: finish round 1 of development and deploy fmn to production by
the end of January. (threebean, 19:44:50)
* Open Floor (nirik, 19:45:43)
* LINK: https://fedoraproject.org/wiki/Jenkins@infra
* LINK: http://fedoraproject.org/wiki/Infrastructure/Architecture
there is a high overview (zemmiph0bia, 19:48:12)
* LINK: https://admin.fedoraproject.org/nagios/
Meeting ended at 19:59:30 UTC.
* nirik to write up a infra list post about how mailing lists are setup
now, so we can decide how to add in mailman3/hyperkitty.
Action Items, by person
* nirik to write up a infra list post about how mailing lists are
setup now, so we can decide how to add in mailman3/hyperkitty.
People Present (lines said)
* nirik (120)
* willo (36)
* threebean (28)
* johe_ (11)
* abompard (10)
* adimania (9)
* zemmiph0bia (8)
* danofsatx (8)
* relrod (6)
* zodbot (4)
* kamina1 (4)
* smooge (4)
* dgilmore (4)
* janeznemanic (3)
* jlar (3)
* tflink (1)
* jsmith (1)
* lmacken (1)
* LoKoMurdoK (1)
* mdomsch (0)
* puiterwijk (0)
* abadger1999 (0)
* pingou (0)
19:00:13 <nirik> #startmeeting Infrastructure (2014-01-16)
19:00:13 <zodbot> Meeting started Thu Jan 16 19:00:13 2014 UTC. The chair is nirik.
Information about MeetBot at http://wiki.debian.org/MeetBot
19:00:13 <zodbot> Useful Commands: #action #agreed #halp #info #idea #link #topic.
19:00:13 <nirik> #meetingname infrastructure
19:00:13 <nirik> #topic welcome all planets of the solar federation
19:00:13 <nirik> #chair smooge relrod nirik abadger1999 lmacken dgilmore mdomsch
threebean pingou puiterwijk
19:00:13 <zodbot> The meeting name has been set to 'infrastructure'
19:00:13 <zodbot> Current chairs: abadger1999 dgilmore lmacken mdomsch nirik pingou
puiterwijk relrod smooge threebean
19:00:26 * relrod waves
19:00:42 * lmacken
19:00:44 * abompard is here
19:00:46 * threebean is here
19:00:50 * adimania is here
19:01:04 * johe_ (new one) is here
19:01:36 * jsmith is lurking
19:01:37 <dgilmore> hola
19:01:42 * jlar is new here
19:01:47 <zemmiph0bia> morning :) (New)
19:01:50 * willo is here
19:02:16 * tflink is lurking
19:02:16 <nirik> cool. Nice crowd today. ;)
19:02:30 <nirik> #topic New folks introductions and Apprentice tasks
19:02:46 <nirik> any new folks like to give a short bio/intro? or apprentices with
questions or comments?
19:02:54 * nirik sees a number of new folks today
19:03:04 <johe_> okay
19:03:39 <johe_> i introduced already to the list, so hi again, i am a 34 year old
systemadministrator and would like to help
19:03:47 <johe_> :-)
19:03:57 <zemmiph0bia> I'm Elizabeta, developer on ratticdb and recovering
sysadmin. Wanting to get added to the fi-apprentice fas group.
19:04:11 <nirik> welcome johe_ and zemmiph0bia!
19:04:19 * threebean waves "hello"
19:04:29 <adimania> Hi. I am Aditya. I graduated to sysadmin-badges a few months ago
but had to take a break.
19:04:38 <nirik> happy to add you folks to our apprentice group... just see me in
#fedora-admin after the meeting.
19:04:39 <threebean> adimania: welcome back :)
19:04:45 <adimania> Will get back in action and start contributing again. :)
19:04:51 <nirik> hey adimania. :) Welcome back
19:05:08 <adimania> threebean, nirik, thanks :)
19:05:25 <jlar> Hi, I'm Jon, I'm looking to keep my skills sharp as my
current job doesn't have me doing a lot of linux stuff. I want to help out any way I
19:05:30 <nirik> johe_: you are more interested in sysadmin tasks? or application
development? or both?
19:05:42 <nirik> zemmiph0bia: you are more interested in sysadmin I guess?
19:05:49 <nirik> welcome jlar.
19:05:58 <smooge> hwelcome
19:06:01 <johe_> nirik, well, the syadmin tasks would fit my skills i guess, as i am
doing it every day :-)
19:06:24 <johe_> but i could/would like to take a look on development also
19:06:26 <zemmiph0bia> nirik, both. sysadmin is certainly what I'm better with
but I'd like to get more into the development
19:06:35 <nirik> jlar: great. See me also after the meeting in #fedora-admin and we
can help get you started. ;)
19:06:40 <nirik> zemmiph0bia: excellent.
19:06:50 <jlar> nirik: sounds good!
19:07:13 <nirik> #fedora-admin is our main channel, we also have #fedora-apps for
development related stuff and #fedora-noc for sysadmin/outages/etc
19:07:34 <willo> nirik: I've got a Q when the intros are done
19:07:48 <nirik> willo: fire away.
19:07:57 <willo> I found a link to this page:
19:07:58 <nirik> any other new folks?
19:08:15 <willo> from the wiki. Can't recall exactly where it was.
19:08:37 <willo> There's an associated fedora hosted project - CSI
19:08:46 <nirik> indeed there is. ;)
19:09:08 <nirik> also, we folded it into the main docs too:
19:09:18 <willo> Is this current policy on how the servers should be setup?
19:09:41 <willo> Or any other guidance on the SHOULD's, SHALL's and
19:09:41 <nirik> in theory. It's something I've really not had time to keep
19:10:06 <nirik> if there's people interested in working on it again we could
revive it somewhat
19:10:56 <nirik> there's good info in there (although probibly dated now) about
how to setup your client machine as an admin... and how to handle security incidents,
19:11:00 <johe_> nirik, i would join in :-)
19:12:32 * nirik ponders the best place to discuss changes there... I guess we could just
use the main infra list at least to start.
19:13:03 <johe_> +1
19:13:22 <willo> I droped from channel for last 2 mins
19:13:25 <nirik> so, yeah, read thu and propose changes. ;) it would be great to get
19:13:27 <nirik> willo: oops. ;(
19:13:31 <nirik> let me repeat:
19:13:39 <nirik> in theory. It's something I've really not had time to keep
19:13:44 <nirik> if there's people interested in working on it again we could
revive it somewhat
19:13:48 <nirik> there's good info in there (although probibly dated now) about
how to setup your client machine as an admin... and how to handle security incidents,
19:14:16 <willo> Cool, great.
19:14:37 <nirik> great topic. Thanks for bringing it up willo
19:15:00 <nirik> #topic Applications status / discussion
19:15:06 <willo> I was thinking that I could wrte some SCAP contentbased on tat
19:15:11 <nirik> any applications news or plans?
19:15:25 <threebean> willo: that would be pretty sweet
19:15:30 <nirik> willo: I think there was some talk about SCAP recently... perhaps
ask Sparks when he's around?
19:15:39 <willo> will do
19:16:17 <threebean> on applications new, a new datagrepper and a new
fedora-packages got deployed at the end of last week
19:16:22 <threebean> https://apps.fedoraproject.org/datagrepper
19:16:28 <threebean> https://apps.fedoraproject.org/packages
19:16:35 <nirik> I have one bit of application news. :) I've been trying to get
the new version of askbot working in our staging setup.
19:16:42 <nirik> threebean: any big changes?
19:16:58 <threebean> enhancements to both. charul's been working on
19:17:13 * LoKoMurdoK here
19:17:16 <threebean> she added that counter to the frontpage and a new html
representation of messages if you look at queries in your browser.
19:17:34 <relrod> threebean: What is the goal date for deploying fmn to prod? Is
19:17:55 <threebean> fedora-packages got a search bugfix - it accepts slashes
19:18:02 <threebean> it also displays EPEL bugs in the bugs list (finally)
19:18:03 <nirik> currently: I have askbot working at
except for the email sending. Need to figure that our
and get it working, then we can repave it and redeploy and make sure it's still good
and then look at updating production
19:18:11 <nirik> welcome LoKoMurdoK
19:18:24 <dgilmore> buenos dias LoKoMurdoK
19:18:42 <dgilmore> nirik: i got some emails yesterday
19:18:49 <threebean> relrod: no concrete goal date for fmn -- we never set one
(although I think we just said "january")
19:19:13 <nirik> dgilmore: yeah, there was a flood of old ones. I wiped those, but
it's not sending any new ones for answers, etc that I can see. ;(
19:19:19 <threebean> pkgdb2 depends on it, but it looks like pingou will be busy
with other stuff before he can get around to the pkgdb2 deployment.
19:19:44 <willo> nirik: I was also looking around at askbot in stg. You saw sme of
the results in fedora-noc channel. I'll try and avoid tripping the orbital defence
lasers in the future. :)
19:20:26 <nirik> note to new folks: do feel free to ask questions about what things
are or what someone means. ;) It's always good to explain in more detail if folks
19:20:35 <nirik> willo: :) no worries.
19:20:54 <willo> I saw trying to find out how my public key gets from the staging
fas into my home dir's authkeys file. Apologies.
19:21:52 <nirik> willo: we can sort it out. I think it's all ok now, but if you
still see any issues please let me know.
19:22:28 <relrod> heh
19:22:57 <nirik> oh, in other application news, abompard and I have been exchanging
some emails on hyperkitty roll out. I'm likely going to make some production instances
soon to start figuring out setup.
19:23:23 <nirik> the biggest problem is going to be setting up just single lists to
forward to the new setup from the old... or otherwise make sure they don't interfere.
19:23:42 <nirik> for both web and smtp.
19:23:53 <smooge> cool
19:24:05 <threebean> abompard: how's the caching situation look these days?
iirc, when we first imported messages from the old lists into the staging hyperkitty
19:24:09 <nirik> so, we may need folks to put on their thinking caps as to how to do
19:24:16 <willo> Dropped again, but just from meeting channel. No message, just
gets quite and then all the nicks disappear. I'll read what I missed in the meeting
19:24:25 <threebean> .. well, you were looking at caching stuff, no?
19:24:35 <abompard> Caching in hyperkitty is waaaay better now
19:24:41 <threebean> \ó/
19:24:50 <nirik> staging instance: https://lists.stg.fedoraproject.org/archives/
19:24:51 <abompard> I'm using memcached
19:25:02 <threebean> oh yeah. that's quick :)
19:25:26 <nirik> #info new packages app ( epel bugs and enhancements) and
datagrepper (new frontpage and fixes) rolled out last week.
19:25:50 <abompard> threebean: actually the best improvement is when archive are
19:25:51 <nirik> #info new askbot in stg is almost working. need to fix email
sending then we can schedule prod upgrade
19:26:23 <nirik> #info hyperkitty roll out moving forward. Will make prod instances
soon and start figuring out forwarding setup.
19:26:27 <abompard> because I was using some tables as a cache, which needed
rebuilding. Now the cache is populated ondemand, from the requests.
19:27:07 <abompard> well there are quite a few non-visible improvements these days.
And nirik is rebuilding my test server, which will be useful
19:27:36 <nirik> abompard: lists-dev should be redone, just needs some tweaks on the
ansible playbook to finish
19:27:44 <abompard> nirik: yeah, will do soon
19:28:03 <abompard> nirik: I just finish my big commit here, and I'll do that
19:28:23 <nirik> excellent.
19:28:29 <threebean> cool :)
19:29:05 <nirik> abompard: I know our schedules are different, but perhaps next week
we could meet up and sort out production forwarding... or perhaps it would be good for me
to write up for the list the way things are now so we can figure it there.
19:29:22 <abompard> nirik: sure
19:29:32 <nirik> let me try and do a list post and we can go from there.
19:29:43 <abompard> nirik: great
19:30:04 <nirik> I am sure we can get it working, just will be a matter of the best
place to splice in the new one without messing with the old.
19:30:38 <nirik> ok, any other applications news?
19:31:03 <nirik> #action nirik to write up a infra list post about how mailing lists
are setup now, so we can decide how to add in mailman3/hyperkitty.
19:31:15 <nirik> oh, another thing:
19:31:24 <nirik> #info new coprs rolled out today
19:31:41 <threebean> relrod: do you have a preference on an fmn deployment goal
19:31:58 <relrod> threebean: Not particularly
19:32:15 <threebean> ok, we'll sort it out in #fedora-apps then I guess.
19:32:19 <relrod> threebean: By the end of January works for me, if that is still
19:32:58 <nirik> for new folks: http://copr.fedoraproject.org/
is copr... fmn is a
messaging preference application, only in staging so far:
19:34:29 <nirik> ok, moving along then...
19:34:32 <nirik> #topic Sysadmin status / discussion
19:34:39 <nirik> lets see...
19:35:07 <nirik> not too much going on right now.
19:35:14 <nirik> I need to finish the ansible cron job...
19:35:26 <nirik> It sounds like folks wanted it to just run --check --diff each day
19:35:46 <nirik> I might try and migrate some more things from puppet to ansible
19:36:12 <nirik> we have setup for those mailman01/02 in production pending
19:36:17 * adimania thinks that is interesting
19:36:45 <nirik> I might file a ticket asking for a script to help you make new
instances in ansible.
19:36:57 <nirik> it's just a matter of copying some templates and editing
19:37:28 <zemmiph0bia> Are there tickets for the remaining stuff that's left in
puppet that needs to get migrated?
19:37:55 * willo was wondering the same thing
19:38:02 <nirik> zemmiph0bia: there's not. it's just been 'as we get to
it'. I posted a thing to the list a few weeks ago about the plan moving forward...
19:38:05 <adimania> I'll give it a shot. I understand puppet but I have very
basic understanding of ansible.
19:38:18 <zemmiph0bia> adimania, ansible is exceptionally easy.
19:38:21 <nirik> if folks want to help out migrating that would be lovely.
19:38:25 <smooge> I am going to order some hardware tonight (whether it gets
approved is another matter). I am doing some certs for coprs and I am catching up with
other stuff from extra long vacation/disease marathon
19:38:31 <zemmiph0bia> I'd be happy to help with that, help my partner write a
book on it :p
19:38:42 <nirik> zemmiph0bia: awesome!
19:38:59 <nirik> smooge: any news on that hw order? or same old same ol?
19:39:05 <willo> I'm keen to assist in the migration
19:39:28 <willo> Ansible learning experience
19:39:28 <smooge> nirik, I have heard nothing so I am going to do it anyway.
19:39:52 <nirik> some things are pretty trivial to move, just takes a bit of time...
some things (like our proxy setup thats in puppet) will likely require more brainpower.
19:40:02 <nirik> smooge: :(
19:40:50 <willo> So, I'm guessing as things are migrated, they're moved out
of puppet pretty quickly
19:40:51 <nirik> so folks interested in ansiblle migrations bug me and I can try and
get you started. :)
19:41:01 <nirik> willo: yep. We are only adding new stuff to ansible...
19:41:09 <threebean> question -> we still don't have a release schedule for
Fedora 21, correct? which means we don't have a foreseeable infrastructure freeze on
the horizon, correct?
19:41:12 <nirik> but we had a lot in puppet, so it's not a quick process
19:41:26 * adimania will go through http://docs.ansible.com/intro_getting_started.html
before bugging nirik
19:41:27 <willo> So, anything that I can see in puppet is a list of things that need
to be migrated, correct
19:41:28 <nirik> threebean: correct. we are no freezes until we know more.
19:41:33 * threebean nods
19:41:44 <threebean> that's a good thing for the migration effort ;)
19:41:56 <nirik> willo: yep. although... there's also some old junk in puppet
that wasn't properly removed when things were retired...
19:42:11 <nirik> for example, I think our asterisk server is still in there, even
tho it went away about 5 years ago
19:42:41 <nirik> anyhow, we will get there i hope. ;)
19:42:41 <dgilmore> there is a ton of legacy things in puppet
19:42:48 <willo> :) just to keep it interesting
19:42:52 <nirik> dgilmore: yeah. just old stuff.
19:43:35 <nirik> I still do want to do a FAD sometime (Fedora activity day, where we
get folks to gather in the same physical location and work on one problem). finishing
ansible migration is one possible topic.
19:43:47 <nirik> #topic Upcoming Tasks/Items
19:43:47 <nirik> https://apps.fedoraproject.org/calendar/list/infrastructure/
19:43:55 <willo> nirik: will hit you up in fedora-admin later tonight or the
19:44:01 <nirik> Anyone have upcoming tasks or items they want to schedule or note?
19:44:05 <nirik> willo: sounds good.
19:44:49 <nirik> okey dokey.
19:44:50 <threebean> #info goal: finish round 1 of development and deploy fmn to
production by the end of January.
19:44:59 <nirik> sounds good. ;)
19:45:43 <nirik> #topic Open Floor
19:45:48 <nirik> any items for open floor?
19:46:00 <nirik> questions, comments, ideas, youtube videos?
19:46:23 <janeznemanic> nirik: may I ask you what is the purpose of jenkins in our
19:46:51 <nirik> https://fedoraproject.org/wiki/Jenkins@infra
19:47:05 <nirik> see the 'what is' there. ;)
19:47:20 <janeznemanic> great
19:47:25 <willo> I'm a picture person. Are there currently any diagrams showing
interactions between systems?
19:47:44 <willo> I understand that it would be complex, I suppose, I'm not sure
just how complex
19:48:12 <zemmiph0bia> http://fedoraproject.org/wiki/Infrastructure/Architecture
there is a high overview
19:48:18 <nirik> well, there's:
talking about our proxy setup,
but ti might be pretty out of date
19:48:24 <nirik> yeah. :)
19:48:26 <zemmiph0bia> :p
19:48:29 <willo> Yeah, saw that one
19:48:38 <willo> Would any more diagrams like this be useful?
19:48:48 <nirik> there was someone interested in making a network diagram a while
back, but they just wandered off.
19:48:57 <willo> Or is it changing to quickly to keep up to date
19:48:59 <nirik> if someone would like to make one, it could be useful yeah
19:49:01 <threebean> heh, there's a fedmsg diagram ->
19:49:05 <threebean> also somewhat out of date
19:49:12 <willo> :)
19:49:27 <danofsatx> Since Nagios is running, can't it spit out a diagram?
19:49:36 <willo> well network is defintely up my alley, I could start there
19:49:40 <nirik> danofsatx: it can yeah.
19:49:51 <nirik> https://admin.fedoraproject.org/nagios/
19:50:15 <johe_> on nagios again? wasnt it zabbix?
19:50:42 <nirik> we were never fully on zabbix. There was an attempted migration,
but it was scrapped.
19:50:43 <willo> Looked at nagios before and I got password prompt
19:50:56 <adimania> won't it depend on the parents which we defined?
19:50:58 <willo> guess fi-apprentice doesn't give me the power...:_
19:51:14 <johe_> nirik, well i was on the noc team when they did the switch, was
just a bit confused
19:51:35 <adimania> the map at https://admin.fedoraproject.org/nagios/
19:51:40 <nirik> willo: you should be able to use your fedora account login/pass
19:51:52 <willo> I think anything out of nagios is only as good as your service
checks and dependancies
19:51:58 <nirik> adimania: yes, it's based on what nagios has... but I think
it's actually reasonably accurate now.
19:52:12 <willo> Which can be hard to describe a lot of interactions about data
19:52:17 <nirik> johe_: yeah, that was before I was too involved...
19:53:28 <nirik> anyhow, does that answer the question?
19:54:03 <janeznemanic> it's complicated and big
19:54:19 <nirik> any other items for open floor?
19:54:23 <danofsatx> Since I'm here, I'd like to give a quick intro of
myself to the infra team. I haven't sent the email to the list yet, though.
19:54:31 <willo> nirik: ah, ok cool.
19:54:36 <kamina1> Hi, I missed the introduction session. So, may I introduce
19:54:36 <johe_> janeznemanic, true, but nagios cant display the relation, such maps
are always huge
19:54:54 <nirik> danofsatx / kamina1: welcome. Please do.
19:54:59 <willo> Didn't try that after my mention in fedora-noc :)
19:55:24 <danofsatx> My name is dan, fas dmossor - I am a 20+ year veteran of the IT
feild, and maintain three separate mixed CentOS/Fedora/Windows networks in my day job.
19:56:06 <kamina1> Thanks. I am Kamina. Currently, I live in Germany. I am fresh
college grad, looking for improving my skills in coding and contribute to open source
19:56:17 <danofsatx> Technically, I am a Cisco certified network engineer, but I can
work with anything
19:56:33 <nirik> excellent. Welcome to both of you.
19:56:38 <kamina1> Thanks
19:56:44 <nirik> are you more interested in sysadmin or application development ? or
19:56:50 <danofsatx> about the only thing I can't do in IT is coding - my job is
to make sure the coders can do theirs.
19:57:04 <kamina1> I am interested in Application Dev
19:57:19 <danofsatx> and I in sysadmin/NOC work
19:57:24 <nirik> cool.
19:57:40 <willo> danofsatx: welcome from a fellow network engineer
19:57:52 <danofsatx> thanks willo ;)
19:58:01 <nirik> for sysadmin, see me after the meeting in #fedora-admin and we can
get you added to the apprentice group, for applications, see #fedora-apps and we can get
you pointed in the right direction. ;)
19:58:20 <nirik> ok, we are almost out of time... anything else before we close out
19:58:33 <threebean> busy meeting today!
19:58:37 <threebean> good to see so many new people :)
19:58:39 <nirik> yeah, great stuff.
19:58:44 <nirik> always good seeing new folks...
19:58:49 <danofsatx> I'll tackle the diagram, in consultation with willo
19:59:14 <relrod> \o/
19:59:16 <nirik> great. Happy to assist.
19:59:22 <nirik> ok, thanks for coming everyone!
19:59:30 <nirik> #endmeeting