#fedora-meeting: Infrastructure (2014-09-04)
Meeting started by nirik at 18:00:18 UTC. The full logs are available at
* welcome (nirik, 18:00:18)
* New folks introductions and Apprentice tasks (nirik, 18:03:02)
* Freeze reminder (nirik, 18:06:32)
* Applications status / discussion (nirik, 18:07:05)
* LINK: http://220.127.116.11/spechub/
under development, feedback welcome
(dist-git frontend) (nirik, 18:16:08)
* taskotron to hit prod after alpha hopefully and replace autoqa once
and for all (nirik, 18:16:25)
* bodhi1 updates this week to handle epel7 (nirik, 18:16:34)
* Sysadmin status / discussion (nirik, 18:19:17)
* Nagios recap (nirik, 18:26:13)
* Upcoming Tasks/Items (nirik, 18:27:07)
* LINK: https://apps.fedoraproject.org/calendar/list/infrastructure/
* Open Floor (nirik, 18:32:49)
* LINK: ssh://firstname.lastname@example.org/fedocal ->
ssh://email@example.com/fedocal (pingou, 18:45:11)
Meeting ended at 18:56:34 UTC.
Action Items, by person
People Present (lines said)
* nirik (75)
* pingou (64)
* bochecha (25)
* mirek-hm (15)
* tflink (13)
* dgilmore (13)
* threebean (12)
* zodbot (5)
* smooge (2)
* akshays (2)
* lmacken (1)
* michel_slm (1)
* oddshocks (1)
* danofsatx (1)
* danielbruno (1)
* relrod (1)
* janeznemanic (1)
* roshi (1)
* puiterwijk (0)
* abadger1999 (0)
* mdomsch (0)
18:00:18 <nirik> #startmeeting Infrastructure (2014-09-04)
18:00:18 <zodbot> Meeting started Thu Sep 4 18:00:18 2014 UTC. The chair is nirik.
Information about MeetBot at http://wiki.debian.org/MeetBot
18:00:18 <zodbot> Useful Commands: #action #agreed #halp #info #idea #link #topic.
18:00:18 <nirik> #meetingname infrastructure
18:00:18 <nirik> #topic welcome
18:00:18 <nirik> #chair smooge relrod nirik abadger1999 lmacken dgilmore mdomsch
threebean pingou puiterwijk
18:00:18 <zodbot> The meeting name has been set to 'infrastructure'
18:00:18 <zodbot> Current chairs: abadger1999 dgilmore lmacken mdomsch nirik pingou
puiterwijk relrod smooge threebean
18:00:51 * threebean is here
18:01:06 * pingou
18:01:14 * relrod here
18:01:16 <janeznemanic> hi
18:01:30 <bochecha> hi there
18:01:41 <dgilmore> hola
18:02:16 <akshays> hello
18:02:21 * mirek-hm is here
18:02:50 <nirik> hello everyone. ;)
18:03:02 <nirik> #topic New folks introductions and Apprentice tasks
18:03:16 <nirik> any new folks like to introduce themselves?
18:03:26 <nirik> or apprentices with questions or comments?
18:04:21 * michel_slm here
18:04:26 <akshays> Hi, i am new here. I am a sys admin.'
18:04:58 <nirik> akshays: welcome.
18:05:32 * oddshocks pops in
18:05:45 * danielbruno here
18:06:29 * roshi listens
18:06:32 <nirik> #topic Freeze reminder
18:06:43 <nirik> Just a reminder we are in freeze until f21 alpha goes out...
18:06:57 <nirik> see the note on the list for more information.
18:07:05 <nirik> #topic Applications status / discussion
18:07:15 <nirik> any application news/status?
18:07:20 <pingou> http://18.104.22.168/spechub/
18:07:24 <threebean> oo
18:07:29 <pingou> based on progit, made for pkgs.fp.o
18:07:46 <pingou> would integrate quite easily with gitolite2
18:08:02 <pingou> we'll have to see when we figure out how to handle gitolite3
18:08:04 <nirik> yeah, I still need to poke around on it.
18:08:45 <pingou> basically, for the integration w/ gitolite2, we can just call
and embed the output in the conf file
with our cron task
18:08:47 <nirik> this (or something like it) might meet the needs mirek-hm was
looking for for copr dist-git perhaps.
18:08:48 <mirek-hm> we (as copr team) started thinking about dist-git for copr, I
sent email to fedora-devel today, so please respond and discuss there
18:08:56 <pingou> oh, and it works fine on the 19,000+ git repos :)
18:09:15 <nirik> mirek-hm: yeah, I wonder if something like this might help that
18:09:18 <threebean> pingou: looks neat
18:09:20 <nirik> but we will have to discuss.
18:09:31 <pingou> threebean: thanks ;-)
18:09:38 <tflink> we've been finding small issues/annoyances with taskotron in
stg and fixing them as we go - no more pto for a while so taskotron production should be
do-able before beta
18:09:44 <mirek-hm> pingou: but that is "just" fronted to dist-git,
18:10:01 <pingou> mirek-hm: it's a front-end to our git repos, allows forking
18:10:28 <pingou> well, I need to work with threebean for the auth part that
requires caching the acls from pkgdb until they change
18:10:52 <mirek-hm> ok, that would be nice one we choose how to do our dist-git,
will check that.
18:10:55 <nirik> pingou: forks and commits to those folks are stored where? in git
with the project forked? or ?
18:11:20 <nirik> tflink: cool. ;) so, after alpha we can nuke autoqa and enable
18:11:24 <pingou> nirik: in a new git repo under the forks/ folder
18:11:34 <mirek-hm> pingou: https://fedorahosted.org/spechub/
does not exist (taken
18:11:43 <pingou> mirek-hm: request pending in trac ;-)
18:11:48 <mirek-hm> ok
18:11:50 <nirik> mirek-hm: I haven't processed his request to add it yet. ;)
18:11:51 <pingou> mirek-hm: code is in github ftm
18:12:00 <threebean> tflink: any chance there will be a new release of resultsdb
before you go to prod?
18:12:09 <tflink> nirik: that's the plan
18:12:17 <tflink> threebean: are there unreleased features?
18:12:20 * threebean nods
18:12:23 <threebean> unreleased bugfix
18:12:24 <nirik> pingou: so, you could fork say kernel, then commit whatever to it
and others can clone that,e tc?
18:12:31 <tflink> I thought that the jsonp stuff was already part of stg
18:12:59 <pingou> nirik: we don't allow forking a fork, but you can clone
locally a fork sure
18:13:02 <tflink> threebean: I'll look into that after the meeting, if we have
unreleased fixes, I'll do a release
18:13:12 <threebean> tflink: yeah, this one in particular ->
18:13:38 <nirik> pingou: ok.
18:14:47 <nirik> alright. Any other application news?
18:14:54 <pingou> oh
18:14:59 * danofsatx is here, but muy busy
18:15:02 <tflink> threebean: I think that's already been released - should be in
dev and stg
18:15:04 <pingou> I'm loading datagrepper's data into mongodb
18:15:21 <lmacken> I upgraded bodhi in production this week with changes for EPEL-7
18:15:23 <pingou> on 22.214.171.124
18:15:42 <pingou> db.messages.count() : 92541 -- it's slow to load
18:15:59 <threebean> clarification -> slow to populate, right?
18:16:08 <nirik> #info http://126.96.36.199/spechub/
under development, feedback
welcome (dist-git frontend)
18:16:25 <nirik> #info taskotron to hit prod after alpha hopefully and replace
autoqa once and for all
18:16:28 <pingou> threebean: yes
18:16:33 <pingou> arg, it failed :(
18:16:34 <nirik> #info bodhi1 updates this week to handle epel7
18:17:10 <threebean> tflink: not sure. I'll send a link in #fedora-apps
18:18:41 <threebean> pingou: yeah, it will be interesting to see if there's any
significant difference in query speed when mongo and postgres have the same number of
18:19:17 <nirik> #topic Sysadmin status / discussion
18:19:37 <nirik> we have had several mirrormanager blowups this week sadly.
18:19:38 <pingou> threebean: looking forward that :)
18:19:56 <mirek-hm> I would like to urge
(/me is looking at puiterwijk)
18:20:02 <nirik> One seems to have been due to bad data in the db causing the update
directory list to fail.
18:20:26 <pingou> a foreign key not enforced
18:20:29 <nirik> mirek-hm: yeah, not sure where puiterwijk is. ;( I will do that
myself after the meeting.
18:20:32 <pingou> and a delete not propagate
18:21:13 <nirik> anyhow, we will keep things going along on it until we replace it.
18:21:39 <nirik> we have also been having some vpn saturation issues...
18:22:06 <nirik> It's possible these are due to high fedmsg busgateway traffic.
So, we moved that to not go over the vpn...
18:22:23 <nirik> but that won't entirely take effect until clients restart
18:22:54 <mirek-hm> nirik: and regards
- nirik do you think 256 IPs
per tenant is enough? I would rather go with 172.N.x.x Class B networks for every tenant.
E.g. 172.1.x.x for copr, 172.2.x.x for permanent, 172.3.x.x for fooo....
18:23:07 <threebean> I'm preparing a freeze break request to push some of that
traffic back to the proxies which should help.
18:23:35 <nirik> mirek-hm: well, if you like, sure. I really don't think we are
going to have more than 256 per client, but I guess I could be wrong. Use Class B if you
18:24:23 <mirek-hm> that give us 16k networks and 65k IP per each network, that is
more then enough in both directions
18:25:24 <nirik> sure, works for me
18:26:13 <nirik> #topic Nagios recap
18:26:26 <nirik> .tiny
18:26:27 <zodbot> nirik: http://tinyurl.com/l3vjae8
18:26:34 <nirik> pretty much all of those are due to the vpn thing.
18:27:07 <nirik> #topic Upcoming Tasks/Items
18:27:07 <nirik> https://apps.fedoraproject.org/calendar/list/infrastructure/
18:27:20 <nirik> anyone have upcoming items they would like to note or schedule?
18:28:13 <pingou> threebean and I prepare an announce about FMN
18:28:21 <nirik> cool.
18:28:30 <nirik> just more widespread? or ?
18:28:35 <pingou> of the 1566 packagers we have in FAS, 1453 are not using FMN yet
18:28:50 <pingou> so we want to create their account automatically
18:29:06 <pingou> so we don't have a clear date for the change yet
18:29:14 <nirik> ok
18:29:15 <pingou> but I'd hope for early october
18:29:16 <mirek-hm> FMN?
18:29:25 <pingou> mirek-hm: https://apps.fedoraproject.org/notifications/
18:29:44 <nirik> basically a way to be notified as you like when fedmsg's
18:29:55 <nirik> instead of always emails
18:30:30 <threebean> you can use it to get emails about coprs, for instance.
18:31:17 <mirek-hm> hmm can someone add it on list of trusted apps for fedoauth?
18:31:30 <nirik> I thought we already did?
18:31:46 <mirek-hm> It just asked me to review the login
18:31:54 <nirik> if not, we can... (althought we are in freeze)
18:32:09 <mirek-hm> true, it can wait
18:32:43 <nirik> anyhow, it's a handy setup. ;)
18:32:49 <nirik> #topic Open Floor
18:32:55 <nirik> ok, any items for open floor?
18:33:22 * tflink is curious if there has been any status change on the new app server and
18:33:49 <nirik> tflink: qa09/virthost-comm03?
18:33:52 <tflink> yep
18:34:02 <nirik> smooge has been working to bring them up.
18:34:19 <nirik> I am not sure what the current status was. I know they are on the
net and all, just need installing (which he was doign)
18:34:32 * tflink just wants to get some stuff moved over to the new hosts before
warranties start expiring
18:34:43 <nirik> yeah, understandable.
18:34:54 <nirik> will poke him when he's around
18:35:04 <bochecha> nirik, what's the next step of moving distgit to ansible? I
kind of lost track after my patches were merged (I see you fixed some errors I made, then
it seems we're halfway through with moving to gitolite3?)
18:35:08 <tflink> we should be getting 2 more hosts in q2 and q3, still in process
18:35:45 <pingou> bochecha: nirik I can help until next week thursday if we want to
move closer at gitolite3
18:36:10 <nirik> bochecha: so yeah, pkgs01.stg is in ansible now. It's rhel7. It
has synced data on it. However, we have no gitolite2 in rhel7, so we wanted to look at
moving to 3, but it's apparently a big jump. pingou looked into it, but I haven't
had time yet.
18:36:30 <nirik> bochecha: also, cgit isn't working right, but thats likely
seliux or something minor
18:36:49 <pingou> nirik: we don't need no cgit, we have spechub :-p
18:37:09 * pingou ducks
18:37:13 <nirik> pingou: :)
18:37:48 <bochecha> pingou, so what's missing to move to gitolite3?
18:38:12 <smooge> ugh sorry
18:38:28 <tflink> unless there's an issue I don't know of, moving to
gitolite3 in the middle of a release that's already chaotic sounds like a bad idea
18:38:34 <nirik> hey smooge. tflink was asking about qa09/virthost-comm03.... any
18:38:34 <pingou> bochecha: basically in gitolite2 we have 1 conf file in /etc
18:38:46 <bochecha> tflink, right, we're only doing this on staging, not
18:38:55 <pingou> bochecha: that conf file defines where the repos are located
18:38:57 <tflink> ok, nvm then :)
18:39:08 <pingou> bochecha: in gitolite3 repos *must* be in ~/repositories
18:39:14 <nirik> tflink: well, mostly because we dont have the old version on
epel7... but of course we can punt and add it
18:39:23 <smooge> The boxes got to mgmt working. I have to start setup and
installation. I expect by Monday
18:39:38 <pingou> so since we use gitolite in such a way that each user has its own
account on the machine, it means that we need each user to have ~/repositories
18:39:47 <pingou> oh and ~/.gitolite.rc btw
18:39:49 <tflink> nirik: I thought that the idea was to move production soon, missed
the bit about stg
18:40:00 <bochecha> ah right
18:40:10 <pingou> so either we mess with $HOME
18:40:14 <bochecha> is there a reason why we don't use the recommended gitolite
setup? (only one user)
18:40:24 <pingou> or we find a way to move to that ^
18:40:35 <nirik> tflink: well, I would like to move prod (but not in freeze of
course). We are mostly just exploring the options in stg right now.
18:40:50 <pingou> which I expect means: 1) a new fedpkg version 2) break all the
clones of all the packages out there
18:41:05 <nirik> yeah, those don't seem... feasable
18:41:06 <bochecha> fwiw, at my previous dayjob I was using the "one user
only" recommended setup of gitolite for my internal distgit, it was working well
18:41:17 <bochecha> pingou, ah right, the fetch/push urls with the user in them :-/
18:41:22 <pingou> yup
18:41:33 <pingou> and that's where I stopped :)
18:41:39 <bochecha> well, a new fedpkg could rewrite those urls
18:41:46 <pingou> ftr
18:42:04 <pingou> they do speak about us in there
18:42:29 <pingou> and they seem to advice to mess with $HOME
18:42:44 <nirik> yeah, so sounding like we should just get v2 branched for now at
18:43:11 <bochecha> yeha, maybe gitolite2 (at least in th einfra repo) for el7 would
make things simpler for now
18:43:12 <pingou> bochecha: which means, breaking all the old fedpkg around :/
18:43:30 <bochecha> pingou, well, yeah, forcing an upgrade
18:43:34 <pingou> I wanted to check how hard it would be to build gitolite2 on koji,
but I haven't checked it yet
18:43:43 <bochecha> (which is coming anyway, when we move to sha512 for the
18:44:02 <bochecha> but yeah, doing it twice is certainly not nice
18:44:06 <pingou> nirik: ^ if we force a fedpkg upgrade already, that might be a
good time to do that
18:44:19 <bochecha> pingou, I don't htink we can do both at the same time
18:44:27 <pingou> that == move to the one gitolite user
18:44:30 <pingou> bochecha: why not?
18:44:45 <dgilmore> what does move to one gitolite user actually mean?
18:45:11 <bochecha> dgilmore, git clone firstname.lastname@example.org:foobar
18:45:11 <pingou> ssh://email@example.com/fedocal ->
18:45:21 <dgilmore> bochecha: yeah no
18:45:31 <dgilmore> bochecha: never will that be okay
18:45:43 * pingou needs more info
18:45:43 <dgilmore> we want to know who commited the changes
18:45:52 <bochecha> we still do with the gitolite@ scheme
18:45:59 <pingou> dgilmore: we do know, gitolite handles that
18:46:03 <nirik> its just at a gitolite layer
18:46:08 <dgilmore> ewww
18:46:11 <dgilmore> no
18:46:12 <bochecha> all users are 'gitolite' on the server, but it's
still mapped to the user's private key
18:46:26 <bochecha> dgilmore, it's the upstream recommended way, though
18:46:33 <dgilmore> bochecha: doesnt make it right
18:46:38 <bochecha> sure
18:46:42 <pingou> s/recommended/&designed/
18:46:52 <bochecha> it's just making it hard/troublesome to do differently
18:47:05 <dgilmore> then we should look at other options
18:47:12 <dgilmore> or work with upstream to make it sane
18:47:38 <pingou> dgilmore: they explicitely removed what allowed to do things the
way we did
18:47:47 <bochecha> knowing we were doing it this way
18:47:51 <dgilmore> pingou: then we find a replacement
18:48:03 <bochecha> pingou, you handle the acls directly in spechub, ok? :)
18:48:13 * pingou jumps
18:48:18 <bochecha> \o/
18:48:28 <pingou> no, ther other jump ;-)
18:49:09 <nirik> yeah, I think v2 for now and look for a longer term solution seems
like what we will have to do.