Summary/Minutes from today's Fedora Infrastructure meeting (2015-01-15)

[mohamed fawzy]

I am so so sorry i received the mail of meeting reminder late i will be in
next week

On Thu, Jan 15, 2015 at 9:07 PM, Kevin Fenzi <kevin at scrye.com> wrote:

> ============================================
> #fedora-meeting: Infrastructure (2015-01-15)
> ============================================
>
>
> Meeting started by nirik at 18:00:27 UTC. The full logs are available at
>
> http://meetbot.fedoraproject.org/fedora-meeting/2015-01-15/infrastructure.2015-01-15-18.00.log.html
> .
>
>
>
> Meeting summary
> ---------------
> * aloha  (nirik, 18:00:27)
>
> * New folks introductions and Apprentice tasks.  (nirik, 18:03:34)
>
> * Applications status / discussion  (nirik, 18:09:01)
>   * blockerbugs nearing rhel7/ansible migration ready  (nirik, 18:10:35)
>   * LINK: http://play.grafana.org/ does look nice  (pingou, 18:23:04)
>
> * Sysadmin status / discussion  (nirik, 18:23:40)
>
> * Upcoming Tasks/Items  (nirik, 18:30:42)
>   * LINK: https://apps.fedoraproject.org/calendar/list/infrastructure/
>     (nirik, 18:30:42)
>
> * An App overview - koji  (nirik, 18:35:34)
>
> * Open Floor  (nirik, 18:58:26)
>   * LINK: https://apps.fedoraproject.org/notifications   (threebean,
>     19:01:14)
>
> Meeting ended at 19:06:54 UTC.
>
>
>
>
> Action Items
> ------------
>
>
>
>
>
> Action Items, by person
> -----------------------
> * **UNASSIGNED**
>   * (none)
>
>
>
>
> People Present (lines said)
> ---------------------------
> * nirik (124)
> * threebean (31)
> * herlo (28)
> * pingou (25)
> * mizdebsk (18)
> * sborza (17)
> * smooge (13)
> * dgilmore (9)
> * zodbot (7)
> * tflink (6)
> * michel_slm (5)
> * mhurron (3)
> * fale (3)
> * webpigeon (1)
> * relrod (1)
> * mirek-hm (1)
> * abompard (1)
> * puiterwijk (0)
> * abadger1999 (0)
> * lmacken (0)
> * mdomsch (0)
> --
> 18:00:27 <nirik> #startmeeting Infrastructure (2015-01-15)
> 18:00:27 <zodbot> Meeting started Thu Jan 15 18:00:27 2015 UTC.  The chair
> is nirik. Information about MeetBot at http://wiki.debian.org/MeetBot.
> 18:00:27 <zodbot> Useful Commands: #action #agreed #halp #info #idea #link
> #topic.
> 18:00:27 <nirik> #meetingname infrastructure
> 18:00:27 <nirik> #topic aloha
> 18:00:27 <nirik> #chair smooge relrod nirik abadger1999 lmacken dgilmore
> mdomsch threebean pingou puiterwijk
> 18:00:27 <zodbot> The meeting name has been set to 'infrastructure'
> 18:00:27 <zodbot> Current chairs: abadger1999 dgilmore lmacken mdomsch
> nirik pingou puiterwijk relrod smooge threebean
> 18:00:32 * pingou here
> 18:00:58 <smooge> here
> 18:01:04 <webpigeon> o/
> 18:01:19 * threebean
> 18:01:39 * relrod here
> 18:02:55 * michel_slm here
> 18:03:22 * sborza here
> 18:03:24 <nirik> alright, I guess lets get started then...
> 18:03:26 * herlo here
> 18:03:34 <nirik> #topic New folks introductions and Apprentice tasks.
> 18:03:42 <nirik> any new folks like to introduce themselves?
> 18:03:48 <nirik> or apprentices with questions or comments?
> 18:04:10 * sborza raises hand
> 18:04:23 * fale is here too
> 18:04:28 * abompard here too :-)
> 18:04:29 <sborza> my ssh access may not be working now that i've moved
> accounts?
> 18:04:50 <nirik> sborza: it should have activated on the new account... we
> can debug after the meeting in #fedora-admin?
> 18:04:55 <mhurron> actually, management has moved from puppet to ansible,
> correct?
> 18:04:56 <sborza> seems lke the pub key is getting denied...i've updated
> the config based on:
> http://infrastructure.fedoraproject.org/infra/docs/sshaccess.txt
> 18:05:04 <sborza> nirik: sure thing, thanks
> 18:05:27 <pingou> mhurron: is moving, yes
> 18:05:34 <nirik> mhurron: we are in the process of doing so yeah...
> 18:05:43 <nirik> we have ~50 machines left in puppet...
> 18:05:45 <mhurron> the apprentice wiki page has a workflow with puppet, is
> there one for ansible? or a page that details what is in puppet and what is
> in ansible?
> 18:06:09 * tflink shows up late :-/
> 18:06:13 <nirik> there's a migration page for puppet->ansible move, but
> yeah we should update things for ansible
> 18:06:44 * herlo blames tflink
> 18:06:48 <herlo> :)
> 18:07:26 <nirik> mhurron: I'll try and update that page, or provide info
> to someone else who wants to. ;)
> 18:07:39 <pingou> <hint> <hint>
> 18:07:47 <tflink> herlo: penalty for showing up late is being blamed for
> ... everything?
> 18:07:53 <herlo> tflink: exactly. :D
> 18:07:59 <herlo> tflink: it shows we love you.
> 18:08:03 <nirik> ha
> 18:08:13 <mhurron> thanks ... i'll do it if I can get the information
> 18:08:29 <tflink> huh, didn't realize it worked that way
> 18:08:37 <nirik> mhurron: excellent. Happy to answer questions in
> #fedora-admin, #fedora-noc, etc. ;)
> 18:08:45 <herlo> tflink: who better than those that care to give you a
> hard time?
> 18:09:01 <nirik> #topic Applications status / discussion
> 18:09:11 <nirik> any application news, status or info this week?
> 18:09:12 * tflink will have to remember that one :)
> 18:09:31 <tflink> the last el7 builds we needed for blockerbugs have been
> done and are in epel7-testing
> 18:09:40 <herlo> nirik: woud new application suggestions go here?
> 18:09:57 <tflink> waiting on final review for application changes to
> support el7 but that should be done this week
> 18:10:19 <nirik> tflink: awesome. Happy to work on migrating it.
> 18:10:21 <nirik> herlo: sure.
> 18:10:35 <nirik> #info blockerbugs nearing rhel7/ansible migration ready
> 18:10:42 <herlo> I wanted to propose we have a look at d-note.
> https://pthree.org/2014/06/09/officially-announcing-d-note-version-1-0/
> 18:10:54 <herlo> This is what it does: https://secrets.xmission.com/
> 18:11:06 <herlo> and it's on github here:
> https://github.com/atoponce/d-note
> 18:11:15 <herlo> I'm working on packaging it for Fedora now.
> 18:11:39 <pingou> I've been mostly working on pkgdb2 this week
> 18:11:55 <nirik> herlo: interesting.
> 18:11:58 * herlo is willing to deploy and maintain it...
> 18:12:07 <pingou> to adjust it as discuss with rel-eng to migrate the new
> package / new branch processes to pkgdb (instead of bugzilla)
> 18:12:17 <herlo> of course, I'll need some ansible guidance. :)
> 18:12:34 <nirik> herlo: well, you need a few more folks. I am very much
> not willing to deploy anything that only one person works on.
> 18:12:43 <nirik> and we need to decide if it would be of use to us.
> 18:12:49 <herlo> nirik: sure... which is why I am bringing it up now.
> 18:12:51 <nirik> but yeah, packaging it up would be first step
> 18:13:12 <sborza> in that line, i'd like to propose we look at
> graphite-web for metrics visualization:
> https://github.com/graphite-project/graphite-web
> 18:13:21 <herlo> nirik: the conversation of usefulness is in the concept
> of being able to share secrets in a place we trust.
> 18:13:21 <sborza> as well as updating our (way out of date) collectd
> configs
> 18:13:29 <herlo> sborza: we use graphite at LF. It's pretty nice.
> 18:13:33 <pingou> herlo: your secrets.xmission link doesn't work here
> 18:13:49 <herlo> pingou: really?
> 18:13:51 <herlo> weird...
> 18:13:57 <sborza> herlo: a lot of changes happening now, would love to get
> this added to the EPEL repos too
> 18:13:57 <fale> it works on my pc
> 18:13:58 <herlo> pingou: it's https.
> 18:14:02 <pingou> yes
> 18:14:06 <herlo> sborza: indeed...
> 18:14:17 <nirik> herlo: can you also use that to make open ones that can
> stay around? or are they always destroyed on one view?
> 18:14:27 * nirik is wondering if it could replace paste.
> 18:14:34 <herlo> pingou: well, you can stand up an instance quickly. from
> my repo at https://github.com/herlo/d-note
> 18:14:42 <sborza> nirik: would be happy to work with someone to get this
> tested/integrated (re: graphite-web)
> 18:14:43 <herlo> nirik: no. It's not intended for that.
> 18:14:49 <nirik> ok.
> 18:14:58 <herlo> nirik: he does have plans for user profiles and such.
> 18:15:09 <threebean> sborza: I'm interested in it, but can't volunteer too
> much time.  there's lots to consider.
> 18:15:18 * nirik notes we have a RFR process for these things. ;)
> 18:15:25 <herlo> nirik: and there's a possibility of allowing multiple
> people to see the note. Only each instance would be deleted upon viewing, I
> think.
> 18:15:26 <threebean> sborza: most of the work would probably be tied up in
> porting our custom plugins over.
> 18:15:39 <herlo> nirik: I just wanted to get a feeling if it would be
> useful first. :)
> 18:15:52 <sborza> threebean: fair enough, doesn't have to be asap but at
> some point soon
> 18:16:00 <nirik> herlo: sure.
> 18:16:10 <fale> It can create both private and public notes, but it
> missing the code highlight feature
> 18:16:27 <nirik> I'm not saying no to anything, just noting that we are
> very busy. :)
> 18:16:39 <threebean> sborza: first thing to look into would be requesting
> that all the components/packages get branched for EPEL7 (they already exist
> in EPEL6 which is nice)
> 18:17:08 <sborza> threebean: sounds good re: collectd...if we want the
> same for graphite-web should I do that myself?
> 18:17:21 <sborza> threebean: full disclosure, i'm part of the graphite-web
> steering committee :)
> 18:17:32 <herlo> nirik: as usual. Busy people are busy. :)
> 18:17:48 <threebean> sborza: cool ;p.  yeah, I actually just meant
> graphite-web needs the branch requests.  we already have collectd on el7.
> 18:17:55 * herlo is done proposing. The interest level is enough to get me
> started for sure.
> 18:18:19 <sborza> threebean: awesome, let's talk more in #fedora-admin (or
> #fedora-devel?)
> 18:18:40 <threebean> sborza: -admin works for this
> 18:18:58 <nirik> herlo: it looks like a cool thing in any case. Even if we
> don't have a instance.
> 18:18:59 <pingou> sborza: can it handle any apps, other than web app?
> 18:19:26 <michel_slm> sborza: are you planning to also package a
> prettified front-end like Team Dashboard or Graphene?
> 18:19:36 <herlo> nirik: right. The point I was hoping to make was that we
> could have 'locations' of trustiness.
> 18:19:49 <nirik> sure, could be handy
> 18:20:47 <nirik> ok, any other application type news?
> 18:21:04 <sborza> pingou: not sure what you're asking, example?
> 18:21:20 <pingou> sborza: we currently use collectd to monitor the load on
> fedmsg
> 18:21:39 <pingou> and on the queue of some application relying on fedmsg
> 18:21:45 <sborza> michel_slm: can do...graphene is awesome, metrilyx is
> decent and grafana is phenomenal
> 18:22:40 <threebean> sborza, pingou: those are the custom plugins I was
> talking about.  I'm almost certain graphite's tools can handle those cases.
> 18:22:47 <sborza> pingou: graphite-web can handle a variety of metrics
> collectors, including anything custom
> 18:22:56 <pingou> cool :)
> 18:23:04 <pingou> http://play.grafana.org/ does look nice
> 18:23:07 <michel_slm> sborza: awesome, let me know if you need reviewers
> 18:23:34 * mirek-hm is late but here
> 18:23:40 <nirik> #topic Sysadmin status / discussion
> 18:23:49 <nirik> ok, on the sysadmin side... the usual stuff. ;)
> 18:24:06 <nirik> We got a few bad drives replaced yesterday. Thanks
> puiterwijk for working with folks to get that done
> 18:24:26 <nirik> I've been working on getting the new rhel7/ansible koji
> hubs done so we can switch to them.
> 18:24:32 <sborza> michel_slm: fo sho, will do
> 18:24:41 <nirik> smooge has been burried in hardware ordering and budget
> fun. ;)
> 18:25:07 <smooge> hardware is ordered
> 18:25:13 <smooge> s390 box is being racked
> 18:25:16 <nirik> we also got a gigantic spike in web requests this morning
> that saturated our proxies for about 10min or so. Not sure what caused it...
> 18:25:20 <smooge> qa boxes will be installed this afternoon
> 18:25:22 <smooge> I broke DNS
> 18:25:42 <nirik> smooge: thats a new s390 koji hub?
> 18:25:54 <smooge> yes. I am getitjng it installed in our new racks
> 18:26:05 <nirik> I really would like to set it up with ansible and in our
> normal machine setup
> 18:27:32 <smooge> I have no idea how the current one is setup
> 18:27:35 <nirik> Nagios has been very noisy of late... hopefully we can
> settle it down some.
> 18:27:47 <smooge> since the s390 is somewhere not in redhat
> 18:27:57 <nirik> smooge: well, when you get the new one installed, let me
> know. I can talk to sharkcz and we can get it sorted. '
> 18:28:15 <nirik> it's up in boston I think...
> 18:28:40 <nirik> anyhow, I want to get all the secondary stuff setup in
> ansible and in our normal processes.
> 18:28:42 <smooge> probably. I thought at one time it was at IBM but not
> sure
> 18:28:48 <nirik> so they have 2fa and get regular updates and all that
> 18:29:01 <smooge> I am fine with that
> 18:29:44 <nirik> Oh, I made a db01.stg instance... and moved some
> databases from the old db02.stg to it.
> 18:30:03 <nirik> there's still a bunch more to move. If any
> sysadmin-mainer folks want to do that, please feel free. ;)
> 18:30:42 <nirik> #topic Upcoming Tasks/Items
> 18:30:42 <nirik>
> https://apps.fedoraproject.org/calendar/list/infrastructure/
> 18:30:52 <nirik> anyone have upcoming tasks they would like to schedule or
> note?
> 18:31:26 <pingou> fosdem and devconf in early Feb
> 18:32:13 <nirik> also smooge and I will be at our main datacenter in feb
> 18:32:27 <nirik> likely we will need some outages then... to add memory to
> machines.
> 18:32:57 <nirik> So, we have some time left in the meeting and I wanted to
> see about trying something new...
> 18:33:15 <threebean> oh?
> 18:33:20 <pingou> suspens :)
> 18:33:38 <nirik> I thought it would be nice when we have extra time to
> talk about one specific app of ours.
> 18:33:45 <nirik> how it's setup, what it does, etc.
> 18:33:55 <michel_slm> +1
> 18:33:58 <nirik> This week I could talk about koji or ansible... anyone
> have a preference?
> 18:35:20 <michel_slm> nirik: seems like you get to decide
> 18:35:20 <threebean> hm.  cool.
> 18:35:27 <nirik> ok, I guess I will pick one... koji, since I have been
> working on it.
> 18:35:34 <nirik> #topic An App overview - koji
> 18:35:54 <nirik> so, koji is our buildsystem. It has several parts...
> 18:36:18 <nirik> a hub that runs a wsgi / httpd application - right now
> this is koji03
> 18:36:48 <nirik> a secondary hub (could take over if primary died) that
> right now runs some cron jobs and kojira
> 18:36:52 <nirik> this is koji04
> 18:37:17 <nirik> kojira is a process that launches things like buildroot
> rebuilds and the like. You can see it in the web interface... jobs owned by
> it.
> 18:37:36 <nirik> There's also a pile of builders. They all talk to the hub
> to get jobs and report on status and such.
> 18:38:06 <nirik> koji uses a self signed cert with its own CA... because
> it uses certs to identify builders and people
> 18:38:33 <nirik> we also have a squid proxy - kojipkgs02 currently that
> sits in front of the koji packages urls...
> 18:38:50 <nirik> builders and other things that download from koji hit
> that squid where it's cached.
> 18:39:00 <nirik> (for packages and most things)
> 18:39:27 <nirik> builders run the 'kojid' process. This talks to the hub,
> and identifies with a cert.
> 18:40:13 <nirik> Current koji (rhel6/puppet) uses heartbeat to keep a
> application ip on the active hub (usually koji03)
> 18:40:28 <nirik> new koji I am working on (rhel7/ansible) will use
> keepalived
> 18:40:51 <nirik> Storage for koji is a nfs mounted volume from the netapp.
> 18:41:12 <nirik> It has to be mounted on the hub and any builders that do
> newrepo tasks (but not on builders that don't)
> 18:41:35 <threebean> question
> 18:41:35 <nirik> Thats kind of the high level... questions?
> 18:41:57 <threebean> those builders that do newrepo tasks
> 18:42:01 <threebean> do they also do other tasks?
> 18:42:04 <nirik> yep.
> 18:42:08 * threebean nods
> 18:42:12 <nirik> koji has the concept of 'channels'
> 18:42:28 <nirik> you can setup channels for things and then tell it
> specific builders do those channels.
> 18:42:32 <threebean> so, a job submitted by joe-user can potentially get
> run on a builder that has write-access to the netapp mount?
> 18:42:55 <nirik> yep. It would be in a mock chroot tho.
> 18:43:02 * threebean nods
> 18:43:08 <threebean> cool.
> 18:43:10 <nirik> that chroot shouldn't have any access...
> 18:43:34 <pingou> !
> 18:43:48 <nirik> We do have some special channels/rules... there's some
> special builders for secure boot for example that only do builds of
> packages that need secure boot signing.
> 18:44:10 <nirik> pingou: yes?
> 18:44:20 <pingou> Are all the koji hosts el6 still? (can the builders be
> moved to el7 separatly from the masters? (Although I imagine
> heartbeet/keepalive might be the problem))
> 18:45:02 <nirik> pingou: most of the builders are now f21. ;)
> 18:45:07 <pingou> oh true :)
> 18:45:36 <nirik> all the arm ones, the buildvm and buildhw ones are all
> f21.
> 18:45:49 <nirik> the bkernel ones are still f20. I need to upgrade them.
> 18:46:09 <nirik> the buildppc ones are rhel6 I think... but we only need
> ppc for epel
> 18:46:53 <nirik> koji03/04 and kojipkgs02 are all rhel6/puppet. I am
> working now to move them to koji01/02, kojipkgs01 that is ansible and rhel7
> 18:48:13 <nirik> oh, there's also a db host... I already moved it to rhel7
> 18:48:54 <nirik> ok, any other questions? does someone want to do some
> other app next week? :)
> 18:49:47 <threebean> oh - here's a question.  the new koschei service
> seems to be hitting koji pretty hard.
> 18:50:13 <threebean> is koji handling that new load OK?
> 18:50:18 <nirik> yeah, it has a limiter... I think it only ever does 40
> jobs at once or something.
> 18:50:25 <pingou> .load
> 18:50:25 <zodbot> pingou: Error: You don't have the owner capability. If
> you think that you should have this capability, be sure that you are
> identified before trying again. The 'whoami' command can tell you if you're
> identified.
> 18:50:28 <nirik> yeah, perfectly fine.
> 18:50:31 <mizdebsk> 30*
> 18:50:54 <nirik> it's actually been nice when I had a problem with the f21
> buildvm's... I could count on it submitting jobs so I could see if we fixed
> it or not.
> 18:51:01 <nirik> mizdebsk: sorry, my mistake. ;)
> 18:51:01 <threebean> ;p
> 18:51:08 <mizdebsk> and koji load must be <50% at the time of executing
> new build by koschei
> 18:51:19 <mizdebsk> all that is configurable, feedback is welcome
> 18:51:42 <nirik> .buildload
> 18:51:43 <zodbot> nirik: Load: 116.9 Total: 272.0 Use: 43.0% (Medium Load)
> 18:52:16 <threebean> mizdebsk: looks fine here.  just curious about how
> its playing out.
> 18:52:16 <nirik> I think it's been fine. I haven't seen any issues with it.
> 18:52:41 <nirik> During mass rebuilds we get a nice backlog (submitting
> faster than it can build), but even then it chruns thru them pretty fast.
> 18:52:48 <nirik> we have 91 builders currently enabled.
> 18:53:06 <pingou> .builders
> 18:53:06 <zodbot> pingou: Enabled: 91 Ready: 78 Disabled: 69
> 18:53:26 <pingou> being used == Enabled - Ready?
> 18:53:42 <nirik> 46 arm, 27 buildvm, 12 buildhw, 2 bkernel, 2 buildppc
> 18:53:58 <mizdebsk> ready means load is low and builder is accepting tasks
> 18:54:12 <nirik> pingou: koji uses a loading setup... so you can tell it
> how much load a builder can handle
> 18:54:22 <nirik> when it gets that load or more it stops being ready to
> accept new jobs.
> 18:54:35 <pingou> ok
> 18:54:54 <nirik> % koji list-hosts
> 18:54:54 <nirik> Hostname                     Enb Rdy Load/Cap Arches
>      Last Update
> 18:54:54 <nirik> arm02-builder00.arm.fedoraproject.org Y   N    6.0/2.0
> armhfp           2015-01-15 18:53:06
> 18:55:09 <mizdebsk> it's obvious that koschei needs hardware - there were
> plans to add new hardware to koji instead having separate pool just for
> koschei
> 18:55:13 <nirik> so that builder can handle a load of 2.0, but it has a
> 6.0 job... so its not ready currently
> 18:55:36 <mizdebsk> also during mass rebuilds i'm suspending koschei
> 18:55:51 <pingou> nirik: cool, thanks
> 18:56:28 <nirik> mizdebsk: I wonder if we could do something with out
> staging koji... (which is currently not working). If it got regular builds
> it would help us know it's working along ok
> 18:57:08 <nirik> just an idle thought.
> 18:57:30 <nirik> pingou / threebean: either of you have an app you might
> want to talk about next week? ;)
> 18:57:34 <mizdebsk> maybe stagging koschei could use it? i started playing
> with ansible setup for koschei, but i don't have access to bastion and such
> 18:57:35 <threebean> sure :P
> 18:57:40 <smooge> mizdebsk, you said there were plans ...
> 18:57:45 <smooge> have they changed?
> 18:57:47 <pingou> nirik: I might be able to find one, maybe two ;-)
> 18:58:17 <nirik> mizdebsk: might work. I need to fix the issue where it's
> not starting builders, but then it would be good to look into
> 18:58:20 <nirik> cool.
> 18:58:26 <nirik> #topic Open Floor
> 18:58:32 <nirik> Anyone have anything for open floor?
> 18:58:33 <mizdebsk> smooge, currently i've been told that there will be
> budget for new hardware if it's needed
> 18:58:56 <mizdebsk> but so far it wasn't necessary - afaict koji is
> handling current load just fain
> 18:59:12 <mizdebsk> fine*
> 18:59:14 <threebean> mizdebsk: would you like a 'koschei' cert for
> koschei, so it can submit builds as itself instead of msimacek ?
> 18:59:27 <mizdebsk> threebean, yes, i've filled ticket for that
> 18:59:33 <threebean> cool, cool.
> 18:59:53 <smooge> mizdebsk, thanks. I just wanted to make sure if I needed
> to put in an order before the end of the year or not (or prepare for boxes
> to arrive)
> 19:00:00 <mizdebsk> threebean,
> https://fedorahosted.org/rel-eng/ticket/5941
> 19:00:14 * pingou gtg
> 19:00:20 <nirik> we really need to get thru more releng tickets in the
> releng meetings. ;) we will get there
> 19:00:25 <pingou> thanks for the meeting everyone, thanks for chairing
> nirik
> 19:00:25 <mizdebsk> i assume cert will be generated as part of RFR
> 19:00:58 <nirik> mizdebsk: yeah, we can get it sorted.
> 19:01:05 <nirik> ok, thanks for coming everyone...
> 19:01:11 <threebean> A couple things I forgot to mention in the apps
> section
> 19:01:12 <nirik> will close in a min if nothing else.
> 19:01:12 <threebean> 1) A new release of FMN got deployed as promised last
> week.  Lots of bugfixes and enhancements, but mostly a new set of nice
> defaults:
> 19:01:14 <threebean> https://apps.fedoraproject.org/notifications
> 19:01:15 <nirik> oh, go ahead
> 19:01:16 <threebean> You there's a button on each messaging context that
> let's you reset your filters to the new defaults if you'd like to test it
> out.
> 19:01:18 <threebean> 2) A new release of the-new-hotness got deployed to
> stg that follows up on bugs about real builds, not just scratch builds.
> There's a little more work to be done before we can put it in production.
> Should be done with that in a week or two.
> 19:01:20 <threebean> 3) We had some issues with FAS in fedmsg the past few
> weeks.  Two of the fas servers weren't publishing any messages.  Should be
> all fixed now.
> 19:02:21 <nirik> threebean: thanks for tracking that down.
> 19:02:21 <threebean> (heh, that's all.. ;p)
> 19:02:33 <threebean> np
> 19:02:36 <dgilmore> mizdebsk: its causeing issues
> 19:02:39 <nirik> I really need to redo fas01 too... but I want to try and
> confirm the cert stuff is all ok in asible
> 19:02:41 <nirik> ansible
> 19:02:53 <dgilmore> mizdebsk: we did some things to have java builds
> mostly go to x86
> 19:03:19 <dgilmore> mizdebsk: and because of the load of the tasks its not
> happening causeing complaints from the java folks
> 19:03:45 <nirik> dgilmore: huh... was this eclipse folks? or ?
> 19:04:01 <dgilmore> nirik: no. the people doing other java builds
> 19:04:04 <mizdebsk> dgilmore, koschei runs some tasks with --arch-override
> x86_64 because arm builders are not usable for java imho
> 19:04:10 <mizdebsk> that may be the reason
> 19:04:36 <dgilmore> mizdebsk: its causing issues
> 19:04:39 <nirik> ok, because one of the arm builders in the eclipse
> channel is down with a dead drive... so it's only got 1 left I think,
> making it slow
> 19:05:06 <dgilmore> java on arm is missing a jit so its slow
> 19:05:08 <mizdebsk> dgilmore, but this is really issue related to arm and
> java, not koschei itself
> 19:05:21 <mizdebsk> are there are any other issues with koschei?
> 19:05:33 <dgilmore> it is something that is going to need some planning
> and coordination
> 19:05:48 <nirik> perhaps we could continue this over in #fedora-releng? we
> are over time on our meeting. ;)
> 19:05:56 <dgilmore> mizdebsk: yes but koschei is breaking what we are
> doing to deal with it
> 19:06:18 * mizdebsk is fine with moving to different channel
> 19:06:49 <dgilmore> nirik: sure
> 19:06:50 <nirik> Thanks for coming folks.
> 19:06:54 <nirik> #endmeeting
>
> _______________________________________________
> infrastructure mailing list
> infrastructure at lists.fedoraproject.org
> https://admin.fedoraproject.org/mailman/listinfo/infrastructure
>



-- 
*Mohamed Fawzy *
Fedora <https://fedoraproject.org/wiki/User:Mohamedfawzy>
<https://fedoraproject.org/wiki/User:Mohamedfawzy>project Ambassador
<https://fedoraproject.org/wiki/User:Mohamedfawzy>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.fedoraproject.org/pipermail/infrastructure/attachments/20150115/aa887eea/attachment-0001.html>