Summary/Minutes from today's Release Engineering Meeting (2014-11-10)

Mon Nov 10 17:14:13 UTC 2014

======================================
#fedora-meeting-1: RELENG (2014-11-10)
======================================

Meeting started by nirik at 14:30:01 UTC. The full logs are available at
http://meetbot.fedoraproject.org/fedora-meeting-1/2014-11-10/releng.2014-11-10-14.30.log.html
.

Meeting summary
---------------
* init process  (nirik, 14:30:02)

* Secondary Architectures update - s390  (nirik, 14:35:15)

* Tickets  (nirik, 14:38:26)

* Secondary Architectures update - ppc  (nirik, 14:38:47)

* Tickets  (nirik, 14:42:14)
  * LINK:
    https://fedorahosted.org/rel-eng/report/10?sort=created&asc=1&page=1
    (nirik, 14:42:23)
  * LINK: http://infrastructure.fedoraproject.org/infra/db-dumps/ (btw)
    (nirik, 14:56:40)

* Open Floor  (nirik, 15:03:26)
  * LINK: http://paste.fedoraproject.org/149368/56331421   (tyll,
    15:25:49)

Meeting ended at 15:34:14 UTC.

Action Items
------------

Action Items, by person
-----------------------
* **UNASSIGNED**
  * (none)

People Present (lines said)
---------------------------
* nirik (85)
* tyll (36)
* sharkcz (16)
* bochecha (6)
* zodbot (4)
* pbabinca (2)
* masta (1)
* pbrobinson (0)
* dgilmore (0)
--
14:30:01 <nirik> #startmeeting RELENG (2014-11-10)
14:30:02 <zodbot> Meeting started Mon Nov 10 14:30:01 2014 UTC.  The chair is nirik. Information about MeetBot at http://wiki.debian.org/MeetBot.
14:30:02 <zodbot> Useful Commands: #action #agreed #halp #info #idea #link #topic.
14:30:02 <nirik> #meetingname releng
14:30:02 <zodbot> The meeting name has been set to 'releng'
14:30:02 <nirik> #chair dgilmore nirik tyll sharkcz bochecha masta pbrobinson
14:30:02 <nirik> #topic init process
14:30:02 <zodbot> Current chairs: bochecha dgilmore masta nirik pbrobinson sharkcz tyll
14:30:10 <nirik> who all is around for a releng meeting?
14:30:15 <bochecha> I am
14:30:42 * pbabinca here
14:32:44 * sharkcz is here
14:33:06 * nirik will wait another minute or two for folks to wander in
14:35:13 <nirik> ok, I guess lets go ahead...
14:35:15 <nirik> #topic Secondary Architectures update - s390
14:35:23 <nirik> sharkcz: any news to report?
14:36:24 <sharkcz> not much, I created a compose a week ago, roughly like beta in primary, works fine, but there is a problem with writing to console in 3.17 kernel, bug is reported and IBM is looking on it
14:36:52 <nirik> ah, ok.
14:37:01 <nirik> so no official beta until thats squashed?
14:37:07 <sharkcz> bug #1158848 for the record, network login works
14:37:20 <sharkcz> yes
14:38:00 <nirik> ok, makes sense.
14:38:19 <nirik> no pbrobinson or masta, I guess we skip ppc and arm for now.
14:38:26 <nirik> #topic Tickets
14:38:36 <sharkcz> I can comment ppc a bit
14:38:41 <nirik> ok, coo.
14:38:47 <nirik> #topic Secondary Architectures update - ppc
14:39:32 <sharkcz> there is beta rc1, but is blocked by 2 issues - cryptsetup fails on arches with page size >=64k (ppc* and aarch64), fix exists upstream so I need to ask jwb to include it
14:40:18 <nirik> ok.
14:40:47 <sharkcz> another issue is multipath install, seems to work on x86 in minimal setup (kvm + 1 multipathed scsi disk), we are reproducing this setup on ppc and will see later today hopefully
14:41:35 <nirik> alright, sounds good.
14:41:42 <nirik> so hopefully soon for a beta
14:41:53 <sharkcz> yep
14:42:07 <nirik> excellent. thanks.
14:42:14 <nirik> #topic Tickets
14:42:23 <nirik> https://fedorahosted.org/rel-eng/report/10?sort=created&asc=1&page=1
14:42:30 <nirik> we have 14 tickets open with meeting...
14:42:39 <nirik> not sure which we can/should go over today
14:42:51 <nirik> (with many folks not here)
14:43:48 <nirik> sharkcz: did https://fedorahosted.org/rel-eng/ticket/6014 get a final list? or do we even have a way to find out?
14:45:01 <sharkcz> I don't have the list at hand, but should be doable by a direct query to koji db
14:45:25 <sharkcz> question is whether it is worth the effort
14:45:39 <nirik> yeah, no idea on that part. perhaps it's just history now.
14:46:11 <sharkcz> yes, I'd close the ticket, very likely there are new builds for the affected packages
14:46:24 <nirik> ok.
14:46:46 <nirik> also https://fedorahosted.org/rel-eng/ticket/6024 can probibly be closed, we discussed it last week and I think pingou already implemented what we talked about.
14:48:55 <nirik> ok, those closed. ;) then there were 12. ;)
14:50:06 <bochecha> 6039 seems like something we could solve quickly?
14:50:42 <nirik> bochecha: off hand I would say we could publish dumps, but I am not fully sure there's no sensitive info in there.
14:50:49 <nirik> I would want to check with dgilmore on that...
14:51:47 <nirik> we have a framework already in place for sharing db's... but not sure koji would be ok to share that way.
14:52:09 <sharkcz> but won't be the dumps too large? I have 7GB+ on s390, the dump, compressed
14:52:39 <sharkcz> and primary has much more info in it I guess
14:53:12 <nirik> huh... primary ones are only 1.3gbish
14:53:14 <bochecha> looking at the db for another koji, I can't think of anything that could be sensitive
14:53:25 <nirik> -rw-r--r--. 1 postgres postgres 1.3G Nov 10 04:55 koji-2014-11-10.dump.xz
14:53:48 <sharkcz> let me check it, I hope I read the scale correctly :-)
14:54:18 <sharkcz> ah, it's 700M ;-)
14:54:24 <nirik> cool.
14:55:13 <nirik> bochecha: I don't know if we setup initial admins when installing with passwords instead of certs? otherwise, yeah, not sure what could be sensitive.
14:55:30 <bochecha> nirik, ah, that maybe
14:56:11 <nirik> anyhow I can check with dgilmore and see what he thinks... it would be pretty trivial to add it to the db's we already make available.
14:56:40 <nirik> http://infrastructure.fedoraproject.org/infra/db-dumps/ (btw)
14:57:34 <nirik> any other tickets we think we can make progress on? or should we call it a short meeting?
14:58:13 <bochecha> about 6016, didn't we say we'd like some data about savings?
14:58:40 <sharkcz> #6023 - should we express our opinions the ticket?
14:59:19 <nirik> bochecha: yeah, you want to add a comment about that there?
14:59:39 <bochecha> pbabinca, did you have any time to look at savings from using fedpkg-minimal? :)
14:59:59 <pbabinca> bochecha, I haven't made any progress there.
15:00:00 <nirik> sharkcz: sure I guess. Again, I would want to check with dgilmore before adding any folks there... it's pretty sensitive...
15:00:12 <nirik> but we should/could add some more folks, especially in .eu
15:03:26 <nirik> #topic Open Floor
15:03:31 <nirik> anyone have items for open floor?
15:04:18 <tyll> can we decide something about https://fedorahosted.org/rel-eng/ticket/6040 - e.g. can an existing systems be used for it?
15:05:09 <nirik> tyll: possibly releng04... but it's rhel6... does that present problems?
15:05:51 <tyll> nirik: uh, if I can login there, I can check - I only ran it on Fedora and RHEL7
15:06:23 <nirik> tyll: I can get you access to it if you don't have it already.
15:06:45 <nirik> thats the machine fedora updates pushes happen on, so it sends already bodhi emails and such.
15:06:52 <tyll> yes, I can access it
15:06:58 <nirik> or we could just make a new releng-backend or something.
15:07:26 <tyll> Can I add an ansible task for the cronjob for it or does it need to go into puppet?
15:08:33 <nirik> sadly, it's in puppet.
15:09:06 <nirik> let me think on it a bit more perhaps...
15:09:07 <tyll> the script does not work out-of-the box on it :-(
15:09:23 * nirik only saw the request this morning before coffee... and have had no time to ponder on it.
15:09:30 <tyll> ok
15:10:02 <nirik> ok, we can make a releng-backend01 perhaps. that makes more sense off hand anyhow.
15:12:09 <nirik> tyll: oh, where are we on rawhide autosigner?
15:13:51 <tyll> I just started looking into it again, it seems that secondary sigul is more stable than a few months ago (or it is because I am monitoring it now)
15:14:44 <nirik> I saw your report of an outage on it... no idea on that. I was asleep... nothing should have been changing on it.
15:14:59 <tyll> It seems to work quite good (still without rawhide gating), except that arm koji shows some issues sometimes
15:15:15 <tyll> ah, there is also something I need to investigate
15:15:37 <tyll> sometimes builds are not properly tagged on secondary kojis, even though a tagging fedmsg was sent earlier
15:15:42 <tyll> not sure if the packages are untagged again
15:16:15 <tyll> or if there is a race condition that makes koji send the fedmsg message before tagging was finished so a query for it fails
15:16:26 <nirik> ah...dunno.
15:16:47 <nirik> one side effect of the rawhide signing is we get the weird metadata problems with pungi making boot.img
15:17:01 <nirik> I think we should put a 'yum clean metadata' in there to work around it.
15:17:26 <tyll> does it fail if RPM checksums fail?
15:17:30 <tyll> checksums change?
15:17:38 <nirik> yeah...
15:18:04 <tyll> I see, this should not happen anymore when there is some gating in koji
15:18:21 <nirik> Error downloading packages:
15:18:21 <nirik> Delta RPM of 1:openssl-libs-1.0.1j-1.fc22.x86_64: Checksum of the delta-rebuilt RPM failed
15:18:35 <nirik> also, probibly it shouldn't be using deltas. ;)
15:18:42 <tyll> hehe
15:19:02 <nirik> anyhow, will see about adding a clean metadata in there.
15:19:32 <tyll> btw. would the releng cron mailing list be the right target to let autosigner report errors via e-mail?
15:20:21 <nirik> Oh, one other tidbit: I got all the builders re-isntalled/updated. they are all runinning 3.17.2-200.fc20 now (aside the ppc and bkernel ones)
15:20:30 <nirik> tyll: sounds good to me. +1
15:21:18 <nirik> ok, if nothing else will close out in a minute.
15:21:35 <tyll> btw. I also added scripts to releng-git a while ago to check whether packages are not signed when mashing them
15:21:50 <tyll> and one script to monitor sigul by checking whether login works every 10 minutes
15:22:01 <tyll> is the last one something that could be added to nagios?
15:22:05 <nirik> mash can check/reject them no?
15:22:10 <nirik> yes, definitely.
15:22:21 <nirik> if it was, then we could more quickly restart them if they got stuck.
15:22:24 <tyll> it looks into the mash logs, because there are only awrnings currently
15:22:53 <nirik> mash has a config option to treat that as a warn or a fail
15:22:58 <tyll> for the check script it would make sense to create a test key that is not used for anything to make it not that critical to use the credentials for it
15:23:12 <tyll> for Rawhide and secondary Branched it is a warning currently
15:23:39 <tyll> the scripts looks into the logs for the warning (I use it to see if autosigner signs all builds)
15:23:44 <nirik> yeah, for bodhi/updates it's strict fail
15:24:02 <nirik> we could make branched that way now too...
15:24:13 <nirik> yeah, test key sounds good.
15:24:17 <tyll> I believe it is for primary Branched
15:24:28 <tyll> ah, this reminds me of one more thing
15:24:56 <tyll> buildbranched and buildrawhide should be merged, but there is at least one diff where I am not sure whether it is intended that they differ
15:25:11 <nirik> yeah, we have wanted to merge them for a long time. ;)
15:25:28 <nirik> whats the diff?
15:25:30 <tyll> I synced most of them
15:25:49 <tyll> http://paste.fedoraproject.org/149368/56331421
15:26:01 <tyll> there is a call for wait
15:26:03 * masta wonders in late
15:28:24 <nirik> huh, no idea why thats there?
15:28:31 <nirik> oh, unless it needs all arches to finish?
15:28:39 <nirik> no, no idea
15:29:56 <tyll> so can it just be synced to buildrawhide?
15:30:34 <nirik> as far as I can tell.
15:30:45 <nirik> there's actually nothing being backgrounded there is there?
15:31:41 <tyll> pungify is
15:31:59 <tyll> and spam-o-matic
15:32:19 <nirik> ok, wonder why rawhide doesn't wait for those and branched does?
15:32:44 <nirik> I would think both should
15:33:23 <tyll> uh, actually both wait before pungify
15:33:47 <nirik> ok... lets continue this after meeting?
15:33:53 <tyll> but rawhide has an extra wait (So I confused both)
15:33:59 <tyll> ok
15:34:12 <nirik> thanks for coming everyone!
15:34:14 <nirik> #endmeeting
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 819 bytes
Desc: OpenPGP digital signature
URL: <http://lists.fedoraproject.org/pipermail/rel-eng/attachments/20141110/65177a45/attachment-0001.sig>