According to the schedule [1], Fedora 26 Candidate RC-1.4 is now available for testing. Please help us complete all the validation testing! For more information on release validation testing, see: https://fedoraproject.org/wiki/QA:Release_validation_test_plan
Test coverage information for the current release can be seen at: https://www.happyassassin.net/testcase_stats/26
You can see all results, find testing instructions and image download locations, and enter results on the Summary page:
https://fedoraproject.org/wiki/Test_Results:Fedora_26_RC_1.4_Summary
The individual test result pages are:
https://fedoraproject.org/wiki/Test_Results:Fedora_26_RC_1.4_Installation https://fedoraproject.org/wiki/Test_Results:Fedora_26_RC_1.4_Base https://fedoraproject.org/wiki/Test_Results:Fedora_26_RC_1.4_Server https://fedoraproject.org/wiki/Test_Results:Fedora_26_RC_1.4_Cloud https://fedoraproject.org/wiki/Test_Results:Fedora_26_RC_1.4_Desktop https://fedoraproject.org/wiki/Test_Results:Fedora_26_RC_1.4_Security_Lab
All Final priority test cases for each of these test pages [2] must pass in order to meet the RC Release Criteria [3].
Help is available on #fedora-qa on irc.freenode.net [4], or on the test list [5].
Current Blocker and Freeze Exception bugs: http://qa.fedoraproject.org/blockerbugs/current
[1] http://fedorapeople.org/groups/schedule/f-26/f-26-quality-tasks.html [2] https://fedoraproject.org/wiki/QA:Release_validation_test_plan [3] https://fedoraproject.org/wiki/Fedora_26_RC_Release_Criteria [4] irc://irc.freenode.net/fedora-qa [5] https://lists.fedoraproject.org/archives/list/test@lists.fedoraproject.org/
Looks like one compose failed *again* for some mirror issue… This time it is Robotic [1] , but i've seen these random failures several times, also for Astronomy for example. So a new compose should be done here to ensure the Spin gets in properly, it is not an issue with the Spin here but an issue somewhere in infrastructure.
[1]: https://koji.fedoraproject.org/koji/taskinfo?taskID=20315960
Error: 00:55:43,639 WARN packaging: Failed to download 'libxshmfence-1.2-4.fc26.i686.rpm': 3 - Curl error (28): Timeout was reached for https://kojipkgs.fedoraproject.org/compose/26/Fedora-26-20170703.0/compose/E... [Operation too slow. Less than 1000 bytes/sec transferred the last 30 seconds] 00:55:43,639 WARN packaging: Failed to download 'libxslt-1.1.29-1.fc26.i686.rpm': 3 - Curl error (28): Timeout was reached for https://kojipkgs.fedoraproject.org/compose/26/Fedora-26-20170703.0/compose/E... [Operation too slow. Less than 1000 bytes/sec transferred the last 30 seconds] 00:55:43,640 WARN packaging: Failed to download 'libxshmfence-1.2-4.fc26.i686.rpm': 1 - No more mirrors to try - All mirrors were already tried without success
Greetings, Christian
On 07/04/2017 08:50 AM, Adam Williamson wrote:
According to the schedule [1], Fedora 26 Candidate RC-1.4 is now available for testing. Please help us complete all the validation testing! For more information on release validation testing, see: https://fedoraproject.org/wiki/QA:Release_validation_test_plan
Test coverage information for the current release can be seen at: https://www.happyassassin.net/testcase_stats/26
You can see all results, find testing instructions and image download locations, and enter results on the Summary page:
https://fedoraproject.org/wiki/Test_Results:Fedora_26_RC_1.4_Summary
The individual test result pages are:
https://fedoraproject.org/wiki/Test_Results:Fedora_26_RC_1.4_Installation https://fedoraproject.org/wiki/Test_Results:Fedora_26_RC_1.4_Base https://fedoraproject.org/wiki/Test_Results:Fedora_26_RC_1.4_Server https://fedoraproject.org/wiki/Test_Results:Fedora_26_RC_1.4_Cloud https://fedoraproject.org/wiki/Test_Results:Fedora_26_RC_1.4_Desktop https://fedoraproject.org/wiki/Test_Results:Fedora_26_RC_1.4_Security_Lab
All Final priority test cases for each of these test pages [2] must pass in order to meet the RC Release Criteria [3].
Help is available on #fedora-qa on irc.freenode.net [4], or on the test list [5].
Current Blocker and Freeze Exception bugs: http://qa.fedoraproject.org/blockerbugs/current
[1] http://fedorapeople.org/groups/schedule/f-26/f-26-quality-tasks.html [2] https://fedoraproject.org/wiki/QA:Release_validation_test_plan [3] https://fedoraproject.org/wiki/Fedora_26_RC_Release_Criteria [4] irc://irc.freenode.net/fedora-qa [5] https://lists.fedoraproject.org/archives/list/test@lists.fedoraproject.org/
On Tue, 2017-07-04 at 09:32 +0200, Christian Dersch wrote:
Looks like one compose failed *again* for some mirror issue… This time it is Robotic [1] , but i've seen these random failures several times, also for Astronomy for example. So a new compose should be done here to ensure the Spin gets in properly,
We don't re-spin for non-blocking images as a matter of course. The compose process takes far too long for that, unfortunately.
I think nirik said he thought he'd fixed this, but obviously he hasn't...
Note this only seems to have affected the i386 image. The x86_64 image is included in the compose.
On 07/04/2017 03:46 PM, Adam Williamson wrote:
On Tue, 2017-07-04 at 09:32 +0200, Christian Dersch wrote:
Looks like one compose failed *again* for some mirror issue… This time it is Robotic [1] , but i've seen these random failures several times, also for Astronomy for example. So a new compose should be done here to ensure the Spin gets in properly,
We don't re-spin for non-blocking images as a matter of course. The compose process takes far too long for that, unfortunately.
I think nirik said he thought he'd fixed this, but obviously he hasn't...
Yeah, I keep doing things that I think will quash this and then it shows up again. :( It has been very hard to track down because it's going through a number of layers... apache to haproxy to varnish to apache. :(
Note this only seems to have affected the i386 image. The x86_64 image is included in the compose.
kevin
On Tue, Jul 4, 2017 at 11:41 PM, Kevin Fenzi kevin@scrye.com wrote:
On 07/04/2017 03:46 PM, Adam Williamson wrote:
On Tue, 2017-07-04 at 09:32 +0200, Christian Dersch wrote:
Looks like one compose failed *again* for some mirror issue… This time it is Robotic [1] , but i've seen these random failures several times, also for Astronomy for example. So a new compose should be done here to ensure the Spin gets in properly,
We don't re-spin for non-blocking images as a matter of course. The compose process takes far too long for that, unfortunately.
I think nirik said he thought he'd fixed this, but obviously he hasn't...
Yeah, I keep doing things that I think will quash this and then it shows up again. :( It has been very hard to track down because it's going through a number of layers... apache to haproxy to varnish to apache. :(
Sadly the aarch64 Minimal image also was hit by this :-(
On Tue, Jul 4, 2017 at 6:41 PM, Kevin Fenzi kevin@scrye.com wrote:
On 07/04/2017 03:46 PM, Adam Williamson wrote:
On Tue, 2017-07-04 at 09:32 +0200, Christian Dersch wrote:
Looks like one compose failed *again* for some mirror issue… This time it is Robotic [1] , but i've seen these random failures several times, also for Astronomy for example. So a new compose should be done here to ensure the Spin gets in properly,
We don't re-spin for non-blocking images as a matter of course. The compose process takes far too long for that, unfortunately.
I think nirik said he thought he'd fixed this, but obviously he hasn't...
Yeah, I keep doing things that I think will quash this and then it shows up again. :( It has been very hard to track down because it's going through a number of layers... apache to haproxy to varnish to apache. :(
Note this only seems to have affected the i386 image. The x86_64 image is included in the compose.
So... this prompts a question in my head. I understand we don't block release for non-blocking spins. That is obvious.
What is not obvious to me is if we would go back and do a compose later for those spins that are impacted by this. It is one thing to skip a spin if the maintainer has not kept up with the release and is otherwise negligent. It is quite another to tell them "too bad..." if the spin isn't composed for something completely out of their control such as an infrastructure issue. That, to me, would be discounting the work they've done throughout the release to maintain and test their spin.
Do we have that ability to do a post-GA compose for those spins in such cases and do we plan to do so?
josh
On Wed, Jul 5, 2017 at 12:50 PM, Josh Boyer jwboyer@fedoraproject.org wrote:
So... this prompts a question in my head. I understand we don't block release for non-blocking spins. That is obvious.
What is not obvious to me is if we would go back and do a compose later for those spins that are impacted by this. It is one thing to skip a spin if the maintainer has not kept up with the release and is otherwise negligent. It is quite another to tell them "too bad..." if the spin isn't composed for something completely out of their control such as an infrastructure issue. That, to me, would be discounting the work they've done throughout the release to maintain and test their spin.
Do we have that ability to do a post-GA compose for those spins in such cases and do we plan to do so?
Nope. This came up last cycle or two where compose had a weird bug preventing umount of the installation root fs image file, resulting in a corrupt file system and ensuing boot failure of the ISO. And actually there were some non-release blocking images that we simply did not ship because of those failures, and they were not respun, and releng said we couldn't even use a prior successfully built ISO.
I don't recall the reason, something to do with either signing or metadata or something... I think it's known an alternative would be nice, especially given the big pile of images we're creating now, but I don't know what work is needed.
On Wed, 2017-07-05 at 15:04 -0600, Chris Murphy wrote:
On Wed, Jul 5, 2017 at 12:50 PM, Josh Boyer jwboyer@fedoraproject.org wrote:
So... this prompts a question in my head. I understand we don't block release for non-blocking spins. That is obvious.
What is not obvious to me is if we would go back and do a compose later for those spins that are impacted by this. It is one thing to skip a spin if the maintainer has not kept up with the release and is otherwise negligent. It is quite another to tell them "too bad..." if the spin isn't composed for something completely out of their control such as an infrastructure issue. That, to me, would be discounting the work they've done throughout the release to maintain and test their spin.
Do we have that ability to do a post-GA compose for those spins in such cases and do we plan to do so?
Nope. This came up last cycle or two where compose had a weird bug preventing umount of the installation root fs image file, resulting in a corrupt file system and ensuing boot failure of the ISO. And actually there were some non-release blocking images that we simply did not ship because of those failures, and they were not respun, and releng said we couldn't even use a prior successfully built ISO.
I don't recall the reason, something to do with either signing or metadata or something... I think it's known an alternative would be nice, especially given the big pile of images we're creating now, but I don't know what work is needed.
Before Pungi 4, the concept of a compose was fairly abstract. There was very little that you could point at and call 'the compose'; because there was no compose-level metadata or PDC or anything like that, we could re-run the compose command for a single specific image and stuff it into the tree with the other images.
With Pungi 4 we really can't do that any more. Composes have a much more solid identity as an actual thing. 'A compose' has a compose ID, production composes have a label, and these can't really be duplicated. There is a bunch of metadata which is produced for the compose *as a whole*; if we do a compose where we try 10 images and 1 fails, the metadata for that compose will list 9 images. If we then ran another compose just to produce that 1 missing image (with properties as similar as possible to the original compose), the metadata for the new compose would include just that 1 image, or all 10 images, and there is no way to somehow 'edit in' the information about the new image to the original compose's metadata (besides doing it by hand or hacking up a script or something, both of which would be error-prone). It's also not actually that easy just to respin a single image; Pungi 4 isn't really built to do that (you'd have to create a new Pungi compose config file each time you wanted to do it), and releng really doesn't want to do it manually below the Pungi level any more in case of inconsistencies or errors.
See the metadata for the RC-1.4 compose here: https://kojipkgs.fedoraproject.org/compose/26/Fedora-26-20170703.0/compose/m...
Note that for dumb reasons that I hate we actually strip the metadata from the final compose when it's actually approved and shipped out to mirrors, but when a compose completes, the metadata related to that compose is uploaded to PDC: https://pdc.fedoraproject.org/ . Once that's done it's not expected, and it may in fact be impossible, for that metadata to be edited, and it's never removed. So even though the metadata for Alpha, Beta and Final composes can't be found on the mirrors with them, it *can* be seen in PDC.
However, as I said before, we *do* have some ways we can unofficially provide 'missing' images. They just can't be a full part of the official release.
On Wed, Jul 05, 2017 at 02:33:01PM -0700, Adam Williamson wrote:
With Pungi 4 we really can't do that any more. Composes have a much more solid identity as an actual thing. 'A compose' has a compose ID, production composes have a label, and these can't really be duplicated. There is a bunch of metadata which is produced for the compose *as a whole*; if we do a compose where we try 10 images and 1 fails, the metadata for that compose will list 9 images. If we then ran another compose just to produce that 1 missing image (with properties as similar as possible to the original compose), the metadata for the new
I understand the value of having something that is an authoritative source of truth about release artifacts. I'm not sure at all about the value of having The Compose like this. Especially since we already do things like assuming that tests which apply to RC 1.4 are probably good for 1.5 since we know none of the relevant packages have changed. I guess we could solve this with another layer of abstraction: a super-compose which could include artifacts from various composes or other compose-like sources.
script or something, both of which would be error-prone). It's also not actually that easy just to respin a single image; Pungi 4 isn't really built to do that (you'd have to create a new Pungi compose config file each time you wanted to do it), and releng really doesn't want to do it manually below the Pungi level any more in case of inconsistencies or errors.
Maybe that just means it needs to be easier to create and update pungi compose configs?
However, as I said before, we *do* have some ways we can unofficially provide 'missing' images. They just can't be a full part of the official release.
But they can still be "official Fedora" even if not part of an "official release".
On Thu, 2017-07-06 at 10:05 -0400, Matthew Miller wrote:
On Wed, Jul 05, 2017 at 02:33:01PM -0700, Adam Williamson wrote:
With Pungi 4 we really can't do that any more. Composes have a much more solid identity as an actual thing. 'A compose' has a compose ID, production composes have a label, and these can't really be duplicated. There is a bunch of metadata which is produced for the compose *as a whole*; if we do a compose where we try 10 images and 1 fails, the metadata for that compose will list 9 images. If we then ran another compose just to produce that 1 missing image (with properties as similar as possible to the original compose), the metadata for the new
I understand the value of having something that is an authoritative source of truth about release artifacts. I'm not sure at all about the value of having The Compose like this. Especially since we already do things like assuming that tests which apply to RC 1.4 are probably good for 1.5 since we know none of the relevant packages have changed. I guess we could solve this with another layer of abstraction: a super-compose which could include artifacts from various composes or other compose-like sources.
Expect my resignation letter attached to a copy of the fedfind source, with a note saying "I'm damned if I'm writing ANOTHER layer of this crap". :P
script or something, both of which would be error-prone). It's also not actually that easy just to respin a single image; Pungi 4 isn't really built to do that (you'd have to create a new Pungi compose config file each time you wanted to do it), and releng really doesn't want to do it manually below the Pungi level any more in case of inconsistencies or errors.
Maybe that just means it needs to be easier to create and update pungi compose configs?
Pungi is inherently a complicated thing, because it's a tool for doing a whole bunch of different tasks for two rather different distributions. There's a limit to how much we can unilaterally mess around with this stuff (because we're sharing a tool with RHEL, here) and a limit to how much we can simplify it, I think.
Adam Williamson wrote:
Note that for dumb reasons that I hate we actually strip the metadata from the final compose when it's actually approved and shipped out to mirrors, but when a compose completes, the metadata related to that compose is uploaded to PDC: https://pdc.fedoraproject.org/ . Once that's done it's not expected, and it may in fact be impossible, for that metadata to be edited, and it's never removed. So even though the metadata for Alpha, Beta and Final composes can't be found on the mirrors with them, it *can* be seen in PDC.
All this does not explain why you cannot just drop an ISO file into the directory to be sent to the mirroring. Just ignore all the compose tools and cp or scp or rsync the file in, how hard can that be?
Kevin Kofler
On Mon, 2017-07-10 at 00:05 +0200, Kevin Kofler wrote:
Adam Williamson wrote:
Note that for dumb reasons that I hate we actually strip the metadata from the final compose when it's actually approved and shipped out to mirrors, but when a compose completes, the metadata related to that compose is uploaded to PDC: https://pdc.fedoraproject.org/ . Once that's done it's not expected, and it may in fact be impossible, for that metadata to be edited, and it's never removed. So even though the metadata for Alpha, Beta and Final composes can't be found on the mirrors with them, it *can* be seen in PDC.
All this does not explain why you cannot just drop an ISO file into the directory to be sent to the mirroring. Just ignore all the compose tools and cp or scp or rsync the file in, how hard can that be?
But we can't "just ignore the compose tools", because we want the metadata to actually be correct and consistent. If we dump an ISO into a compose link this, it won't be.
Adam Williamson wrote:
But we can't "just ignore the compose tools", because we want the metadata to actually be correct and consistent. If we dump an ISO into a compose link this, it won't be.
Real users don't care about metadata that you upload to some obscure server like pdc.fp.o that most users don't even know exists (this thread was the first time that *I* heard of it, and I am in no way a new user ;-) ). Real users care about what ends up on the mirrors. Dropping in the ISO into the mirrored directory perfectly addresses the expectations of actual users.
Kevin Kofler
On Tue, Jul 11, 2017 at 01:10:52PM +0200, Kevin Kofler wrote:
But we can't "just ignore the compose tools", because we want the metadata to actually be correct and consistent. If we dump an ISO into a compose link this, it won't be.
Real users don't care about metadata that you upload to some obscure server like pdc.fp.o that most users don't even know exists (this thread was the first time that *I* heard of it, and I am in no way a new user ;-) ). Real users care about what ends up on the mirrors. Dropping in the ISO into the mirrored directory perfectly addresses the expectations of actual users.
I get that having correct metadata is useful for a lot of programmatic things (like maybe finally making https://pagure.io/releng/issue/5805 a reality), but I agree about the results. Also, we shouldn't have this dictated to us by the design of the compose tools. We should look at what we want the user-facing results to be and work back from that.
On Tue, 2017-07-11 at 11:24 -0400, Matthew Miller wrote:
On Tue, Jul 11, 2017 at 01:10:52PM +0200, Kevin Kofler wrote:
But we can't "just ignore the compose tools", because we want the metadata to actually be correct and consistent. If we dump an ISO into a compose link this, it won't be.
Real users don't care about metadata that you upload to some obscure server like pdc.fp.o that most users don't even know exists (this thread was the first time that *I* heard of it, and I am in no way a new user ;-) ). Real users care about what ends up on the mirrors. Dropping in the ISO into the mirrored directory perfectly addresses the expectations of actual users.
I get that having correct metadata is useful for a lot of programmatic things (like maybe finally making https://pagure.io/releng/issue/5805 a reality), but I agree about the results. Also, we shouldn't have this dictated to us by the design of the compose tools. We should look at what we want the user-facing results to be and work back from that.
We do: we want the user-facing results to be a consistent set of deliverables, reliably built using the same process and thus to the same specifications every time. Throwing together images by hand and stuffing them into a directory together does not reliably achieve that, which is why we made things like Pungi.
The best resolution to a situation like this is simply to fix the problem that's causing images to fail to build in the first place.
On Tue, 2017-07-11 at 11:12 -0700, Adam Williamson wrote:
The best resolution to a situation like this is simply to fix the problem that's causing images to fail to build in the first place.
I should note for the record that both the images missing from RC-1.4 were present in RC-1.5, which is what we actually released.
On Wed, Jul 05, 2017 at 03:04:36PM -0600, Chris Murphy wrote:
Nope. This came up last cycle or two where compose had a weird bug preventing umount of the installation root fs image file, resulting in a corrupt file system and ensuing boot failure of the ISO. And actually there were some non-release blocking images that we simply did not ship because of those failures, and they were not respun, and releng said we couldn't even use a prior successfully built ISO.
I believe we ended up linking to a nightly on the spins or labs website. These don't get a lot of downloads so it's not super-important that they be on the mirrors.
On Wed, 2017-07-05 at 14:50 -0400, Josh Boyer wrote:
On Tue, Jul 4, 2017 at 6:41 PM, Kevin Fenzi kevin@scrye.com wrote:
On 07/04/2017 03:46 PM, Adam Williamson wrote:
On Tue, 2017-07-04 at 09:32 +0200, Christian Dersch wrote:
Looks like one compose failed *again* for some mirror issue… This time it is Robotic [1] , but i've seen these random failures several times, also for Astronomy for example. So a new compose should be done here to ensure the Spin gets in properly,
We don't re-spin for non-blocking images as a matter of course. The compose process takes far too long for that, unfortunately.
I think nirik said he thought he'd fixed this, but obviously he hasn't...
Yeah, I keep doing things that I think will quash this and then it shows up again. :( It has been very hard to track down because it's going through a number of layers... apache to haproxy to varnish to apache. :(
Note this only seems to have affected the i386 image. The x86_64 image is included in the compose.
So... this prompts a question in my head. I understand we don't block release for non-blocking spins. That is obvious.
What is not obvious to me is if we would go back and do a compose later for those spins that are impacted by this. It is one thing to skip a spin if the maintainer has not kept up with the release and is otherwise negligent. It is quite another to tell them "too bad..." if the spin isn't composed for something completely out of their control such as an infrastructure issue. That, to me, would be discounting the work they've done throughout the release to maintain and test their spin.
Do we have that ability to do a post-GA compose for those spins in such cases and do we plan to do so?
We can do this on a kind of 'unofficial' basis, and I believe we have done so before (in fact I think we just wound up taking the image in question from a nightly compose done around the time of release).
-
Adam Williamson -
Chris Murphy -
Christian Dersch -
Josh Boyer -
Kevin Fenzi -
Kevin Kofler -
Matthew Miller -
Peter Robinson