Greetings.
We have enabled the koji 'save-failed-tree' plugin in koji.fedoraproject.org. This plugin allows you to tell koji to bundle up a failed official builds chroot (either partly or fully) and download it to investigate it locally.
This plugin should only be used for the case where you cannot determine the cause of a build failure by any other means and need to examine specific files in the chroot to do so.
A few things to note:
* This will only work on failed official builds that have failed recently enough to still have their chroot on the builder where they failed (default: 1 day) Not scratch builds. Not canceled builds. The chroot downloads are REALLY LARGE. Please use this sparingly.
* This will only work on buildArch tasks, not images, etc
* Saved tree .tar.gz's are deleted from koji after 14 days.
* You need to have python3-koji-cli-plugins subpackage installed to use the command.
* You run the command as: koji save-failed-tree <failed-taskid>
I hope that this will be of use to help maintainers track down hard to isolate bugs when all other means fail.
kevin
_______________________________________________ devel-announce mailing list -- devel-announce@lists.fedoraproject.org To unsubscribe send an email to devel-announce-leave@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel-announce@lists.fedorapro... Do not reply to spam on the list, report it: https://pagure.io/fedora-infrastructure
On Thu, Feb 11, 2021, at 2:27 PM, Kevin Fenzi wrote:
Greetings.
We have enabled the koji 'save-failed-tree' plugin in koji.fedoraproject.org. This plugin allows you to tell koji to bundle up a failed official builds chroot (either partly or fully) and download it to investigate it locally.
If Koji natively used Kubernetes/podman, then this would just be a container image. And we know how to mirror, sign, version and promote those. For example, it'd also be easy and effortless to spin it up as a shell on a remote Kubernetes cluster.
(This isn't some theoretical thing for me, I do this *all the time* - our upstream CoreOS CI works this way)
Hi Kevin,
is there specific reason, why scratch builds cannot do it? I met hard to reproduce issues in builds several times. I would prefer to analyse scratch builds instead of production builds.
Could for example just single per-user scratch build archive be stored for some short time? Might be just a day or so. It might be useful to add some tweaks and retry. Keeping previous scratch build is not needed, but ability to extract information from non-production builds seems vital to me.
Keeping record of failed production builds, which will stay there forever, does not look good to me.
Cheers, Petr
On 2/11/21 8:27 PM, Kevin Fenzi wrote:
Greetings.
We have enabled the koji 'save-failed-tree' plugin in koji.fedoraproject.org. This plugin allows you to tell koji to bundle up a failed official builds chroot (either partly or fully) and download it to investigate it locally.
This plugin should only be used for the case where you cannot determine the cause of a build failure by any other means and need to examine specific files in the chroot to do so.
A few things to note:
- This will only work on failed official builds that have failed
recently enough to still have their chroot on the builder where they failed (default: 1 day) Not scratch builds. Not canceled builds. The chroot downloads are REALLY LARGE. Please use this sparingly.
This will only work on buildArch tasks, not images, etc
Saved tree .tar.gz's are deleted from koji after 14 days.
You need to have python3-koji-cli-plugins subpackage installed to use
the command.
- You run the command as: koji save-failed-tree <failed-taskid>
I hope that this will be of use to help maintainers track down hard to isolate bugs when all other means fail.
kevin
devel-announce mailing list -- devel-announce@lists.fedoraproject.org To unsubscribe send an email to devel-announce-leave@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel-announce@lists.fedorapro... Do not reply to spam on the list, report it: https://pagure.io/fedora-infrastructure
devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-leave@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org Do not reply to spam on the list, report it: https://pagure.io/fedora-infrastructure
On Fri, Feb 12, 2021 at 06:17:01PM +0100, Petr Menšík wrote:
Hi Kevin,
is there specific reason, why scratch builds cannot do it? I met hard to reproduce issues in builds several times. I would prefer to analyse scratch builds instead of production builds.
Could for example just single per-user scratch build archive be stored for some short time? Might be just a day or so. It might be useful to add some tweaks and retry. Keeping previous scratch build is not needed, but ability to extract information from non-production builds seems vital to me.
Keeping record of failed production builds, which will stay there forever, does not look good to me.
I think this is a limitation of the way koji handles scratch builds. koji keeps track of exact buildroots used for offical builds, but scratch builds are just against whatever buildroot is current at the time and I don't think koji keeps failed scratch builds around on the builder.
But I agree, having some way to do this for scratch builds might be nice. Could you file a koji RFE? or would you like me to do so?
kevin --
Cheers, Petr
On 2/11/21 8:27 PM, Kevin Fenzi wrote:
Greetings.
We have enabled the koji 'save-failed-tree' plugin in koji.fedoraproject.org. This plugin allows you to tell koji to bundle up a failed official builds chroot (either partly or fully) and download it to investigate it locally.
This plugin should only be used for the case where you cannot determine the cause of a build failure by any other means and need to examine specific files in the chroot to do so.
A few things to note:
- This will only work on failed official builds that have failed
recently enough to still have their chroot on the builder where they failed (default: 1 day) Not scratch builds. Not canceled builds. The chroot downloads are REALLY LARGE. Please use this sparingly.
This will only work on buildArch tasks, not images, etc
Saved tree .tar.gz's are deleted from koji after 14 days.
You need to have python3-koji-cli-plugins subpackage installed to use
the command.
- You run the command as: koji save-failed-tree <failed-taskid>
I hope that this will be of use to help maintainers track down hard to isolate bugs when all other means fail.
kevin
devel-announce mailing list -- devel-announce@lists.fedoraproject.org To unsubscribe send an email to devel-announce-leave@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel-announce@lists.fedorapro... Do not reply to spam on the list, report it: https://pagure.io/fedora-infrastructure
devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-leave@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org Do not reply to spam on the list, report it: https://pagure.io/fedora-infrastructure
-- Petr Menšík Software Engineer Red Hat, http://www.redhat.com/ email: pemensik@redhat.com PGP: DFCF908DB7C87E8E529925BC4931CA5B6C9FC5CB
devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-leave@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org Do not reply to spam on the list, report it: https://pagure.io/fedora-infrastructure