koji save-failed-tree enabled

[Kevin Fenzi]

Greetings. 

We have enabled the koji 'save-failed-tree' plugin in
koji.fedoraproject.org. This plugin allows you to tell koji to bundle up
a failed official builds chroot (either partly or fully) and download it
to investigate it locally. 

This plugin should only be used for the case where you cannot determine
the cause of a build failure by any other means and need to examine
specific files in the chroot to do so. 

A few things to note: 

* This will only work on failed official builds that have failed
recently enough to still have their chroot on the builder where they
failed (default: 1 day)  Not scratch builds. Not canceled builds.
The chroot downloads are REALLY LARGE. Please use this sparingly. 

* This will only work on buildArch tasks, not images, etc

* Saved tree .tar.gz's are deleted from koji after 14 days.

* You need to have python3-koji-cli-plugins subpackage installed to use
the command. 

* You run the command as: koji save-failed-tree <failed-taskid>

I hope that this will be of use to help maintainers track down hard to
isolate bugs when all other means fail. 

kevin

_______________________________________________
devel-announce mailing list -- devel-announce(a)lists.fedoraproject.org
To unsubscribe send an email to devel-announce-leave(a)lists.fedoraproject.org
Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives:
https://lists.fedoraproject.org/archives/list/[e-mail]...
Do not reply to spam on the list, report it: https://pagure.io/fedora-infrastructure

[Colin Walters]

On Thu, Feb 11, 2021, at 2:27 PM, Kevin Fenzi wrote:
>  Greetings. 
>  
>  We have enabled the koji 'save-failed-tree' plugin in
>  koji.fedoraproject.org. This plugin allows you to tell koji to bundle up
>  a failed official builds chroot (either partly or fully) and download it
>  to investigate it locally.  
If Koji natively used Kubernetes/podman, then this would just be a container image.  And
we know how to mirror, sign, version and promote those.  For example, it'd also be
easy and effortless to spin it up as a shell on a remote Kubernetes cluster.

(This isn't some theoretical thing for me, I do this *all the time* - our upstream
CoreOS CI works this way)

[Petr Menšík]

Hi Kevin,

is there specific reason, why scratch builds cannot do it? I met hard to
reproduce issues in builds several times. I would prefer to analyse
scratch builds instead of production builds.

Could for example just single per-user scratch build archive be stored
for some short time? Might be just a day or so. It might be useful to
add some tweaks and retry. Keeping previous scratch build is not needed,
but ability to extract information from non-production builds seems
vital to me.

Keeping record of failed production builds, which will stay there
forever, does not look good to me.

Cheers,
Petr

On 2/11/21 8:27 PM, Kevin Fenzi wrote:
>  Greetings. 
>  
>  We have enabled the koji 'save-failed-tree' plugin in
>  koji.fedoraproject.org. This plugin allows you to tell koji to bundle up
>  a failed official builds chroot (either partly or fully) and download it
>  to investigate it locally. 
>  
>  This plugin should only be used for the case where you cannot determine
>  the cause of a build failure by any other means and need to examine
>  specific files in the chroot to do so. 
>  
>  A few things to note: 
>  
>  * This will only work on failed official builds that have failed
>  recently enough to still have their chroot on the builder where they
>  failed (default: 1 day)  Not scratch builds. Not canceled builds.
>  The chroot downloads are REALLY LARGE. Please use this sparingly. 
>  
>  * This will only work on buildArch tasks, not images, etc
>  
>  * Saved tree .tar.gz's are deleted from koji after 14 days.
>  
>  * You need to have python3-koji-cli-plugins subpackage installed to use
>  the command. 
>  
>  * You run the command as: koji save-failed-tree <failed-taskid>
>  
>  I hope that this will be of use to help maintainers track down hard to
>  isolate bugs when all other means fail. 
>  
>  kevin
>  
>  
>  _______________________________________________
>  devel-announce mailing list -- devel-announce(a)lists.fedoraproject.org
>  To unsubscribe send an email to devel-announce-leave(a)lists.fedoraproject.org
>  Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/
>  List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
>  List Archives:
> https://lists.fedoraproject.org/archives/list/[e-mail]...
>  Do not reply to spam on the list, report it: https://pagure.io/fedora-infrastructure
>  
>  
>  _______________________________________________
>  devel mailing list -- devel(a)lists.fedoraproject.org
>  To unsubscribe send an email to devel-leave(a)lists.fedoraproject.org
>  Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/
>  List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
>  List Archives:
> https://lists.fedoraproject.org/archives/list/[e-mail]
>  Do not reply to spam on the list, report it: https://pagure.io/fedora-infrastructure
>   
-- 
Petr Menšík
Software Engineer
Red Hat, http://www.redhat.com/
email: pemensik(a)redhat.com
PGP: DFCF908DB7C87E8E529925BC4931CA5B6C9FC5CB

[Kevin Fenzi]

On Fri, Feb 12, 2021 at 06:17:01PM +0100, Petr Menšík wrote:
>  Hi Kevin,
>  
>  is there specific reason, why scratch builds cannot do it? I met hard to
>  reproduce issues in builds several times. I would prefer to analyse
>  scratch builds instead of production builds.
>  
>  Could for example just single per-user scratch build archive be stored
>  for some short time? Might be just a day or so. It might be useful to
>  add some tweaks and retry. Keeping previous scratch build is not needed,
>  but ability to extract information from non-production builds seems
>  vital to me.
>  
>  Keeping record of failed production builds, which will stay there
>  forever, does not look good to me. 
I think this is a limitation of the way koji handles scratch builds. 
koji keeps track of exact buildroots used for offical builds, but
scratch builds are just against whatever buildroot is current at the
time and I don't think koji keeps failed scratch builds around on the
builder.

But I agree, having some way to do this for scratch builds might be
nice. Could you file a koji RFE? or would you like me to do so?

kevin
--
>  
>  Cheers,
>  Petr
>  
>  On 2/11/21 8:27 PM, Kevin Fenzi wrote:
>  > Greetings. 
>  > 
>  > We have enabled the koji 'save-failed-tree' plugin in
>  > koji.fedoraproject.org. This plugin allows you to tell koji to bundle up
>  > a failed official builds chroot (either partly or fully) and download it
>  > to investigate it locally. 
>  > 
>  > This plugin should only be used for the case where you cannot determine
>  > the cause of a build failure by any other means and need to examine
>  > specific files in the chroot to do so. 
>  > 
>  > A few things to note: 
>  > 
>  > * This will only work on failed official builds that have failed
>  > recently enough to still have their chroot on the builder where they
>  > failed (default: 1 day)  Not scratch builds. Not canceled builds.
>  > The chroot downloads are REALLY LARGE. Please use this sparingly. 
>  > 
>  > * This will only work on buildArch tasks, not images, etc
>  > 
>  > * Saved tree .tar.gz's are deleted from koji after 14 days.
>  > 
>  > * You need to have python3-koji-cli-plugins subpackage installed to use
>  > the command. 
>  > 
>  > * You run the command as: koji save-failed-tree <failed-taskid>
>  > 
>  > I hope that this will be of use to help maintainers track down hard to
>  > isolate bugs when all other means fail. 
>  > 
>  > kevin
>  > 
>  > 
>  > _______________________________________________
>  > devel-announce mailing list -- devel-announce(a)lists.fedoraproject.org
>  > To unsubscribe send an email to devel-announce-leave(a)lists.fedoraproject.org
>  > Fedora Code of Conduct:
> https://docs.fedoraproject.org/en-US/project/code-of-conduct/
>  > List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
>  > List Archives:
> https://lists.fedoraproject.org/archives/list/[e-mail]...
>  > Do not reply to spam on the list, report it:
> https://pagure.io/fedora-infrastructure
>  > 
>  > 
>  > _______________________________________________
>  > devel mailing list -- devel(a)lists.fedoraproject.org
>  > To unsubscribe send an email to devel-leave(a)lists.fedoraproject.org
>  > Fedora Code of Conduct:
> https://docs.fedoraproject.org/en-US/project/code-of-conduct/
>  > List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
>  > List Archives:
> https://lists.fedoraproject.org/archives/list/[e-mail]
>  > Do not reply to spam on the list, report it:
> https://pagure.io/fedora-infrastructure
>  > 
>  
>  -- 
>  Petr Menšík
>  Software Engineer
>  Red Hat, http://www.redhat.com/
>  email: pemensik(a)redhat.com
>  PGP: DFCF908DB7C87E8E529925BC4931CA5B6C9FC5CB
>   



>  _______________________________________________
>  devel mailing list -- devel(a)lists.fedoraproject.org
>  To unsubscribe send an email to devel-leave(a)lists.fedoraproject.org
>  Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/
>  List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
>  List Archives:
> https://lists.fedoraproject.org/archives/list/[e-mail]
>  Do not reply to spam on the list, report it: https://pagure.io/fedora-infrastructure