Fedora Cloud rawhide images not updating?

[Ondrej Mosnacek]

Hi,

I noticed that the image available at [1] has been stuck at the
20201004 version for quite some time now. AFACIT, it used to be
updated approx. every day before. Does anyone know what happened? I
understand that the rawhide content might not be 100% reliable, but 17
days without updating the image seems a bit long...

Thanks,

[1]
https://download.fedoraproject.org/pub/fedora/linux/development/rawhide/C...

-- 
Ondrej Mosnacek
Software Engineer, Platform Security - SELinux kernel
Red Hat, Inc.

[Kevin Fenzi]

On Wed, Oct 21, 2020 at 12:29:14PM +0200, Ondrej Mosnacek wrote:
>  Hi,
>  
>  I noticed that the image available at [1] has been stuck at the
>  20201004 version for quite some time now. AFACIT, it used to be
>  updated approx. every day before. Does anyone know what happened? I
>  understand that the rawhide content might not be 100% reliable, but 17
>  days without updating the image seems a bit long...
>  
>  Thanks,
>  
>  [1]
> https://download.fedoraproject.org/pub/fedora/linux/development/rawhide/C...
>   
Indeed they aren't. ;( 

They are (with the exception of ppc64le) being composed, they just
aren't being synced over. 

I note that we updated productmd on that day and when we sync we call
compose-partial-copy from compose-utils (which uses productmd), so the
only thought I have now is that there is a bug there. I downgraded back
to the previous version and we can see if it updates correctly tonight?

kevin

[Adam Williamson]

On Sun, 2020-10-25 at 11:23 -0700, Kevin Fenzi wrote:
>  On Wed, Oct 21, 2020 at 12:29:14PM +0200, Ondrej Mosnacek wrote:
>  > Hi,
>  > 
>  > I noticed that the image available at [1] has been stuck at the
>  > 20201004 version for quite some time now. AFACIT, it used to be
>  > updated approx. every day before. Does anyone know what happened? I
>  > understand that the rawhide content might not be 100% reliable, but 17
>  > days without updating the image seems a bit long...
>  > 
>  > Thanks,
>  > 
>  > [1]
> https://download.fedoraproject.org/pub/fedora/linux/development/rawhide/C...
>  > 
>  Indeed they aren't. ;( 
>  
>  They are (with the exception of ppc64le) being composed, they just
>  aren't being synced over. 
>  
>  I note that we updated productmd on that day and when we sync we call
>  compose-partial-copy from compose-utils (which uses productmd), so the
>  only thought I have now is that there is a bug there. I downgraded back
>  to the previous version and we can see if it updates correctly tonight? 
I believe mboddu had already looked at this a bit and identified
productmd as a suspect, but hadn't figured out exactly what the issue
was yet. He said it seems like any variant with an 'images/' folder
doesn't get synced, I think.
-- 
Adam Williamson
Fedora QA Community Monkey
IRC: adamw | Twitter: AdamW_Fedora | XMPP: adamw AT happyassassin . net
http://www.happyassassin.net

[Kevin Fenzi]

On Sun, Oct 25, 2020 at 12:30:13PM -0700, Adam Williamson wrote:
>  On Sun, 2020-10-25 at 11:23 -0700, Kevin Fenzi wrote:
>  > On Wed, Oct 21, 2020 at 12:29:14PM +0200, Ondrej Mosnacek wrote:
>  > > Hi,
>  > > 
>  > > I noticed that the image available at [1] has been stuck at the
>  > > 20201004 version for quite some time now. AFACIT, it used to be
>  > > updated approx. every day before. Does anyone know what happened? I
>  > > understand that the rawhide content might not be 100% reliable, but 17
>  > > days without updating the image seems a bit long...
>  > > 
>  > > Thanks,
>  > > 
>  > > [1]
> https://download.fedoraproject.org/pub/fedora/linux/development/rawhide/C...
>  > > 
>  > Indeed they aren't. ;( 
>  > 
>  > They are (with the exception of ppc64le) being composed, they just
>  > aren't being synced over. 
>  > 
>  > I note that we updated productmd on that day and when we sync we call
>  > compose-partial-copy from compose-utils (which uses productmd), so the
>  > only thought I have now is that there is a bug there. I downgraded back
>  > to the previous version and we can see if it updates correctly tonight?
>  
>  I believe mboddu had already looked at this a bit and identified
>  productmd as a suspect, but hadn't figured out exactly what the issue
>  was yet. He said it seems like any variant with an 'images/' folder
>  doesn't get synced, I think. 
Yeah, but I downgraded python3-productmd, (to the version it was when it
was working) and it still didn't work... so it's somthing less obvious.
;( 

kevin

[Lubomír Sedlář]

Kevin Fenzi píše v Ne 25. 10. 2020 v 13:56 -0700:
>  On Sun, Oct 25, 2020 at 12:30:13PM -0700, Adam Williamson wrote:
>  > On Sun, 2020-10-25 at 11:23 -0700, Kevin Fenzi wrote:
>  > > On Wed, Oct 21, 2020 at 12:29:14PM +0200, Ondrej Mosnacek wrote:
>  > > > Hi,
>  > > > 
>  > > > I noticed that the image available at [1] has been stuck at the
>  > > > 20201004 version for quite some time now. AFACIT, it used to be
>  > > > updated approx. every day before. Does anyone know what
>  > > > happened? I
>  > > > understand that the rawhide content might not be 100% reliable,
>  > > > but 17
>  > > > days without updating the image seems a bit long...
>  > > > 
>  > > > Thanks,
>  > > > 
>  > > > [1] 
>  > > >
> https://download.fedoraproject.org/pub/fedora/linux/development/rawhide/C...
>  > > > 
>  > > Indeed they aren't. ;( 
>  > > 
>  > > They are (with the exception of ppc64le) being composed, they
>  > > just
>  > > aren't being synced over. 
>  > > 
>  > > I note that we updated productmd on that day and when we sync we
>  > > call
>  > > compose-partial-copy from compose-utils (which uses productmd),
>  > > so the
>  > > only thought I have now is that there is a bug there. I
>  > > downgraded back
>  > > to the previous version and we can see if it updates correctly
>  > > tonight?
>  > 
>  > I believe mboddu had already looked at this a bit and identified
>  > productmd as a suspect, but hadn't figured out exactly what the
>  > issue
>  > was yet. He said it seems like any variant with an 'images/' folder
>  > doesn't get synced, I think.
>  
>  Yeah, but I downgraded python3-productmd, (to the version it was when
>  it was working) and it still didn't work... so it's somthing less
>  obvious.
>  ;(  
It's very much not obvious. I believe the root cause is that there are
two raw.xz images for Server on armhfp, which are indistinguishable in
the metadata. This causes loading of the metadata to fail. The script
that partial copy script needs the information from the metadata to
know what it should copy. Since loading the metadata fails, it ignores
the file and images are not copied.

My theory is that the image is updated only when the one of the armhfp
images fail and is missing from the compose.

I believe that in order to fix this, the ambiguity in metadata has to
be resolved in some way, like this:
https://pagure.io/pungi-fedora/pull-request/923

Lubomír


>  kevin
>

[Kevin Fenzi]

On Mon, Oct 26, 2020 at 12:38:25PM +0100, Lubomír Sedlář wrote:
>  
>  It's very much not obvious. I believe the root cause is that there are
>  two raw.xz images for Server on armhfp, which are indistinguishable in
>  the metadata. This causes loading of the metadata to fail. The script
>  that partial copy script needs the information from the metadata to
>  know what it should copy. Since loading the metadata fails, it ignores
>  the file and images are not copied.
>  
>  My theory is that the image is updated only when the one of the armhfp
>  images fail and is missing from the compose.
>  
>  I believe that in order to fix this, the ambiguity in metadata has to
>  be resolved in some way, like this:
>  https://pagure.io/pungi-fedora/pull-request/923 
That did fix it. Thanks!

kevin

[Ondrej Mosnacek]

On Wed, Oct 28, 2020 at 10:25 PM Kevin Fenzi <kevin(a)scrye.com&gt; wrote:
>  On Mon, Oct 26, 2020 at 12:38:25PM +0100, Lubomír Sedlář wrote:
>  >
>  > It's very much not obvious. I believe the root cause is that there are
>  > two raw.xz images for Server on armhfp, which are indistinguishable in
>  > the metadata. This causes loading of the metadata to fail. The script
>  > that partial copy script needs the information from the metadata to
>  > know what it should copy. Since loading the metadata fails, it ignores
>  > the file and images are not copied.
>  >
>  > My theory is that the image is updated only when the one of the armhfp
>  > images fail and is missing from the compose.
>  >
>  > I believe that in order to fix this, the ambiguity in metadata has to
>  > be resolved in some way, like this:
>  > https://pagure.io/pungi-fedora/pull-request/923
>
>  That did fix it. Thanks! 
Indeed, the images are now being updated. Thanks to everyone involved
for fixing this!

-- 
Ondrej Mosnacek
Software Engineer, Platform Security - SELinux kernel
Red Hat, Inc.

[Adam Williamson]

On Mon, 2020-10-26 at 12:38 +0100, Lubomír Sedlář wrote:
>  
>  It's very much not obvious. I believe the root cause is that there are
>  two raw.xz images for Server on armhfp, which are indistinguishable in
>  the metadata. This causes loading of the metadata to fail. The script
>  that partial copy script needs the information from the metadata to
>  know what it should copy. Since loading the metadata fails, it ignores
>  the file and images are not copied.
>  
>  My theory is that the image is updated only when the one of the armhfp
>  images fail and is missing from the compose.
>  
>  I believe that in order to fix this, the ambiguity in metadata has to
>  be resolved in some way, like this:
>  https://pagure.io/pungi-fedora/pull-request/923 
This does make me think of a few other questions...

1) If we consider this an Unacceptable State of Affairs, could we not
set things up such that pungi refuses to run such a compose at all?

2) Does productmd *really* have to choke on this case? I suppose it
depends exactly how it wants to parse the metadata, but fedfind parses
the same metadata and it doesn't choke on this; if you asked it to find
the 'Minimal armhfp raw-xz' image and expected to get only one result
you might have fun, but as long as you aren't actually doing something
that happens to exactly hit the 'duplicate' images, you can otherwise
work with the metadata OK.
-- 
Adam Williamson
Fedora QA Community Monkey
IRC: adamw | Twitter: AdamW_Fedora | XMPP: adamw AT happyassassin . net
http://www.happyassassin.net