openQA failures caused by arch mismatch

[Adam Williamson]

Hi, folks! This is just a heads-up for anyone who's been watching the
openQA test results for an update, or something, and been confused.

A few days ago we added a new ppc64 openQA worker host box; ultimately
we're aiming to enable some of the openQA tests to run on ppc64 as well
as x86_64 and i686. Unfortunately, it seems that openQA's protection
against running jobs on workers of the wrong arch isn't working
properly in our configuration, and since then, openQA has tried to run
quite a lot of x86_64 jobs on ppc64 workers, which of course doesn't
work at all (it blows up immediately because qemu-kvm isn't available,
in point of fact, but it could go wrong for *all kinds* of other
reasons too!)

I'm just now putting a change in place which should prevent this
happening again, and will try to re-run the tests for all updates which
were affected by this. I do apologize for the bogus failures.
-- 
Adam Williamson
Fedora QA Community Monkey
IRC: adamw | Twitter: AdamW_Fedora | XMPP: adamw AT happyassassin . net
http://www.happyassassin.net

[Adam Williamson]

On Wed, 2017-07-26 at 16:17 -0700, Adam Williamson wrote:
>  Hi, folks! This is just a heads-up for anyone who's been watching
> the
>  openQA test results for an update, or something, and been confused.
>  
>  A few days ago we added a new ppc64 openQA worker host box; ultimately
>  we're aiming to enable some of the openQA tests to run on ppc64 as well
>  as x86_64 and i686. Unfortunately, it seems that openQA's protection
>  against running jobs on workers of the wrong arch isn't working
>  properly in our configuration, and since then, openQA has tried to run
>  quite a lot of x86_64 jobs on ppc64 workers, which of course doesn't
>  work at all (it blows up immediately because qemu-kvm isn't available,
>  in point of fact, but it could go wrong for *all kinds* of other
>  reasons too!)
>  
>  I'm just now putting a change in place which should prevent this
>  happening again, and will try to re-run the tests for all updates which
>  were affected by this. I do apologize for the bogus failures. 
Oh, while I'm at it: you'll also currently get a perfect set of openQA
failures for *any* Fedora 24 update. This is because the base disk
images for Fedora 24 aren't there any more; I wrote the script that
looks after the disk images to only keep images for the last two stable
releases around, and forgot about the one-month transition period at
the start of a cycle when there are still updates for the last *three*
stable releases. I'll try and figure out something to do about this too
(it should be possible). Again, sorry for the bogus failures.
-- 
Adam Williamson
Fedora QA Community Monkey
IRC: adamw | Twitter: AdamW_Fedora | XMPP: adamw AT happyassassin . net
http://www.happyassassin.net

[Adam Williamson]

On Wed, 2017-07-26 at 16:27 -0700, Adam Williamson wrote:
>  On Wed, 2017-07-26 at 16:17 -0700, Adam Williamson wrote:
>  > Hi, folks! This is just a heads-up for anyone who's been watching the
>  > openQA test results for an update, or something, and been confused.
>  > 
>  > A few days ago we added a new ppc64 openQA worker host box; ultimately
>  > we're aiming to enable some of the openQA tests to run on ppc64 as well
>  > as x86_64 and i686. Unfortunately, it seems that openQA's protection
>  > against running jobs on workers of the wrong arch isn't working
>  > properly in our configuration, and since then, openQA has tried to run
>  > quite a lot of x86_64 jobs on ppc64 workers, which of course doesn't
>  > work at all (it blows up immediately because qemu-kvm isn't available,
>  > in point of fact, but it could go wrong for *all kinds* of other
>  > reasons too!)
>  > 
>  > I'm just now putting a change in place which should prevent this
>  > happening again, and will try to re-run the tests for all updates which
>  > were affected by this. I do apologize for the bogus failures.
>  
>  Oh, while I'm at it: you'll also currently get a perfect set of openQA
>  failures for *any* Fedora 24 update. This is because the base disk
>  images for Fedora 24 aren't there any more; I wrote the script that
>  looks after the disk images to only keep images for the last two stable
>  releases around, and forgot about the one-month transition period at
>  the start of a cycle when there are still updates for the last *three*
>  stable releases. I'll try and figure out something to do about this too
>  (it should be possible). Again, sorry for the bogus failures. 
Re-runs of all the bogus failures for the ppc64 issue are complete.
I've now tweaked the disk image script so that the F24 base disk images
are available again, and have re-scheduled the tests for all F24
updates, so those are running now. Again, sorry for any inconvenience.
-- 
Adam Williamson
Fedora QA Community Monkey
IRC: adamw | Twitter: AdamW_Fedora | XMPP: adamw AT happyassassin . net
http://www.happyassassin.net

[Josh Boyer]

On Wed, Jul 26, 2017 at 7:17 PM, Adam Williamson
<adamwill(a)fedoraproject.org&gt; wrote:
>  Hi, folks! This is just a heads-up for anyone who's been watching
> the
>  openQA test results for an update, or something, and been confused.
>
>  A few days ago we added a new ppc64 openQA worker host box; ultimately 
Out of curiosity, do you mean ppc64, ppc64le, or both?

If ppc64le isn't the main target for openQA, i'd strongly recommend
switching to that.

josh

[Adam Williamson]

On Thu, 2017-07-27 at 06:09 -0400, Josh Boyer wrote:
>  On Wed, Jul 26, 2017 at 7:17 PM, Adam Williamson
>  <adamwill(a)fedoraproject.org&gt; wrote:
>  > Hi, folks! This is just a heads-up for anyone who's been watching the
>  > openQA test results for an update, or something, and been confused.
>  > 
>  > A few days ago we added a new ppc64 openQA worker host box; ultimately
>  
>  Out of curiosity, do you mean ppc64, ppc64le, or both?
>  
>  If ppc64le isn't the main target for openQA, i'd strongly recommend
>  switching to that. 
I can't remember which arch we installed the host with, but it doesn't
actually matter much because all openQA tests run in VMs, and
apparently you can get native speed virt for both ppc64 and ppc64le no
matter which the host is running. At least, so I was told.
-- 
Adam Williamson
Fedora QA Community Monkey
IRC: adamw | Twitter: AdamW_Fedora | XMPP: adamw AT happyassassin . net
http://www.happyassassin.net

[Josh Boyer]

On Thu, Jul 27, 2017 at 12:02 PM, Adam Williamson
<adamwill(a)fedoraproject.org&gt; wrote:
>  On Thu, 2017-07-27 at 06:09 -0400, Josh Boyer wrote:
> > On Wed, Jul 26, 2017 at 7:17 PM, Adam Williamson
> > <adamwill(a)fedoraproject.org&gt; wrote:
> > > Hi, folks! This is just a heads-up for anyone who's been watching the
> > > openQA test results for an update, or something, and been confused.
> > >
> > > A few days ago we added a new ppc64 openQA worker host box; ultimately
> >
> > Out of curiosity, do you mean ppc64, ppc64le, or both?
> >
> > If ppc64le isn't the main target for openQA, i'd strongly recommend
> > switching to that.
>
>  I can't remember which arch we installed the host with, but it doesn't
>  actually matter much because all openQA tests run in VMs, and
>  apparently you can get native speed virt for both ppc64 and ppc64le no
>  matter which the host is running. At least, so I was told. 
OK, let me elaborate as to why I recommend focusing on ppc64le.

Big-endian ppc64 has been essentially put in maintenance mode.  This
means that we'll make efforts to keep the base OS functional, but it
will not get new shiny things like golang, containers, Atomic, etc.
At some point we will likely deprecate it completely, but we aren't at
that point yet.

So if you have installed the host with ppc64, that would be somewhat
regrettable.  As for which is being *tested* in openQA, it'd be great
to do both so I fully support that.

josh

[Adam Williamson]

On Thu, 2017-07-27 at 13:32 -0400, Josh Boyer wrote:
>  On Thu, Jul 27, 2017 at 12:02 PM, Adam Williamson
>  <adamwill(a)fedoraproject.org&gt; wrote:
>  > On Thu, 2017-07-27 at 06:09 -0400, Josh Boyer wrote:
>  > > On Wed, Jul 26, 2017 at 7:17 PM, Adam Williamson
>  > > <adamwill(a)fedoraproject.org&gt; wrote:
>  > > > Hi, folks! This is just a heads-up for anyone who's been watching the
>  > > > openQA test results for an update, or something, and been confused.
>  > > > 
>  > > > A few days ago we added a new ppc64 openQA worker host box; ultimately
>  > > 
>  > > Out of curiosity, do you mean ppc64, ppc64le, or both?
>  > > 
>  > > If ppc64le isn't the main target for openQA, i'd strongly recommend
>  > > switching to that.
>  > 
>  > I can't remember which arch we installed the host with, but it doesn't
>  > actually matter much because all openQA tests run in VMs, and
>  > apparently you can get native speed virt for both ppc64 and ppc64le no
>  > matter which the host is running. At least, so I was told.
>  
>  OK, let me elaborate as to why I recommend focusing on ppc64le.
>  
>  Big-endian ppc64 has been essentially put in maintenance mode.  This
>  means that we'll make efforts to keep the base OS functional, but it
>  will not get new shiny things like golang, containers, Atomic, etc.
>  At some point we will likely deprecate it completely, but we aren't at
>  that point yet.
>  
>  So if you have installed the host with ppc64, that would be somewhat
>  regrettable. 
It's not really a big deal either way; all the openQA boxes can be
reinstalled at the drop of a hat in any case.
-- 
Adam Williamson
Fedora QA Community Monkey
IRC: adamw | Twitter: AdamW_Fedora | XMPP: adamw AT happyassassin . net
http://www.happyassassin.net