Proposal to modify release criteria for fwraid

[Stephen Gallagher]

At yesterday's F29 Go/No-Go meeting, we discussed the blocker status
of BZ #1628192 - Fedora 29 installation cannot see a firmware RAID
device. While the blocker criteria clearly states that this should be
a blocker for Beta, many of the people present at the meeting
disagreed, for a variety of reasons.

* Hardware supporting fwraid is considerably less pervasive than it
was when the criterion was written

* Testing this criterion can only be done with install media, which
limits our testing pool to the very dedicated members of Fedora QA.
Yes, anyone *can* download a nightly compose and try it, but in
practice this tends to be limited to the core testers. The majority of
testing that this feature will get will tend to happen as people try
out the Beta release.

To that end, I'd like to propose that we make the following change to
the criteria going forward:

"The blocking criterion for successful installation atop a firmware
RAID array is moved to the GA release criteria."

[Petr Šabata]

On Fri, Sep 14, 2018 at 10:22:12AM -0400, Stephen Gallagher wrote:
>  At yesterday's F29 Go/No-Go meeting, we discussed the blocker
> status
>  of BZ #1628192 - Fedora 29 installation cannot see a firmware RAID
>  device. While the blocker criteria clearly states that this should be
>  a blocker for Beta, many of the people present at the meeting
>  disagreed, for a variety of reasons.
>  
>  * Hardware supporting fwraid is considerably less pervasive than it
>  was when the criterion was written
>  
>  * Testing this criterion can only be done with install media, which
>  limits our testing pool to the very dedicated members of Fedora QA.
>  Yes, anyone *can* download a nightly compose and try it, but in
>  practice this tends to be limited to the core testers. The majority of
>  testing that this feature will get will tend to happen as people try
>  out the Beta release.
>  
>  To that end, I'd like to propose that we make the following change to
>  the criteria going forward:
>  
>  "The blocking criterion for successful installation atop a firmware
>  RAID array is moved to the GA release criteria." 
I hope this won't just mean we discover the problems later, in practice.
P

[Stephen Gallagher]

On Mon, Sep 17, 2018 at 5:16 AM Petr Šabata <contyk(a)redhat.com&gt; wrote:
>  On Fri, Sep 14, 2018 at 10:22:12AM -0400, Stephen Gallagher wrote:
>  > At yesterday's F29 Go/No-Go meeting, we discussed the blocker status
>  > of BZ #1628192 - Fedora 29 installation cannot see a firmware RAID
>  > device. While the blocker criteria clearly states that this should be
>  > a blocker for Beta, many of the people present at the meeting
>  > disagreed, for a variety of reasons.
>  >
>  > * Hardware supporting fwraid is considerably less pervasive than it
>  > was when the criterion was written
>  >
>  > * Testing this criterion can only be done with install media, which
>  > limits our testing pool to the very dedicated members of Fedora QA.
>  > Yes, anyone *can* download a nightly compose and try it, but in
>  > practice this tends to be limited to the core testers. The majority of
>  > testing that this feature will get will tend to happen as people try
>  > out the Beta release.
>  >
>  > To that end, I'd like to propose that we make the following change to
>  > the criteria going forward:
>  >
>  > "The blocking criterion for successful installation atop a firmware
>  > RAID array is moved to the GA release criteria."
>
>  I hope this won't just mean we discover the problems later, in practice. 

The policy of Fedora QA is to run all of the GA-blocking tests at Beta as well.

[Kamil Paral]

On Mon, Sep 17, 2018 at 2:45 PM Stephen Gallagher <sgallagh(a)redhat.com&gt;
wrote:

>  On Mon, Sep 17, 2018 at 5:16 AM Petr Šabata <contyk(a)redhat.com&gt;
> wrote:
>
>  >
>  > I hope this won't just mean we discover the problems later, in practice.
>
>
>  The policy of Fedora QA is to run all of the GA-blocking tests at Beta as
>  well.
>  
My goal has always been to run all Final testcases with the publicly
released Beta image as soon as possible. I don't think we're too successful
at running all Final testcases with pre-Beta images or Beta RCs. I'm not
aware of any policy per say. But we certainly try.

[Adam Williamson]

On Mon, 2018-09-17 at 17:25 +0200, Kamil Paral wrote:
>  On Mon, Sep 17, 2018 at 2:45 PM Stephen Gallagher
> <sgallagh(a)redhat.com&gt;
>  wrote:
>  
>  > On Mon, Sep 17, 2018 at 5:16 AM Petr Šabata <contyk(a)redhat.com&gt; wrote:
>  > 
>  > > I hope this won't just mean we discover the problems later, in practice.
>  > 
>  > The policy of Fedora QA is to run all of the GA-blocking tests at Beta as
>  > well.
>  > 
>  
>  My goal has always been to run all Final testcases with the publicly
>  released Beta image as soon as possible. I don't think we're too successful
>  at running all Final testcases with pre-Beta images or Beta RCs. I'm not
>  aware of any policy per say. But we certainly try. 
I've been saying for a long time that we ought to be doing that, and
it's why we started doing validation events for nightly composes
throughout the cycle (instead of starting shortly before Alpha as we
used to). I definitely would like us to be running the validation tests
a long time before Beta freeze.
-- 
Adam Williamson
Fedora QA Community Monkey
IRC: adamw | Twitter: AdamW_Fedora | XMPP: adamw AT happyassassin . net
http://www.happyassassin.net

[Adam Williamson]

On Mon, 2018-09-17 at 08:56 -0700, Adam Williamson wrote:
>  On Mon, 2018-09-17 at 17:25 +0200, Kamil Paral wrote:
>  > On Mon, Sep 17, 2018 at 2:45 PM Stephen Gallagher <sgallagh(a)redhat.com&gt;
>  > wrote:
>  > 
>  > > On Mon, Sep 17, 2018 at 5:16 AM Petr Šabata <contyk(a)redhat.com&gt; wrote:
>  > > 
>  > > > I hope this won't just mean we discover the problems later, in
> practice.
>  > > 
>  > > The policy of Fedora QA is to run all of the GA-blocking tests at Beta as
>  > > well.
>  > > 
>  > 
>  > My goal has always been to run all Final testcases with the publicly
>  > released Beta image as soon as possible. I don't think we're too successful
>  > at running all Final testcases with pre-Beta images or Beta RCs. I'm not
>  > aware of any policy per say. But we certainly try.
>  
>  I've been saying for a long time that we ought to be doing that, and
>  it's why we started doing validation events for nightly composes
>  throughout the cycle (instead of starting shortly before Alpha as we
>  used to). I definitely would like us to be running the validation tests
>  a long time before Beta freeze. 
Sorry, though to clarify a bit more: we *aim* to run as many of the
tests as possible, as early as possible. But it's an aspirational
thing. We don't *commit* to having the Final tests run by Beta.
-- 
Adam Williamson
Fedora QA Community Monkey
IRC: adamw | Twitter: AdamW_Fedora | XMPP: adamw AT happyassassin . net
http://www.happyassassin.net

[Chris Murphy]

On Mon, Sep 17, 2018 at 9:56 AM, Adam Williamson
<adamwill(a)fedoraproject.org&gt; wrote:
>  On Mon, 2018-09-17 at 17:25 +0200, Kamil Paral wrote:
> > On Mon, Sep 17, 2018 at 2:45 PM Stephen Gallagher <sgallagh(a)redhat.com&gt;
> > wrote:
> >
> > > On Mon, Sep 17, 2018 at 5:16 AM Petr Šabata <contyk(a)redhat.com&gt; wrote:
> > >
> > > > I hope this won't just mean we discover the problems later, in
> practice.
> > >
> > > The policy of Fedora QA is to run all of the GA-blocking tests at Beta as
> > > well.
> > >
> >
> > My goal has always been to run all Final testcases with the publicly
> > released Beta image as soon as possible. I don't think we're too successful
> > at running all Final testcases with pre-Beta images or Beta RCs. I'm not
> > aware of any policy per say. But we certainly try.
>
>  I've been saying for a long time that we ought to be doing that, and
>  it's why we started doing validation events for nightly composes
>  throughout the cycle (instead of starting shortly before Alpha as we
>  used to). I definitely would like us to be running the validation tests
>  a long time before Beta freeze. 
Maybe there needs to be a milestone for it? Something like alpha in
terms of the timing, but non-blocking since there's no release
specific to this hypothetical milestone, but does give you the
resources to plan for and implement it.


-- 
Chris Murphy

[Adam Williamson]

On Fri, 2018-09-14 at 10:22 -0400, Stephen Gallagher wrote:
>  At yesterday's F29 Go/No-Go meeting, we discussed the blocker
> status
>  of BZ #1628192 - Fedora 29 installation cannot see a firmware RAID
>  device. While the blocker criteria clearly states that this should be
>  a blocker for Beta, many of the people present at the meeting
>  disagreed, for a variety of reasons.
>  
>  * Hardware supporting fwraid is considerably less pervasive than it
>  was when the criterion was written
>  
>  * Testing this criterion can only be done with install media, which
>  limits our testing pool to the very dedicated members of Fedora QA.
>  Yes, anyone *can* download a nightly compose and try it, but in
>  practice this tends to be limited to the core testers. The majority of
>  testing that this feature will get will tend to happen as people try
>  out the Beta release.
>  
>  To that end, I'd like to propose that we make the following change to
>  the criteria going forward:
>  
>  "The blocking criterion for successful installation atop a firmware
>  RAID array is moved to the GA release criteria." 
There were no real objections to this when it was proposed, so I'm
going to go ahead and implement it in the F30 criteria. Thanks,
Stephen.
-- 
Adam Williamson
Fedora QA Community Monkey
IRC: adamw | Twitter: AdamW_Fedora | XMPP: adamw AT happyassassin . net
http://www.happyassassin.net