Release criteria: kickstart

[Adam Williamson]

On Thu, 2011-08-25 at 14:35 -0600, Stephen John Smoogen wrote:
> On Thu, Aug 25, 2011 at 14:25, Chris Lumens <clumens at redhat.com> wrote:
> >> kickstart is a very broad area; you can write extremely complex
> >> kickstart files that do a lot of stuff. So broadly what we'd need to do
> >> is define a subset of kickstart functionality that we expect to work,
> >> and then possibly divide that up by release phase (so some stuff must
> >> work by Beta, the rest by Final, for e.g.)
> >>
> >> anaconda devs following this list, do you have any existing expectations
> >> as to what level of kickstart functionality ought to be in place for
> >> releases, and when you think would be appropriate?
> >>
> >> So far it seems everyone more or less agrees that it should be possible
> >> to do at least a basic unattended kickstart install by Beta.
> >
> > You're right, kickstart is incredibly broad.  I don't think we could
> > ever hope to come up with criteria to cover all of it.  I guess the best
> > we can do is define criteria in terms of something else we already have.
> 
> I would go for the classical kickstart test for Alpha:
> 
> Does it take a minimal kickstart and build a default system. The
> minimal being the exact stuff that would be created if a person just
> clicked through a release.

Oh, I like that. That's very good.

> For Beta
> 
> Take these X broken kickstarts, does it bail at the appropriate places.
> Take these Y working kickstarts, does it work.
> Where Y is a set of items that can be tested on say a KVM or a
> "default" desktop defined somewhere.

I'm not so keen on that; it's a bit specific. One of my fetishes with
the criteria is to keep them generic so they don't have to keep being
changed all the time; I wouldn't want a criterion to rely on some
specific kickstart file we keep lying around somewhere.

> For Final
> The above and some subset of obscure items that can be tested reliably
> somewhere that development and QA can replicate.

again, seems a bit arbitrary; what we really need is the kickstart
functionality we actually believe has to work, not an arbitrary set of
interesting bugs we happen to be able to test.

a minimal answer to this question would be fairly valid, I guess. at
least in the short term. sometimes working up criteria in response to
real-world cases - i.e. someone finds a kickstart problem, files a bug,
proposes it as blocker, we agree it should be one and work up a
criterion in response to it - has proved a decent way of doing things in
the past.
-- 
Adam Williamson
Fedora QA Community Monkey
IRC: adamw | Twitter: AdamW_Fedora | identi.ca: adamwfedora
http://www.happyassassin.net