Review Request: Initial Fedup CLI Test Case

[Tim Flink]

I realize that I'm bypassing the usual process for test case review
here but considering the fact that we need an upgrade test case and
that F18 beta go/no-go is tomorrow, I didn't think that there would be
much objection to directly creating the wiki page.

With the release of F18 beta RC1 and some fedup related updates to fix
major issues, we can start doing more testing of upgrades using fedup.

I've written a test case for the basic 'upgrade F17 graphical desktop
with fedup-cli' test case. It is available at:

https://fedoraproject.org/wiki/QA:Testcase_upgrade_fedup_cli_previous_des...

If there are no objections today, I'll add it to the install testing
matrix later today so it'll be included for tomorrow's go/no-go meeting.

Comments and/or (reasonable) change requests would be much appreciated.

Thanks,

Tim

[Martin Krizek]

Hi Tim,

Is this "...default partitioning (no less than 500MB for /boot), selecting the 
default package set." really necessary? I'd say that a more valid use case 
would be upgrading with any partitioning and any package set. I mean, who uses 
just a default package set? Would it make sense to remove that part?

Thanks,
Martin

On Wednesday 21 of November 2012 13:21:50 Tim Flink wrote:
>  I realize that I'm bypassing the usual process for test case
> review
>  here but considering the fact that we need an upgrade test case and
>  that F18 beta go/no-go is tomorrow, I didn't think that there would be
>  much objection to directly creating the wiki page.
>  
>  With the release of F18 beta RC1 and some fedup related updates to fix
>  major issues, we can start doing more testing of upgrades using fedup.
>  
>  I've written a test case for the basic 'upgrade F17 graphical desktop
>  with fedup-cli' test case. It is available at:
>  
>  https://fedoraproject.org/wiki/QA:Testcase_upgrade_fedup_cli_previous_deskto
>  p
>  
>  If there are no objections today, I'll add it to the install testing
>  matrix later today so it'll be included for tomorrow's go/no-go meeting.
>  
>  Comments and/or (reasonable) change requests would be much appreciated.
>  
>  Thanks,
>  
>  Tim

[Tim Flink]

On Thu, 22 Nov 2012 12:00:20 +0100
Martin Krizek <mkrizek(a)redhat.com&gt; wrote:

>  Hi Tim,
>  
>  Is this "...default partitioning (no less than 500MB for /boot),
>  selecting the default package set." really necessary? I'd say that a
>  more valid use case would be upgrading with any partitioning and any
>  package set. I mean, who uses just a default package set? Would it
>  make sense to remove that part? 
Honestly, it depends on how you look at it. I agree that there needs to
be more variation in the system to be upgraded in order to test
upgrades more completely. It's a side effect of how we write test cases
and the resources we have to do the testing.

I think it really comes down to practicality. When you loosen up the
test case like that, you have a combinatorial explosion of potential
test cases that could cause problems. Since the fedup process uses yum
at it's core - most of those bugs would come down to the individual
packages which cause problems. If upgradepath was enforced, it would be
less of a potential issue but that's not an option at the moment.

If we look at real world use cases; using rpmfusion is common, using
some arbitrary package set is almost definitely the case; do we test
upgrades with all of those possibilities?

The problem with our release validation process is where to draw the
line between what we test and what we don't. Our test cases are pretty
much human bash scripts and that seems to be what people expect. If
you follow all the steps of a test case, you can mark it pass/fail and
be done with it; no improvisation or filling in unwritten stuff should
be required. I'm personally not a fan of this testing style but it is
what it is and I choose my battles - test case writing style is not one
that I'm choosing to fight right now.

With the testing resources we have right now, I think that limiting the
upgrade test case to the default package set is our best option. Does
it cover everything? No. Does it cover the most common issues? Probably.
It does make some assurances that the fedup process is at least working
for a known base case.

I'm not saying that people shouldn't test upgrades with something other
than the base package set or shouldn't file bugs if they hit issues
outside of the base package set. I encourage that kind of testing and
I'd love to see more acceptance of reasonable improvisation during
testing. I just don't think that the release validation test cases is
the best place to be doing that right now.

Tim

[Jóhann B. Guðmundsson]

On 11/22/2012 11:00 AM, Martin Krizek wrote:
>  Hi Tim,
>
>  Is this "...default partitioning (no less than 500MB for /boot), selecting the
>  default package set." really necessary? I'd say that a more valid use case
>  would be upgrading with any partitioning and any package set. I mean, who uses
>  just a default package set? Would it make sense to remove that part? 
We limit our test cases to only defaults as in minimal and DE not 
everything nor anything else there in between.

JBG

[Adam Williamson]

On Thu, 2012-11-22 at 11:03 -0700, Tim Flink wrote:

>  The problem with our release validation process is where to draw the
>  line between what we test and what we don't. Our test cases are pretty
>  much human bash scripts and that seems to be what people expect. If
>  you follow all the steps of a test case, you can mark it pass/fail and
>  be done with it; no improvisation or filling in unwritten stuff should
>  be required. I'm personally not a fan of this testing style but it is
>  what it is and I choose my battles - test case writing style is not one
>  that I'm choosing to fight right now. 
There's a clear benefit to the human bash script style which I think is
important in the context of release validation. It ensures (at least,
tries to - despite our relatively strict test cases, we _still_ have
problems with this) that despite the fact we have different people in
different places in different time zones running the tests on different
machines, they're all broadly testing the same thing.

If our test cases were less deterministic, it'd be very difficult to
compare results. If the upgrade test case just said effectively 'run an
upgrade however you like', and we saw 'Fail' from Bob for TC2, 'Pass'
from Anna for TC4, 'Pass' from James for TC6, 'Fail' from Bob for TC8,
and 'Pass' from Anna for RC1, what conclusions could we draw about how
the install test case behaviour had changed over those releases? Very
few or none, without careful analysis of any filed bugs, because we'd
have no idea if all those people had been testing the same thing, or
testing really different configurations.

Even though we try to be human-bash-script-y at present, the fact that
it's really difficult to nail down all environmental factors already
often gives us trouble replicating each other's results - we have lots
of 'how did you set up XYZ? was your repo on this server or that
server?' stuff flying around on IRC. If our tests were less
deterministic I think we'd find it even harder. We might gain a bit more
coverage, but at the cost of making it more or less impossible to
confidently compare test results from different testers across releases.

>  With the testing resources we have right now, I think that limiting
> the
>  upgrade test case to the default package set is our best option. Does
>  it cover everything? No. Does it cover the most common issues? Probably.
>  It does make some assurances that the fedup process is at least working
>  for a known base case. 
Right, this is the main point here - as both you and Johann said, the
limitation of the scope of the upgrade test is entirely intentional, and
an acknowledgement of the size of the problem space and the resources we
have to apply to it.
-- 
Adam Williamson
Fedora QA Community Monkey
IRC: adamw | Twitter: AdamW_Fedora | identi.ca: adamwfedora
http://www.happyassassin.net