N-1 upgrade test case proposal
Hi folks! So at the meeting this week we agreed to add test cases for
N-1 upgrades. I've adjusted the templates used to generate the upgrade
test cases to allow for this, but now I notice there's kind of a naming
conflict.
All the current upgrade test cases are named upgrade_dnf_previous_foo ,
with 'previous' meaning 'upgrade from the previous release relative to
the one being tested'. But the FedoraVersion template that's used in
the upgrade test case templates also uses the term 'previous', only it
uses it to mean 'previous stable release'. In other words, right now
for the upgrade test case names 'previous' means 'Fedora 23', but for
the FedoraVersion template it means 'Fedora 22'.
The way I implemented source release choice in the upgrade test case
template is to let you set a value that'll be passed through to the
FedoraVersion template. So we're in the kinda odd position right now
that our 'previous' test cases pass the value 'current' to
FedoraVersion, and to add N-1 test cases we'd have to name them
something like 'upgrade_dnf_previous2_foo' but have them pass the value
'previous'.
That seems like it'd be confusing for anyone who came along in a year
or two and wanted to figure out what was going on, so I'm proposing
that instead, we rename the existing upgrade tests to follow the
FedoraVersion template naming. We'd have 'upgrade_dnf_current_foo' test
cases which test upgrade from current stable to release under test (23
-> 24 at present), and 'upgrade_dnf_previous_foo' test cases which
would test upgrade from previous stable to release under test (22 -> 24
at present), and the names of the test cases would match the values
they use for the template params.
The only slight drawback I can see is that it'd lead to a slightly odd
effect in testcase_stats: as the *newly added* test cases would have
the same name as the *current* test cases currently have,
testcase_stats would group them together - so testcase_stats would show
a full series of 'upgrade_dnf_previous' results for F24, but in fact
the first few of those would be F23->F24 upgrade tests, while the rest
would be F22->F24 upgrade tests. This doesn't really seem like a big
problem, though, we're very early in the cycle and testcase_stats is
just an aid for testers, it's OK for it to have this kind of wrinkle.
So does anyone have any objections to changing things so we have
'upgrade_dnf_current_foo' test cases which are for F23->F24, and
'upgrade_dnf_previous_foo' test cases which are for F22->F24? If no-one
sees any issues I'll go ahead and do that by the end of the week, and
make appropriate adjustments to the matrices and the openQA wiki
reporting code.
As discussed at the meeting, the N-1 test cases would be marked
'Optional' for now, as we want to talk to FPC about packaging
guidelines for N-1 upgrades before we make them part of the release
criteria.
Thanks folks!
--
Adam Williamson
Fedora QA Community Monkey
IRC: adamw | Twitter: AdamW_Fedora | XMPP: adamw AT happyassassin . net
http://www.happyassassin.net