Re: N-1 upgrade test case proposal
Hi folks! So at the meeting this week we agreed to add test cases
for
N-1 upgrades. I've adjusted the templates used to generate the upgrade
test cases to allow for this, but now I notice there's kind of a naming
conflict.
All the current upgrade test cases are named upgrade_dnf_previous_foo ,
with 'previous' meaning 'upgrade from the previous release relative to
the one being tested'. But the FedoraVersion template that's used in
the upgrade test case templates also uses the term 'previous', only it
uses it to mean 'previous stable release'. In other words, right now
for the upgrade test case names 'previous' means 'Fedora 23', but for
the FedoraVersion template it means 'Fedora 22'.
The way I implemented source release choice in the upgrade test case
template is to let you set a value that'll be passed through to the
FedoraVersion template. So we're in the kinda odd position right now
that our 'previous' test cases pass the value 'current' to
FedoraVersion, and to add N-1 test cases we'd have to name them
something like 'upgrade_dnf_previous2_foo' but have them pass the value
'previous'.
That seems like it'd be confusing for anyone who came along in a year
or two and wanted to figure out what was going on, so I'm proposing
that instead, we rename the existing upgrade tests to follow the
FedoraVersion template naming. We'd have 'upgrade_dnf_current_foo' test
cases which test upgrade from current stable to release under test (23
-> 24 at present), and 'upgrade_dnf_previous_foo' test cases which
would test upgrade from previous stable to release under test (22 -> 24
at present), and the names of the test cases would match the values
they use for the template params.
Honestly, I've always found it confusing to refer to last stable release as
"previous" in the test case name. So I welcome this change, because I think it
makes more sense and is more in line with our terminology.
The only slight drawback I can see is that it'd lead to a slightly odd
effect in testcase_stats: as the *newly added* test cases would have
the same name as the *current* test cases currently have,
testcase_stats would group them together - so testcase_stats would show
a full series of 'upgrade_dnf_previous' results for F24, but in fact
the first few of those would be F23->F24 upgrade tests, while the rest
would be F22->F24 upgrade tests. This doesn't really seem like a big
problem, though, we're very early in the cycle and testcase_stats is
just an aid for testers, it's OK for it to have this kind of wrinkle.
I think this is not important at this moment.
Thanks.