Our current upgrade criterion says:
"For each one of the release-blocking package sets, it must be possible to
successfully complete an upgrade from a fully updated installation of the previous stable
Fedora release with that package set installed."
https://fedoraproject.org/wiki/Fedora_24_Beta_Release_Criteria#Upgrade_re...
Currently we have no criterion that would cover upgrading across 2 releases, e.g. from F21
to F23 (directly, not one by one). But in the real world this very often happens. It's
even one of the reasons we support our releases until N+2 release is available + 1 month
(i.e. F21 is supported until F23 is out + 1 month). The often cited reason is for people
to be upgrading just once per year (and have one month to do that). And of course many
(probably most) of them don't upgrade one by one, but skip a release.
I feel that for something as important as system upgrade, we should provide a better level
of quality and assurance for upgrading across 2 releases. Currently we have no criterion
and testing it is just an afterthought, not even tracked anywhere. I'd like to amend
the existing criterion to include N-2 release as well, i.e.:
"For each one of the release-blocking package sets, it must be possible to
successfully complete an upgrade from a fully updated installation of any of the two
previous stable Fedora releases with that package set installed."
(language corrections very welcome)
We can discuss whether N+2 upgrading should be a separate Final criterion, not joined with
the Beta one. I don't feel strongly either way.
I'd also set up a new test case in our installation matrix in the upgrade section:
https://fedoraproject.org/wiki/Template:Installation_test_matrix#Upgrade
Something like QA:Testcase_upgrade_dnf_skip_release. The question is whether to have just
a single test case and let people choose which package set they test, or whether to pick
some particular package set. We probably don't want to test all combinations, at least
not manually. Just a single "please test something" test case would be
satisfactory here, I think. Something will get tested, and we will block on important bugs
we discover, that's the important change.
If we decide to not go this route for some reason, I think we should adjust our tools
(system-upgrade) and documentation (wiki, fedora docs) and provide very clear and visible
warning that the only officially supported means of upgrading is to go up releases one by
one. And that skipping releases might be dangerous (considerably more than doing it the
recommended way). Because I feel we would be doing our users a disservice if we neither
tested skipping releases nor warned them against doing that.
Thoughts?
Kamil