Release criterion proposal: upgrade methods

[Adam Williamson]

On Fri, 2012-10-05 at 04:55 -0400, Kamil Paral wrote:

> > I think we have to rejig the whole thing somehow. Um.
> > 
> > For each one of the release-blocking package sets ('minimal', and the
> > package sets for each one of the release-blocking desktops), it must
> > be
> > possible to successfully complete an upgrade from a fully updated
> > installation of the previous stable Fedora release with that package
> > set
> > installed, using any officially recommended upgrade mechanism. The
> > upgraded system must meet all release criteria."
> > 
> > I _think_ that gets it (with, again, the obvious corresponding
> > criterion
> > for Final). English, eh. Who'd use it.
> 
> That almost begs for a mathematical formula instead.
> 
> Now seriously - after reading the email, I wonder whether non-native
> English speakers will ever be able to distinguish these subtleties. I
> wanted to say "especially for those outside the QA team, who don't
> read this list often", but then I realized that I'll forget the
> correct meaning soon myself, and I'll have to ask for confirmation
> again and again in the future.
> 
> If we want to engage more community and link to these criteria, so
> that they tag blocker bugs, they have to be as accessible as possible.
> Maybe we could put explanations right into the criteria text? Like
> "any of supported mechanism (all of them must work)" or "any of
> supported mechanism (at least one must work)". It doesn't look pretty
> and it makes the text longer, but it makes it pretty clear.
> 
> Of course if we intend to use these criteria only in the core QA team,
> and ask community to tag blocker bugs intuitively (without really
> reading those criteria), then it's probably fine. But one day, one
> day, we will be deciding blocker bug status and no English language
> expert will be around, and then it's gonna be tough.
> 
> I'd rather have it longer and clearer.

So I think you have some good points for sure. Talking about the
criteria in general, I'd say we have the problem that we have four
requirements for the criteria which are sort of competing against each
other:

* Clarity / ease of understanding
* Precision
* Length
* Comprehensiveness

It's very hard to write criteria that are short, clear, precise and
comprehensive. We started out after the big revision for F13 with
criteria that were short and clear:

https://fedoraproject.org/wiki/Fedora_13_Alpha_Release_Criteria

but the problem is that over time it became clear they weren't precise
enough. We want the criteria to say what they mean, so when we had to
take a tricky case and decide exactly what we meant to cover with the
criteria, we've written that into the criteria, and we've added quite a
few over the years.

So I think we've definitely come to a point where the simple format we
currently have - a single numbered list of plain text rules - is getting
a little hard to manage, and it's certainly becoming a bit of a
wall-o-text. I think what we probably need to do is look at doing a
reshuffle of the overall presentation of the criteria, after F18. I'll
put it on the retrospective so someone can take it as a task for F19.
I'm happy to do it, but if anyone else wants to, please do - the
criteria have been kind of an 'adamw thing' and I think it's usually
healthy when you get a fresh perspective on this kind of thing.

Ideas I've had include splitting the criteria up into groups, having a
proper glossary of those terms we've given specific meaning, and having
a 'concordance' which explains particular conventions we've developed in
interpreting the criteria, like the conventions we have for
hardware-specific bugs and so on.

Coming back to this particular case, I left my thought process in there
for a bit of fun, but I don't think the final draft I came up with is
terrible. Looking at it linguistically, I think the worries about 'or'
and 'each' and 'any' and so on were almost red herrings: the key problem
is that we need to address in the criterion a somewhat complex
hypothetical scenario. The way the criterion was written at first, from
which we were trying to 'patch' it, was looking at a hypothetical
scenario where there's a single notional installation of Fedora 17 and
we're talking about the conditions for upgrading it to Fedora 18, say.
It started out with the text "It must be possible to successfully
complete an upgrade from a fully updated installation of the previous
stable Fedora release". The mental picture that gives you is, well, just
as it says: 'a fully updated installation...'. One single fully updated
installation.

All the trouble we had with the update came down to that limitation of
the original wording, because that's not actually the concept we really
want to express. We want to express the hypothetical scenario of
_several_ fully updated installations of the previous stable release,
and apply a certain condition to each one of them: the mental picture we
needed the criterion to express is 'you're sitting in a room with three
systems, one a Fedora 17 GNOME install, one a Fedora 17 KDE install, one
a Fedora 17 minimal install, and you're going to try and upgrade them
all to Fedora 18 in the same way'. As long as we kept the "It must be
possible to successfully complete an upgrade from a fully updated
installation of the previous stable Fedora release" wording as our
'scene setter' it was never going to work. That's what my last attempt
(I hope) fixes: by starting out with "For each one of the
release-blocking package sets" instead, it immediately sets up this
'multiple starting point' scenario, which I hope makes the whole thing
clearer. I actually think it's fine, now I finally got my head around
that - if you forget all the troubles we had with the rewrite, and
forget the rest of my mail, and just read the final proposal I came up
with, is it actually very difficult to understand?
-- 
Adam Williamson
Fedora QA Community Monkey
IRC: adamw | Twitter: AdamW_Fedora | identi.ca: adamwfedora
http://www.happyassassin.net