On Thu, 2017-07-06 at 21:15 -0400, Matthew Miller wrote:
First, there is gating from rel-eng and QA in progress here:
(Note that this is
compose/validation gating, not the CI stuff we're also talking about
separately.) That's key in keeping the release basically stable. But
there's another part:
So, I guess I should set some more detailed expectations here. At least
from my perspective on it.
Compose gating is something that, in principle, we *can* do already.
It's not very difficult. We have quite extensive automated testing of
each compose, between openQA and autocloud. We have the results
reported to resultsdb. It is not fundamentally difficult to write a
thing which queries resultsdb for the test results for a given compose
and makes a decision about whether that compose should be released,
based on those results.
However, there are some counterpoints. One, as always, there's devilry
in the details. We have to decide *what* the criteria are, exactly. For
autocloud, it's sort of 'easy', because autocloud results are not very
granular at all - it's basically a straight up pass/fail for each image
in a compose.
But for openQA...well, I can draw up a list of the openQA tests that
correspond to Alpha criteria, easily enough. But do we want to go
straight there, or start smaller?
Also, what do we do about...*complex* failures? For instance, I already
know, right now, that every so often, anaconda just crashes in the
middle of an install. It's been doing that for a year or so. It's a
mysterious python crash, and it happens very rarely. There are also
similar known 'it occasionally just crashes' bugs in GNOME and KDE -
sometimes KDE startup just fails and the system sits at a black screen
forever, occasionally GNOME crashes back to gdm shortly after login.
And of course sometimes openQA just screws up (it's not perfect), and
sometimes some network or mirror issue produces a failure, etc etc.
So do we introduce some sort of 'fuzz factor' and say 'it's OK if 90%
of the tests pass'? Do we set things up so the 'is it OK to ship?'
calculation automatically re-runs every time a test is restarted, so we
can just manually restart these kind of tests and if they pass, the
compose will ship? Do I start writing some kind of complex openQA
plugin to try and identify 'known intermittent failures' and
automatically restart such tests? (this is the sort of thing that's
possible, but could also eat my life.)
There's another big point: I suspect that doing this kind of compose
gating is going to *feel* like quite a big change to the distro
development process. releng has been kinda gradually introducing more
and more hurdles to the compose and sync process, but it's still *more
or less* the case that people expect a Rawhide compose to succeed and
sync every day - when composes fail for more than two or three days,
people start getting antsy. If we add compose gating, I really don't
think there's any possible outcome except that 'successful' composes
get noticeably rarer. I can't put numbers on that yet; we could write a
script which applies whatever criteria we decide on to the last, say,
year of Rawhide composes and gives us an idea what percentage would've
met the criteria, but of course that's not really a true representation
because the act of introducing criteria provides a powerful incentive
for people to fix problems which wasn't there before. All we can really
say is that it's pretty likely that 'successful' Rawhide composes will
become to some noticeable degree rarer. And I'm pretty sure that will
have other consequences on how the whole process of working on Fedora
feels. But it's difficult to be too specific until we actually do this.
Honestly - I was kinda banking on us having a reasonable amount of the
usual 'slack time' at the start of a release cycle to try and do a bit
of a 'soft launch' of compose gating, with at least a few weeks for us
to shake down all the details and get a feel for how significant the
impact on the development process is. What you're talking about feels
somewhat different. I'm not necessarily comfortable with us banking on
the idea that compose gating is going to be something we can kick in
*immediately* with complete success, to the degree of basing F28/F29
plans on it.
Fedora QA Community Monkey
IRC: adamw | Twitter: AdamW_Fedora | XMPP: adamw AT happyassassin . net