Plan for tomorrow's FESCo meeting (2010-11-17)

[François Cami]

On Sat, Nov 20, 2010 at 5:44 PM, Adam Williamson <awilliam at redhat.com> wrote:
> On Sat, 2010-11-20 at 11:23 +0100, Till Maas wrote:
>> On Fri, Nov 19, 2010 at 10:18:38PM -0800, Adam Williamson wrote:
>>
>> > place. The idea was never that some magic independent group of testers
>> > would spend the rest of their lives doing nothing but test updates.
>>
>> This idea was never prominently communicated as the default
>> situation. Iirc it was said that there are lots of people who want the
>> update criteria and will test updates. Making package maintainers now
>> start to beg for their updates to be tested is imho just a big waste of
>> time.
>>
>> Also there is no dispensable manpower from package maintainers
>> available, so requiring them to additionally test each other updates
>> manually and to maintain test machines is not a good idea. The whole
>> update criteria enforcement only works if there are enough dedicated
>> testers that provide extra manpower. Or if the testing is all automated.

There is also no other choice if we want to reduce the probability of
introducing regressions in updates.
Testing complex packages (esp. network services, virtualization, and
even whole desktop environments, see below) requires some technical
skills and also knowledge of these packages. In other word, we need
technical-minded, knowledgeable people in the proventesters group.
Note that the dedicated proventesters (apparently) provide extra
manpower, possibly not enough yet for the goals we've set for
ourselves.

> I'm looking back through the hideous fesco meeting discussions of this
> stuff and I don't see any suggestion that there would be "lots of people
> who want the update criteria and will test updates". Granted I may be
> missing it, but I can't find it. We set up the proven testers group to
> test *critpath* updates, they're not really expected to be any more
> likely to test non-critpath updates than anyone else; even there, the
> whole point of the proven testers group is to get as many people as
> possible to join, and from outside of QA. The last time we went through
> this I suggested to the KDE SIG that they have their members sign up as
> proventesters so they could test each other's packages, and I know that
> at least some of them did sign up; however, it appears that now they
> don't want to be bothered to do the testing, if what you're saying is
> true for the team.
>
> I really don't see how you can say that testing updates is a 'waste of
> time' with a straight face. It *takes* time, yes. It may be boring
> sometimes, yes. But a waste of time? Do you write your code perfectly
> first time, every time? Does it never have any bugs in it? Are you 100%
> confident when you run 'make' that everything's going to pop out the
> other end in perfect working order? Or sometimes, just sometimes, do you
> screw up, and when the compile fails or the built binary fails to run,
> you go 'oops, yeah, better fix that'? That's 'testing', and everyone
> does that (at least I really hope they do). Why wouldn't you want to do
> the same for packages? Just like code doesn't always come out right the
> first time, neither do packages. I really don't see why you think it's a
> great idea to push an update of the entire KDE desktop to a stable
> Fedora release without any of you actually installing it and checking
> that it *works*.

A Desktop Environment being a complex piece of code (this is
especially true for the biggest DEs, KDE and GNOME), I think someone
else, and preferably a lot of people, than the maintainer needs to
test it before it being pushed to stable. It's all about having
different use cases. For instance, I know that I only use a fairly
small subset of XFCE. So while I, as a proventester, could confidently
say that the parts I use do work after a XFCE update, I would never go
as far as pretend each and every plugin or feature works. So we
probably need more than a single tester to feel confident about a DE
update.

I also happen to believe that pushing a new version (not a dot
release) of an entire DE (or any complex software for that matter) to
a stable Fedora release is a very bad idea. There might obviously be
new features and bugfixes but there *will* be other bugs. It's a fact.
Since we're apparently not always prepared to backport fixes from
upstream each time we find bugs, I fail to see why replacing a release
by another, each with its different bugs, is a win, from the user's
point of view. IMHO this is far too much disruptive a decision, since
users who rely on specific features to work could get their desktops
hosed between releases. How are they supposed to handle that?

François