Review of Fedora 18 Release Criteria

[Adam Williamson]

On Wed, 2012-10-10 at 16:43 -0400, David Cantrell wrote:

> > I don't think my disagreement is necessarily a problem for you,
> > though...let me see if I can put it a different way:
> > 
> > Take the example you just gave. My position isn't 'we should take that
> > bug as a blocker both four weeks out and one week out'. My position is
> > 'we shouldn't take it as a blocker four weeks out _or_ one week out'.
> > 
> > I'd phrase the problem more as 'there is a tendency in blocker review to
> > accept things too casually early on'. We should only accept things as
> > blockers that we really believe we should hold the release up for - even
> > if it's six weeks until release and we know the bug is going to be fixed
> > in two days, we should not accept it as a blocker unless it passes the
> > 'if today was go/no-go and this was the last bug, would we block the
> > release' smell test.
> > 
> > Rather than throw almost everything on the blocker list when it's early
> > and get more rigorous later on, I'd rather we just be rigorous the whole
> > time. The blocker list isn't a work list (using the blocker list as a
> > 'todo list' / reminder list is a tendency I've noticed in both RHEL and
> > Fedora, and something I think we should fight), it is a list of bugs
> > that must be fixed for the release to go out. No more, no less. If we
> > wouldn't really block the release for a bug, it doesn't belong on the
> > list at any point in time. I think we've been making a good effort to
> > improve this in recent releases; it's not just an aspiration.
> 
> That's fine, but how is that not a to do list?

I should have said not _solely_ a to-do list. The phenomenon I had in
mind was when people throw stuff on the blocker list not because they
really think it should block release, but just because they know if it's
on the blocker list it'll get treated with high priority (either a user
trying to 'game the system' into getting their bug fixed quickly, or a
developer abusing the blocker list as their todo list). That happens,
and it shouldn't (we've been pretty solid about rejecting such proposals
recently).

So yeah, the blocker list ultimately winds up functioning as a to-do
list, but that's a consequence, not what it's actually _for_. 'I want to
make sure we don't forget about this' isn't a valid reason for a bug to
be on the blocker list. People should be using another tracker bug for
that, or the 'Priority' field, or anything at all else that they like,
but not the blocker list :)

> > So if there are concrete cases where you think bugs have been accepted
> > as blockers that you wouldn't be comfortable taking as blockers the day
> > before release, that's _always a problem_, and please highlight those
> > bugs to us. I don't think it's 'not a problem' if we accepted them as
> > blockers a long time before release and they got fixed before any chance
> > of causing a slip, it's still a problem. We should stick firmly to our
> > tight, minimal definition of what constitutes a blocker, regardless of
> > the point in the release schedule.
> 
> The big difference here and what we see on the RHEL side is that RHEL allows
> for blocker bugs to be re-evaluated and booted if a later workaround or some
> other acceptable solution besides a bug fix is found.  It becomes a schedule
> thing, which I am routinely told is very important for Fedora.

This is certainly the case for Fedora too. Accepted blockers are
reviewed at each meeting - usually just to check progress, but the
blocker status of the bug can also be reassessed if new information that
changes the equation has showed up. We've certainly done this on many
occasions (too often, for some people's liking).

> If something marked as a blocker early on is still not fixed through several
> test releases, does it really matter anymore for the release?  If not, why
> not boot it off the blocker list?

Again that doesn't seem like a good factor to take into account. The
fact that it hasn't been fixed doesn't tell us anything about how bad a
bug it is. If it hasn't been fixed _and no-one has complained_, then
that might be something to take into account, as it would indicate the
impact of the bug is narrow. But if it hasn't been fixed, but there are
200 people on the bug report baying for blood, taking it off the blocker
list doesn't seem indicated :)

> As we have both already stated and agreed on, the installer is unique.  Our
> development process is so very much tied to the Fedora release schedule.
> There is no easy way for us to develop something large like a new installer
> and not have it tied to a release.  This became harder for us when we
> stopped doing rawhide nightly composes many years ago (and I have no idea if
> those are happening again these days).  People don't go and download a new
> installer to try it out.  You download a release of something and try
> installing it.  Our only way to get that to users is to follow the Fedora
> release schedule, unless we want to start making our own distribution.
> 
> I know what you're getting at and it's that newui was a huge feature and
> that's made our lives hard.  Unfortunately, the Fedora release schedule is
> such that work like this is always painful for us.

Well, there's a case there that we should change the release schedule,
then. It's a Perennial Topic, I know, but it could happen. We have a
three sided triangle here:

	Shiny New Features


Quality				Length of Release Schedule

You can pick any two, basically. You can have a short release schedule
and lots of shiny new features. You can have a short release cycle and
great quality. You can have great quality and shiny new features. But it
does seem rather difficult to achieve a short release schedule, great
quality, and lots of shiny new features. Broadly, we compromise on
quality to achieve this - our standards are a hell of a lot slacker
than, oh, say Debian's. But if we get to the point where we'd have to
compromise our quality standards too heavily to achieve the shiny new
features we want on the current release cycle, we have to bring the
release cycle point of the triangle into the equation too.

> > It might be an idea to start CCing the blocker meeting announcements out
> > a bit more widely, to help with this. Or heck, we could also have some
> > non-QA person run the meetings for a while, give it to FPL or FPM or
> > someone. That would liven things up.
> 
> The SOP can say whatever it wants, but what actually happens in practice
> matters.  

Sure, I agree, and as I said, I can see two or three practical things we
could do to try and address this.

> People don't go to the meetings because they take too long.  I'm
> going to make a comparison to RHEL process again, but I think it's useful
> here.  Meetings have a hard time limit.  There is only so much work a person
> can do in a day and it's ridiculous to keep going through bug status over
> and over and over just because it's there.  When RHEL meetings get in to
> "bugzilla bingo", they cover what can be covered in the allotted time.  What
> isn't covered isn't forgotten, just brought up at the next meeting.

We've started doing that this week, actually, and limiting the meetings
to three hours. But we're not just delaying stuff to the next meeting
because it'd just keep piling up, we're just running another meeting the
next day if we overrun. So we're doing another review meeting tomorrow
A.M. because we didn't get through all the bugs today in 3 hours.

> And from my own point of view, I've never seen your invites in #anaconda
> simply because I do not look at IRC unless addressed by my nick.  I can't.
> I don't have time to sit and watch the text scroll.  The length of this
> reply that you sent....I get hundreds of those per day that I'm expected to
> read and respond to.  Plus all the meetings I'm scheduled for on a daily
> basis.  This is not me whining, just trying to explain how I make the most
> effective use of the resources I have.  Direct invites to these meetings
> would help me work them in to my schedule and perhaps even attend.

I know I get wordy, sorry, it's a tendency of mine...I like to think
about things on a very broad / long-term / comprehensive scale and I
tend to develop those thoughts in discussions like this (which I find
really useful, btw, so thanks).

I'm not that happy with having a giant list of direct CCs on
announcement mails because it's just inelegant and bitrots quickly (you
wind up with announcements going to the person who used to be the FPL
three years ago). It's what announcement MLs are for, after all: posting
important announcements and nothing else. test-announce is very low
traffic, I don't think it's a huge burden to subscribe to. We could
certainly start CCing devel-announce and a couple other major announce
lists if we aren't already. Is there an anaconda-announce?
-- 
Adam Williamson
Fedora QA Community Monkey
IRC: adamw | Twitter: AdamW_Fedora | identi.ca: adamwfedora
http://www.happyassassin.net