F13 Schedule Proposal--please RESPOND - 'test compose'

[James Laska]

On Wed, 2009-11-11 at 18:31 -0800, John Poelstra wrote:
> On 11/06/2009 12:53 PM, Bill Nottingham wrote:
> > Sorry about the delay - been drowning in F12 final hair-on-fire
stuff.
> >
> > John Poelstra (poelstra at redhat.com) said:
> >>>> I'm not sure what period
> >>>> you believe should be a month.  Can you be more specific?
> >>>
> >>> The alpha/beta interim time; i.e., the time between corresponding
> >>> alpha and beta milestones.
> >>
> >> Which alpha and beta milestones--Start/End of Alpha freeze? Alpha
> >> Public Release? Beta Freeze? Beta Public Release?  Specific date
> >> comparisons would be helpful.
> >
> > The interim period between the alpha and beta public release is the
> > one that was most obvious to me. In every prior release except F-12
> > (because F-12 slipped), it's been right around a month. In your
schedule,
> > it's set at 6 weeks. Hence my query as if you were just
incorporating
> > the F-12 slip time.
> 
> The duration between public alpha and beta has been all over and I'm 
> unsure of its significance (if I understand you correctly below,
you're 
> suggesting that measuring the time between milestones is meaningless):
> (in days)
> F8 -  21
> F9 -  23
> F10 - 35
> F11 - 28
> F12 - 56
> 
> F13 - 42 (original proposal)
> 
> I think a better way to break this down is to be more specific about
the 
> tasks in between because saying "one month in between" is too vague.
To 
> make it four weeks we'd have to set:
> 
> Alpha Public Release Available
>    2 weeks
>     --followed by--
> Beta Freeze
>    2 weeks
>     --followed by--
> Beta Public Release Available
> 
> I'd propose making the Alpha Public Release three weeks--the original 
> proposal was four.
> 
> For the past couple of releases there was been two weeks between the 
> freeze date and the public release date. We need two weeks to include
a 
> "Test Compose"--granted I don't believe we were able to create any of 
> the Test Composes on time (something we should examine when creating
the 
> F13 schedule).

That is something I'd like to wiggle into the RC process for F-13.  For
each of the F-12 milestones, we had the following quality milestones:

      * Pre-RC installation verification (non-media, rawhide-based)
      * Test compose verification (media provided by rel-eng)
      * Release candidate verification (media provided by rel-eng)

We *informally* did this for the RC phase.  But this was adhoc and I
think we could improve our coverage if we can plan to this milestone.

Can we add a 'test compose' to the release candidate phase?

> > If that's adjusted down to 4 weeks, the other related dates move a
> > similar amount. Alpha Release ->  2/23, Alpha Freeze ->  2/9,
Feature
> > Freeze ->  2/1, Feature submission ->  1/19.
> >
> > (As a random practical note, feature submission right after New
> > Years may not be best.)
> 
> I agree.  I was following the methodology set forth by FESCo that the 
> submission deadline should be two weeks before Feature freeze.  I'll 
> keep it in mind as other dates move around.
> 
> <cut>
> 
> >> concern around being eaten by raptors.  Hopefully it can also
reduce
> >> the amount of discussion about future schedules because after 12
> >> releases we've finally figured out a few things that work best.
> >>
> >> I am concerned that the Release Engineering team is not growing and
> >> new people do not come to our meetings or look for ways to get
> >> involved and help.  I thought it would help to create a better
trail
> >> of documentation about our repeatable processes so that others can
> >> know how and why we do what we do.
> >
> > No, more documentation is good; it just surprised me that it was all
> > in one place as a methodology, as that's not the sort of thing I'm
> > going to go randomly looking for if it's not pointed out.
> >
> > One note as to the methodology as denoted on the spreadsheet - we
> > *do* need to define whether we are time-based with specific target
> > dates, or time-based with a specific length of cycle. If we are
> 
> Good distinction.  You are right, we can't be both :)
> 
> > time-based with specific target dates (May 1/Nov 1), tracking
> > 'start ->  <foo>  milestone' durations, with 'start' being the
actual
> > release date of the prior release, is meaningless to creating the
> > next schedule. If we're time based with 'a 6 month cycle, however
> > it lands', then those 'start ->  milestone' durations become a
> > meaningful thing to track again.
> >
> 
> Meaning can be found in how long it actually takes us to complete 
> certain tasks and the overall durations of our schedules.  Because
when 
> we estimate these things wrong or don't meet the originally planned 
> schedule we slip (ship late).  If we find that certain tasks are
usually 
> complete on time or within an acceptable variance we know they they
are 
> good durations to use in the next schedule.
> 
> Whether we target a specific GA date or say we are a '6 month cycle'
we 
> still have to make decisions about how long each of the tasks can
take 
> within the overall alloted time period.  This results in trade-offs
and 
> decisions about what parts of the schedule should remain the same for 
> each release and which parts can change without adversely affecting
our 
> ability to meet the scheduled GA date.
> 
> We've generally found that we need a certain amount of time between 
> freeze and public release (2 weeks).  We've also found that for a
public 
> release to be worth creating it needs to be available for a period of 
> time before creating a new one.  With these task durations less
flexible 
> the schedule has to give somewhere.  In the past several releases
we've 
> been taking that off the development time (start to first freeze).
> 
> >>> 'work all done' is the RC freeze. Not all work is equal, though;
there's
> >>> UI changes that affect collateral, rebases of fringe leaf node
packages,
> >>>
> >>
> >> It would seem to me that this isn't really working for us.  We
> >> haven't been able to compose an RC on time for the Alpha, Beta or
> >> Final.
> >
> > Part of this is a more strict handling of RCs. Before, we'd just
> > compose anyway, and not track the blocker list with such severity.
> > Now that we're actually treating the blocker list with a little more
> > process, we're slipping more.
> 
> Interesting observation.  I hadn't considered it this way.  Are you 
> suggesting that we should handle blockers differently?  Is this is a 
> good or bad thing to you?

Great thing imo!  I think this results in improved quality, improved
communication between all parties, and rewards testers for finding high
severity/impact issues that the group feels should be blockers.

That said ... Adam and I have been discussing some ideas on how we can
improve the blocker process to make for meetings that aren't as
intensive.  Expect more from QA on this soon.

> >> Is there any substance to the idea that if the "work"
> >> stopped just a little earlier (5-7 days) there might be more time
> >> for the developers to focus on the blockers so the RCs could be
> >> built in time and it wouldn't be such a last minute scramble?
> >
> > That would just be moving the freeze date, wouldn't it? As a
> > counter-example, here in F-12 we've been frozen and just taking
> > fixes, and we still had blockers when it came time to compose the
> > RC.  I'm not sure that more freeze time would help here. Part of
> > me thinks that what would help the most is more attention to
blockers
> > *throughout* the process.
> >
> 
> We reviewed the blocker list every Friday starting at least two weeks 
> before every freeze until release.  What would you suggest as an 
> additional or alternate way to focus on blockers?

For me ... see above.  There is room for improving the process of
proposing blockers to reduce the average time spent per bug in meeting.

Thanks,
James
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 198 bytes
Desc: This is a digitally signed message part
Url : http://lists.fedoraproject.org/pipermail/rel-eng/attachments/20091112/5eac9698/attachment.bin