F19 Final criteria revamp

[Adam Williamson]

On Tue, 2013-06-11 at 08:50 -0400, Kamil Paral wrote:
> > Hey folks - so just ahead of the blocker meeting tomorrow, I'm done with
> > the Final criteria rewrite.
> 
> Thanks a lot, I reviewed them and they seem to be fine. Some comments below.
> 
> I noticed that the upgrade criterion went missing. But we already have
> one in Beta, and it seems to be the same. What am I missing?

Sorry, I should have noted that. You're missing that the 'old' criteria
were tuned for the pre-F18 situation where we had multiple possible
'recommended' upgrade methods, to whit, preupgrade and 'boot an image
and pick upgrade'. The Beta criteria required any single 'recommended'
method to work. The Final criteria required all 'recommended' upgrade
methods to work.

Now we only have one 'recommended' upgrade method, so it seems
unnecessary to keep that complexity. We can always add it back if we
grow another 'recommended' upgrade method, which I sincerely hope we
don't :)

> > * I wrote an exception for hardware-based services when the hardware
> > isn't present into the 'services' criterion: we waived a couple of
> > 'blockers' for F18 on the basis that it's okay if a service fails if
> > it's for hardware that isn't present, so I thought it made sense to
> > write that into the criterion. (Obviously it's best if we can make the
> > service not fire unless the hardware is present, but I don't think it
> > makes sense to block the release on that kind of thing).
> 
> The "unless they require hardware which is not present" phrase could
> be separated into a comment box, so that the original sentence is
> cleaner. Just an idea.

True, it could. I'll try writing it up both ways and see which one looks
best...

> >  SELinux and crash notifications
> > There must be no SELinux denial notifications or crash notifications
> after boot of a release-blocking live image or at first login after a
> default install of a release-blocking desktop. 
> 
> Should we add "during installation" as well? Wrt the bug discussed in
> the last blocker bug meeting.

Hum, somehow I thought I had, but I must've got confused (I was juggling
a few things while I was doing the rewrite). Indeed we should add that.
Thanks. The non-live installer does not have the facility to display AVC
alerts so we're covered there, but can crop up during live installs.

> > * I watered down the 'desktop' criteria quite a bit. Looking at them
> > afresh I really think we were kind of over-reaching when we wrote them
> > for F13: I remember we were thinking about 'polish criteria', but now
> > we've had this process in place for a while, it really doesn't make
> > sense to block the release on fairly trivial 'polish' issues (especially
> > when we happily ship with much bigger issues in slightly different other
> > areas). So I nuked the 'icons can't be fuzzy' requirement, the
> > requirement for Help and About menus to be present, and the bit about
> > apps not showing up twice in the menus. Those are all nice things to
> > check, but I really don't think we need to be blocking releases on them.
> 
> I agree. These issues were too trivial to block release on. Still, I'd
> like to see /someone/ to ensure the polish. It would be great to have
> both QA team and Polish (UX) team. At present, we need to handle that.
> But this change is fine. I'm looking forward to simplifying the test
> cases.

Yes, I agree we shouldn't lose the goal of trying to take care of polish
entirely. Pity the desktop team isn't going to be at Flock :(

> > * I watered down the artwork criterion yet further: the initial idea was
> > that we were going to be super-keen about having a consistent background
> > graphic from bootloader to desktop, but that's really fallen by the
> > wayside since F14 or so. Given our overall boot process nowadays, it
> > only seems particularly important to make sure we use the right image
> > for the desktop background. It'd be great if the project goes back to
> > that goal, but clearly so far there hasn't been the development
> > commitment to keep it going.
> 
> " All Fedora artwork visible in critical path actions on
> release-blocking desktops must be consistent with the proposed final
> theme." 
> 
> Maybe "must not be inconsistent"? They are hardly consistent now.
> Plymouth theme is the same for years, GDM background has nothing to do
> with desktop background.
> 
> Maybe drop the sentence completely?

Probably best to run it by the artwork team. I thought the 'must be
consistent' wording was sufficiently weaselly to let us wave through
stuff that isn't outrageously incongruent, but maybe it doesn't serve
much of a purpose any more.

Thanks for your thoughts! I'm planning to go back through and twiddle it
based on all the feedback from various people at some point this week,
then present a revised draft for review.
-- 
Adam Williamson
Fedora QA Community Monkey
IRC: adamw | Twitter: AdamW_Fedora | identi.ca: adamwfedora
http://www.happyassassin.net