4:55 p.m.
During FUDCon, we've been working on revising the Fedora release criteria.
John Poelstra had already fleshed out a structure and much of the final
content, and we've been revising and tweaking it in conjunction with QA
(myself, Will Woods and James Laska), release engineering (Jesse Keating),
anaconda team (especially Denise Dumas and Peter Jones) and desktop team
(Christopher Aillon and Matthias Clasen, who provided suggestions at an
earlier stage).
The new structure is based around a general page and specific pages for the
Fedora 13 Alpha, Beta and Final releases (which have been written
generically so they can easily be converted into pages for F14 and all
future releases just by copying and pasting). You can find the criteria
here:
https://fedoraproject.org/wiki/Fedora_Release_Criteria
https://fedoraproject.org/wiki/Fedora_13_Alpha_Release_Criteria
https://fedoraproject.org/wiki/Fedora_13_Beta_Release_Criteria
https://fedoraproject.org/wiki/Fedora_13_Final_Release_Criteria
they should contain everything you need to know. We based most of the
criteria around testing that was already being carried out but with no
formal policy basis, with additional suggestions from the anaconda and
desktop teams.
We will follow these criteria for the Fedora 13 release process. So if you
can see any problems or potential trouble with any of this, please do reply
and let us know!
Desktop team - can you please let us know of any additional things that you
would expect to be working at each point during the release cycle? Note
that only things that *must* be working at each point should be listed on
these pages, not nice-to-haves. You must be able to commit to the idea
that, if any criterion on the page is not met, we would slip the release in
question.
--
Adam Williamson
Fedora QA Community Monkey
IRC: adamw | Fedora Talk: adamwill AT fedoraproject DOT org
http://www.happyassassin.net
7:38 p.m.
On Tue, Dec 8, 2009 at 3:55 AM, Adam Williamson <awilliam(a)redhat.com> wrote:
https://fedoraproject.org/wiki/Fedora_Release_Criteria
https://fedoraproject.org/wiki/Fedora_13_Alpha_Release_Criteria
https://fedoraproject.org/wiki/Fedora_13_Beta_Release_Criteria
My comments:
"10. # The installer must be able to successfully complete an upgrade
installation from a clean, fully updated default installation of the
previous stable Fedora release, either via preupgrade or by booting to
the installer manually "
It should be from at least ten other rawhide stages too.
Why is it so weak?
Criteria 11 of beta should figure in alpha. The rescue feature should
be absolutely robust.
Criteria 15 can be more specific. Which desktops?.. all of Kde, Xfce and Gnome
Software Compilation from source and important rpm-src packages should
be ok at beta stage.
Best
A. Mani
--
A. Mani
ASL, CLC, AMS, CMS
http://www.logicamani.co.cc
2:55 p.m.
On Tue, 2009-12-08 at 06:38 +0500, A. Mani wrote:
On Tue, Dec 8, 2009 at 3:55 AM, Adam Williamson
<awilliam(a)redhat.com> wrote:
>
> https://fedoraproject.org/wiki/Fedora_Release_Criteria
> https://fedoraproject.org/wiki/Fedora_13_Alpha_Release_Criteria
> https://fedoraproject.org/wiki/Fedora_13_Beta_Release_Criteria
My comments:
"10. # The installer must be able to successfully complete an upgrade
installation from a clean, fully updated default installation of the
previous stable Fedora release, either via preupgrade or by booting to
the installer manually "
It should be from at least ten other rawhide stages too.
Why is it so weak?
That's all we can commit to test at present. Also, upgrading from
Rawhide is something we care far less about; after all, all Rawhide
installations should be disposable, so one not being upgradeable really
isn't a big deal.
Criteria 11 of beta should figure in alpha. The rescue feature
should
be absolutely robust.
That's a possibility - James, John, what do you guys think of this?
Anyone else?
Criteria 15 can be more specific. Which desktops?.. all of Kde, Xfce
and Gnome
It specifically says 'default desktop' and that is all it's intended to
cover. See John Poelstra's post to -devel-list about the difficulties of
covering multiple desktops in the criteria.
Software Compilation from source and important rpm-src packages
should
be ok at beta stage.
that sounds like a good idea too - again, any other opinions on this?
--
Adam Williamson
Fedora QA Community Monkey
IRC: adamw | Fedora Talk: adamwill AT fedoraproject DOT org
http://www.happyassassin.net
6:48 a.m.
On Wed, 2009-12-09 at 12:55 -0800, Adam Williamson wrote:
On Tue, 2009-12-08 at 06:38 +0500, A. Mani wrote:
> On Tue, Dec 8, 2009 at 3:55 AM, Adam Williamson <awilliam(a)redhat.com> wrote:
> >
> > https://fedoraproject.org/wiki/Fedora_Release_Criteria
> > https://fedoraproject.org/wiki/Fedora_13_Alpha_Release_Criteria
> > https://fedoraproject.org/wiki/Fedora_13_Beta_Release_Criteria
>
> My comments:
>
> "10. # The installer must be able to successfully complete an upgrade
> installation from a clean, fully updated default installation of the
> previous stable Fedora release, either via preupgrade or by booting to
> the installer manually "
>
> It should be from at least ten other rawhide stages too.
> Why is it so weak?
That's all we can commit to test at present. Also, upgrading from
Rawhide is something we care far less about; after all, all Rawhide
installations should be disposable, so one not being upgradeable really
isn't a big deal.
> Criteria 11 of beta should figure in alpha. The rescue feature should
> be absolutely robust.
That's a possibility - James, John, what do you guys think of this?
Anyone else?
Good catch, I don't have any objections. I've moved the rescue mode
criteria to the previous milestone. So for Alpha, rescue mode will
detect and mount default installations. For Beta, rescue mode will
detect and mount more complex installation scenarios.
> Criteria 15 can be more specific. Which desktops?.. all of Kde,
Xfce and Gnome
It specifically says 'default desktop' and that is all it's intended to
cover. See John Poelstra's post to -devel-list about the difficulties of
covering multiple desktops in the criteria.
+1 to John's thread ... we'll tackle this in a future update.
> Software Compilation from source and important rpm-src packages
should
> be ok at beta stage.
that sounds like a good idea too - again, any other opinions on this?
Hmm, no strong opinions from me. If such a criteria existed, what would
be the verification steps to ensure it is upheld?
2:16 a.m.
On 12/07/2009 10:55 PM, Adam Williamson wrote:
During FUDCon, we've been working on revising the Fedora release
criteria.
John Poelstra had already fleshed out a structure and much of the final
content, and we've been revising and tweaking it in conjunction with QA
(myself, Will Woods and James Laska), release engineering (Jesse Keating),
anaconda team (especially Denise Dumas and Peter Jones) and desktop team
(Christopher Aillon and Matthias Clasen, who provided suggestions at an
earlier stage).
Is this discussion available somewhere online?
The new structure is based around a general page and specific pages
for the
Fedora 13 Alpha, Beta and Final releases (which have been written
generically so they can easily be converted into pages for F14 and all
future releases just by copying and pasting). You can find the criteria
here:
https://fedoraproject.org/wiki/Fedora_Release_Criteria
https://fedoraproject.org/wiki/Fedora_13_Alpha_Release_Criteria
https://fedoraproject.org/wiki/Fedora_13_Beta_Release_Criteria
Category 10 in the Beta Release Criteria.
"The installer must be able to successfully complete an upgrade
installation from a clean, fully updated default installation of the
previous stable Fedora release, either via preupgrade or by booting to
the installer manually"
I think we first need to get established if Fedora officially "supports"
upgrades now. If it does we need create and add several upgrade test
cases and to do so we need to know what "default installation" is. What
packages that installation contains and since Releng needs to provide us
with that list it would be good if they document and explain at the same
time how and why they choose to make that selection the default. (
Default dvd install from my perspective should just be secure base no
auto selection for the end user in the installer. )
https://fedoraproject.org/wiki/Fedora_13_Final_Release_Criteria
they should contain everything you need to know. We based most of the
criteria around testing that was already being carried out but with no
formal policy basis, with additional suggestions from the anaconda and
desktop teams.
Where there representatives from all the *DE GNOME KDE, XFCE, LXDE SUGAR
MOBLIN ( I would not be surprise if I'm forgetting some ) present and
chimed in on this? What where these suggestion that Christopher and
Matthias and others made encase any one wants to chime in and provide
additional feedback?
Desktop team - can you please let us know of any additional things
that you
would expect to be working at each point during the release cycle? Note
that only things that *must* be working at each point should be listed on
these pages, not nice-to-haves. You must be able to commit to the idea
that, if any criterion on the page is not met, we would slip the release in
question.
Was this sent to all the *DE lists? ( Noticed that only Gnome was cc on
this ).
JBG
7:21 a.m.
On Tue, 2009-12-08 at 08:16 +0000, "Jóhann B. Guðmundsson" wrote:
Where there representatives from all the *DE GNOME KDE, XFCE, LXDE SUGAR
MOBLIN ( I would not be surprise if I'm forgetting some ) present and
chimed in on this? What where these suggestion that Christopher and
Matthias and others made encase any one wants to chime in and provide
additional feedback?
My comments were all on this list, so you should have no difficulty
finding them.
8 a.m.
On 12/08/2009 01:21 PM, Matthias Clasen wrote:
On Tue, 2009-12-08 at 08:16 +0000, "Jóhann B. Guðmundsson"
wrote:
> Where there representatives from all the *DE GNOME KDE, XFCE, LXDE SUGAR
> MOBLIN ( I would not be surprise if I'm forgetting some ) present and
> chimed in on this? What where these suggestion that Christopher and
> Matthias and others made encase any one wants to chime in and provide
> additional feedback?
>
My comments were all on this list, so you should have no difficulty
finding them.
Interesting reply however I'm only interested in the ones that they took
as your suggestions. As soon as I know that I can look through your
comments on the list to find them :)
If you are unaware of which one(s) it was it's perhaps best to get Adam,
Will or James to chime that into this thread.
JBG
9:27 a.m.
On Tue, 2009-12-08 at 14:00 +0000, "Jóhann B. Guðmundsson" wrote:
Interesting reply however I'm only interested in the ones that
they took
as your suggestions. As soon as I know that I can look through your
comments on the list to find them :)
John actually added Matthias' suggestions to the page in bullet point
form before FUDCon happened; we only fleshed them out into better
language during FUDCon. If you look at the state of the pages from
_before_ FUDCon in the Wiki history, you should be able to see quite
easily the quick-n-dirty list of Matthias' points :)
--
Adam Williamson
Fedora QA Community Monkey
IRC: adamw | Fedora Talk: adamwill AT fedoraproject DOT org
http://www.happyassassin.net
9:26 a.m.
On Tue, 2009-12-08 at 08:16 +0000, "Jóhann B. Guðmundsson" wrote:
On 12/07/2009 10:55 PM, Adam Williamson wrote:
> During FUDCon, we've been working on revising the Fedora release criteria.
> John Poelstra had already fleshed out a structure and much of the final
> content, and we've been revising and tweaking it in conjunction with QA
> (myself, Will Woods and James Laska), release engineering (Jesse Keating),
> anaconda team (especially Denise Dumas and Peter Jones) and desktop team
> (Christopher Aillon and Matthias Clasen, who provided suggestions at an
> earlier stage).
>
Is this discussion available somewhere online?
It was a hackfest, not a presentation, so it wasn't recorded - we were
moving around and talking to different people and things, it took a day
and a half, so it wouldn't have been practical. Remember it was
discussed for quite a while before FUDCon too.
> The new structure is based around a general page and specific
pages for the
> Fedora 13 Alpha, Beta and Final releases (which have been written
> generically so they can easily be converted into pages for F14 and all
> future releases just by copying and pasting). You can find the criteria
> here:
>
> https://fedoraproject.org/wiki/Fedora_Release_Criteria
> https://fedoraproject.org/wiki/Fedora_13_Alpha_Release_Criteria
> https://fedoraproject.org/wiki/Fedora_13_Beta_Release_Criteria
>
Category 10 in the Beta Release Criteria.
"The installer must be able to successfully complete an upgrade
installation from a clean, fully updated default installation of the
previous stable Fedora release, either via preupgrade or by booting to
the installer manually"
I think we first need to get established if Fedora officially "supports"
upgrades now. If it does we need create and add several upgrade test
cases and to do so we need to know what "default installation" is. What
packages that installation contains and since Releng needs to provide us
with that list it would be good if they document and explain at the same
time how and why they choose to make that selection the default. (
Default dvd install from my perspective should just be secure base no
auto selection for the end user in the installer. )
'Supports' is kind of a vague term and not really useful. We don't
'support' anything in the sense of guaranteeing that it'll work - it's
not like we give you a refund if your system breaks :) So I try to avoid
the word when talking formally about this sort of thing.
What it basically means is that at beta stage, we test that if you just
install the previous release into a VM with all default options, update
it, and then run an upgrade, it works. This isn't the same as
'supporting' upgrades, because it only tests a single very simple case.
It's more of a basic check that the fundamental bits of upgrade code do
their job, and there's no really huge bug that everyone who tries to
upgrade is going to hit. We're not committing to testing and supporting
every conceivable combination of packages for upgrading.
This isn't really a change - we've always been doing this test anyway,
it's part of the installation test matrix, and if it had failed, we
would have considered that a blocker bug. A lot of what this page does
is just properly document and justify the tests that we've already been
running in QA, with no official policy document to define what tests we
_should_ run.
Where there representatives from all the *DE GNOME KDE, XFCE, LXDE
SUGAR
MOBLIN ( I would not be surprise if I'm forgetting some ) present and
chimed in on this? What where these suggestion that Christopher and
Matthias and others made encase any one wants to chime in and provide
additional feedback?
The initial session was mostly poelcat, me and jlaska working the tests
we have into the policy and cleaning up the high-level explanation
stuff. At that point it had all the anaconda stuff in it, plus a few
desktop things Matthias had suggested before FUDCon. We then took it to
the anaconda team for their feedback, and to Christopher Aillon for a
desktop team perspective, and added their suggestions. There were KDE
people at FUDCon and the criteria hackfest was announced so they should
have known about it, but they didn't come to offer suggestions and I
didn't run into any of them while I was working on it to ask for their
suggestions. Adam Miller was around and knows about this stuff from an
XFCE position, but again he hasn't given anything the XFCE team would
like to list. Moblin and Sugar are different as they don't get released
as part of the Fedora release, so they probably don't go into these
criteria.
You can see the suggestions made by Chris and Matthias pretty easily:
all the desktop-related criteria in the pages come from them. All the
stuff about what has to work in the 'default desktop' (so currently
GNOME) is feedback from them.
> Desktop team - can you please let us know of any additional
things that you
> would expect to be working at each point during the release cycle? Note
> that only things that *must* be working at each point should be listed on
> these pages, not nice-to-haves. You must be able to commit to the idea
> that, if any criterion on the page is not met, we would slip the release in
> question.
>
Was this sent to all the *DE lists? ( Noticed that only Gnome was cc on
this ).
Unfortunately not yet, just because I'm not subscribed to them and I was
sending that message from sucktastic FUDCon wireless so it would have
been pain to subscribe to them to send it :) I will definitely get in
touch with each spin SIG to see what they want to feed back into the
criteria, though. The pages as they currently exist aren't set in stone,
though there is an issue about the relative importance of the spins
which we really can't answer in a discussion simply about the release
criteria, because it's a much bigger issue than that.
--
Adam Williamson
Fedora QA Community Monkey
IRC: adamw | Fedora Talk: adamwill AT fedoraproject DOT org
http://www.happyassassin.net
9:53 a.m.
On Mon, 2009-12-07 at 14:55 -0800, Adam Williamson wrote:
During FUDCon, we've been working on revising the Fedora release
criteria.
John Poelstra had already fleshed out a structure and much of the final
content, and we've been revising and tweaking it in conjunction with QA
(myself, Will Woods and James Laska), release engineering (Jesse Keating),
anaconda team (especially Denise Dumas and Peter Jones) and desktop team
(Christopher Aillon and Matthias Clasen, who provided suggestions at an
earlier stage).
The new structure is based around a general page and specific pages for the
Fedora 13 Alpha, Beta and Final releases (which have been written
generically so they can easily be converted into pages for F14 and all
future releases just by copying and pasting). You can find the criteria
here:
https://fedoraproject.org/wiki/Fedora_Release_Criteria
https://fedoraproject.org/wiki/Fedora_13_Alpha_Release_Criteria
https://fedoraproject.org/wiki/Fedora_13_Beta_Release_Criteria
https://fedoraproject.org/wiki/Fedora_13_Final_Release_Criteria
they should contain everything you need to know. We based most of the
criteria around testing that was already being carried out but with no
formal policy basis, with additional suggestions from the anaconda and
desktop teams.
We will follow these criteria for the Fedora 13 release process. So if you
can see any problems or potential trouble with any of this, please do reply
and let us know!
Desktop team - can you please let us know of any additional things that you
would expect to be working at each point during the release cycle? Note
that only things that *must* be working at each point should be listed on
these pages, not nice-to-haves. You must be able to commit to the idea
that, if any criterion on the page is not met, we would slip the release in
question.
Not sure if this has been raised yet, but are we specifying when in the
release that packages should be signed with a valid signature? I
believe packages are signed at all release milestones, but I'd like to
clear up that assumption.
Thanks,
James
10:10 a.m.
On Fri, Dec 11, 2009 at 10:53:40 -0500,
James Laska <jlaska(a)redhat.com> wrote:
Not sure if this has been raised yet, but are we specifying when in the
release that packages should be signed with a valid signature? I
believe packages are signed at all release milestones, but I'd like to
clear up that assumption.
I belive the plan is that all official koji builds are going to be signed
with the same key. The key will just provide assurance that the rpms were
official builds from our koji and not that they are tied to a particular
release or release type.
10:20 a.m.
On Fri, 2009-12-11 at 10:53 -0500, James Laska wrote:
Not sure if this has been raised yet, but are we specifying when in
the
release that packages should be signed with a valid signature? I
believe packages are signed at all release milestones, but I'd like to
clear up that assumption.
Do you think that's a criteria issue, i.e. something to which there's an
innate correct answer which can be defined and which shouldn't change?
I'd think of it more as a process issue, but IMBW.
--
Adam Williamson
Fedora QA Community Monkey
IRC: adamw | Fedora Talk: adamwill AT fedoraproject DOT org
http://www.happyassassin.net
10:52 a.m.
On Fri, 2009-12-11 at 08:20 -0800, Adam Williamson wrote:
On Fri, 2009-12-11 at 10:53 -0500, James Laska wrote:
> Not sure if this has been raised yet, but are we specifying when in the
> release that packages should be signed with a valid signature? I
> believe packages are signed at all release milestones, but I'd like to
> clear up that assumption.
Do you think that's a criteria issue, i.e. something to which there's an
innate correct answer which can be defined and which shouldn't change?
I'd think of it more as a process issue, but IMBW.
Yeah, that's my question ... is there an assumption that all packages
will be signed? Does this assumption need to be validated?
Looking at our current test plans for the release, I don't see anything
where we confirm that packages are properly signed. Should we be
testing this, and if so ... does it map back to a specific release
criteria?
11:49 a.m.
On 12/11/2009 08:52 AM, James Laska wrote:
On Fri, 2009-12-11 at 08:20 -0800, Adam Williamson wrote:
> On Fri, 2009-12-11 at 10:53 -0500, James Laska wrote:
>
>> Not sure if this has been raised yet, but are we specifying when in the
>> release that packages should be signed with a valid signature? I
>> believe packages are signed at all release milestones, but I'd like to
>> clear up that assumption.
>
> Do you think that's a criteria issue, i.e. something to which there's an
> innate correct answer which can be defined and which shouldn't change?
> I'd think of it more as a process issue, but IMBW.
Yeah, that's my question ... is there an assumption that all packages
will be signed? Does this assumption need to be validated?
Looking at our current test plans for the release, I don't see anything
where we confirm that packages are properly signed. Should we be
testing this, and if so ... does it map back to a specific release
criteria?
The way we've approached the Release Criteria is that we are only
capturing things that *must* be present to ship.
If the answer to the question: "If for whatever reason, packages were
not signed with the correct key (or at all), would we delay shipping the
release until they were signed correctly?"... is "yes" then I think it
should be added to the criteria.
I would propose the answer should be "yes, all packages must be properly
signed." I'm sure people will let me know if they disagree ;-)
John
5248
days inactive
5252
days old
13 comments
7 participants
participants (7)
-
"Jóhann B. Guðmundsson" -
A. Mani -
Adam Williamson -
Bruno Wolff III -
James Laska -
John Poelstra -
Matthias Clasen