Re: QA process was Re: RPM submission procedure

[Timothy John Giese]

I'm brand new to the whole process so I thought it might be useful for 
me to chime in from my perspective regarding the QA process.

I like checklists.  It gives me a good idea of what the expectations 
are.  I followed the instructions at 
http://www.fedora.us/wiki/PackageSubmissionQAPolicy without difficulty.  
Granted, I do not aspire to package hundreds of packages, so I find the 
procedure to not be an obstacle.

Again, I find the checklists useful in reviewing submitted packages.  I 
have followed the instructions from 
http://www.fedora.us/wiki/QAChecklist and used them as a basis for 
reviewing other people's packages.

On the other hand, I feel deterred from reviewing packages.  I am new, 
so I am not a "trusted" developer.  In order for a package to be 
published, it must be approved by 1 trusted developer or 2 untrusted 
developers.  Given the large number of packages and the relatively few 
number of reviewers, I feel that my review provides little help in 
speeding the overall publication of a package (because I suspect other 
untrusted developers are also deterred).  Instead, I feel like any 
review I provide will be available to a *trusted* developer whenever the 
trusted developer gets the time to review the package and will provide 
some assistance to them.... but in terms of the time it takes to publish 
the package, it most likely will not speed up the process significatly 
(the rate-limiting-step is probably the speed at which a trusted 
developer reviews the package). 

An obvious solution would be to have untrusted developers dilligently 
review packages and thus bypass the trusted devlopers; however, I don't 
know what would act to catalyze the motivation for such a response (the 
motivation being: feeling as though the contribution leads to a 
significant speeding of the process).

I believe that the QA submission procedure may change when fedora.us 
becomes fedora extras, so it may be a rash to suggest changes to the 
current submission procedure.  I can only hope that any procedure used 
upon transition of the infrastructure will attempt to address this concern.

Cheers,
Tim Giese

QA process was Re: RPM submission procedure

[Michael Schwendt]

On Thu, 08 Jan 2004 19:56:40 -0600, Timothy John Giese wrote:

>  On the other hand, I feel deterred from reviewing packages.  I am
> new, 
>  so I am not a "trusted" developer.  In order for a package to be 
>  published, it must be approved by 1 trusted developer or 2 untrusted 
>  developers.  
I think it's "OR": 2 untrusted developers OR 1 trusted developer.

>  Given the large number of packages and the relatively few 
>  number of reviewers, I feel that my review provides little help in 
>  speeding the overall publication of a package (because I suspect other 
>  untrusted developers are also deterred).  
We're missing a keyword (or a marker in the summary field) that flags
package request tickets which have at least one publish vote. Right now,
your best bet is to sort by "last changed" and visit recently changed
tickets.

--

QA process was Re: RPM submission procedure

[Panu Matilainen]

On Fri, 9 Jan 2004, Michael Schwendt wrote:

>  On Thu, 08 Jan 2004 19:56:40 -0600, Timothy John Giese wrote:
>  
>  > On the other hand, I feel deterred from reviewing packages.  I am new, 
>  > so I am not a "trusted" developer.  In order for a package to be 
>  > published, it must be approved by 1 trusted developer or 2 untrusted  
        
                                                ^^^^
>  > developers. 
>  
>  I think it's "OR": 2 untrusted developers OR 1 trusted developer. 
That's what Timothy said :)

>  
>  > Given the large number of packages and the relatively few 
>  > number of reviewers, I feel that my review provides little help in 
>  > speeding the overall publication of a package (because I suspect other 
>  > untrusted developers are also deterred). 
>  
>  We're missing a keyword (or a marker in the summary field) that flags
>  package request tickets which have at least one publish vote. Right now,
>  your best bet is to sort by "last changed" and visit recently changed
>  tickets. 
I don't think bugzilla keywords and such are going to help. For any given 
package there are only so many people who a) are interested in it b) are 
willing and capable of doing QA on it. The fact that you need two such 
users outside of "trusted developers" to get the package *anywhere* 
doesn't do much to encourage people to do QA because it feels like totally 
wasted effort since the package isn't still moving. 

I've been lobbying lowering the entry bar to testing/unstable for a while
now... Stable repository should IMHO remain basically as it is, eg you
can't get your package there until it's seen a full QA review under
current rules. However allowing just one "untrusted" developer QA review
to get package into testing or unstable would at least give people the
feeling that their effort is valuable and actually helps. I've also 
suggested various other schemes to lower the bar of getting 
package into testing/unstable in the "source code audit proposal" thread 
in fedora.us devel list - a thread which unfortunately just died and 
absolutely nothing came out of all the ideas thrown around. Sure, we don't 
want to implement any major systems for fedora.us right now because of the 
looming official Extras merge but some minor tweaks to the policies would 
help, I believe.

	- Panu -

QA process was Re: RPM submission procedure

[Toshio Kuratomi]

On Fri, 2004-01-09 at 04:54, Panu Matilainen wrote:
>  On Fri, 9 Jan 2004, Michael Schwendt wrote:
>  
>  > On Thu, 08 Jan 2004 19:56:40 -0600, Timothy John Giese wrote:
>  > We're missing a keyword (or a marker in the summary field) that flags
>  > package request tickets which have at least one publish vote. Right now,
>  > your best bet is to sort by "last changed" and visit recently changed
>  > tickets.
>  
>  I don't think bugzilla keywords and such are going to help. For any given 
>  package there are only so many people who a) are interested in it b) are 
>  willing and capable of doing QA on it. The fact that you need two such 
>  users outside of "trusted developers" to get the package *anywhere* 
>  doesn't do much to encourage people to do QA because it feels like totally 
>  wasted effort since the package isn't still moving. 
>   
As someone who has just started doing QA and hasn't gotten jaded yet
*grin*, I think a keyword would be helpful.  I see many packages that I
don't know anything about except the name in the queue but think I might
like to try.  If I knew that one person had already asserted that it was
publishable, I'd be more confident in spending a little time trying it
out.

One problem I have with the current keyword situation, though, is that
the package queue bugzilla does not show them.  I've clicked through
numerous packages only to find they have NEEDSWORK set.  I've set up a
query to screen this, but I think that one of the uses of a
"HALF-PUBLISH" type keyword is to let people new to QA get a feel for
what good packages should look like by adding their knowledge to someone
else's QA knowledge.  For that audience, I don't think we should assume
a familiarity with bugzilla, running queries, etc.

-Toshio
-- 
Toshio <toshio(a)tiki-lounge.com&gt;

QA process was Re: RPM submission procedure

[Michael Schwendt]

On Fri, 9 Jan 2004 11:54:51 +0200 (EET), Panu Matilainen wrote:

>  > On Thu, 08 Jan 2004 19:56:40 -0600, Timothy John Giese wrote:
>  > 
>  > > On the other hand, I feel deterred from reviewing packages.  I am new, 
>  > > so I am not a "trusted" developer.  In order for a package to be 
>  > > published, it must be approved by 1 trusted developer or 2 untrusted 
>                                                           ^^^^
>  > > developers. 
>  > 
>  > I think it's "OR": 2 untrusted developers OR 1 trusted developer.
>  
>  That's what Timothy said :) 
*Ooops* :)

>  The fact that you need two such 
>  users outside of "trusted developers" to get the package *anywhere* 
>  doesn't do much to encourage people to do QA because it feels like totally 
>  wasted effort since the package isn't still moving.  
But if the first reviewer has done most of the work already, a second
review can be like a piece of cake. Or is everyone waiting for someone
else to be the early bird?

With regard the notion of "untrusted/trusted developers", I favour the
concept of giving "untrusted developers" the chance to take responsibility
early by relying on their review. That means, if there's a package request
ticket with a PUBLISH vote, I assume the stuff has been reviewed actually,
so let's apply some of the mandatory sanity checks only, skim over the
rest and get it published, taking the first "untrusted" review into
account.  Heck, if that first review was half-hearted, there's still the
PENDING step, and the packager shouldn't "sleep" either and be somewhat
familiar with a package, too, before it is approved.

*Sometimes* the packager does most of the QA already, anyway.

>  I've been lobbying lowering the entry bar to testing/unstable for
> a while
>  now... Stable repository should IMHO remain basically as it is, eg you
>  can't get your package there until it's seen a full QA review under
>  current rules. However allowing just one "untrusted" developer QA review
>  to get package into testing or unstable would at least give people the
>  feeling that their effort is valuable and actually helps.  
Bears the problem that people would be entirely satisfied with getting
their packages published in testing/unstable easily and don't care to get
them into "stable". It shouldn't end like RHContrib, but it would be
worth a try: +1

--

QA process was Re: RPM submission procedure

[Panu Matilainen]

On Fri, 9 Jan 2004, Michael Schwendt wrote:
>  On Fri, 9 Jan 2004 11:54:51 +0200 (EET), Panu Matilainen wrote:
>  
>  > The fact that you need two such 
>  > users outside of "trusted developers" to get the package *anywhere* 
>  > doesn't do much to encourage people to do QA because it feels like totally 
>  > wasted effort since the package isn't still moving. 
>  
>  But if the first reviewer has done most of the work already, a second
>  review can be like a piece of cake. Or is everyone waiting for someone
>  else to be the early bird? 
Sure it's a piece of cake, but *somebody* has to do it. If you look at the 
average package in fedora.us QA queue there are *very* esoteric packages 
in there.. in many cases you'll be lucky to find just that one person who 
understands/is interested enough to learn what the package is about *and* 
willing to do the few steps of QA review.

>  
>  With regard the notion of "untrusted/trusted developers", I favour the
>  concept of giving "untrusted developers" the chance to take responsibility
>  early by relying on their review. That means, if there's a package request
>  ticket with a PUBLISH vote, I assume the stuff has been reviewed actually,
>  so let's apply some of the mandatory sanity checks only, skim over the
>  rest and get it published, taking the first "untrusted" review into
>  account.  Heck, if that first review was half-hearted, there's still the
>  PENDING step, and the packager shouldn't "sleep" either and be somewhat
>  familiar with a package, too, before it is approved. 
I agree - but the way it's expressed in our current policies doesn't come 
out sounding like that. At least to me, the policy sounds basically like 
"you have to do the same checks the other guy did" meaning pretty 
redundant and boring work somebody already did... So if others agree, we 
should at least rephrase the "publish vote from two untrusted developers" 
to something which expresses what you say above better than the current 
policy.

>  
>  *Sometimes* the packager does most of the QA already, anyway.
>  
>  > I've been lobbying lowering the entry bar to testing/unstable for a while
>  > now... Stable repository should IMHO remain basically as it is, eg you
>  > can't get your package there until it's seen a full QA review under
>  > current rules. However allowing just one "untrusted" developer QA review
>  > to get package into testing or unstable would at least give people the
>  > feeling that their effort is valuable and actually helps. 
>  
>  Bears the problem that people would be entirely satisfied with getting
>  their packages published in testing/unstable easily and don't care to get
>  them into "stable". It shouldn't end like RHContrib, but it would be
>  worth a try: +1 
Oh, I certainly don't want another RH contrib. I want to *lower* the bar
to get packages out in the field, not remove it :) People are not going to
be satisfied by just getting their packages into testing/unstable but it's
a *start* - it's much easier to get users to test something they don't
have to build themselves. How the actual move happens from testing to
stable.. I don't know really, certainly it shouldn't be of the form "if
nobody complains in 10 days move it a level upwards". We've had the
discussion a few times, always non-conclusive, as to how to actually use
the stable/testing/unstable structure beyond what the packager originally
asks for. Just an off-the-cuff idea: if one trusted, or two untrusted
developers vote a package ready to move to "upwards" it should be moved ?  
(assuming that one PUBLISH vote from untrusted developer is enough to get
a package into testing/unstable)

I would really, really like to see this discussed to whatever depth it 
takes and the outcome actually taken into action...

	- Panu -

QA process was Re: RPM submission procedure

[Michael Schwendt]

On Fri, 9 Jan 2004 18:13:13 +0200 (EET), Panu Matilainen wrote:

>  On Fri, 9 Jan 2004, Michael Schwendt wrote:
>  > On Fri, 9 Jan 2004 11:54:51 +0200 (EET), Panu Matilainen wrote:
>  > 
>  > > The fact that you need two such 
>  > > users outside of "trusted developers" to get the package *anywhere* 
>  > > doesn't do much to encourage people to do QA because it feels like totally
>
>  > > wasted effort since the package isn't still moving. 
>  > 
>  > But if the first reviewer has done most of the work already, a second
>  > review can be like a piece of cake. Or is everyone waiting for someone
>  > else to be the early bird?
>  
>  Sure it's a piece of cake, but *somebody* has to do it. If you look at the 
>  average package in fedora.us QA queue there are *very* esoteric packages 
>  in there.. in many cases you'll be lucky to find just that one person who 
>  understands/is interested enough to learn what the package is about *and* 
>  willing to do the few steps of QA review. 
And there are no two users who can build and test-drive such packages?

I mean the packager does the low-level stuff. And the more "esoteric" a
package is, the more familiar he ought to be with it, shouldn't he?  It
would just need an "okay" from two users that the built software works and
that the included sources have been verified with any available
signatures, trusted sources, via diffs, with confirmation from upstream
authors or other ways. We should not assume that every packager's work
needs to be verified at a low level. Actually some packagers get it right
from the early beginning. Other even stick very close to the spec
template. Whether the software works and whether upgrades don't result in
regression, is much more important. Such QA can be and should be performed
by actual users of a given piece of software. Let packagers and users
team up. There are no users? Ugh. Then who cares? Let the package
rest in peace until someone steps up who's interested in it.

>  > With regard the notion of "untrusted/trusted
> developers", I favour the
>  > concept of giving "untrusted developers" the chance to take
> responsibility
>  > early by relying on their review. That means, if there's a package request
>  > ticket with a PUBLISH vote, I assume the stuff has been reviewed actually,
>  > so let's apply some of the mandatory sanity checks only, skim over the
>  > rest and get it published, taking the first "untrusted" review into
>  > account.  Heck, if that first review was half-hearted, there's still the
>  > PENDING step, and the packager shouldn't "sleep" either and be
> somewhat
>  > familiar with a package, too, before it is approved.
>  
>  I agree - but the way it's expressed in our current policies doesn't come 
>  out sounding like that. At least to me, the policy sounds basically like 
>  "you have to do the same checks the other guy did" meaning pretty 
>  redundant and boring work somebody already did... So if others agree, we 
>  should at least rephrase the "publish vote from two untrusted developers" 
>  to something which expresses what you say above better than the current 
>  policy. 
But we don't have a complete and accurate list of what must be included in
a package approval, other than MD5 checksums of source tarballs and
src.rpm, e.g. as mentioned here
http://www.fedora.us/wiki/PackageSubmissionQAPolicy 

So how do you know what exactly a reviewer has reviewed? He could post
checksums, vote for publish and be done. Sure, the QA checklist is
available. But it's incomplete and includes steps not needed for many
packages. Turns out it really only needs two people to approve a
package. If no two people have interest in a package, can we expect that
there would be any users of a published binary package? Would they submit
bug reports or move on to another source of prebuilt package in case of
problems?

>  > *Sometimes* the packager does most of the QA already, anyway.
>  > 
>  > > I've been lobbying lowering the entry bar to testing/unstable for a while
>  > > now... Stable repository should IMHO remain basically as it is, eg you
>  > > can't get your package there until it's seen a full QA review under
>  > > current rules. However allowing just one "untrusted" developer QA
> review
>  > > to get package into testing or unstable would at least give people the
>  > > feeling that their effort is valuable and actually helps. 
>  > 
>  > Bears the problem that people would be entirely satisfied with getting
>  > their packages published in testing/unstable easily and don't care to get
>  > them into "stable". It shouldn't end like RHContrib, but it would be
>  > worth a try: +1
>  
>  Oh, I certainly don't want another RH contrib. I want to *lower* the bar
>  to get packages out in the field, not remove it :) People are not going to
>  be satisfied by just getting their packages into testing/unstable but it's
>  a *start* - it's much easier to get users to test something they don't
>  have to build themselves. 
Hence my old suggestion that packagers offer not just src.rpms but also
prebuilt rpms *if* users show interest in helping to approve them. Whether
you trust such packages or not, is up to you.

The problem I see is that if nobody cares to get a new package published,
he does not care about submitting bug reports either. He rather will be
found bitching in some message board about the non-working packages found
in some repository and offering his own packages from private web space.

>  How the actual move happens from testing to
>  stable.. I don't know really, certainly it shouldn't be of the form "if
>  nobody complains in 10 days move it a level upwards". We've had the
>  discussion a few times, always non-conclusive, as to how to actually use
>  the stable/testing/unstable structure beyond what the packager originally
>  asks for. Just an off-the-cuff idea: if one trusted, or two untrusted
>  developers vote a package ready to move to "upwards" it should be moved ?  
There's no way to avoid a voting based system IMO. Again, if you care for
a "free" distribution, you support it somehow, at least with feedback and
bug reports.

>  (assuming that one PUBLISH vote from untrusted developer is enough to
> get
>  a package into testing/unstable)
>  
>  I would really, really like to see this discussed to whatever depth it 
>  takes and the outcome actually taken into action... 
--

QA process was Re: RPM submission procedure

[Panu Matilainen]

On Fri, 9 Jan 2004, Michael Schwendt wrote:

>  On Fri, 9 Jan 2004 18:13:13 +0200 (EET), Panu Matilainen wrote:
>  
>  > On Fri, 9 Jan 2004, Michael Schwendt wrote:
>  > > On Fri, 9 Jan 2004 11:54:51 +0200 (EET), Panu Matilainen wrote:
>  > > 
>  > > > The fact that you need two such 
>  > > > users outside of "trusted developers" to get the package
> *anywhere* 
>  > > > doesn't do much to encourage people to do QA because it feels like
> totally 
>  > > > wasted effort since the package isn't still moving. 
>  > > 
>  > > But if the first reviewer has done most of the work already, a second
>  > > review can be like a piece of cake. Or is everyone waiting for someone
>  > > else to be the early bird?
>  > 
>  > Sure it's a piece of cake, but *somebody* has to do it. If you look at the 
>  > average package in fedora.us QA queue there are *very* esoteric packages 
>  > in there.. in many cases you'll be lucky to find just that one person who 
>  > understands/is interested enough to learn what the package is about *and* 
>  > willing to do the few steps of QA review.
>  
>  And there are no two users who can build and test-drive such packages?
>  
>  I mean the packager does the low-level stuff. And the more "esoteric" a
>  package is, the more familiar he ought to be with it, shouldn't he?  It
>  would just need an "okay" from two users that the built software works and
>  that the included sources have been verified with any available
>  signatures, trusted sources, via diffs, with confirmation from upstream
>  authors or other ways. We should not assume that every packager's work
>  needs to be verified at a low level. Actually some packagers get it right
>  from the early beginning. Other even stick very close to the spec
>  template. Whether the software works and whether upgrades don't result in
>  regression, is much more important. Such QA can be and should be performed
>  by actual users of a given piece of software. Let packagers and users
>  team up.  
By all means, but the current fedora.us policies and practises hardly 
*encourage* that. The amount of nitpicking trusted developers produce 
(among themselves) is enough to scare off anybody starting in packaging 
I'm willing to bet :)

>  There are no users? Ugh. Then who cares? Let the package
>  rest in peace until someone steps up who's interested in it. 
Sure.

>  
>  > > With regard the notion of "untrusted/trusted developers", I favour
> the
>  > > concept of giving "untrusted developers" the chance to take
> responsibility
>  > > early by relying on their review. That means, if there's a package request
>  > > ticket with a PUBLISH vote, I assume the stuff has been reviewed actually,
>  > > so let's apply some of the mandatory sanity checks only, skim over the
>  > > rest and get it published, taking the first "untrusted" review into
>  > > account.  Heck, if that first review was half-hearted, there's still the
>  > > PENDING step, and the packager shouldn't "sleep" either and be
> somewhat
>  > > familiar with a package, too, before it is approved.
>  > 
>  > I agree - but the way it's expressed in our current policies doesn't come 
>  > out sounding like that. At least to me, the policy sounds basically like 
>  > "you have to do the same checks the other guy did" meaning pretty 
>  > redundant and boring work somebody already did... So if others agree, we 
>  > should at least rephrase the "publish vote from two untrusted developers"
>
>  > to something which expresses what you say above better than the current 
>  > policy.
>  
>  But we don't have a complete and accurate list of what must be included in
>  a package approval, other than MD5 checksums of source tarballs and
>  src.rpm, e.g. as mentioned here
>  http://www.fedora.us/wiki/PackageSubmissionQAPolicy 
>  
>  So how do you know what exactly a reviewer has reviewed? He could post
>  checksums, vote for publish and be done. Sure, the QA checklist is
>  available. But it's incomplete and includes steps not needed for many
>  packages. Turns out it really only needs two people to approve a
>  package. If no two people have interest in a package, can we expect that
>  there would be any users of a published binary package? Would they submit
>  bug reports or move on to another source of prebuilt package in case of
>  problems? 
Probably not.. or then they would. Do the freshrpms, atrpms, dag, newrpms 
"gang" get reports of faulty packages, without the users actually looking 
inside specs? Sure they do.

>  
>  > > *Sometimes* the packager does most of the QA already, anyway.
>  > > 
>  > > > I've been lobbying lowering the entry bar to testing/unstable for a
> while
>  > > > now... Stable repository should IMHO remain basically as it is, eg you
>  > > > can't get your package there until it's seen a full QA review
> under
>  > > > current rules. However allowing just one "untrusted" developer
> QA review
>  > > > to get package into testing or unstable would at least give people the
>  > > > feeling that their effort is valuable and actually helps. 
>  > > 
>  > > Bears the problem that people would be entirely satisfied with getting
>  > > their packages published in testing/unstable easily and don't care to get
>  > > them into "stable". It shouldn't end like RHContrib, but it would
> be
>  > > worth a try: +1
>  > 
>  > Oh, I certainly don't want another RH contrib. I want to *lower* the bar
>  > to get packages out in the field, not remove it :) People are not going to
>  > be satisfied by just getting their packages into testing/unstable but it's
>  > a *start* - it's much easier to get users to test something they don't
>  > have to build themselves.
>  
>  Hence my old suggestion that packagers offer not just src.rpms but also
>  prebuilt rpms *if* users show interest in helping to approve them. Whether
>  you trust such packages or not, is up to you. 
That's what I do personally for the more popular packages like apt. The
problem with that is that it requires some place to put the packages into;
not everybody is willing or can afford to pay for the space and bandwidth
for the purpose. You could just as well have a common repository (be it 
RPMS.unverified-use-at-your-own-risk or whatever) where each package is 
signed by the packagers GPG-key - if you choose to trust some packager
you import his/her GPG key and don't worry about getting whatever cruft 
out of that repo on your system. 

	- Panu -

QA process was Re: RPM submission procedure

[Michael Schwendt]

On Fri, 9 Jan 2004 20:05:06 +0200 (EET), Panu Matilainen wrote:

>  The amount of nitpicking trusted developers produce 
>  (among themselves) is enough to scare off anybody starting in packaging 
>  I'm willing to bet :) 
This must change, although often it is separated between suggestions and
blocker criteria. But at the same time, new packagers should not come
with slightly modified packages from e.g. Mandrake Cooker which bzip2 even
the smallest patch, or generic packages which contain dozens of lines of
conditional code which tries to adapt to a build environment.

A fundamental problem is "packager mentality". If a packager has the
impression that a QA person is the nitpicking bad guy who's nothing else
than a PITA, then the whole concept of working together on a community
maintained repository is doomed to fail. If on the other hand, the
packager is at least a bit open for suggestions or established common
practise, everything works better.

--

QA process was Re: RPM submission procedure

[Karl DeBisschop]

On Fri, 2004-01-09 at 13:59, Michael Schwendt wrote:
>  On Fri, 9 Jan 2004 20:05:06 +0200 (EET), Panu Matilainen wrote:
>  
>  > The amount of nitpicking trusted developers produce 
>  > (among themselves) is enough to scare off anybody starting in packaging 
>  > I'm willing to bet :)
>  
>  This must change, although often it is separated between suggestions and
>  blocker criteria. But at the same time, new packagers should not come
>  with slightly modified packages from e.g. Mandrake Cooker which bzip2 even
>  the smallest patch, or generic packages which contain dozens of lines of
>  conditional code which tries to adapt to a build environment. 
Is this rejection of generic packages official policy?

Somehow it seems to me contrary to the idea that Fedora stresses
upstream bugfixes.

I think of it this way: as a developer, I cannot possibly package well
for all distros, but I may be able to package well for the one or ones I
use heavily. Nonetheless, as a developer, I have to maintain the code in
single units that is readily packaged by many distros, not just the one
I happen to use.

Do other people see this tension?

I'm not saying that any distros packaging should be compromised by the
packaging of another. But it seems that the idea of pushing work
upstream would favor work with developer/packagers to make releases that
are suitable for direct use as much as possible.

-- 
Karl DeBisschop <kdebisschop(a)alert.infoplease.com&gt;
Pearson Education/Information Please

QA process was Re: RPM submission procedure

[Alan Cox]

On Fri, Jan 09, 2004 at 02:20:16PM -0500, Karl DeBisschop wrote:
>  Somehow it seems to me contrary to the idea that Fedora stresses
>  upstream bugfixes. 
Do you want 35 rival competing updated copies of perl that all
conflict ?

>  Do other people see this tension? 
Not really. Firstly you should be working with the package maintainer,
and secondly if you can't work with them then Fedora has another category
to deal with stuff that might cause conflicts, and lets people be aware
that they may get them.

QA process was Re: RPM submission procedure

[Karl DeBisschop]

On Fri, 2004-01-09 at 16:13, Alan Cox wrote:
>  On Fri, Jan 09, 2004 at 02:20:16PM -0500, Karl DeBisschop wrote:
>  > Somehow it seems to me contrary to the idea that Fedora stresses
>  > upstream bugfixes.
>  
>  Do you want 35 rival competing updated copies of perl that all
>  conflict ? 
Not at all. In fact I don't really see how you read that from what I
wrote, so I must be missing something or I must not have been clear.

What I'm suggesting is that if every distro has entirely separate specs,
you are tending torward these sorts of conflicts. But if the distros are
less insistent that a spec is for their distro alone, then we have less
tendency to diverge.

>  > Do other people see this tension?
>  
>  Not really. Firstly you should be working with the package maintainer,
>  and secondly if you can't work with them then Fedora has another category
>  to deal with stuff that might cause conflicts, and lets people be aware
>  that they may get them. 
Generally, it's hard as a developer to get feedback from packagers. They
tend to do what they want and never feed it back to the developer in my
experience.

And even if you do, it's not tenable to ask a developer to coordinate
independently with packagers from all the distros out there. So what I'm
suggesting is that if the distros work from the idea of a reference spec
that is somewhat distro agnostic, then the delevopr coordinates with the
.deb packaging community, the rpm packaging community, and the ports
community. That way, as packagers contribute thier feedback to the
devlopers, it's more focused on what make the application better.

If packagers from different distros eschew that tpe of cooperation, then
the developer either sees silence or a "do things my way because that's
our policy" from 20 different camps pulling the project in all different
ways.

-- 
Karl DeBisschop <kdebisschop(a)alert.infoplease.com&gt;
Pearson Education/Information Please

QA process was Re: RPM submission procedure

[Michael Schwendt]

On Fri, 09 Jan 2004 16:32:49 -0500, Karl DeBisschop wrote:

>  What I'm suggesting is that if every distro has entirely separate
> specs,
>  you are tending torward these sorts of conflicts. But if the distros are
>  less insistent that a spec is for their distro alone, then we have less
>  tendency to diverge. 
Fine. Get all distributors to agree on common package naming guidelines,
on a common set of RPM macros, on a common set of helper tools, etc. etc.

--

QA process was Re: RPM submission procedure

[Karl DeBisschop]

On Fri, 2004-01-09 at 16:47, Michael Schwendt wrote:
>  On Fri, 09 Jan 2004 16:32:49 -0500, Karl DeBisschop wrote:
>  
>  > What I'm suggesting is that if every distro has entirely separate specs,
>  > you are tending torward these sorts of conflicts. But if the distros are
>  > less insistent that a spec is for their distro alone, then we have less
>  > tendency to diverge.
>  
>  Fine. Get all distributors to agree on common package naming guidelines,
>  on a common set of RPM macros, on a common set of helper tools, etc. etc. 
At this point, as you seem to feel, that is probably not feasible.

But is it not possible to handle this divergence in some way?

For instance, I can easily imagine making several specs from one spec.in
via automake. But AFAIK RPM doesn't have a way to choose between
"nagios-plugins.fedora.spec" and "nagios-plugins.mandrake.spec".

I don't see how the genie can be put back in the bottle. But I do wonder
if we have a chance to stop the genie from getting bigger than
necessary.

-- 
Karl DeBisschop <kdebisschop(a)alert.infoplease.com&gt;
Pearson Education/Information Please

QA process was Re: RPM submission procedure

[Miloslav Trmač]

On Fri, Jan 09, 2004 at 05:29:25PM -0500, Karl DeBisschop wrote:
>  For instance, I can easily imagine making several specs from one
> spec.in
>  via automake. But AFAIK RPM doesn't have a way to choose between
>  "nagios-plugins.fedora.spec" and "nagios-plugins.mandrake.spec".
It doesn't need to. You unpack the package, run
	./configure && make nagios-plugins.spec
and you have what you need. You lose rpmbuild -t*, but that is
no big deal IMHO.
	Mirek

QA process was Re: RPM submission procedure

[Lamar Owen]

On Friday 09 January 2004 05:36 pm, Miloslav Trmac wrote:
>  It doesn't need to. You unpack the package, run
>  	./configure && make nagios-plugins.spec
>  and you have what you need. You lose rpmbuild -t*, but that is
>  no big deal IMHO.
>  	Mirek 
You also lose rpmbuild --rebuild *.src.rpm with a single .src.rpm that works 
everywhere for buildhost convenience.  You can't then distribute a single 
.src.rpm for everything you support.
-- 
Lamar Owen
Director of Information Technology
Pisgah Astronomical Research Institute
1 PARI Drive
Rosman, NC  28772
[phone]
www.pari.edu

QA process was Re: RPM submission procedure

[Miloslav Trmač]

On Fri, Jan 09, 2004 at 05:47:56PM -0500, Lamar Owen wrote:
>  You also lose rpmbuild --rebuild *.src.rpm with a single .src.rpm
> that works 
>  everywhere for buildhost convenience.  You can't then distribute a single 
>  .src.rpm for everything you support. 
I don't want to use a distribution where
all the distribution maker does
is take the upstream src.rpm without ever looking at the contents.

The perspective is different for individual users, but an unified way
to create rpms would still be an improvement over the current unified
way to just install the files.
	Mirek

QA process was Re: RPM submission procedure

[Michael Schwendt]

On Fri, 09 Jan 2004 17:29:25 -0500, Karl DeBisschop wrote:

>  On Fri, 2004-01-09 at 16:47, Michael Schwendt wrote:
>  > On Fri, 09 Jan 2004 16:32:49 -0500, Karl DeBisschop wrote:
>  > 
>  > > What I'm suggesting is that if every distro has entirely separate specs,
>  > > you are tending torward these sorts of conflicts. But if the distros are
>  > > less insistent that a spec is for their distro alone, then we have less
>  > > tendency to diverge.
>  > 
>  > Fine. Get all distributors to agree on common package naming guidelines,
>  > on a common set of RPM macros, on a common set of helper tools, etc. etc.
>  
>  At this point, as you seem to feel, that is probably not feasible.
>  
>  But is it not possible to handle this divergence in some way?
>  
>  For instance, I can easily imagine making several specs from one spec.in
>  via automake. But AFAIK RPM doesn't have a way to choose between
>  "nagios-plugins.fedora.spec" and "nagios-plugins.mandrake.spec".
>  
>  I don't see how the genie can be put back in the bottle. But I do wonder
>  if we have a chance to stop the genie from getting bigger than
>  necessary. 
Why do you want to merge distribution-specific build scripts upstream?

--

QA process was Re: RPM submission procedure

[Dag Wieers]

On Fri, 9 Jan 2004, Karl DeBisschop wrote:

>  On Fri, 2004-01-09 at 16:47, Michael Schwendt wrote:
>  > On Fri, 09 Jan 2004 16:32:49 -0500, Karl DeBisschop wrote:
>  > 
>  > > What I'm suggesting is that if every distro has entirely separate specs,
>  > > you are tending torward these sorts of conflicts. But if the distros are
>  > > less insistent that a spec is for their distro alone, then we have less
>  > > tendency to diverge.
>  > 
>  > Fine. Get all distributors to agree on common package naming guidelines,
>  > on a common set of RPM macros, on a common set of helper tools, etc. etc.
>  
>  At this point, as you seem to feel, that is probably not feasible.
>  
>  But is it not possible to handle this divergence in some way?
>  
>  For instance, I can easily imagine making several specs from one spec.in
>  via automake. But AFAIK RPM doesn't have a way to choose between
>  "nagios-plugins.fedora.spec" and "nagios-plugins.mandrake.spec".
Yes, in most cases it are different people anyway that maintain the 
different SPEC files for the same project. And since so many things differ 
between eg. SuSE, Mandrake and Red Hat it may be wiser to have them in 
different files. (Dependencies, Init scripts, Group-tag, %pre*/%post*)

And you could still allow looking for eg.

	nagios-plugins.fedora.spec
	nagios-plugins.redhat.spec
	nagios-plugins.spec

I once mentioned such a scheme on the distcc ml. Since currently 
developers don't add SPEC files, as they tend to favor only one 
distribution.

--   dag wieers,  dag(a)wieers.com,  http://dag.wieers.com/   --
[Any errors in spelling, tact or fact are transmission errors]

QA process was Re: RPM submission procedure

[Nicolas Mailhot]

Le ven 09/01/2004 à 23:29, Karl DeBisschop a écrit :
>  On Fri, 2004-01-09 at 16:47, Michael Schwendt wrote:
>  > On Fri, 09 Jan 2004 16:32:49 -0500, Karl DeBisschop wrote:
>   
>  > Fine. Get all distributors to agree on common package naming guidelines,
>  > on a common set of RPM macros, on a common set of helper tools, etc. etc.
>  
>  At this point, as you seem to feel, that is probably not feasible. 
The FHS success proves it can be done. But it needs a real commitment
from the distributions themselves. And you won't get it unless a large
body of packages are done in a distro-independent way (distributions bow
to external trends, they rarely create them).

This means a large self-consistent repository that uses file
dependencies for stuff that differ in distributions, work around
distribution differences by repackaging essential facilities for
distributions that lack them, and enforce uniform policy by using a
single srpm for all distros (eventually rebuild for each distribution).

Tools like apt/yum and the common repository format will probably
produce something like this sooner or later. With other distribution
packages a yum/apt entry away there will be increased pressure for
distro interoperability - just like the free availability of very
different (historically) pieces of software under Linux created common
file layout standards.

The Rh/Mdk drift would certainly have been smaller had both
distributions used a common network update tool at the time.

Cheers,

-- 
Nicolas Mailhot

QA process was Re: RPM submission procedure

[Lamar Owen]

On Friday 09 January 2004 04:47 pm, Michael Schwendt wrote:
>  On Fri, 09 Jan 2004 16:32:49 -0500, Karl DeBisschop wrote:
>  > What I'm suggesting is that if every distro has entirely separate specs,
>  > you are tending torward these sorts of conflicts. But if the distros are
>  > less insistent that a spec is for their distro alone, then we have less
>  > tendency to diverge. 
>  Fine. Get all distributors to agree on common package naming
> guidelines,
>  on a common set of RPM macros, on a common set of helper tools, etc. etc. 
That will happen about the same time that all packages become trivial to 
package.

This is one area that I know very well, having built PostgreSQL RPMs for 
several versions on many different distributions.  Even among historical Red 
Hat versions there are issues.  For several reasons my PostgreSQL packages 
need to be able to build on Red Hats 6.2, 7.3, 8.0, 9, AS 2.1, and EL3.  As 
well as building on Aurora 1.0 (which is virtually the same thing as building 
on RH7.3, just on SPARC) and Fedora Core.  This means that there is a mess, 
thanks to Red Hat's own inconsistencies (such as krb5 directory, the presence 
of krb5 at all, amongst others).  So I get into a situation of passing 
command line parameters telling the spec file what to do.  Autoselecting the 
distro is more difficult, but less time consuming for the build.

I _have_ made packages for older PostgreSQL that built without command line 
parameters or specfile edits on all of Caldera, SuSE, Red Hat, Mandrake, and 
TurboLinux.  More recent PostgreSQL is more friendly to packaging in many 
ways, but it is far from trivial.  At least that perl client was pulled out; 
it required building as root due to broken makemaker installations on older 
Red Hat and other dists.  Now I have better packages; still could use some 
work, but better.

Karl might have been meaning me when he alluded to upstream RPM packagers, 
which I am.  He has helped me a great deal in that packaging.  I remember the 
nightmare of PostgreSQL packaging that I picked up in summer of 1999.  It's a 
little better now: most distributions at least synchronize every so often, 
and I try to pull in the best of the individual distribution's patches and 
ideas.  Mandrake does things so differently that I don't bother looking at 
theirs; SuSE likewise due to significant differences.  I could make a fully 
cross-distribution spec (as I _have_ done this for hire before), but it would 
be a feeping creaturistic kludge of the first magnitude.  So I make as 
generic a package as I can that I personally can use, which in practice means 
it has a Red Hat flavor.  I then post it and let people do with it what they 
may.

But there is a great deal of NotInventedHere between distributions.
-- 
Lamar Owen
Director of Information Technology
Pisgah Astronomical Research Institute
1 PARI Drive
Rosman, NC  28772
[phone]
www.pari.edu

QA process was Re: RPM submission procedure

[Alan Cox]

On Fri, Jan 09, 2004 at 04:32:49PM -0500, Karl DeBisschop wrote:
>  > Do you want 35 rival competing updated copies of perl that all
>  > conflict ?
>  
>  Not at all. In fact I don't really see how you read that from what I
>  wrote, so I must be missing something or I must not have been clear. 
The problem is with a lot of packages new versions are not always
compatible with old ones. So you might use a new library version and find
your app built with it won't run on another setup. So for that reason you
want people to know there is a definitive version of a package. There might
be other versions but you always want to know the definitive one.

>  suggesting is that if the distros work from the idea of a reference
> spec
>  that is somewhat distro agnostic, then the delevopr coordinates with the
>  .deb packaging community, the rpm packaging community, and the ports
>  community. That way, as packagers contribute thier feedback to the
>  devlopers, it's more focused on what make the application better. 
Ah now I understand where you are going. That makes complete sense and some
packages do come with reference spec files.

>  If packagers from different distros eschew that tpe of cooperation,
> then
>  the developer either sees silence or a "do things my way because that's
>  our policy" from 20 different camps pulling the project in all different
>  ways. 
True.

Alan

QA process was Re: RPM submission procedure

[Michael Schwendt]

On Fri, 09 Jan 2004 14:20:16 -0500, Karl DeBisschop wrote:

>  On Fri, 2004-01-09 at 13:59, Michael Schwendt wrote:
>  > On Fri, 9 Jan 2004 20:05:06 +0200 (EET), Panu Matilainen wrote:
>  > 
>  > > The amount of nitpicking trusted developers produce 
>  > > (among themselves) is enough to scare off anybody starting in packaging 
>  > > I'm willing to bet :)
>  > 
>  > This must change, although often it is separated between suggestions and
>  > blocker criteria. But at the same time, new packagers should not come
>  > with slightly modified packages from e.g. Mandrake Cooker which bzip2 even
>  > the smallest patch, or generic packages which contain dozens of lines of
>  > conditional code which tries to adapt to a build environment.
>  
>  Is this rejection of generic packages official policy? 
There is nothing like "rejection". Unless it's software which cannot be
included due to patenting or licencing issues. 

But back to generic packages as described by me roughly, find someone
who's willing to review them on a low level. It really is no fun to spend
time on something which looks unnecessary. We build for Fedora Core (and
Red Hat Linux) only, not for Distribution ABC which requires different
packaging and which uses additional helper tools.

>  Somehow it seems to me contrary to the idea that Fedora stresses
>  upstream bugfixes. 
I fail to see the connection between a spec file mess and integrating
bug-fixes upstream.

Apart from that, not everything you ship upstream is included.

>  I think of it this way: as a developer, I cannot possibly package
> well
>  for all distros, but I may be able to package well for the one or ones I
>  use heavily. Nonetheless, as a developer, I have to maintain the code in
>  single units that is readily packaged by many distros, not just the one
>  I happen to use. 
Then -- as a developer -- make your software be trivial to package.

--

QA process was Re: RPM submission procedure

[Karl DeBisschop]

On Fri, 2004-01-09 at 16:21, Michael Schwendt wrote:
>  On Fri, 09 Jan 2004 14:20:16 -0500, Karl DeBisschop wrote:
>  
>  > On Fri, 2004-01-09 at 13:59, Michael Schwendt wrote:
>  > > On Fri, 9 Jan 2004 20:05:06 +0200 (EET), Panu Matilainen wrote:
>  > > 
>  > > > The amount of nitpicking trusted developers produce 
>  > > > (among themselves) is enough to scare off anybody starting in packaging 
>  > > > I'm willing to bet :)
>  > > 
>  > > This must change, although often it is separated between suggestions and
>  > > blocker criteria. But at the same time, new packagers should not come
>  > > with slightly modified packages from e.g. Mandrake Cooker which bzip2 even
>  > > the smallest patch, or generic packages which contain dozens of lines of
>  > > conditional code which tries to adapt to a build environment.
>  > 
>  > Is this rejection of generic packages official policy?
>  
>  There is nothing like "rejection". Unless it's software which cannot be
>  included due to patenting or licencing issues. 
>  
>  But back to generic packages as described by me roughly, find someone
>  who's willing to review them on a low level. It really is no fun to spend
>  time on something which looks unnecessary. We build for Fedora Core (and
>  Red Hat Linux) only, not for Distribution ABC which requires different
>  packaging and which uses additional helper tools.
>  
>  > Somehow it seems to me contrary to the idea that Fedora stresses
>  > upstream bugfixes.
>  
>  I fail to see the connection between a spec file mess and integrating
>  bug-fixes upstream. 
The connection is that I see an application that requires a mess to
package as broken. That brokenness should ideally be communicated
upstream so the developer can address if for the benefit of all distros.
When a packager's first choice is to create that spec file mess, rather
than work the developer to avoid it, the community as a whole is being
shortchanged. 

>  Apart from that, not everything you ship upstream is included.
I don't see what you're getting at here. But I think we're more in
agreement than I thought at first, so it may not be worth dragging out.

>  > I think of it this way: as a developer, I cannot possibly
> package well
>  > for all distros, but I may be able to package well for the one or ones I
>  > use heavily. Nonetheless, as a developer, I have to maintain the code in
>  > single units that is readily packaged by many distros, not just the one
>  > I happen to use.
>  
>  Then -- as a developer -- make your software be trivial to package. 
I agree fully with this.

And what I was trying to say is that feedback from the packagers should
help the developers do that. If the process of generating a package is
hard, then I want to know how to make it easier, so that the spec can be
trivial.

But if the packager viewpoint is completely ignorant of the fact that
the developer caters to many distros, then the developer gets at best
"do x for me, even if some other distro wants conflicting y"

As a developer, I want to make packaging as easy as possible across the
board. Which means I need to be able to support not only Fedora, even
that that is my distro of choice.

I was probably just picking up on a different emphasis. Where you said
'new packagers should not come with ... generic packages' I read that to
mean all distros should have very unique packages, and inferred that
would be at the expense of communication with the developer so that
unique distributions goals can be met with minimal added work at the
packaging end. But that doesn't seem to be what you're saying.

-- 
Karl DeBisschop <kdebisschop(a)alert.infoplease.com&gt;
Pearson Education/Information Please

QA process was Re: RPM submission procedure

[Michael Schwendt]

On Fri, 09 Jan 2004 16:57:34 -0500, Karl DeBisschop wrote:

>  > But back to generic packages as described by me roughly, find
> someone
>  > who's willing to review them on a low level. It really is no fun to spend
>  > time on something which looks unnecessary. We build for Fedora Core (and
>  > Red Hat Linux) only, not for Distribution ABC which requires different
>  > packaging and which uses additional helper tools.
>  > 
>  > > Somehow it seems to me contrary to the idea that Fedora stresses
>  > > upstream bugfixes.
>  > 
>  > I fail to see the connection between a spec file mess and integrating
>  > bug-fixes upstream.
>  
>  The connection is that I see an application that requires a mess to
>  package as broken. That brokenness should ideally be communicated
>  upstream so the developer can address if for the benefit of all distros.
>  When a packager's first choice is to create that spec file mess, rather
>  than work the developer to avoid it, the community as a whole is being
>  shortchanged.  
I've killed some quotes deliberately because they haven't help in avoiding
a misunderstanding. In my original comment on "generic packages" I refer
to spec files which attempt at getting it right for more than a specific
distribution. With "spec file mess" I refer to them using lots of %define
constants or macros to be set manually, lots of conditional code to do
some things differently for different distributions and an additional
multitude of conditional %_with/%_without checks to build the package in a
variety of configurations. While the latter may be useful for some users,
the rest is just bloat, a "mess" that is not fun to review, especially not
when patches are bzipped or if conditional code is also in %post scriplets
[and calls distribution-specific helper tools] and if it takes extra
effort to see *how* the software is built.

>  > Apart from that, not everything you ship upstream is included.
>  
>  I don't see what you're getting at here. But I think we're more in
>  agreement than I thought at first, so it may not be worth dragging out. 
Patches, patches, patches, but also packaging style. Distributors apply
some customizations, improvements, tweaks and integration work which an
upstream maintainer would reject to include by default. With regard to
spec files, some upstream maintainers would also insist on modifying
contributed spec files in a way they find "smart" or elegant, but which
would obfuscate the packaging (e.g. generated %files lists, overuse of
macros, heavy inline creation of extra files within spec files, ...).

>  > > I think of it this way: as a developer, I cannot possibly
> package well
>  > > for all distros, but I may be able to package well for the one or ones I
>  > > use heavily. Nonetheless, as a developer, I have to maintain the code in
>  > > single units that is readily packaged by many distros, not just the one
>  > > I happen to use.
>  > 
>  > Then -- as a developer -- make your software be trivial to package.
>  
>  I agree fully with this.
>  
>  And what I was trying to say is that feedback from the packagers should
>  help the developers do that. If the process of generating a package is
>  hard, then I want to know how to make it easier, so that the spec can be
>  trivial. 
That would not solve the problem of the "packager freak" who loves to turn
his spec file into a little uncommented program and who does enough
lobbying to get his spec file included as the upstream default.
 
>  As a developer, I want to make packaging as easy as possible across
> the
>  board. Which means I need to be able to support not only Fedora, even
>  that that is my distro of choice. 
But that means that you may need to spend additional time on tweaking your
"configure" scripts and aclocal code to deal with possible detection of
differences between distributions. Something developers with a "works for
me" mentality would rather avoid, in particular if they don't like other
distributions. It ends with them accepting contributed patches
reluctantly, if at all.

--

QA process was Re: RPM submission procedure

[Gene C.]

On Friday 09 January 2004 13:59, Michael Schwendt wrote:
>  On Fri, 9 Jan 2004 20:05:06 +0200 (EET), Panu Matilainen wrote:
>  > The amount of nitpicking trusted developers produce
>  > (among themselves) is enough to scare off anybody starting in packaging
>  > I'm willing to bet :)
>
>  This must change, although often it is separated between suggestions and
>  blocker criteria. But at the same time, new packagers should not come
>  with slightly modified packages from e.g. Mandrake Cooker which bzip2 even
>  the smallest patch, or generic packages which contain dozens of lines of
>  conditional code which tries to adapt to a build environment.
>
>  A fundamental problem is "packager mentality". If a packager has the
>  impression that a QA person is the nitpicking bad guy who's nothing else
>  than a PITA, then the whole concept of working together on a community
>  maintained repository is doomed to fail. If on the other hand, the
>  packager is at least a bit open for suggestions or established common
>  practise, everything works better. 
I agree about a change being needed.

Something that is unclear to me is the purpose of the QA.  If it is to ensure 
that the package is constructed properly, etc., then this make sense.  It it 
is targeting the software being packaged then I am sceptical that this is 
going to work.  Yes, some effort by packagers and reviewers should be made to 
address obvious security errors/problems.  But expecting a packager (as 
oppose to an upstream developer) to do everything is (IMHO) not going to 
work.  While source code audits can be useful, they are also very expensive 
and require highly skilled (and scarce) people to do them or the result is 
meaningless.

I also like the categorizing approach that Alex does for ATrpms.  If there is 
some expectation of having people actually test and QA some new packages, 
then there should be a place to put new packages ... reading bugzilla reports 
to find out about new packages just does not cut it.

I am hopeful that the Red Hat folks will speak on the Fedora Extras subject 
soon (their lack of comment is very noticeable).  Some of this discussion 
leads me to believe that the QA expectations (for fedora.us and Fedora 
Extras) exceeds that for Fedora Core packages by Red Hat.  Yes, I am sure 
that some packages get a lot of scrutiny (the kernel, glibc, gcc, etc.) but a 
lot do not (e.g., gftp).
-- 
Gene

QA process was Re: RPM submission procedure

[Michael K. Johnson]

On Fri, Jan 09, 2004 at 02:42:38PM -0500, Gene C. wrote:
>  I am hopeful that the Red Hat folks will speak on the Fedora Extras
> subject 
>  soon (their lack of comment is very noticeable).  Some of this discussion  
Well, I have two reasons for not commenting:
 o  We don't have the CVS server up yet, and I believe that action
    should come before talk here.
 o  I'm doing a lot of listening.  Overall it is important to me that
    it not be so hard or confusing to do that people don't do it.  I
    think that the rules can be simple and exceptions worked out on
    an individual basis; that's what we've done internally.  We need
    rules, with rationale, better codified -- which is one thing we
    liked about fedora.us in the first place -- but I think that we
    need to make sure that people aren't put off too much by the
    rules we have, and I think that I've got more listening to do
    before I speak.

michaelkjohnson

 "He that composes himself is wiser than he that composes a book."
 Linux Application Development                     -- Ben Franklin
 http://people.redhat.com/johnsonm/lad/

QA process was Re: RPM submission procedure

[Gene C.]

On Friday 09 January 2004 19:22, Michael K. Johnson wrote:
>  On Fri, Jan 09, 2004 at 02:42:38PM -0500, Gene C. wrote:
>  > I am hopeful that the Red Hat folks will speak on the Fedora Extras
>  > subject soon (their lack of comment is very noticeable).  Some of this
>  > discussion
>
>  Well, I have two reasons for not commenting:
>   o  We don't have the CVS server up yet, and I believe that action
>      should come before talk here.
>   o  I'm doing a lot of listening.  Overall it is important to me that
>      it not be so hard or confusing to do that people don't do it.  I
>      think that the rules can be simple and exceptions worked out on
>      an individual basis; that's what we've done internally.  We need
>      rules, with rationale, better codified -- which is one thing we
>      liked about fedora.us in the first place -- but I think that we
>      need to make sure that people aren't put off too much by the
>      rules we have, and I think that I've got more listening to do
>      before I speak. 
Great!  This sounds reasonable.

I am a bit disturbed by some of this discussion and the egos that are 
expressed ... it seems like there is a lot of not-invented-here stuff.  Right 
now I believe that ESR (and the 40 packages he maintains) has been definitely 
alienated by this discussion.

I really like Alex's bleeding/testing/good/stable classification of packages.  
It should be easy to get packages into the bleeding or testing categories but 
to get into either good or stable should require lots more QA.  By QA I mean 
TESTING by lots of people ... not code reviews by a couple of "experts".  My 
experience is that very subtle bugs (or intentional compromises) can fool 
even experts who have access to source code but lots of testing (running the 
software) by lots of people can reveal that the software does have problems 
(even if they cannot point to specific causes).  Your do not get lots of 
testing if the package is not "available".

I am also seeing QA requirements well beyond what Red Hat does internally 
(from my perspective) and what I believe is reasonable.  While I believe that 
some QA rules are needed, lets make them realistic ... lots of rules about 
package format are reasonable but the quality of the code in the package will 
only be discovered through testing (actually running the software).

As I see it, a lot of the Extras (fedora.us) packages are more like the gftp 
package rather than the kernel, gcc, or glibc which need lots of testing 
before becoming stable.  For gftp, Red Hat has taken the software package 
from upstream and then packaged it for Red Hat / Fedora Core making sure that 
it puts stuff into the correct directories, etc.  However, the Red Hat folks 
do not make sure that the software is working as it should ... that is an 
upstream responsibility.  The 2.0.14 version of gftp which is part of FC-1 
has problems with http.  I worked with the upstream maintainer/developer to 
fix these problems and most of the fixes are in 2.0.16 (with the remainder in 
CVS).  While there is no gftp update for FC1, 2.0.16 is available in rawhide 
(development) for those with the need.

IMHO, the objective for Extras should be for a set of packages which do not 
conflict with anything in Core (or other Extras packages) and which have a 
classification which reflects actual user experience with that software.  The 
Extras process should honor and promote this objective.
-- 
Gene

QA process was Re: RPM submission procedure

[Warren Togami]

Gene C. wrote:
>  I am also seeing QA requirements well beyond what Red Hat does
> internally 
>  (from my perspective) and what I believe is reasonable.  While I believe that 
>  some QA rules are needed, lets make them realistic ... lots of rules about 
>  package format are reasonable but the quality of the code in the package will 
>  only be discovered through testing (actually running the software). 
Anyone that truly does QA for a real job would know that this is false. 
    fedora.us does not even come close to what real QA people do, stuff 
like writing regression tests to make sure all library functions do as 
they are expected.

Warren

QA process was Re: RPM submission procedure

[Michael Schwendt]

On Sat, 10 Jan 2004 04:00:29 -0500, Gene C. wrote:

-- snip --

Disclaimer in front: I've posted so much to this topic, I think I need to
point out that all this is my personal opinion and in no way official for
fedora.us. Consult the fedora.us leadership for official statements.

>  Right now I believe that ESR (and the 40 packages he maintains) has
> been
>  definitely alienated by this discussion. 
So? Is that a problem? What he has in mind is simply not possible yet.
Fedora.us is damned short on human resources. There's no automated build
system. There's no automated system where ESR could submit package updates
and get them built. Fedora.us does scale, provided that with each new
package, additional people join the "fun".

>  I really like Alex's bleeding/testing/good/stable classification
> of packages.  
>  It should be easy to get packages into the bleeding or testing categories 
*What* is so difficult to get a package into fedora.us "testing" or
"unstable"? I still don't understand it.

>  but 
>  to get into either good or stable should require lots more QA.  By QA I mean 
>  TESTING by lots of people ... not code reviews by a couple of "experts".
Where do you take these things from? Who does "code reviews" [*]? You can
be happy if one or two people test a package prior to publishing it. A
rock solid package can be in testing for a long time without a single bug
report being submitted. What does that tell you? Nothing. The next update
already could introduce a showstopper bug which must be caught prior to
release. But sometimes these are very difficult to find and affect only a
small target group. Even the code maintainers miss them until they receive
a bug report.

[*] Except maybe skimming over context diffs upon upgrades. Have you ever
watched the sized of a diff between two releases of e.g. the GIMP 2
development branch? It would be a full-time job to "review" something like
that at the level of source code.

>  My 
>  experience is that very subtle bugs (or intentional compromises) can fool 
>  even experts who have access to source code but lots of testing (running the 
>  software) by lots of people can reveal that the software does have problems 
>  (even if they cannot point to specific causes).   
So what? Do you consider the upstream source tarball checksum verification
step unnecessary?

>  Your do not get lots of 
>  testing if the package is not "available". 
Then let users vote on what they want to see published and what they will
test-drive. Package requests can be prioritized based on demand.

Btw, do you wait for any specific software to be published? Or what
is your intention?

>  I am also seeing QA requirements well beyond what Red Hat does
> internally 
>  (from my perspective) and what I believe is reasonable.  
*Where*? I don't see that. I see mostly low level technical reviews which
want to ensure that the software builds and installs correctly, is set up
properly, has good dependencies and that it "seems to work" (if the
reviewer does not consider himself qualified enough to do expert
tests). Sure, for some packages it is different. Some reviewers are very
familiar with some software and test-drive the software before they would
approve a release. But there is no such definition of QA at fedora.us.

>  While I believe that 
>  some QA rules are needed, lets make them realistic ... lots of rules about 
>  package format are reasonable but the quality of the code in the package will 
>  only be discovered through testing (actually running the software). 
Package format is not an issue at all. Packagers are encouraged to keep
their spec files readable and easy to maintain and adapt what is
considered common practise. They are not forced to do that.

>  As I see it, a lot of the Extras (fedora.us) packages are more like
> the gftp 
>  package rather than the kernel, gcc, or glibc which need lots of testing 
>  before becoming stable.  For gftp, Red Hat has taken the software package 
>  from upstream and then packaged it for Red Hat / Fedora Core making sure that 
>  it puts stuff into the correct directories, etc.  However, the Red Hat folks 
>  do not make sure that the software is working as it should ... that is an 
>  upstream responsibility.  The 2.0.14 version of gftp which is part of FC-1 
>  has problems with http.  I worked with the upstream maintainer/developer to 
>  fix these problems and most of the fixes are in 2.0.16 (with the remainder in 
>  CVS).  While there is no gftp update for FC1, 2.0.16 is available in rawhide 
>  (development) for those with the need. 
Red Hat QA is not bullet-proof either. And they may have different
priorities from time to time.

A gftp 2.0.16 package can also be found in fedora.us "patches". It could
have been released as an official Update to FC1 if there had not been a
misunderstanding. The fedora.us package was "tweaked", i.e. based on a
different build script than Red Hat's package. In the future, it will be
better if external updates to packages in FC are based on Red Hat's
package and only modified where absolutely necessary.
 
--

QA process was Re: RPM submission procedure

[Michael K. Johnson]

On Sat, Jan 10, 2004 at 04:00:29AM -0500, Gene C. wrote:
>  I am also seeing QA requirements well beyond what Red Hat does
> internally 
>  (from my perspective) and what I believe is reasonable.  While I believe that 
>  some QA rules are needed, lets make them realistic ... lots of rules about 
>  package format are reasonable but the quality of the code in the package will 
>  only be discovered through testing (actually running the software). 
There are many levels of QA.  We don't have to have the same kind of rules
for QA that, say, a proprietary software company does.  I think that package
QA is primarily to make sure that the packaging has not been screwed up,
and secondarily to look for faults in the software itself.

There will always be bugs.  The point isn't to get rid of all bugs before
declaring the software usable.  The point is to avoid disaster while
keeping up with the amazing development speed of open source software....
Open source gives us a better opportunity to fix bugs without waiting
through a whole cycle.

So exactly what needs to be done for QA depends on what has been done
in development.  If a previously OK'ed package has had one minor patch
added to fix a bug, then QA just does not have to be extensive.  If it
is an entirely new version that has had very little upstream testing,
then it needs more.

This seems like common sense to me.  :-)

michaelkjohnson

 "He that composes himself is wiser than he that composes a book."
 Linux Application Development                     -- Ben Franklin
 http://people.redhat.com/johnsonm/lad/

QA process was Re: RPM submission procedure

[Gene C.]

On Monday 12 January 2004 12:50, Michael K. Johnson wrote:
>  On Sat, Jan 10, 2004 at 04:00:29AM -0500, Gene C. wrote:
>  > I am also seeing QA requirements well beyond what Red Hat does internally
>  > (from my perspective) and what I believe is reasonable.  While I believe
>  > that some QA rules are needed, lets make them realistic ... lots of rules
>  > about package format are reasonable but the quality of the code in the
>  > package will only be discovered through testing (actually running the
>  > software).
>
>  There are many levels of QA.  We don't have to have the same kind of rules
>  for QA that, say, a proprietary software company does.  I think that
>  package QA is primarily to make sure that the packaging has not been
>  screwed up, and secondarily to look for faults in the software itself.
>
>  There will always be bugs.  The point isn't to get rid of all bugs before
>  declaring the software usable.  The point is to avoid disaster while
>  keeping up with the amazing development speed of open source software....
>  Open source gives us a better opportunity to fix bugs without waiting
>  through a whole cycle.
>
>  So exactly what needs to be done for QA depends on what has been done
>  in development.  If a previously OK'ed package has had one minor patch
>  added to fix a bug, then QA just does not have to be extensive.  If it
>  is an entirely new version that has had very little upstream testing,
>  then it needs more.
>
>  This seems like common sense to me.  :-) 

Agreed!!  Unfortunately, this is not my understanding of what is being said in 
some of the messages in this thread.  I am hoping that it is simply a 
misunderstanding by me.
-- 
Gene

QA process was Re: RPM submission procedure

[Michael Schwendt]

On Tue, 13 Jan 2004 01:50:04 -0500, Gene C. wrote:

>  On Monday 12 January 2004 12:50, Michael K. Johnson wrote:
>  > On Sat, Jan 10, 2004 at 04:00:29AM -0500, Gene C. wrote:
>  > > I am also seeing QA requirements well beyond what Red Hat does internally
>  > > (from my perspective) and what I believe is reasonable.  While I believe
>  > > that some QA rules are needed, lets make them realistic ... lots of rules
>  > > about package format are reasonable but the quality of the code in the
>  > > package will only be discovered through testing (actually running the
>  > > software).
>  >
>  > There are many levels of QA.  We don't have to have the same kind of rules
>  > for QA that, say, a proprietary software company does.  I think that
>  > package QA is primarily to make sure that the packaging has not been
>  > screwed up, and secondarily to look for faults in the software itself.
>  >
>  > There will always be bugs.  The point isn't to get rid of all bugs before
>  > declaring the software usable.  The point is to avoid disaster while
>  > keeping up with the amazing development speed of open source software....
>  > Open source gives us a better opportunity to fix bugs without waiting
>  > through a whole cycle.
>  >
>  > So exactly what needs to be done for QA depends on what has been done
>  > in development.  If a previously OK'ed package has had one minor patch
>  > added to fix a bug, then QA just does not have to be extensive.  If it
>  > is an entirely new version that has had very little upstream testing,
>  > then it needs more.
>  >
>  > This seems like common sense to me.  :-)
>  
>  
>  Agreed!!  Unfortunately, this is not my understanding of what is being said in 
>  some of the messages in this thread.  I am hoping that it is simply a 
>  misunderstanding by me. 
Maybe we should discuss a specific package request ticket and how to QA
it.

--