QA-process for new packages

[Silke Reimer]

Hallo!

Some time ago I submitted a few packages on fedora.us. One of them
(gdal, Bug #1964) got lots of comments so I rebuilt the package and
announced it today. Due to several reasons it took me some time to
rebuild the package and meanwhile I have been set as owner of the
bug (and of all my other bugs (#1965, #2000 and #2001) as well).

Since I am not member of the QA team I don't really understand this
action. I thought that people that are new to fedora can submit
packages thus being submitter of a bug. Afterwards the QA team
assigns someone to do the quality assurance and the submitter will
have to fix the package if there are problems.

So, my question is: Did I misunderstand the process? And what should
I do to ensure that a QA team member might look at my packages? Or
is this perhaps the normal process and I don't have to do anything
at all?

Thanks for all explanation (or some hints to any documention [1]),

	Silke

[1] Yes, I already read: http://www.fedora.us/wiki/PackageSubmissionQAPolicy

-- 
Silke Reimer

Intevation GmbH                      http://intevation.de/
FreeGIS                                http://freegis.org/

[Michael Schwendt]

On Thu, 30 Sep 2004 19:26:37 +0200, Silke Reimer wrote:

>  Some time ago I submitted a few packages on fedora.us. One of them
>  (gdal, Bug #1964) got lots of comments so I rebuilt the package and
>  announced it today. Due to several reasons it took me some time to
>  rebuild the package and meanwhile I have been set as owner of the
>  bug (and of all my other bugs (#1965, #2000 and #2001) as well). 
The assignment of package request tickets to package owners has been
announced and explained on this list around two weeks ago.
 
>  Since I am not member of the QA team 
Who said that? 

You _are_ a member of the QA team. The community does QA on packages
in the queue. That's pre-release QA. Other members of the community do
post-release QA and submit bug reports when they find something in
the published packages.

>  I don't really understand this
>  action. I thought that people that are new to fedora can submit
>  packages thus being submitter of a bug. Afterwards the QA team
>  assigns someone to do the quality assurance and the submitter will
>  have to fix the package if there are problems. 
There's no such system. Doing QA on new packages and package updates
is done by volunteers. And they are not assigned packages to QA, but
they choose interesting submissions themselves. This system is flawed.
Because if I reviewed and approved 200 different new packages, I would
need to assure that any future update requests for those 200 packages
are QA'ed, too, until the package developer reaches "trusted" status.

So, what I've been doing recently is to pick packages from new
contributors and give them the chance to get a package published. In
return, however, I'd like to see that they engage in QA and help other
contributors. I've counted more than 60 different names in the queue,
so theoretically, there are enough different people to choose from.
Further, I monitor the REVIEWED queue, and I take a look at older
package requests from the trusted developers, too, because they don't
need QA for updates.

>  So, my question is: Did I misunderstand the process? And what should
>  I do to ensure that a QA team member might look at my packages? Or
>  is this perhaps the normal process and I don't have to do anything
>  at all?
>  
>  Thanks for all explanation (or some hints to any documention [1]),
>  
>  	Silke
>  
>  [1] Yes, I already read: http://www.fedora.us/wiki/PackageSubmissionQAPolicy 
The first paragraph here,

  http://www.fedora.us/wiki/PackageSubmissionQAPolicy#review

contains an important piece of information:

  [...] encourage other packagers to thoroughly check your package by
  doing a good job in QA testing of their package. [...]


Might be or might not be that all this changes when the merger with
official Fedora Extras is complete. But again, according to the
package submission and QA policy at fedora.us, it doesn't take much to
get a package published...

-- 
Fedora Core release 2.91 (FC3 Test 2) - Linux 2.6.8-1.541
loadavg: 1.60 1.92 1.70

[Michael Tiemann]

Silke, I'll volunteer to be one of your reviewers.

M

On Thu, 2004-09-30 at 14:39, Michael Schwendt wrote:
>  On Thu, 30 Sep 2004 19:26:37 +0200, Silke Reimer wrote:
>  
>  > Some time ago I submitted a few packages on fedora.us. One of them
>  > (gdal, Bug #1964) got lots of comments so I rebuilt the package and
>  > announced it today. Due to several reasons it took me some time to
>  > rebuild the package and meanwhile I have been set as owner of the
>  > bug (and of all my other bugs (#1965, #2000 and #2001) as well).
>  
>  The assignment of package request tickets to package owners has been
>  announced and explained on this list around two weeks ago.
>   
>  > Since I am not member of the QA team
>  
>  Who said that? 
>  
>  You _are_ a member of the QA team. The community does QA on packages
>  in the queue. That's pre-release QA. Other members of the community do
>  post-release QA and submit bug reports when they find something in
>  the published packages.
>  
>  > I don't really understand this
>  > action. I thought that people that are new to fedora can submit
>  > packages thus being submitter of a bug. Afterwards the QA team
>  > assigns someone to do the quality assurance and the submitter will
>  > have to fix the package if there are problems.
>  
>  There's no such system. Doing QA on new packages and package updates
>  is done by volunteers. And they are not assigned packages to QA, but
>  they choose interesting submissions themselves. This system is flawed.
>  Because if I reviewed and approved 200 different new packages, I would
>  need to assure that any future update requests for those 200 packages
>  are QA'ed, too, until the package developer reaches "trusted" status.
>  
>  So, what I've been doing recently is to pick packages from new
>  contributors and give them the chance to get a package published. In
>  return, however, I'd like to see that they engage in QA and help other
>  contributors. I've counted more than 60 different names in the queue,
>  so theoretically, there are enough different people to choose from.
>  Further, I monitor the REVIEWED queue, and I take a look at older
>  package requests from the trusted developers, too, because they don't
>  need QA for updates.
>  
>  > So, my question is: Did I misunderstand the process? And what should
>  > I do to ensure that a QA team member might look at my packages? Or
>  > is this perhaps the normal process and I don't have to do anything
>  > at all?
>  > 
>  > Thanks for all explanation (or some hints to any documention [1]),
>  > 
>  > 	Silke
>  > 
>  > [1] Yes, I already read: http://www.fedora.us/wiki/PackageSubmissionQAPolicy
>  
>  The first paragraph here,
>  
>    http://www.fedora.us/wiki/PackageSubmissionQAPolicy#review
>  
>  contains an important piece of information:
>  
>    [...] encourage other packagers to thoroughly check your package by
>    doing a good job in QA testing of their package. [...]
>  
>  
>  Might be or might not be that all this changes when the merger with
>  official Fedora Extras is complete. But again, according to the
>  package submission and QA policy at fedora.us, it doesn't take much to
>  get a package published...
>  
>  -- 
>  Fedora Core release 2.91 (FC3 Test 2) - Linux 2.6.8-1.541
>  loadavg: 1.60 1.92 1.70

[Silke Reimer]

Hallo Michael and Michael,

On Thu, Sep 30, 2004 at 02:47:25PM -0400, Michael Tiemann wrote:
>  Silke, I'll volunteer to be one of your reviewers. 
Thank you very much :-)

>  
>  On Thu, 2004-09-30 at 14:39, Michael Schwendt wrote:
>  > On Thu, 30 Sep 2004 19:26:37 +0200, Silke Reimer wrote:
>  > 
>  > > Some time ago I submitted a few packages on fedora.us. One of them
>  > > (gdal, Bug #1964) got lots of comments so I rebuilt the package and
>  > > announced it today. Due to several reasons it took me some time to
>  > > rebuild the package and meanwhile I have been set as owner of the
>  > > bug (and of all my other bugs (#1965, #2000 and #2001) as well).
>  > 
>  > The assignment of package request tickets to package owners has been
>  > announced and explained on this list around two weeks ago. 
Yes, but it doesn't help me to understand the implications of this
reassignement. Together with your additional information from below
it does make lots more sense to me.

>  >  
>  > > Since I am not member of the QA team
>  > 
>  > Who said that? 
>  > 
>  > You _are_ a member of the QA team. The community does QA on packages
>  > in the queue. That's pre-release QA. Other members of the community do
>  > post-release QA and submit bug reports when they find something in
>  > the published packages. 
Thanks! This explanation (toghether with you remarks below) helped me to
understand better the philosophy of the Fedora project. Perhaps I
should add that I am coming from the debian world. Thus it is quite
confusing from time to time to understand what similar and what is
different in the Fedora way to set up the project. So please excuse
my sometimes ignorant questions. (And I prefer to ask stupid
questions instead of doing something in the "wrong" way.)

>  > > I don't really understand this
>  > > action. I thought that people that are new to fedora can submit
>  > > packages thus being submitter of a bug. Afterwards the QA team
>  > > assigns someone to do the quality assurance and the submitter will
>  > > have to fix the package if there are problems.
>  > 
>  > There's no such system. Doing QA on new packages and package updates
>  > is done by volunteers. And they are not assigned packages to QA, but
>  > they choose interesting submissions themselves. This system is flawed.
>  > Because if I reviewed and approved 200 different new packages, I would
>  > need to assure that any future update requests for those 200 packages
>  > are QA'ed, too, until the package developer reaches "trusted" status.
>  > 
>  > So, what I've been doing recently is to pick packages from new
>  > contributors and give them the chance to get a package published. In
>  > return, however, I'd like to see that they engage in QA and help other
>  > contributors. I've counted more than 60 different names in the queue,
>  > so theoretically, there are enough different people to choose from.
>  > Further, I monitor the REVIEWED queue, and I take a look at older
>  > package requests from the trusted developers, too, because they don't
>  > need QA for updates. 
OK. I could of course go and review some packages that I already use
or might use if they are in Fedora. But right now I feel that I
would like to have one of my package fully reviewed before I look at
other pacakges. Thus I could  see what are the issues that I should
look at and I can become more comfortable with the Fedora way of
thinking (which also has impact on how to build a package).

	Silke

-- 
Silke Reimer

Intevation GmbH                      http://intevation.de/
FreeGIS                                http://freegis.org/

[Michael Schwendt]

On Fri, 1 Oct 2004 11:32:32 +0200, Silke Reimer wrote:

>  Thanks! This explanation (toghether with you remarks below) helped me
> to
>  understand better the philosophy of the Fedora project. Perhaps I
>  should add that I am coming from the debian world. Thus it is quite
>  confusing from time to time to understand what similar and what is
>  different in the Fedora way to set up the project. 
Well, I'm not a spokesman of the Fedora Project. Actually, fedora.us
is still a separate project inspite of the announced merger with the
Fedora Project and the ongoing preparations to import it into Fedora
Extras CVS. Except for semi-public draft documents or proposals I've
not seen anything concrete on what the final "Fedora way" will look
like.

So, currently, there's no organized QA team or anything like that,
which processes package requests in FIFO order or who are assigned
tickets by project leadership or team manager(s). Community commitment
is the key.

I wouldn't mind seeing some people who go through the queue and
re-prioritise packages based on popularity, importance (e.g.
dependencies and project objectives), or other factors.

I've also mentioned before that I'd like package developers and users
to build small QA teams or reduce QA efforts to a minimum (= security
relevant checks and some items from the QA checklist) and start
publishing in "unstable" or "testing" repository rather than
"stable".

>  So please excuse
>  my sometimes ignorant questions. (And I prefer to ask stupid
>  questions instead of doing something in the "wrong" way.) 
No problem. At least you do ask. That's far better than not asking at
all and complaining based on misunderstandings.

>  OK. I could of course go and review some packages that I already use
>  or might use if they are in Fedora. But right now I feel that I
>  would like to have one of my package fully reviewed before I look at
>  other pacakges. Thus I could  see what are the issues that I should
>  look at and I can become more comfortable with the Fedora way of
>  thinking (which also has impact on how to build a package). 
True. But by taking look at other packager's packages, you learn
how they do it. Could be helpful.

Primary objectives are to verify source checksums and to get packages
to build (in 'mach' / in the fedora.us build-system / in a system
cleaned up with fedora-rmdevelrpms).  Too many packages don't even
build. Once they build, it's possible to examine package contents and
to perform run-time checks. When, however, a package fails to build,
and when you take a closer look and you notice lots of mistakes,
pitfalls, and potentially broken things, you post your findings as
comments. Ssometimes that looks as if a reviewer picks on a package
developer.

-- 
Fedora Core release 2.91 (FC3 Test 2) - Linux 2.6.8-1.590
loadavg: 3.19 3.10 1.98

[Silke Reimer]

On Fri, Oct 01, 2004 at 12:33:13PM +0200, Michael Schwendt wrote:
>  On Fri, 1 Oct 2004 11:32:32 +0200, Silke Reimer wrote:
>  
>  > Thanks! This explanation (toghether with you remarks below) helped me to
>  > understand better the philosophy of the Fedora project. Perhaps I
>  > should add that I am coming from the debian world. Thus it is quite
>  > confusing from time to time to understand what similar and what is
>  > different in the Fedora way to set up the project.
>  
>  Well, I'm not a spokesman of the Fedora Project. Actually, fedora.us
>  is still a separate project inspite of the announced merger with the
>  Fedora Project and the ongoing preparations to import it into Fedora
>  Extras CVS. Except for semi-public draft documents or proposals I've
>  not seen anything concrete on what the final "Fedora way" will look
>  like. 
This makes it of course more difficult to understand how the project
works. But it seems to me the the "Fedora way" is just evolving and
such strongly dependent on the actions of all volunteers. Thanks for
all your explanations. I have now a very much better feeling of how
it works.

>  
>  I wouldn't mind seeing some people who go through the queue and
>  re-prioritise packages based on popularity, importance (e.g.
>  dependencies and project objectives), or other factors.
>  
>  I've also mentioned before that I'd like package developers and users
>  to build small QA teams or reduce QA efforts to a minimum (= security
>  relevant checks and some items from the QA checklist) and start
>  publishing in "unstable" or "testing" repository rather than
> "stable".
>  
>  > So please excuse
>  > my sometimes ignorant questions. (And I prefer to ask stupid
>  > questions instead of doing something in the "wrong" way.)
>  
>  No problem. At least you do ask. That's far better than not asking at
>  all and complaining based on misunderstandings.
>  
>  > OK. I could of course go and review some packages that I already use
>  > or might use if they are in Fedora. But right now I feel that I
>  > would like to have one of my package fully reviewed before I look at
>  > other pacakges. Thus I could  see what are the issues that I should
>  > look at and I can become more comfortable with the Fedora way of
>  > thinking (which also has impact on how to build a package).
>  
>  True. But by taking look at other packager's packages, you learn
>  how they do it. Could be helpful. 
That's a good point. I hope I will have some time to engage in the
near future. (Right now my company is preparing a booth at a fair so
I am quite stressed. Hope this will be better end of next month
:-))

	Silke

-- 
Silke Reimer

Intevation GmbH                      http://intevation.de/
FreeGIS                                http://freegis.org/

[Michael Schwendt]

On Thu, 30 Sep 2004 14:47:25 -0400, Michael Tiemann wrote:

>  Silke, I'll volunteer to be one of your reviewers. 
Good to hear that. Though, Silke's packages are held up by packages
at the bottom of their dependency chain, e.g. shapelib:

  https://bugzilla.fedora.us/show_bug.cgi?id=920

That package blocks four other tickets, so one would guess the
packagers would try to get it released with higher priority. Would be
much appreciated to get your comments there.

-- 
Fedora Core release 2 (Tettnang) - Linux 2.6.8-1.521
loadavg: 2.50 2.22 1.98

[Michael Tiemann]

Please tell me what I need to do.  If I need to be a package reviewer
for shapelib, I'll do that.  If I need to lobby people inside Red Hat,
I'll try to do that.  I think it's hugely important that Fedora be a
good host for GIS software users and developers.

M

On Tue, 2004-10-05 at 08:49, Michael Schwendt wrote:
>  On Thu, 30 Sep 2004 14:47:25 -0400, Michael Tiemann wrote:
>  
>  > Silke, I'll volunteer to be one of your reviewers.
>  
>  Good to hear that. Though, Silke's packages are held up by packages
>  at the bottom of their dependency chain, e.g. shapelib:
>  
>    https://bugzilla.fedora.us/show_bug.cgi?id=920
>  
>  That package blocks four other tickets, so one would guess the
>  packagers would try to get it released with higher priority. Would be
>  much appreciated to get your comments there.

[Michael Schwendt]

On Tue, 05 Oct 2004 09:01:08 -0400, Michael Tiemann wrote:

>  > > Silke, I'll volunteer to be one of your reviewers.
>  > 
>  > Good to hear that. Though, Silke's packages are held up by packages
>  > at the bottom of their dependency chain, e.g. shapelib:
>  > 
>  >   https://bugzilla.fedora.us/show_bug.cgi?id=920
>  > 
>  > That package blocks four other tickets, so one would guess the
>  > packagers would try to get it released with higher priority. Would be
>  > much appreciated to get your comments there.
>
>  Please tell me what I need to do.  If I need to be a package reviewer
>  for shapelib, I'll do that.  If I need to lobby people inside Red Hat,
>  I'll try to do that.  I think it's hugely important that Fedora be a
>  good host for GIS software users and developers. 
[Actually, this particular package is called "proj" (shapelib is
another one in the dependency chain).]

Bug 920 would need somebody to answer the questions raised in the
last comment and preferably post a GPG clearsigned approval in
accordance with the fedora.us package submission and QA policy
( http://www.fedora.us/wiki/PackageSubmissionQAPolicy ). [1]

Currently, there is no other way to get packages published. And for
new contributors--package developers as well as reviewers--there is no
other way to earn trust, well, except for submitting flawless
packages. While it would be easy for me, or one of the other trusted
developers, to verify whether a new contributor is who (s)he pretends
to be, it would need much more before I would declare that I trust him
or her with regard to creating packages and/or direct access to the
build system and repository. Release manager resources are finite,
too, and for instance, I wouldn't like to see the build team encounter
an increasing number of failed builds (or updates which would be
published without QA and would break the repository). Definition of
"trust" is one of the major problems.

What holds up many packages in the fedora.us QA queue is lack of
community commitment. For the few people, who review and approve
packages regularly, reviewing and approving hundreds of special
purpose packages (e.g. educational programming languages) is beyond
their time and motivation. Especially, since many packages don't even
build, or don't seem to work when they build, or raise many
questions. More than 60 people have submitted packages. But only few
of them help eachother to get the stuff approved.

Seeing progress in terms of official Fedora Extras, and a contributor
recruitment process guided by Red Hat, would be helpful, too. With
some people at fedora.us it seems as if they would like to get a
package included, but when they are told that the package doesn't
build or has bugs/problems in several places, they don't respond or
sound as if they accept suggestions only reluctantly. As if they think
they can save themselves the work because everything will become more
easy (like the old Red Hat Contrib mess) with official Fedora Extras.

-----

[1] As a side-note, one can argue about alternative ways how to
package "proj". For instance, in the NetCDF package request, the
issue of polluting the /usr/include namespace has come up.
Installing header files into an own directory below /usr/include,
or headers and libraries into an own tree below /usr/lib/foo,
is a common technique to avoid namespace pollution and to allow
for parallel installation of multiple library versions. But
usually, such issues don't block a package from being published
as long as there are no file conflicts [yet].

-- 
Fedora Core release 2 (Tettnang) - Linux 2.6.8-1.521
loadavg: 1.38 1.38 1.41

[Ed Hill]

On Tue, 2004-10-05 at 10:09, Michael Schwendt wrote:
>  On Tue, 05 Oct 2004 09:01:08 -0400, Michael Tiemann wrote:
>  
>  > > > Silke, I'll volunteer to be one of your reviewers.
>  > > 
>  > > Good to hear that. Though, Silke's packages are held up by packages
>  > > at the bottom of their dependency chain, e.g. shapelib:
>  > > 
>  > >   https://bugzilla.fedora.us/show_bug.cgi?id=920
>  > > 
>  > > That package blocks four other tickets, so one would guess the
>  > > packagers would try to get it released with higher priority. Would be
>  > > much appreciated to get your comments there.
>  >
>  > Please tell me what I need to do.  If I need to be a package reviewer
>  > for shapelib, I'll do that.  If I need to lobby people inside Red Hat,
>  > I'll try to do that.  I think it's hugely important that Fedora be a
>  > good host for GIS software users and developers.
>  
>  [Actually, this particular package is called "proj" (shapelib is
>  another one in the dependency chain).]
>  
>  Bug 920 would need somebody to answer the questions raised in the
>  last comment and preferably post a GPG clearsigned approval in
>  accordance with the fedora.us package submission and QA policy
>  ( http://www.fedora.us/wiki/PackageSubmissionQAPolicy ). [1] 

Hi Michael (& Michael),

As someone relatively new to packaging, thank you for the discussion on
how to get software into Fedora Extras.  I'd also *very* much like to
see Fedora become a good distro for GIS (and climate and weather and
ocean modeling).

And as the NetCDF and NCO package submitter, I'm dedicating a few hours
every week to work on updates and QA.  Hopefully, I'll get faster as I
get more experience.  So please keep the reviews coming!  I promise to
enthusiastically [not reluctantly! ;-)] make changes when the
suggestions make sense.

Ed

ps - It would be great to have an OPeNDAP (formerly "DODS") RPM 
   (or collection of RPMs since it has many components) for 
   Fedora.  Is anyone else interested in tackling this large-ish
   package?  Or has someone already done it?

     http://opendap.org/

-- 
Edward H. Hill III, PhD
office:  MIT Dept. of EAPS;  Rm 54-1424;  77 Massachusetts Ave.
             Cambridge, MA 02139-4307
emails:  eh3(a)mit.edu                ed(a)eh3.com
URLs:    http://web.mit.edu/eh3/    http://eh3.com/
phone:   [phone]
fax:     [phone]