RFC: fedora.us QA approval format

[Aurelien Bompard]

Hi all,

Erik and myself have been working on a QA check script to automate the QA
process at fedora.us. The script is getting quite correct, and can even be
useful ;-)
At the end of the checks, the script outputs a QA approval report, which can
be edited, gpg-signed, and pasted into bugzilla. The question is: what
should a QA approval look like to have all the minimum QA requirements and
be as parseable as possible by a publishing script for the release manager.
Erik and I have different minds on what has to be in the review, so we
though we should discuss it here, especially with the release managers.
I have setup a wiki page with a primary format proposal, and I invite you to
have a look at it and comment on it: http://www.fedora.us/wiki/QAFormat

If the script is to get into fedora-rpmdevtools, a complete newbie could
contribute meaningful QA's in a format which would be useful to the release
manager. This would lower the bar to QA, and standardize the process a
little bit more.

Please comment


Aurélien
-- 
http://gauret.free.fr   ~~~~   Jabber : gauret(a)amessage.info
If you wish to live wisely, ignore sayings -- including this one.

[Panu Matilainen]

On Thu, 1 Apr 2004, Aurelien Bompard wrote:

>  Hi all,
>  
>  Erik and myself have been working on a QA check script to automate the QA
>  process at fedora.us. The script is getting quite correct, and can even be
>  useful ;-)
>  At the end of the checks, the script outputs a QA approval report, which can
>  be edited, gpg-signed, and pasted into bugzilla. The question is: what
>  should a QA approval look like to have all the minimum QA requirements and
>  be as parseable as possible by a publishing script for the release manager.
>  Erik and I have different minds on what has to be in the review, so we
>  though we should discuss it here, especially with the release managers.
>  I have setup a wiki page with a primary format proposal, and I invite you to
>  have a look at it and comment on it: http://www.fedora.us/wiki/QAFormat 
For one it should be such that it's easy to verify that two reviews got 
the same results. Currently since everybody uses slightly different format 
of QA reviews you need to look very carefully to spot possible differences 
(meaning the package has been changed since previous check which could 
mean something nasty is going on) in two reviews and is prone to 
human error.

For source md5sum's I'd say mark any source checked against upstream with 
(ok) or such, for sources that can't be verified from web do include 
md5sum for it anyway, it allows checking if the file has changed from one 
version to another for example.

>  
>  If the script is to get into fedora-rpmdevtools, a complete newbie could
>  contribute meaningful QA's in a format which would be useful to the release
>  manager. This would lower the bar to QA, and standardize the process a
>  little bit more. 
This is very welcome.. while the QA cannot be completely automated (eg you 
can't just blindly trust whatever happens to read as the package source 
url, it could be somebody's own website with hacked tarball, human sanity 
check is needed) removing boring, error prone manual steps from it is a 
Good Thing. 

	- Panu -

[Aurelien Bompard]

Panu Matilainen wrote:
>  For one it should be such that it's easy to verify that two
> reviews got
>  the same results. Currently since everybody uses slightly different format
>  of QA reviews you need to look very carefully to spot possible differences
>  (meaning the package has been changed since previous check which could
>  mean something nasty is going on) in two reviews and is prone to
>  human error. 
This is the point of a standardized format, and the QA script can make it
easier.

>  For source md5sum's I'd say mark any source checked against
> upstream with
>  (ok) or such, for sources that can't be verified from web do include
>  md5sum for it anyway, it allows checking if the file has changed from one
>  version to another for example. 
OK, I've applied your suggestions on the wiki page. Does this fill your
needs ?

>  This is very welcome.. while the QA cannot be completely automated
> (eg you
>  can't just blindly trust whatever happens to read as the package source
>  url, it could be somebody's own website with hacked tarball, human sanity
>  check is needed) removing boring, error prone manual steps from it is a
>  Good Thing. 
Totally agreed. Thanks for your support.

Aurélien
-- 
http://gauret.free.fr   ~~~~   Jabber : gauret(a)amessage.info
L'expérience est quelquechose que l'on acquiert juste après en avoir eu
besoin.

[Toshio Kuratomi]

On Thu, 2004-04-01 at 11:23, Aurelien Bompard wrote:
>  Hi all,
>  
>  Erik and myself have been working on a QA check script to automate the QA
>  process at fedora.us. The script is getting quite correct, and can even be
>  useful ;-) 
Hi Aurelian, Erik,

I've been working on something similar and it's also just about ready
for a first look.  Could you tell me more about yours so I know if I
should join your effort, merge ideas back and forth with you, or keep
working independently?

Mine is a little pygnome app that basically presents the user with a
checklist that they then fill out in order to generate the QA Review. 
It's working well enough for a testing release (one work-aroundable
bug)  but I have to finish adding checklist items from Fedora.us today.

It looks (from seeing your sample review) that you've got a more
streamlined, automated script but I'm not sure.

-Toshio

-- 
Toshio <toshio(a)tiki-lounge.com&gt;

[Aurelien Bompard]

Hi Toshio

>  I've been working on something similar and it's also just
> about ready
>  for a first look.  Could you tell me more about yours so I know if I
>  should join your effort, merge ideas back and forth with you, or keep
>  working independently? 
You can find our script here:
http://gauret.free.fr/fichiers/rpms/fedora/fedora-startqa
(that's a snapshot of the CVS)

>  Mine is a little pygnome app that basically presents the user with a
>  checklist that they then fill out in order to generate the QA Review.
>  It's working well enough for a testing release (one work-aroundable
>  bug)  but I have to finish adding checklist items from Fedora.us today. 
Hmm, interesting... Ours is a console only script which generates a pastable
output of the report as well as a few files containing the todo list,
rpmlint's output and the report itself. Its main features are:
- Download of the srpm, given the bug ID
- Check of the srpm md5sum (if it exists)
- Download of the sources, with md5sum check
- Check of the srpm GPG signature
- Build of the srpm in mach
- Display of rpmlint's "advice"
- Display of the files in the resulting binary rpm
- Display of the report, to be edited, signed and pasted in bugzilla
- Display of the TODO list (check that were not done, installation, etc...)
- Interactive mode where you can choose which tests to run (default)
- Batch mode where all the tests are run
- Debug mode
- Support for livna packages
(Erik please complete the list if I forgot something)

It's getting pretty usable, and needs some testing. The next big item in our
(at least my) TODO list is the fork the build in mach in order to save
time.
 
>  It looks (from seeing your sample review) that you've got a more
>  streamlined, automated script but I'm not sure. 
You're very welcome to look at it, and even test it if you can ;-)

Thanks

Aurélien
-- 
http://gauret.free.fr   ~~~~   Jabber : gauret(a)amessage.info
There are only 10 types of people in the world :
those who understand binary and those who don't.

[Toshio Kuratomi]

On Thu, 2004-04-01 at 14:47, Aurelien Bompard wrote:
>  Hi Toshio
>  
>  > I've been working on something similar and it's also just about ready
>  > for a first look.  Could you tell me more about yours so I know if I
>  > should join your effort, merge ideas back and forth with you, or keep
>  > working independently?
>  
>  You can find our script here:
>  http://gauret.free.fr/fichiers/rpms/fedora/fedora-startqa
>  (that's a snapshot of the CVS)
>   
Looking at it now.  It looks like we're working on two different sides
of the problem (There's still review work left to do after you're script
runs.  There's considerable automation that my script could do but
doesn't.)  Although I probably wouldn't have started if this had been
available before :-)  I'll do some testing later today and let you know
if I have any feedback.

I'd like to get the functionality you have into what I'm working on at
some point.  After we're both up and running, let's talk about how I
could best contribute to your effort while improving mine.

-Toshio
-- 
Toshio <toshio(a)tiki-lounge.com&gt;

fedora-startqa

[Toshio Kuratomi]

Looks good (although mach is giving me problems again so I can't test
all of it.)

Some feedback:
- A NEEDSWORK review is just as valuable as a PUBLISH +1 review.  I'd
like to see the script generate that as well.
- (Showing my ignorance of mach) How safe is it to build untrusted
sources within mach?  since mach builds the package before the user gets
a chance to go look at whether the Source URL is canonical, I was
wondering....
- Review has "Installs, runs, and uninstalls fine on FC1" but I haven't
done any of that yet -- should it be in TODO?
- The first time I ran it, the script errored out because there was an
old version of an md5sum file on the server that didn't have the package
version I had up there.  However, GPG signed SRPMs are equivalent to
checking a GPG signed md5sum file that has an  md5sum for the SRPM.  So
my view is if the GPG signature on the SRPM is good and the MD5SUM file
doesn't contradict it (ie: different signing keys, different MD5Sums for
the same file) it shouldn't error out.
- I'd like to be able to point at an SRPM instead of into bugzilla in
case I have an SRPM already on my machine that I'd like to check.

-Toshio
-- 
Toshio <toshio(a)tiki-lounge.com&gt;

fedora-startqa

[Aurelien Bompard]

Thanks for the feedback Toshio !

Toshio wrote:

>  - A NEEDSWORK review is just as valuable as a PUBLISH +1
> review.  I'd
>  like to see the script generate that as well. 
Good idea, right now, the idea is to stop if a QA showstopper is found (no
signature, build fails in mach), and let the QA'er write the NEEDSWORK
review. This can be automated a little I think. Added on the TODO list.

>  - (Showing my ignorance of mach) How safe is it to build untrusted
>  sources within mach?  since mach builds the package before the user gets
>  a chance to go look at whether the Source URL is canonical, I was
>  wondering.... 
Well, you can read the spec file before building in mach, so you can look at
the URLs for the sources, start you browser and have a look. Is that what
you mean ?

>  - Review has "Installs, runs, and uninstalls fine on FC1"
> but I haven't
>  done any of that yet -- should it be in TODO? 
It is always in the TODO anyway. Erik also thinks that it should not be
there, so I'll remove it, but I've put it there to remember the user to
tell which distro he has tested the package on, and to check
uninstallation. I think that nothing prevents a user from doing a false
review anyway, and I wanted to make a template where nothing but the
"notes" had to be added. Anyway, if the majority thinks it's wrong,
let's
remove it.

>  - The first time I ran it, the script errored out because there was
> an
>  old version of an md5sum file on the server that didn't have the package
>  version I had up there. 
Can you give me a bug id ?

>  However, GPG signed SRPMs are equivalent to 
>  checking a GPG signed md5sum file that has an  md5sum for the SRPM.  So
>  my view is if the GPG signature on the SRPM is good and the MD5SUM file
>  doesn't contradict it (ie: different signing keys, different MD5Sums for
>  the same file) it shouldn't error out. 
Yes, there is this -c option to disable srpm md5sum checking.

>  - I'd like to be able to point at an SRPM instead of into
> bugzilla in
>  case I have an SRPM already on my machine that I'd like to check. 
This is already on my TODO list :-)

Thanks for your review


Aurélien
-- 
http://gauret.free.fr   ~~~~   Jabber : gauret(a)amessage.info
Hacker vaillant, rien d'impossible.

fedora-startqa

[Toshio Kuratomi]

On Fri, 2004-04-02 at 01:43, Aurelien Bompard wrote:
>  > - (Showing my ignorance of mach) How safe is it to build
> untrusted
>  > sources within mach?  since mach builds the package before the user gets
>  > a chance to go look at whether the Source URL is canonical, I was
>  > wondering....
>  
>  Well, you can read the spec file before building in mach, so you can look at
>  the URLs for the sources, start you browser and have a look. Is that what
>  you mean ? 
Two problems:
1) In batch mode, the human element is missing.  If it is insecure,
there needs to be a way to disable mach building from the commandline.

2) If the script is aimed at newbies, there should be a warning of the
potential dangers of building the source package and what can be done to
reduce that risk.  In qa-assistant's checklist, I tried to create a list
of High Security items that should be evaluate before the reviewer
started doing anything else.  Maybe a list like that (minus things that
are checked automatically) spit out to the screen before viewing the
spec file?

>  > - The first time I ran it, the script errored out because there
> was an
>  > old version of an md5sum file on the server that didn't have the package
>  > version I had up there.
>  
>  Can you give me a bug id ?
>   
I corrected the out of date md5sum file (It was with a package that I
had control over.)  I'll try re-provoking the bug (or tracing it in the
code) when I have a bit of time.

>  > However, GPG signed SRPMs are equivalent to 
>  > checking a GPG signed md5sum file that has an  md5sum for the SRPM.  So
>  > my view is if the GPG signature on the SRPM is good and the MD5SUM file
>  > doesn't contradict it (ie: different signing keys, different MD5Sums for
>  > the same file) it shouldn't error out.
>  
>  Yes, there is this -c option to disable srpm md5sum checking.
>   
I'll give this a try too.  I think, though, what I want is for the
script to automatically make a decision that an SRPM with a valid GPG
does not have to have it's md5sum checked.

Slightly more paranoid is to make the following checks:
1] GPG signature of SRPM
2] Is the md5sum of the relevant SRPM in the md5sum file?
3] GPG signature of md5sum file
4] Did the same key sign both files?

If all pass, then pass the test.
If 1] Pass and 2] Is fail, pass the test.
All other cases fail.
-- 
_______S________U________B________L________I________M________E_______
  t  o  s  h  i  o  +  t  i  k  i  -  l  o  u  n  g  e  .  c  o  m
                                                          GA->ME 1999

[Patrice Dumas]

>  - Check of the srpm md5sum (if it exists) 
Maybe here there could be a check that the srpm doesn't install anything wrong
(that is doesn't contain paths) ?

>  - Download of the sources, with md5sum check 
Maybe the download should't be automatic, such that it is possible to check
that the download url is really the right url (presumably searching first the
project home page with google, in order not to use the url provided in the
srpm, and verifying that it is the right download page), and not one with 
bad package ?

>  - Support for livna packages 
What does this mean ?


Anyway this script is a very good idea.

Pat

[Aurelien Bompard]

Hi Patrice,

> > - Check of the srpm md5sum (if it exists)
>  
>  Maybe here there could be a check that the srpm doesn't install anything
>  wrong (that is doesn't contain paths) ? 
Could you clarify that ? I thought the srpm could only contain the sources
and the spec file, without paths. Can it be different ?

> > - Download of the sources, with md5sum check
>  
>  Maybe the download should't be automatic, such that it is possible to
>  check that the download url is really the right url (presumably searching
>  first the project home page with google, in order not to use the url
>  provided in the srpm, and verifying that it is the right download page),
>  and not one with bad package ? 
Right now, you get to look at the spec file after the sources check. Then
you can paste the url in you browser and check the location.

> > - Support for livna packages
>  
>  What does this mean ? 
Packages in the non-us section of fedora.us are at rpm.livna.org (usualy,
the problem is software patents or DMCA). This is equivalent to non-us in
Debian and PLF in Mandrake. They use a bugzilla at bugzilla.livna.org and
have the same procedure as fedora.us

>  Anyway this script is a very good idea. 
Thanks for you support !

Aurélien
-- 
http://gauret.free.fr   ~~~~   Jabber : gauret(a)amessage.info
"They that can give up essential liberty to obtain a little temporary
safety, deserve neither liberty nor safety." -- Benjamin Franklin

[Michael Schwendt]

On Fri, 2 Apr 2004 10:38:54 +0200, Patrice Dumas wrote:

>  > - Download of the sources, with md5sum check
>  
>  Maybe the download should't be automatic, such that it is possible to check
>  that the download url is really the right url (presumably searching first the
>  project home page with google, in order not to use the url provided in the
>  srpm, and verifying that it is the right download page), and not one with 
>  bad package ? 
Reviewers should also notice when upstream projects provide detached GPG
signatures, which can be used to verify the published tarballs.

[Warren Togami]

Michael Schwendt wrote:
>  On Fri, 2 Apr 2004 10:38:54 +0200, Patrice Dumas wrote:
>  
>  
> >>- Download of the sources, with md5sum check
> >
> >Maybe the download should't be automatic, such that it is possible to check
> >that the download url is really the right url (presumably searching first the
> >project home page with google, in order not to use the url provided in the
> >srpm, and verifying that it is the right download page), and not one with 
> >bad package ?
>  
>  
>  Reviewers should also notice when upstream projects provide detached GPG
>  signatures, which can be used to verify the published tarballs.
>  
>   
Reviewers should also harass upstream projects into providing GPG 
signatures "or else". =)

We managed to convince gaim and scribus, but few other people...

Warren

[Toshio Kuratomi]

On Thu, 2004-04-01 at 11:23, Aurelien Bompard wrote: 
>  I have setup a wiki page with a primary format proposal, and I invite
> you to
>  have a look at it and comment on it: http://www.fedora.us/wiki/QAFormat 
Comments on the Review Format:
I like lists and more succinct information.  So I'd slim the output as
much as possible.

header:
Checked <SRPM>
<RESOLUTION>

* I would get rid of the first line altogether.  Because these reviews
are going into bugzilla, the package name is already available.  And it
doesn't provide any information the build system needs.  Alternately,
you could make the first line:

<HASH> <SRPM>

so that it's useful for the build system.  (But see my next entry.)

Files:
<hash> <filename>

* I would change Files to MD5sums (or MD5SUMS) because at sometime in
the future the build system may support other hash types and it would be
good for it to be able to easily tell which is which one this is.

* I would specify that the <SRPM> always comes first in the HASH
section.  This makes it easier for the release managers and the
buildsystem to parse the HASH-SRPM pairing from the other files.
It could also be separated by a blank line or other visible 
demarcation.

Body:

I favor a Good/Blocker/Non-Blocker style with bulletted lists.  For your
sample, I'd implement this as:

  Good:
  * Sources: bash-completion-20040331.tar.bz2 is valid 
  * Sources: bash-completion.profile is not web-accessible
  * Signature: VALID - bcd241cb 
  * Installs, runs and uninstalls fine on FC1
  * Spec file looks good.

  Needswork:

  Minor:

  Notes:
  [Put here the things you want to add about the package]

  --TODO--
  {Things todo when editing the review}

* Regarding the Sources lines: I'd include the full URL for the
tarball and say it comes from a canonical source rather than simply
"is valid".
  * http://www.caliban.org/files/bash/bash-completion-20040331.tar.gz is
    the canonical Source location

For the other source line, I'd want something like this:
  * Sources: bash-completion.profile appears to be correct and proper

My reasoning is that I don't care so much about whether I can download
the files off the internet  (More precisely: I only care if I can't.)
I do want to know what works  been done verifying the sources (which
canonicalness of source tarballs helps for the first one and looking
at the file helps for the second.)

These could both go in the TODO section rather than the completed
review.

Anyhow, this looks like it'll be a tremendously helpful tool when
it's finished.  Good work!

-Toshio

-- 
_______S________U________B________L________I________M________E_______
  t  o  s  h  i  o  +  t  i  k  i  -  l  o  u  n  g  e  .  c  o  m
                                                          GA->ME 1999

[Aurelien Bompard]

Hi,

Sorry to write back so late about that.

Toshio wrote:
>  * I would get rid of the first line altogether.  Because these
> reviews
>  are going into bugzilla, the package name is already available.  And it
>  doesn't provide any information the build system needs.  Alternately,
>  you could make the first line:
>  <HASH> <SRPM>
>  so that it's useful for the build system.  (But see my next entry.)
>  
>  * I would change Files to MD5sums (or MD5SUMS) because at sometime in
>  the future the build system may support other hash types and it would be
>  good for it to be able to easily tell which is which one this is.
>  
>  * I would specify that the <SRPM> always comes first in the HASH
>  section.  This makes it easier for the release managers and the
>  buildsystem to parse the HASH-SRPM pairing from the other files.
>  It could also be separated by a blank line or other visible
>  demarcation. 
OK. I've updated http://fedora.us/wiki/QAFormat. How do you like it now ?

>  * Regarding the Sources lines: I'd include the full URL for the
>  tarball and say it comes from a canonical source rather than simply
>  "is valid".
>    * http://www.caliban.org/files/bash/bash-completion-20040331.tar.gz is
>      the canonical Source location 
Maybe the full URL is not necessary in the review, we could just add that
canonicalness has been checked.
I'm worried about the fact that bugzilla's mails are reformated to 80 (or
some) characters columns, thus breaking GPG check. I try to avoid long
lines, and an URL will probably be too long.

>  For the other source line, I'd want something like this:
>    * Sources: bash-completion.profile appears to be correct and proper
>  
>  My reasoning is that I don't care so much about whether I can download
>  the files off the internet  (More precisely: I only care if I can't.)
>  I do want to know what works  been done verifying the sources (which
>  canonicalness of source tarballs helps for the first one and looking
>  at the file helps for the second.) 
Good point. Done.

>  Anyhow, this looks like it'll be a tremendously helpful tool
> when
>  it's finished.  Good work! 
Thanks :-)

Can you (and any fedora-devel subscriber) give me your comments about the
new page at http://fedora.us/wiki/QAFormat ?

Aurélien
-- 
http://gauret.free.fr   ~~~~   Jabber : gauret(a)amessage.info
Accroche-toi au prompt, j'enlève le shell.

[Toshio Kuratomi]

On Sat, 2004-04-10 at 08:04, Aurelien Bompard wrote:
>  Toshio wrote:
>  > * Regarding the Sources lines: I'd include the full URL for the
>  > tarball and say it comes from a canonical source rather than simply
>  > "is valid".
>  >   * http://www.caliban.org/files/bash/bash-completion-20040331.tar.gz is
>  >     the canonical Source location
>  
>  Maybe the full URL is not necessary in the review, we could just add that
>  canonicalness has been checked.
>  I'm worried about the fact that bugzilla's mails are reformated to 80 (or
>  some) characters columns, thus breaking GPG check. I try to avoid long
>  lines, and an URL will probably be too long.
>   
An alternate solution to leaving it out is to put the URL on its own
line.  (Was going to verify this wouldn't munge things on
https://bugzilla.redhat.com/bugzilla-devel, but that seems to be
non-existant right now.)

OTOH your point that the full URL might not be necessary is true.  It's
in the SRPM and any second reviewer really should be getting it from
there, not the bugzilla entry.

New thoughts:
* I like bullets for lists of things.  Helps to delimit separate entries
on a page.

* I'd leave the src.rpm under MD5Sums so if the hash algorithm gets
updated a program will parse the type of hash before encountering its
first hash.

* I'd put the recommendation (PUBLISH +1| NEEDSWORK) as the first line. 
It's the most important thing to pick out of the clutter of the review. 
All the rest is "supporting evidence" of the recommendation.

* What are your thoughts on which parts of the review will be used by
automated tools?  I would say the Recommendation and MD5Sum section are
definites.  The Sources section a maybe, and the rest probably not.  In
any case, if it is going to be machine parsed then I'd use one phrase
for "Positive result" and one for "Negative result" (possibly one for
"Not-Applicable") (OK | NOT OK | N/A).  That way a parser will have
fewer words to search for.  After the result keyword there could be any
other words as comments.  If an entry isn't going to be machine parsed,
then I'd stick it into a Good: vs NEEDSWORK: section and forgo the
(OK|NOT OK)

* It doesn't show up in this review -- I'd put any patches in the
Sources section, would you agree?

Here's my take on what I'd like a review to look like
http://www.fedora.us/wiki/QAFormatAlt

This assumes that automated tools aren't going to check whether Package
Name/SRPM Signature/etc were verified.  If you think they should be then
I'd take your example with a header (ex: Essential Checks) and bullets
and normalize all the entries to OK instead of OK, VALID, and YES.

Hope some of my comments sound reasonable to you,
-Toshio
-- 
_______S________U________B________L________I________M________E_______
  t  o  s  h  i  o  +  t  i  k  i  -  l  o  u  n  g  e  .  c  o  m
                                                          GA->ME 1999

[Aurelien Bompard]

Toshio wrote:

>  New thoughts:
>  * I like bullets for lists of things.  Helps to delimit separate entries
>  on a page. 
Agreed. I like that too :)

>  * I'd leave the src.rpm under MD5Sums so if the hash algorithm
> gets
>  updated a program will parse the type of hash before encountering its
>  first hash. 
Agreed, that's a good point. 

>  * I'd put the recommendation (PUBLISH +1| NEEDSWORK) as the first
> line.
>  It's the most important thing to pick out of the clutter of the review.
>  All the rest is "supporting evidence" of the recommendation. 
OK, good point too.

>  * What are your thoughts on which parts of the review will be used
> by
>  automated tools?  I would say the Recommendation and MD5Sum section are
>  definites.  The Sources section a maybe, and the rest probably not. 
This looks reasonable to me. The mandatory lines are already listed on
PackageSubmissionQAPolicy:
--8<------
In cases where a QA message would lead to package approval or publication,
the message MUST be GPG clearsigned, and contain md5sums of the reviewed
source RPM as well as preferably all upstream sources.
--8<------

So that's as you wrote: recommandation and MD5SUMs. The Sources section
seems not to be mandatory, but it can do no harm if it is easily parseable.
On that point I agree on a "OK" + any explanation. NOT OK and N/A look fine
to me too.

>  If an entry isn't going to be machine parsed,
>  then I'd stick it into a Good: vs NEEDSWORK: section and forgo the
>  (OK|NOT OK) 
OK by me. In this case, since it is not going to be parsed, the reviewer is
totally free to choose his prose ;-) The Good and Needswork sections are
clearer and should be respected though.
 
>  * It doesn't show up in this review -- I'd put any patches in
> the
>  Sources section, would you agree? 
Agreed, they are more or less like a local source anyway.
 
>  Here's my take on what I'd like a review to look like
>  http://www.fedora.us/wiki/QAFormatAlt 
Looks good to me. I'd remove the blank line between the srpm's md5sum and
the rest though. Since it's the first one anyway, we can save space there.
But that's totally cosmetic, and if you like it a lot, we'll keep it. (it's
just to have something to improve on your version ;-p )
I'm going to merge that into QAFormat.

>  This assumes that automated tools aren't going to check whether
> Package
>  Name/SRPM Signature/etc were verified.  If you think they should be then
>  I'd take your example with a header (ex: Essential Checks) and bullets
>  and normalize all the entries to OK instead of OK, VALID, and YES. 
The question actually is: Should the automated tool control that the
approval have the minimum required items, or should it also check that the
QA Checklist has been gone through. Because of the lenght and the nature of
the QA Checklist, I think it is impossible to make sure that the QA check
was actually correctly done. I think we have to trust a bit the reviewer,
and that's why we need at least two QA approvals.
So I don't mind having the tool check only the minimum, and having an human
eye deciding if the QA check was correctly done. OTOH, I'm not the one who
will be operating the tool, and I'm not the one doing the job. If the
release managers want the (yet-to-be-written) tool to be more automatic,
and thus to remove more of their load, they sould impose a parseable format
for the rest of the checklist too. Can any of the release managers tell us
a bit about that please ?

Thanks a lot for your comments Toshio, I'm happy that we're reaching a
satisfying QA template.

Bye,

Aurélien
-- 
http://gauret.free.fr   ~~~~   Jabber : gauret(a)amessage.info
"First they ignore you, then they laugh at you, then they fight you,
then you win. " -- Mahatma Ghandi               ^^^^^^^^^^^^^^^^^^^
                                                   Linux is here.