Quake3 security issue and non-responsive maintainer: Xavier Lamien

[Przemek Klosowski]

On 05/11/2010 07:30 PM, Jeff Spaleta wrote:
> On Tue, May 11, 2010 at 3:10 PM, Przemek Klosowski
> <przemek.klosowski at nist.gov>  wrote:
>> This probably means at least a rudimentary application testing rig
>> and a discipline that identifies and deals with distressed packages.
>
> Does the ongoing work with AutoQA provide the solution you are looking for?
>
> http://fedoraproject.org/wiki/AutoQA

Yes, it looks like the right approach, and reasonable infrastructure,
but it needs a lot of work on tests, and on the policy of integrating
the tests with the packaging and release process.

Currently the tests seem to only address the packaging issues; tallying 
the tests from 
https://fedorahosted.org/pipermail/autoqa-results/2010-May/thread.html 
gives this distribution:

     23 rats_install
      45 rats_sanity
      98 conflicts:
     171 repoclosure:
     673 rpmguard:
     799 rpmlint:

We  need a lot of application-specific tests, which at least presently
would have to be hand-rolled. A reasonable future approach might be the 
regression testing, i.e. tests generated out of bug reports. In 
principle, the bugzilla informal query on how to replicate the bug could
be broken out and formalized to a point where the data might be usable 
to a testing harness.

The policy question is pretty complicated and multifaceted:

* errors have to be classified as to severity/impact;

* each failure requires a decision by the package manager on what is the 
appropriate action, and who should be responsible (upstream? manager?);

* such decisions should probably be remembered from test run to test run 
to mask test noise, but probably should be reviewed periodically

* what to do if the maintainer or upstream are not responsive

It's a large and complicated task; even carefully maintained projects 
have test errors, and in fact such  projects tend to have more extensive 
test suites so they may end up being noisier at least initially.