source audit

[Kevin Fenzi]

Since folks are checking over their packages for the correct license
tags and rebuilding for various other reasons, I thought I would add
another one to the mix. ;) 

I wrote up a quick and dirty script to check the sources that are in
the cvs look aside cache against the upstream source of the package as
pulled from the URI(s) in the Sources line(s). 

Of course this has a number of limitations: 

- Only Sources lines with full URI's can be checked. 
- I'm not currently checking Patches with full URI's, but I can add
that if there is interest. 

You can find the results file at: 

http://www.scrye.com/~kevin/fedora/sourcecheck/sourcecheck.out

And also attached to this mail. 

Lines in the output are of three forms: 

- BADURL:base-file-name:$PACKAGENAME

This means that the URI provided in the Source(s) line didn't result in
a download of the source. This could be any of: URL changed, version
changed and URL wasn't updated, Site is down, Site is gone, etc. 
Also there are a number of packages with incorrect sourceforge links. 
(BTW, there are still some packages with ftp://people.redhat.com/
URLs). 

- BADSOURCE:$SOURCENAME:$PACKAGENAME

This means that the source was downloaded ok from the upstream site,
but doesn't match the md5sum given in the sources file. 
This could be due to needing to strip out content that fedora cannot
ship (but in that case you shouldn't have the full URI in the Source
line). Or upstream following poor release practices and updating
without changing their release.

- BAD_CVS_SOURCE:$SOURCENAME:$PACKAGENAME

This means that the file was downloaded from the URI given, and the
md5sum did not match the file thats present in CVS (not the lookaside).
This might be due to timestamps, or any of the above reasons. 

Needless to say, I think all of these cases should be fixed. 

Does anyone find this useful? Should I run it on a periodic basis? 
Shall I file bugs or spam owners after some period of time?

Comments, bugs, suggestions? 

kevin

[Andy Shevchenko]

Hi Kevin Fenzi!

 On Tue, Aug 21, 2007 at 10:30:57PM -0600, Kevin Fenzi wrote next:

>  You can find the results file at: 
>  
>  http://www.scrye.com/~kevin/fedora/sourcecheck/sourcecheck.out 
Not Found
The requested URL /~kevin/fedora/sourcecheck/sourcecheck.out was not found
on this server.

>  And also attached to this mail.  
It's cool.


-- 
With best regards,
Andy Shevchenko.      mailto: andy(a)smile.org.ua

[Andy Shevchenko]

Hi Andy Shevchenko!

 On Wed, Aug 22, 2007 at 07:40:23AM +0300, Andy Shevchenko wrote next:

>  Hi Kevin Fenzi!
>  > And also attached to this mail. 
>  It's cool. 
Missed words.

I'd like to get that info periodicaly (such as once per mounth).
However I have an idea to add the username (user who mainly response on the
package). And may be good to add small description (like in your letter) to
the head of the generated stuff. 

Thanks for your efforts.

-- 
With best regards,
Andy Shevchenko.      mailto: andy(a)smile.org.ua

[Kevin Fenzi]

On Wed, 22 Aug 2007 07:46:01 +0300
andy(a)smile.org.ua (Andy Shevchenko) wrote:

>  Hi Andy Shevchenko!
>  
>   On Wed, Aug 22, 2007 at 07:40:23AM +0300, Andy Shevchenko wrote next:
>  
>  > Hi Kevin Fenzi!
>  > > And also attached to this mail. 
>  > It's cool.
>  Missed words.
>  
>  I'd like to get that info periodicaly (such as once per mounth). 
ok. Will see if there are enough folks who find it useful to keep doing
it. I would hope we could reduce the list down by quite a bit as well. 

>  However I have an idea to add the username (user who mainly response
>  on the package). And may be good to add small description (like in
>  your letter) to the head of the generated stuff.  
Yeah, there can be lots of improvements for the format. 
Will see what I can add. 

>  Thanks for your efforts. 
No problem

kevin

[Kevin Fenzi]

On Wed, 22 Aug 2007 07:40:23 +0300
andy(a)smile.org.ua (Andy Shevchenko) wrote:

>  Hi Kevin Fenzi!
>  
>   On Tue, Aug 21, 2007 at 10:30:57PM -0600, Kevin Fenzi wrote next:
>  
>  > You can find the results file at: 
>  > 
>  > http://www.scrye.com/~kevin/fedora/sourcecheck/sourcecheck.out
>  Not Found
>  The requested URL /~kevin/fedora/sourcecheck/sourcecheck.out was not
>  found on this server. 
Rats. Copied it to the wrong place. 
It should be there now. 

>  > And also attached to this mail. 
>  It's cool. 
kevin

[Eric Sandeen]

Kevin Fenzi wrote:
>  Since folks are checking over their packages for the correct license
>  tags and rebuilding for various other reasons, I thought I would add
>  another one to the mix. ;) 
>  
>  I wrote up a quick and dirty script to check the sources that are in
>  the cvs look aside cache against the upstream source of the package as
>  pulled from the URI(s) in the Sources line(s). 
>  
>  Of course this has a number of limitations: 
>  
>  - Only Sources lines with full URI's can be checked. 
>  - I'm not currently checking Patches with full URI's, but I can add
>  that if there is interest. 
>  
>  You can find the results file at: 
>  
>  http://www.scrye.com/~kevin/fedora/sourcecheck/sourcecheck.out 
or maybe:
http://www.scrye.com/~kevin/fedora/sourcecheck.out   :)

>  And also attached to this mail. 
>  
>  Lines in the output are of three forms: 
>  
>  - BADURL:base-file-name:$PACKAGENAME
>  
>  This means that the URI provided in the Source(s) line didn't result in
>  a download of the source. This could be any of: URL changed, version
>  changed and URL wasn't updated, Site is down, Site is gone, etc. 
>  Also there are a number of packages with incorrect sourceforge links.  
What constitutes a good sourceforge link these days?

Thanks,
-Eric

[Todd Zullinger]

Eric Sandeen wrote:
>  What constitutes a good sourceforge link these days? 
According to http://fedoraproject.org/wiki/Packaging/SourceURL:

http://downloads.sourceforge.net/%{name}/%{name}-%{version}.tar.gz

-- 
Todd        OpenPGP -> KeyID: 0xBEAF0CE3 | URL: www.pobox.com/~tmz/pgp
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Does it follow that I reject all authority? Perish the thought. In the
matter of boots, I defer to the authority of the boot-maker.
    -- Mikhail Bakunin

[Debarshi Ray]

>  According to http://fedoraproject.org/wiki/Packaging/SourceURL:
>
>  http://downloads.sourceforge.net/%{name}/%{name}-%{version}.tar.gz 
One of my packages, gnome-password-generator, does not follow this. It
uses http://downloads.sourceforge.net/gnome-password/%{name}-%{version}.tar.gz

Is it fine?

Cheerio,
Debarshi
-- 
GPG key ID: 63D4A5A7
Key server: pgp.mit.edu

[Todd Zullinger]

Debarshi 'Rishi' Ray wrote:
> > According to http://fedoraproject.org/wiki/Packaging/SourceURL:
> >
> > http://downloads.sourceforge.net/%{name}/%{name}-%{version}.tar.gz
>  
>  One of my packages, gnome-password-generator, does not follow this. It
>  uses http://downloads.sourceforge.net/gnome-password/%{name}-%{version}.tar.gz
>  
>  Is it fine? 
I would say it is.  I don't think the SourceURL page is there to be
enforced as law or gospel.  The main point of using that form of URL
is the downloads.sourceforge.net part, as opposed to dl.sf.net or one
of the various mirrors (which may change over time).

Packages that are part of an upstream project may not always have same
tarball name as the main project, so yeah, you'll have things like the
example you gave.  I help out with libgpod and gtkpod upstream.  Both
are under the same project space, so the URL for libgpod is
http://downloads.sourceforge.net/gtkpod/%{name}-%{version}.tar.gz

At least, that's my take on it.

-- 
Todd        OpenPGP -> KeyID: 0xBEAF0CE3 | URL: www.pobox.com/~tmz/pgp
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
A vacuum is a hell of a lot better than some of the stuff that nature
replaces it with.
    -- Tennessee Williams

[Michael Schwendt]

On Wed, 22 Aug 2007 11:30:06 +0530, Debarshi 'Rishi' Ray wrote:

>  > According to http://fedoraproject.org/wiki/Packaging/SourceURL:
>  >
>  > http://downloads.sourceforge.net/%{name}/%{name}-%{version}.tar.gz
>  
>  One of my packages, gnome-password-generator, does not follow this. It
>  uses http://downloads.sourceforge.net/gnome-password/%{name}-%{version}.tar.gz
>  
>  Is it fine? 
Sure. Macros in there are just out of convenience for the packager.
Being able to point a non-interactive download tool (e.g. curl, wget)
and fetch the tarball [from one of sourceforge's servers] directly is
the goal. With other urls, the interactive SF.net mirror chooser makes
that impossible.

[Kevin Fenzi]

On Tue, 21 Aug 2007 23:52:13 -0500
esandeen(a)redhat.com (Eric Sandeen) wrote:

>  > http://www.scrye.com/~kevin/fedora/sourcecheck/sourcecheck.out
>  
>  or maybe:
>  http://www.scrye.com/~kevin/fedora/sourcecheck.out   :) 
Yeah, copied to the wrong dir. ;( 
Should be in the first link now. 

kevin

[Hans de Goede]

Kevin Fenzi wrote:
>  Since folks are checking over their packages for the correct license
>  tags and rebuilding for various other reasons, I thought I would add
>  another one to the mix. ;) 
>  
>  I wrote up a quick and dirty script to check the sources that are in
>  the cvs look aside cache against the upstream source of the package as
>  pulled from the URI(s) in the Sources line(s). 
>   
Thanks, this is very usefull. But as usual, with all these usefull checking 
scripts people are making, can you please add the owner of the package to the 
list and sort by owner. I maintain 150 packages, so with these kinda checks 
there are always a few that need fixing, the problem is finding all my packages 
in the (somewhat long) list.

Thanks & Regards,

Hans

[Kevin Fenzi]

On Wed, 22 Aug 2007 08:35:41 +0200
j.w.r.degoede(a)hhs.nl (Hans de Goede) wrote:

>  Kevin Fenzi wrote:
>  > Since folks are checking over their packages for the correct license
>  > tags and rebuilding for various other reasons, I thought I would add
>  > another one to the mix. ;) 
>  > 
>  > I wrote up a quick and dirty script to check the sources that are in
>  > the cvs look aside cache against the upstream source of the package
>  > as pulled from the URI(s) in the Sources line(s). 
>  > 
>  
>  Thanks, this is very usefull. But as usual, with all these usefull
>  checking scripts people are making, can you please add the owner of
>  the package to the list and sort by owner. I maintain 150 packages,
>  so with these kinda checks there are always a few that need fixing,
>  the problem is finding all my packages in the (somewhat long) list. 
Yeah, good point. 

I will run it again here soon and add package owner and sort by that.

>  
>  Thanks & Regards,
>  
>  Hans
>   
kevin

[Patrice Dumas]

On Tue, Aug 21, 2007 at 10:30:57PM -0600, Kevin Fenzi wrote:
>  I wrote up a quick and dirty script to check the sources that are in
>  the cvs look aside cache against the upstream source of the package as
>  pulled from the URI(s) in the Sources line(s).  
This is a very interesting input in my opinion.
 
>  - I'm not currently checking Patches with full URI's, but I
> can add
>  that if there is interest.  
It seems to me that checking patchs would be of interest too, at least
to me. It allows to check, for example that the debian patches are still
here.

--
Pat

[Kevin Fenzi]

On Wed, 22 Aug 2007 13:09:20 +0200
pertusus(a)free.fr (Patrice Dumas) wrote:

>  On Tue, Aug 21, 2007 at 10:30:57PM -0600, Kevin Fenzi wrote:
>  > I wrote up a quick and dirty script to check the sources that are in
>  > the cvs look aside cache against the upstream source of the package
>  > as pulled from the URI(s) in the Sources line(s). 
>  
>  This is a very interesting input in my opinion. 
Thanks. I hope we can get this cleaned up. We should be matching
upsteam all the time. 
>   
>  > - I'm not currently checking Patches with full URI's, but I can add
>  > that if there is interest. 
>  
>  It seems to me that checking patchs would be of interest too, at least
>  to me. It allows to check, for example that the debian patches are
>  still here. 
Yeah, I will add that in the next run. I just noticed after it was
running that it wasn't doing Patch: lines. It took about 12 hours to
run last time. ;( 

>  Pat 
kevin

[Hans de Goede]

Kevin Fenzi wrote:
>  On Wed, 22 Aug 2007 13:09:20 +0200
>  pertusus(a)free.fr (Patrice Dumas) wrote:
> > It seems to me that checking patchs would be of interest too, at least
> > to me. It allows to check, for example that the debian patches are
> > still here.
>  
>  Yeah, I will add that in the next run. I just noticed after it was
>  running that it wasn't doing Patch: lines. It took about 12 hours to
>  run last time. ;( 
>   
Will the next run also list and be sorted by owner, please?

Regards,

Hans

[Kevin Fenzi]

On Thu, 23 Aug 2007 20:03:54 +0200
j.w.r.degoede(a)hhs.nl (Hans de Goede) wrote:

>  Kevin Fenzi wrote:
>  > On Wed, 22 Aug 2007 13:09:20 +0200
>  > pertusus(a)free.fr (Patrice Dumas) wrote:
>  >> It seems to me that checking patchs would be of interest too, at
>  >> least to me. It allows to check, for example that the debian
>  >> patches are still here.
>  > 
>  > Yeah, I will add that in the next run. I just noticed after it was
>  > running that it wasn't doing Patch: lines. It took about 12 hours to
>  > run last time. ;( 
>  > 
>  
>  Will the next run also list and be sorted by owner, please? 
Yes, thats my plan. 

>  Regards,
>  
>  Hans 
kevin

[Truch, Matthew]

In the case of gpsd, your file reports:

BAD_CVS_SOURCE:gpsd-2.34.tar.gz:gpsd

But gpsd-2.34.tar.gz is not in cvs, it's in the lookaside cache.  And
furthermore, the md5sum between the file in the lookaside cache matches
the file pointed to by the Source: URL in the spec file.  

-- 
"Things should be made as simple as possible, but not any simpler. --
Einstein."
--------------------------
Matthew Truch
Department of Physics and Astronomy
University of Pennsylvania
matt(a)truch.net
http://matt.truch.net/

[Kevin Fenzi]

On Fri, 24 Aug 2007 11:03:07 -0400
matt(a)truch.net (Matthew D Truch) wrote:

>  In the case of gpsd, your file reports:
>  
>  BAD_CVS_SOURCE:gpsd-2.34.tar.gz:gpsd
>  
>  But gpsd-2.34.tar.gz is not in cvs, it's in the lookaside cache.  And
>  furthermore, the md5sum between the file in the lookaside cache
>  matches the file pointed to by the Source: URL in the spec file.  
>   
Yeah, this is a false positive/error. 

When my script ran this is what I got from a spectool -g: 

--13:52:29--  http://download.berlios.de/gpsd/gpsd-2.34.tar.gz
           => `./gpsd-2.34.tar.gz'
Resolving download.berlios.de... 195.37.77.141
Connecting to download.berlios.de|195.37.77.141|:80... connected.
HTTP request sent, awaiting response... 302 Moved Temporarily
Location:
http://download.berlios.de/error/HTTP_SERVICE_UNAVAILABLE.html.var
[following] --13:52:29--
http://download.berlios.de/error/HTTP_SERVICE_UNAVAILABLE.html.var =>
`./HTTP_SERVICE_UNAVAILABLE.html.var' Connecting to
download.berlios.de|195.37.77.141|:80... connected. HTTP request sent,
awaiting response... 200 OK Length: unspecified [text/html]

    0K .                                                        16.34
MB/s

Last-modified header missing -- time-stamps turned off.
13:52:30 (16.34 MB/s) - `./HTTP_SERVICE_UNAVAILABLE.html.var' saved
[1058]

I will try and correct things to detect this and not report it. 

Sorry for the noise. 

kevin