Announcing Fedora Tracker -- comments?

[Brad Smith]

All, 

About six months ago, as the result of a conversation on IRC, I began
playing with the idea of a "tracker" for Fedora: Something to tie
together the many apt and yum repositories out there into a single,
simple to use framework. 

There is, of course, a great deal of software available for Fedora
outside of the core package set. But the question I kept running into
was "How do I figure out which repository, if any, has the package I'm
looking for, assuming I even know the name of the package in the first
place?". 

None of the answers to this question that I'd come across were really
satisfactory. They either required me to already have a comprehensive
list of third-party repositories (synaptic) or were too general (google,
rpmfind, etc). So, partly to scratch this itch and partly to teach
myself Python, I started work on a tool for easily indexing and
searching Fedora repositories. 

I've been quiet about things because until now the Fedora Tracker was
mostly an educational project. But it's now at a point where I think it
could be of use to others and I'd appreciate some constructive
criticism. I'm announcing it on this list instead of fedora-list because
I'd don't know how much traffic my webhost can handle and I'd like to
stress-test it with a smaller group of well-clued people first. 

The url is:

http://academy.phpwebhosting.com/cgi-bin/tracker/tracker.py

Since this is my first python project, any feedback regarding the
quality of the code would also be appreciated. I figured out a lot of
this stuff, like how to dissect the rpm headers, "by ear", so no doubt
there are many places where my approach could be improved upon. Should
anyone care to peruse it, the code is available here:

http://academy.phpwebhosting.com/tracker/tracker-2004032400.tar.bz2

The app basically consists of two components: The web-based frontend
allows users to search indexed repositories and submit repos for
indexing. Another tool called tracker-process.py runs as a cron job and
pulls down the headers for each queued repo, storing that info in the
database. 

There is more that I plan to do, but since my work at Red Hat is
unrelated to the Tracker project (or development of any sort) how soon I
can do more is dependent upon how much free time I have (not much at the
moment). That said, here are the main items on my TODO list:

- Support for repo-side configuration. I would implement this by having
tracker-process.py look for a "tracker.conf" file in the headers or base
directory of a repo. By modifying this file, the administrator of a repo
could specify a list of mirrors for the repository, alter the repository
description, indicate when the repo should be re-indexed, etc. 

- Support for updating a repository that has been indexed (see above)

- Support for storing a list of mirrors for a repository (see above)

- Support for the proposed common XML-based metadata format
(http://linux.duke.edu/projects/metadata/readme.metadata)

- Improvements to the searching mechanism, such as protecting quoted
strings (currently the search is either done as a series of AND'ed
keyword matches or as one big regex match). 

- Ability to display dependencies (for the life of me I cannot figure
out how to interpret the dependency data in the rpm headers). Any
reference to documentation would be appreciated.

- Improvements to the user interface. For example, currently if a search
yields multiple versions of the same package they are all listed. When I
have some time I'm going to change this so that packages with multiple
versions/architectures are displayed under one package name and only
expanded when the package is selected.

- Index Fedora Core packages. Currently this is a problem because the
default repository is too busy for tracker-process.py to be able to
reliably index it. Once support for mirrors is set up, this should not
be a problem. Alternately, I could just index a mirror and then change
the url in the database. =;) 

- Comment things more thoroughly and convert existing comments to pydoc
or similar 

- Re-arrange files so that the url is more accessible

- Whatever else comes to mind or is suggested to me =;)


Thanks to everybody who helps make Fedora such a fun distro to use!

--Brad

[seth vidal]

On Wed, 2004-03-24 at 18:12 -0500, Brad Smith wrote:

>  All, 
>  
>  About six months ago, as the result of a conversation on IRC, I began
>  playing with the idea of a "tracker" for Fedora: Something to tie
>  together the many apt and yum repositories out there into a single,
>  simple to use framework.  
That's cool,

Hi Brad,
 I took a look through your code, you should hop over to the
rpm-python-list:
https://lists.dulug.duke.edu/mailman/listinfo/rpm-python-list

There are some problems in the code you put together wrt dealing with
the package information but nothing that's not fixable.

However, I think looking at the code in the createrepo scripts for the
new xml metadata would help out a bit. B/c I did a lot of the same stuff
in there.



>  None of the answers to this question that I'd come across were
> really
>  satisfactory. They either required me to already have a comprehensive
>  list of third-party repositories (synaptic) or were too general (google,
>  rpmfind, etc). So, partly to scratch this itch and partly to teach
>  myself Python, I started work on a tool for easily indexing and
>  searching Fedora repositories. 
>  The url is:
>  
>  http://academy.phpwebhosting.com/cgi-bin/tracker/tracker.py 
>  The app basically consists of two components: The web-based frontend
>  allows users to search indexed repositories and submit repos for
>  indexing. Another tool called tracker-process.py runs as a cron job and
>  pulls down the headers for each queued repo, storing that info in the
>  database. 
>   
>  - Support for repo-side configuration. I would implement this by
> having
>  tracker-process.py look for a "tracker.conf" file in the headers or base
>  directory of a repo. By modifying this file, the administrator of a repo
>  could specify a list of mirrors for the repository, alter the repository
>  description, indicate when the repo should be re-indexed, etc.  
>  - Support for updating a repository that has been indexed (see
> above)
>  
>  - Support for storing a list of mirrors for a repository (see above)
>  
>  - Support for the proposed common XML-based metadata format
>  (http://linux.duke.edu/projects/metadata/readme.metadata) 
This might be a good place to start to make 


>  - Ability to display dependencies (for the life of me I cannot
> figure
>  out how to interpret the dependency data in the rpm headers). Any
>  reference to documentation would be appreciated. 
rpm-python-list is a good place to go - but also the xml-metadata has
that information stored in the primary.xml.gz file - you could just
steal the code that extracts that information and shove it into a db
instead of into the xml. Or read it from the xml and dump it into a db -
probably much easier and faster!


>  - Improvements to the user interface. For example, currently if a
> search
>  yields multiple versions of the same package they are all listed. When I
>  have some time I'm going to change this so that packages with multiple
>  versions/architectures are displayed under one package name and only
>  expanded when the package is selected. 
You should use the package ID (and md5sum or sha1sum) to unique the
packages that are dupes. Again - you can look at the primary.xml.gz to
do this too.

>  - Comment things more thoroughly and convert existing comments to
> pydoc
>  or similar  
just use docstrings in your function defs:

def function():
    """this is my docstring, there are many others like it but this one 
       is mine"""


then you can do pydoc ./myfile

and it will print out the docstring :)

Maybe jumping on over to a quieter list like rpm-python-list would be an
easier discussion forum but I think this could do some very neat things
for generating mirror lists for people

or autogenerating yum.conf or up2date sources files.


-sv

[Brad Smith]

>  * http://rpm.livna.org repositories are missing 
I've added this, but note that the "submit" link will allow anyone to
queue up a repository for indexing. 

>  * "testing" and "unstable" repositories at
> http://fedora.us are be
>  missing, too 
I'll have to do these later, unless someone else beats me to it
*hinthint* =;)
 
>  * While the current package search interface is powerful and
> flexible, I
>  would highly recommend a default simplified interface for the ordinary
>  user where to enter a package name only and be done. Because once the
>  service gets known more widely, it likely will be used as a replacement
>  for sites like rpmseek.com or rpmfind.net. User can leave fields blank in
>  the current search interface, but it is personal experience that the
>  visual complexity of the interface either confuses or scares off visitors. 
This is a very good suggestion. I mentioned before that my time is
limited, but I'll add this to the stack. Thanks for the suggestion.

--Brad

[Brad Smith]

On Thu, 2004-03-25 at 02:02, Laurent GUERBY wrote:
>  Really nice! A detail, I believe the Location: and baseurl: data
> when
>  displayed should be links (all other displayed URLs are links
>  but those two :).
>  
>  Laurent 
Thanks for the encouragement. I'm in the process of having all displayed
repo urls link to the list of packages for that repo within the Tracker
interface.

--Brad

[Michael Schwendt]

On Wed, 24 Mar 2004 18:12:51 -0500, Brad Smith wrote:

>  http://academy.phpwebhosting.com/cgi-bin/tracker/tracker.py 
* http://rpm.livna.org repositories are missing

* "testing" and "unstable" repositories at http://fedora.us are be
missing, too
 
* While the current package search interface is powerful and flexible, I
would highly recommend a default simplified interface for the ordinary
user where to enter a package name only and be done. Because once the
service gets known more widely, it likely will be used as a replacement
for sites like rpmseek.com or rpmfind.net. User can leave fields blank in
the current search interface, but it is personal experience that the
visual complexity of the interface either confuses or scares off visitors.

[Laurent GUERBY]

Really nice! A detail, I believe the Location: and baseurl: data when
displayed should be links (all other displayed URLs are links
but those two :).

Laurent

[Warren Togami]

Michael Schwendt wrote:
>  On Wed, 24 Mar 2004 18:12:51 -0500, Brad Smith wrote:
>  
>  
> >http://academy.phpwebhosting.com/cgi-bin/tracker/tracker.py
>  
>  
>  * http://rpm.livna.org repositories are missing
>  
>  * "testing" and "unstable" repositories at http://fedora.us are be
>  missing, too 
Packages in the authoritative fedora.us and rpm.livna.org repositories 
will overlap with and conflict with some packages in 3rd party 
repositories.  For this reason search results should be displayed in a 
set order of precedence.  Something like the below.

Order of Precedent
------------------
1) Fedora Core
2) Fedora Extras (currently at fedora.us)
3) Non-US Extras (rpm.livna.org)
4) Everything Else
    a) Fedora Alternatives (Does not exist yet)
    b) 3rd party repositories

[Alan Cox]

On Wed, Mar 24, 2004 at 10:21:45PM -1000, Warren Togami wrote:
>  Order of Precedent
>  ------------------
>  1) Fedora Core
>  2) Fedora Extras (currently at fedora.us)
>  3) Non-US Extras (rpm.livna.org)
>  4) Everything Else
>     a) Fedora Alternatives (Does not exist yet)
>     b) 3rd party repositories 
I'd have thought the order of precedence was a user matter at the end of
the day. Certainly I'd want 0) My own collection and US folks probably
wouldn't want 3) at all

As a rather nifty visualisation tool there is no point crippling it

Alan

[Warren Togami]

Alan Cox wrote:
>  On Wed, Mar 24, 2004 at 10:21:45PM -1000, Warren Togami wrote:
>  
> >Order of Precedent
> >------------------
> >1) Fedora Core
> >2) Fedora Extras (currently at fedora.us)
> >3) Non-US Extras (rpm.livna.org)
> >4) Everything Else
> >   a) Fedora Alternatives (Does not exist yet)
> >   b) 3rd party repositories
>  
>  
>  I'd have thought the order of precedence was a user matter at the end of
>  the day. Certainly I'd want 0) My own collection and US folks probably
>  wouldn't want 3) at all 
#0 is a good point.
#3 obviously requires geolocation services.  In order to use the tracker 
you must plug a GPS device into your computer.  No turning off the 
Trusted Computing hardware enforcement or the black helicopters come to 
get you.

Over time, random other countries fall into the #3 blackhole.

Warren

[Brad Smith]

I think he was suggesting that the user should be able to specify a
hierarchy. 

I think what I'll do is this (subject to time constraints, so it
probably won't happen until the weeekend): It should be reasonably
simple to have the default search output look something like this:

=========================================================================
----------------------------------------------------
                 Matches in Fedora Core 
- This is the main set of official Fedora packages -
----------------------------------------------------
package
package

-------------------------------------------------------------------
		     Matches in Fedora Extras
- Supplementary packages managed by the Fedora steering committee -
-------------------------------------------------------------------
package
package
package

-----------------------------------------------------------
	     Matches in non-US Fedora Extras 
- These packages may violate DMCA restrictions in the US! -
-----------------------------------------------------------
package
package

--------------------------------------------------------------------------------------------------------------
                    	      Matches in third-party Fedora repositories
- These packages are independently maintained and quality controlled outside of the
official Fedora Project. -
--------------------------------------------------------------------------------------------------------------
package
package
package
=========================================================================


Later, a feature could be added to allow users to log in and have
personalized preferences including a customized hierarchy by repository
and repository type. Another useful setting would be custom search
criteria (ie only show yum repos by default)

--Brad

On Thu, 2004-03-25 at 07:06, Warren Togami wrote:
>  Alan Cox wrote:
>  > On Wed, Mar 24, 2004 at 10:21:45PM -1000, Warren Togami wrote:
>  > 
>  >>Order of Precedent
>  >>------------------
>  >>1) Fedora Core
>  >>2) Fedora Extras (currently at fedora.us)
>  >>3) Non-US Extras (rpm.livna.org)
>  >>4) Everything Else
>  >>   a) Fedora Alternatives (Does not exist yet)
>  >>   b) 3rd party repositories
>  > 
>  > 
>  > I'd have thought the order of precedence was a user matter at the end of
>  > the day. Certainly I'd want 0) My own collection and US folks probably
>  > wouldn't want 3) at all
>  
>  #0 is a good point.
>  #3 obviously requires geolocation services.  In order to use the tracker 
>  you must plug a GPS device into your computer.  No turning off the 
>  Trusted Computing hardware enforcement or the black helicopters come to 
>  get you.
>  
>  Over time, random other countries fall into the #3 blackhole.
>  
>  Warren
>

[Chip Turner]

Brad Smith <brads(a)redhat.com&gt; writes:

>  All, 
>  
>  About six months ago, as the result of a conversation on IRC, I began
>  playing with the idea of a "tracker" for Fedora: Something to tie
>  together the many apt and yum repositories out there into a single,
>  simple to use framework.  
Very cool stuff.  A couple comments:

1) Package names seem to have the first char uppercased and the rest
   lowercased in the package results page?  package name case should
   be preserved
2) The package search results are hard to read through.  Maybe an
   option to only show package nvre and summary?  That way searches
   that result in a large set can be more quickly parsed.  Maybe if
   there are more than a handful of results, it switches to a
   condensed view?

Chip

-- 
Chip Turner                   cturner(a)redhat.com
                              Red Hat, Inc.

[Scott Sloan]

>  > About six months ago, as the result of a conversation on IRC, I
> began
>  > playing with the idea of a "tracker" for Fedora: Something to tie
>  > together the many apt and yum repositories out there into a single,
>  > simple to use framework.  

I would be most interested in seeing how it has all come together.
Someone want to point me in the right direction on where I can find it?

-- 
Scott Sloan
------------
"I'm not a genius. I'm just passionately curious" -- Einstein