Automatic Provides and Requires for Python modules

[Toshio Kuratomi]

In general, +1 to this.

On Sun, Mar 21, 2010 at 11:25:18AM +0100, Aurelien Bompard wrote:
> Hi people,
> 
> At the top of our Python packaging page 
> (https://fedoraproject.org/wiki/Packaging:Python), there's a note which 
> reads 'In theory /usr/lib/rpm/pythondeps.sh would also automatically 
> generate "Provides" lines'
> 
> This is also true for python modules : distutils and setuptools have a way 
> to specify provides and requires, but those are not reflected in the RPM.
> It means that they must be entered and maintained by hand in the spec file. 
> In my opinion, this is not optimal.
> 
> The perl modules have an interesting way of reflecting their dependencies in 
> the rpm: they add a dependency on "perl(Module::Name)". I believe the same 
> system can be applicable to python modules.
> 
> Currently, there are two main packaging systems in python : distutils and 
> setuptools. Distutils declares the dependencies in an egg-info file, which 
> is RFC-822-formatted. Setuptools turns this file into a PKG-INFO file in a 
> subdirectory, and adds a requires.txt files with additional dependencies, in 
> a different format.
> 
Note that in some corner cases, the egg-info might not give you what you
need.

For instance, if we were to subpackage pkg_resources separately from
setuptools we would only have one upstream egg-info that referenced
setuptools, nothing referencing pkg_resources.

Like I say, these are corner cases, though.

> The pythondeps.sh script should be able to extract requires from these 
> files. To match the dependencies, pythondeps.sh should create virtual 
> provides like "python(ModuleName) = version".
> 
In describing this, module name is probably not the best word.  ProjectName
or EggName might be better.  (Because there can multiple modules but they
are all described by a single name in the egg-info).

Also versions are more problematic than naming's corner cases.

* If we backport a bugfix or feature to an old version, the EggInfo may
  require a newer version than we provide even though it would work fine
  with our version.
* Upstreams frequently put a minimum version in that they have tested with
  rather than the true minimum version that the package will work with.
  That means the setup.py file (and thus egginfo)  may say it requires
  SqlAlchemy-0.5.5 but it really works with the SQLAlchemy-0.5.2 that we
  ship.

> The good news is : I've written it already ;-)
> It's based on the pythondeps.sh script from Git master (which changed a 
> little bit due to python3, see bug 532118). Also, the script does not try to 
> be too smart with versionned dependencies, because the format is a little 
> bit different in python and in rpm. For those complicated cases, handwritten 
> requirements can still be added to the spec file. The script only covers the 
> usual cases.
> 
> For reference, the dependency format in distutils is described here:
> http://www.python.org/dev/peps/pep-0314/
> The dependency format in setuptools is described here:
> http://peak.telecommunity.com/DevCenter/setuptools#declaring-dependencies
> 

Note: PEP about changing the version strings in distutils:
http://www.python.org/dev/peps/pep-0386/

It's accepted but not yet implemented.  When it is we will have something
that we can map upstream versions to our version + release format (although,
we'd probably only use the version portion in autodeps).

Also PEPs that change the metadata format in distutils:
http://www.python.org/dev/peps/pep-0376/
http://www.python.org/dev/peps/pep-0390/

> A patch would not make much sense due to the size of the addition, so here's 
> the full script: http://aurelien.bompard.org/projects/divers/pythondeps.sh
> 
Some specifics:
    # Handle alpha and rc releases: the version comparator will be rpm, not
    # python, so 1.0rc1 > 1.0. Deal with it by turning 1.0rc1 into 1.0-0.rc1
    # (which is the recommended naming scheme anyway)
    echo $pyver | sed -e 's/\([0-9.]\+\)\([a-z].*\)/\1-0.\2/g'

Actually, the naming scheme for Fedora would be:
  1.0-0.1.rc1, 1.0-0.2.rc1, etc.

If I'm reading the code correctly, these versions just get put in the
autodeps, though, so that shouldn't matter.  However, what does matter is
that this sequence won't be parsed so that the sequence of versions is
correct:

0.9     => 0.9
1.0rc1  => 1.0-0.rc1
1.0     => 1.0
1.0post1=> 1.0-0.post1

That would order as 0.9, 1.0, 1.0-0.rc1, 1.0-0.post1

You want something more like this:
0.9     => 0.9-1
1.0rc1  => 1.0-0.rc1
1.0     => 1.0-1
1.0post1=> 1.0-1.post1

0 and 1 are always added and denote pre release versus post release.

The next problem is deciding which version strings you're going to target.
Current distutils, setuptools, and the distutils PEP have three different
algorithms for comparing versions.  You can throw out current distutils
because no one use it.  Setuptools is something that the python community is
trying to get rid of but it's the current de facto standard.  It's version
algorithm is a huge mess because it makes a lot of guesses about what you
might mean.  The distutils PEP is not in a python release yet but it is much
better -- there's a few discreet keywords that mean pre release or post
release and some methods of combining them.

>
> I've tested it with quite a few packages, but to make sure I've written a 
> few unit tests (very simple, bash-based) :
> http://aurelien.bompard.org/projects/divers/test-pythondeps.sh
> 
> I do believe it would be a valuable addition to RPM (of course, even if 
> accepted shortly, I don't expect it to land in F-13 since it requires 
> recompiling all the python packages).
> 
> Do you think it's a good idea ? What about the implementation ?

Idea for Provides: +1

Idea for Requires: I think the versioning landscape in python is pretty
crazy right now.  I'd leave off versions in Requires altogether until
(hopefully) PEP-386 is implemented and becomes standard. (But including
Requires: python(SQLAlchemy) will be an improvement by itself)

Implementation: The only bug I saw was how you were changing upstream
version to an rpm orderable version-release string and any changes needed
to remove version from the Requires generation.

-Toshio
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 198 bytes
Desc: not available
Url : http://lists.fedoraproject.org/pipermail/python-devel/attachments/20100322/070ffea2/attachment.bin