Upstream packaging feedback

[Toshio Kuratomi]

On Wed, Sep 12, 2012 at 01:41:22PM +1000, Nick Coghlan wrote:
> I'll likely be helping to guide updates to the Python packaging format
> standards over the coming months. While they won't hit the standard
> library until 3.4, there will likely be third party tool support in
> earlier versions (since the whole point of the exercise is to eliminate
> the current implementation coupling to distutils and setuptools in
> favour of better defined metadata standards for communication between
> multiple tools).
> 
> The first step will be reviewing the status quo and then creating a
> plausible road map (as well as describing current efforts for various
> aspects). I've started on that here:
> http://python-notes.boredomandlaziness.org/en/latest/pep_ideas/core_packaging_api.html
> 
> One thing I would *love* to be able to enable is adding support for
> automatic mapping of PyPI distribution names (similar to what already
> exists for Perl and CPAN) where (for example), a developer could just
> write "Requires: python(south)" instead of having to figure out manually
> the name of the appropriate RPM package in Fedora.
> 
> I believe that the new metadata fields defined in PEP 345 and PEP 426
> should be enough to support that when generating a SPEC file from the
> PyPI metadata.

Hey Nick!  Thanks for working on this whole thing.  I haven't been looking
at things in the past year but was active on it a few years ago if you want
to bounce any ideas around, get some idea of what's been discussed i nthe
past by whom, or talk about what Fedora-specifically is currently doing/has
found troublesome in the past.

Mapping of pypi distribution names was something that I looked into a small
bit with some Canonical people at PyCon two years ago.  IIRC, it was part of
trying to map distribution packages with each other and with pypi in order
to figure out the state of python3 porting.  Barry Warsaw still works there
and might know more about what happened to it -- Allison Randal is who I was
working with but I got the impression that she isn't working on that anymore
so I don't know if she'll still have code around or not.

There are numerous caveats to trying to do this, none of them
insurmountable.  I believe we were trying to compare versions as well as
package names which was even tougher.  Some things I can remember off the
top of my head:

* Multiple names for a package
  * pypi usually has the same name as the setup.py name field which is also
    encoded in the egg file name and metadata
  * The name of the module that is imported
* One upstream package having multiple downstream names -- for instance,
  Debian has setuptools and pkg_resources as two separate binary packages.
  However, they're provided by the setuptools pypi entry.
* A module that is provided by multiple upstreams:  For instance, setuptools
  is provided by setuptools and distribute.  pexpect is provided by pexpect
  and pexpect-u.
* Some packages aren't present on pypi.  Many library bindings
  provided by the libraries are this way, for instance libselinux-python.
* Some libraries have conflicting names upstream (mock was one example in
  the past.  ming still is one: http://www.libming.org/ and
  http://merciless.sourceforge.net/
* The naming for packages in the distribution isn't always simple.  In
  Fedora, for instance, we have several styles and not all of them are
  mutually exclusive:
  * python-foo is the common case for libraries (python-docutils)
  * foo-python are the majority (but not all) of libraries which are
    bindings to C libraries (selinux-python)
  * Fedora allows modules that have py as a prefix to not add python- to
    their name. (pygtk2)
  * Applications may provide libraries that other packages need but since
    they're "primarily applications" they may simply bear the name of the
    application. (bzr)
  * Case sensitivity can get you.  There are some maintainers that prefer to
    lowercase the package names and others who prefer to do exactly what
    upstream did.
  * dashes, periods, and other punctuation can also get you.  Sometimes
    those are translated into dashes, other times they're left alone, and
    other times they're omitted.
* Even today, not all packages provide egg-info.  For instance, a useful
  python module might consist of a single .py file so upstream might not
  provide a setup.py for it.  We install it by copying the .py file to
  site-packages.
* We'll need to differentiate between things provided for python2 and python3
  (and python26 in EPEL)

All that said.... you can probably sidestep some of these issues by having
python packages contain explicit virtual Provides.  These might be manually
added or automatically generated by a tool like pypi2rpm with the
maintainer editting them afterwards to make sure they didn't hit any of the
above cornercases.

There's some prior work done by other people:
* http://www.rpm.org/ticket/154
* http://lists.fedoraproject.org/pipermail/packaging/2008-June/004715.html
  (Despite my having written that email, the hard parts were all dmalcolm :-)

-Toshio
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 198 bytes
Desc: not available
URL: <http://lists.fedoraproject.org/pipermail/python-devel/attachments/20120912/a4d3ecfe/attachment.sig>