Upstream packaging feedback

Fri Sep 21 02:19:54 UTC 2012

This is a timely discussion for me because I need to get back to packaging
some tools for RHEL/Centos that we have as dependencies.  The tools are
available on pypi or from a git repository.  They seem to be available in
older versions for Fedora 14 but not in EPEL or other sites for
RHEL/Centos.

One issue I encountered when previously working on the packaging for
Fedora was that pypi packages do not necessarily follow the Linux
Filesystem Hierarchy Standard.  In fact, the packages may be implicitly
organized for something like Windows where documentation goes in the same
directory as code.  I found I needed to use pypi2rpm bdist_rpm2 and modify
the spec file to get things into the right places.

BTW, the tools right now use python 2.7.

So, please add the FHS issue to your list.

I've also found that everything seems to be in a state of flux right now,
with setuptools, distribute, distutils, distutils2, and
pypi2rpm/bdist_rpm2 all out there and bumping into each other.  When I
tried to do a yum update to my Fedora 14 system, I ran into trouble
because of conflicts among these packages.  I hope this all settles down
soon.

Stan Klein

On Thu, September 13, 2012 8:00 am, Toshio Kuratomi <a.badger at gmail.com>
wrote:

>
> Message: 1
> Date: Wed, 12 Sep 2012 15:14:07 -0700
> From: Toshio Kuratomi <a.badger at gmail.com>
> To: Fedora Python SIG <python-devel at lists.fedoraproject.org>
> Subject: Re: Upstream packaging feedback

>
> On Wed, Sep 12, 2012 at 01:41:22PM +1000, Nick Coghlan wrote:
>> I'll likely be helping to guide updates to the Python packaging format
>> standards over the coming months. While they won't hit the standard
>> library until 3.4, there will likely be third party tool support in
>> earlier versions (since the whole point of the exercise is to eliminate
>> the current implementation coupling to distutils and setuptools in
>> favour of better defined metadata standards for communication between
>> multiple tools).
>>
>> The first step will be reviewing the status quo and then creating a
>> plausible road map (as well as describing current efforts for various
>> aspects). I've started on that here:
>> http://python-notes.boredomandlaziness.org/en/latest/pep_ideas/core_packaging_api.html
>>
>> One thing I would *love* to be able to enable is adding support for
>> automatic mapping of PyPI distribution names (similar to what already
>> exists for Perl and CPAN) where (for example), a developer could just
>> write "Requires: python(south)" instead of having to figure out manually
>> the name of the appropriate RPM package in Fedora.
>>
>> I believe that the new metadata fields defined in PEP 345 and PEP 426
>> should be enough to support that when generating a SPEC file from the
>> PyPI metadata.
>
> Hey Nick!  Thanks for working on this whole thing.  I haven't been looking
> at things in the past year but was active on it a few years ago if you
> want
> to bounce any ideas around, get some idea of what's been discussed i nthe
> past by whom, or talk about what Fedora-specifically is currently
> doing/has
> found troublesome in the past.
>
> Mapping of pypi distribution names was something that I looked into a
> small
> bit with some Canonical people at PyCon two years ago.  IIRC, it was part
> of
> trying to map distribution packages with each other and with pypi in order
> to figure out the state of python3 porting.  Barry Warsaw still works
> there
> and might know more about what happened to it -- Allison Randal is who I
> was
> working with but I got the impression that she isn't working on that
> anymore
> so I don't know if she'll still have code around or not.
>
> There are numerous caveats to trying to do this, none of them
> insurmountable.  I believe we were trying to compare versions as well as
> package names which was even tougher.  Some things I can remember off the
> top of my head:
>
> * Multiple names for a package
>   * pypi usually has the same name as the setup.py name field which is
> also
>     encoded in the egg file name and metadata
>   * The name of the module that is imported
> * One upstream package having multiple downstream names -- for instance,
>   Debian has setuptools and pkg_resources as two separate binary packages.
>   However, they're provided by the setuptools pypi entry.
> * A module that is provided by multiple upstreams:  For instance,
> setuptools
>   is provided by setuptools and distribute.  pexpect is provided by
> pexpect
>   and pexpect-u.
> * Some packages aren't present on pypi.  Many library bindings
>   provided by the libraries are this way, for instance libselinux-python.
> * Some libraries have conflicting names upstream (mock was one example in
>   the past.  ming still is one: http://www.libming.org/ and
>   http://merciless.sourceforge.net/
> * The naming for packages in the distribution isn't always simple.  In
>   Fedora, for instance, we have several styles and not all of them are
>   mutually exclusive:
>   * python-foo is the common case for libraries (python-docutils)
>   * foo-python are the majority (but not all) of libraries which are
>     bindings to C libraries (selinux-python)
>   * Fedora allows modules that have py as a prefix to not add python- to
>     their name. (pygtk2)
>   * Applications may provide libraries that other packages need but since
>     they're "primarily applications" they may simply bear the name of the
>     application. (bzr)
>   * Case sensitivity can get you.  There are some maintainers that prefer
> to
>     lowercase the package names and others who prefer to do exactly what
>     upstream did.
>   * dashes, periods, and other punctuation can also get you.  Sometimes
>     those are translated into dashes, other times they're left alone, and
>     other times they're omitted.
> * Even today, not all packages provide egg-info.  For instance, a useful
>   python module might consist of a single .py file so upstream might not
>   provide a setup.py for it.  We install it by copying the .py file to
>   site-packages.
> * We'll need to differentiate between things provided for python2 and
> python3
>   (and python26 in EPEL)
>
> All that said.... you can probably sidestep some of these issues by having
> python packages contain explicit virtual Provides.  These might be
> manually
> added or automatically generated by a tool like pypi2rpm with the
> maintainer editting them afterwards to make sure they didn't hit any of
> the
> above cornercases.
>
> There's some prior work done by other people:
> * http://www.rpm.org/ticket/154
> * http://lists.fedoraproject.org/pipermail/packaging/2008-June/004715.html
>   (Despite my having written that email, the hard parts were all dmalcolm
> :-)
>
> -Toshio
> -------------- next part --------------
> A non-text attachment was scrubbed...
> Name: not available
> Type: application/pgp-signature
> Size: 198 bytes
> Desc: not available
> URL:
> <http://lists.fedoraproject.org/pipermail/python-devel/attachments/20120912/a4d3ecfe/attachment-0001.sig>