serious conflicts between python pks installed via yum vs pip

[Neal Becker]

I've seen this repeatedly, with often very serious consequences (complete 
failure of update from f15->f16 for example).

Between packages installed via pip, and packages installed via yum, some 
packages seem to switch between

e.g.,
numpy-1.6.1-py2.7.egg-info

being installed as a file and the same name being installed as a directory.  
This can cause subsequent rpm update (via cpio) to abort.

I don't know what's causing this, but IMO this is a serious problem.

[Haïkel Guémar]

2012/2/10 Neal Becker <ndbecker2(a)gmail.com&gt;:
>  I've seen this repeatedly, with often very serious consequences
> (complete
>  failure of update from f15->f16 for example).
>
>  Between packages installed via pip, and packages installed via yum, some
>  packages seem to switch between
>
>  e.g.,
>  numpy-1.6.1-py2.7.egg-info
>
>  being installed as a file and the same name being installed as a directory.
>  This can cause subsequent rpm update (via cpio) to abort.
>
>  I don't know what's causing this, but IMO this is a serious problem.
>
>  --
>  devel mailing list
>  devel(a)lists.fedoraproject.org
>  https://admin.fedoraproject.org/mailman/listinfo/devel 
Never use pip outside an isolated environment (use virtualenv)

H.

[Neal Becker]

80 wrote:

>  2012/2/10 Neal Becker <ndbecker2(a)gmail.com&gt;:
> > I've seen this repeatedly, with often very serious consequences (complete
> > failure of update from f15->f16 for example).
> >
> > Between packages installed via pip, and packages installed via yum, some
> > packages seem to switch between
> >
> > e.g.,
> > numpy-1.6.1-py2.7.egg-info
> >
> > being installed as a file and the same name being installed as a directory.
> > This can cause subsequent rpm update (via cpio) to abort.
> >
> > I don't know what's causing this, but IMO this is a serious problem.
> >
> > --
> > devel mailing list
> > devel(a)lists.fedoraproject.org
> > https://admin.fedoraproject.org/mailman/listinfo/devel
>  
>  Never use pip outside an isolated environment (use virtualenv)
>  
>  H. 
Really?  This is the only answer?  Can't we tweek rpm/yum to accomodate this?  
Does anyone understand what is causing it?  Why would pip install the egg-info 
differently than rpm?

[Haïkel Guémar]

2012/2/10 Neal Becker <ndbecker2(a)gmail.com&gt;:
>  Really?  This is the only answer?  Can't we tweek rpm/yum to accomodate this?
>  Does anyone understand what is causing it?  Why would pip install the egg-info
>  differently than rpm?
>
>  --
>  devel mailing list
>  devel(a)lists.fedoraproject.org
>  https://admin.fedoraproject.org/mailman/listinfo/devel 
Python guidelines recommends that packagers installs python eggs using
distutils (python setup.py install as recommended in guidelines) while
pip use the same install method as easy_install (provided by
setuptools/distribute). The former one install egg metadata as a file,
the latter as a directory, that's not a packaging/rpm issue.

@+
H.

[Neal Becker]

80 wrote:

>  2012/2/10 Neal Becker <ndbecker2(a)gmail.com&gt;:
> >
> > Really?  This is the only answer?  Can't we tweek rpm/yum to accomodate this?
> > Does anyone understand what is causing it?  Why would pip install the
> > egg-info differently than rpm?
> >
> > --
> > devel mailing list
> > devel(a)lists.fedoraproject.org
> > https://admin.fedoraproject.org/mailman/listinfo/devel
>  
>  Python guidelines recommends that packagers installs python eggs using
>  distutils (python setup.py install as recommended in guidelines) while
>  pip use the same install method as easy_install (provided by
>  setuptools/distribute). The former one install egg metadata as a file,
>  the latter as a directory, that's not a packaging/rpm issue.
>  
>  @+
>  H. 
Yes, but trying to be helpful to users is a fedora packaging/rpm issue.

[Haïkel Guémar]

Le 10/02/2012 19:20, Neal Becker a écrit :
>  Yes, but trying to be helpful to users is a fedora packaging/rpm issue.
>  
Generally speaking, mixing RPM packages with <insert your favorite
development platform here> native packages is not doing a favor to our
users.
I agree that some users may need more up-to-date modules, but isolated
environments are a better alternative in these cases.

Off course, as a distribution, we could try to leverage Fedora personal
repositories for such uses or try to help maintainer to provide
up-to-date packages if possible.

My 2cts,
H.

[Neal Becker]

Haïkel Guémar wrote:

>  Le 10/02/2012 19:20, Neal Becker a écrit :
> >
> > Yes, but trying to be helpful to users is a fedora packaging/rpm issue.
> >
>  
>  Generally speaking, mixing RPM packages with <insert your favorite
>  development platform here> native packages is not doing a favor to our
>  users.
>  I agree that some users may need more up-to-date modules, but isolated
>  environments are a better alternative in these cases.
>  
>  Off course, as a distribution, we could try to leverage Fedora personal
>  repositories for such uses or try to help maintainer to provide
>  up-to-date packages if possible.
>  
>  My 2cts,
>  H. 
I believe we can try to cooperate better here, or at least fail more gracefully.

[Matej Cepl]

On 10.2.2012 18:09, 80 wrote:
>  Python guidelines recommends that packagers installs python eggs
> using
>  distutils (python setup.py install as recommended in guidelines) while
>  pip use the same install method as easy_install (provided by
>  setuptools/distribute). The former one install egg metadata as a file,
>  the latter as a directory, that's not a packaging/rpm issue. 
a) I don't think the answer “Then don’t do it” is a good one. Some other 
Fedora-packaged languages (Perl comes to mind) allow three levels (or 
maybe even four) of installation of packages (in CPAN meaning of the 
word), system-wide-RPM-packaged, system-wide-unpackaged (to 
/usr/local/*), and per-user-in-$HOME. Not sure how it is with Ruby and 
PHP, but I believe this should be a standard in all major 
Fedora-packaged languages.
b) distutils v. setuptools conflict is just an unfortunate testimony of 
immature bad state of the Python upstream packaging, but it seems to me 
that generally Python world is moving towards setuptools. Shouldn't we 
follow the suite and move towards setuptools as well?
c) If we want to have as many Python packages packaged in RPMs (the 
terminology is going to kill me soon) do we have some pip2spec (in the 
same manner as there is cpan2spec)?

Best,

Matěj

[Neal Becker]

Matej Cepl wrote:

>  On 10.2.2012 18:09, 80 wrote:
> > Python guidelines recommends that packagers installs python eggs using
> > distutils (python setup.py install as recommended in guidelines) while
> > pip use the same install method as easy_install (provided by
> > setuptools/distribute). The former one install egg metadata as a file,
> > the latter as a directory, that's not a packaging/rpm issue.
>  
>  a) I don't think the answer “Then don’t do it” is a good one. Some other
>  Fedora-packaged languages (Perl comes to mind) allow three levels (or
>  maybe even four) of installation of packages (in CPAN meaning of the
>  word), system-wide-RPM-packaged, system-wide-unpackaged (to
>  /usr/local/*), and per-user-in-$HOME. Not sure how it is with Ruby and
>  PHP, but I believe this should be a standard in all major
>  Fedora-packaged languages.
>  b) distutils v. setuptools conflict is just an unfortunate testimony of
>  immature bad state of the Python upstream packaging, but it seems to me
>  that generally Python world is moving towards setuptools. Shouldn't we
>  follow the suite and move towards setuptools as well?
>  c) If we want to have as many Python packages packaged in RPMs (the
>  terminology is going to kill me soon) do we have some pip2spec (in the
>  same manner as there is cpan2spec)?
>  
>  Best,
>  
>  Matěj
>   
We will never have all python packages that users may want to try out packaged, 
nor should we want to.  I will only package items for Fedora that I have used 
enough to convince myself that they are:

1. useful enough in general
2. likely to continue to be developed/maintained

I always try out packages first via easy_install or pip (after checking they are 
not already available via yum).

Yes, I could try things via virtualenv, but I'm just not in that habit - and I 
suspect many others are in the same boat.

[Toshio Kuratomi]

On Sat, Feb 11, 2012 at 09:00:55AM -0500, Neal Becker wrote:
>  
>  I always try out packages first via easy_install or pip (after checking they are 
>  not already available via yum).
>  
>  Yes, I could try things via virtualenv, but I'm just not in that habit - and I 
>  suspect many others are in the same boat.
>   
For this use case, why are you installing into
/usr/lib{64,}/python2.7/site-packages ?

~/.local/lib/python2.7/site-packages sounds like it would suit this use much
better.

For pip, the command line to do that is:

pip-python --user PACKAGENAME

(I think there's an easy_install equivalent but I try to avoid setuptools
stuff when I can :-)

I believe that that directory is set by upstream python so it should work
out of the box on Fedora and other systems.

-Toshio

[Neal Becker]

Toshio Kuratomi wrote:

>  On Sat, Feb 11, 2012 at 09:00:55AM -0500, Neal Becker wrote:
> > 
> > I always try out packages first via easy_install or pip (after checking they
> > are not already available via yum).
> > 
> > Yes, I could try things via virtualenv, but I'm just not in that habit - and
> > I suspect many others are in the same boat.
> > 
>  For this use case, why are you installing into
>  /usr/lib{64,}/python2.7/site-packages ?
>  
>  ~/.local/lib/python2.7/site-packages sounds like it would suit this use much
>  better.
>  
>  For pip, the command line to do that is:
>  
>  pip-python --user PACKAGENAME
>  
>  (I think there's an easy_install equivalent but I try to avoid setuptools
>  stuff when I can :-)
>  
>  I believe that that directory is set by upstream python so it should work
>  out of the box on Fedora and other systems.
>  
>  -Toshio 
That sounds like a good suggestion - thanks!

[Toshio Kuratomi]

On Sat, Feb 11, 2012 at 01:52:08PM +0100, Matej Cepl wrote:
>  b) distutils v. setuptools conflict is just an unfortunate testimony
>  of immature bad state of the Python upstream packaging, but it seems
>  to me that generally Python world is moving towards setuptools.
>  Shouldn't we follow the suite and move towards setuptools as well?
>  
Upstream (python core) is doing everything within its power to move away
from setuptools, actually.  The problem with setuptools is that the upstream
author is a pain to work with and the upstream code is poorly written.

The problem with moving away from setuptools is that setuptools puts several
different types of things together (metadata, plugins, package building and
installing, selection and loading between different versions of a library,
installation of pre-packaged modules)... and there's only clear replacements
for some of those.

Python-3.3 offers the packaging module in the stdlib to replace the metadata
and package building/installing portion.  There's a backport of that for
python2 and earlier python3 which is the distutils2 module.  I looked at it
but the last release doesn't work with multilib.  I haven't had time to look
at a snapshot yet (people installing on MacOSX report that the latest
release doesn't work for them but the latest snapshot does... so a snapshot
may be the way to go anyhow.)

-Toshio

[Pierre-Yves Chibon]

On Sat, 2012-02-11 at 13:52 +0100, Matej Cepl wrote:
>  c) If we want to have as many Python packages packaged in RPMs (the 
>  terminology is going to kill me soon) do we have some pip2spec (in the
>  same manner as there is cpan2spec)?  
Reading this I wrote that: https://github.com/pypingou/pypi2spec
It is still pretty rough and far from polish but it does the basis
things (with the exception of retrieving the License information for
which I'm thinking that I'll have to parse the setup.py).

Hope this can help,
Pierre

[Toshio Kuratomi]

On Fri, Feb 10, 2012 at 11:38:20AM -0500, Neal Becker wrote:
>  80 wrote:
>  
>  > 2012/2/10 Neal Becker <ndbecker2(a)gmail.com&gt;:
>  >> I've seen this repeatedly, with often very serious consequences (complete
>  >> failure of update from f15->f16 for example).
>  >>
>  >> Between packages installed via pip, and packages installed via yum, some
>  >> packages seem to switch between
>  >>
>  >> e.g.,
>  >> numpy-1.6.1-py2.7.egg-info
>  >>
>  >> being installed as a file and the same name being installed as a directory.
>  >> This can cause subsequent rpm update (via cpio) to abort.
>  >>
>  >> I don't know what's causing this, but IMO this is a serious problem.
>  >>
>  >> --
>  >> devel mailing list
>  >> devel(a)lists.fedoraproject.org
>  >> https://admin.fedoraproject.org/mailman/listinfo/devel
>  > 
>  > Never use pip outside an isolated environment (use virtualenv)
>  > 
>  > H.
>  
>  Really?  This is the only answer?  Can't we tweek rpm/yum to accomodate this?  
>  Does anyone understand what is causing it?  Why would pip install the egg-info 
>  differently than rpm?
>  
If a package uses distutils (from the stdlib) to build, it will create an
egg-info file with the metadata.  If the package is installed with
setuptools (from python-setuptools), it will create and egg-info directory
with the metadata.

Some upstream packages allow you to use either distutils or setuptools with
code like:

try:
    from setuptools import setup
except ImportError:
    from distutils.core import setup

(numpy is more complex than this.... looks like with numpy, pip imports
setuptools and therefore the numpy setup.py uses setuptools' setup whereas
python setup.py [COMMAND] does not import setuptools so it uses distutils)

This has the unhappy side effect that sometimes the package is installed
with the egg-info as a file adn sometimes as a directory.

rpm/yum cannot accomodate this easily.  I thought that Panu had posted
a good explanation to the mailing lists many years ago but my search skills
are failing me right now.  IIRC, it's basically that in your rpm
transaction, rpm must install the new archive before removing the old one.
If the archive includes a file that has to replace a directory, this has no
way of succeeding.  (What if the directory contains files, for instance).

Now, there is a way out in this case that doesn't involve the rpm program.
In this case, the numpy rpm could contain either a file or a directory for
the egg-info.  It presently contains a file and thus the failure.  If you
can figure out how to make numpy create directory egg-info in this case,
you'll be able to make an rpm that will install over a pip local install.

Another option would be to configure python (via the PYTHONPATH environment
variable probably) to read modules from
/usr/local/lib{64,}/python2.7/site-packages/ or another directory that isn't
for the OS vendor.  Then have pip install numpy into there.

-Toshio

[Alexander Boström]

fre 2012-02-10 klockan 11:38 -0500 skrev Neal Becker: 
>  80 wrote: 
>  > Never use pip outside an isolated environment (use virtualenv)
>  > 
>  > H.
>  
>  Really?  This is the only answer? 
Does it help if we add that virtualenv is really useful? You can use it
to install a completely new copy of an application, including upgraded
dependencies, and then switch to that new virtualenv atomically and
reversibly.

/Alexander

[Adam Jackson]

On 2/10/12 11:32 AM, Neal Becker wrote:
>  I've seen this repeatedly, with often very serious consequences
> (complete
>  failure of update from f15->f16 for example).
>
>  Between packages installed via pip, and packages installed via yum, some
>  packages seem to switch between
>
>  e.g.,
>  numpy-1.6.1-py2.7.egg-info
>
>  being installed as a file and the same name being installed as a directory.
>  This can cause subsequent rpm update (via cpio) to abort.
>
>  I don't know what's causing this, but IMO this is a serious problem. 
Doctor, it hurts when I do this.

- ajax

[Pete Zaitcev]

On Fri, 10 Feb 2012 11:32:31 -0500
Neal Becker <ndbecker2(a)gmail.com&gt; wrote:

>  I've seen this repeatedly, with often very serious consequences
> (complete 
>  failure of update from f15->f16 for example). 
Yeah... My answer to this was:
   \o/ PACKAGE ALL THE THIIIINGS !!! O//

So I got git-review included into Fedora, which was the only pip-installed
thing on my laptop, then did yum remove python-pip, voila!

If every concerned hacker took one package... well, you know. Think globally,
act locally.

-- Pete