Why do Fedora python packages install in site-packages?

[Richard W.M. Jones]

I'm confused why Fedora Python packages get installed in
/usr/lib/python*/site-packages/.

Looking at the upstream Python documentation[1] it seems this
directory is reserved for packages installed by the local system
administrator.  (Similar to the Perl distinction between
INSTALLDIRS=vendor / INSTALLDIRS=site / INSTALLDIRS=core [2])

Not directly relevant, but Debian reserves[3] site-packages for the
system administrator, and puts Python packages in its own directory
(/usr/lib/python*/dist-packages).

Is this a mistake or am I not understanding the distinction?

Rich.

[1] http://docs.python.org/install/index.html
[2] http://rt.cpan.org/Public/Bug/Display.html?id=47396#txn-629736
[3] http://www.debian.org/doc/packaging-manuals/python-policy/ch-python.html

-- 
Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones
Read my programming blog: http://rwmj.wordpress.com
Fedora now supports 80 OCaml packages (the OPEN alternative to F#)
http://cocan.org/getting_started_with_ocaml_on_red_hat_and_fedora

[Toshio Kuratomi]

On Thu, Dec 09, 2010 at 12:44:36PM +0000, Richard W.M. Jones wrote:
>  
>  I'm confused why Fedora Python packages get installed in
>  /usr/lib/python*/site-packages/.
>  
>  Looking at the upstream Python documentation[1] it seems this
>  directory is reserved for packages installed by the local system
>  administrator.  (Similar to the Perl distinction between
>  INSTALLDIRS=vendor / INSTALLDIRS=site / INSTALLDIRS=core [2])
>   
I don't see where the distinction is being made between local system admin
and vendor in that document.  Could you give a better link or a search term?

Also, on an FHS compliant Linux system, the system administrator would not
use /usr/lib/pythonX.Y/site-packages anyway.  They'd use something like
/usr/local/lib/pythonX.Y/site-packages.  We should, perhaps be arranging for
such a directory to be i nthe default path -- if so, probably just need
a bug opened against the python package.  (Although that will slow pyhton
down.... python will then have another directory that it has to search
through to find its modules.)

>  Not directly relevant, but Debian reserves[3] site-packages for the
>  system administrator, and puts Python packages in its own directory
>  (/usr/lib/python*/dist-packages).
>  
>  Is this a mistake or am I not understanding the distinction?
>   
You're not understanding the distinction but I'm not either...  If you
read
their documentation again you'll see a few things:

Debian switches to dist-packages with python2.6; python2.5 and below
continues to use site-packages.

System admins are actually reserved directories in /usr/local; dist-packages
and site-packages has nothing to do with system admins in their guidelines
as this directory follows the same rule as the one for packagers (python2.6+
vs python2.5-)

In this setup, it seems like Debian is not putting anything in
site-packages.  Without more information, their use of dist-packages just
seems arbitrary and non-standard.

-Toshio

[Richard W.M. Jones]

On Thu, Dec 09, 2010 at 09:40:04AM -0800, Toshio Kuratomi wrote:
>  On Thu, Dec 09, 2010 at 12:44:36PM +0000, Richard W.M. Jones wrote:
>  > 
>  > I'm confused why Fedora Python packages get installed in
>  > /usr/lib/python*/site-packages/.
>  > 
>  > Looking at the upstream Python documentation[1] it seems this
>  > directory is reserved for packages installed by the local system
>  > administrator.  (Similar to the Perl distinction between
>  > INSTALLDIRS=vendor / INSTALLDIRS=site / INSTALLDIRS=core [2])
>  > 
>  I don't see where the distinction is being made between local system admin
>  and vendor in that document.  Could you give a better link or a search term? 
Maybe I can explain the Perl situation better.

"site" (on Fedora this is /usr/local/share/perl5) is where the system
admin installs packages using:

  perl Makefile.PL ; make ; sudo make install

"vendor" (/usr/share/perl5/vendor_perl from Rawhide onwards) is where
the package manager puts Perl packages which aren't in the core Perl.

"core" (/usr/share/perl5) is where core parts of Perl go and packages
which are shipped in the perl tarball.

The idea AIUI is that admin-installed packages survive Perl upgrades
or cases where Perl is removed and reinstalled.  If you use the 'cpan'
command line tool on Fedora, the packages get installed in "site".

>  Also, on an FHS compliant Linux system, the system administrator
> would not
>  use /usr/lib/pythonX.Y/site-packages anyway.  They'd use something like
>  /usr/local/lib/pythonX.Y/site-packages. 
Indeed, and Debian puts site-packages under /usr/local as you've
described, which is where I came across the problem in the first place
(trying to build a package for Debian and noticing this is different
from Fedora).

Debian also creates /usr/share/pyshared but I've yet to understand
exactly why.

[...]

>  In this setup, it seems like Debian is not putting anything in
>  site-packages.  Without more information, their use of dist-packages just 
>  seems arbitrary and non-standard. 
I think the Perl case makes some sense, if one accepts that there is a
difference between "core" and "vendor" (maybe this difference is too
artificial).  Upstream Python documentation doesn't really cover the
case of package managers at all.

Rich.

-- 
Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones
libguestfs lets you edit virtual machines.  Supports shell scripting,
bindings from many languages.  http://et.redhat.com/~rjones/libguestfs/
See what it can do: http://et.redhat.com/~rjones/libguestfs/recipes.html

[Toshio Kuratomi]

On Thu, Dec 09, 2010 at 08:30:40PM +0000, Richard W.M. Jones wrote:
>  On Thu, Dec 09, 2010 at 09:40:04AM -0800, Toshio Kuratomi wrote:
>  > On Thu, Dec 09, 2010 at 12:44:36PM +0000, Richard W.M. Jones wrote:
>  > > 
>  > > I'm confused why Fedora Python packages get installed in
>  > > /usr/lib/python*/site-packages/.
>  > > 
>  > > Looking at the upstream Python documentation[1] it seems this
>  > > directory is reserved for packages installed by the local system
>  > > administrator.  (Similar to the Perl distinction between
>  > > INSTALLDIRS=vendor / INSTALLDIRS=site / INSTALLDIRS=core [2])
>  > > 
>  > I don't see where the distinction is being made between local system admin
>  > and vendor in that document.  Could you give a better link or a search term?
>  
>  Maybe I can explain the Perl situation better.
>  
>  "site" (on Fedora this is /usr/local/share/perl5) is where the system
>  admin installs packages using:
>  
>    perl Makefile.PL ; make ; sudo make install
>  
>  "vendor" (/usr/share/perl5/vendor_perl from Rawhide onwards) is where
>  the package manager puts Perl packages which aren't in the core Perl.
>  
>  "core" (/usr/share/perl5) is where core parts of Perl go and packages
>  which are shipped in the perl tarball.
>  
>  The idea AIUI is that admin-installed packages survive Perl upgrades
>  or cases where Perl is removed and reinstalled.  If you use the 'cpan'
>  command line tool on Fedora, the packages get installed in "site".
>   
Oops -- I shouldn't have quoted the perl stuff in that -- My question was
targetted at the upstream python documentation.  I'll talk a little more
about the perl comparison at the bottom.

>  > Also, on an FHS compliant Linux system, the system administrator
> would not
>  > use /usr/lib/pythonX.Y/site-packages anyway.  They'd use something like
>  > /usr/local/lib/pythonX.Y/site-packages.
>  
>  Indeed, and Debian puts site-packages under /usr/local as you've
>  described, which is where I came across the problem in the first place
>  (trying to build a package for Debian and noticing this is different
>  from Fedora).
>  
>  Debian also creates /usr/share/pyshared but I've yet to understand
>  exactly why.
>   
I'd have to look and check if this is the same thing I'm thinking of but
Debian regularly ships multiple versions of the python interpreter.  As part
of that, they ship the *.py files in /usr/share/pyshared.  Then they have
a scriptlet that runs at install time to symlink those into all the
/usr/lib/pythonX.Y that are installed and byte compile them.

>  [...]
>  
>  > In this setup, it seems like Debian is not putting anything in
>  > site-packages.  Without more information, their use of dist-packages just 
>  > seems arbitrary and non-standard.
>  
>  I think the Perl case makes some sense, if one accepts that there is a
>  difference between "core" and "vendor" (maybe this difference is too
>  artificial).  Upstream Python documentation doesn't really cover the
>  case of package managers at all.
>   
<nod>  In python the vendor and core stuff is separated but the vendor
stuff
nests within the core directory.  If you take a look at /usr/lib64/pythonX.Y
you'll see a bunch of *.py files directly in there.  Those are all "core".
Many of the directories in there also contain "core" files.  The directory
/usr/lib64/pythonX.Y/site-packages is where vendor supplied third party
stuff is stored.

From a brief look we don't seem to have configured a default directory for
system admins to use in /usr/local/ but that seems like a good addition to
me.  Open a bug for dmalcolm to look at for that.

(And yes, upstream python, like many upstreams that don't target Linux
primarily, doesn't think of the FHS and just assumesthe sysadmin installs to
/usr/lib/pythonX.Y/site-packages.)

-Toshio

[Richard W.M. Jones]

On Thu, Dec 09, 2010 at 03:38:42PM -0800, Toshio Kuratomi wrote:
>  I'd have to look and check if this is the same thing I'm
> thinking of
>  but Debian regularly ships multiple versions of the python
>  interpreter.  As part of that, they ship the *.py files in
>  /usr/share/pyshared.  Then they have a scriptlet that runs at
>  install time to symlink those into all the /usr/lib/pythonX.Y that
>  are installed and byte compile them. 
I see, that makes more sense now.

>  From a brief look we don't seem to have configured a default
>  directory for system admins to use in /usr/local/ but that seems
>  like a good addition to me.  Open a bug for dmalcolm to look at for
>  that. 
https://bugzilla.redhat.com/show_bug.cgi?id=662034

Rich.

-- 
Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones
Read my programming blog: http://rwmj.wordpress.com
Fedora now supports 80 OCaml packages (the OPEN alternative to F#)
http://cocan.org/getting_started_with_ocaml_on_red_hat_and_fedora