Some Python Packaging Questions

[Jeff Pitman]

Hi:

I was wondering what your opinions on a going-forward solution to 
Requires in Python packages.  Especially with respect to the newer 
"Provides: python-abi = %{pybasever}".

This previous thread is relevant:
http://www.redhat.com/archives/fedora-devel-list/2004-May/msg00698.html

Some packages are using this as a Requires:, but not all.  Fedora.us 
implements a slightly different scheme in the spec template.  If you 
were coming from a "forget what was done in past Python packaging" 
viewpoint, would you move everything over to "python-abi"?  If not, 
why?

Oh, and what's the difference between %{pybasever} and %{pyver}?  Is it 
like this:

%{pyver} = 2.3.4
%{pybasever} = 2.3

Lastly, in the fedora.us spec template, I still don't see how 
differentiation on the directories produces anything useful:
# For noarch packages:
Requires:       %{python_sitelib}
# For arch-dependent packages:
Requires:       %{python_sitearch}

On my systems, this will always expand to the same versioned library 
directory. Does this have to do with 64-bit somehow?

Thanks for your input!

take care,
-- 
-jeff

[Ville Skyttä]

On Mon, 2004-08-23 at 09:27, Jeff Pitman wrote:

>  This previous thread is relevant:
>  http://www.redhat.com/archives/fedora-devel-list/2004-May/msg00698.html
>  
>  Some packages are using this as a Requires:, but not all.  Fedora.us 
>  implements a slightly different scheme in the spec template. 
FYI: the spec template used distutils.sysconfig.get_python_version()
which is available only in Python >= 2.3; that has been changed to the
sys.version[:3] approach in the latest version which should be published
soon (bugzilla.fedora.us 1401).  (The fedora.us python-forward-compat
package provides python-abi stuff for < FC2.)

>  Lastly, in the fedora.us spec template, I still don't see how 
>  differentiation on the directories produces anything useful:
>  # For noarch packages:
>  Requires:       %{python_sitelib}
>  # For arch-dependent packages:
>  Requires:       %{python_sitearch}
>  
>  On my systems, this will always expand to the same versioned library 
>  directory. Does this have to do with 64-bit somehow? 
Yes.  sitelib should always point to the architecture independent
location (/usr/lib/... in current python packages), and sitearch to the
arch-dependent one (%{_libdir}/... in current python packages).  With
x86_64, there was a bug in the python package that resulted both of the
above being %{_libdir} which again caused borkage (in a way that those
macros expanded to wrong locations even if the sofware was installed to
the correct ones) when a noarch Python extension package was built on
x86_64.  That has been fixed in Rawhide.

Note that these variables are usually for convenience only for eg
%install and %files sections, they are not passed to builds using
distutils.  They are there to increase the probability that extension
packages just need to be rebuilt if there are changes in the way Python
itself is packaged wrt directory layout, or to build for a custom Python
setup, without having to do any specfile tweaking.

As of now, one could fairly safely use /usr/lib*/python%{pyver}/...
everywhere, but I guess one beautiful day arch-independent Python
extensions will be installed in /usr/share/python%{pyver}/...  And as
noted above, one can redefine %__python to build for a custom Python
installation in which case we can't assume much at all.

[Jeff Pitman]

On Monday 23 August 2004 15:19, Ville Skyttä wrote:
>  On Mon, 2004-08-23 at 09:27, Jeff Pitman wrote:
>  > Some packages are using this as a Requires:, but not all. 
>  > Fedora.us implements a slightly different scheme in the spec
>  > template.
>
>  FYI: the spec template used distutils.sysconfig.get_python_version()
>  which is available only in Python >= 2.3; that has been changed to
>  the sys.version[:3] approach in the latest version which should be
>  published soon (bugzilla.fedora.us 1401).  (The fedora.us
>  python-forward-compat package provides python-abi stuff for < FC2.) 
Fair enough.  Can you elaborate on the python-forward-compat package a 
little bit more?  And why would it be useful for < FC2 where python 2.2 
never used python-abi = 2.2, etc.?

>  > On my systems, this will always expand to the same versioned
>  > library directory. Does this have to do with 64-bit somehow?
>
>  Yes.  sitelib should always point to the architecture independent
>  location (/usr/lib/... in current python packages), and sitearch to
>  the arch-dependent one (%{_libdir}/... in current python packages). 
>  With x86_64, there was a bug in the python package that resulted both
>  of the above being %{_libdir} which again caused borkage (in a way
>  that those macros expanded to wrong locations even if the sofware was
>  installed to the correct ones) when a noarch Python extension package
>  was built on x86_64.  That has been fixed in Rawhide. 
So, out of my ignorance, %{_libdir} = /usr/lib64 on 64-bit platforms AND 
we have to worry about .so(64bit) for lib suffixes.  I guess I should 
get a 64-bit system to check this out, but from a surface-level it 
seems to be an awful hack.  I guess the suffixes are needed for pkgs 
that require /usr/lib/ or something...

Anyway, the reason I think it's the same is because i386 == noarch under 
this scheme.

>  As of now, one could fairly safely use /usr/lib*/python%{pyver}/...
>  everywhere, but I guess one beautiful day arch-independent Python
>  extensions will be installed in /usr/share/python%{pyver}/...  And as
>  noted above, one can redefine %__python to build for a custom Python
>  installation in which case we can't assume much at all. 
Why would this be even useful?  Segregation of compiled extensions (.so) 
and pure-python (.py) files?  sys.path would have to be modified anyway 
and certainly the user really doesn't care.  Zero gains, really, at a 
high-level view.  But, what do we gain really at a low-level view?  I 
don't do nefarious /usr NFS mounts and segregate based on architectures 
in a disparate environment, so I don't know what the advantages are.  
Help me out with why would this be a "beautiful day"?

take care,
-- 
-jeff

[Ville Skyttä]

On Mon, 2004-08-23 at 11:34, Jeff Pitman wrote:

>  Can you elaborate on the python-forward-compat package a 
>  little bit more?  And why would it be useful for < FC2 where python 2.2 
>  never used python-abi = 2.2, etc.? 
python-forward-compat provides python-abi = 2.2.  It is _only_ useful
for < FC2 where the python package does not provide any python-abi, and
it "binds" itself to the installed python version, IIRC currently by
requiring /usr/bin/python2.2.  The usefulness is that one can use the
same specfiles/SRPMS of python module packages regarding this bit for
many distro versions.  AFAIK, depending on python-abi seems to be the
"official" way of requiring a certain version of Python, or a compatible
one (my guess: in the sense that a "compatible" one would have eg. the
site-packages dir of the version it is compatible with in its default
load path).

>  So, out of my ignorance, %{_libdir} = /usr/lib64 on 64-bit platforms
Yes.

>  AND we have to worry about .so(64bit) for lib suffixes. I guess I
> should 
>  get a 64-bit system to check this out, but from a surface-level it 
>  seems to be an awful hack.  I guess the suffixes are needed for pkgs 
>  that require /usr/lib/ or something... 
I'm not sure I understand the above.  But see below.

>  Anyway, the reason I think it's the same is because i386 ==
> noarch under 
>  this scheme. 
i386 == noarch?  You mean that both i386 and noarch python packages
usually install into /usr/lib/python%{pyver}/... ?  The point is that if
one builds the same noarch package on i386 or x86_64 or whatever using
the default configurations of the same distro version, the results
should be the same, including the install locations (otherwise it's not
a noarch package).  This does not hold true for arch-dependent packages,
and this is nothing specific to Python extension packages.

>  > As of now, one could fairly safely use
> /usr/lib*/python%{pyver}/...
>  > everywhere, but I guess one beautiful day arch-independent Python
>  > extensions will be installed in /usr/share/python%{pyver}/...  And as
>  > noted above, one can redefine %__python to build for a custom Python
>  > installation in which case we can't assume much at all.
>  
>  Why would this be even useful?  Segregation of compiled extensions (.so) 
>  and pure-python (.py) files? 
Whatever the reasons, the decision has already been made.  AFAIK
upstream Python distutils has differentiated between the two already for
a long time, and it certainly does so nowadays.  It seems to be just a
coincidence that the two point to the same location for 32-bit archs in
Python packages distributed by Red Hat (and perhaps it's the default in
upstream Python sources too, haven't checked, but nevertheless these
locations are two different "entities").  There's no such coincidence
with Python stuff from Red Hat for 64-bit archs AFAIK (this info is from
reading the patches RH applies to the Python package for 64-bit archs, I
don't have such a box either).

>    sys.path would have to be modified anyway 
>  and certainly the user really doesn't care. 
No, the user should not care and nobody has to modify anything to take
this into account, things must Just Work.  The locations where distutils
installs extensions by default need to be in the default load paths of
that Python, otherwise I'd argue it's a bug in the Python package.

>    Zero gains, really, at a 
>  high-level view.  But, what do we gain really at a low-level view?  I 
>  don't do nefarious /usr NFS mounts and segregate based on architectures 
>  in a disparate environment, so I don't know what the advantages are. 
I don't have that experience either, so someone else will have to fill
in here.  Evil or not, if one does not do those mounts, I guess there's
not much to gain at all.  But the reality is that the distinction is
there already, and spec templates and "best practices"-like packaging
instructions should take that into account.

>  Help me out with why would this be a "beautiful day"?
"beautiful" did not come through as intended, it was a direct
translation from a Finnish saying.  Think "One day you wake up and
notice that ...".

[Jeff Pitman]

On Monday 23 August 2004 21:56, Ville Skyttä wrote:
>  On Mon, 2004-08-23 at 11:34, Jeff Pitman wrote:
>  > AND we have to worry about .so(64bit) for lib suffixes. I guess I
>  > should get a 64-bit system to check this out, but from a
>  > surface-level it seems to be an awful hack.  I guess the suffixes
>  > are needed for pkgs that require /usr/lib/ or something...
>
>  I'm not sure I understand the above.  But see below. 
I've seen this floating around a few spec files:
%ifarch ia64
%define depsuffix ()(64bit)
%else
%define depsuffix %{nil}
%endif

Then, later:
libwhatever.so%{depsuffix}

But, maybe this is a very old way to do it and they now pile it on 
inside of /usr/lib64/.  I have no idea.  Someone needs to donate 64-bit 
systems to both of us!! ;)

>  > > As of now, one could fairly safely use
>  > > /usr/lib*/python%{pyver}/... everywhere, but I guess one
>  > > beautiful day arch-independent Python extensions will be
>  > > installed in /usr/share/python%{pyver}/...  And as noted above,
>  > > one can redefine %__python to build for a custom Python
>  > > installation in which case we can't assume much at all.
>  >
>  > Why would this be even useful?  Segregation of compiled extensions
>  > (.so) and pure-python (.py) files?
>
>  Whatever the reasons, the decision has already been made.   
I would never aspire to change any decisions.  I am just trying to 
understand the policy a little bit better.

>  AFAIK upstream Python distutils has differentiated between the two
>  already for a long time, and it certainly does so nowadays.   
I would actually presume that the differentiation is rooted in autoconf 
and just carried over into distutils as a matter of reference.  The 
get_python_lib() code calculates what the prefix is by the following 
statement:

	prefix = plat_specific and EXEC_PREFIX or PREFIX

Running ./configure --help on any autoconf system, you see that:

Installation directories:
--prefix=PREFIX     install architecture-independent files in PREFIX
                          [/usr/local]
--exec-prefix=EPREFIX   install architecture-dependent files in EPREFIX
                          [PREFIX]

I have only been working the Unix scene for 7, 8 years and I've not seen 
much use of ./configure --exec-prefix.  Are we to presume that RPM 
proper should support this as a specific tag since autoconf has had 
this for a long time?  %{_exec_prefix} == %{_prefix}

>  It seems to be 
>  just a coincidence that the two point to the same location for 32-bit
>  archs in Python packages distributed by Red Hat (and perhaps it's the
>  default in upstream Python sources too, haven't checked, but
>  nevertheless these locations are two different "entities").  There's
>  no such coincidence with Python stuff from Red Hat for 64-bit archs
>  AFAIK (this info is from reading the patches RH applies to the Python
>  package for 64-bit archs, I don't have such a box either). 
Why?  Is it because 64-bit can simultaneously run 32-bit libraries and 
64-bit libraries at the same time and they're concerned over name 
clashes?  Do they segregate 32-bit and 64-bit libs into different dirs?  
Maybe some programs can only have half their libs in 32-bit and the 
other half in 64-bit because of compilation issues.

It just seems all a waste of time, from a ignorant persons point of 
view.  You've really perked my interest ... now I have to get my paws 
on a 64-bit box.  I've got to see why anyone would care to make such 
seemingly anal retentive distinctions and 
why /usr/lib64/python2.3/site-packages 
or /usr/lib/python2.3/site-packages are somehow different entities.

I've not seen anyone use --exec-prefix, even in the Rawhide python.spec. 
So, not sure how get_python_lib() != get_python_lib(1) would ever occur 
in a standard distribution.

>  >   sys.path would have to be modified anyway
>  > and certainly the user really doesn't care.
>
>  No, the user should not care and nobody has to modify anything to
>  take this into account, things must Just Work.  The locations where
>  distutils installs extensions by default need to be in the default
>  load paths of that Python, otherwise I'd argue it's a bug in the
>  Python package. 
*Sigh*  I've still have not put a finger on this.  It seems people like 
to throw Python "apps" in /usr/share/app_name and libraries 
into /usr/lib/python2.3/site-packages.  I'm not 
sure /usr/share/app_name really makes any sense all the time.  Trick 
question: is epydoc an app or is it a library?

thanks,
-- 
-jeff

[Jeff Pitman]

On Monday 23 August 2004 21:56, Ville Skyttä wrote:
>  But the reality is that the distinction is
>  there already, and spec templates and "best practices"-like packaging
>  instructions should take that into account. 
Oh, and one other quick question.  

Why not use --record instead of worrying about python_sitelib and 
python_sitearch?

%install
%{__python} setup.py install --root=$RPM_BUILD_ROOT 
--record=INSTALLED_FILES

%files -f INSTALLED_FILES
%defattr(-,root,root)
%doc README INSTALL etc. etc.


take care,
-- 
-jeff

[Michael Schwendt]

On Tue, 24 Aug 2004 10:17:48 +0800, Jeff Pitman wrote:

>  Oh, and one other quick question.  
>  
>  Why not use --record instead of worrying about python_sitelib and 
>  python_sitearch?
>  
>  %install
>  %{__python} setup.py install --root=$RPM_BUILD_ROOT 
>  --record=INSTALLED_FILES
>  
>  %files -f INSTALLED_FILES
>  %defattr(-,root,root)
>  %doc README INSTALL etc. etc. 
That doesn't include directories (IIRC) and would need extra effort
when you want to %ghost *.pyo files.

[Jeff Pitman]

On Tuesday 24 August 2004 15:36, Michael Schwendt wrote:
>  That doesn't include directories (IIRC) and would need extra
> effort
>  when you want to %ghost *.pyo files. 
That is the suck!  Thanks for pointing that out.  That would cause all 
sorts of littering.  Maybe we should push a patch to upstream distutils 
that includes directories or a more well-rounded approach that includes 
file options.  --rpm-record or something would be a neat option to 
setup.py.

I was going to recommend this one liner:

sed 's/\(.*\.pyo\)/%ghost \1/' < INSTALLED_FILES > 
INSTALLED_FILES.ghostbusted

%files -f INSTALLED_FILES.ghostbusted
%doc README INSTALL LICENSE
%dir %{_libdir}/python?.?/site-packages/blah
%dir %{_libdir}/python?.?/site-packages/blah/subblah

But, the '%files -f' directive doesn't process %ghost as of RPM 4.3.1.  
Oh well, it's just more disk space anyway... still not convinced we 
absolutely need to ghost pyo files.

thanks,
-- 
-jeff

[Jeff Pitman]

On Tuesday 24 August 2004 17:10, Jeff Pitman wrote:
>  I was going to recommend this one liner:
>
>  sed 's/\(.*\.pyo\)/%ghost \1/' < INSTALLED_FILES >
>  INSTALLED_FILES.ghostbusted
>
>  %files -f INSTALLED_FILES.ghostbusted
>  %doc README INSTALL LICENSE
>  %dir %{_libdir}/python?.?/site-packages/blah
>  %dir %{_libdir}/python?.?/site-packages/blah/subblah 
For directories, I found something similar to this useful:

find %{_datadir}/%{name}-%{version} -type d |
            while read d; do echo "%dir $d"; done >>INSTALLED_FILES
find %{_libdir}/python%{pybasever}/site-packages/foo -type d |
            while read d; do echo "%dir $d"; done >>INSTALLED_FILES

take care,
-- 
-jeff

[Ville Skyttä]

On Mon, 2004-08-23 at 18:32, Jeff Pitman wrote:

>  I've seen this floating around a few spec files:
>  %ifarch ia64
>  %define depsuffix ()(64bit)
>  %else
>  %define depsuffix %{nil}
>  %endif
>  
>  Then, later:
>  libwhatever.so%{depsuffix} 
Oh.  

>  But, maybe this is a very old way to do it and they now pile it on 
>  inside of /usr/lib64/.  I have no idea. 
It probably has something to do with x86_64 which can use 32-bit and
64-bit apps/libs but I guess they cannot be mixed or something (eg. a
64-bit Mozilla probably cannot use 32-bit plugins).

(But this stuff deserves a new thread, I bet it's not getting the
attention of the people who know it here.)

>  Someone needs to donate 64-bit systems to both of us!! ;) 
Yeah, send 'em over!

>  > AFAIK upstream Python distutils has differentiated between the
> two
>  > already for a long time, and it certainly does so nowadays.  
>  
>  I would actually presume that the differentiation is rooted in autoconf 
>  and just carried over into distutils as a matter of reference. 
Or Perl.

>  I have only been working the Unix scene for 7, 8 years and I've
> not seen 
>  much use of ./configure --exec-prefix. 
Me neither but the support is there for those who want to use it, and I
guess there's a reason to use it in some situations.  BTW --exec-prefix
seems to be fully supported also in rpm's default config, see %configure
and %{_*dir} in /usr/lib*/rpm/macros.

>  Why?  Is it because 64-bit can simultaneously run 32-bit libraries
> and 
>  64-bit libraries at the same time and they're concerned over name 
>  clashes?  Do they segregate 32-bit and 64-bit libs into different dirs?  
>  Maybe some programs can only have half their libs in 32-bit and the 
>  other half in 64-bit because of compilation issues. 
Perhaps.  AFAIK only x86_64 can use both 32- and 64-bit stuff though,
ia64 only the latter.

>  *Sigh*  I've still have not put a finger on this.  It seems
> people like 
>  to throw Python "apps" in /usr/share/app_name and libraries 
>  into /usr/lib/python2.3/site-packages.  I'm not 
>  sure /usr/share/app_name really makes any sense all the time. 
Certainly not all the time, but if that stuff is intended to be used
only within a certain executable and is not generally useful,
site-packages would not sound good to me.

>  Trick question: is epydoc an app or is it a library? 
I don't know epydoc, but based on the description on the homepage, I
could imagine both (eg. basic command line executables in /usr/bin, some
Python modules in site-packages that could be used by imaginary
additional frontends).

[Ville Skyttä]

On Tue, 2004-08-24 at 12:10, Jeff Pitman wrote:

>  But, the '%files -f' directive doesn't process %ghost as
> of RPM 4.3.1.   
How?  It works nicely in the fedora-rpmdevtools and cscope packages from
fedora.us on FC2.

[Jeff Pitman]

On Wednesday 25 August 2004 06:54, Ville Skyttä wrote:
>  > But, the '%files -f' directive doesn't process
> %ghost as of RPM
>  > 4.3.1.
>
>  How?  It works nicely in the fedora-rpmdevtools and cscope packages
>  from fedora.us on FC2. 
Sorry, my scriptlet was messed up.  It works, actually.

take care,
-- 
-jeff