Re: More Re: [Distutils] Compatibility of bdist_rpm with Fedora packaging instructions
by Stanley A. Klein
I developed a fix that can byte compile only the .py files in
site-packages. It involved creating my own version of
brp-python-bytecompile, preventing brp-python-bytecompile from running,
and running my-python-bytecompile instead.
The relevant statements in the spec file are as follows:
Before the description insert (from the Fedora packaging python page):
# Turn off the brp-python-bytecompile script
%global __os_install_post %(echo '%{__os_install_post}' | sed -e
's!/usr/lib[^[:space:]]*/brp-python-bytecompile[[:space:]].*$!!g')
At the end of the %install (following copying of the doc directories into
the buildroot) insert:
~/my-rpm-macros/my-python-bytecompile
The code for my-python-bytecompile is simply a copy of
brp-python-bytecompile with the final section commented out. That section
does files in directories other than site-packages. The code is below.
Stan Klein
----------------------------------------------------------------------
my-python-bytecompile
#!/bin/bash
errors_terminate=$2
# If using normal root, avoid changing anything.
if [ -z "$RPM_BUILD_ROOT" -o "$RPM_BUILD_ROOT" = "/" ]; then
exit 0
fi
# If we don't have a python interpreter, avoid changing anything.
default_python=${1:-/usr/bin/python}
if [ ! -x "$default_python" ]; then
exit 0
fi
# Figure out how deep we need to descend. We could pick an insanely high
# number and hope it's enough, but somewhere, somebody's sure to run into it.
depth=`(find $RPM_BUILD_ROOT -type f -name "*.py" -print0 ; echo /) | \
xargs -0 -n 1 dirname | sed 's,[^/],,g' | sort -u | tail -n 1 | wc -c`
if [ -z "$depth" -o "$depth" -le "1" ]; then
exit 0
fi
# .pyc/.pyo files embed a "magic" value, identifying the ABI version of
Python
# bytecode that they are for.
#
# The files below RPM_BUILD_ROOT could be targetting multiple versions of
# python (e.g. a single build that emits several subpackages e.g. a
# python26-foo subpackage, a python31-foo subpackage etc)
#
# Support this by assuming that below each /usr/lib/python$VERSION/, all
# .pyc/.pyo files are to be compiled for /usr/bin/python$VERSION.
#
# For example, below /usr/lib/python2.6/, we're targetting /usr/bin/python2.6
# and below /usr/lib/python3.1/, we're targetting /usr/bin/python3.1
shopt -s nullglob
for python_libdir in $RPM_BUILD_ROOT/usr/lib{,64}/python[0-9].[0-9]/ ;
do
python_binary=/usr/bin/$(basename $python_libdir)
real_libdir=${python_libdir/$RPM_BUILD_ROOT/}
echo "Bytecompiling .py files below $python_libdir using $python_binary"
# Generate normal (.pyc) byte-compiled files.
$python_binary -c 'import compileall, sys; sys.exit(not
compileall.compile_dir("'"$python_libdir"'", '"$depth"',
"'"$real_libdir"'", force=1, quiet=1))'
if [ $? -ne 0 -a 0$errors_terminate -ne 0 ]; then
# One or more of the files had a syntax error
exit 1
fi
# Generate optimized (.pyo) byte-compiled files.
$python_binary -O -c 'import compileall, sys; sys.exit(not
compileall.compile_dir("'"$python_libdir"'", '"$depth"',
"'"$real_libdir"'", force=1, quiet=1))'
if [ $? -ne 0 -a 0$errors_terminate -ne 0 ]; then
# One or more of the files had a syntax error
exit 1
fi
done
# Handle other locations in the filesystem using the default python
# implementation:
# Generate normal (.pyc) byte-compiled files.
#$default_python -c 'import compileall, re, sys; sys.exit (not
compileall.compile_dir("'"$RPM_BUILD_ROOT"'", '"$depth"', "/", 1,
re.compile(r"'"/bin/|/sbin/|/usr/lib(64)?/python[0-9]\.[0-9]"'"),
quiet=1))'
#if [ $? -ne 0 -a 0$errors_terminate -ne 0 ]; then
# # One or more of the files had a syntax error
# exit 1
#fi
#
# Generate optimized (.pyo) byte-compiled files.
#$default_python -O -c 'import compileall, re, sys; sys.exit(not
compileall.compile_dir("'"$RPM_BUILD_ROOT"'", '"$depth"', "/", 1,
re.compile(r"'"/bin/|/sbin/|/usr/lib(64)?/python[0-9]\.[0-9]"'"),
quiet=1))' > /dev/null
#if [ $? -ne 0 -a 0$errors_terminate -ne 0 ]; then
# # One or more of the files had a syntax error
# exit 1
#fi
exit 0
12 years
- 1
- 0
More Re: [Distutils] Compatibility of bdist_rpm with Fedora packaging instructions
by Stanley A. Klein
One additional issue: The system produces pyo and pyc files for all the
.py files it finds. That is good for the files that go into site-packages
because they are intended to be executed from there, but might not be so
good for documentation files such as examples and code-snippets that are
intended to be run or otherwise used in user-space.
The commands:
find . -type f -name *.pyc -exec rm -f {} \;
find . -type f -name *.pyo -exec rm -f {} \;
executed at some point in the process at the root of the default
documentation directory after the .pyc and .pyo files have been created
can remove them. However, I can't seem to figure out where to put the
statements. Also, might there be a way to prevent the byte compiling of
documentation files?
Stan Klein
12 years
- 1
- 0
Re: [Distutils] Compatibility of bdist_rpm with Fedora packaging instructions
by Stanley A. Klein
Tarek -
Thanks.
I got it working and want to document some of my findings. I'm cc:'ing
the Fedora Python list so they can take these issues into account in their
Python packaging instructions. My findings are as follows:
1. The source code management system was git. I needed to install
setuptools-git to get files recognized that were being maintained under
git.
2. I also needed to establish a MANIFEST.in file to ensure all relevant
files were included.
3. The setup.cfg statement under [bdist_rpm] of "doc_files =" doesn't
work if there are directories involved. This is a documented "gotcha" in
http://fedoraproject.org/wiki/How_to_create_an_RPM_package
They also advise avoiding use of INSTALLED_FILES.
Here is what I used in the spec file (it had to be edited for the
directories to be included in docs):
%install
python setup.py install --root=$RPM_BUILD_ROOT
cd $RPM_BUILD_ROOT
mkdir -p %{buildroot}%{_defaultdocdir}/%{name}-%{version}/
mkdir -p %{buildroot}%{_defaultdocdir}/%{name}-%{version}/docs
mkdir -p %{buildroot}%{_defaultdocdir}/%{name}-%{version}/examples
cd %{_builddir}/%{name}-%{version}
cp -p *.txt %{buildroot}%{_defaultdocdir}/%{name}-%{version}/
cp -rp docs/ %{buildroot}%{_defaultdocdir}/%{name}-%{version}/docs
cp -rp examples/ %{buildroot}%{_defaultdocdir}/%{name}-%{version}/examples
%clean
rm -rf $RPM_BUILD_ROOT
%files
%defattr(-,root,root)
%{_defaultdocdir}/%{name}-%{version}/
%{python_sitelib}/%{name}/
%{python_sitelib}/%{name}-%{version}-py%{python_version}.egg-info/
4. The overall approach I used was to run
$ python setup.py --command-packages=pypi2rpm.command bdist_rpm2
--spec-only
That brought in items such as the description specified in setup.py. I
then moved the spec file up a directory level from dist, edited in the
above parts relevant to the doc files, and finally ran
$ python setup.py --command-packages=pypi2rpm.command bdist_rpm2
> --spec-file=SPECFILE
I also found it very useful to actually run
python setup.py --command-packages=pypi2rpm.command bdist_rpm2 --spec-file
SPECFILE >buildout 2>&1
Recording the build output enabled me to find details of errors that
occurred during the build. There were a number of those, and the output
flies by too quickly to notice them for diagnosis of problems that need to
be fixed. (an approach I've also used in the past is to tee the output,
so I can see if the build was successful when it completes). Also, the -k
option came in useful in figuring out the documentation-related fixes to
the spec file, because it helped seeing what was included in the build and
what was available in the build but not getting copied to the install.
5. I inquired about how the project produced its sdist files for pypi,
and was told that they had so many problems related to their switch to git
that they simply did a gzipped tar of their repository.
6. The project used a different extension on their README file from the
expected values of README and README.txt. I needed to change that to
README.txt to get it to process properly.
Again, thanks.
Stan Klein
On Fri, November 11, 2011 6:00 am, Tarek Ziad? <ziade.tarek(a)gmail.com> wrote:
>
> Message: 3
> Date: Thu, 10 Nov 2011 18:42:08 +0100
> From: Tarek Ziad? <ziade.tarek(a)gmail.com>
> To: "Stanley A. Klein" <sklein(a)cpcug.org>
> Cc: distutils-sig(a)python.org
> Subject: Re: [Distutils] Compatibility of bdist_rpm with Fedora
> packaging instructions
>
> On Thu, Nov 10, 2011 at 6:11 PM, Stanley A. Klein <sklein(a)cpcug.org>
> wrote:
>> Tarek -
>>
>> I downloaded pypi2rpm and built it using bdist_rpm.
>
> you built it ? pypi2rpm is a script and a bidist_rpm2 command, no need
> to build, just install
>
>
>> ?The instructions on how to use it are rather thin.
>
> Yes there's no doc, as its mostly use in our own build tools for now.
>
>> My best guess is to run it in spec-only
>> mode, modify the spec, and then run it with the modified spec specified.
>> I tried to get a help listing but that was also think and I couldn't
>> quite
>> figure out how to run it.
>>
>> Could you provide further information on how to use it.
>>
>> Thanks.
>
> Once it's installed you can build a rpm in your project, using:
>
> $ python setup.py --command-packages=pypi2rpm.command bdist_rpm2
> --spec-file=SPECFILE
>
> (there are other options you can find with --help)
>
>
>>
>> Stan Klein
>>
>>
>>
>> On Thu, November 10, 2011 5:06 am, Tarek Ziad? wrote:
>>> On Thu, Nov 10, 2011 at 10:04 AM, Paul Nasrat <pnasrat(a)gmail.com>
>>> wrote:
>>>> I don't think bdist_rpm should track vendor packaging requirements,
>>>> purely as those recommendations may change faster than the release
>>>> process of distutils. I also believe bdist_rpm may be going away in
>>>> the future:
>>>
>>> Yes I confirm this. We removed it in packaging because we believe it
>>> should be maintained by the RPM communities -- with their own release
>>> cycles etc.
>>>
>>> FWIW I have a custom version in the pypi2rpm project where I just feed
>>> a .spec file to the bdist_rpm command, so I can do proper RHEL or
>>> Fedora packaging.
>>>
>>>> For Fedora have you considered rpmdev-newspec which can creates a
>>>> templated python spec file for your packages.
>>>>
>>>> http://fedoraproject.org/wiki/How_to_create_an_RPM_package
>>>>
>>>> Paul
>>>>
>>>> On 8 November 2011 21:40, Stanley A. Klein <sklein(a)cpcug.org> wrote:
>>>>> I will need to build some Python packages for Fedora and Centos. ?The
>>>>> spec
>>>>> file produced by bdist_rpm automatically includes the statement
>>>>> %files -f INSTALLED_FILES
>>>>>
>>>>> The Fedora Python packaging instruction includes a recommendation to
>>>>> avoid
>>>>> use of INSTALLED_FILES and provides some alternatives. ?That is the
>>>>> first
>>>>> incompatibility I've encountered, but there may be more.
>>>>>
>>>>> The bdist_rpm code probably should be changed to enable
>>>>> compatibility.
>>>>> Meanwhile, is there a workaround?
>>>>>
>>>>>
>>>>> Stan Klein
12 years
- 2
- 2
- ← Newer
- 1
- Older →