%pyproject_install: .dist-info vs .egg-info
by Felix Schwarz
Hey,
I just noticed that the new packaging macros create a .dist-info directory
instead of .egg-info.
Just to be sure: There is no incompatibility between these two, right? So
setuptools-based code can still retrieve all the package metadata in .dist-info
directories?
(If so I can just update the package in Fedora as that dist-info change itself
does not break any backwards compatibility as far as other packages are concerned.)
Felix
3 months
Re: Flatpak and __pycache__
by Miro Hrončok
On 11/29/20 3:22 PM, Kalev Lember wrote:
> Can do. However, one question: When we find code in /app/lib(64)/pythonX.Y, do
> we bytecompile with /usr/bin/pythonX.Y or /app/bin/pythonX.Y?
>
>
> Awesome, thanks!
>
> It depends: for python2.7 (gimp flatpak) we use python2.7 re-built for /app
> prefix (so it's bundled with the app's flatpak), but for regular python3 we just
> use the /usr-installed one that's part of the flatpak runtime (the runtime uses
> /usr prefix and app flatpaks use /app prefix).
Here's a draft PR that should fix the Python 3 case:
https://src.fedoraproject.org/rpms/redhat-rpm-config/pull-request/113
The Python 2 case is IMHO not worth it. It doesn't have __pycache__ or %pycached.
It might be a good idea to call %py_byte_compile in the spec files of flatpaked
Python 2 RPMs if the bytecache is desired.
--
Miro Hrončok
--
Phone: +420777974800
IRC: mhroncok
3 months
Re: Flatpak and __pycache__
by Miro Hrončok
On 11/29/20 3:22 PM, Kalev Lember wrote:
> > > for python_libdir in `find "$RPM_BUILD_ROOT" -type d|grep -E
> > "/usr/lib(64)?/python[0-9]\.[0-9]$"`;
> >
> > ... which should use prefix instead of hardcoding /usr (or alternatively
> scan
> > both /usr and /app).
>
> Can do. However, one question: When we find code in /app/lib(64)/pythonX.Y, do
> we bytecompile with /usr/bin/pythonX.Y or /app/bin/pythonX.Y?
>
>
> Awesome, thanks!
>
> It depends: for python2.7 (gimp flatpak) we use python2.7 re-built for /app
> prefix (so it's bundled with the app's flatpak), but for regular python3 we just
> use the /usr-installed one that's part of the flatpak runtime (the runtime uses
> /usr prefix and app flatpaks use /app prefix).
>
> Would it be possible to just use %__python2 and %__python3 macros for
> byte-compiling? These are always set correctly by the flatpak macros, no matter
> if the interpreter is in /app or /usr.
Not in the way this is currently done, no. The script detects a
Python-version-specific path and uses that Python version.
E.g. even if %__python3 is set to /usr/bin/pypy3, when the script fins files in
/usr/lib(64)/python3.9, it compiles them with python3.9.
--
Miro Hrončok
--
Phone: +420777974800
IRC: mhroncok
3 months
Re: Flatpak and __pycache__
by Miro Hrončok
On 11/29/20 7:35 AM, Kalev Lember wrote:
> On Sun, Nov 29, 2020 at 1:55 AM Miro Hrončok <mhroncok(a)redhat.com
> <mailto:mhroncok@redhat.com>> wrote:
>
> On 11/28/20 10:06 PM, Ville-Pekka Vainio wrote:
> > Hi,
> >
> > I'm slowly working on reviving the Finnish spell-checking stack. When
> > working on the libvoikko package, I noticed the Python module now has
> > this in the file list:
> >
> > %if ! 0%{?flatpak}
> > %{python3_sitelib}/__pycache__/*
> > %endif
> >
> > Git blame takes me to this commit:
> >
> https://src.fedoraproject.org/rpms/libvoikko/c/e1b9941462b82f208b814fc2f6...
> <https://src.fedoraproject.org/rpms/libvoikko/c/e1b9941462b82f208b814fc2f6...>
> >
> > Apparently Flatpak could not handle __pycache__ stuff about six months ago.
> >
> > According to the packaging guidelines I should be using something like
> >
> > %pycached %{python3_sitelib}/%{name}.py
> >
> > This macro is defined in /usr/lib/rpm/macros.d/macros.python3 and it
> > seems like it does not take the Flatpak issue into account.
> > Should I just leave those lines as they are? Should the %pycached
> > macro be improved?
>
> You should be able to use the %pycached macro and if that breaks flatpaks, we
> should amend that macro to support that instead of adding `%if ! 0%{?flatpak}`
> to individual spec files. The idea behind this macro is to be able to do
> changes
> in one place.
>
> However, it would help to know the reason why flatpaks don't have bytecode
> caches. This is the first time I've seen this mentioned. It will require other
> code to be adapted as well, for example %pyproject_save_files.
>
>
> I believe this is because flatpaks are installed into /app, but the python
> bytecode compiler only does it for /usr.
>
> brp-python-bytecompile has:
>
> > for python_libdir in `find "$RPM_BUILD_ROOT" -type d|grep -E
> "/usr/lib(64)?/python[0-9]\.[0-9]$"`;
>
> ... which should use prefix instead of hardcoding /usr (or alternatively scan
> both /usr and /app).
Can do. However, one question: When we find code in /app/lib(64)/pythonX.Y, do
we bytecompile with /usr/bin/pythonX.Y or /app/bin/pythonX.Y?
--
Miro Hrončok
--
Phone: +420777974800
IRC: mhroncok
3 months
Re: Flatpak and __pycache__
by Miro Hrončok
On 11/28/20 10:06 PM, Ville-Pekka Vainio wrote:
> Hi,
>
> I'm slowly working on reviving the Finnish spell-checking stack. When
> working on the libvoikko package, I noticed the Python module now has
> this in the file list:
>
> %if ! 0%{?flatpak}
> %{python3_sitelib}/__pycache__/*
> %endif
>
> Git blame takes me to this commit:
> https://src.fedoraproject.org/rpms/libvoikko/c/e1b9941462b82f208b814fc2f6...
>
> Apparently Flatpak could not handle __pycache__ stuff about six months ago.
>
> According to the packaging guidelines I should be using something like
>
> %pycached %{python3_sitelib}/%{name}.py
>
> This macro is defined in /usr/lib/rpm/macros.d/macros.python3 and it
> seems like it does not take the Flatpak issue into account.
> Should I just leave those lines as they are? Should the %pycached
> macro be improved?
You should be able to use the %pycached macro and if that breaks flatpaks, we
should amend that macro to support that instead of adding `%if ! 0%{?flatpak}`
to individual spec files. The idea behind this macro is to be able to do changes
in one place.
However, it would help to know the reason why flatpaks don't have bytecode
caches. This is the first time I've seen this mentioned. It will require other
code to be adapted as well, for example %pyproject_save_files.
--
Miro Hrončok
--
Phone: +420777974800
IRC: mhroncok
3 months
Re: Python dependency missing during RPM build
by Miro Hrončok
On 11/19/20 6:02 PM, Richard W.M. Jones wrote:
> It's not a big deal because I added the Python Requires: line by hand,
> but I want to make sure I understand why it happens and whether
> there's an easy fix.
>
> https://koji.fedoraproject.org/koji/taskinfo?taskID=55889416
> https://src.fedoraproject.org/rpms/nbdkit/blob/master/f/nbdkit.spec#_475
>
> This subpackage contains a Python file:
>
> $ rpm -qlp ./nbdkit-S3-plugin-1.23.9-1.fc34.x86_64.rpm
> /usr/lib64/nbdkit/plugins/nbdkit-S3-plugin <--- this one
> /usr/share/doc/nbdkit-S3-plugin
> /usr/share/doc/nbdkit-S3-plugin/README
> /usr/share/licenses/nbdkit-S3-plugin
> /usr/share/licenses/nbdkit-S3-plugin/LICENSE
> /usr/share/man/man1/nbdkit-S3-plugin.1.gz
>
> The automatically generated dependencies don't pick up the need for
> python3-boto3.
>
> $ grep ^import nbdkit-S3-plugin
> import nbdkit
> import boto3
>
> $ rpm -qRp ./nbdkit-S3-plugin-1.23.9-1.fc34.x86_64.rpm
> /usr/sbin/nbdkit
> nbdkit-python-plugin >= 1.22
> rpmlib(CompressedFileNames) <= 3.0.4-1
> rpmlib(FileDigests) <= 4.6.0-1
> rpmlib(PayloadFilesHavePrefix) <= 4.0-1
> rpmlib(PayloadIsZstd) <= 5.4.18-1
>
> (As I said above, I added it explicitly, which is why the RPM built in
> Koji above _does_ contain the boto3 dependency).
>
> Now admittedly the Python file doesn't end in .py and doesn't have a
> python shebang at the top. But:
>
> $ file nbdkit-S3-plugin
> nbdkit-S3-plugin: Python script, ASCII text executable
>
> which seems as if it matches the %__python_magic regexp in
> /usr/lib/rpm/fileattrs/python.attr. So perhaps it _ought_ to work and
> something is wrong on the machine I'm using to reproduce this?
>
> Alternatively is there another way to tell the dependency generator to
> take a special look at this file?
There are three dependency generators for Python and neither of them does what
you ask here.
/usr/lib/rpm/fileattrs/python.attr
This file makes sure that files installed into /usr/lib(64)/pythonX.Y require
python(abi) = X.Y.
/usr/lib/rpm/fileattrs/pythondist.attr
This file makes sure that Python packages (upstream term) require other Python
packages. E.g. that requests requires idna, chardet and urllib3. This is read
from upstream meatadata (.dist-info or egg-info directories/files). It has
nothing to do with imports. See for example data in
/usr/lib/python3.9/site-packages/requests-2.24.0-py3.9.egg-info/requires.txt.
/usr/lib/rpm/fileattrs/pythonname.attr
This file makes sure that packages called python3-requests provide
python-requests and python3.9-requests.
There is no generator that parses Python imports (and never has been), sorry.
--
Miro Hrončok
--
Phone: +420777974800
IRC: mhroncok
3 months, 1 week
help wanted with modern Python packaging macros
(%generate_buildrequires, %tox)
by Felix Schwarz
Hi,
I wanted to update python-dns-lexicon. Version 3.4 uses poetry and tox so I
thought it would be a good idea to get a grip on %generate_buildrequires, %tox
etc.
Unfortunately I'm a bit stuck at the moment. Maybe this is just because I'm
starring for too long on some spec file (and probably my knowledge of
tox+macros is lacking).
I pushed the current state to:
https://src.fedoraproject.org/fork/fschwarz/rpms/python-dns-lexicon/commi...
My main problem is shown at the end of this mail: Somehow the Python extras
subpackage name is bad but I don't know how to debug this (without spending a
lot of time).
Additionally (but lower priority) just using %tox in %check fails (I think it
tries to download dependencies?).
Any help/pointers appreciated.
Felix
$ fedpkg --release=f33 mockbuild
…
Obsoletes: python-dns-lexicon < 3.4.0-1.fc33
Processing files: python3-dns-lexicon+easyname-3.4.0-1.fc33.noarch
Error: The package name contains an extras name `easyname` that was not found
in the metadata.
Check if the extras were removed from the project. If so, consider removing
the subpackage and obsoleting it from another.
error: Dependency tokens must begin with alpha-numeric, '_' or '/': ***
PYTHON_EXTRAS_NOT_FOUND_ERROR___SEE_STDERR ***
Error: The package name contains an extras name `easyname` that was not found
in the metadata.
Check if the extras were removed from the project. If so, consider removing
the subpackage and obsoleting it from another.
error: Dependency tokens must begin with alpha-numeric, '_' or '/': ***
PYTHON_EXTRAS_NOT_FOUND_ERROR___SEE_STDERR ***
Provides: python-dns-lexicon+easyname = 3.4.0-1.fc33
python3-dns-lexicon+easyname = 3.4.0-1.fc33 python3.9-dns-lexicon+easyname =
3.4.0-1.fc33
Requires(rpmlib): rpmlib(CompressedFileNames) <= 3.0.4-1 rpmlib(FileDigests)
<= 4.6.0-1 rpmlib(PayloadFilesHavePrefix) <= 4.0-1
Obsoletes: python-dns-lexicon+easyname < 3.4.0-1.fc33
RPM build errors:
absolute symlink: /usr/bin/lexicon-3 -> /usr/bin/lexicon-3.9
Dependency tokens must begin with alpha-numeric, '_' or '/': ***
PYTHON_EXTRAS_NOT_FOUND_ERROR___SEE_STDERR ***
Dependency tokens must begin with alpha-numeric, '_' or '/': ***
PYTHON_EXTRAS_NOT_FOUND_ERROR___SEE_STDERR ***
Finish: rpmbuild python-dns-lexicon-3.4.0-1.fc33.src.rpm
Finish: build phase for python-dns-lexicon-3.4.0-1.fc33.src.rpm
3 months, 2 weeks
