Python 3.7's Deterministic pycs
by Petr Viktorin
Hello!
The first beta for Python 3.7 is out. It will hopefully get into Fedora
soon as python37.
After it comes out of beta, we'll upgrade python3 to it.
The What's New list is at: https://docs.python.org/3.7/whatsnew/3.7.html
One thing that's interesting for packagers is PEP 552: Deterministic
pycs: https://www.python.org/dev/peps/pep-0552/
Let me summarize in my own words.
A new opt-in mode for byte-compilation makes .pyc (bytecode cache) files
depend only on the contents of the corresponding source file.
If we use this, it will slow down imports, because the whole source file
would need to be read and hashed in order to verify if a .pyc file is
valid. (Currently, metadata like the modification time and file size is
used.)
To speed things up, there's an option, UNCHECKED_HASH, which skips cache
validation entirely. Using this would mean that if you modify a .py
source file installed by RPM, the changes wouldn't take effect (the .py
contents would only be shown in tracebacks).
Modifying installed files in production is extremely bad practice, of
course, but it's quite useful for debugging on throw-away systems. If we
adopt UNCHECKED_HASH, anyone doing it will have to remember to remove
the corresponding .pyc file.
Honestly, I'm not sure we want to use this in Fedora. Is anyone here
into reproducible builds, to make a better argument for this?
--
Petr Viktorin
3 years, 10 months
EPEL7: Adapting %python_provide to provide python3- for python36-
by Miro Hrončok
Hey,
since the plan is to have some python3-... packages in RHEL proper, should we
adapt the %python_provide macro to provide python3-... when it gets python36-...?
%{python_provide python36-foo} currently does nothing.
I propose to change it to do: Provides: python3-foo = %{version}-%{release}.
Note: %{python_provide python2-foo} currently adds obsoletes, let's not add them
for the python36 case (there is nothing to obsolete here, quite the opossite -
python3-foo from RHEL would in theory obsolete python36-foo from EPEL).
Arguably, this discussion should have happened before we did the mass rebuild
for 3.4 -> 3.6 transition, however I don't consider such provides essential. The
idea is to change the macro, but don't mass rebuild - instead start providing
the provides gradually.
Note that not all EPEL7 Python 3 packages use this macro, but many do.
--
Miro Hrončok
--
Phone: +420777974800
IRC: mhroncok
4 years, 1 month
PYTEST.pyc files
by Orion Poplawski
It appears that in at least some situations pytest will create
-PYTEST.pyc files, and sometimes (always?) with weird permissions:
-rw-------. 1 root root 1614 Jul 13 2018
/usr/lib64/python3.7/site-packages/cytoolz/__pycache__/utils_test.cpython-37-PYTEST.pyc
I've noticed the following packages have them:
pytest-4.4.1-1.fc31.src.rpm
python-astropy-healpix-0.4-1.fc31.src.rpm
python-cytoolz-0.9.0.1-3.fc30.src.rpm
python-healpy-1.12.9-1.fc31.src.rpm
python-pytest-repeat-0.8.0-1.fc31.src.rpm
python-pytest-rerunfailures-6.0-1.fc31.src.rpm
python-pytest-shutil-1.6.0-2.fc31.src.rpm
python-reproject-0.4-6.fc30.src.rpm
python3-pytest-asyncio-0.10.0-1.fc31.src.rpm
scipy-1.2.1-1.fc31.src.rpm
These can be prevented by setting PYTHONDONTWRITEBYTECODE=1 when run pytest.
Can anyone else shed more light on this? Should we add this to the
guidelines? (Possibly not since there do not appear to be many packages
like this). I suspect it comes in when has to set
PYTHONPATH=%{buildroot}%{python3_sitearch} due to needing to load
compiled modules.
- Orion
--
Orion Poplawski
Manager of NWRA Technical Systems 720-772-5637
NWRA, Boulder/CoRA Office FAX: 303-415-9702
3380 Mitchell Lane orion(a)nwra.com
Boulder, CO 80301 https://www.nwra.com/
4 years, 1 month
Issues with egg-info switching between directory and file
by Orion Poplawski
I've noticed a few times now where a python package unexpectedly
produces an egg-info file instead of a directory. This is particularly
troubling in light of rpm's inability to replace a directory with a file:
https://fedoraproject.org/wiki/Packaging:Directory_Replacement
This seems to come about in the case of a project using distutils with
incorrect options:
running install_egg_info
Writing
/builddir/build/BUILDROOT/python-pkgconfig-1.5.1-1.fc31.x86_64/usr/lib/python3.7/site-packages/pkgconfig-1.5.1-py3.7.egg-info
/usr/lib64/python3.7/distutils/dist.py:274: UserWarning: Unknown
distribution option: 'python_requires'
warnings.warn(msg)
In the case of pkgconfig, this appears to be coming about due to the use
of "poetry" which seems to incorrectly be using distutils instead of
setuptools:
https://github.com/sdispater/poetry/issues/866
In any case - please be careful with changing egg-info format - and do
use ....egg-info/ in your specs to ensure that the format is a
directory as expected.
- Orion
--
Orion Poplawski
Manager of NWRA Technical Systems 720-772-5637
NWRA, Boulder/CoRA Office FAX: 303-415-9702
3380 Mitchell Lane orion(a)nwra.com
Boulder, CO 80301 https://www.nwra.com/
4 years, 2 months
