Hello all,
I've recently noticed that in some of the python packages I maintain, there are now additionally byte-compiled files that appear to be from pytest:
[swt2c@rawhide-test ~][PROD]$ rpm -ql python3-pytest-xdist | grep PYTEST /usr/lib/python3.6/site-packages/xdist/__pycache__/_version.cpython-36-PYTEST.pyc /usr/lib/python3.6/site-packages/xdist/__pycache__/dsession.cpython-36-PYTEST.pyc /usr/lib/python3.6/site-packages/xdist/__pycache__/looponfail.cpython-36-PYTEST.pyc /usr/lib/python3.6/site-packages/xdist/__pycache__/newhooks.cpython-36-PYTEST.pyc /usr/lib/python3.6/site-packages/xdist/__pycache__/plugin.cpython-36-PYTEST.pyc /usr/lib/python3.6/site-packages/xdist/__pycache__/remote.cpython-36-PYTEST.pyc /usr/lib/python3.6/site-packages/xdist/__pycache__/report.cpython-36-PYTEST.pyc /usr/lib/python3.6/site-packages/xdist/__pycache__/slavemanage.cpython-36-PYTEST.pyc /usr/lib/python3.6/site-packages/xdist/scheduler/__pycache__/each.cpython-36-PYTEST.pyc /usr/lib/python3.6/site-packages/xdist/scheduler/__pycache__/load.cpython-36-PYTEST.pyc /usr/lib/python3.6/site-packages/xdist/scheduler/__pycache__/loadscope.cpython-36-PYTEST.pyc
It seems that these files must be generated when running the tests for the package. I have a hunch they might be problematic, though, because they contain the buildroot's path in them:
[swt2c@rawhide-test ~][PROD]$ strings /usr/lib/python3.6/site-packages/xdist/__pycache__/_version.cpython-36-PYTEST.pyc | grep builddir t/builddir/build/BUILDROOT/python-pytest-xdist-1.20.1-1.fc28.noarch/usr/lib/python3.6/site-packages/xdist/_version.py
What are these special PYTEST byte compiled files? Should they be packaged in the first place? And if so, how can I avoid having the build root path hard-coded in them?
Thanks, Scott
On 11/10/2017 04:58 AM, Scott Talbert wrote:
Hello all,
I've recently noticed that in some of the python packages I maintain, there are now additionally byte-compiled files that appear to be from pytest:
[swt2c@rawhide-test ~][PROD]$ rpm -ql python3-pytest-xdist | grep PYTEST /usr/lib/python3.6/site-packages/xdist/__pycache__/_version.cpython-36-PYTEST.pyc
/usr/lib/python3.6/site-packages/xdist/__pycache__/dsession.cpython-36-PYTEST.pyc
[...etc...]
It seems that these files must be generated when running the tests for the package. I have a hunch they might be problematic, though, because they contain the buildroot's path in them:
[swt2c@rawhide-test ~][PROD]$ strings /usr/lib/python3.6/site-packages/xdist/__pycache__/_version.cpython-36-PYTEST.pyc | grep builddir t/builddir/build/BUILDROOT/python-pytest-xdist-1.20.1-1.fc28.noarch/usr/lib/python3.6/site-packages/xdist/_version.py
What are these special PYTEST byte compiled files?
Pytest compiles test files slightly differently than normal Python. That's what allows it to do its "assert rewriting", i.e. telling you what went wrong assert. I *assume* this is what's happening here: they started caching the results of that compilation. I might be wrong, but that's the direction I would look in if I had time to research this issue today :)
I'm adding Thomas, pytest's maintainer, to the thread. Thomas, do you know more about what's happening?
Should they be packaged in the first place?
I don't think they should. They're cache files; nothing should break if they're if not present. Pytest will probably try to write them when it runs, but should silently do nothing if it can't write into /usr/lib. So, the only time not packaging them would be a problem is if someone runs the tests as root, as that would create unpackaged files in /usr/lib. But I don't think that's a use case we need to support.
Anyway, thanks for letting us know! This might turn out to be something we'll need to solve at the Python SIG level. I'll start with a brainstorm.
How to best make sure those files aren't packaged? %ignore? Or rm them at end of %check? Or is there some Pytest config option or env variable to prevent writing them, which we could enable in the rpm macros?
For the record, here are packages that have the issue now:
$ sudo dnf repoquery --releasever 27 -s -f '*PYTEST.pyc' pytest-3.2.1-3.fc27.src.rpm python-astropy-2.0.2-1.fc27.src.rpm python-camel-0.1.2-2.fc27.src.rpm python-flask-0.11.1-6.fc27.src.rpm python-healpy-1.11.0-1.fc27.src.rpm python-joblib-0.11-1.fc27.src.rpm python-pyqtgraph-0.10.0-1.fc27.src.rpm python-pytest-timeout-1.2.0-3.fc27.src.rpm python-pytest-xdist-1.18.2-1.fc27.src.rpm python-wcsaxes-0.9-5.fc26.src.rpm
On Fri, 10 Nov 2017, Petr Viktorin wrote:
[swt2c@rawhide-test ~][PROD]$ strings /usr/lib/python3.6/site-packages/xdist/__pycache__/_version.cpython-36-PYTEST.pyc | grep builddir t/builddir/build/BUILDROOT/python-pytest-xdist-1.20.1-1.fc28.noarch/usr/lib/python3.6/site-packages/xdist/_version.py
What are these special PYTEST byte compiled files?
Pytest compiles test files slightly differently than normal Python. That's what allows it to do its "assert rewriting", i.e. telling you what went wrong assert. I *assume* this is what's happening here: they started caching the results of that compilation. I might be wrong, but that's the direction I would look in if I had time to research this issue today :)
I'm adding Thomas, pytest's maintainer, to the thread. Thomas, do you know more about what's happening?
Yes, that does appear to be what's happening: https://github.com/pytest-dev/pytest/blob/master/_pytest/assertion/rewrite.p...
Should they be packaged in the first place?
I don't think they should. They're cache files; nothing should break if they're if not present. Pytest will probably try to write them when it runs, but should silently do nothing if it can't write into /usr/lib. So, the only time not packaging them would be a problem is if someone runs the tests as root, as that would create unpackaged files in /usr/lib. But I don't think that's a use case we need to support.
Anyway, thanks for letting us know! This might turn out to be something we'll need to solve at the Python SIG level. I'll start with a brainstorm.
How to best make sure those files aren't packaged? %ignore? Or rm them at end of %check? Or is there some Pytest config option or env variable to prevent writing them, which we could enable in the rpm macros?
I ended up setting PYTHONDONTWRITEBYTECODE=1 when calling py.test as a workaround for now.
For the record, here are packages that have the issue now:
$ sudo dnf repoquery --releasever 27 -s -f '*PYTEST.pyc' pytest-3.2.1-3.fc27.src.rpm python-astropy-2.0.2-1.fc27.src.rpm python-camel-0.1.2-2.fc27.src.rpm python-flask-0.11.1-6.fc27.src.rpm python-healpy-1.11.0-1.fc27.src.rpm python-joblib-0.11-1.fc27.src.rpm python-pyqtgraph-0.10.0-1.fc27.src.rpm python-pytest-timeout-1.2.0-3.fc27.src.rpm python-pytest-xdist-1.18.2-1.fc27.src.rpm python-wcsaxes-0.9-5.fc26.src.rpm
It seems like there really should be more packages affected.
Scott
python-devel@lists.fedoraproject.org
-
Petr Viktorin -
Scott Talbert