python-devel December 2016

python-devel@lists.fedoraproject.org

20 participants
15 discussions

Python 3.6, Fedora, and the "C" locale
by Nick Coghlan 24 Feb '17

24 Feb '17

Hi folks, One of the minor irritations with running Python 3 applications in Fedora containers is that those containers still default to the "C" local by default - you have to force "C.UTF-8" or "en_US.UTF-8" in order to tell Python that it should use UTF-8 for communicating with system interfaces. However, since 3.5, the CPython start-up sequence has treated claims from the OS that it should use ASCII more skeptically, and enabled the "surrogateescape" error handling by default on the standard streams. Along similar lines, what do folks think of the idea of patching Python 3.6 in Fedora to assume UTF-8 if it's told that it should use ASCII to communicate with the OS? That changes the failure mode from "interface problems happen any time you fail to configure your locale correctly" to "interface problems happen any time you fail to configure your locale correctly *and* your default locale uses an encoding other than UTF-8". That latter case then further breaks down into the following two situations: - you use an ASCII-incompatible encoding like Shift-JIS or GB-18030, in which case the old POSIX default of ASCII was already broken, and the surrogateescape workaround in 3.5 probably didn't help much - you use an ASCII-compatible encoding like koi8-r, which means the surrogateescape workaround in 3.5 probably helped a bit, but libraries like `click` would still have complained and refused to run So I think doing this would be a nice usability improvement for users of the system Python 3 installation on Fedora. I also have an upstream motive for suggesting this, though: if we do this in the more constrained environment of "Fedora users" and it doesn't break the world, then that will help build a case for making it the default upstream behaviour in Python 3.7. Regards, Nick. -- Nick Coghlan | ncoghlan(a)gmail.com | Brisbane, Australia

7 19

Packages FTBFS with Python 3.6
by Miro Hrončok 07 Jan '17

07 Jan '17

Hi all, We've recently tried to rebuild all Python packages with Python 3.6. However, we currently have bunch of packages that simply fail to build. As the list contains >200 packages, it would be very helpful, if you (package maintainers) could help us solve the issues, as we cannot go one by one to investigate the issue. I've attached the list of failed packages (failed.txt). You can search Koji to find out, what went wrong. Sometimes, it's just missing dependency. That dependency should be on this list as well. If it is not, maybe we lost it, please tell us if that's the case. Maybe the dependency is circular and something needs to be bootstrapped. If you need provenpackager powers, let me know. Everything currently happens in a side tag. I will notify you when the side tag gets merged. To build your package in mock, I've attached a configuration file for it (fedora-rawhide-x86_64-p36.cfg). Some useful commands: fedpkg build --target=f26-python3 # build in Koji from git fedpkg build --target=f26-python3 --scratch # scratch build in Koji from git koji build f26-python3 --scratch <srpm> # scratch build in Koji from SRPM fedpkg srpm && mock -r fedora-rawhide-x86_64-p36 <srpm> # mock build Thanks for your help. -- Miro Hrončok -- Phone: +420777974800 IRC: mhroncok

3 4

Python 3.6 and blocking-by-default os.urandom()
by Nick Coghlan 04 Jan '17

04 Jan '17

Hi folks, The long discussions upstream regarding the default behaviour of Python's os.urandom() API in Python 3.6 have come to an end, with Guido opting to make os.urandom() block implicitly waiting for the system RNG to be ready, even on Linux (where /dev/urandom doesn't do that). While I can see his point from a cross-platform language runtime design perspective, I'm still not sure it's the right answer for the Fedora system Python, as it doesn't play nice with ABRT, and goes against Ted T'so's decision to keep /dev/urandom non-blocking at the kernel level (risking confusion for folks that are Linux developers & operators first, and Pythonistas second). If "os.urandom() was called before system RNG is ready" throws an uncaught BlockingIOError, then we'll get a nice ABRT-friendly Python stack trace for people to debug (and perhaps figure out their VM entropy pool isn't being seeded properly, or their ARM hardware design needs a better source of entropy). By contrast, if it just blocks, then folks will be faced with a system hang, which they'll need to trace back to CPython being blocked on a kernel getrandom() call, and then infer from that that the system RNG isn't ready (for whatever reason), and they should probably do something about that. Patching this behaviour should be relatively straightforward - Python 3.5 currently makes this call in non-blocking mode and falls back to reading /dev/urandom directly in that case, so we'd just be forward porting the same logic to 3.6 and raising an exception instead of falling back to the file descriptor. Accordingly, what I propose we do is as follows: 1. Raise the concern in the F26 system-wide change proposal for migrating to Python 3.6 2. Apply the patch when the 3.6 beta releases are added to Fedora Rawhide 3. Decide whether or not to keep the patch based on ABRT results and other feedback on the Rawhide builds. Note that if the feedback on Rawhide shows that the change is helping people to find and diagnose VMs and hardware with improperly seeded entropy pools, that's a *good* thing: this proposed change is replacing Python 3.5's "/dev/urandom and os.urandom() may silently return statistically less-than-fully-random random numbers if the kernel entropy pool isn't seeded properly" with "os.urandom() will fail noisily in those cases, so you can either switch to the random module, or fix your entropy pool seeding". Cheers, Nick. -- Nick Coghlan | ncoghlan(a)gmail.com | Brisbane, Australia

4 9

List of packages owning %{python3_sitelib}/__pycache__
by Athos Ribeiro 30 Dec '16

30 Dec '16

Hello, Guidelines say that %{python3_sitelib}/__pycache__ should not be owned by python packages because python3-libs already owns it [1]. That directory is actually owned by system-python-libs. While going through a package review, I realized that 50+ packages own %{python3_sitelib}/__pycache__. To avoid generating noise to packagers, I am just listing those packages here [2]. [1] https://fedoraproject.org/wiki/Packaging:Python#Byte_compiling [2] - List of packages (rawhide) owning %{python3_sitelib}/__pycache__: cairo-dock-plug-ins netstat-monitor pydot pyparsing python-apipkg python-args python-autopep8 python-bottle python-cma python-cmdln python-configobj python-configparser python-cookies python-cram python-cycler python-debian python-demjson python-dialog python-docopt python-empy python-entrypoints python-feedparser python-flask-assets python-flask-login python-flask-principal python-flask-whooshee python-flexmock python-fuse python-hwdata python-ipgetter python-jupyter-core python-markdown2 python-mimeparse python-MultipartPostHandler2 python-novaclient-os-networks python-novaclient-os-virtual-interfaces python-optcomplete python-pandocfilters python-path python-pefile python-pickleshare python-pika_pool python-plaintable python-polib python-pylcdsysinfo python-pyphen python-pytest-flakes python-pytest-pep8 python-q python-random2 python-responses python-scp python-setuptools_hg python-simplemediawiki root uwsgi -- Athos Ribeiro http://www.ime.usp.br/~athoscr

7 6

portingdb.xyz is back!
by Tomas Orsava 27 Dec '16

27 Dec '16

We're pleased to announce that the Python 3 PortingDB is back at it's usual location: http://portingdb.xyz/ Happy New Year! Tomas Orsava

1 0

Tackling cert bundling with install-time symlink maps?
by Nick Coghlan 25 Dec '16

25 Dec '16

While hacking on Anitya yesterday, an idea occurred to me that may help address the SSL/TLS cert bundling problem in a way that doesn't require patching of modules and hence can be made compatible with virtual environments. Specifically, I had the idea of adding a new "--symlink <file-pattern>@<link-target>" option to pip, such that you could do things like: "pip install -r requirements.txt --symlink cacerts.txt@ /etc/pki/tls/certs/ca-bundle.crt" to replace any file called "cacerts.txt" in the packages being installed with a symlink to "/etc/pki/tls/certs/ca-bundle.crt" instead (cacerts.txt is the name httplib2 uses for its cert bundle). Since we patch `python3-certifi` to use the system bundle, and running `certifi` as a script prints the location of the cert bundle it is using, that would mean you could do things like: sudo dnf install python3-certifi python -m pip install requests --symlink cacert.pem@`/usr/bin/python3 -m certifi` and end up with an *unpatched* requests in the virtual environment that was nevertheless still using the system certificate store. I filed that idea on the pip issue tracker at https://github.com/pypa/pip/issues/4197 but figured I should raise it here as well, as if something like this was added, then Fedora could be updated to use a standard symlink map when building RPMs, and the developer portal could be updated with suggest `pip.conf` settings to use the system cert bundle by default. Cheers, Nick. -- Nick Coghlan | ncoghlan(a)gmail.com | Brisbane, Australia

2 2

Re: Packages FTBFS with Python 3.6
by Miro Hrončok 21 Dec '16

21 Dec '16

On 21.12.2016 14:52, Zdenek Dohnal wrote: > On 12/21/2016 02:39 PM, Miro Hrončok wrote: >> On 21.12.2016 14:11, Zdenek Dohnal wrote: >>> I think I encountered same issue with libpython3.5m.so.1.0: >>> >>> Example: >>> >>> Error: nothing provides libpython3.5m.so.1.0 needed by >>> python3-cups-1.9.72-9.fc25.i686. >>> nothing provides libpython3.5m.so.1.0 needed by >>> unbound-libs-1.5.10-1.fc26.i686 >> >> Translates to: unbound-libs (unbound) and python3-cups (python-cups) >> where not built with Python 3.6 == (in this case) they failed. > python-cups build failed because of unbound-libs. Kind of circle :) . Well, not really a circular dependency. unbound needs to be build first. >> >>> >>> On 12/21/2016 09:53 AM, Richard W.M. Jones wrote: >>>> On Wed, Dec 21, 2016 at 12:18:47AM +0100, Miro Hrončok wrote: >>>>> libguestfs >>>> As should be obvious from the (multiple) build failures when you've >>>> been trying to build it over the past few days, this fails because of: >>>> >>>> DEBUG util.py:426: Error: nothing provides >>>> libpython3.5m.so.1.0()(64bit) needed by >>>> libvirt-python3-2.5.0-1.fc26.x86_64 >>>> >>>> which is nothing to do with libguestfs. >>>> >>>> Rich. >>>> >>> >> > -- Miro Hrončok -- Phone: +420777974800 IRC: mhroncok

1 0

Re: Packages FTBFS with Python 3.6
by Miro Hrončok 21 Dec '16

21 Dec '16

On 21.12.2016 14:11, Zdenek Dohnal wrote: > I think I encountered same issue with libpython3.5m.so.1.0: > > Example: > > Error: nothing provides libpython3.5m.so.1.0 needed by python3-cups-1.9.72-9.fc25.i686. > nothing provides libpython3.5m.so.1.0 needed by unbound-libs-1.5.10-1.fc26.i686 Translates to: unbound-libs (unbound) and python3-cups (python-cups) where not built with Python 3.6 == (in this case) they failed. > > On 12/21/2016 09:53 AM, Richard W.M. Jones wrote: >> On Wed, Dec 21, 2016 at 12:18:47AM +0100, Miro Hrončok wrote: >>> libguestfs >> As should be obvious from the (multiple) build failures when you've >> been trying to build it over the past few days, this fails because of: >> >> DEBUG util.py:426: Error: nothing provides libpython3.5m.so.1.0()(64bit) needed by libvirt-python3-2.5.0-1.fc26.x86_64 >> >> which is nothing to do with libguestfs. >> >> Rich. >> > -- Miro Hrončok -- Phone: +420777974800 IRC: mhroncok

1 0

Re: Packages FTBFS with Python 3.6
by Miro Hrončok 21 Dec '16

21 Dec '16

On 21.12.2016 09:53, Richard W.M. Jones wrote: > On Wed, Dec 21, 2016 at 12:18:47AM +0100, Miro Hrončok wrote: >> libguestfs > > As should be obvious from the (multiple) build failures when you've > been trying to build it over the past few days, this fails because of: > > DEBUG util.py:426: Error: nothing provides libpython3.5m.so.1.0()(64bit) needed by libvirt-python3-2.5.0-1.fc26.x86_64 > > which is nothing to do with libguestfs. I'm sorry that I wasn't able to read all of the hundreds build/root logs to see what is the cause of the failures. I was simply building the packages over and over until no more succeeded builds appeared. libvirt-python is failing, so libguestfs cannot be built. Sorry for bothering you and thanks for the info. -- Miro Hrončok -- Phone: +420777974800 IRC: mhroncok

1 0

Re: Packages FTBFS with Python 3.6
by Miro Hrončok 21 Dec '16

21 Dec '16

On 21.12.2016 07:41, Pavel Raiskup wrote: > On Wednesday, December 21, 2016 12:18:47 AM CET Miro Hrončok wrote: >> Hi all, >> We've recently tried to rebuild all Python packages with Python 3.6. >> However, we currently have bunch of packages that simply fail to build. >> ... >> Everything currently happens in a side tag. I will notify you when the >> side tag gets merged. > > The 'gettext' FTBFS is not related only to Python 3.6 tag [1, FWIW, still > without answer]. Once I resolve those issues, am I expected to build the > package against both 'f26' and (with bumped version) 'f26-python3' targets? I've built it in side tag by trial and error. > When do you expect merging 'f26-python3' into f26? ASAP, already requested it. > > [1] https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org… > > Pavel > -- Miro Hrončok -- Phone: +420777974800 IRC: mhroncok

1 0

2025

2024

2023

2022

2021

2020

2019

2018

2017

2016

2015

2014

2013

2012

2011

2010

2009

2008

python-devel December 2016