Python 3.6, Fedora, and the "C" locale
by Nick Coghlan
Hi folks,
One of the minor irritations with running Python 3 applications in
Fedora containers is that those containers still default to the "C"
local by default - you have to force "C.UTF-8" or "en_US.UTF-8" in
order to tell Python that it should use UTF-8 for communicating with
system interfaces.
However, since 3.5, the CPython start-up sequence has treated claims
from the OS that it should use ASCII more skeptically, and enabled the
"surrogateescape" error handling by default on the standard streams.
Along similar lines, what do folks think of the idea of patching
Python 3.6 in Fedora to assume UTF-8 if it's told that it should use
ASCII to communicate with the OS? That changes the failure mode from
"interface problems happen any time you fail to configure your locale
correctly" to "interface problems happen any time you fail to
configure your locale correctly *and* your default locale uses an
encoding other than UTF-8".
That latter case then further breaks down into the following two situations:
- you use an ASCII-incompatible encoding like Shift-JIS or GB-18030,
in which case the old POSIX default of ASCII was already broken, and
the surrogateescape workaround in 3.5 probably didn't help much
- you use an ASCII-compatible encoding like koi8-r, which means the
surrogateescape workaround in 3.5 probably helped a bit, but libraries
like `click` would still have complained and refused to run
So I think doing this would be a nice usability improvement for users
of the system Python 3 installation on Fedora.
I also have an upstream motive for suggesting this, though: if we do
this in the more constrained environment of "Fedora users" and it
doesn't break the world, then that will help build a case for making
it the default upstream behaviour in Python 3.7.
Regards,
Nick.
--
Nick Coghlan | ncoghlan(a)gmail.com | Brisbane, Australia
6 years, 9 months
Packages FTBFS with Python 3.6
by Miro Hrončok
Hi all,
We've recently tried to rebuild all Python packages with Python 3.6.
However, we currently have bunch of packages that simply fail to build.
As the list contains >200 packages, it would be very helpful, if you
(package maintainers) could help us solve the issues, as we cannot go
one by one to investigate the issue.
I've attached the list of failed packages (failed.txt). You can search
Koji to find out, what went wrong. Sometimes, it's just missing
dependency. That dependency should be on this list as well. If it is
not, maybe we lost it, please tell us if that's the case. Maybe the
dependency is circular and something needs to be bootstrapped. If you
need provenpackager powers, let me know.
Everything currently happens in a side tag. I will notify you when the
side tag gets merged.
To build your package in mock, I've attached a configuration file for it
(fedora-rawhide-x86_64-p36.cfg).
Some useful commands:
fedpkg build --target=f26-python3 # build in Koji from git
fedpkg build --target=f26-python3 --scratch # scratch build in Koji from git
koji build f26-python3 --scratch <srpm> # scratch build in Koji from SRPM
fedpkg srpm && mock -r fedora-rawhide-x86_64-p36 <srpm> # mock build
Thanks for your help.
--
Miro Hrončok
--
Phone: +420777974800
IRC: mhroncok
6 years, 11 months
Python 3.6 and blocking-by-default os.urandom()
by Nick Coghlan
Hi folks,
The long discussions upstream regarding the default behaviour of Python's
os.urandom() API in Python 3.6 have come to an end, with Guido opting to
make os.urandom() block implicitly waiting for the system RNG to be ready,
even on Linux (where /dev/urandom doesn't do that).
While I can see his point from a cross-platform language runtime design
perspective, I'm still not sure it's the right answer for the Fedora system
Python, as it doesn't play nice with ABRT, and goes against Ted T'so's
decision to keep /dev/urandom non-blocking at the kernel level (risking
confusion for folks that are Linux developers & operators first, and
Pythonistas second).
If "os.urandom() was called before system RNG is ready" throws an uncaught
BlockingIOError, then we'll get a nice ABRT-friendly Python stack trace for
people to debug (and perhaps figure out their VM entropy pool isn't being
seeded properly, or their ARM hardware design needs a better source of
entropy). By contrast, if it just blocks, then folks will be faced with a
system hang, which they'll need to trace back to CPython being blocked on a
kernel getrandom() call, and then infer from that that the system RNG isn't
ready (for whatever reason), and they should probably do something about
that.
Patching this behaviour should be relatively straightforward - Python 3.5
currently makes this call in non-blocking mode and falls back to reading
/dev/urandom directly in that case, so we'd just be forward porting the
same logic to 3.6 and raising an exception instead of falling back to the
file descriptor.
Accordingly, what I propose we do is as follows:
1. Raise the concern in the F26 system-wide change proposal for migrating
to Python 3.6
2. Apply the patch when the 3.6 beta releases are added to Fedora Rawhide
3. Decide whether or not to keep the patch based on ABRT results and other
feedback on the Rawhide builds.
Note that if the feedback on Rawhide shows that the change is helping
people to find and diagnose VMs and hardware with improperly seeded entropy
pools, that's a *good* thing: this proposed change is replacing Python
3.5's "/dev/urandom and os.urandom() may silently return statistically
less-than-fully-random random numbers if the kernel entropy pool isn't
seeded properly" with "os.urandom() will fail noisily in those cases, so
you can either switch to the random module, or fix your entropy pool
seeding".
Cheers,
Nick.
--
Nick Coghlan | ncoghlan(a)gmail.com | Brisbane, Australia
6 years, 11 months
List of packages owning %{python3_sitelib}/__pycache__
by Athos Ribeiro
Hello,
Guidelines say that %{python3_sitelib}/__pycache__ should not be owned
by python packages because python3-libs already owns it [1]. That
directory is actually owned by system-python-libs.
While going through a package review, I realized that 50+ packages own
%{python3_sitelib}/__pycache__. To avoid generating noise to packagers,
I am just listing those packages here [2].
[1] https://fedoraproject.org/wiki/Packaging:Python#Byte_compiling
[2] - List of packages (rawhide) owning %{python3_sitelib}/__pycache__:
cairo-dock-plug-ins
netstat-monitor
pydot
pyparsing
python-apipkg
python-args
python-autopep8
python-bottle
python-cma
python-cmdln
python-configobj
python-configparser
python-cookies
python-cram
python-cycler
python-debian
python-demjson
python-dialog
python-docopt
python-empy
python-entrypoints
python-feedparser
python-flask-assets
python-flask-login
python-flask-principal
python-flask-whooshee
python-flexmock
python-fuse
python-hwdata
python-ipgetter
python-jupyter-core
python-markdown2
python-mimeparse
python-MultipartPostHandler2
python-novaclient-os-networks
python-novaclient-os-virtual-interfaces
python-optcomplete
python-pandocfilters
python-path
python-pefile
python-pickleshare
python-pika_pool
python-plaintable
python-polib
python-pylcdsysinfo
python-pyphen
python-pytest-flakes
python-pytest-pep8
python-q
python-random2
python-responses
python-scp
python-setuptools_hg
python-simplemediawiki
root
uwsgi
--
Athos Ribeiro
http://www.ime.usp.br/~athoscr
6 years, 11 months
Tackling cert bundling with install-time symlink maps?
by Nick Coghlan
While hacking on Anitya yesterday, an idea occurred to me that may help
address the SSL/TLS cert bundling problem in a way that doesn't require
patching of modules and hence can be made compatible with virtual
environments.
Specifically, I had the idea of adding a new "--symlink
<file-pattern>@<link-target>" option to pip, such that you could do things
like:
"pip install -r requirements.txt --symlink cacerts.txt@
/etc/pki/tls/certs/ca-bundle.crt"
to replace any file called "cacerts.txt" in the packages being installed
with a symlink to "/etc/pki/tls/certs/ca-bundle.crt" instead (cacerts.txt
is the name httplib2 uses for its cert bundle).
Since we patch `python3-certifi` to use the system bundle, and running
`certifi` as a script prints the location of the cert bundle it is using,
that would mean you could do things like:
sudo dnf install python3-certifi
python -m pip install requests --symlink cacert.pem@`/usr/bin/python3
-m certifi`
and end up with an *unpatched* requests in the virtual environment that was
nevertheless still using the system certificate store.
I filed that idea on the pip issue tracker at
https://github.com/pypa/pip/issues/4197 but figured I should raise it here
as well, as if something like this was added, then Fedora could be updated
to use a standard symlink map when building RPMs, and the developer portal
could be updated with suggest `pip.conf` settings to use the system cert
bundle by default.
Cheers,
Nick.
--
Nick Coghlan | ncoghlan(a)gmail.com | Brisbane, Australia
6 years, 11 months
Re: Packages FTBFS with Python 3.6
by Miro Hrončok
On 21.12.2016 14:52, Zdenek Dohnal wrote:
> On 12/21/2016 02:39 PM, Miro Hrončok wrote:
>> On 21.12.2016 14:11, Zdenek Dohnal wrote:
>>> I think I encountered same issue with libpython3.5m.so.1.0:
>>>
>>> Example:
>>>
>>> Error: nothing provides libpython3.5m.so.1.0 needed by
>>> python3-cups-1.9.72-9.fc25.i686.
>>> nothing provides libpython3.5m.so.1.0 needed by
>>> unbound-libs-1.5.10-1.fc26.i686
>>
>> Translates to: unbound-libs (unbound) and python3-cups (python-cups)
>> where not built with Python 3.6 == (in this case) they failed.
> python-cups build failed because of unbound-libs. Kind of circle :) .
Well, not really a circular dependency. unbound needs to be build first.
>>
>>>
>>> On 12/21/2016 09:53 AM, Richard W.M. Jones wrote:
>>>> On Wed, Dec 21, 2016 at 12:18:47AM +0100, Miro Hrončok wrote:
>>>>> libguestfs
>>>> As should be obvious from the (multiple) build failures when you've
>>>> been trying to build it over the past few days, this fails because of:
>>>>
>>>> DEBUG util.py:426: Error: nothing provides
>>>> libpython3.5m.so.1.0()(64bit) needed by
>>>> libvirt-python3-2.5.0-1.fc26.x86_64
>>>>
>>>> which is nothing to do with libguestfs.
>>>>
>>>> Rich.
>>>>
>>>
>>
>
--
Miro Hrončok
--
Phone: +420777974800
IRC: mhroncok
6 years, 11 months
Re: Packages FTBFS with Python 3.6
by Miro Hrončok
On 21.12.2016 14:11, Zdenek Dohnal wrote:
> I think I encountered same issue with libpython3.5m.so.1.0:
>
> Example:
>
> Error: nothing provides libpython3.5m.so.1.0 needed by python3-cups-1.9.72-9.fc25.i686.
> nothing provides libpython3.5m.so.1.0 needed by unbound-libs-1.5.10-1.fc26.i686
Translates to: unbound-libs (unbound) and python3-cups (python-cups)
where not built with Python 3.6 == (in this case) they failed.
>
> On 12/21/2016 09:53 AM, Richard W.M. Jones wrote:
>> On Wed, Dec 21, 2016 at 12:18:47AM +0100, Miro Hrončok wrote:
>>> libguestfs
>> As should be obvious from the (multiple) build failures when you've
>> been trying to build it over the past few days, this fails because of:
>>
>> DEBUG util.py:426: Error: nothing provides libpython3.5m.so.1.0()(64bit) needed by libvirt-python3-2.5.0-1.fc26.x86_64
>>
>> which is nothing to do with libguestfs.
>>
>> Rich.
>>
>
--
Miro Hrončok
--
Phone: +420777974800
IRC: mhroncok
6 years, 11 months
Re: Packages FTBFS with Python 3.6
by Miro Hrončok
On 21.12.2016 09:53, Richard W.M. Jones wrote:
> On Wed, Dec 21, 2016 at 12:18:47AM +0100, Miro Hrončok wrote:
>> libguestfs
>
> As should be obvious from the (multiple) build failures when you've
> been trying to build it over the past few days, this fails because of:
>
> DEBUG util.py:426: Error: nothing provides libpython3.5m.so.1.0()(64bit) needed by libvirt-python3-2.5.0-1.fc26.x86_64
>
> which is nothing to do with libguestfs.
I'm sorry that I wasn't able to read all of the hundreds build/root logs
to see what is the cause of the failures. I was simply building the
packages over and over until no more succeeded builds appeared.
libvirt-python is failing, so libguestfs cannot be built.
Sorry for bothering you and thanks for the info.
--
Miro Hrončok
--
Phone: +420777974800
IRC: mhroncok
6 years, 11 months
Re: Packages FTBFS with Python 3.6
by Miro Hrončok
On 21.12.2016 07:41, Pavel Raiskup wrote:
> On Wednesday, December 21, 2016 12:18:47 AM CET Miro Hrončok wrote:
>> Hi all,
>> We've recently tried to rebuild all Python packages with Python 3.6.
>> However, we currently have bunch of packages that simply fail to build.
>> ...
>> Everything currently happens in a side tag. I will notify you when the
>> side tag gets merged.
>
> The 'gettext' FTBFS is not related only to Python 3.6 tag [1, FWIW, still
> without answer]. Once I resolve those issues, am I expected to build the
> package against both 'f26' and (with bumped version) 'f26-python3' targets?
I've built it in side tag by trial and error.
> When do you expect merging 'f26-python3' into f26?
ASAP, already requested it.
>
> [1] https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.o...
>
> Pavel
>
--
Miro Hrončok
--
Phone: +420777974800
IRC: mhroncok
6 years, 11 months
