Python 3.7's Deterministic pycs
by Petr Viktorin
Hello!
The first beta for Python 3.7 is out. It will hopefully get into Fedora
soon as python37.
After it comes out of beta, we'll upgrade python3 to it.
The What's New list is at: https://docs.python.org/3.7/whatsnew/3.7.html
One thing that's interesting for packagers is PEP 552: Deterministic
pycs: https://www.python.org/dev/peps/pep-0552/
Let me summarize in my own words.
A new opt-in mode for byte-compilation makes .pyc (bytecode cache) files
depend only on the contents of the corresponding source file.
If we use this, it will slow down imports, because the whole source file
would need to be read and hashed in order to verify if a .pyc file is
valid. (Currently, metadata like the modification time and file size is
used.)
To speed things up, there's an option, UNCHECKED_HASH, which skips cache
validation entirely. Using this would mean that if you modify a .py
source file installed by RPM, the changes wouldn't take effect (the .py
contents would only be shown in tracebacks).
Modifying installed files in production is extremely bad practice, of
course, but it's quite useful for debugging on throw-away systems. If we
adopt UNCHECKED_HASH, anyone doing it will have to remember to remove
the corresponding .pyc file.
Honestly, I'm not sure we want to use this in Fedora. Is anyone here
into reproducible builds, to make a better argument for this?
--
Petr Viktorin
4 years, 4 months
EPEL: Python 3.4 will be EOL in March 2019
by Miro Hrončok
On 13.8.2018 11:49, Larry Hastings wrote on python-dev(a)python.org:
> We of the core dev community commit to supporting Python releases for
> five years. Releases get eighteen months of active bug fixes, followed
> by three and a half years of security fixes. Python 3.4 turns 5 next
> March--at which point we'll stop supporting it, and I'll retire as 3.4
> release manager.
>
> My plan is to make one final release on or around its fifth birthday
> containing the last round of security fixes. That's about seven
> months from now.
See also PEP 429 -- Python 3.4 Release Schedule [0].
We have python34 and python36 in EPEL7. python34 being the "main" one
and python36 the "other" one. The original plan [1] was that once the
ecosystem is adapted well enough, we can switch what "main" and "other"
is and eventually drop python34, creating room for maybe another python3
release to be the "other" next (python38 maybe?). Originally this was
supposed to happen for each release [2], but this was later abandoned
due to lack of manpower.
Do we want to ever switch "main" with "other"? The problem here is:
$ dnf repoquery --repo=epel7 --whatrequires 'python(abi) = 3.4' |
pkgname | sort | uniq | wc -l
243
$ dnf repoquery --repo=epel7 --whatrequires 'python(abi) = 3.6' |
pkgname | sort | uniq | wc -l
15
The original idea is that all packages would have the python3_other
macros in them [2] and all we need is rebuild everything in proper
order, maybe with some bootstrapping. However that never happened and
most of the packages I've seen lack the python3_other bits (no
statistics, just my impression).
Is this something we want? If so, are the packagers willing to adapt
their packages (as much as I'd like to do this, I lack the resources to
hack on 228 packages)?
The Python Maint team in Red Hat is here to help. However the drive
would need to come from the EPEL community, as now we are only guessing
what are the needs here.
[0] https://www.python.org/dev/peps/pep-0429/
[1]
https://lists.fedoraproject.org/archives/list/epel-devel@lists.fedoraproj...
[2] https://fedoraproject.org/wiki/User:Bkabrda/EPEL7_Python3
--
Miro Hrončok
--
Phone: +420777974800
IRC: mhroncok
5 years, 1 month
Mass Python 2 packages Removal
by Miro Hrončok
Hi, this is about:
https://fedoraproject.org/wiki/Changes/Mass_Python_2_Package_Removal
We have ~300 bugz open and blocking
https://bugzilla.redhat.com/show_bug.cgi?id=PY2REMOVAL
Some of them are > 1 week old and hence anyone can go and remove the
mentioned (sub)packages. I've retired the packages that were for
retirement and I will retire more once they are unblocked (needs to go
to compose -> portingdb before the automation recognizes it).
I open more bugs as they are closed (I don't wan to deal with 1000 bugs
at once).
Retirement is easy. The problematic part is removing subpackages. I go
and manually choose the packages that appear not to be just simple "py
spec generates py2 and py3 subpackages" combo, but a bit more
complicated, and I leave the "simple" ones alone for now.
I'm theoretically able to remove a simple py2 subpackage in about a
minute, yet it is tedious and after ~20 packages I need to stop (before
I go crazy).
We have something in between 900 and 2000 packages to remove (there are
900 leaves to remove and we may make another 1100 into leaves as we go).
Is anyone willing to look into automating this? Write a spec depython2izer?
This usually means:
- removing python2- and python- BRs
- removing %package -n python2 and it's metadata, description & %files
- removing all %py2_build, %py2_install or their older manual variants
- removing python2 tests from %check
- (sometimes) removing %py3dir or python3 dir pushd/popd dance
- (sometimes) removing with_python3 conditionals as they make no sense
IIRC Zbyszek said at Flock that he might work on that. Is that still
true? If not, shall I draft something?
--
Miro Hrončok
--
Phone: +420777974800
IRC: mhroncok
5 years, 1 month
Plans for Matplotlib 3.0+
by Elliott Sales de Andrade
Hi all,
In case you weren't aware, Matplotlib 3.0 will drop support for Python 2.
The rc1 was just released today/yesterday, though I have not yet attempted
to see how that update will go. The 2.2.x line will continue to be
supported and receive bug fixes until Python 2 is EOL.
So I'm wondering what exactly our plans are going forward for when the
final 3.0 release is made?
--
Elliott
5 years, 2 months
Re: Python Classroom spin dep issue: scanning
by Miro Hrončok
On 26.9.2018 02:41, Adam Williamson wrote:
> Hi, Miro!
Hi Adam. CCing Python Devel, so others can hop in the discussion.
> The Python Classroom spin images no longer build in recent Rawhide,
> because of a dnf behaviour change that exposes dependency issues as
> failures rather than hiding them by silently excluding packages. (This
> change will likely get into F29 soon too).
>
> The issue is this. fedora-live-python-classroom.ks includes
> fedora-live-minimization.ks , which does this:
>
> -sane-backends
>
> but it also includes fedora-live-workstation.ks, which includes fedora-
> workstation-common.ks, which does this:
>
> @gnome-desktop
>
> ...and @gnome-desktop includes sane-backends-drivers-scanners and
> libsane-hpaio, which both require sane-backends.
>
> Until recently dnf was simply silently excluding sane-backends-drivers-
> scanners and libsane-hpaio from the image, to 'resolve' this. Now it
> fails on the problem, and we get to make a choice.
>
> You basically have three options:
>
> 1. Just drop fedora-live-minimization.ks from fedora-live-python-
> classroom.ks . It's pretty old and random at this point. It's basically
> trying to drop printing and scanning stuff to save a bit of space. Do
> you actually want printing and scanning not to work on your image?
The idea behind this was:
Python Classroom (GNOME) is used for (usually one-time of)
installations/live on classroom computers for a workshop at a conference
or an actual course at school or university, or similar. Most of
classrooms I've seen do not have printers and scanners attached to the
computers. Hence I decided to save space while removing this. If anybody
want's to use the Classroom Lab in an environment other than a classroom
(such as a Python developer who finds it a good balance of default
packages) or a printer + scanner is connected to a classroom computer,
they can install the packages.
However I didn't examine how much space this actually saves and if it's
not a significant amount, we can simply go with 1.
> 2. Keep fedora-live-minimization.ks but explicitly add back `sane-
> backends` to %packages in fedora-live-python-classroom.ks . This would
> override the exclusion and fix the dep issue, and include the scanning
> bits in the image.
Having scanning and not having printing seems a bit weird. But again, if
printing is huge and scanning is not, we can use 2.
> 3. Keep fedora-live-minimization.ks and add excludes for sane-backends-
> drivers-scanners and libsane-hpaio to fedora-live-python-classroom.ks .
> This would lose scanning support, and save a bit of space.
I'd use this if we realize the space savings to be distinguishable.
> What would you like to do? Thanks!
Before I build 3 images and compare their sizes, do you have any idea
about a better way to find this out?
--
Miro Hrončok
--
Phone: +420777974800
IRC: mhroncok
5 years, 2 months
python3-debug: Should we use -Og rather than -O0?
by Petr Viktorin
Hi,
I've been looking at documentation of these GCC flags:
-O0
Reduce compilation time and make debugging produce the expected
results. This is the default
-Og
Optimize debugging experience. -Og enables optimizations that do
not interfere with debugging. It should be the optimization level of
choice for the standard edit-compile-debug cycle, offering a reasonable
level of optimization while maintaining fast compilation and a good
debugging experience.
(from https://gcc.gnu.org/onlinedocs/gcc/Optimize-Options.html)
Currently, we use -O0 for the debug build of Python, but it seems that
-Og would actually be a better option. Is there something I'm missing?
5 years, 2 months
Future of Python 2.6 and Jython in Fedora
by Miro Hrončok
In Fedora, we advertise "all the Pythons" are available and developers
may use them with venv, virtualenv, tox.
https://developer.fedoraproject.org/tech/languages/python/multiple-python...
Two of the interpreters we ship are quite problematic in this regard.
### Python 2.6
- virtualenv 16.x dropped support for 2.6 and we dirty patched it back
- tox 3 dropped support for 2.6
- pip/setuptools dropped support for 2.6 and our patched virtualenv
uses an older version of pip and setuptools to make work
However, I know there are developers who use Python 2.6 in Fedora
(namely because their software maintains RHEL 6 compatibility).
I propose that we indicate on Fedora Developer Portal that Python 2.6 is
provided as is and that it may not work with virtualenv/tox. We maintain
the current patch of virtualenv at least for the lifetime of Fedora 29,
but if an upgrade of virtualenv is done, we ditch that patch from Fedora
30+.
We also mark it deprecated and schedule the removal for EPEL 6 EOL
(AFAIK late 2020).
### Jython
- virtualenv somehow works but is not officially supported
- pip is not supported and has hesisenbugs
I know nobody who uses Jython with virtualenv and tox.
I propose that we indicate on Fedora Developer Portal that Jython is
provided as is and that it may not work with virtualenv/tox.
(No scheduled removal.)
Thoughts?
--
Miro Hrončok
--
Phone: +420777974800
IRC: mhroncok
5 years, 2 months
release-monitoring for python projects
by Martin Bukatovic
Dear python devel,
While I was considering packaging of tmuxp tool, I noticed that
dependencies of this package provided by Fedora are bit old. Eg.
python-libtmux is 2 years old, which is old enough to prevent packaging
any reasonable recent version of tmuxp.
Do I read this right that we need to update both python-libtmux and
other libraries in rawhide first so that review request for tmuxp can be
created, and when it will be completed, the new package will land only
in rawhide?
I checked release-monitoring.org tool and noticed that wrong backed is
selected for both packages. When I fixed this, the tool created "new
version is available" bugs for both packages.
Considering the information (most?) python packages have in it's spec
file, would it make sense to preconfigure monitoring details based on
information from the spec file when the package is added or updated? Or
maybe just mandate that the release monitoring information are correct
(and that the integration works) during review request? Or was there
some bulk change some time ago related which could break this?
--
Martin Bukatovič
5 years, 2 months
Python 2 Package Removal and when to use fedora-obsolete-packages
by Scott Talbert
I'm thinking about removing some of my Python 2 subpackages (that are no
longer used). However, I'm wondering about the case where the removed
packages should be added to fedora-obsolete-packages. I don't _think_ the
removal of my packages would cause any problems upon upgrade.
Can someone provide an example of a situation where a removed package
could cause a problem on upgrade (and should thus be added to
fedora-obsolete-packages).
Thanks,
Scott
5 years, 2 months
Re: DNF: "There are following alternatives to this package"
by Tomas Orsava
On 09/13/2018 06:43 PM, Mathieu Bridon wrote:
> On Thu, 2018-09-13 at 18:17 +0200, Tomas Orsava wrote:
>> We'd like to propose a new functionality for dnf: When a user tries
>> to install a package XYZ and dnf doesn't find it, dnf would recommend
>> them alternative packages. These offered packages would advertise
>> that they are an alternative for XYZ using a specially formatted
>> Provides tag.
>>
>> For example, packages `python2-requests` and `python3-requests`
>> would both have the following tag:
>>
>> Provides: alternative-for(python-requests) = %{version}-
>> %{release}
>>
>> (Possibly via the already existing and widespread %python_provide
>> macro in the Python case.)
>>
>> And when the user would try to install `python-requests`, dnf would
>> look for packages with the relevant Provides tag and display them:
>>
>> # dnf install python-requests
>> * There are following alternatives to this package:
>> python2-requests python3-requests
>> No match for argument: python-requests
>> Error: Unable to find a match
>>
>> This would be very similar to an already existing functionality that
>> searches for lowercase package names:
>>
>> # dnf install python3-REQUESTS
>> * Maybe you meant: python3-requests
>> No match for argument: python3-REQUESTS
>> Error: Unable to find a match
>>
>> (That functionality is broken in some versions—RHBZ#1628514—so you
>> might not see this result at the moment.)
>>
>> What are your thoughts?
> It's neat, but it doesn't help catch typos, which seems like the most
> probably case where I'd want such a feature.
>
> How about instead of a scheme based on provides, just quickly check if
> a package has a "similar" name?
>
> Basically extend the existing check for lowercase you mention.
Catching typos would be useful, and I'd certainly support that addition,
but this doesn't really scratch the itch I'm trying to solve.
We want to tell the user "these are exactly the alternatives available
to you". Not guess, not rely on some typo resolution. We know what the
alternatives are, here they are.
I've talked with people working on dnf and this looks like an easy
mechanism to add, and it will be more reliable for the use cases it covers.
Tomas
>
> $ git stats
> git: 'stats' is not a git command. See 'git --help'.
>
> The most similar command is
> status
>
>
5 years, 2 months
