F29 Self Contained Change: No more automagic Python bytecompilation

[Jan Kurik]

Proposed Self Contained Change: No more automagic Python bytecompilation
https://fedoraproject.org/wiki/Changes/No_more_automagic_Python_bytecompi...


Owner(s):
  * Miro Hrončok <mhroncok at redhat dot com>
  * Petr Viktorin <pviktori at redhat dot com>


The current way of automatic Python byte-compiling of files outside
Python-specific directories is too magical and error-prone. It is
built on heuristics that are increasingly wrong.
We will provide a way to opt-out of it and adjust the guidelines to
prefer explicit bytecompilation of such files. Later, the old behavior
will be opt-in only or will cease to exist.
Note that bytecompilation in Python-specific directories (e.g.
/usr/lib/python3.6/) is not affected.



== Detailed description ==
[Change wrangler note]: As the detailed description of this Change
proposal is quite extensive I am just referring to the Change Proposal
wiki page:
https://fedoraproject.org/wiki/Changes/No_more_automagic_Python_bytecompi...
. Please review the proposal following the link.


== Scope ==
* Proposal owners:
Make it work technically, propose the new guidelines, file pull
requests for python3 modules that follow the current guidelines.

* Other developers:
Maintainers of python3 packages that redefine %__python should merge
provided pull requests. Others may opt-in for the new behavior or
explicitly stick with the old one (not a System Wide Change, they
don't have to do anything).

* Release engineering:
[https://pagure.io/releng/issue/7315 #7315]

* Policies and guidelines:
will be changed as described in description

* Trademark approval:
not needed
-- 
Jan Kuřík
Platform & Fedora Program Manager
Red Hat Czech s.r.o., Purkynova 99/71, 612 45 Brno, Czech Republic

[Tomasz Kłoczko]

On 16 February 2018 at 11:25, Jan Kurik <jkurik(a)redhat.com&gt; wrote:

>  Proposed Self Contained Change: No more automagic Python
> bytecompilation
>  https://fedoraproject.org/wiki/Changes/No_more_automagic_Python_
>  bytecompilation
>
>
>  Owner(s):
>    * Miro Hrončok <mhroncok at redhat dot com>
>    * Petr Viktorin <pviktori at redhat dot com>
>
>
>  The current way of automatic Python byte-compiling of files outside
>  Python-specific directories is too magical and error-prone. It is
>  built on heuristics that are increasingly wrong.
>  
Maybe instead throwing the baby out with the bathwater it would be better
to spend some time on prepare set of rules/advises/policies about set of
conditions which needs to be fulfilled about such scripts, and/or even
promote pushing necessary changes to original packages source trees?
Is it good to tolerate install for example python modules
outside Python-specific directories?

BTW python bytecode files. Three questions:
1) What do you guys are thinking about remove source .py files and move
them to debuginfo/debugsource packages?

2) Currently seems are generates .pyc and .pyo files. Why not only .pyo
files? (python optimized bytecode). If .pyo files are installes .pyc are
redundant. Isn't it?

3) Currently all .pyc/.pyo/.py files are installed in
%{_libdir}/python<py.ver> tree. In reality all those files are arch
independent. Why not installed those files in %{_datadir}/python<py.ver>?
This would allow build many/most of the python modules as noarch packages.

kloczek
-- 
Tomasz Kłoczko | LinkedIn: *http://lnkd.in/FXPWxH <http://lnkd.in/FXPWxH>*

[Miro Hrončok]

On 16.2.2018 14:09, Tomasz Kłoczko wrote:
>  On 16 February 2018 at 11:25, Jan Kurik <jkurik(a)redhat.com 
>  <mailto:[e-mail]>> wrote:
>  
>      Proposed Self Contained Change: No more automagic Python bytecompilation
>      https://fedoraproject.org/wiki/Changes/No_more_automagic_Python_bytecompi...
>     
> <https://fedoraproject.org/wiki/Changes/No_more_automagic_Python_bytecompi...
>  
>  
>      Owner(s):
>         * Miro Hrončok <mhroncok at redhat dot com>
>         * Petr Viktorin <pviktori at redhat dot com>
>  
>  
>      The current way of automatic Python byte-compiling of files outside
>      Python-specific directories is too magical and error-prone. It is
>      built on heuristics that are increasingly wrong.
>  
>  
>  Maybe instead throwing the baby out with the bathwater it would be 
>  better to spend some time on prepare set of rules/advises/policies about 
>  set of conditions which needs to be fulfilled about such scripts, and/or 
>  even promote pushing necessary changes to original packages source trees? 
I'm afraid I don't understand this part.

>  Is it good to tolerate install for examplepython modules 
>  outside Python-specific directories? 
It is good to tolerate private python modules in application specific 
directories, such as /usr/share/coolapp/plugins/. It's completely valid 
use case.

>  BTW python bytecode files. Three questions:
>  1) What do you guys are thinking about remove source .py files and move 
>  them to debuginfo/debugsource packages? 
This was considered several times in the past, but usually never 
seriously. The possible problems overweight the benefit. Users (and 
sysadmins, and we) are used to those files being there.

>  2) Currently seems are generates .pyc and .pyo files. Why not only
> .pyo 
>  files? (python optimized bytecode). If .pyo files are installes .pyc are 
>  redundant. Isn't it? 
I believe this depends on how you invoke your Python when your import 
those modules.

>  3) Currently all .pyc/.pyo/.py files are installed in 
>  %{_libdir}/python<py.ver> tree. 
Actually, for noarch packages this is /usr/lib/pythonX.Y even on 64bits 
system.

>  In reality all those files are arch 
>  independent. 
Yes, see above.

>  Why not installed those files in 
>  %{_datadir}/python<py.ver>? 
There is a bug open for this for 8 years. Some day, maybe.

https://bugzilla.redhat.com/show_bug.cgi?id=543756

>  This would allow build many/most of the 
>  python modules as noarch packages. 
We do that, see above.

-- 
Miro Hrončok
--
Phone: [phone]
IRC: mhroncok