Python 3.4, ensurepip and wheels

[Bohuslav Kabrda]

Hi all,
so I started to work on packaging Python 3.4 beta 1 and I wanted to create ensurepip patch also acceptable for upstream - Toshio and Nick talked about that at [1]. I tried to analyze the situation and here is what I have. First some facts:
- with ensurepip, Python 3.4 bundles setuptools and pip wheel archives
- when ensurepip is run, it installs (or upgrades) currently installed pip from the bundled wheels

A way to solve this "properly" in a distro (as mentioned in [1]) would be:
- throw away bundled setuptools and pip wheels when building Python
- making python3 require python-pip
- patching ensurepip (this is the fun part):
-- if ensurepip is run in venv:
--- we will need to reconstruct the wheels and install them into venv; or just grab the RECORD files and use them as a list of files we need to copy into venv
--- to reconstruct the wheels from installed RPMs, it'd be a good idea to package python-{setuptools,pip} as wheels (with the ".dist-info" directory and all the wheel goodies inside) so that we have the RECORD files
--- to package setuptools and pip as wheels, they'll have to BR: python3-pip (I saw Toshio's notes from flock and I agree with the part about how Fedora should use wheels [2])
-- if ensurepip is run systemwide:
--- if "--upgrade" is used, should this run "yum update python3-pip"?

Now, when new X.Y version Python is introduced, we will have to:
- bootstrap the new python3 interpreter to not require python3-pip, since python3-pip will require python(abi)=X.(Y-1)
- then we will need to build setuptools and pip *not as wheels*, because we still won't have pip that would work with our new Python X.Y - that means boostrapping them too and building them using the current "setup.py install" approach
- when we will have the new pip, we will be able to reenable python3 requires for python3-pip
- and we will also be able to rebuild setuptools and pip *as wheels*

There are also some more minor issues, that are worth mentioning:
- We typically won't update pip after Fedora is released - if there are security issues, we will backport patches and bump the release. The problem with this is that "ensurepip --upgrade" is not able to see Fedora releases and wouldn't work in venv, leaving it vulnerable. I think we have three options here:
-- Either we use some Fedora-specific RPM magic, which won't be acceptable for upstream
-- Or we will use a nice feature of wheel RECORD files - they keep file hashes, so we can check venv pip against system pip and see if something has changed when "ensurepip --upgrade" is run
-- Or we can just copy whole pip again into venv, regardless of what is currently there (doesn't seem very nice..., but it's simple and in the end it works as the above method)
- We can't copy the *.pyc and *.pyo files from system to venv, because they have file locations hardcoded in them, that are used in tracebacks. We can solve this by just copying *.py files and running py_compile in our ensurepip.

I hope I covered all the important points. Basically, we can make this work in a way acceptable for upstream, if we package setupttols and pip as wheels. It'll require some extra effort, but I think it's worth it.
Thoughts? Anyone has better/simpler ideas?

Thanks a lot,
Slavek.

[1] https://lists.fedoraproject.org/pipermail/python-devel/2013-November/000535.html
[2] https://fedoraproject.org/wiki/User:Toshio/Flock2013_Python_Guidelines#Wheel:_the_new_upstream_distribution_format