Python 3.4, ensurepip and wheels

[Nick Coghlan]

On 12/11/2013 01:28 AM, Bohuslav Kabrda wrote:
>     > Actually, there seems to be a much simpler way of doing this in Fedora (and any distro more generally):
>     > - setuptools and pip RPMs will carry the wheel inside them and drop it into ensurepip/_bundled
>     > - the wheels will be rebuilt during every RPM build everytime *after patching*, so they will carry security patches etc.
>     > - we will use the RPM release as the "build tag" mentioned in PEP 427 [1], so that when we e.g. fix a security bug but don't bump the version, "ensurepip --upgrade" will still see that the wheel has to be reinstalled (otherwise it'd say think the version is already there and wouldn't reinstall)
>     > So the only thing we will need to implement will be autodiscovery of the wheels, since they will change names independently on python3 package, but I think we can do that :) From upstream point of view this shouldn't break anything, but it'd also probably not have any benefit. Would you still accept such patch?
>     >
> 
>     Note that this misses one thing mentioned in my mail: local changes.
> 
>     Because scripting languages are readable and modifiable, sysadmins
>     will sometimes fix bugs that affect them by deploying just the
>     changed .py files (rather than rebuilding an rpm package). 
>     Configuration management (puppet, Ansible, chef, etc) makes it easy
>     to deploy and track hotfixes this way.  If wheels are cached at rpm
>     buildtime then this workflow will cease to function correctly for
>     virtualenv and these bundled libraries.  Since the bundling is an
>     implementation detail that sysadmins will have no reason to know
>     about until they want to hotfix whichever libraries are bundled,
>     this is likely to take them a while to figure out.
> 
> The upstream solution has this problem, too. In fact, this approach will
> be much closer to the upstream solution in the sense that it uses
> prebuilt wheels to install/update virtualenv. The wheels will just come
> from different packages than python3 itself. IMO it'd be sufficient to
> place a README file into the ensurepip/_bundled dir that would explain
> this. Actually, diverging from upstream in this way is an argument for
> not doing the "files copying" solution now.

Well, we only went for the full bundling solution upstream as an
expedient hack that we *knew* we could get working in time for Python
3.4. I'd actually prefer it if, given a system install of pip (and
setuptools), ensurepip bootstrapped those into new environments rather
than the bundled ones.

At the moment, if you do "pip install --upgrade pip" in your system
Python (regardless of platform), future virtual environments will still
get the old pip until the next CPython maintenance release. Now, in
theory, there will be new CPython releases shortly after each new pip
release, but going to a new CPython maintenance release is a higher
impact change than just updating pip, so we have to expect substantial
lags between pip updates and CPython updates in many environments.

> While trying to create the patch that would rebuild the wheels from
> system and install them in venv, I bumped into few ugly problems, which
> is why I came up with this "wheels in RPMs" solution in the first place.
> I'd like to do this for 3.4 in Fedora and then start working on the
> "file copying" solution that would be acceptable for upstream Python
> 3.5. When this is done upstream, we will move to there with 3.5.

Alas, it sounds like the pip folks were right when they said "we're not
sure that can actually work at this point" (something I hadn't thought
about previously: at the very least, the install time shebang line
rewriting and wrapper script generation will break. There's also the
problem of *which* scripts get installed, since a system pip for Python
3 likely won't have the bare "pip" command, but we *do* want that in a
virtual environment).

>     Creating the wheels from the files on the system keeps the number of
>     copies of files to a minimum.  That, in turn, makes it easier for
>     everyone involved to discover which files are the ones that do
>     something.

So, here's a crazy thought: what if, rather than copying the installed
files directly into the virtual environment, we reverse engineered a
wheel archive *dynamically* from the system install and then installed
from that? That would avoid the problems with trying to bypass pip's
script generation, while still bootstrapping new virtual environments
based on the installed versions of pip and setuptools.

Cheers,
Nick.

-- 
Nick Coghlan
Red Hat Hosted & Shared Services
Software Engineering & Development, Brisbane

Testing Solutions Team Lead
Beaker Development Lead (http://beaker-project.org/)