Python 3.4, ensurepip and wheels

[Nick Coghlan]

On 12/11/2013 04:52 PM, Bohuslav Kabrda wrote:
> ----- Original Message -----
>>> While trying to create the patch that would rebuild the wheels from
>>> system and install them in venv, I bumped into few ugly problems, which
>>> is why I came up with this "wheels in RPMs" solution in the first place.
>>> I'd like to do this for 3.4 in Fedora and then start working on the
>>> "file copying" solution that would be acceptable for upstream Python
>>> 3.5. When this is done upstream, we will move to there with 3.5.
>>
>> Alas, it sounds like the pip folks were right when they said "we're not
>> sure that can actually work at this point" (something I hadn't thought
>> about previously: at the very least, the install time shebang line
>> rewriting and wrapper script generation will break. There's also the
>> problem of *which* scripts get installed, since a system pip for Python
>> 3 likely won't have the bare "pip" command, but we *do* want that in a
>> virtual environment).
>>
>>>     Creating the wheels from the files on the system keeps the number of
>>>     copies of files to a minimum.  That, in turn, makes it easier for
>>>     everyone involved to discover which files are the ones that do
>>>     something.
>>
>> So, here's a crazy thought: what if, rather than copying the installed
>> files directly into the virtual environment, we reverse engineered a
>> wheel archive *dynamically* from the system install and then installed
>> from that? That would avoid the problems with trying to bypass pip's
>> script generation, while still bootstrapping new virtual environments
>> based on the installed versions of pip and setuptools.
> 
> There is a slight problem here:
> In a stable Fedora release, we typically won't change pip's version, but rather backport security patches and the version will stay the same. So if you reconstruct the wheel from that, running "ensurepip --upgrade" in virtualenvs won't actually do anything, since pip will see the same version in virtualenv as is in the system and do nothing. There are few possible solutions to this:
> - if version in system is the same as in virtualenv, check file checksums and if some differ, reinstall
> - if version in system is the same as in virtualenv, uninstall pip in virtualenv and then reinstall the one from the system
> - (ugly, not upstream acceptable) do "rpm -q python3-pip" and use the Fedora release as a "build tag" for reconstructed wheel (similar to what I proposed in the "bundled wheels" solution)
> If upstream (you :)) accepts any of these types of behaviour (the second one seems the easiest and should work fine) in the PEP, I'll work on that.
> Quite literally this time, we would reinvent the wheel :)

Ah, true, this is the thing we're planning to fix by formally adding
local version numbering as part of the upstream data model in metadata
2.0 (that is, pip, setuptools, etc would actually be *required* to cope
properly if the version metadata contained a number like "1.5-2" rather
than just "1.5", so that patched downstream rebuilds can have different
version numbers in the metadata than vanilla upstream versions). (This
is already in the draft metadata spec, but I haven't quite worked
through all the implications for version pinning yet)

You may even want to try that out (patching the pip and setuptools
version numbers to include a release suffix) and see if it already
works, since the scheme we're making official in metadata 2.0 is based
on the way pkg_resources.parse_version() already behaves:

>>> pkg_resources.parse_version("1.5") < pkg_resources.parse_version("1.5-1") < pkg_resources.parse_version("1.5-2")
True

Failing that, I think the uninstall/reinstall trick (preferably combined
with some kind of marker so it only happens when there *is* an upgrade
to be installed) sounds like a reasonable workaround until we get
metadata 2.0 formalised upstream (which will hopefully happen before pip
1.6 is released mid next year).

Cheers,
Nick.

-- 
Nick Coghlan
Red Hat Hosted & Shared Services
Software Engineering & Development, Brisbane

Testing Solutions Team Lead
Beaker Development Lead (http://beaker-project.org/)