[Fwd: Python libraries and backwards compat [was Re: What would it take to make Software Collections work in Fedora?]]

[Nick Coghlan]

On 03/05/2013 08:53 AM, Mark McLoughlin wrote:
>> I'd appreciate it if someone else with a Fedora Python packaging
>> background could look into this and, hopefully, explain how the
>> discussion on distutils-sig isn't so terrifying after all.

As far as I can see, your concerns are valid in the near term, but 
manageable in the longer term. The Linux distros have tried to advocate 
system-wide dynamic linking for years, and ISVs have overwhelmingly 
responded by choosing not to support the platform, rather than by 
embracing dynamic linking. ISVs have instead voted with their feet by 
embracing Microsoft, Google and Apple, all OS vendors that explicitly 
encourage bundling of dependencies within an application. The Java and 
Ruby communities don't care, and the Python community doesn't actually 
care either (we're just a lot more conservative in general about making 
backwards incompatible changes in the first place). In the fight between 
easier security updates for system administrators and easier 
cross-platform and cross-OS-version support for developers, shared 
dependencies have lost, and lost comprehensively, amongst all but the 
largest software vendors.

It's *not* a coincidence that CPython publishes pre-built binaries for 
Windows and Mac OS X, but not for any Linux distro. Cross-distro 
packaging is just too hard, and not worth the effort, so we just publish 
the source archive and tell the individual distro communities "you 
figure it out" (and, to their credit, they generally do).

Even amongst the Linux community, the popularity of virtual appliances 
and virtual machines in general show that people recognise the serious 
architectural problems created by coupling nominally independent 
components together by forcing them to share dependencies. The response 
to that has not been "virtual machines are evil, because they bundle 
dependencies" (even though that's exactly what they do - they even 
bundle the underlying OS!), it has been, "we need to create the 
appropriate tools to easily update all these virtual machines when it is 
time to deploy a security fix".

Language specific virtual environments are no different: the response 
should NOT be "virtual environments are evil, we need to discourage 
developers from using them", it should be "what tools do we need to 
create to make it easy to deploy security fixes to language specific 
virtual environments?". When dependencies are bundled without metadata, 
that is indeed truly evil, as there's no way to know what needs to be 
updated to install a security fix. Virtual environments aren't like that 
- the necessary dependency information is there, the deployment tools to 
manage security updates just need to be written (and perhaps the 
behaviour of some existing tools adjusted to make it easier to deploy 
those fixes).

We can either dig in our heels, demanding that every language include 
support for dynamic linking against a version of a library at runtime 
without the distros doing any extra work (and demand that each 
individual upstream project do the necessary work to select the correct 
version at runtime), or we can try to create shared dependency systems 
that *don't suck* from the point of view of a single-developer ISV that 
wants to easily support arbitrary Linux distros (who may all be shipping 
different versions of the application's dependencies), as well as Mac OS 
X and Windows (who probably aren't shipping shared versions of the 
application's dependencies at all).

Specifically in the context of Python's virtual environments, Fedora and 
other distros definitely have scope to make contributions that improve 
the security update handling for system administrators, *without* 
compromising on ease of deployment for cross-platform developers. For 
example:

* A Python venv may include *.pth files in the venv's site-packages 
directory that add more directories to sys.path. This means it is 
possible to enhance pip and other installers to use a shared version 
store, *without* needing to use anything other than virtual environments 
to expose the appropriate versions of the shared dependencies to a given 
application. This can be done purely through installation tools and 
Python's existing import infrastructure *without* the application 
needing to do anything special.
* Similar effects can also be achieved through symlinks (a *.pth based 
approach will likely be an easier sell upstream, though, since "doesn't 
work properly on Windows" is not an acceptable limitation when it comes 
to Python's packaging infrastructure)

Python upstream is heavily focused on cross platform development. If 
distros want to resolve sysadmin specific problems (such as handling 
security updates to shared dependencies), they need to bring a solution 
that doesn't make life harder for developers and also works on Mac OS X 
and Windows, or upstream will ignore you.

If you want me to flesh out the "shared versions for Python virtual 
environments" idea, so you can pitch it to the pip and virtualenv 
developers, I can certainly do that, but I don't have time to work on 
it, or advocate for it myself.

Regards,
Nick.

-- 
Nick Coghlan
Red Hat Infrastructure Engineering & Development, Brisbane

Python Applications Team Lead
Beaker Development Lead (http://beaker-project.org/)
PulpDist Development Lead (http://pulpdist.readthedocs.org)