[Fwd: Python libraries and backwards compat [was Re: What would it take to make Software Collections work in Fedora?]]
Nick Coghlan
ncoghlan at redhat.com
Tue Mar 5 02:57:43 UTC 2013
On 03/05/2013 08:53 AM, Mark McLoughlin wrote:
>> I'd appreciate it if someone else with a Fedora Python packaging
>> background could look into this and, hopefully, explain how the
>> discussion on distutils-sig isn't so terrifying after all.
As far as I can see, your concerns are valid in the near term, but
manageable in the longer term. The Linux distros have tried to advocate
system-wide dynamic linking for years, and ISVs have overwhelmingly
responded by choosing not to support the platform, rather than by
embracing dynamic linking. ISVs have instead voted with their feet by
embracing Microsoft, Google and Apple, all OS vendors that explicitly
encourage bundling of dependencies within an application. The Java and
Ruby communities don't care, and the Python community doesn't actually
care either (we're just a lot more conservative in general about making
backwards incompatible changes in the first place). In the fight between
easier security updates for system administrators and easier
cross-platform and cross-OS-version support for developers, shared
dependencies have lost, and lost comprehensively, amongst all but the
largest software vendors.
It's *not* a coincidence that CPython publishes pre-built binaries for
Windows and Mac OS X, but not for any Linux distro. Cross-distro
packaging is just too hard, and not worth the effort, so we just publish
the source archive and tell the individual distro communities "you
figure it out" (and, to their credit, they generally do).
Even amongst the Linux community, the popularity of virtual appliances
and virtual machines in general show that people recognise the serious
architectural problems created by coupling nominally independent
components together by forcing them to share dependencies. The response
to that has not been "virtual machines are evil, because they bundle
dependencies" (even though that's exactly what they do - they even
bundle the underlying OS!), it has been, "we need to create the
appropriate tools to easily update all these virtual machines when it is
time to deploy a security fix".
Language specific virtual environments are no different: the response
should NOT be "virtual environments are evil, we need to discourage
developers from using them", it should be "what tools do we need to
create to make it easy to deploy security fixes to language specific
virtual environments?". When dependencies are bundled without metadata,
that is indeed truly evil, as there's no way to know what needs to be
updated to install a security fix. Virtual environments aren't like that
- the necessary dependency information is there, the deployment tools to
manage security updates just need to be written (and perhaps the
behaviour of some existing tools adjusted to make it easier to deploy
those fixes).
We can either dig in our heels, demanding that every language include
support for dynamic linking against a version of a library at runtime
without the distros doing any extra work (and demand that each
individual upstream project do the necessary work to select the correct
version at runtime), or we can try to create shared dependency systems
that *don't suck* from the point of view of a single-developer ISV that
wants to easily support arbitrary Linux distros (who may all be shipping
different versions of the application's dependencies), as well as Mac OS
X and Windows (who probably aren't shipping shared versions of the
application's dependencies at all).
Specifically in the context of Python's virtual environments, Fedora and
other distros definitely have scope to make contributions that improve
the security update handling for system administrators, *without*
compromising on ease of deployment for cross-platform developers. For
example:
* A Python venv may include *.pth files in the venv's site-packages
directory that add more directories to sys.path. This means it is
possible to enhance pip and other installers to use a shared version
store, *without* needing to use anything other than virtual environments
to expose the appropriate versions of the shared dependencies to a given
application. This can be done purely through installation tools and
Python's existing import infrastructure *without* the application
needing to do anything special.
* Similar effects can also be achieved through symlinks (a *.pth based
approach will likely be an easier sell upstream, though, since "doesn't
work properly on Windows" is not an acceptable limitation when it comes
to Python's packaging infrastructure)
Python upstream is heavily focused on cross platform development. If
distros want to resolve sysadmin specific problems (such as handling
security updates to shared dependencies), they need to bring a solution
that doesn't make life harder for developers and also works on Mac OS X
and Windows, or upstream will ignore you.
If you want me to flesh out the "shared versions for Python virtual
environments" idea, so you can pitch it to the pip and virtualenv
developers, I can certainly do that, but I don't have time to work on
it, or advocate for it myself.
Regards,
Nick.
--
Nick Coghlan
Red Hat Infrastructure Engineering & Development, Brisbane
Python Applications Team Lead
Beaker Development Lead (http://beaker-project.org/)
PulpDist Development Lead (http://pulpdist.readthedocs.org)
More information about the python-devel
mailing list