Python libraries and backwards compat [was Re: What would it take to make Software Collections work in Fedora?]

[Mark McLoughlin]

On Thu, 2012-12-06 at 07:06 -0800, Adam Williamson wrote:
> On Thu, 2012-12-06 at 15:30 +0100, Nicolas Mailhot wrote:
> > IMHO use of software collections is a symptom of a badly run organisation
> > not devoting enough cycles to maintain the software it uses, and hoping
> > (as in wishful thinking) no problem will go critical before the product
> > they built on top of those collections is end-of-lifed
> > 
> > I completely fail to see how entities with that problem will manage to
> > maintain the package number explosion creating software collections will
> > induce.
> 
> On the one hand, I agree completely - I think the 'share all
> dependencies dynamically' model that Linux distros have traditionally
> embraced is the right one, and that we're a strong vector for spreading
> the gospel when it comes to that model, and it'd be a shame to
> compromise that.
> 
> On the other hand, we've been proselytizing the Java heretics for over a
> decade now, and the Ruby ones for a while, and neither shows any signs
> of conversion or just plain going away, so we may have to call it an
> ecumenical matter and deal with their models somehow. Sucky as it may
> be. I don't know, I'm a bit conflicted.

It's interesting that you call out Java and Ruby folks as being
heretics. I guess that means all is kosher with Python?

OpenStack is getting burned by API instability in some Python packages,
so I've started a thread on Python's distutils-sig to try and guage the
level of heresy amongst Python folks :)

It started here:

  http://mail.python.org/pipermail/distutils-sig/2013-February/020030.html

and now we're talking about Software Collections here:

  http://mail.python.org/pipermail/distutils-sig/2013-March/020074.html

Two things I'm picking up from the thread:

  - A trend towards "semantic versioning" and, implicit in that, an 
    acceptance of API breakages so long as the major number of a library
    version is incremented

  - Supporting the parallel installation of incompatible versions of 
    libraries isn't seen as an issue because you can "just use virtual 
    environments" ... which amounts to Python Software Collections.

The combination of those two things suggests to me that the Python world
will start looking a lot less sane to packagers - i.e. library
maintainers breaking API compatibility more often and assuming we can
just use SC or similar to have multiple incompatible versions installed.

I can see OpenStack upstream reacting to this by "capping" its required
version range for each library it depends so that if the library does
release an incompatible version, OpenStack sticks with the latest
compatible version.

If that happens, I think OpenStack packagers will need to look seriously
at using Software Collections. Basically, we'd look to package and
maintain our own stack of all the Python libraries we need above the
core Python libraries.

So, you'd have openstack-nova, openstack-glance, etc. all installed as
normal in /usr, /var, etc. but they'd require python libraries from the
openstack-grizzly SC like openstack-grizzly-python-eventlet which would
be installed in /opt/fedora/openstack-grizzly/root/usr/lib/python.

I'd appreciate it if someone else with a Fedora Python packaging
background could look into this and, hopefully, explain how the
discussion on distutils-sig isn't so terrifying after all.

Cheers,
Mark.