Python libraries and backwards compat [was Re: What would it take to make Software Collections work in Fedora?]

[Bohuslav Kabrda]

----- Original Message -----
> On Thu, 2012-12-06 at 07:06 -0800, Adam Williamson wrote:
> > On Thu, 2012-12-06 at 15:30 +0100, Nicolas Mailhot wrote:
> > > IMHO use of software collections is a symptom of a badly run
> > > organisation
> > > not devoting enough cycles to maintain the software it uses, and
> > > hoping
> > > (as in wishful thinking) no problem will go critical before the
> > > product
> > > they built on top of those collections is end-of-lifed
> > > 
> > > I completely fail to see how entities with that problem will
> > > manage to
> > > maintain the package number explosion creating software
> > > collections will
> > > induce.
> > 
> > On the one hand, I agree completely - I think the 'share all
> > dependencies dynamically' model that Linux distros have
> > traditionally
> > embraced is the right one, and that we're a strong vector for
> > spreading
> > the gospel when it comes to that model, and it'd be a shame to
> > compromise that.
> > 
> > On the other hand, we've been proselytizing the Java heretics for
> > over a
> > decade now, and the Ruby ones for a while, and neither shows any
> > signs
> > of conversion or just plain going away, so we may have to call it
> > an
> > ecumenical matter and deal with their models somehow. Sucky as it
> > may
> > be. I don't know, I'm a bit conflicted.
> 
> It's interesting that you call out Java and Ruby folks as being
> heretics. I guess that means all is kosher with Python?
> 

Python makes it a bit hard to install and use multiple parallel versions of libraries with just setuptools/distutils, so it's a bit more kosher :)
Since I do both Ruby and Python packaging and I can compare them, it seems to me that the biggest difference is that Python folks aren't used to do specific version requirements (not judging what pros and cons that brings).

> OpenStack is getting burned by API instability in some Python
> packages,
> so I've started a thread on Python's distutils-sig to try and guage
> the
> level of heresy amongst Python folks :)
> 
> It started here:
> 
>   http://mail.python.org/pipermail/distutils-sig/2013-February/020030.html
> 
> and now we're talking about Software Collections here:
> 
>   http://mail.python.org/pipermail/distutils-sig/2013-March/020074.html
> 
> Two things I'm picking up from the thread:
> 
>   - A trend towards "semantic versioning" and, implicit in that, an
>     acceptance of API breakages so long as the major number of a
>     library
>     version is incremented
> 
>   - Supporting the parallel installation of incompatible versions of
>     libraries isn't seen as an issue because you can "just use
>     virtual
>     environments" ... which amounts to Python Software Collections.
> 
> The combination of those two things suggests to me that the Python
> world
> will start looking a lot less sane to packagers - i.e. library
> maintainers breaking API compatibility more often and assuming we can
> just use SC or similar to have multiple incompatible versions
> installed.
> 
> I can see OpenStack upstream reacting to this by "capping" its
> required
> version range for each library it depends so that if the library does
> release an incompatible version, OpenStack sticks with the latest
> compatible version.
> 

There have been similar issues with distro-packaging of Ruby applications using bundler for quite a while now [1], [2], so it's not something unseen.

> If that happens, I think OpenStack packagers will need to look
> seriously
> at using Software Collections. Basically, we'd look to package and
> maintain our own stack of all the Python libraries we need above the
> core Python libraries.
> 
> So, you'd have openstack-nova, openstack-glance, etc. all installed
> as
> normal in /usr, /var, etc. but they'd require python libraries from
> the
> openstack-grizzly SC like openstack-grizzly-python-eventlet which
> would
> be installed in /opt/fedora/openstack-grizzly/root/usr/lib/python.
> 
> I'd appreciate it if someone else with a Fedora Python packaging
> background could look into this and, hopefully, explain how the
> discussion on distutils-sig isn't so terrifying after all.
> 

IMHO that depends how the upstreams will actually use the new functionality. It can be used both for good (better semantics of versioning making it easier for packagers to discover API changes) or bad (too tight dependencies of packages, relying on fact that users will just use venv) from Fedora's POV.

> Cheers,
> Mark.
> 
> --
> devel mailing list
> devel at lists.fedoraproject.org
> https://admin.fedoraproject.org/mailman/listinfo/devel

-- 
Regards,
Bohuslav "Slavek" Kabrda.

[1] http://lists.fedoraproject.org/pipermail/devel/2012-December/174912.html
[2] http://lists.fedoraproject.org/pipermail/ruby-sig/2012-July/001073.html