Language specific mirrors for Fedora Playground compliant packages

[Nick Coghlan]

On 09/15/2014 11:34 PM, Vít Ondruch wrote:
> 1) Speaking of Rubyists, you would need to convince them to start using
> "source "https://fedora-rubygems-mirror.org" instead of standard "source
> "https://rubygems.org"" in their Gemfiles. Not sure how you would like
> to achieve this. We could patch Bundler somehow to substitute the
> original source on Fedora, but I don't want to imagine what source of
> problems it would bring.

I'm not suggesting making this mandatory for anything (except perhaps as
part of the package review process at some indeterminate point in the
future). It would simply be about providing developers with the *option*
of pulling Python packages, Ruby gems, etc from a language specific
mirror if they so chose.

For many open source folks, pulling direct from upstream is going to be
simpler and easier. For us, pulling directly from upstream means having
to repeat the licensing review work that the Fedora community would
otherwise handle.

Accordingly, this new service would be primarily for the benefit of
folks in environments like ours where licensing compliance is of greater
concern, by providing a way for us to collaborate on that work *without*
having to muck about with changing package formats.

> 2) In Fedora, there are packages, which are modified from upstream
> versions. For example, rubygem-listen [1] provides universal API to
> provide notifications about file changes on file system. Upstream
> decided, that they will have hard dependencies on other platform
> specific libraries, which supports FS level notifications. We decided to
> drop them, since it does not make sense to support libraries which does
> nothing on Linux. Sometimes, we are forced to override overly tight
> dependencies on other libraries. And now to the question. What would
> happened with such gem? What the mirror should provide in such case?
> Fedora's version or upstream version of gem? The former is quite
> dangerous idea, since it would end up in two different packages of the
> same version.

In Python's case, we've deliberately redesigned our versioning system to
cope with the needs of redistributors like Fedora to indicate patched
versions (see http://www.python.org/dev/peps/pep-0440/ for details).
That would allow us to publish the Fedora-specific patched versions on a
PyPI mirror without version conflicts.

In general, however, I would expect the packages on the language mirrors
to be unchanged from their upstream counterparts. If we need to apply
patches, then that's what the distro packaging formats are for.

> 3) You need additional human/infrastructure resources.

We (as in the software development teams for internal services at Red
Hat) are going to have to do this work for our own internal application
deployments anyway. My proposal is that instead of doing that in such a
way that it only benefits Red Hat associates, we should instead make it
possible to move that work upstream into Fedora where it can benefit the
entire Fedora/RHEL/CentOS ecosystem.

> Additionally to the point 3, if you have such resources, I would be
> better to invest them into development of proper support of multiple
> versions of packages in DNF. That would remove the need for most of the
> SCLs and additional repositories in Copr, at least speaking of Ruby
> ecosystem.

Better for who? Parallel installs within the system directories aren't
particularly interesting to me. I *like* the software collections model,
because it decouples the language runtime lifecycle from that of the
underlying OS. It doesn't matter whether I'm deploying to Fedora, RHEL
6/7 or CentOS 6/7, the "py27" software collection is the "py27" software
collection. The same goes for all the other language runtimes, database
runtimes and web servers that are available from softwarecollections.org
or commercially through Red Hat.

That reduced level of coupling is hugely beneficial for application
portability between dev workstations, community environments and
corporate environments.

> Nowadays, people are not using ruby193 collection, because they are in
> love with Ruby 1.9.3. They are using the collection since it is bundled
> with Ruby on Rails 3.2. And they are using SCLs just because it allows
> them to install more versions of RoR on single system, not because they
> believe that their worflow is now much easier. But you can achieve it
> just fine with rpm -i having the RPMs. It is just not supported by
> DNF/YUM, since DNF/YUM understands just "upgrade, don't leave old
> version on system".

My interest is in managing the maintenance lifecycle of long lived
production services, not ad hoc parallel installs on developer
workstations. With software collections, I get a steady platform upgrade
cadence independent of the OS lifecycle, as well as a smooth layering of
packages updated at different rates. Roughly speaking, I'll be aiming
for a lifecycle like:

Application layer: our responsibility, upgrade every 2-12 weeks
Software collections layer: upgrade every 1-2 years
Operating system layer: upgrade every 3-5 years

The software collection layer also decouples our applications to some
degree from the OS layer, abstracting away many of the differences
between Fedora & the various versions of RHEL/CentOS that software
collections support.

Cheers,
Nick.

-- 
Nick Coghlan
Red Hat Hosted & Shared Services
Software Engineering & Development, Brisbane

HSS Provisioning Architect