RPM and GEMS - Unholy marriage

[Jeroen van Meeuwen]

Bert Meerman wrote:
> Hi everyone,
> 
> I've been following the discussion on RPM and GEMS conflict of interest. 
> I found it quite interesting, and I would like to discuss some packaging 
> options already present within ruby and rails.
> 
> There are three things I would like to add to Jeroen's page:
> 1) A valid Rails RPM package *explicit* links to gems required.
> Strictly speaking, you can lazy load the gems an application needs. This 
> might work, especially if you use gems that are very common (like 
> Rails). Problem is, that a critical dependency is missing. You can add 
> this info in the RPM spec, but I think it is better to make it mandatory 
> more downstream. For RPM packaging, all you need to do is ask for the 
> dependent gems.
> My question: what are your thoughts on this?
> 

With a packaged Rails product, for example, one aspect of the RPM package will 
most likely be:

  Requires:  rubygem(rails)

One can enforce a specific version of the Ruby gem:

  Requires:  rubygem(rails) = 2.3.5

in order to prevent breakage on platform or application upgrades, through 
system capabilities. Pretty much the same goes for Ruby gems, not just 
packaged applications from downstream consumers or vendors.

Normally, in cases where the upstream of the dependent package is using sane 
versioning, one could even:

  Requires: rubygem(rails) >= 2.3, rubygem(rails) < 2.4

which, logically, should result in the Rails RPM package changing for bug- and 
security-fixes only. However, such seems to not be common practice in the 
larger Ruby community. This is a problem mid-streams like Fedora will have to 
deal with -or let it be a Free For All but then what are we doing here?

Anyway, while the application may have (with emphasis on "may have"):

  RAILS_GEM_VERSION = '2.3.2'

this does not correspond with the capabilities provided by the system in any 
way. It locks the application down to using 2.3.2 and 2.3.2 only, which should 
thus also not be used lightly in any situation.

Because of it's own little package management, we have two completely separate 
dependency paths each of them searching to resolve dependencies (but only one 
of them through global dependencies), and both are as dependent on a closed 
dependency tree as another.

However, if you install a Ruby gem RPM, and gem update the system, your RPM is 
still there, and you can depend on it, and you can be sure that, should the 
system-wide capability be attempted to be removed in any case, the transaction 
is going to fail because of missing dependencies.

Imagine you gem install rails but then yum remove ruby -just to emphasize the 
point I'm trying to make here. 

In another example, gem install mysql and then yum update mysql from 5.0 to 
5.1, or yum update ruby from 1.8.6 to 1.8.7.

There can only be one package manager doing all the work. It's RPM.

Does that limit anyone from using Gem? No, we're shipping it and in the end, 
it's your system. Knock yourself out.

Do you have the liberty to derive from what we provide, either by using gem or 
apt or ...? Sure! Have fun.

Does that mean the RPMs provided are worthless? Euh, not so much.

For one, RPM builds are reproducible. If a test fails on one build system, it 
will fail on another build system in very much the same way (exceptions 
apply). If compilation fails because it's using a dynamic library instead of a 
static library to compile against, or it uses the latest and greatest version 
rather then the compat- version. We'll figure that out and fix it. 
Documentation for the upstream gem only accounts for so much of so many system 
aspects.

> 2) A valid Rails RPM package must contain the version when specifying 
> gems and the version must be limited.
> Example:
> # require 'aws/s3'
> config.gem 'aws-s3', :lib => 'aws/s3', :version => '0.4.0',      #     
> :source => "http://code.whytheluckystiff.net"

Actually, this is not the case. in a config.gem statement, one only is 
required to list the name, and all other parameters are optional.

For example, one my require a mysql gem, but not care about the version - 
because the platforms the package or application is distributed for includes 
the correct version and the mysql gem doesn't change that much.

> (see also 
> http://api.rubyonrails.org/classes/Rails/Configuration.html#M002064)
> This means that the module will use any version greater than 0.4.0. You 
> can also specify smaller, a range or a specific number. The trick is 
> this: if you do this correctly, you can use multiple gem versions on the 
> same system. In theory, this must work. This is what was causing the 
> issue with Lighthouse. I think you should fix the version or make it a 
> range, never a 'greater than' (as in the example and with Lighthouse).
> My question: what are your thoughts on this, especially the mandatory 
> 'limit the version number'?
> 

Multiple versions of a single applications are useful in a situation in which, 
from the perspective of a midstream distributor, each and every version can be 
appropriately maintained. Nevermind the bug-fixes and let's go straight to 
security fixes. We, the mid-stream distributor, have a tendency to solve most 
of those security issues for you, and give you the appropriate, fixed version, 
without breaking anything. It's something virtually unheard of in the greater 
Ruby community, where feature enhancements and API breakage go hand-in-hand in 
a release marked as a Security related release.

That said, we do have mechanisms in place that allow us to install more then 
one version of an application through RPM. We currently use that mechanism for 
kernel packages. It has a setting that is called "installonlyn_limit", so you 
may choose how many versions (or actually, RPM releases, not application 
versions) of the package should be preserved.

> 3) Gems can be vendorized.
> OK, this is the moment where a lot of people are going to be amazed, 
> mouth open and staring at the screen for the next five minutes: you can 
> make a Rails application and include the gems in it, as it where local 
> code. This is called vendorization.

</sarcasm> ;-)

> A vendorized gem is only available 
> to the application using it: it is part of the source code of that 
> application and located in the /vendor directory of the application root 
> (not in the normal gem directory). In fact, you can vendorize a gem and 
> then even modify the code.
> On one hand, if you say all gems should be vendorized, you don't have 
> any version conflicts. On the other hand, I heard someone complain about 
> static linking...
> Personally, I disagree with the vendorization of gems. It obfuscates 
> version management.
> Question: does Fedore prefer (or mandatorize) a specific approach? Why?
> 

We do not allow the shipping of vendored code or libraries. A good explanation 
of what vendored code entails is at 
https://bugzilla.redhat.com/show_bug.cgi?id=470696#c37

That does not mean, however, that downstream consumers or Independent Software 
Vendors are not allowed to vendor parts of their project or product. We 
discourage it, for all the right reasons might I add, but we can not (and 
should not) prohibit it.

Vendored code is not allowed for the parts that the Fedora Project 
distributes, hence if a product is ever to be a part of the distribution, the 
vendored code needs to go.

-- Jeroen