RPM and GEMS - Unholy marriage

[Bert Meerman]

On 30-6-2010 14:19, Mamoru Tasaka wrote:
> Bert Meerman wrote, at 06/30/2010 02:39 AM +9:00:
>    
>> On 28-6-2010 10:42, Jeroen van Meeuwen wrote:
>>      
>>> Bert Meerman wrote:
>>>
>>>        
>>>> Hi everyone,
>>>>
>>>> I've been following the discussion on RPM and GEMS conflict of interest.
>>>> I found it quite interesting, and I would like to discuss some packaging
>>>> options already present within ruby and rails.
>>>>
>>>> There are three things I would like to add to Jeroen's page:
>>>> 1) A valid Rails RPM package *explicit* links to gems required.
>>>> Strictly speaking, you can lazy load the gems an application needs. This
>>>> might work, especially if you use gems that are very common (like
>>>> Rails). Problem is, that a critical dependency is missing. You can add
>>>> this info in the RPM spec, but I think it is better to make it mandatory
>>>> more downstream. For RPM packaging, all you need to do is ask for the
>>>> dependent gems.
>>>> My question: what are your thoughts on this?
>>>>
>>>>
>>>>          
>>> With a packaged Rails product, for example, one aspect of the RPM package will
>>> most likely be:
>>>
>>>      Requires:  rubygem(rails)
>>>
>>> One can enforce a specific version of the Ruby gem:
>>>
>>>      Requires:  rubygem(rails) = 2.3.5
>>>
>>> in order to prevent breakage on platform or application upgrades, through
>>> system capabilities. Pretty much the same goes for Ruby gems, not just
>>> packaged applications from downstream consumers or vendors.
>>>
>>> Normally, in cases where the upstream of the dependent package is using sane
>>> versioning, one could even:
>>>
>>>      Requires: rubygem(rails)>= 2.3, rubygem(rails)<    2.4
>>>
>>> which, logically, should result in the Rails RPM package changing for bug- and
>>> security-fixes only. However, such seems to not be common practice in the
>>> larger Ruby community. This is a problem mid-streams like Fedora will have to
>>> deal with -or let it be a Free For All but then what are we doing here?
>>>
>>> Anyway, while the application may have (with emphasis on "may have"):
>>>
>>>      RAILS_GEM_VERSION = '2.3.2'
>>>
>>> this does not correspond with the capabilities provided by the system in any
>>> way. It locks the application down to using 2.3.2 and 2.3.2 only, which should
>>> thus also not be used lightly in any situation.
>>>
>>> Because of it's own little package management, we have two completely separate
>>> dependency paths each of them searching to resolve dependencies (but only one
>>> of them through global dependencies), and both are as dependent on a closed
>>> dependency tree as another.
>>>
>>> However, if you install a Ruby gem RPM, and gem update the system, your RPM is
>>> still there, and you can depend on it, and you can be sure that, should the
>>> system-wide capability be attempted to be removed in any case, the transaction
>>> is going to fail because of missing dependencies.
>>>
>>> Imagine you gem install rails but then yum remove ruby -just to emphasize the
>>> point I'm trying to make here.
>>>
>>> In another example, gem install mysql and then yum update mysql from 5.0 to
>>> 5.1, or yum update ruby from 1.8.6 to 1.8.7.
>>>
>>> There can only be one package manager doing all the work. It's RPM.
>>>
>>> Does that limit anyone from using Gem? No, we're shipping it and in the end,
>>> it's your system. Knock yourself out.
>>>
>>> Do you have the liberty to derive from what we provide, either by using gem or
>>> apt or ...? Sure! Have fun.
>>>
>>> Does that mean the RPMs provided are worthless? Euh, not so much.
>>>
>>> For one, RPM builds are reproducible. If a test fails on one build system, it
>>> will fail on another build system in very much the same way (exceptions
>>> apply). If compilation fails because it's using a dynamic library instead of a
>>> static library to compile against, or it uses the latest and greatest version
>>> rather then the compat- version. We'll figure that out and fix it.
>>> Documentation for the upstream gem only accounts for so much of so many system
>>> aspects.
>>>
>>>
>>>        
>>>> 2) A valid Rails RPM package must contain the version when specifying
>>>> gems and the version must be limited.
>>>> Example:
>>>> # require 'aws/s3'
>>>> config.gem 'aws-s3', :lib =>    'aws/s3', :version =>    '0.4.0',      #
>>>> :source =>    "http://code.whytheluckystiff.net"
>>>>
>>>>          
>>> Actually, this is not the case. in a config.gem statement, one only is
>>> required to list the name, and all other parameters are optional.
>>>
>>> For example, one my require a mysql gem, but not care about the version -
>>> because the platforms the package or application is distributed for includes
>>> the correct version and the mysql gem doesn't change that much.
>>>
>>>
>>>        
>>>> (see also
>>>> http://api.rubyonrails.org/classes/Rails/Configuration.html#M002064)
>>>> This means that the module will use any version greater than 0.4.0. You
>>>> can also specify smaller, a range or a specific number. The trick is
>>>> this: if you do this correctly, you can use multiple gem versions on the
>>>> same system. In theory, this must work. This is what was causing the
>>>> issue with Lighthouse. I think you should fix the version or make it a
>>>> range, never a 'greater than' (as in the example and with Lighthouse).
>>>> My question: what are your thoughts on this, especially the mandatory
>>>> 'limit the version number'?
>>>>
>>>>
>>>>          
>>> Multiple versions of a single applications are useful in a situation in which,
>>> from the perspective of a midstream distributor, each and every version can be
>>> appropriately maintained. Nevermind the bug-fixes and let's go straight to
>>> security fixes. We, the mid-stream distributor, have a tendency to solve most
>>> of those security issues for you, and give you the appropriate, fixed version,
>>> without breaking anything. It's something virtually unheard of in the greater
>>> Ruby community, where feature enhancements and API breakage go hand-in-hand in
>>> a release marked as a Security related release.
>>>
>>> That said, we do have mechanisms in place that allow us to install more then
>>> one version of an application through RPM. We currently use that mechanism for
>>> kernel packages. It has a setting that is called "installonlyn_limit", so you
>>> may choose how many versions (or actually, RPM releases, not application
>>> versions) of the package should be preserved.
>>>        
>> 3) Gems can be vendorized.
>>      
>>>> OK, this is the moment where a lot of people are going to be amazed,
>>>> mouth open and staring at the screen for the next five minutes: you can
>>>> make a Rails application and include the gems in it, as it where local
>>>> code. This is called vendorization.
>>>>
>>>>          
>>> </sarcasm>    ;-)
>>>
>>>
>>>        
>>>> A vendorized gem is only available
>>>> to the application using it: it is part of the source code of that
>>>> application and located in the /vendor directory of the application root
>>>> (not in the normal gem directory). In fact, you can vendorize a gem and
>>>> then even modify the code.
>>>> On one hand, if you say all gems should be vendorized, you don't have
>>>> any version conflicts. On the other hand, I heard someone complain about
>>>> static linking...
>>>> Personally, I disagree with the vendorization of gems. It obfuscates
>>>> version management.
>>>> Question: does Fedore prefer (or mandatorize) a specific approach? Why?
>>>>
>>>>
>>>>          
>>> We do not allow the shipping of vendored code or libraries. A good explanation
>>> of what vendored code entails is at
>>> https://bugzilla.redhat.com/show_bug.cgi?id=470696#c37
>>>
>>> That does not mean, however, that downstream consumers or Independent Software
>>> Vendors are not allowed to vendor parts of their project or product. We
>>> discourage it, for all the right reasons might I add, but we can not (and
>>> should not) prohibit it.
>>>
>>> Vendored code is not allowed for the parts that the Fedora Project
>>> distributes, hence if a product is ever to be a part of the distribution, the
>>> vendored code needs to go.
>>>
>>> -- Jeroen
>>>
>>>        
>> Nice comments! I think we're getting to a nice fundamental problem here:
>> Given the facts:
>> 1) Higher versions of gems (or Ruby itself) can have changed API's. In
>> fact, Ruby 1.9 is very different (just about everything breaks) from
>> Ruby 1.8.x. Also, have a look at the change in
>> http://redmine.ruby-lang.org/issues/show/2713. Yes, it's not a bug but a
>> feature. It also means that just about every gem will break (this is
>> like deleting ncurses from a C development system).
>> 2) Mid stream distributor (Fedora) wants to be able to solve security
>> issues.
>> 3) A principle of mid stream distributor (Fedora) is to be able to patch
>> in a transparent fashion. That is, code may not be obfuscated. This
>> principle means, among other things, that vendorized code will not be
>> packaged.
>> 4) A principle of Ruby is to have your code tested. In fact: 'Writing
>> applications without tests makes you a bad person, incapable of love.'
>> (Wilson - The Rails Way). IMHO this can be used to Fedora advantage:
>> remember, you can confront every Ruby developer with this principle -
>> and they *will* appreciate you for doing so! (This is some of the most
>> valued principles of the Ruby community - in fact, the first point made
>> above is not an issue for ruby developers as they will point out that
>> it's all about the tests, not the code)
>>
>> This leads me to suggesting the following solution:
>> 1) Given a codebase C, with gems G1..Gn. Let's say one of the gems is
>> the CBAC gem, version 1.0.
>> 2) No gems are vendorized.
>> 2) Run tests and verify application.
>> 3) Package the gems.
>> 4) Package the application. Change the application so the gems are fixed
>> to a specific version.
>> 5) Test and verify. We now have a stable build. Package.
>> Let's say the following happens:
>> 6) A security flaw is detected in the CBAC gem, fixed in version 1.7
>> 7) Change the application to use CBAC gem version 1.1 instead of 1.0.
>> Run tests to verify application.
>> 8) Do this until you reach version 1.7.
>> 9) If everything goes OK, commit the change (eg the version number) to
>> the application, repackage and you're done.
>> 10) If the application breaks (let's say at version 1.4), either do one
>> of the following:
>>        - Fix the app so it can work with 1.7 (can be done by the app
>> developer)
>>        - Fix the security flaw in version 1.4 (can be done by the gem
>> developer)
>> In both cases, the security flaw is fixed, tested, and packaged.
>> ...
>> Packaging of the applications and gems can be done using a sandbox system:
>> - Basic Fedora version x system
>> - Ruby interpreter
>> - Run test of the application/ gem
>>
>> Remember: in Ruby land, if the tests pass and the app/ gem fails, the
>> code is bad. You can use Wilson's quote to make your point across. Also,
>> this way Fedora would encourage Ruby principles. Personally, as a
>> Rubyist, I would love this approach.
>>
>> What do you guys think?
>>
>>      
> Well, if I understand what you want to say here correctly, all
> you want to discuss here is not limited to ruby or rubygems, and
> we follow the package updates policy in the distribution (Fedora)
>
>
> Regards
> Mamoru
> _______________________________________________
> ruby-sig mailing list
> ruby-sig at lists.fedoraproject.org
> https://admin.fedoraproject.org/mailman/listinfo/ruby-sig
>    
I can't quite follow you on this one Mamoru, but what I'm trying to say 
is that there may be a way to use the Ruby development chain to the 
advantage of Distro packaging. The test philosophy used by Ruby allows a 
totally different perspective on development and maintenance. Without 
acknowledgment of this philosophy, I think it will become next to 
impossible to maintain the packages. Of course, before doing all sorts 
of stuff, I would like to know what you think of this.

Regards,
    Bert