nodejs-packagiing improvements

[T.C. Hollingsworth]

On Fri, Mar 21, 2014 at 10:59 AM, Jamie Nguyen <j at jamielinux.com> wrote:
> I'm not sure I'd use or promote such a feature.

And I don't blame you given I did such a poor job at putting forward
my proposal.  :-)  I'm singing the benefits of the new dependency
style without explaining what's so much better about them.  Let me try
again...

At the moment, most node modules use the tilde form of expressing
dependencies.  This means they use something like ~1.2.3, which
translates to >1.2.3 <1.3.  This shorthand was stolen from rubygems.

As it turns out, this highly restrictive form of dependency has been
criticized in both the ruby and nodejs communities as being too
restrictive.  Espeically in the nodejs community it's unwarranted; the
semantic versioning standard [1] that node modules are supposed to
follow (and many do in practice) dictates that the major version must
be bumped when backwards incompatible changes are made to a library.
Thus, regardless of what the dependency says it's generally possible
to upgrade to a new minor version of a library while keeping its
dependents working.

A year ago, npm upstream introduced the caret (^) operator as an
alternative to the tilde operator.  With this operator, something like
^1.2.3 translates to >1.2.3 <2.  This form of dependency much more
accurately reflects what will actually work, and is more aligned with
typical Fedora practice, where we often want to use newer compatible
versions of libraries, First being one of our foundations and all.

In the last month, npm upstream recently switched 'npm install --save'
to use carets instead of tildes, given that a year has passed since
support for it was introduced so most npm installations can reasonably
be expected to support it.  Unfortunately, as the recent attempts to
kill off the unfortunate use of a self-signed certificate for
registry.npmjs.org indicate, there are still a lot of old npm
installations in the wild (mostly thanks to PaaS providers), so many
module authors are going to choose not to use the caret form to retain
compatibility with these ancient npm versions for some time to come.

Of course, in Fedora we control the npm version provided, and indeed
we've had an npm version that supports this (even if the dependency
generator does not) for a year now as well.  So we're free to
encourage the use of it whenever we want, and using them in place of a
tilde likely reflects what upstream would love to do if they didn't
have to support PaaS providers that move at glacial paces.

Let's step back and take a look at what we're doing now.  We often
want to introduce newer versions of libraries to the package
collection than are typically defined in dependents' package.json
files.  When we do that, we have to bump the dependency in all
dependent packages.  So if we want to update nodejs-foomodule from
1.2.3 to 1.3.4, we often have to go add '%nodejs_fixdep ~1.3.4' to all
dependent packages.  For certain core modules used by a lot of
dependents, this could be a lot of work for what should be a simple
upgrade.

If we instead used something like '%nodejs_fixdep ^1.2.3' in those
dependent packages, we eliminate the need to push updates that consist
solely of modifying package.json to potentially dozens of dependent
modules whenever we update a heavily-depended-upon module.  We can
instead just update it and everything will Just Work.  Much nicer,
IMHO.

But doing this with classic %nodejs_fixdep is tedious.  Now instead of
just updating a handful of %nodejs_fixdep calls in %prep, you'd have
to double-check every one against the new package.json and make sure
you update the lower bound if it moved up.

Enter '%nodejs_fixdep --caretify'. Instead of manually keeping up with
the tilde->caret conversion, it's just done for you automatically at
build time.  It's explicitly saying "don't allow dependencies that are
of a lower version than is defined in package.json, but allow later
versions up to but not including the next major version".  It does
what's right for Fedora without second-guessing upstream too much,
unlike our current policy which overrides upstream to a greater
degree.

I should note that in practical terms, 'sed -i s/~/^/g package.json'
would accomplish the same thing, but by adding a special option to
%nodejs_fixdep we can introduce error-checking that makes packagers
less likely to shoot themselves in the foot.  (I'm not entirely happy
with --caretify as the argument, I'm open to better suggestions.)

I don't think it reduces the ability of packagers to know what's going
on in a package.  With our current practice, you either need to look
at the combination of the package.json pre-%prep and the spec file to
know how they're going to turn out, or else just look at package.json
post-%prep.  With my proposal, the vast majority of packages will just
use '--tildefy' and nothing else, so you can just look at package.json
pre-%prep and know all those tildes are going to be converted to
carets later on, or look at package.json post-%prep as per usual.

That being said, a possible alternative would be to just pretend
tildes are actually carets in the dependency generator.  Then no spec
changes are required ever, but package.json won't accurately reflect
what could be possibly installed on the system.  I rather like that
package.json and RPM are in sync at all times when it comes to modules
in the package collection.  Explicit is better than implicit.  ;-)

tl;dr
- tilde dependencies are evil
- we have something better now with carets
- we really should use them in Fedora, it'll make our lives so much easier ;-)
- given the above, --tildefy makes sense and eliminates tedious,
unnecessary manual labor on our parts

I really think there's no point in us playing dependency cops to the
degree we have been with current practices, and adopting the above
will allow us to bring much more awesomeness to the distribution much
more quickly without sacrificing quality in any way.

-T.C.

[1] http://semver.org/