Node.js module packaging [was Re: Summary/Minutes for today's FESCo meeting (2013-01-16)]

Wed Jan 16 21:21:17 UTC 2013

On 1/16/13, Marcela Mašláňová <mmaslano at redhat.com> wrote:
> * #992 F19 Feature: NodeJS -
>    https://fedoraproject.org/wiki/Features/NodeJS  (mmaslano, 18:04:56)
>    * AGREED: NodeJS is approved; for the feature to be considered
>      complete, packaging guidelines must be approved (+8,-0)  (mmaslano,
>      18:17:50)
>    * sgallagh will speak about guidelines with FPC  (mmaslano, 18:18:29)

Now that we're discussing features in fedora-devel prior to FESCo
voting on them, it would be nice if FESCo members could raise some of
their concerns in the e-mail discussion prior to the meeting.  I
couldn't really make the meeting and it seemed like the most of the
concerns around node modules had been addressed already in that
thread, so I wasn't too concerned about it.

That way, most of your questions would have already been answered.  ;-)

> 18:04:56 <mmaslano> #topic #992 F19 Feature: NodeJS - https://fedoraproject.org/wiki/Features/NodeJS
> 18:05:03 <mmaslano> .fesco 992
> 18:05:05 <zodbot> mmaslano: #992 (F19 Feature: NodeJS - https://fedoraproject.org/wiki/Features/NodeJS) – FESCo - https://fedorahosted.org/fesco/ticket/992
> 18:05:49 <sgallagh> I proposed this feature, but the budding Node.js community within Fedora has really taken off with it.
> 18:06:09 <sgallagh> A few dozen node packages are in the review queue already, several of them approved.
> 18:06:19 <mmaslano> sgallagh: I know you was working with upstream, so there shouldn't be any serious problems
> 18:06:20 <jwb> one sec
> 18:06:33 * nirik is a bit worried about npm package manager.
> 18:06:36 <jwb> last week we decided to vote en bloc for features that have no objections
> 18:06:47 <sgallagh> We need to sort out an official packaging policy, which I'm going to start brainstorming at FUDCon, but the packages that have gone in so far are taking the slightly hackish symlink approach that avoids bundling.

As I mentioned to sgallagh in one of the node module review bugs
earlier, symlinking to deps isn't a "hack that avoids bundling"--it's
the way upstream intends things to work.  See the npm FAQ [1] for some
of the rationale behind it.

> 18:07:02 <notting> nirik: how does that work exactly? rpm-installed modules are registered in a local npm database too?

npm doesn't use a database, it just checks a set of well-known paths
(listed in `man npm-folders` [3]).  My RPMs install into the global
module path, /usr/lib/node_modules, so npm finds them just fine.

> 18:07:04 <t8m> jwb, I suppose this one is not such case - I think some discussion about npm is relevant
> 18:07:20 <abadger1999> sgallagh: <nod>  I am a little afraid of the explosion of js packages without JS guidelines.
> 18:07:30 <jwb> t8m, sure.  but we could get those without objection out of the way first...  otherwise we're just going to go through them all anyway
> 18:07:38 <sgallagh> Yeah, I can understand that. I'm learning as I go here as well.
> 18:07:48 <nirik> notting: my understanding is that it's a per user install type of thing... or per app tree.
> 18:08:06 <mmaslano> sgallagh: so how the npm is working?
> 18:08:09 <nirik> so, no different than some of the other $lang things.
> 18:08:34 <t8m> mmaslano, could you please propose which features are uncontroversial and can be voted together after we close discussing this feature?
> 18:08:38 <sgallagh> We haven't gotten that far yet. We've (Well, mostly T.C. Hollingsworth) been packaging the dependencies for npm
> 18:08:44 <mmaslano> t8m: yes
> 18:09:01 <mitr> AFAICS there isn't a local database - but a per-project dependency subtree into which a command can symlink the globally-installed libraries

Exactly.

> 18:09:03 <sgallagh> We're trying to keep things to a common location and we're looking into working with upstream to add a set of search paths rather than the hackish symlinking we're doing in the packages.

Again, as I mentioned in the bug [1], node supports search paths just
fine, but our packages will infect the node module search path with
all RPM-installed modules and break upstream's desired module
isolation.

> 18:10:16 * nirik is conditionally +1, but would like guidelines and the packages that meet those guidelines.
> 18:10:18 <abadger1999> sgallagh: k.  If we can get to the point of common location, documentation on how to do the symlinking, and naming/versioning that's probably enough to get an initial set of guidelines.

The common location is defined by the %{nodejs_sitelib} macro as
%{_prefix}/lib/node_modules, and is equivalent to where `npm install
-g` would install packages, which it defines as "$(npm prefix
-g)/node_modules".  Even a vanilla npm installation from upstream will
work fine with this.

The symlinking is achieved using the %nodejs_symlink_deps RPM macro,
which points to a trivial Python script that reads the dependencies
out of the JSON metadata included with every node module and does the
necessary symlinking and should be run at the end of the %install
section of every node module RPM.

I figured the naming guidelines would follow python, ruby, et. al.
with "nodejs-" prefixes for libraries and regular names for apps.

This stuff is mostly already covered by the draft packaging guidelines
[4] I wrote and linked to in the Scope section of the NodeJS Feature
page.  They're a little rough around the edges at the moment but I
hope to clean them up and provide a detailed rationale for some of the
stuff unique to Node in the next few days.

> 18:10:22 <t8m> Can we tentatively approve the feature requesting packaging guidelines be finished before some point of time? For example F19 branch point.
> 18:10:40 <sgallagh> nirik: I'm okay with updating the Feature page to require approved guidelines as part of the feature completion criteria
> 18:10:41 <t8m> or maybe even earlier
> 18:11:29 <t8m> or we can postpone the feature approval until such guide exists
> 18:11:42 <sgallagh> t8m: As I said above, I'm going to try to hammer out the basics of this at FUDCon this week, so I think branch is achievable :)
> 18:12:03 <mitr> I can't see that postponing the approval would really change anything - the guidelines are not yet approved and packages are getting into the distribution anyway
> 18:12:23 <mitr> I don't think our decision depends on what the guidelines look like exactly, or does it?
> 18:12:30 <nirik> unless we want to say: don't add anymore of them until we have basic guidelines?
> 18:12:31 * abadger1999 would be okay with nodejs as a feature now but would like to avoid having many server-side javascript packages until we have guidelines.
> 18:12:47 <mitr> Right
> 18:13:41 <t8m> nirik, Yes, that's what I'd like to say by the postponed approval
> 18:13:46 <notting> i'd agree with mitr though - if all we want is approved guidelines before packging, i thik we can still approve the feature
> 18:14:04 <abadger1999> nirik: +1
> 18:14:34 <abadger1999> mitr: Well... in the past FPC + fesco has put freezes on packages getting in that were written in a language that didn't have guidelines yet.
> 18:15:12 <mitr> Proposal: NodeJS is approved; for the feature to be considered complete, packaging guidelines must be approved by feature freeze and any preexisting packages modified to comply (if necessary) by beta freeze.  Feature owners are strongly encouraged to get the guidelines approved before adding more packages.

I'll refrain from pushing stable any node modules until guidelines are
approved and my packages are modified to suit them.  Will that work?

> 18:15:45 <notting> +1
> 18:15:46 <abadger1999> mitr: +1
> 18:15:50 <jwb> sure +1
> 18:15:50 <pjones> mitr: +1
> 18:15:51 <sgallagh> mitr: I'm +1 to that (and speaking as the feature owner, sure)
> 18:15:55 <nirik> +1
> 18:16:12 <t8m> +1
> 18:16:28 <mmaslano> +1
> 18:16:35 <mitr> (that's the FESCo /feature side; given that these are new packages, there's little integration concern.  OTOH, perhaps FPC might want to object to adding _any_ node.js packages now when guidelines don't exist)
> 18:17:12 <mitr> I'm not sure we have a precedent/established practice WRT new languages
> 18:17:25 <sgallagh> mitr: I think that's a bit of a chicken-and-egg problem. As this is new territory, some of the issues we're discovering will only be possible to document when we have nested deps.
> 18:17:28 <mitr> perhaps we should establish one?

+1, it would be nice to have a procedure for doing this kind of stuff
in the future.

> 18:17:28 <abadger1999> with my FPC hat, I do kinda -- but hopefully sgallagh can sit down with enough of us FPC people at fudcon to get something workable by next week.
> 18:17:35 <pjones> mitr: I think we just did
> 18:17:47 <mitr> pjones: only as long as someone remembers this meeting
> 18:17:50 <mmaslano> #agreed NodeJS is approved; for the feature to be considered complete, packaging guidelines must be approved (+8,-0)
> 18:18:02 <mitr> sgallagh: yes

Thanks, everyone!

-T.C.

[1] https://bugzilla.redhat.com/show_bug.cgi?id=891213#c5
[2] https://npmjs.org/doc/faq.html#I-installed-something-globally-but-I-can-t-require-it
[3] https://fedoraproject.org/wiki/PackagingDrafts:Node.js