9:39 a.m.
I am sending this to the Fedora nodejs community / SIG before sending
it to fedora-devel and/or the packaging committee. I'm doing that so
we (the people who actually maintain, or try to maintain, the nodejs
packages) have a chance to talk and discuss things without too much
thread jumping.
Problem1:
The current way of packaging every nodejs module/library as it's own
package has created library stagnation. It is extremely hard to
update any of these modules due to other modules depending on already
released versions. It is also very hard to add any Fedora package
that requires new nodejs libraries due to the volume of package
reviews that have to happen.
Problem2:
Due to problem1, and other reasons, some of our major nodejs packagers
are orphaning their nodejs packages. This is leading to many packages
being unable to install. This is starting a downward spiral of
uninstallable nodejs packages.
Proposed Solution:
Keep nodejs and npm in Fedora. Possibly yarn too.
Get rid of all non-essential nodejs libraries. Those that are
essential, bundle all their build, test, and runtime nodejs libraries.
Create scripts that make it easy to bundle, and maintain, these
essential nodejs libraries.
If needed, create scripts that make it easy for non-nodejs packages to
bundle their nodejs libraries properly.
LIST OF PROBLEMS TO WORK OUT:
Problem1: Currently, all the libraries/modules are links under
<package>/node_modules/ and dnf doesn't like changing links into
directories.
Solution1: Bundled runtime libraries/modules will be in
<package>/bundled_node_modules/ and all the links in
<package>/node_modules/ will point there.
Problem2: Many of the bundled libraries are archful, meaning they need
to be compiled. What if an essential nodejs library was originally a
noarch?
Solution2: All bundled nodejs library packages will be treached as
archful, meaning they will need to be built on each supported arch.
Problem3: TBD
Solution3: TBD
Problem4: TBD
Problem4: TBD
What do you, the Fedora nodejs packagers and community think of this proposal?
Does it sound ok?
If so, let's start working through all the problems, finding solutions.
If not, then let's find a different solution to our current crisis.
Thanks for your time.
Troy Dawson
9:55 a.m.
On 21/07/2020 15:39, Troy Dawson wrote:
Problem2: Many of the bundled libraries are archful, meaning they
need
to be compiled. What if an essential nodejs library was originally a
noarch?
Solution2: All bundled nodejs library packages will be treached as
archful, meaning they will need to be built on each supported arch.
I think "many" is stretching things.
There were never more than about a dozen or so archful modules in
Fedora and most of them were not widely used.
I suspect that archful modules are probably best kept packaged as
they are now mostly just because trying to build them as part of
a large bundled tree is going to be incredibly painful.
Tom
--
Tom Hughes (tom(a)compton.nu)
http://compton.nu/
8:31 a.m.
On Tue, Jul 21, 2020 at 10:56 AM Tom Hughes via nodejs
<nodejs(a)lists.fedoraproject.org> wrote:
On 21/07/2020 15:39, Troy Dawson wrote:
> Problem2: Many of the bundled libraries are archful, meaning they need
> to be compiled. What if an essential nodejs library was originally a
> noarch?
> Solution2: All bundled nodejs library packages will be treached as
> archful, meaning they will need to be built on each supported arch.
I think "many" is stretching things.
There were never more than about a dozen or so archful modules in
Fedora and most of them were not widely used.
I suspect that archful modules are probably best kept packaged as
they are now mostly just because trying to build them as part of
a large bundled tree is going to be incredibly painful.
I think this makes sense as well. Essentially, we should just say that
archful nodejs modules are "essential" (in the words of the proposal)
and therefore need to be packaged and shipped separately.
I think a "npm2rpm" script would have to do a couple things-- does
this sound about right?
a) recursively parse package.json to get a list of all dependencies
and tarballs, and use this to write out a list of Sources.
b) for each package, also look at package.json to see what files need
to be installed, and write out instructions for the %install section.
c) generate a "License" field for the package and also install all the
relevant licenses in %license.
There should probably be a way to specify that a given dependency is
not being bundled too, to stop the parser from descending into that
dependency tree. I imagine that (at least during the transition to
this and maybe afterwards, in the case of archful or essential
modules) it would be very useful to have packages with a mixture of
bundled and unbundled dependencies.
Ben Rosser
10:24 a.m.
On Wed, Jul 22, 2020 at 6:31 AM Ben Rosser <rosser.bjr(a)gmail.com> wrote:
On Tue, Jul 21, 2020 at 10:56 AM Tom Hughes via nodejs
<nodejs(a)lists.fedoraproject.org> wrote:
>
> On 21/07/2020 15:39, Troy Dawson wrote:
>
> > Problem2: Many of the bundled libraries are archful, meaning they need
> > to be compiled. What if an essential nodejs library was originally a
> > noarch?
> > Solution2: All bundled nodejs library packages will be treached as
> > archful, meaning they will need to be built on each supported arch.
>
> I think "many" is stretching things.
>
> There were never more than about a dozen or so archful modules in
> Fedora and most of them were not widely used.
>
> I suspect that archful modules are probably best kept packaged as
> they are now mostly just because trying to build them as part of
> a large bundled tree is going to be incredibly painful.
I think this makes sense as well. Essentially, we should just say that
archful nodejs modules are "essential" (in the words of the proposal)
and therefore need to be packaged and shipped separately.
I think a "npm2rpm" script would have to do a couple things-- does
this sound about right?
a) recursively parse package.json to get a list of all dependencies
and tarballs, and use this to write out a list of Sources.
b) for each package, also look at package.json to see what files need
to be installed, and write out instructions for the %install section.
c) generate a "License" field for the package and also install all the
relevant licenses in %license.
There should probably be a way to specify that a given dependency is
not being bundled too, to stop the parser from descending into that
dependency tree. I imagine that (at least during the transition to
this and maybe afterwards, in the case of archful or essential
modules) it would be very useful to have packages with a mixture of
bundled and unbundled dependencies.
Ben Rosser
You are both right.
Odd as it seems, I kept seeing nodejs-sqlite3 come up, which is archful.
I had never done a real search.
It turns out that the various packages I had been working on require
it, and only two other packages.
So ya, Ben, I like you idea. I was looking at a very skewed view.
Troy
4:38 p.m.
On Wed, Jul 22, 2020 at 8:24 AM Troy Dawson <tdawson(a)redhat.com> wrote:
On Wed, Jul 22, 2020 at 6:31 AM Ben Rosser <rosser.bjr(a)gmail.com> wrote:
>
> On Tue, Jul 21, 2020 at 10:56 AM Tom Hughes via nodejs
> <nodejs(a)lists.fedoraproject.org> wrote:
> >
> > On 21/07/2020 15:39, Troy Dawson wrote:
> >
> > > Problem2: Many of the bundled libraries are archful, meaning they need
> > > to be compiled. What if an essential nodejs library was originally a
> > > noarch?
> > > Solution2: All bundled nodejs library packages will be treached as
> > > archful, meaning they will need to be built on each supported arch.
> >
> > I think "many" is stretching things.
> >
> > There were never more than about a dozen or so archful modules in
> > Fedora and most of them were not widely used.
> >
> > I suspect that archful modules are probably best kept packaged as
> > they are now mostly just because trying to build them as part of
> > a large bundled tree is going to be incredibly painful.
>
> I think this makes sense as well. Essentially, we should just say that
> archful nodejs modules are "essential" (in the words of the proposal)
> and therefore need to be packaged and shipped separately.
>
> I think a "npm2rpm" script would have to do a couple things-- does
> this sound about right?
>
> a) recursively parse package.json to get a list of all dependencies
> and tarballs, and use this to write out a list of Sources.
> b) for each package, also look at package.json to see what files need
> to be installed, and write out instructions for the %install section.
> c) generate a "License" field for the package and also install all the
> relevant licenses in %license.
>
> There should probably be a way to specify that a given dependency is
> not being bundled too, to stop the parser from descending into that
> dependency tree. I imagine that (at least during the transition to
> this and maybe afterwards, in the case of archful or essential
> modules) it would be very useful to have packages with a mixture of
> bundled and unbundled dependencies.
>
> Ben Rosser
>
You are both right.
Odd as it seems, I kept seeing nodejs-sqlite3 come up, which is archful.
I had never done a real search.
It turns out that the various packages I had been working on require
it, and only two other packages.
So ya, Ben, I like you idea. I was looking at a very skewed view.
Troy
Sorry for being so quiet.
I was going through all of Fedora's dependencies, and double checking
my results because I was having a hard time believing what I was
seeing.
I wanted to get a list of essential nodejs packages, and I thought of
getting a list of nodejs packages that are required by non-nodejs
packages.
That list is amazingly small.
# Binaries / Runtime packages
statsd -> nodejs-generic-pool
R-shiny -> nodejs-showdown
notepadqq -> nodejs-shelljs
mocha -> uglify-js and nodejs-jade
kosmtik -> 11 nodejs libraries
jake -> 5 nodejs libraries
elasticdump -> 3 nodejs libraries
discord-irc -> 10 nodejs libraries
carto -> 4 nodejs libraries
# Source Dependencies
statsd
R-shiny
python-webassets
perl-Mojolicious-Plugin-AssetPack
mocha
kosmtik
jake
elasticdump
discord-irc
carto
Note: Due to the current high volume of uninstallable packages, and
being in the middle of a mass rebuild, it's possible I'm off by a few
packages.
I also left off js-jquery*. But even if you think we should include
the js-* packages, that's still only about 5-10 more packages.
Most of the binary/runtime dependencies did not have many duplicates.
So we are looking at 30-40 packages if we wanted to consider all of
them "essential".
I would think it would be better to have the high volume ones (jake,
kosmtik, discord-irc) bundle their nodejs dependencies.
So, after this exercise, I'm sort of at a loss as to what we would
consider essential. So I'm going to throw out a short list. Please
add or subtract from it.
Once we figure on a list, we can start figuring out a plan on how to
trim Fedora down to that list.
nodejs (Have to have)
npm (Have to have)
yarnpkg
js-jquery
uglify-js
Thanks
Troy
4:44 p.m.
On 29/07/2020 22:38, Troy Dawson wrote:
I wanted to get a list of essential nodejs packages, and I thought
of
getting a list of nodejs packages that are required by non-nodejs
packages.
How have you defined essential?
That list is amazingly small.
# Binaries / Runtime packages
statsd -> nodejs-generic-pool
R-shiny -> nodejs-showdown
notepadqq -> nodejs-shelljs
mocha -> uglify-js and nodejs-jade
kosmtik -> 11 nodejs libraries
jake -> 5 nodejs libraries
elasticdump -> 3 nodejs libraries
discord-irc -> 10 nodejs libraries
carto -> 4 nodejs libraries
As both kosmtik and carto originate with me - indeed kosmtik
is kind of the original cause of my involvement in Node.js
pacaging I can comment a bit on those.
I wouldn't actually worry too much about keeping kosmtik to
be honest.
While carto is probably more useful the current version is
rather out of date, because of tricky dependency issues in
the newer versions so that dependency count will go up if it
is updated I think.
I also wonder if you missed some build/test toolchains
that are required? Only that is where of the nasty trees
of dependencies are I think.
Tom
--
Tom Hughes (tom(a)compton.nu)
http://compton.nu/
5:27 p.m.
On Wed, Jul 29, 2020 at 2:44 PM Tom Hughes <tom(a)compton.nu> wrote:
On 29/07/2020 22:38, Troy Dawson wrote:
> I wanted to get a list of essential nodejs packages, and I thought of
> getting a list of nodejs packages that are required by non-nodejs
> packages.
How have you defined essential?
That's what I still don't know and I'd love some feedback.
nodejs is one of the few languages in Fedora that allows bundling.
It's fairly new (a year or two) and I'd like to push it to it's extreme.
Ideal world (not gonna happen, but this is my 'everything works' goal)
Fedora only has nodejs and npm as packages.
If a package needs a nodejs library, they bundle it, and there is an
easy way to bundle it.
If a developer and/or end user needs a nodejs library, they use npm,
and not an rpm.
With that as the goal, my definition of essential is any nodejs
library that so many other packages require, that it is more work to
bundle it everywhere, versus having a Fedora package for it.
I'm willing to to change that definition of essential, but that's what
I'm starting off with.
> That list is amazingly small.
>
> # Binaries / Runtime packages
> statsd -> nodejs-generic-pool
> R-shiny -> nodejs-showdown
> notepadqq -> nodejs-shelljs
> mocha -> uglify-js and nodejs-jade
> kosmtik -> 11 nodejs libraries
> jake -> 5 nodejs libraries
> elasticdump -> 3 nodejs libraries
> discord-irc -> 10 nodejs libraries
> carto -> 4 nodejs libraries
As both kosmtik and carto originate with me - indeed kosmtik
is kind of the original cause of my involvement in Node.js
pacaging I can comment a bit on those.
I wouldn't actually worry too much about keeping kosmtik to
be honest.
While carto is probably more useful the current version is
rather out of date, because of tricky dependency issues in
the newer versions so that dependency count will go up if it
is updated I think.
I also wonder if you missed some build/test toolchains
that are required? Only that is where of the nasty trees
of dependencies are I think.
Actually, that is one of the points of this. Making those build/test
dependency trees go away.
Not that the tests would go away, but that the build/test dependencies
would be bundled in a tarball.
Each time you update your nodejs library, you run the script that
re-makes the build-test tarball.
During the build, they would be untarred, build and tests would be
run, and they go away when the rpm is packaged.
Troy
10:09 a.m.
Hi!
I'm one of the maintainers of Grafana in Fedora.
Not that the tests would go away, but that the build/test
dependencies would be bundled in a tarball.
I really like the idea, however (using latest Grafana):
$ du -sch node_modules
1,7G node_modules
Some of the largest dependencies:
$ du -am node_modules | sort -n -r | head -n 20
1719 node_modules
291 node_modules/@storybook
85 node_modules/@babel
81 node_modules/@storybook/core
78 node_modules/@storybook/core/node_modules
73 node_modules/monaco-editor
66 node_modules/@types
58 node_modules/@microsoft
54 node_modules/@microsoft/api-extractor
53 node_modules/typescript
52 node_modules/@microsoft/api-extractor/node_modules
50 node_modules/typescript/lib
49 node_modules/@microsoft/api-extractor/node_modules/typescript
46 node_modules/@microsoft/api-extractor/node_modules/typescript/lib
38 node_modules/@svgr
36 node_modules/@svgr/webpack/node_modules
36 node_modules/@svgr/webpack
35 node_modules/@svgr/webpack/node_modules/@babel
33 node_modules/monaco-editor/dev/vs
33 node_modules/monaco-editor/dev
$ ls node_modules | wc -l
2017
Compressing the node_modules folder using zstd:
default level: 179 MB
zstd level 19: 108 MB
How can/should we bundle that? Is this size acceptable in a SRPM?
Bundling it would allow us to build the frontend and run the frontend tests as part of the
spec, and we can apply patches properly (without (re)generating anything locally). Most of
the patches at least, in case some security patch requires an upgrade of a npm dependency,
we also need to recreate the vendor tarball.
Cheers,
Andreas
3:15 p.m.
On Wed, Jul 22, 2020 at 6:31 AM Ben Rosser <rosser.bjr(a)gmail.com> wrote:
On Tue, Jul 21, 2020 at 10:56 AM Tom Hughes via nodejs
<nodejs(a)lists.fedoraproject.org> wrote:
>
> On 21/07/2020 15:39, Troy Dawson wrote:
>
> > Problem2: Many of the bundled libraries are archful, meaning they need
> > to be compiled. What if an essential nodejs library was originally a
> > noarch?
> > Solution2: All bundled nodejs library packages will be treached as
> > archful, meaning they will need to be built on each supported arch.
>
> I think "many" is stretching things.
>
> There were never more than about a dozen or so archful modules in
> Fedora and most of them were not widely used.
>
> I suspect that archful modules are probably best kept packaged as
> they are now mostly just because trying to build them as part of
> a large bundled tree is going to be incredibly painful.
I think this makes sense as well. Essentially, we should just say that
archful nodejs modules are "essential" (in the words of the proposal)
and therefore need to be packaged and shipped separately.
I think a "npm2rpm" script would have to do a couple things-- does
this sound about right?
a) recursively parse package.json to get a list of all dependencies
and tarballs, and use this to write out a list of Sources.
b) for each package, also look at package.json to see what files need
to be installed, and write out instructions for the %install section.
c) generate a "License" field for the package and also install all the
relevant licenses in %license.
There should probably be a way to specify that a given dependency is
not being bundled too, to stop the parser from descending into that
dependency tree. I imagine that (at least during the transition to
this and maybe afterwards, in the case of archful or essential
modules) it would be very useful to have packages with a mixture of
bundled and unbundled dependencies.
Ben Rosser
Hi Ben,
Sorry it's taken so long for me to properly reply to this.
I'm going to refer to your last paragraph (having a mixed bundled and
unbundled) as (d)
For section (a) and (b), would you rather write our own script, or use
npm or yarn ?
In my mind I was thinking of doing something like "npm install" then
using the resulting node_modules directory.
Turn that into one tarball.
It sounds like you would like a seperate source for each package, at
least for the top level package.
I can see some pro's and con's for both.
Single Tarball - Good Things:
- Don't have to write a new script for downloading.
- Everything in one place.
- It's laid out how npm would lay it out.
Single Tarball - Bad Things:
- Harder to do mixed bundles (d)
- Have to redo the entire tarball for any update to any part of it.
Multi Sources - Good Things:
- Have everything parsed and labeled at creation time.
- Easier to do "this is only for build/test" and remove them for runtime.
- Easier to do mixed bundles (d)
Multi Sources - Bad Things:
- Have to create our own script (?)
- Complicated "Sources" section in spec file (100's to 1000)
I'm going to research this some more. But if you already have some
idea's, I'd love to hear them.
Troy
5:11 p.m.
On Thu, Jul 30, 2020 at 1:15 PM Troy Dawson <tdawson(a)redhat.com> wrote:
On Wed, Jul 22, 2020 at 6:31 AM Ben Rosser <rosser.bjr(a)gmail.com> wrote:
>
> On Tue, Jul 21, 2020 at 10:56 AM Tom Hughes via nodejs
> <nodejs(a)lists.fedoraproject.org> wrote:
> >
> > On 21/07/2020 15:39, Troy Dawson wrote:
> >
> > > Problem2: Many of the bundled libraries are archful, meaning they need
> > > to be compiled. What if an essential nodejs library was originally a
> > > noarch?
> > > Solution2: All bundled nodejs library packages will be treached as
> > > archful, meaning they will need to be built on each supported arch.
> >
> > I think "many" is stretching things.
> >
> > There were never more than about a dozen or so archful modules in
> > Fedora and most of them were not widely used.
> >
> > I suspect that archful modules are probably best kept packaged as
> > they are now mostly just because trying to build them as part of
> > a large bundled tree is going to be incredibly painful.
>
> I think this makes sense as well. Essentially, we should just say that
> archful nodejs modules are "essential" (in the words of the proposal)
> and therefore need to be packaged and shipped separately.
>
> I think a "npm2rpm" script would have to do a couple things-- does
> this sound about right?
>
> a) recursively parse package.json to get a list of all dependencies
> and tarballs, and use this to write out a list of Sources.
> b) for each package, also look at package.json to see what files need
> to be installed, and write out instructions for the %install section.
> c) generate a "License" field for the package and also install all the
> relevant licenses in %license.
>
> There should probably be a way to specify that a given dependency is
> not being bundled too, to stop the parser from descending into that
> dependency tree. I imagine that (at least during the transition to
> this and maybe afterwards, in the case of archful or essential
> modules) it would be very useful to have packages with a mixture of
> bundled and unbundled dependencies.
>
> Ben Rosser
>
Hi Ben,
Sorry it's taken so long for me to properly reply to this.
I'm going to refer to your last paragraph (having a mixed bundled and
unbundled) as (d)
For section (a) and (b), would you rather write our own script, or use
npm or yarn ?
In my mind I was thinking of doing something like "npm install" then
using the resulting node_modules directory.
Turn that into one tarball.
It sounds like you would like a seperate source for each package, at
least for the top level package.
I can see some pro's and con's for both.
Single Tarball - Good Things:
- Don't have to write a new script for downloading.
- Everything in one place.
- It's laid out how npm would lay it out.
Single Tarball - Bad Things:
- Harder to do mixed bundles (d)
- Have to redo the entire tarball for any update to any part of it.
Multi Sources - Good Things:
- Have everything parsed and labeled at creation time.
- Easier to do "this is only for build/test" and remove them for runtime.
- Easier to do mixed bundles (d)
Multi Sources - Bad Things:
- Have to create our own script (?)
- Complicated "Sources" section in spec file (100's to 1000)
I'm going to research this some more. But if you already have some
idea's, I'd love to hear them.
Troy
I've got a package that won't install anymore, and finally got some
time, so I've been taking a swipe at a bundling script, see how it
goes.
This isn't a complete npm2rpm. I already have a spec for this rpm.
But here is what I currently have
PACKAGE="node-static"
VERSION="$(npm view ${PACKAGE} version)"
TMP_DIR=$(mktemp -d -t ci-XXXXXXXXXX)
mkdir ${TMP_DIR}
pushd ${TMP_DIR}
npm pack ${PACKAGE}
tar xfz *.tgz
cd package
npm install --no-optional --only=prod
# FIND SOMEWHERE TO PUT LICENSES
# FOR NOW JUST PRINT THEM OUT
find . -name "package.json" -exec jq .license {} \; | sort -u
mv node_modules/ node_modules_prod
npm install --no-optional --only=dev
mv node_modules/ node_modules_dev
tar cfz ../${PACKAGE}-${VERSION}-nm-prod.tgz node_modules_prod
tar cfz ../${PACKAGE}-${VERSION}-nm-dev.tgz node_modules_dev
cd ..
cp ${PACKAGE}-${VERSION}* $HOME/rpmbuild/SOURCES
popd
rm -rf ${TMP_DIR}
This is a completely noarch package.
What I have in my %install section is this.
%install
mkdir -p %{buildroot}%{nodejs_sitelib}/%{npm_name}
cp -pr benchmark bin examples lib package.json test
%{buildroot}%{nodejs_sitelib}/%{npm_name}
# Setup Binaries
mkdir %{buildroot}%{_bindir}
ln -s %{nodejs_sitelib}/%{npm_name}/bin/cli.js %{buildroot}%{_bindir}/static
# Setup node modules
cd %{buildroot}%{nodejs_sitelib}/%{npm_name}
tar xfz %{SOURCE1}
mkdir -p node_modules
cd node_modules
ln -s ../node_modules_prod/* .
ln -s ../node_modules_prod/.bin .
This builds just fine, which is great, except for one thing.
The current rpm macros file that is figuring out the bundled
dependencies, is only looking to see if something is a link or not.
If it's a link, it thinks it isn't a dependency.
Why do I have links to this node_modules_prod directory?
That allows updates for non-bundled rpm's that have a node_modules directory.
For dnf, if a directory was a link, and you try to replace it with a
regular directory, it's not happy and fails. But if it's a link and
you replace it with another link to someplace else, everyone is happy.
Anyway, this is working fine, but the provides script needs a bit of
tweaking to handle the links.
Troy
4:15 p.m.
On Thu, Aug 27, 2020 at 3:11 PM Troy Dawson <tdawson(a)redhat.com> wrote:
On Thu, Jul 30, 2020 at 1:15 PM Troy Dawson <tdawson(a)redhat.com> wrote:
>
> On Wed, Jul 22, 2020 at 6:31 AM Ben Rosser <rosser.bjr(a)gmail.com> wrote:
> >
> > On Tue, Jul 21, 2020 at 10:56 AM Tom Hughes via nodejs
> > <nodejs(a)lists.fedoraproject.org> wrote:
> > >
> > > On 21/07/2020 15:39, Troy Dawson wrote:
> > >
> > > > Problem2: Many of the bundled libraries are archful, meaning they
need
> > > > to be compiled. What if an essential nodejs library was originally
a
> > > > noarch?
> > > > Solution2: All bundled nodejs library packages will be treached as
> > > > archful, meaning they will need to be built on each supported arch.
> > >
> > > I think "many" is stretching things.
> > >
> > > There were never more than about a dozen or so archful modules in
> > > Fedora and most of them were not widely used.
> > >
> > > I suspect that archful modules are probably best kept packaged as
> > > they are now mostly just because trying to build them as part of
> > > a large bundled tree is going to be incredibly painful.
> >
> > I think this makes sense as well. Essentially, we should just say that
> > archful nodejs modules are "essential" (in the words of the
proposal)
> > and therefore need to be packaged and shipped separately.
> >
> > I think a "npm2rpm" script would have to do a couple things-- does
> > this sound about right?
> >
> > a) recursively parse package.json to get a list of all dependencies
> > and tarballs, and use this to write out a list of Sources.
> > b) for each package, also look at package.json to see what files need
> > to be installed, and write out instructions for the %install section.
> > c) generate a "License" field for the package and also install all
the
> > relevant licenses in %license.
> >
> > There should probably be a way to specify that a given dependency is
> > not being bundled too, to stop the parser from descending into that
> > dependency tree. I imagine that (at least during the transition to
> > this and maybe afterwards, in the case of archful or essential
> > modules) it would be very useful to have packages with a mixture of
> > bundled and unbundled dependencies.
> >
> > Ben Rosser
> >
>
> Hi Ben,
> Sorry it's taken so long for me to properly reply to this.
> I'm going to refer to your last paragraph (having a mixed bundled and
> unbundled) as (d)
>
> For section (a) and (b), would you rather write our own script, or use
> npm or yarn ?
>
> In my mind I was thinking of doing something like "npm install" then
> using the resulting node_modules directory.
> Turn that into one tarball.
>
> It sounds like you would like a seperate source for each package, at
> least for the top level package.
> I can see some pro's and con's for both.
>
> Single Tarball - Good Things:
> - Don't have to write a new script for downloading.
> - Everything in one place.
> - It's laid out how npm would lay it out.
> Single Tarball - Bad Things:
> - Harder to do mixed bundles (d)
> - Have to redo the entire tarball for any update to any part of it.
>
> Multi Sources - Good Things:
> - Have everything parsed and labeled at creation time.
> - Easier to do "this is only for build/test" and remove them for runtime.
> - Easier to do mixed bundles (d)
> Multi Sources - Bad Things:
> - Have to create our own script (?)
> - Complicated "Sources" section in spec file (100's to 1000)
>
> I'm going to research this some more. But if you already have some
> idea's, I'd love to hear them.
>
> Troy
I've got a package that won't install anymore, and finally got some
time, so I've been taking a swipe at a bundling script, see how it
goes.
This isn't a complete npm2rpm. I already have a spec for this rpm.
But here is what I currently have
PACKAGE="node-static"
VERSION="$(npm view ${PACKAGE} version)"
TMP_DIR=$(mktemp -d -t ci-XXXXXXXXXX)
mkdir ${TMP_DIR}
pushd ${TMP_DIR}
npm pack ${PACKAGE}
tar xfz *.tgz
cd package
npm install --no-optional --only=prod
# FIND SOMEWHERE TO PUT LICENSES
# FOR NOW JUST PRINT THEM OUT
find . -name "package.json" -exec jq .license {} \; | sort -u
mv node_modules/ node_modules_prod
npm install --no-optional --only=dev
mv node_modules/ node_modules_dev
tar cfz ../${PACKAGE}-${VERSION}-nm-prod.tgz node_modules_prod
tar cfz ../${PACKAGE}-${VERSION}-nm-dev.tgz node_modules_dev
cd ..
cp ${PACKAGE}-${VERSION}* $HOME/rpmbuild/SOURCES
popd
rm -rf ${TMP_DIR}
This is a completely noarch package.
What I have in my %install section is this.
%install
mkdir -p %{buildroot}%{nodejs_sitelib}/%{npm_name}
cp -pr benchmark bin examples lib package.json test
%{buildroot}%{nodejs_sitelib}/%{npm_name}
# Setup Binaries
mkdir %{buildroot}%{_bindir}
ln -s %{nodejs_sitelib}/%{npm_name}/bin/cli.js %{buildroot}%{_bindir}/static
# Setup node modules
cd %{buildroot}%{nodejs_sitelib}/%{npm_name}
tar xfz %{SOURCE1}
mkdir -p node_modules
cd node_modules
ln -s ../node_modules_prod/* .
ln -s ../node_modules_prod/.bin .
This builds just fine, which is great, except for one thing.
The current rpm macros file that is figuring out the bundled
dependencies, is only looking to see if something is a link or not.
If it's a link, it thinks it isn't a dependency.
Why do I have links to this node_modules_prod directory?
That allows updates for non-bundled rpm's that have a node_modules directory.
For dnf, if a directory was a link, and you try to replace it with a
regular directory, it's not happy and fails. But if it's a link and
you replace it with another link to someplace else, everyone is happy.
Anyway, this is working fine, but the provides script needs a bit of
tweaking to handle the links.
Troy
Nevermind. It works much better if I build it on Rawhide
(nodejs-packaging-23) instead of RHEL8 (nodejs-packaging-17)
Thank you to those who helped offline.
Troy
1312
days inactive
1367
days old
nodejs@lists.fedoraproject.org
10 comments
4 participants
participants (4)
-
Andreas Gerstmayr -
Ben Rosser -
Tom Hughes -
Troy Dawson