Hi
I gave it a go at packaging pgadmin4 [1], to replace the current obsolete and abandoned (and crashy) pgadmin3.
pgadmin4 is a mix of Python/Flask for serverside and HTML/JS for client side, for which I wrote a minimal Qt5WebEngineView wrapper to make it appear like a desktop application. The spec is here: [2]
I have a couple of questions to eventually be able to put it up for review:
- The client side uses node/package.json to build static assets. Currently I have this in the spec
# git clonehttps://github.com/postgres/pgadmin4.git # cd pgadmin4/web # yarn install # find node_modules/webpack/lib/ -type f -exec sed -i 's|md4|sha256|g' {} ; # yarn run bundle # cd .. # tar cfJ pgadmin4-static-js-generated-$(git rev-parse --short=7 HEAD).tar.xz web/pgadmin/static/js/generated/ Source1: %{name}-static-js-generated-%{shortcommit}.tar.xz
Is this in any way acceptable? Is this compatible with "You can provide a package that uses nodejs, but you should bundle all the nodejs libraries that are needed."[3]
- I install the actual pgadmin4 files /usr/lib/pgadmin4/ and then install my custom qt wrapper to %{_bindir} which then internally invokes "python3 /usr/lib/pgadmin4/pgadmin4.py". Is /usr/lib/pgadmin4/ an appropriate directory for this?
- I currently have to manually specify the requires, which I can generate using
python3 /usr/lib/rpm/redhat/pyproject_buildrequires.py -N requirements.txt 2>/dev/null | awk '{print "Requires: "$0}'
Is there a way to automatically generate the runtime Requires, similar to the buildtime BuildRequires?
Thanks Sandro
[1] https://copr.fedorainfracloud.org/coprs/smani/pgAdmin4 [2] https://smani.fedorapeople.org/pgadmin4.spec [3] https://docs.fedoraproject.org/en-US/packaging-guidelines/Node.js/
I set this up recently using the packages from here -
No idea if you've looked at those but thought I'd point them out in case they have any good practise you can "borrow" :-)
Graham
-----Original Message----- From: Sandro Mani manisandro@gmail.com Reply-To: Development discussions related to Fedora devel@lists.fedoraproject.org To: Development discussions related to Fedora devel@lists.fedoraproject.org Subject: Packaging pgAdmin4 Date: Thu, 9 Dec 2021 14:17:41 +0100
Hi I gave it a go at packaging pgadmin4 [1], to replace the current obsolete and abandoned (and crashy) pgadmin3. pgadmin4 is a mix of Python/Flask for serverside and HTML/JS for client side, for which I wrote a minimal Qt5WebEngineView wrapper to make it appear like a desktop application. The spec is here: [2] I have a couple of questions to eventually be able to put it up for review: - The client side uses node/package.json to build static assets. Currently I have this in the spec # git clone https://github.com/postgres/pgadmin4.git # cd pgadmin4/web # yarn install # find node_modules/webpack/lib/ -type f -exec sed -i 's|md4|sha256|g' {} ; # yarn run bundle # cd .. # tar cfJ pgadmin4-static-js-generated-$(git rev-parse --short=7 HEAD).tar.xz web/pgadmin/static/js/generated/ Source1: %{name}-static-js-generated-%{shortcommit}.tar.xz
Is this in any way acceptable? Is this compatible with "You can provide a package that uses nodejs, but you should bundle all the nodejs libraries that are needed."[3]
- I install the actual pgadmin4 files /usr/lib/pgadmin4/ and then install my custom qt wrapper to %{_bindir} which then internally invokes "python3 /usr/lib/pgadmin4/pgadmin4.py". Is /usr/lib/pgadmin4/ an appropriate directory for this?
- I currently have to manually specify the requires, which I can generate using
python3 /usr/lib/rpm/redhat/pyproject_buildrequires.py -N requirements.txt 2>/dev/null | awk '{print "Requires: "$0}'
Is there a way to automatically generate the runtime Requires, similar to the buildtime BuildRequires?
Thanks Sandro [1] https://copr.fedorainfracloud.org/coprs/smani/pgAdmin4 [2] https://smani.fedorapeople.org/pgadmin4.spec [3] https://docs.fedoraproject.org/en-US/packaging-guidelines/Node.js/ _______________________________________________ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-leave@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org Do not reply to spam on the list, report it: https://pagure.io/fedora-infrastructure
On 09/12/2021 14:17, Sandro Mani wrote:
Is this in any way acceptable?
All packages must be built completely from sources.
Is this compatible with
You must remove all pre-built artifacts and minified sources before creating a vendored tarball.
NW.js must be built from sources.
On 09.12.21 16:12, Vitaly Zaitsev via devel wrote:
On 09/12/2021 14:17, Sandro Mani wrote:
Is this in any way acceptable?
All packages must be built completely from sources.
This does not appear to be accurate for nodejs packages - take i.e. node-svgo, which compliant with the guidelines bundles node_modules dir in svgo-2.8.0-nm-dev.tgz resp svgo-2.8.0-nm-prod.tgz. The difference in pgadmin is that it is not a nodejs package published on npmjs.com, so AFAICS I cannot use nodejs-packaging-bundler for case.
Is this compatible with
You must remove all pre-built artifacts and minified sources before creating a vendored tarball.
NW.js must be built from sources.
NW.js isn't used in this package.
On 09/12/2021 16:56, Sandro Mani wrote:
This does not appear to be accurate for nodejs packages - take i.e. node-svgo, which compliant with the guidelines bundles node_modules dir in svgo-2.8.0-nm-dev.tgz resp svgo-2.8.0-nm-prod.tgz.
You can vendor only sources. No prebuilt assets are allowed.
NW.js isn't used in this package.
https://github.com/postgres/pgadmin4/blob/master/pkg/linux/build-functions.s...
https://github.com/postgres/pgadmin4/blob/master/runtime/yarn.lock#L1011-L10...
I haven’t looked at pgAdmin4, but I’m the current maintainer of nodejs-svgo and several other Fedora packages that use the new NodeJS packaging guidelines. I’m not the original packager for nodejs-svgo, and I wasn’t the first to convert it to the new NodeJS guidelines. I welcome further community discussion on this topic.
The current NodeJS guidelines seem to be intended to provide a clear and relatively straightforward process for bundling NodeJS dependencies using packages from NPM.
They don’t (explicitly) ask packagers to:
- audit possibly hundreds of recursive production and dev dependencies to see if any of their contents are generated, bundled, or transpiled - figure out if the affected files are actually needed and either (1) strip them, or (2) add additional source archives and somehow build the affected dependencies from scratch as part of the RPM build, then repeat the process recursively for all the new dev dependencies required to do so - repeat the whole process every time the package is updated
It seems like if this were the typical case, it would have deserved a mention in the NodeJS guidelines.
I understand your position that this is what would be required for bundled dependencies in the general case. On the other hand, if this is truly the standard that is intended to apply to NodeJS dependency bundles, then I think it’s not an exaggeration or an overreaction to say that the new NodeJS packaging guidelines are a mistake: NodeJS packaging would be at least as unworkable as it was under the old guidelines that required unbundling everything, and nearly every package using NodeJS would need to be retired, since maintaining one package would be nearly as much effort as maintaining separate packages for its entire dependency tree.
Consider, too, the example spec file[1] in the NodeJS guidelines, which would package “tape”. You can generate the prod and dev bundles with “nodejs-packaging-bundler tape 5.1.1”, unpack them, and take a look: even the prod tarball contains a bundled browser build “object.assign/dist/browser.js”, and the dev bundle contains 28 directories named “dist”. (The prod bundle is ~2k files and the dev bundle is ~12k files including ~800 package.json’s.)
Surely the example spec file is meant to be compliant with the guidelines it illustrates?
– Ben Beasley
[1] https://docs.fedoraproject.org/en-US/packaging-guidelines/Node.js/#_example_...
On 12/9/21 11:31, Vitaly Zaitsev via devel wrote:
On 09/12/2021 16:56, Sandro Mani wrote:
This does not appear to be accurate for nodejs packages - take i.e. node-svgo, which compliant with the guidelines bundles node_modules dir in svgo-2.8.0-nm-dev.tgz resp svgo-2.8.0-nm-prod.tgz.
You can vendor only sources. No prebuilt assets are allowed.
NW.js isn't used in this package.
https://github.com/postgres/pgadmin4/blob/master/pkg/linux/build-functions.s...
https://github.com/postgres/pgadmin4/blob/master/runtime/yarn.lock#L1011-L10...
On Thursday, December 9, 2021 6:59:17 PM CET Ben Beasley wrote:
I haven’t looked at pgAdmin4, but I’m the current maintainer of nodejs-svgo and several other Fedora packages that use the new NodeJS packaging guidelines. I’m not the original packager for nodejs-svgo, and I wasn’t the first to convert it to the new NodeJS guidelines. I welcome further community discussion on this topic.
Hi,
if the packages is using yarn, you can simply use its caching mechanism for bundling the needed node modules. You will have an offline cache where you can install the modules from!
Example: https://src.fedoraproject.org/rpms/nodejs-bash-language-server
Take a look at prepare_vendor.sh which creates a tarball with the yarn cache for you.
Best regards
Andreas
Hi,
if the packages is using yarn, you can simply use its caching mechanism for bundling the needed node modules. You will have an offline cache where you can install the modules from!
Example: https://src.fedoraproject.org/rpms/nodejs-bash-language-server
Take a look at prepare_vendor.sh which creates a tarball with the yarn cache for you.
Thanks!
Sandro
On 09.12.21 17:31, Vitaly Zaitsev via devel wrote:
On 09/12/2021 16:56, Sandro Mani wrote:
This does not appear to be accurate for nodejs packages - take i.e. node-svgo, which compliant with the guidelines bundles node_modules dir in svgo-2.8.0-nm-dev.tgz resp svgo-2.8.0-nm-prod.tgz.
You can vendor only sources. No prebuilt assets are allowed.
Which would basically mean bundling the node_modules folder?
NW.js isn't used in this package.
https://github.com/postgres/pgadmin4/blob/master/pkg/linux/build-functions.s...
https://github.com/postgres/pgadmin4/blob/master/runtime/yarn.lock#L1011-L10...
Right, but I'm not packaging the nwjs runtime.
On 12/9/21 1:05 PM, Sandro Mani wrote:
On 09.12.21 17:31, Vitaly Zaitsev via devel wrote:
On 09/12/2021 16:56, Sandro Mani wrote:
This does not appear to be accurate for nodejs packages - take i.e. node-svgo, which compliant with the guidelines bundles node_modules dir in svgo-2.8.0-nm-dev.tgz resp svgo-2.8.0-nm-prod.tgz.
You can vendor only sources. No prebuilt assets are allowed.
Which would basically mean bundling the node_modules folder?
No, it would mean bundling the source from which the stuff in node_modules is generated.
On 10.12.21 01:54, Demi Marie Obenour wrote:
On 12/9/21 1:05 PM, Sandro Mani wrote:
On 09.12.21 17:31, Vitaly Zaitsev via devel wrote:
On 09/12/2021 16:56, Sandro Mani wrote:
This does not appear to be accurate for nodejs packages - take i.e. node-svgo, which compliant with the guidelines bundles node_modules dir in svgo-2.8.0-nm-dev.tgz resp svgo-2.8.0-nm-prod.tgz.
You can vendor only sources. No prebuilt assets are allowed.
Which would basically mean bundling the node_modules folder?
No, it would mean bundling the source from which the stuff in node_modules is generated.
Well this isn't what is the current nodejs packaging guidelines state and as noted by Ben elsewhere in this thread would make it prohibitive to package anything but the most trivial nodejs library.
On Fri, Dec 10, 2021 at 12:57 PM Sandro Mani manisandro@gmail.com wrote:
On 10.12.21 01:54, Demi Marie Obenour wrote:
On 12/9/21 1:05 PM, Sandro Mani wrote:
On 09.12.21 17:31, Vitaly Zaitsev via devel wrote:
On 09/12/2021 16:56, Sandro Mani wrote:
This does not appear to be accurate for nodejs packages - take i.e. node-svgo, which compliant with the guidelines bundles node_modules dir in svgo-2.8.0-nm-dev.tgz resp svgo-2.8.0-nm-prod.tgz.
You can vendor only sources. No prebuilt assets are allowed.
Which would basically mean bundling the node_modules folder?
No, it would mean bundling the source from which the stuff in node_modules is generated.
Well this isn't what is the current nodejs packaging guidelines state and as noted by Ben elsewhere in this thread would make it prohibitive to package anything but the most trivial nodejs library.
Well, not even NodeJS packaging can ignore basic "legal" requirements (such as: not shipping pre-built binaries etc.). I always assumed that to be an implicit rule, which doesn't need to be repeated on every domain-specific Packaging Guidelines page ... So, if bundled / vendored dependencies contain objectionable content, those files needs to be removed, preferably before building the vendor tarball. If that's currently not the case, then NodeJS packaging is more broken than I thought.
Fabio
On 10.12.21 22:05, Fabio Valentini wrote:
On Fri, Dec 10, 2021 at 12:57 PM Sandro Mani manisandro@gmail.com wrote:
On 10.12.21 01:54, Demi Marie Obenour wrote:
On 12/9/21 1:05 PM, Sandro Mani wrote:
On 09.12.21 17:31, Vitaly Zaitsev via devel wrote:
On 09/12/2021 16:56, Sandro Mani wrote:
This does not appear to be accurate for nodejs packages - take i.e. node-svgo, which compliant with the guidelines bundles node_modules dir in svgo-2.8.0-nm-dev.tgz resp svgo-2.8.0-nm-prod.tgz.
You can vendor only sources. No prebuilt assets are allowed.
Which would basically mean bundling the node_modules folder?
No, it would mean bundling the source from which the stuff in node_modules is generated.
Well this isn't what is the current nodejs packaging guidelines state and as noted by Ben elsewhere in this thread would make it prohibitive to package anything but the most trivial nodejs library.
Well, not even NodeJS packaging can ignore basic "legal" requirements (such as: not shipping pre-built binaries etc.). I always assumed that to be an implicit rule, which doesn't need to be repeated on every domain-specific Packaging Guidelines page ... So, if bundled / vendored dependencies contain objectionable content, those files needs to be removed, preferably before building the vendor tarball. If that's currently not the case, then NodeJS packaging is more broken than I thought.
The license review is outlined in [1] and not the issue at hand here - clearly, bundled content needs to satisfy the licensing requirements or otherwise removed. The issue here concerns bundling node_modules vs unbundling and building from source every single dependency and dependency of dependency.
Thanks Sandro
[1] https://docs.fedoraproject.org/en-US/packaging-guidelines/Node.js/#_bundled_...
On 12/10/21 6:56 AM, Sandro Mani wrote:
On 10.12.21 01:54, Demi Marie Obenour wrote:
On 12/9/21 1:05 PM, Sandro Mani wrote:
On 09.12.21 17:31, Vitaly Zaitsev via devel wrote:
On 09/12/2021 16:56, Sandro Mani wrote:
This does not appear to be accurate for nodejs packages - take i.e. node-svgo, which compliant with the guidelines bundles node_modules dir in svgo-2.8.0-nm-dev.tgz resp svgo-2.8.0-nm-prod.tgz.
You can vendor only sources. No prebuilt assets are allowed.
Which would basically mean bundling the node_modules folder?
No, it would mean bundling the source from which the stuff in node_modules is generated.
Well this isn't what is the current nodejs packaging guidelines state and as noted by Ben elsewhere in this thread would make it prohibitive to package anything but the most trivial nodejs library.
If some of the dependencies are unnecessary, the package maintainers could patch the code to not use them, and send the patches upstream. That said, this really needs to be solved at the NPM level, by having NPM packages include machine-extractable source code.
In any case, node_modules is not source code, since it is not “the preferred form of the work for making modifications to it.” (quoting LGPLv2.1 here, but I believe Fedora uses an equivalent definition). The question then becomes whether it is more like bundling a prebuilt binary, which is not acceptable, or like the bundling of the output of lex, yacc, or pandoc in autotools-generated tarballs, which I consider fine. One distinction might be whether the output files are portable and can be automatically regenerated, which is invariably true in the latter case.
On Thursday, 16 December 2021 23:59:23 CET Demi Marie Obenour wrote:
On 12/10/21 6:56 AM, Sandro Mani wrote:
On 10.12.21 01:54, Demi Marie Obenour wrote:
On 12/9/21 1:05 PM, Sandro Mani wrote:
On 09.12.21 17:31, Vitaly Zaitsev via devel wrote:
On 09/12/2021 16:56, Sandro Mani wrote:
This does not appear to be accurate for nodejs packages - take i.e. node-svgo, which compliant with the guidelines bundles node_modules dir in svgo-2.8.0-nm-dev.tgz resp svgo-2.8.0-nm-prod.tgz.
You can vendor only sources. No prebuilt assets are allowed.
Which would basically mean bundling the node_modules folder?
No, it would mean bundling the source from which the stuff in node_modules is generated.
Well this isn't what is the current nodejs packaging guidelines state and as noted by Ben elsewhere in this thread would make it prohibitive to package anything but the most trivial nodejs library.
If some of the dependencies are unnecessary, the package maintainers could patch the code to not use them, and send the patches upstream. That said, this really needs to be solved at the NPM level, by having NPM packages include machine-extractable source code.
In any case, node_modules is not source code, since it is not “the preferred form of the work for making modifications to it.” (quoting LGPLv2.1 here, but I believe Fedora uses an equivalent definition). The question then becomes whether it is more like bundling a prebuilt binary, which is not acceptable, or like the bundling of the output of lex, yacc, or pandoc in autotools-generated tarballs, which I consider fine. One distinction might be whether the output files are portable and can be automatically regenerated, which is invariably true in the latter case.
I don't see a problem if the node modules don't ship prebuilt libraries or binaries. If you look at my scripts they remove all of this.
https://src.fedoraproject.org/rpms/nodejs-bash-language-server/blob/rawhide/... prepare_vendor.sh#_55
On 12/18/21 3:04 AM, Andreas Schneider wrote:
On Thursday, 16 December 2021 23:59:23 CET Demi Marie Obenour wrote:
On 12/10/21 6:56 AM, Sandro Mani wrote:
On 10.12.21 01:54, Demi Marie Obenour wrote:
On 12/9/21 1:05 PM, Sandro Mani wrote:
On 09.12.21 17:31, Vitaly Zaitsev via devel wrote:
On 09/12/2021 16:56, Sandro Mani wrote: > This does not appear to be accurate for nodejs packages - take i.e. > node-svgo, which compliant with the guidelines bundles node_modules > dir in svgo-2.8.0-nm-dev.tgz resp svgo-2.8.0-nm-prod.tgz.
You can vendor only sources. No prebuilt assets are allowed.
Which would basically mean bundling the node_modules folder?
No, it would mean bundling the source from which the stuff in node_modules is generated.
Well this isn't what is the current nodejs packaging guidelines state and as noted by Ben elsewhere in this thread would make it prohibitive to package anything but the most trivial nodejs library.
If some of the dependencies are unnecessary, the package maintainers could patch the code to not use them, and send the patches upstream. That said, this really needs to be solved at the NPM level, by having NPM packages include machine-extractable source code.
In any case, node_modules is not source code, since it is not “the preferred form of the work for making modifications to it.” (quoting LGPLv2.1 here, but I believe Fedora uses an equivalent definition). The question then becomes whether it is more like bundling a prebuilt binary, which is not acceptable, or like the bundling of the output of lex, yacc, or pandoc in autotools-generated tarballs, which I consider fine. One distinction might be whether the output files are portable and can be automatically regenerated, which is invariably true in the latter case.
I don't see a problem if the node modules don't ship prebuilt libraries or binaries. If you look at my scripts they remove all of this.
https://src.fedoraproject.org/rpms/nodejs-bash-language-server/blob/rawhide/...
As long as everything left is in fact (human-editable) source code, this is fine. This excludes both minified JavaScript and the output of TypeScript, Babel, or other transpilers. All of those would need to be run during the package build.
I looked at the nodejs-back-language-server source package and it does not meet this requirement. The file .package-cache/v6/npm-object-assign-4.1.0-968bf1100d7956bb3ca086f006f846b3bc4008da-integrity/node_modules/object.assign/dist/browser.js in the vendor tarball is obviously transpiler or bundler output. There are also several .min.js files and source maps, none of which should be present in a Fedora source package. This was just a cursory inspection; I would expect a thorough review before a new package is uploaded.
On 12/9/21 10:12 AM, Vitaly Zaitsev via devel wrote:
On 09/12/2021 14:17, Sandro Mani wrote:
Is this in any way acceptable?
All packages must be built completely from sources.
Is this compatible with
You must remove all pre-built artifacts and minified sources before creating a vendored tarball.
Does this apply to stuff like bison or flex output that is supposed to be in autotools-generated release tarballs, or can those tarballs be used as-is?
-
Andreas Schneider -
Ben Beasley -
Demi Marie Obenour -
Fabio Valentini -
graham_alton＠hotmail.com -
Sandro Mani -
Vitaly Zaitsev