As many of you know, as packages change, so do their BuildRequires. In the current state, maintaining them requires some manual work from the maintainer.
1. So I got around the idea of a simple tool that checks file accesses during the build and using RPM queries, detects whether some package's files are not accessed at all therefore the package is not needed for the build. To my knowledge there is no such project. The project is here: https://github.com/mkoncek/unbreq
It may not be completely reliable, but it also may be good enough to catch simple mistakes.
2. At least in the case of maven build system, this tool does not help with `mvn(foo:bar)` dependencies, as maven unconditionally reads all the files present in /usr/share/maven-metadata, from which it deduces the associations between jars and artifact coordinates. I imagine other build systems employ a similar strategy.
3. In the case of maven, we have a manual tool: xmvn-builddep, which reads the build.log and constructs the actual BuildRequires from it, using knowledge about the build procedure. This could be used as an additional step of this tool, having similar tools for other languages.
Ultimately, I am interested in the possibility of having automated unused BuildRequires detection as part of rpmbuild / mockbuild.
On Wed, 2024-07-03 at 18:02 +0200, Marián Konček wrote:
As many of you know, as packages change, so do their BuildRequires. In the current state, maintaining them requires some manual work from the maintainer.
- So I got around the idea of a simple tool that checks file
accesses during the build and using RPM queries, detects whether some package's files are not accessed at all therefore the package is not needed for the build. To my knowledge there is no such project.
yes , no such project and I also think that project may be useful .
I notice with [1] and with new [2] we can check if src requires something-devel and the binary don't requires the libsomthing.so.etc
to exemplify (result in attachment) : ./dep_checker.py --max_deps 0 glib gtk+ libbonoboui libgnome libgnomeui GConf2
we got may cases of somepackge.src requires GConf2-devel but his bin doesn't requires libgconf-2.so
I can deduce that we can remove the "Builrequires: GConf2-devel" . But we have other cases that is not true , like headers packages and things that used scripts and more cases that I don't recall ATM . So I did't find any formula or linear method , but I hope that dep_checker.py can help you , I hadn't finish the development yet ...
[1] https://pagure.io/releng/blob/main/f/scripts/orphaned-packages-process [2] https://pagure.io/fork/sergiomb/releng/blob/main/f/scripts/orphaned-packages...
The project is here: https://github.com/mkoncek/unbreq
It may not be completely reliable, but it also may be good enough to catch simple mistakes.
- At least in the case of maven build system, this tool does not
help with `mvn(foo:bar)` dependencies, as maven unconditionally reads all the files present in /usr/share/maven-metadata, from which it deduces the associations between jars and artifact coordinates. I imagine other build systems employ a similar strategy.
- In the case of maven, we have a manual tool: xmvn-builddep,
which reads the build.log and constructs the actual BuildRequires from it, using knowledge about the build procedure. This could be used as an additional step of this tool, having similar tools for other languages.
Ultimately, I am interested in the possibility of having automated unused BuildRequires detection as part of rpmbuild / mockbuild.
-- Marián Konček
Dne 03. 07. 24 v 6:02 odp. Marián Konček napsal(a):
As many of you know, as packages change, so do their BuildRequires. In the current state, maintaining them requires some manual work from the maintainer.
- So I got around the idea of a simple tool that checks file accesses during the build and using RPM queries, detects
whether some package's files are not accessed at all therefore the package is not needed for the build. To my knowledge there is no such project. The project is here: https://github.com/mkoncek/unbreq
Great idea. I am not aware of anything similar.
It may not be completely reliable, but it also may be good enough to catch simple mistakes.
*nod* but it can be improved later :)
It would be good if you do not enforce changes in spec file. This is big block for any testing/prototyping. I highly recommend to implement it as Mock plugins. Here are some pointers
https://rpm-software-management.github.io/mock/#plugins
https://github.com/rpm-software-management/mock/tree/main/mock/py/mockbuild/...
You run the `resolve` at the end of %install but some BuildRequires are needed because of %check which is run after %install
- In the case of maven, we have a manual tool: xmvn-builddep, which reads the build.log and constructs the actual
BuildRequires from it, using knowledge about the build procedure. This could be used as an additional step of this tool, having similar tools for other languages.
See
https://rpm-software-management.github.io/rpm/manual/spec.html#generate_buil...
You can check https://docs.fedoraproject.org/en-US/packaging-guidelines/Python/ how Python uses this.
Ultimately, I am interested in the possibility of having automated unused BuildRequires detection as part of rpmbuild / mockbuild.
rpmbuild does not have any isolation and uses any other package installed on system beside these specified in BuildRequires.
Mock on the other hand install only minimal systems + BuildRequires. So it is more suitable for this task and check.
If you need some assistance or guidance with integration to Mock please contact me off-list and I will be happy to assist you.
On Wed, Jul 03, 2024 at 06:02:01PM +0200, Marián Konček wrote:
As many of you know, as packages change, so do their BuildRequires. In the current state, maintaining them requires some manual work from the maintainer.
- So I got around the idea of a simple tool that checks file
accesses during the build and using RPM queries, detects whether some package's files are not accessed at all therefore the package is not needed for the build. To my knowledge there is no such project. The project is here: https://github.com/mkoncek/unbreq
For completeness, this is a tool I wrote a hundred years ago that does the opposite:
https://people.redhat.com/~rjones/auto-buildrequires/
Rich.
It may not be completely reliable, but it also may be good enough to catch simple mistakes.
- At least in the case of maven build system, this tool does not
help with `mvn(foo:bar)` dependencies, as maven unconditionally reads all the files present in /usr/share/maven-metadata, from which it deduces the associations between jars and artifact coordinates. I imagine other build systems employ a similar strategy.
- In the case of maven, we have a manual tool: xmvn-builddep, which
reads the build.log and constructs the actual BuildRequires from it, using knowledge about the build procedure. This could be used as an additional step of this tool, having similar tools for other languages.
Ultimately, I am interested in the possibility of having automated unused BuildRequires detection as part of rpmbuild / mockbuild.
-- Marián Konček
-- _______________________________________________ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-leave@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org Do not reply to spam, report it: https://pagure.io/fedora-infrastructure/new_issue
On 03. 07. 24 18:02, Marián Konček wrote:
As many of you know, as packages change, so do their BuildRequires. In the current state, maintaining them requires some manual work from the maintainer.
Thanks!
(As a side note, I encourage everybody to use %genearte_buildrequires from upstream metadata/code as much as possible, it helps remove stale BuildRequires quite a lot. I can help you design a macro for this for your ecosystem (e.g. Perl, Maven)).
- So I got around the idea of a simple tool that checks file accesses during
the build and using RPM queries, detects whether some package's files are not accessed at all therefore the package is not needed for the build. To my knowledge there is no such project. The project is here: https://github.com/mkoncek/unbreq
It may not be completely reliable, but it also may be good enough to catch simple mistakes.
I read the documentation. Should this be executed after %check rather than after %install?
- At least in the case of maven build system, this tool does not help with
`mvn(foo:bar)` dependencies, as maven unconditionally reads all the files present in /usr/share/maven-metadata, from which it deduces the associations between jars and artifact coordinates. I imagine other build systems employ a similar strategy.
I imagine that for Python packages, this will be similar, as Python tools would likely read all the installed .dist-info, .egg-info metadata regardless of whether they actually need those packages.
Perhaps there could be a regex/glob based ignore-list of files that should not count?
- In the case of maven, we have a manual tool: xmvn-builddep, which reads the
build.log and constructs the actual BuildRequires from it, using knowledge about the build procedure. This could be used as an additional step of this tool, having similar tools for other languages.
Ultimately, I am interested in the possibility of having automated unused BuildRequires detection as part of rpmbuild / mockbuild.
I +1 Mirek's opinion to make this a mock plugin. That way, we can run it in bulk without modifying the specs.
On 03. 07. 24 18:02, Marián Konček wrote:
As many of you know, as packages change, so do their BuildRequires. In the current state, maintaining them requires some manual work from the maintainer.
- So I got around the idea of a simple tool that checks file accesses during
the build and using RPM queries, detects whether some package's files are not accessed at all therefore the package is not needed for the build. To my knowledge there is no such project. The project is here: https://github.com/mkoncek/unbreq
It may not be completely reliable, but it also may be good enough to catch simple mistakes.
- At least in the case of maven build system, this tool does not help with
`mvn(foo:bar)` dependencies, as maven unconditionally reads all the files present in /usr/share/maven-metadata, from which it deduces the associations between jars and artifact coordinates. I imagine other build systems employ a similar strategy.
- In the case of maven, we have a manual tool: xmvn-builddep, which reads the
build.log and constructs the actual BuildRequires from it, using knowledge about the build procedure. This could be used as an additional step of this tool, having similar tools for other languages.
Ultimately, I am interested in the possibility of having automated unused BuildRequires detection as part of rpmbuild / mockbuild.
Amongst others, I get:
warning: BuildRequires make is not needed
While I call /usr/bin/make which is owned by make.
Any idea what could cause this?
I have only overridden the `open*` family of functions, whereas `exec*` should be as well, I think that is the reason.
On 4. 7. 2024 13:56, Miro Hrončok wrote:
On 03. 07. 24 18:02, Marián Konček wrote:
As many of you know, as packages change, so do their BuildRequires. In the current state, maintaining them requires some manual work from the maintainer.
- So I got around the idea of a simple tool that checks file
accesses during the build and using RPM queries, detects whether some package's files are not accessed at all therefore the package is not needed for the build. To my knowledge there is no such project. The project is here: https://github.com/mkoncek/unbreq
It may not be completely reliable, but it also may be good enough to catch simple mistakes.
- At least in the case of maven build system, this tool does not
help with `mvn(foo:bar)` dependencies, as maven unconditionally reads all the files present in /usr/share/maven-metadata, from which it deduces the associations between jars and artifact coordinates. I imagine other build systems employ a similar strategy.
- In the case of maven, we have a manual tool: xmvn-builddep, which
reads the build.log and constructs the actual BuildRequires from it, using knowledge about the build procedure. This could be used as an additional step of this tool, having similar tools for other languages.
Ultimately, I am interested in the possibility of having automated unused BuildRequires detection as part of rpmbuild / mockbuild.
Amongst others, I get:
warning: BuildRequires make is not needed
While I call /usr/bin/make which is owned by make.
Any idea what could cause this?
On Wed, 3 Jul 2024 18:02:01 +0200, Marián Konček wrote:
To my knowledge there is no such project.
Only highly experimental ones have been used many, many years ago. For example, in Fedora land I've caught some unused BuildRequires with a script that checks whether shared libs provided by build dependencies reappear as SONAME deps in the built packages.
Reviving this thread. I have had success with this project implemented as a mock plugin [1] (available in copr: [2]). I intend to communicate with mock developers to see if it can be integrated into mock [3]. Implementation notes aside, this may be a good place to discuss the algorithm:
On input we have the *.src.rpm file(s) and a list of files that have been accessed during the build. * From the source RPM we extract the BuildRequires strings. * Then we use `dnf --installed --whatprovides $BR` to see what RPM provides each BR. This itself presents an interesting question: what does it mean if a single BR is provided by multiple installed RPMs? We assume it must be because some of them must have been required by different packages. * For each such RPM we query `dnf --assumeno remove $RPMS` to see what packages would have been removed. * For each RPM that would have been removed even transitively we query what files it contains and check whether its files have been accessed. If not, it is added to the list of RPMs that can be removed. * The next `dnf --assumeno remove $RPMS` query is run with RPMS containing all the previous RPM that we concluded we can remove as well as the one from current iteration.
This is a conservative approach that will not include false positives. However it will not warn about cases when the package has a BR on `foo`, `foo` requires `bar`, `bar` is really needed but `foo` is not. This would require actually traversing the dependency graph manually, which is possible but needs implementation.
[1] https://github.com/mkoncek/unbreq [2] https://copr.fedorainfracloud.org/coprs/mkoncek/unbreq/ [3] https://github.com/rpm-software-management/mock/issues/1562
On 5. 7. 2024 11:23, Michael Schwendt wrote:
On Wed, 3 Jul 2024 18:02:01 +0200, Marián Konček wrote:
To my knowledge there is no such project.
Only highly experimental ones have been used many, many years ago. For example, in Fedora land I've caught some unused BuildRequires with a script that checks whether shared libs provided by build dependencies reappear as SONAME deps in the built packages.
-
Marián Konček -
Michael Schwendt -
Miro Hrončok -
Miroslav Suchý -
Richard W.M. Jones -
Sérgio Basto