Script to get all deps of a package (for soname bumps etc.)

[Ankur Sinha]

Hi folks,

I'm sure this exists somewhere but I couldn't find one, so with the help
of folks on #fedora-devel, I hacked up this simple shell script to get
the all the deps of a package. It's useful when your package update
includes a soname bump, and you need to figure out what packages need to
be rebuilt etc.

https://pagure.io/fedora-get-package-dependencies/blob/main/f/get_deps.sh

It lists all the capabilities of the package, and then asks dnf to list
what packages require any of them. It should cover most cases.

If someone has a better script, please do share it, and please feel free
to improve this one too. I'm more than happy to give everyone access to
the repo and/or hand it over to a particular package maintainers related
pagure group.

Ideally, a script of this form should be noted in the package-maintainer
docs so we can all use it.

-- 
Thanks,
Regards,
Ankur Sinha "FranciscoD" (He / Him / His) |
https://fedoraproject.org/wiki/User:Ankursinha
Time zone: Europe/London

[Sérgio Basto]

On Wed, 2022-07-13 at 14:19 +0100, Ankur Sinha wrote:
>  Hi folks,
>  
>  I'm sure this exists somewhere but I couldn't find one,  
(IMHO) The script that give really all deps is
find_unblocked_orphans.py from
https://pagure.io/releng/blob/main/f/scripts/find_unblocked_orphans.py
, you can use it with option --skip-orphans 
for example :
find_unblocked_orphans.py --max_deps 30 --skip-orphans glib

It is used by Miro to generate the weekly reports for orphans packages
.

I'm still working in some improvements of the script , which aren't
finished yet (neither published)

>  so with the help
>  of folks on #fedora-devel, I hacked up this simple shell script to
>  get
>  the all the deps of a package. It's useful when your package update
>  includes a soname bump, and you need to figure out what packages need
>  to
>  be rebuilt etc.
>  
>  https://pagure.io/fedora-get-package-dependencies/blob/main/f/get_deps.sh
>  
>  It lists all the capabilities of the package, and then asks dnf to
>  list
>  what packages require any of them. It should cover most cases.
>  
>  If someone has a better script, please do share it, and please feel
>  free
>  to improve this one too. I'm more than happy to give everyone access
>  to
>  the repo and/or hand it over to a particular package maintainers
>  related
>  pagure group.
>  
>  Ideally, a script of this form should be noted in the package-
>  maintainer
>  docs so we can all use it.
>  
>  _______________________________________________
>  devel mailing list -- devel(a)lists.fedoraproject.org
>  To unsubscribe send an email to devel-leave(a)lists.fedoraproject.org
>  Fedora Code of Conduct:
>  https://docs.fedoraproject.org/en-US/project/code-of-conduct/
>  List Guidelines:
>  https://fedoraproject.org/wiki/Mailing_list_guidelines
>  List Archives:
>  https://lists.fedoraproject.org/archives/list/[e-mail]
>  Do not reply to spam on the list, report it:
>  https://pagure.io/fedora-infrastructure 
-- 
Sérgio M. B.

[Maxwell G]

On 22/07/13 02:19PM, Ankur Sinha wrote:
>  Hi folks,
>  
>  I'm sure this exists somewhere but I couldn't find one, so with the help
>  of folks on #fedora-devel, I hacked up this simple shell script to get
>  the all the deps of a package. It's useful when your package update
>  includes a soname bump, and you need to figure out what packages need to
>  be rebuilt etc.
>  
>  https://pagure.io/fedora-get-package-dependencies/blob/main/f/get_deps.sh
>  
>  It lists all the capabilities of the package, and then asks dnf to list
>  what packages require any of them. It should cover most cases.
>  
>  If someone has a better script, please do share it, and please feel free
>  to improve this one too. I'm more than happy to give everyone access to
>  the repo and/or hand it over to a particular package maintainers related
>  pagure group. 
I don't think you need to do all of this. With

```
sudo dnf repoquery --repo=rawhide -q --whatrequires dcmtk | xargs sudo dnf repoquery
--repo=rawhide -q --latest-limit 1 --source
```

AFAIK, dnf repoquery is smart enough to figure out the virtual provides
by itself. You can install fedora-repos-rawhide on a stable Fedora
release to get the rawhide repo definitions. (They're disabled by
default, so don't worry about your packages getting updated to rawhide
versions!) `--release rawhide` is not what you want. On my system, it
adds a few seconds to the script's execution, as it tries to load the
rawhide version for all of the enabled repositories. You only want to
query the rawhide repo itself.

A better version of the above would be

```
sudo dnf repoquery --repo=rawhide -q --whatrequires dcmtk | xargs sudo dnf repoquery
--repo=rawhide -q --source | pkgname | sort | uniq
```

which just includes the *names* of the affected source packages. This is
most likely what you want for doing rebuilds.

If you're trying to find which source packages *Build*Require something,
you can use

```
sudo dnf repoquery --repo=rawhide{,-source} -q --whatrequires dcmtk-devel | grep
'\.src' | pkgname | sort | uniq
```

You can add --recursive to either command if appropriate.

You can also add a .repo file for the koji buildroot repository and
query that

```
$ cat /etc/distro.repos.d/koji.repo
[koji]
name=koji
baseurl=http://kojipkgs.fedoraproject.org/repos/rawhide/latest/$basearch/
enabled=0


[koji-source]
name=koji-source
baseurl=http://kojipkgs.fedoraproject.org/repos/rawhide/latest/src/
enabled=0
```

I believe Miro uses this for the FTI bugs. It tends to be more accurate,
especially when there hasn't been a compose for a couple days. If this
is something people are interested in, I think it's worthwhile to
include this repo definition in fedora-packager.

-- 
Thanks,

Maxwell G (@gotmax23)
Pronouns: He/Him/His

[Robbie Harwood]

Maxwell G via devel <devel(a)lists.fedoraproject.org&gt; writes:

>  I believe Miro uses this for the FTI bugs. It tends to be more
> accurate,
>  especially when there hasn't been a compose for a couple days. If this
>  is something people are interested in, I think it's worthwhile to
>  include this repo definition in fedora-packager. 
An authoritative, preferably single, simple command to run to answer the
question "across all architectures, what needs my package and in what
form?" would be really helpful.  Even if there's a "correct" way to do
it today, it's not discoverable (as evidenced by these threads) nor is
it easy to remember.  Something like `dnf whatuses src:mypackage` (or of
similar simplicity) would be appreciated.

Be well,
--Robbie

[Iñaki Ucar]

On Wed, 13 Jul 2022 at 18:13, Robbie Harwood <rharwood(a)redhat.com&gt; wrote:
>  Maxwell G via devel <devel(a)lists.fedoraproject.org&gt; writes:
>
>  > I believe Miro uses this for the FTI bugs. It tends to be more accurate,
>  > especially when there hasn't been a compose for a couple days. If this
>  > is something people are interested in, I think it's worthwhile to
>  > include this repo definition in fedora-packager.
>
>  An authoritative, preferably single, simple command to run to answer the
>  question "across all architectures, what needs my package and in what
>  form?" would be really helpful.  Even if there's a "correct" way to
> do
>  it today, it's not discoverable (as evidenced by these threads) nor is
>  it easy to remember.  Something like `dnf whatuses src:mypackage` (or of
>  similar simplicity) would be appreciated. 
If this functionality requires defining and enabling additional repos,
maybe dnf is not the proper place? But then maybe fedpkg is. Anyway, I
agree that a simple command would be very much appreciated.

-- 
Iñaki Úcar

[Stephen Smoogen]

On Wed, 13 Jul 2022 at 12:25, Iñaki Ucar <iucar(a)fedoraproject.org&gt; wrote:

>  On Wed, 13 Jul 2022 at 18:13, Robbie Harwood
> <rharwood(a)redhat.com&gt; wrote:
>  >
>  > Maxwell G via devel <devel(a)lists.fedoraproject.org&gt; writes:
>  >
>  > > I believe Miro uses this for the FTI bugs. It tends to be more
>  accurate,
>  > > especially when there hasn't been a compose for a couple days. If this
>  > > is something people are interested in, I think it's worthwhile to
>  > > include this repo definition in fedora-packager.
>  >
>  > An authoritative, preferably single, simple command to run to answer the
>  > question "across all architectures, what needs my package and in what
>  > form?" would be really helpful.  Even if there's a "correct" way
> to do
>  > it today, it's not discoverable (as evidenced by these threads) nor is
>  > it easy to remember.  Something like `dnf whatuses src:mypackage` (or of
>  > similar simplicity) would be appreciated.
>
>  If this functionality requires defining and enabling additional repos,
>  maybe dnf is not the proper place? But then maybe fedpkg is. Anyway, I
>  agree that a simple command would be very much appreciated.
>
>  
fedpkg would only call dnf to try and get the same information. It doesn't
have any better window on things


-- 
Stephen Smoogen, Red Hat Automotive
Let us be kind to one another, for most of us are fighting a hard battle.
-- Ian MacClaren

[Iñaki Ucar]

On Wed, 13 Jul 2022 at 18:43, Stephen Smoogen <ssmoogen(a)redhat.com&gt; wrote:
>  On Wed, 13 Jul 2022 at 12:25, Iñaki Ucar <iucar(a)fedoraproject.org&gt; wrote:
> >
> > On Wed, 13 Jul 2022 at 18:13, Robbie Harwood <rharwood(a)redhat.com&gt; wrote:
> > >
> > > Maxwell G via devel <devel(a)lists.fedoraproject.org&gt; writes:
> > >
> > > > I believe Miro uses this for the FTI bugs. It tends to be more accurate,
> > > > especially when there hasn't been a compose for a couple days. If this
> > > > is something people are interested in, I think it's worthwhile to
> > > > include this repo definition in fedora-packager.
> > >
> > > An authoritative, preferably single, simple command to run to answer the
> > > question "across all architectures, what needs my package and in what
> > > form?" would be really helpful.  Even if there's a "correct"
> way to do
> > > it today, it's not discoverable (as evidenced by these threads) nor is
> > > it easy to remember.  Something like `dnf whatuses src:mypackage` (or of
> > > similar simplicity) would be appreciated.
> >
> > If this functionality requires defining and enabling additional repos,
> > maybe dnf is not the proper place? But then maybe fedpkg is. Anyway, I
> > agree that a simple command would be very much appreciated.
> >
>
>  fedpkg would only call dnf to try and get the same information. It doesn't have any
> better window on things 
It could e.g. package the necessary "development" repos and activate
them when calling dnf.

-- 
Iñaki Úcar

[Ankur Sinha]

Hello,

Thanks for that---some really useful bits in there that I wasn't aware
of.

On Wed, Jul 13, 2022 10:09:10 -0500, Maxwell G wrote:
>  I don't think you need to do all of this. With
>  
>  ```
>  sudo dnf repoquery --repo=rawhide -q --whatrequires dcmtk | xargs sudo dnf repoquery
> --repo=rawhide -q --latest-limit 1 --source
>  ```
>  
>  AFAIK, dnf repoquery is smart enough to figure out the virtual provides
>  by itself.  
This is what we (I) aren't sure of, and that's why I first obtain the
capabilities manually and then query for them. If someone can confirm
that this is indeed the case, that would certainly simplify things.


-- 
Thanks,
Regards,
Ankur Sinha "FranciscoD" (He / Him / His) |
https://fedoraproject.org/wiki/User:Ankursinha
Time zone: Europe/London

[Maxwell G]

On 22/07/13 05:35PM, Ankur Sinha wrote:
>  > This is what we (I) aren't sure of, and that's why I
> first obtain the
>  capabilities manually and then query for them. If someone can confirm
>  that this is indeed the case, that would certainly simplify things. 
Here is confirmation:

```
sudo dnf repoquery --repo=rawhide{,-source} -q --whatrequires python3-setuptools | grep
'\.src' | grep yt-dlp
yt-dlp-0:2022.06.29-2.fc37.src

$ sudo dnf repoquery --repo=rawhide-source -q --requires yt-dlp | grep setuptools
python3dist(setuptools) >= 40.8
```

yt-dlp depends on one of `python3-setuptool`'s virtual provides, and it is still
found when using the command I gave to find which packages BuildRequire
a certain other package. This also works when querying runtime
dependencies:

```
$ sudo dnf repoquery -q --repo=rawhide --requires reuse | grep setuptools
python3.11dist(setuptools)
$ sudo dnf repoquery -q --repo=rawhide --whatrequires python3-setuptools | grep reuse
reuse-0:1.0.0-2.fc37.noarch
```

-- 
Thanks,

Maxwell G (@gotmax23)
Pronouns: He/Him/His

[Ankur Sinha]

On Wed, Jul 13, 2022 12:54:37 -0500, Maxwell G wrote:
>  On 22/07/13 05:35PM, Ankur Sinha wrote:
>  > > This is what we (I) aren't sure of, and that's why I first obtain the
>  > capabilities manually and then query for them. If someone can confirm
>  > that this is indeed the case, that would certainly simplify things.
>  
>  Here is confirmation:
>  
>  ```
>  sudo dnf repoquery --repo=rawhide{,-source} -q --whatrequires python3-setuptools | grep
> '\.src' | grep yt-dlp
>  yt-dlp-0:2022.06.29-2.fc37.src
>  
>  $ sudo dnf repoquery --repo=rawhide-source -q --requires yt-dlp | grep setuptools
>  python3dist(setuptools) >= 40.8
>  ```
>  
>  yt-dlp depends on one of `python3-setuptool`'s virtual provides, and it is still
>  found when using the command I gave to find which packages BuildRequire
>  a certain other package. This also works when querying runtime
>  dependencies:
>  
>  ```
>  $ sudo dnf repoquery -q --repo=rawhide --requires reuse | grep setuptools
>  python3.11dist(setuptools)
>  $ sudo dnf repoquery -q --repo=rawhide --whatrequires python3-setuptools | grep reuse
>  reuse-0:1.0.0-2.fc37.noarch
>  ``` 
Thanks very much!

That really simplifies it to a single command.


-- 
Thanks,
Regards,
Ankur Sinha (He / Him / His) | https://ankursinha.in
Time zone: Europe/London

[Richard Shaw]

On Wed, Jul 13, 2022 at 8:19 AM Ankur Sinha <sanjay.ankur(a)gmail.com&gt; wrote:

>  Hi folks,
>
>  I'm sure this exists somewhere but I couldn't find one, so with the help
>  of folks on #fedora-devel, I hacked up this simple shell script to get
>  the all the deps of a package. It's useful when your package update
>  includes a soname bump, and you need to figure out what packages need to
>  be rebuilt etc.
>  
I use this script which I'm pretty sure I pieced together from some of
Sergio's logic some time ago. I like that it saves them to a file so I can
iterate over them with other scripts.

$ cat ~/.local/bin/needs_rebuilding
#!/bin/bash

if [ $# -eq 0 ]
  then
    echo "No arguments supplied"
    echo "Usage: $0 <pkgname> [pkgname] ..."
    exit 0
fi

provides=$(mktemp -t provides-XXXXXX)
deps=$(mktemp -t rawdeps-XXXXXX)

# Only break on newline not space or tab in for loops
IFS=$'\n'

for pkg in "$@"; do
echo "Checking for provides in $pkg."
dnf --quiet repoquery --repoid=rawhide --provides "$pkg" >> $provides
done

sed -i "/^bundled/d" $provides
sort -u -o $provides $provides

echo "Found $(wc -l $provides | awk '{print $1}') provides to be
evaluated."

for dep in $(cat $provides); do
echo "Checking requirements for $dep"
dnf --quiet repoquery --repoid=rawhide --qf "%{source_name}" --whatrequires
"$dep" >> $deps
done

sort -u -o ./needs_rebuilding $deps

# Remove the packages being checked
for pkgname in "$@"; do
sed -i "/$pkgname/d" needs_rebuilding
done

echo "Rebuild dependencies saved to ./needs_rebuilding"


Thanks,
Richard

[Ankur Sinha]

On Mon, Jul 18, 2022 06:52:26 -0500, Richard Shaw wrote:
>  <snip> 
Hi Richard,

Thanks very much for sharing your script.

>  I use this script which I'm pretty sure I pieced together from
> some of Sergio's
>  logic some time ago. I like that it saves them to a file so I can iterate over
>  them with other scripts.
>  
>  $ cat ~/.local/bin/needs_rebuilding
>  #!/bin/bash
>  
>  if [ $# -eq 0 ]
>    then
>      echo "No arguments supplied"
>      echo "Usage: $0 <pkgname> [pkgname] ..."
>      exit 0
>  fi
>  
>  provides=$(mktemp -t provides-XXXXXX)
>  deps=$(mktemp -t rawdeps-XXXXXX)
>  
>  # Only break on newline not space or tab in for loops
>  IFS=$'\n'
>  
>  for pkg in "$@"; do
>  echo "Checking for provides in $pkg."
>  dnf --quiet repoquery --repoid=rawhide --provides "$pkg" >> $provides
>  done
>  
>  sed -i "/^bundled/d" $provides
>  sort -u -o $provides $provides
>  
>  echo "Found $(wc -l $provides | awk '{print $1}') provides to be
> evaluated." 
^
I think this bit isn't required because as Maxwell confirmed, repoquery
is smart enough to take all provides into account. However, if one does
want to see what capability causes the dependency, I think we must go
over each capability individually.

So, after the discussion here, I've shamelessly picked bits from all
your responses and scripts and have come up with these two:

https://pagure.io/fedora-get-package-dependencies/blob/main/f/get_deps.sh
gives just a list of all deps, without going over each capability
individually

https://pagure.io/fedora-get-package-dependencies/blob/main/f/get_deps_ve...
gives a more verbose report listing what capability of the package other
packages are dependent on.

If folks can please test these out and make more improvements, I can
open a PR in the package-maintainer docs to list these there to give
them more visibility.

-- 
Thanks,
Regards,
Ankur Sinha "FranciscoD" (He / Him / His) |
https://fedoraproject.org/wiki/User:Ankursinha
Time zone: Europe/London