Automating R package dependencies

[Elliott Sales de Andrade]

Hi R-interested packagers and others,

Recently, I've been looking at how RPM can automatically determine
Provides and Requires [1]. I have since implemented this for R using
an R script [2] and some file attributes [3]. Following other
languages' Provides, I have namespaced them as R(packageName). It then
adds corresponding Requires, Suggests, and Enhances.

Additionally, R package versions commonly contain dashes. In order to
work in RPM, these are replaced with dots. For the automated
Provides/Requires, I have used the *real* versions instead.

So now the question is how to apply this. I expect there are social
concerns, i.e., discussing with the R maintainer, making a
Self-contained Change, etc. But for this email, I am mostly concerned
with the technical aspects:

1. Is R-devel the right place to put the script and RPM attribute file
(all R packages would normally depend on this)?
2. Does this namespacing make sense?
3. Are dashes in *namespaced* versions going to be a problem?
4. Python had a flag to enable the automatic generator; do we need
this for R, and how was it implemented?
5. I expect this would need a rebuild of all packages to get the
dependencies right (because the regular rebuild is unordered); would
this need a side tag? Or would leaving it for the normal mass rebuild
just be fine?
6. R only has two levels of dependencies (hard-require or suggested,
but not installed by default). Thus both build- and runtime-optional
packages are in Suggests; do we care about the extra Suggests?

[1] https://rpm.org/user_doc/dependency_generators.html
[2] https://src.fedoraproject.org/fork/qulogic/rpms/R/blob/autodeps/f/R-deps.R
[3] https://src.fedoraproject.org/fork/qulogic/rpms/R/blob/autodeps/f/R.attr

--
Elliott

[Iñaki Ucar]

On Sun, 16 Jun 2019 at 05:07, Elliott Sales de Andrade
<quantum.analyst(a)gmail.com&gt; wrote:
>  Hi R-interested packagers and others,
>
>  Recently, I've been looking at how RPM can automatically determine
>  Provides and Requires [1]. I have since implemented this for R using
>  an R script [2] and some file attributes [3]. Following other
>  languages' Provides, I have namespaced them as R(packageName). It then
>  adds corresponding Requires, Suggests, and Enhances.
>
>  Additionally, R package versions commonly contain dashes. In order to
>  work in RPM, these are replaced with dots. For the automated
>  Provides/Requires, I have used the *real* versions instead.
>
>  So now the question is how to apply this. I expect there are social
>  concerns, i.e., discussing with the R maintainer, making a
>  Self-contained Change, etc. But for this email, I am mostly concerned
>  with the technical aspects:
>
>  1. Is R-devel the right place to put the script and RPM attribute file
>  (all R packages would normally depend on this)? 
I believe R-SIG-Fedora is the right place.

>  2. Does this namespacing make sense?
>  3. Are dashes in *namespaced* versions going to be a problem?
>  4. Python had a flag to enable the automatic generator; do we need
>  this for R, and how was it implemented?
>  5. I expect this would need a rebuild of all packages to get the
>  dependencies right (because the regular rebuild is unordered); would
>  this need a side tag? Or would leaving it for the normal mass rebuild
>  just be fine?
>  6. R only has two levels of dependencies (hard-require or suggested,
>  but not installed by default). Thus both build- and runtime-optional
>  packages are in Suggests; do we care about the extra Suggests? 
Sometimes these extra Suggests are not satisfiable, but they can be
removed after disabling vignette rebuilding. In other cases,
Recommends are more appropriate for suggested packages.

Iñaki

> [1] https://rpm.org/user_doc/dependency_generators.html
> [2] https://src.fedoraproject.org/fork/qulogic/rpms/R/blob/autodeps/f/R-deps.R
> [3] https://src.fedoraproject.org/fork/qulogic/rpms/R/blob/autodeps/f/R.attr

[Elliott Sales de Andrade]

On Sun, Jun 16, 2019, 10:00 AM Iñaki Ucar, <iucar(a)fedoraproject.org&gt; wrote:

>  On Sun, 16 Jun 2019 at 05:07, Elliott Sales de Andrade
>  <quantum.analyst(a)gmail.com&gt; wrote:
>  >
>  > Hi R-interested packagers and others,
>  >
>  > Recently, I've been looking at how RPM can automatically determine
>  > Provides and Requires [1]. I have since implemented this for R using
>  > an R script [2] and some file attributes [3]. Following other
>  > languages' Provides, I have namespaced them as R(packageName). It then
>  > adds corresponding Requires, Suggests, and Enhances.
>  >
>  > Additionally, R package versions commonly contain dashes. In order to
>  > work in RPM, these are replaced with dots. For the automated
>  > Provides/Requires, I have used the *real* versions instead.
>  >
>  > So now the question is how to apply this. I expect there are social
>  > concerns, i.e., discussing with the R maintainer, making a
>  > Self-contained Change, etc. But for this email, I am mostly concerned
>  > with the technical aspects:
>  >
>  > 1. Is R-devel the right place to put the script and RPM attribute file
>  > (all R packages would normally depend on this)?
>
>  I believe R-SIG-Fedora is the right place.
>  
I meant the package, not the mailing list... But I guess I should cc them
as well when this is more ready.


>  > 2. Does this namespacing make sense?
>  > 3. Are dashes in *namespaced* versions going to be a problem?
>  > 4. Python had a flag to enable the automatic generator; do we need
>  > this for R, and how was it implemented?
>  > 5. I expect this would need a rebuild of all packages to get the
>  > dependencies right (because the regular rebuild is unordered); would
>  > this need a side tag? Or would leaving it for the normal mass rebuild
>  > just be fine?
>  > 6. R only has two levels of dependencies (hard-require or suggested,
>  > but not installed by default). Thus both build- and runtime-optional
>  > packages are in Suggests; do we care about the extra Suggests?
>
>  Sometimes these extra Suggests are not satisfiable, but they can be
>  removed after disabling vignette rebuilding. In other cases,
>  Recommends are more appropriate for suggested packages. 

Yes, but there's no way to know that automatically. Suggests are weak
hints, so the fact that they're not satisfiable shouldn'tbe a problem.
Packagers can always add explicit Recommends if they think it's useful.

Iñaki
>  > [1] https://rpm.org/user_doc/dependency_generators.html
>  > [2]
>  https://src.fedoraproject.org/fork/qulogic/rpms/R/blob/autodeps/f/R-deps.R
>  > [3]
>  https://src.fedoraproject.org/fork/qulogic/rpms/R/blob/autodeps/f/R.attr
>  _______________________________________________
>  devel mailing list -- devel(a)lists.fedoraproject.org
>  To unsubscribe send an email to devel-leave(a)lists.fedoraproject.org
>  Fedora Code of Conduct:
>  https://docs.fedoraproject.org/en-US/project/code-of-conduct/
>  List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
>  List Archives:
>  https://lists.fedoraproject.org/archives/list/[e-mail]
>

[Iñaki Ucar]

On Sun, 16 Jun 2019 at 22:52, Elliott Sales de Andrade
<quantum.analyst(a)gmail.com&gt; wrote:
> > > 6. R only has two levels of dependencies (hard-require or suggested,
> > > but not installed by default). Thus both build- and runtime-optional
> > > packages are in Suggests; do we care about the extra Suggests?
> >
> > Sometimes these extra Suggests are not satisfiable, but they can be
> > removed after disabling vignette rebuilding. In other cases,
> > Recommends are more appropriate for suggested packages.
>
>  Yes, but there's no way to know that automatically. Suggests are weak hints, so the
> fact that they're not satisfiable shouldn'tbe a problem. Packagers can always add
> explicit Recommends if they think it's useful. 
Not being satisfiable shouldn't be a problem, but it goes against the
packaging guidelines:

"All package dependencies (build-time or runtime, regular, weak or
otherwise) MUST ALWAYS be satisfiable within the official Fedora
repositories."

Iñaki

[José Abílio Matos]

On Sunday, 16 June 2019 03.09.41 WEST Elliott Sales de Andrade wrote:
>  Hi R-interested packagers and others,
>  
>  So now the question is how to apply this. I expect there are social
>  concerns, i.e., discussing with the R maintainer, making a
>  Self-contained Change, etc. But for this email, I am mostly concerned
>  with the technical aspects:
>  
>  1. Is R-devel the right place to put the script and RPM attribute file
>  (all R packages would normally depend on this)? 
Either that or create a new package R-rpm-macros and have R-devel depending on 
it (just like it happens for python).

>  2. Does this namespacing make sense? 
What are your doubts here?

1) use a versioned dependency like R3.6dist(packageName);
or
2) refer to the repository R_cran(packageName)?

>  3. Are dashes in *namespaced* versions going to be a problem? 
I do not think so. I remember in python one case where this happens:
# rpm -q --provides python3-dateutil
python3-dateutil = 1:2.8.0-1.fc30
python3.7dist(python-dateutil) = 2.8.0
python3dist(python-dateutil) = 2.8.0

>  4. Python had a flag to enable the automatic generator; do we need
>  this for R, and how was it implemented?
>  5. I expect this would need a rebuild of all packages to get the
>  dependencies right (because the regular rebuild is unordered); would
>  this need a side tag? Or would leaving it for the normal mass rebuild
>  just be fine? 
I tend to the last option if we ensure that all R packages are rebuilt.

>  6. R only has two levels of dependencies (hard-require or suggested,
>  but not installed by default). Thus both build- and runtime-optional
>  packages are in Suggests; do we care about the extra Suggests? 
Does dnf cares about Suggests?

According to dnf.conf the option install_weak_deps:
"When this option is set to True and a new package is about to be installed, 
all packages linked by weak dependency relation (Recommends  or  Supplements  
flags) with this package will pulled into the transaction.  Default is True."

It does not refers Suggests...

-- 
José Abílio

[Elliott Sales de Andrade]

FYI, I have put together a Change proposal for this work, which you
may have seen announced [1].

On Mon, 17 Jun 2019 at 09:35, José Abílio Matos <jamatos(a)fc.up.pt&gt; wrote:
>  On Sunday, 16 June 2019 03.09.41 WEST Elliott Sales de Andrade wrote:
>  > Hi R-interested packagers and others,
>  >
>  > So now the question is how to apply this. I expect there are social
>  > concerns, i.e., discussing with the R maintainer, making a
>  > Self-contained Change, etc. But for this email, I am mostly concerned
>  > with the technical aspects:
>  >
>  > 1. Is R-devel the right place to put the script and RPM attribute file
>  > (all R packages would normally depend on this)?
>
>  Either that or create a new package R-rpm-macros and have R-devel depending on
>  it (just like it happens for python).
>  
Coupled with Florian's suggestion to place the files in the RPM
ecosystem, I plan to make an R-rpm-macros package for this.

>  > 2. Does this namespacing make sense?
>
>  What are your doubts here?
>
>  1) use a versioned dependency like R3.6dist(packageName);
>  or
>  2) refer to the repository R_cran(packageName)?
>  
I'm just looking for any issues that may arise. I don't believe these
latter two would work though. For 1, encoding the minor version seems
unnecessary. While 3.4->3.5 required rebuilding compiled libraries,
3.5->3.6 did not. And non-compiled libraries did not need a rebuild at
all. So adding the minor version would cause extraneous rebuilds when
new R versions come out. For 2, I'm not sure indicating CRAN source
adds any clarity.

>  > 3. Are dashes in *namespaced* versions going to be a problem?
>
>  I do not think so. I remember in python one case where this happens:
>  # rpm -q --provides python3-dateutil
>  python3-dateutil = 1:2.8.0-1.fc30
>  python3.7dist(python-dateutil) = 2.8.0
>  python3dist(python-dateutil) = 2.8.0
>  
I was asking about versions, not names.

>  > 4. Python had a flag to enable the automatic generator; do we
> need
>  > this for R, and how was it implemented?
>  > 5. I expect this would need a rebuild of all packages to get the
>  > dependencies right (because the regular rebuild is unordered); would
>  > this need a side tag? Or would leaving it for the normal mass rebuild
>  > just be fine?
>
>  I tend to the last option if we ensure that all R packages are rebuilt.
>  
I've discussed with releng and have a simple plan to build without side tag [2].

>  > 6. R only has two levels of dependencies (hard-require or
> suggested,
>  > but not installed by default). Thus both build- and runtime-optional
>  > packages are in Suggests; do we care about the extra Suggests?
>
>  Does dnf cares about Suggests?
>
>  According to dnf.conf the option install_weak_deps:
>  "When this option is set to True and a new package is about to be installed,
>  all packages linked by weak dependency relation (Recommends  or  Supplements
>  flags) with this package will pulled into the transaction.  Default is True."
>
>  It does not refers Suggests...
>  
Not directly at this time. It is simply an unused hint. It is possible
for dnf to print them out for the user to further decide, but nothing
is implemented right now.

>  --
>  José Abílio
>  
[1] https://fedoraproject.org/wiki/Changes/Automatic_R_runtime_dependencies
[2] https://pagure.io/releng/issue/8501

-- 
Elliott

[Florian Festi]

It would be really great to have a proper upstream repository for these
scripts so they don't just live inside of distgit. They could have their
own repository in https://github.com/rpm-software-management or could be
placed in https://github.com/rpm-software-management/rpm-extras. What
ever you like more.

Please contract me off list for setting up the repository/permissions.

Florian


-- 

Red Hat GmbH, http://www.de.redhat.com/, Registered seat: Grasbrunn,
Commercial register: Amtsgericht Muenchen, HRB 153243,
Managing Directors: Paul Argiry, Charles Cachera, Michael Cunningham,
Michael O'Neill