F39 proposal: Changes of defaults in createrepo_c-1.0.0 (System-Wide Change proposal)

[Ben Cotton]

https://fedoraproject.org/wiki/Changes/createrepo_c_1.0.0

This document represents a proposed Change. As part of the Changes
process, proposals are publicly announced in order to receive
community feedback. This proposal will only be implemented if approved
by the Fedora Engineering Steering Committee.


== Summary ==
Update createrepo_c to 1.0.0, new release will include change of
default compression to zstd, no longer generating metadata in sqlite
database format by default and simplified comps xml type in repodata.

== Owner ==
* Name: [[User:Amatej | Aleš Matěj]]
* Email: amatej(a)redhat.com



== Detailed Description ==
- Switch default compression from gz to zstd. DNF is able to
decompress zstd via libsolv since Fedora 30 and RHEL 8.4. It will
still be possible to create gz compressed repodata by manually
specifying gz compression.
Further information:
https://github.com/rpm-software-management/createrepo_c/issues/82,
https://bugzilla.redhat.com/show_bug.cgi?id=1914876

- Stop generating metadata in sqlite database format by default. It
will still be possible to create them using a --database switch or via
sqliterepo_c tool. (It looks like the only user left is mdapi:
https://github.com/fedora-infra/mdapi/issues/97).
Further information:
https://github.com/rpm-software-management/createrepo_c/issues/338 and
https://pagure.io/releng/issue/10745

- When adding groups.xml to repodata createrepo_c currently adds two
variants to repomd.xml. The specified file as is, uncompressed, with
the type "group" and also a compressed variant with type "group_XX",
where XX is compression suffix. This is atypical and unexpected. We
propose to include just one variant of groups.xml using specified
compression and repomd.xml type "group". This is not compatible with
yum in RHEL 7. If required users will still be able to create
repositories with the old layout using modifyrepo_c.
Further information: https://bugzilla.redhat.com/show_bug.cgi?id=2056318

- To reflect API and overall stability increase major version above 0.

== Feedback ==


== Benefit to Fedora ==
* Using zstd compression provides smaller metadata that are faster to
decompress.
* Generating sqlite metadata is expensive, its slowing createrepo_c
runs and it takes up space on mirrors. (For F37 updates repo:
primary_db: 12.8 mb, filelists_db: 22.0 mb, other_db: 4.2 mb.)
* More consistent metadata, groups use the same logic as other
metadata types. Fixes:
https://bugzilla.redhat.com/show_bug.cgi?id=1904360

== Scope ==
* Proposal owners: Implement specified changes, mostly changes of default.
* Other developers: There are no requirements on other developers.

* Release engineering:
* Policies and guidelines: N/A (not needed for this Change)
* Trademark approval: N/A (not needed for this Change)
* Alignment with Objectives:


== Upgrade/compatibility impact ==
Repodata created with createrepo_c's new defaults will not be
compatible with fedora < 30 and RHEL < 8.4 (including RHEL 7, 6).
This is mainly due to zstd compression but old YUM also cannot handle
the changed mdtype of groups (it is compatible only with uncompressed
- "group" and compressed - "group_XX" not compressed -
"group").

Regarding the sqlite databases if some other users would require them
they can always download the xml metadata and use sqliterepo_c to
generate them locally.

== How To Test ==
Repositories generated with updated createrepo_c have to work with
dnf. Specifically we should ensure that groups are still loaded.

== User Experience ==
The only noticeable effect should be smaller metadata thanks to zstd
compression resulting in faster downloads (when zchunk is not used).
The (de)compression speed up is not very noticeable.

For example for F37 fedora repo:
* primary.xml.gz: 18M -> primary.xml.zst: 14M
* filelists.xml.gz: 52M -> filelists.xml.zst: 41M
* other.xml.gz: 6.9M -> other.xml.zst: 5.4M

== Dependencies ==
Updating createrepo_c does not require any other change requests to
complete first.

Metadata generated by the following tools could be affected by the
described changes if they use createrepo_c's default configuration. It
is hard to tell what effect will this have on the tools. If they
create metadata for dnf it will work fine. In case there are any
problems it is possible to manually setup the old behavior (as
described above) or revert the changes.

 bodhi-server
 copr-backend
 libguestfs
 modulemd-tools
 prunerepo
 pungi
 retrace-server
 swid-tools
 anaconda-install-env-deps
 cobbler
 ddiskit
 koji-builder
 mock
 module-build
 perl-Fedora-Rebuild
 python3-dlrn
 python3-dnf-plugin-local
 python3-dnf-plugin-modulesync
 python3-rpmfluff
 retrace-server

== Contingency Plan ==
* Contingency mechanism: If we cannot finish the changes in time there
would be no action necessary. We can even ship only part of the
changes as they don't depend on each other. In case there are
additional unexpected issues with the changes we can revert them in
createrepo_c.
* Contingency deadline: 2023-08-01
* Blocks release? No

== Documentation ==
Createrepo_c documentation will be updated accordingly.

== Release Notes ==



-- 
Ben Cotton
He / Him / His
Fedora Program Manager
Red Hat
TZ=America/Indiana/Indianapolis

[Zbigniew Jędrzejewski-Szmek]

On Thu, Mar 23, 2023 at 02:52:50PM -0400, Ben Cotton wrote:
>  https://fedoraproject.org/wiki/Changes/createrepo_c_1.0.0 
>  == Summary ==
>  Update createrepo_c to 1.0.0, new release will include change of
>  default compression to zstd, no longer generating metadata in sqlite
>  database format by default and simplified comps xml type in repodata. 
LGTM. Thank you for taking the time to figure out all the dependencies.

While at it, would it be possible to narrow down the check for which
file paths are included in primary filelists to match the packaging docs?

Currently [1] the pattern is implemented as '(^/etc|^/usr/lib/sendmail$|bin/)'
(note that "bin/" can apply anywhere in the string).
The comment says that this is for yum compat [2].
Our guidelines only mention files under /usr/bin, /usr/sbin, /etc [3].

I think that the number of files affected is small, but it is a
potential source of confusion and a hard-to-explain discrepancy between
what the guidelines say and actual dnf behaviour. Yum is long gone
and we clearly intended to clean this up many years ago but never
got round to it.

[1] https://github.com/rpm-software-management/createrepo_c/blob/master/src/m...
[2] https://github.com/rpm-software-management/createrepo_c/blob/master/src/m...
[3]
https://docs.fedoraproject.org/en-US/packaging-guidelines/#_file_and_dire...

Zbyszek

[Neal Gompa]

On Fri, Mar 24, 2023 at 3:42 AM Zbigniew Jędrzejewski-Szmek
<zbyszek(a)in.waw.pl&gt; wrote:
>  On Thu, Mar 23, 2023 at 02:52:50PM -0400, Ben Cotton wrote:
>  > https://fedoraproject.org/wiki/Changes/createrepo_c_1.0.0
>
>  > == Summary ==
>  > Update createrepo_c to 1.0.0, new release will include change of
>  > default compression to zstd, no longer generating metadata in sqlite
>  > database format by default and simplified comps xml type in repodata.
>
>  LGTM. Thank you for taking the time to figure out all the dependencies.
>
>  While at it, would it be possible to narrow down the check for which
>  file paths are included in primary filelists to match the packaging docs?
>
>  Currently [1] the pattern is implemented as '(^/etc|^/usr/lib/sendmail$|bin/)'
>  (note that "bin/" can apply anywhere in the string).
>  The comment says that this is for yum compat [2].
>  Our guidelines only mention files under /usr/bin, /usr/sbin, /etc [3].
>
>  I think that the number of files affected is small, but it is a
>  potential source of confusion and a hard-to-explain discrepancy between
>  what the guidelines say and actual dnf behaviour. Yum is long gone
>  and we clearly intended to clean this up many years ago but never
>  got round to it.
>
>  [1] https://github.com/rpm-software-management/createrepo_c/blob/master/src/m...
>  [2] https://github.com/rpm-software-management/createrepo_c/blob/master/src/m...
>  [3]
> https://docs.fedoraproject.org/en-US/packaging-guidelines/#_file_and_dire...
>  
I'd rather it be bin/|lib*/|^/etc so that binaries and libraries of
all primary paths (the majority of file deps) are covered in primary.

We should not assume that everything is in /usr because createrepo_c
is used for non-Linux repos that use different prefixes (hence why
"bin/").



-- 
真実はいつも一つ！/ Always, there's only one truth!

[Zbigniew Jędrzejewski-Szmek]

On Sat, Mar 25, 2023 at 06:29:14AM -0400, Neal Gompa wrote:
>  On Fri, Mar 24, 2023 at 3:42 AM Zbigniew Jędrzejewski-Szmek
>  <zbyszek(a)in.waw.pl&gt; wrote:
>  >
>  > On Thu, Mar 23, 2023 at 02:52:50PM -0400, Ben Cotton wrote:
>  > > https://fedoraproject.org/wiki/Changes/createrepo_c_1.0.0
>  >
>  > > == Summary ==
>  > > Update createrepo_c to 1.0.0, new release will include change of
>  > > default compression to zstd, no longer generating metadata in sqlite
>  > > database format by default and simplified comps xml type in repodata.
>  >
>  > LGTM. Thank you for taking the time to figure out all the dependencies.
>  >
>  > While at it, would it be possible to narrow down the check for which
>  > file paths are included in primary filelists to match the packaging docs?
>  >
>  > Currently [1] the pattern is implemented as
> '(^/etc|^/usr/lib/sendmail$|bin/)'
>  > (note that "bin/" can apply anywhere in the string).
>  > The comment says that this is for yum compat [2].
>  > Our guidelines only mention files under /usr/bin, /usr/sbin, /etc [3].
>  >
>  > I think that the number of files affected is small, but it is a
>  > potential source of confusion and a hard-to-explain discrepancy between
>  > what the guidelines say and actual dnf behaviour. Yum is long gone
>  > and we clearly intended to clean this up many years ago but never
>  > got round to it.
>  >
>  > [1]
> https://github.com/rpm-software-management/createrepo_c/blob/master/src/m...
>  > [2]
> https://github.com/rpm-software-management/createrepo_c/blob/master/src/m...
>  > [3]
> https://docs.fedoraproject.org/en-US/packaging-guidelines/#_file_and_dire...
>  >
>  
>  I'd rather it be bin/|lib*/|^/etc so that binaries and libraries of
>  all primary paths (the majority of file deps) are covered in primary. 
Sorry, but that'd be a (big) step backwards. `lib*/` would hugely expand the
list of files in primary, while the goal is to make it smaller. Also,
for libraries we already have a better mechanism: provides based on SONAME
and symbol versions.

>  We should not assume that everything is in /usr because createrepo_c
>  is used for non-Linux repos that use different prefixes (hence why
>  "bin/"). 
We should probably make it ^(/etc/|/bin/|/sbin/|/usr/bin/|/usr/sbin/). Split-usr
distros are a blast from the past, but adding the extra paths wouldn't change
anything for us, since we don't use them.

Zbyszek

[Kalev Lember]

On Sun, Mar 26, 2023 at 5:06 PM Zbigniew Jędrzejewski-Szmek <
zbyszek(a)in.waw.pl&gt; wrote:

>  We should probably make it
> ^(/etc/|/bin/|/sbin/|/usr/bin/|/usr/sbin/).
>  Split-usr
>  distros are a blast from the past, but adding the extra paths wouldn't
>  change
>  anything for us, since we don't use them.
>  
Please add /app/bin/ and /app/sbin/ as well to that list for flatpak rpm
builds.

-- 
Kalev

[Neal Gompa]

On Sun, Mar 26, 2023 at 12:27 PM Kalev Lember <kalevlember(a)gmail.com&gt; wrote:
>  On Sun, Mar 26, 2023 at 5:06 PM Zbigniew Jędrzejewski-Szmek <zbyszek(a)in.waw.pl&gt;
> wrote:
> >
> > We should probably make it ^(/etc/|/bin/|/sbin/|/usr/bin/|/usr/sbin/). Split-usr
> > distros are a blast from the past, but adding the extra paths wouldn't change
> > anything for us, since we don't use them.
>
>
>  Please add /app/bin/ and /app/sbin/ as well to that list for flatpak rpm builds.
>  
This shouldn't be needed because the buildsystem can happily pull
filelists.xml. That said, this is part of the reason why we don't
mandate a prefix in createrepo_c. We can't know what the prefix is
going to be for RPMs for projects.



-- 
真実はいつも一つ！/ Always, there's only one truth!

[Fabio Valentini]

On Mon, Mar 27, 2023 at 2:10 PM Neal Gompa <ngompa13(a)gmail.com&gt; wrote:
>  On Sun, Mar 26, 2023 at 12:27 PM Kalev Lember <kalevlember(a)gmail.com&gt; wrote:
>  >
>  > On Sun, Mar 26, 2023 at 5:06 PM Zbigniew Jędrzejewski-Szmek
> <zbyszek(a)in.waw.pl&gt; wrote:
>  >>
>  >> We should probably make it ^(/etc/|/bin/|/sbin/|/usr/bin/|/usr/sbin/).
> Split-usr
>  >> distros are a blast from the past, but adding the extra paths wouldn't
> change
>  >> anything for us, since we don't use them.
>  >
>  >
>  > Please add /app/bin/ and /app/sbin/ as well to that list for flatpak rpm builds.
>  >
>
>  This shouldn't be needed because the buildsystem can happily pull
>  filelists.xml. That said, this is part of the reason why we don't
>  mandate a prefix in createrepo_c. We can't know what the prefix is
>  going to be for RPMs for projects. 
If this is a difficult topic because different distros have different
expectations here, would it be possible to make this configurable
(either at runtime or at build time)?
That way createrepo_c wouldn't need to be opinionated ...

Fabio

[Neal Gompa]

On Mon, Mar 27, 2023 at 11:24 AM Fabio Valentini <decathorpe(a)gmail.com&gt; wrote:
>  On Mon, Mar 27, 2023 at 2:10 PM Neal Gompa <ngompa13(a)gmail.com&gt; wrote:
>  >
>  > On Sun, Mar 26, 2023 at 12:27 PM Kalev Lember <kalevlember(a)gmail.com&gt; wrote:
>  > >
>  > > On Sun, Mar 26, 2023 at 5:06 PM Zbigniew Jędrzejewski-Szmek
> <zbyszek(a)in.waw.pl&gt; wrote:
>  > >>
>  > >> We should probably make it ^(/etc/|/bin/|/sbin/|/usr/bin/|/usr/sbin/).
> Split-usr
>  > >> distros are a blast from the past, but adding the extra paths wouldn't
> change
>  > >> anything for us, since we don't use them.
>  > >
>  > >
>  > > Please add /app/bin/ and /app/sbin/ as well to that list for flatpak rpm
> builds.
>  > >
>  >
>  > This shouldn't be needed because the buildsystem can happily pull
>  > filelists.xml. That said, this is part of the reason why we don't
>  > mandate a prefix in createrepo_c. We can't know what the prefix is
>  > going to be for RPMs for projects.
>
>  If this is a difficult topic because different distros have different
>  expectations here, would it be possible to make this configurable
>  (either at runtime or at build time)?
>  That way createrepo_c wouldn't need to be opinionated ...
>  
Just don't anchor the regex with "^" the beginning. Then it works for
anything.

That is: "/bin/|/lib(|32|64|exec)/|/etc/"




--
真実はいつも一つ！/ Always, there's only one truth!

[Zbigniew Jędrzejewski-Szmek]

On Mon, Mar 27, 2023 at 11:46:49AM -0400, Neal Gompa wrote:
>  On Mon, Mar 27, 2023 at 11:24 AM Fabio Valentini
> <decathorpe(a)gmail.com&gt; wrote:
>  >
>  > On Mon, Mar 27, 2023 at 2:10 PM Neal Gompa <ngompa13(a)gmail.com&gt; wrote:
>  > >
>  > > On Sun, Mar 26, 2023 at 12:27 PM Kalev Lember <kalevlember(a)gmail.com&gt;
> wrote:
>  > > >
>  > > > On Sun, Mar 26, 2023 at 5:06 PM Zbigniew Jędrzejewski-Szmek
> <zbyszek(a)in.waw.pl&gt; wrote:
>  > > >>
>  > > >> We should probably make it ^(/etc/|/bin/|/sbin/|/usr/bin/|/usr/sbin/).
> Split-usr
>  > > >> distros are a blast from the past, but adding the extra paths
> wouldn't change
>  > > >> anything for us, since we don't use them.
>  > > >
>  > > >
>  > > > Please add /app/bin/ and /app/sbin/ as well to that list for flatpak rpm
> builds.
>  > > >
>  > >
>  > > This shouldn't be needed because the buildsystem can happily pull
>  > > filelists.xml. That said, this is part of the reason why we don't
>  > > mandate a prefix in createrepo_c. We can't know what the prefix is
>  > > going to be for RPMs for projects.
>  >
>  > If this is a difficult topic because different distros have different
>  > expectations here, would it be possible to make this configurable
>  > (either at runtime or at build time)?
>  > That way createrepo_c wouldn't need to be opinionated ...
>  >
>  
>  Just don't anchor the regex with "^" the beginning. Then it works for
> anything.
>  
>  That is: "/bin/|/lib(|32|64|exec)/|/etc/" 
I want to keep it clear that _my_ proposal is to make this narrower,
to match the packaging guidelines. The guidelines say that packages
SHOULD NOT use paths outside of /usr/bin|/usr/sbin|/etc, so any other
paths that we put in primary just make things less efficient.

And as mentioned previously, adding libs would blow up the number
of paths in primary, even though they are not useful because we
have a dependency generator.

Making this configurable is probably easier than discussing the
"correct" path endlessly. Then the setting for Fedora can easily
be made to match the guidelines. What other distros do will be up
to them.

Zbyszek

[Petr Pisar]

V Mon, Mar 27, 2023 at 11:46:49AM -0400, Neal Gompa napsal(a):
>  On Mon, Mar 27, 2023 at 11:24 AM Fabio Valentini
> <decathorpe(a)gmail.com&gt; wrote:
>  >
>  > On Mon, Mar 27, 2023 at 2:10 PM Neal Gompa <ngompa13(a)gmail.com&gt; wrote:
>  > >
>  > > On Sun, Mar 26, 2023 at 12:27 PM Kalev Lember <kalevlember(a)gmail.com&gt;
> wrote:
>  > > >
>  > > > On Sun, Mar 26, 2023 at 5:06 PM Zbigniew Jędrzejewski-Szmek
> <zbyszek(a)in.waw.pl&gt; wrote:
>  > > >>
>  > > >> We should probably make it ^(/etc/|/bin/|/sbin/|/usr/bin/|/usr/sbin/).
> Split-usr
>  > > >> distros are a blast from the past, but adding the extra paths
> wouldn't change
>  > > >> anything for us, since we don't use them.
>  > > >
>  > > >
>  > > > Please add /app/bin/ and /app/sbin/ as well to that list for flatpak rpm
> builds.
>  > > >
>  > >
>  > > This shouldn't be needed because the buildsystem can happily pull
>  > > filelists.xml. That said, this is part of the reason why we don't
>  > > mandate a prefix in createrepo_c. We can't know what the prefix is
>  > > going to be for RPMs for projects.
>  >
>  > If this is a difficult topic because different distros have different
>  > expectations here, would it be possible to make this configurable
>  > (either at runtime or at build time)?
>  > That way createrepo_c wouldn't need to be opinionated ...
>  >
>  
>  Just don't anchor the regex with "^" the beginning. Then it works for
> anything.
>  
>  That is: "/bin/|/lib(|32|64|exec)/|/etc/"
>   
I don't think it's wise to match /usr/libexec which is defined as
<https://refspecs.linuxfoundation.org/FHS_3.0/fhs/ch04s07.html>;:

    /usr/libexec includes internal binaries that are not intended to be
    executed directly by users or shell scripts.

-- Petr

[Mattia Verga]

Il 23/03/23 19:52, Ben Cotton ha scritto:
>  https://fedoraproject.org/wiki/Changes/createrepo_c_1.0.0
>  
A quick look about this change impact on bodhi-server, as I understand
from the relevant module at
https://github.com/fedora-infra/bodhi/blob/develop/bodhi-server/bodhi/ser...
:
>  == Detailed Description ==
>  - Switch default compression from gz to zstd. DNF is able to
>  decompress zstd via libsolv since Fedora 30 and RHEL 8.4. It will
>  still be possible to create gz compressed repodata by manually
>  specifying gz compression. 
As I understand, bodhi sets XZ compression on all repos
but those with a
release prefix set to 'FEDORA-EPEL' (EPEL7, EPEL8 and EPEL9) where it
uses BZ2.
Do we want to continue using XZ or switch to the new ZSTD default?
Do we want to continue using BZ for EPEL9 or switch to ZSTD/XZ? (EPEL
Next, both 8 and 9, are already using XZ if I understand correctly)
>  - Stop generating metadata in sqlite database format by default. It
>  will still be possible to create them using a --database switch or via
>  sqliterepo_c tool. (It looks like the only user left is mdapi:
>  https://github.com/fedora-infra/mdapi/issues/97). 
I think this haven't got
impact on bodhi-server.
>  - When adding groups.xml to repodata createrepo_c currently adds two
>  variants to repomd.xml. The specified file as is, uncompressed, with
>  the type "group" and also a compressed variant with type "group_XX",
>  where XX is compression suffix. This is atypical and unexpected. We
>  propose to include just one variant of groups.xml using specified
>  compression and repomd.xml type "group". This is not compatible with
>  yum in RHEL 7. If required users will still be able to create
>  repositories with the old layout using modifyrepo_c. 
I'm not sure if this has impact on bodhi-server. We're composing
repodata for EPEL7 and EPEL8 which still use yum, but I don't think
we're adding groups metadata?

Mattia

[Mattia Verga]

> > - Stop generating metadata in sqlite database format by default.
> It
> > will still be possible to create them using a --database switch or via
> > sqliterepo_c tool. (It looks like the only user left is mdapi:
> > https://github.com/fedora-infra/mdapi/issues/97). 
>  I think this haven't got
>  impact on bodhi-server. 
Correction: bodhi-server generates sqlite database format. Do we want to continue
distributing that?

Mattia

[Kevin Fenzi]

On Fri, Mar 24, 2023 at 05:25:17PM +0000, Mattia Verga via devel wrote:
>  Il 23/03/23 19:52, Ben Cotton ha scritto:
>  > https://fedoraproject.org/wiki/Changes/createrepo_c_1.0.0
>  >
>  A quick look about this change impact on bodhi-server, as I understand
>  from the relevant module at
>  https://github.com/fedora-infra/bodhi/blob/develop/bodhi-server/bodhi/ser...
This is the part of bodhi that inserts metadata into the repodata.
ie, 'this is a security update' etc. ie, updateinfo.xml
Right?

>  > == Detailed Description ==
>  > - Switch default compression from gz to zstd. DNF is able to
>  > decompress zstd via libsolv since Fedora 30 and RHEL 8.4. It will
>  > still be possible to create gz compressed repodata by manually
>  > specifying gz compression.
>  As I understand, bodhi sets XZ compression on all repos but those with a
>  release prefix set to 'FEDORA-EPEL' (EPEL7, EPEL8 and EPEL9) where it
>  uses BZ2.
>  Do we want to continue using XZ or switch to the new ZSTD default?
>  Do we want to continue using BZ for EPEL9 or switch to ZSTD/XZ? (EPEL
>  Next, both 8 and 9, are already using XZ if I understand correctly) 
Do note that bodhi just calls pungi to make the updates repos, so it's
really about pungi config here right? Of course the questions are still
needing answers if we want to change/adjust the pungi config.

This should be:
https://pagure.io/fedora-infra/ansible/blob/main/f/roles/bodhi2/backend/t...
and
https://pagure.io/fedora-infra/ansible/blob/main/f/roles/bodhi2/backend/t...
for modules.

So from that it looks like epel8 is xz specifically, but the rest are
default. 

>  > - Stop generating metadata in sqlite database format by default.
> It
>  > will still be possible to create them using a --database switch or via
>  > sqliterepo_c tool. (It looks like the only user left is mdapi:
>  > https://github.com/fedora-infra/mdapi/issues/97).
>  I think this haven't got impact on bodhi-server.
>  >
>  > - When adding groups.xml to repodata createrepo_c currently adds two
>  > variants to repomd.xml. The specified file as is, uncompressed, with
>  > the type "group" and also a compressed variant with type
> "group_XX",
>  > where XX is compression suffix. This is atypical and unexpected. We
>  > propose to include just one variant of groups.xml using specified
>  > compression and repomd.xml type "group". This is not compatible with
>  > yum in RHEL 7. If required users will still be able to create
>  > repositories with the old layout using modifyrepo_c.
>  
>  I'm not sure if this has impact on bodhi-server. We're composing
>  repodata for EPEL7 and EPEL8 which still use yum, but I don't think
>  we're adding groups metadata? 
We are I am pretty sure. ;) This is from fedora-comps, we gather that
and pungi does some processing on it, but then inserts it in the end as
group data. 

I have a few additional questions:

* How does this impact zchunk? or does it?

* Kind of unrealted to this change, but I'll ask here as createrepo_c
folks might know off hand: Is it possible to decouple drpms from a repo?
ie, if we had a updates repo and a completely seperate updates-drpms
repo with just drpms in it, what would it take to use that? Changes in
dnf I guess? but also changes in createrepo_c to generate repodata for a
repo that was only drpms?

Thanks for working on this!

kevin

[Mattia Verga]

Il 25/03/23 21:34, Kevin Fenzi ha scritto:
>  On Fri, Mar 24, 2023 at 05:25:17PM +0000, Mattia Verga via devel
> wrote:
> > Il 23/03/23 19:52, Ben Cotton ha scritto:
> >> https://fedoraproject.org/wiki/Changes/createrepo_c_1.0.0
> >>
> > A quick look about this change impact on bodhi-server, as I understand
> > from the relevant module at
> >
> https://github.com/fedora-infra/bodhi/blob/develop/bodhi-server/bodhi/ser...
>  This is the part of bodhi that inserts metadata into the repodata.
>  ie, 'this is a security update' etc. ie, updateinfo.xml
>  Right? 
Ah, yes, it is relevant for updateinfo.xml only.

https://github.com/fedora-infra/bodhi/blob/1a2b192929fa1836ed04e7a0660c74...

> >> == Detailed Description ==
> >> - Switch default compression from gz to zstd. DNF is able to
> >> decompress zstd via libsolv since Fedora 30 and RHEL 8.4. It will
> >> still be possible to create gz compressed repodata by manually
> >> specifying gz compression.
> > As I understand, bodhi sets XZ compression on all repos but those with a
> > release prefix set to 'FEDORA-EPEL' (EPEL7, EPEL8 and EPEL9) where it
> > uses BZ2.
> > Do we want to continue using XZ or switch to the new ZSTD default?
> > Do we want to continue using BZ for EPEL9 or switch to ZSTD/XZ? (EPEL
> > Next, both 8 and 9, are already using XZ if I understand correctly)
>  Do note that bodhi just calls pungi to make the updates repos, so it's
>  really about pungi config here right? Of course the questions are still
>  needing answers if we want to change/adjust the pungi config.
>
>  This should be:
>
> https://pagure.io/fedora-infra/ansible/blob/main/f/roles/bodhi2/backend/t...
>  and
>
> https://pagure.io/fedora-infra/ansible/blob/main/f/roles/bodhi2/backend/t...
>  for modules.
>
>  So from that it looks like epel8 is xz specifically, but the rest are
>  default. 
So, we're using different settings for pungi config and for
updateinfo.xml generated by Bodhi.

I don't think there's a (easy) way to have bodhi inherit configuration
from pungi config file.

I would first suggest to "fix" Bodhi metadata generation to use the same
compression setting (which is now XZ) for everything except EPEL7 and
EPEL8 (with the current conditional it's using BZ2 for EPEL9 also). Then
we can discuss about switching from XZ to ZSTD, but I think the choice
should be mirrored on pungi side.

This would be a kind of hard coded change in bodhi code. If there's a
rationale we could modify bodhi to have those parameters set in database
for each Release object, but does it worth the work?

> >> - Stop generating metadata in sqlite database format by
> default. It
> >> will still be possible to create them using a --database switch or via
> >> sqliterepo_c tool. (It looks like the only user left is mdapi:
> >> https://github.com/fedora-infra/mdapi/issues/97).
> > I think this haven't got impact on bodhi-server.
> >> - When adding groups.xml to repodata createrepo_c currently adds two
> >> variants to repomd.xml. The specified file as is, uncompressed, with
> >> the type "group" and also a compressed variant with type
> "group_XX",
> >> where XX is compression suffix. This is atypical and unexpected. We
> >> propose to include just one variant of groups.xml using specified
> >> compression and repomd.xml type "group". This is not compatible with
> >> yum in RHEL 7. If required users will still be able to create
> >> repositories with the old layout using modifyrepo_c.
> > I'm not sure if this has impact on bodhi-server. We're composing
> > repodata for EPEL7 and EPEL8 which still use yum, but I don't think
> > we're adding groups metadata?
>  We are I am pretty sure. ;) This is from fedora-comps, we gather that
>  and pungi does some processing on it, but then inserts it in the end as
>  group data.
>
>  I have a few additional questions:
>
>  * How does this impact zchunk? or does it? 
AFAIK, zchunck is not enabled for EPEL, so it should not have impact.

Mattia

[Kevin Fenzi]

On Sun, Mar 26, 2023 at 04:54:37PM +0000, Mattia Verga via devel wrote:
>  
>  So, we're using different settings for pungi config and for
>  updateinfo.xml generated by Bodhi.
>  
>  I don't think there's a (easy) way to have bodhi inherit configuration
>  from pungi config file.
>  
>  I would first suggest to "fix" Bodhi metadata generation to use the same
>  compression setting (which is now XZ) for everything except EPEL7 and
>  EPEL8 (with the current conditional it's using BZ2 for EPEL9 also). Then
>  we can discuss about switching from XZ to ZSTD, but I think the choice
>  should be mirrored on pungi side. 
Sounds reasonable.

>  This would be a kind of hard coded change in bodhi code. If
> there's a
>  rationale we could modify bodhi to have those parameters set in database
>  for each Release object, but does it worth the work? 
I'm not sure. I suppose down the road fedora 45 or something might
change again, but we could deal with the hard coded change then, or
implement the release object thing.

kevin

[Aleš Matěj]

>  On Fri, Mar 24, 2023 at 05:25:17PM +0000, Mattia Verga via devel
> wrote:
>  I have a few additional questions:
>  
>  * How does this impact zchunk? or does it? 
I also think there shouldn't be any impact on zchunk.

>  * Kind of unrealted to this change, but I'll ask here as
> createrepo_c
>  folks might know off hand: Is it possible to decouple drpms from a repo?
>  ie, if we had a updates repo and a completely seperate updates-drpms
>  repo with just drpms in it, what would it take to use that? Changes in
>  dnf I guess? but also changes in createrepo_c to generate repodata for a
>  repo that was only drpms? 
Such repos can be created by createrepo_c and modifyrepo_c but it is a bit awkward.
Basically first create normal (coupled) repo and then extract the prestodelta.xml and
drpms
into a new repo. Additionally createrepo_c API could be used to make it more straight
forward.
However you are right dnf can't handle that. It looks for deltas in the same repo
where it finds 
the normal update package. It would require changes in dnf and libdnf.

[Kevin Fenzi]

On Mon, Mar 27, 2023 at 11:57:04AM -0000, Aleš Matěj wrote:
>  > On Fri, Mar 24, 2023 at 05:25:17PM +0000, Mattia Verga via devel
> wrote:
>  > I have a few additional questions:
>  > 
>  > * How does this impact zchunk? or does it?
>  
>  I also think there shouldn't be any impact on zchunk. 
ok. Great to hear. 

>  > * Kind of unrealted to this change, but I'll ask here as
> createrepo_c
>  > folks might know off hand: Is it possible to decouple drpms from a repo?
>  > ie, if we had a updates repo and a completely seperate updates-drpms
>  > repo with just drpms in it, what would it take to use that? Changes in
>  > dnf I guess? but also changes in createrepo_c to generate repodata for a
>  > repo that was only drpms?
>  
>  Such repos can be created by createrepo_c and modifyrepo_c but it is a bit awkward.
>  Basically first create normal (coupled) repo and then extract the prestodelta.xml and
> drpms
>  into a new repo. Additionally createrepo_c API could be used to make it more straight
> forward. 
Well, I was thinking of a completely disconnected process. ie, the drpms
are created compltely seperately from the main repo and a
prestodelta.xml is created there in the seperate repo.

>  However you are right dnf can't handle that. It looks for deltas
> in the same repo where it finds 
>  the normal update package. It would require changes in dnf and libdnf. 
ok. Thanks for the info. 

If we want to bring drpms back to useful, I think this would probibly be the
best way to go. Then we could have some app that creates them async and
dnf could use them if available.

kevin

[Stephen Smoogen]

On Mon, 27 Mar 2023 at 10:29, Kevin Fenzi <kevin(a)scrye.com&gt; wrote:

>  On Mon, Mar 27, 2023 at 11:57:04AM -0000, Aleš Matěj wrote:
>  > > On Fri, Mar 24, 2023 at 05:25:17PM +0000, Mattia Verga via devel wrote:
>  > > I have a few additional questions:
>  > >
>  > > * How does this impact zchunk? or does it?
>  >
>  > I also think there shouldn't be any impact on zchunk.
>
>  ok. Great to hear.
>
>  > > * Kind of unrealted to this change, but I'll ask here as createrepo_c
>  > > folks might know off hand: Is it possible to decouple drpms from a
>  repo?
>  > > ie, if we had a updates repo and a completely seperate updates-drpms
>  > > repo with just drpms in it, what would it take to use that? Changes in
>  > > dnf I guess? but also changes in createrepo_c to generate repodata for
>  a
>  > > repo that was only drpms?
>  >
>  > Such repos can be created by createrepo_c and modifyrepo_c but it is a
>  bit awkward.
>  > Basically first create normal (coupled) repo and then extract the
>  prestodelta.xml and drpms
>  > into a new repo. Additionally createrepo_c API could be used to make it
>  more straight forward.
>
>  Well, I was thinking of a completely disconnected process. ie, the drpms
>  are created compltely seperately from the main repo and a
>  prestodelta.xml is created there in the seperate repo.
>
>  
Or hear me out here for this wild thought: We stop offering drpms.



-- 
Stephen Smoogen, Red Hat Automotive
Let us be kind to one another, for most of us are fighting a hard battle.
-- Ian MacClaren

[Kevin Fenzi]

On Mon, Mar 27, 2023 at 10:32:13AM -0400, Stephen Smoogen wrote:
>  Or hear me out here for this wild thought: We stop offering drpms.
yes, yes, I just wanted to know if there was a plan forward for people
interested in doing the work to support them. I am not committing to
doing that work. ;) 

kevin

[Maxwell G]

On Mon Mar 27, 2023 at 07:27 -0700, Kevin Fenzi wrote:
>  On Mon, Mar 27, 2023 at 11:57:04AM -0000, Aleš Matěj wrote:
<snip>
>  > However you are right dnf can't handle that. It looks for
> deltas in the same repo where it finds 
>  > the normal update package. It would require changes in dnf and libdnf.
>
>  ok. Thanks for the info. 
>
>  If we want to bring drpms back to useful, I think this would probibly be the
>  best way to go. Then we could have some app that creates them async and
>  dnf could use them if available.
>
>  kevin 
Can you or someone else more familiar with DRPMs than I file a dnf5 RFE?

https://github.com/rpm-software-management/dnf5/issues/new

--
Maxwell G (@gotmax23)
Pronouns: He/They

[Kevin Fenzi]

On Mon, Mar 27, 2023 at 03:19:00PM +0000, Maxwell G wrote:
>  On Mon Mar 27, 2023 at 07:27 -0700, Kevin Fenzi wrote:
>  > On Mon, Mar 27, 2023 at 11:57:04AM -0000, Aleš Matěj wrote:
>  <snip>
>  > > However you are right dnf can't handle that. It looks for deltas in the
> same repo where it finds 
>  > > the normal update package. It would require changes in dnf and libdnf.
>  >
>  > ok. Thanks for the info. 
>  >
>  > If we want to bring drpms back to useful, I think this would probibly be the
>  > best way to go. Then we could have some app that creates them async and
>  > dnf could use them if available.
>  >
>  > kevin
>  
>  Can you or someone else more familiar with DRPMs than I file a dnf5 RFE?
>  
>  https://github.com/rpm-software-management/dnf5/issues/new 
Well, I don't know that I have any time to drive this, 
and I am not sure if it should be on dnf5 or libdnf or librepo or all of
them?

Anyhow, filed https://github.com/rpm-software-management/dnf5/issues/408

kevin

[Daniel Alley]

>  - When adding groups.xml to repodata createrepo_c currently adds two
>  variants to repomd.xml. The specified file as is, uncompressed, with
>  the type "group" and also a compressed variant with type "group_XX",
>  where XX is compression suffix. This is atypical and unexpected. We
>  propose to include just one variant of groups.xml using specified
>  compression and repomd.xml type "group". This is not compatible with
>  yum in RHEL 7. If required users will still be able to create
>  repositories with the old layout using modifyrepo_c.
>  Further information: https://bugzilla.redhat.com/show_bug.cgi?id=2056318 
If we're planning to drop by-default compatibility with EL7 anyway and bump to 1.0, we
could go a little bit further.

* "createrepo_c" doesn't need to support sqlite metadata generation at all,
so long as the repository producer can run "sqliterepo_c".  If I recall
correctly, yum on EL7 doesn't actually need the sqlite metadata to be generated in any
case, it can produce that metadata itself so long as it has the XML.  Making it available
only avoids a local processing step.  But if "modifyrepo_c" will be necessary to
work with most EL7 repos, then making "sqliterepo_c" necessary for a minor
optimization isn't a big leap.

* createrepo_c currently has two different switches for compression types,
"compress-type" and "general-compress-type", where the latter applies
to everything and the former only applies to files that are *not* "primary",
"filelists", and "other".  there should be one compression type option
and it should be applied uniformly to all metadata produced by the
"createrepo_c" tool.  Yum would have choked on this because of the special
handling around "comps", but I believe it should no longer be a concern?

[Aleš Matěj]

>  If we're planning to drop by-default compatibility with EL7
> anyway and bump to 1.0, we
>  could go a little bit further.
>  
>  * "createrepo_c" doesn't need to support sqlite metadata generation at
> all,
>  so long as the repository producer can run "sqliterepo_c".  If I recall
>  correctly, yum on EL7 doesn't actually need the sqlite metadata to be generated in
> any
>  case, it can produce that metadata itself so long as it has the XML.  Making it
> available
>  only avoids a local processing step.  But if "modifyrepo_c" will be necessary
> to
>  work with most EL7 repos, then making "sqliterepo_c" necessary for a minor
>  optimization isn't a big leap. 
We were also discussing adding some "mode" options that could be set to EL7 to
generate
compatible metadata for EL7 with just createrepo_c.
 
>  * createrepo_c currently has two different switches for compression
> types,
>  "compress-type" and "general-compress-type", where the latter
> applies
>  to everything and the former only applies to files that are *not* "primary",
>  "filelists", and "other".  there should be one compression type
> option
>  and it should be applied uniformly to all metadata produced by the
>  "createrepo_c" tool.  Yum would have choked on this because of the special
>  handling around "comps", but I believe it should no longer be a concern?
Yes that is a good suggestion. There are also other command line options that
could use some (re)work. For example the unique --xz option which bypasses
the mentioned compression options. 
Unfortunately I am not sure if we will be able to prioritize all the improvements.