F39 proposal: Changes of defaults in createrepo_c-1.0.0 (System-Wide Change proposal)

[Ben Cotton]

https://fedoraproject.org/wiki/Changes/createrepo_c_1.0.0

This document represents a proposed Change. As part of the Changes
process, proposals are publicly announced in order to receive
community feedback. This proposal will only be implemented if approved
by the Fedora Engineering Steering Committee.


== Summary ==
Update createrepo_c to 1.0.0, new release will include change of
default compression to zstd, no longer generating metadata in sqlite
database format by default and simplified comps xml type in repodata.

== Owner ==
* Name: [[User:Amatej | Aleš Matěj]]
* Email: amatej(a)redhat.com



== Detailed Description ==
- Switch default compression from gz to zstd. DNF is able to
decompress zstd via libsolv since Fedora 30 and RHEL 8.4. It will
still be possible to create gz compressed repodata by manually
specifying gz compression.
Further information:
https://github.com/rpm-software-management/createrepo_c/issues/82,
https://bugzilla.redhat.com/show_bug.cgi?id=1914876

- Stop generating metadata in sqlite database format by default. It
will still be possible to create them using a --database switch or via
sqliterepo_c tool. (It looks like the only user left is mdapi:
https://github.com/fedora-infra/mdapi/issues/97).
Further information:
https://github.com/rpm-software-management/createrepo_c/issues/338 and
https://pagure.io/releng/issue/10745

- When adding groups.xml to repodata createrepo_c currently adds two
variants to repomd.xml. The specified file as is, uncompressed, with
the type "group" and also a compressed variant with type "group_XX",
where XX is compression suffix. This is atypical and unexpected. We
propose to include just one variant of groups.xml using specified
compression and repomd.xml type "group". This is not compatible with
yum in RHEL 7. If required users will still be able to create
repositories with the old layout using modifyrepo_c.
Further information: https://bugzilla.redhat.com/show_bug.cgi?id=2056318

- To reflect API and overall stability increase major version above 0.

== Feedback ==


== Benefit to Fedora ==
* Using zstd compression provides smaller metadata that are faster to
decompress.
* Generating sqlite metadata is expensive, its slowing createrepo_c
runs and it takes up space on mirrors. (For F37 updates repo:
primary_db: 12.8 mb, filelists_db: 22.0 mb, other_db: 4.2 mb.)
* More consistent metadata, groups use the same logic as other
metadata types. Fixes:
https://bugzilla.redhat.com/show_bug.cgi?id=1904360

== Scope ==
* Proposal owners: Implement specified changes, mostly changes of default.
* Other developers: There are no requirements on other developers.

* Release engineering:
* Policies and guidelines: N/A (not needed for this Change)
* Trademark approval: N/A (not needed for this Change)
* Alignment with Objectives:


== Upgrade/compatibility impact ==
Repodata created with createrepo_c's new defaults will not be
compatible with fedora < 30 and RHEL < 8.4 (including RHEL 7, 6).
This is mainly due to zstd compression but old YUM also cannot handle
the changed mdtype of groups (it is compatible only with uncompressed
- "group" and compressed - "group_XX" not compressed -
"group").

Regarding the sqlite databases if some other users would require them
they can always download the xml metadata and use sqliterepo_c to
generate them locally.

== How To Test ==
Repositories generated with updated createrepo_c have to work with
dnf. Specifically we should ensure that groups are still loaded.

== User Experience ==
The only noticeable effect should be smaller metadata thanks to zstd
compression resulting in faster downloads (when zchunk is not used).
The (de)compression speed up is not very noticeable.

For example for F37 fedora repo:
* primary.xml.gz: 18M -> primary.xml.zst: 14M
* filelists.xml.gz: 52M -> filelists.xml.zst: 41M
* other.xml.gz: 6.9M -> other.xml.zst: 5.4M

== Dependencies ==
Updating createrepo_c does not require any other change requests to
complete first.

Metadata generated by the following tools could be affected by the
described changes if they use createrepo_c's default configuration. It
is hard to tell what effect will this have on the tools. If they
create metadata for dnf it will work fine. In case there are any
problems it is possible to manually setup the old behavior (as
described above) or revert the changes.

 bodhi-server
 copr-backend
 libguestfs
 modulemd-tools
 prunerepo
 pungi
 retrace-server
 swid-tools
 anaconda-install-env-deps
 cobbler
 ddiskit
 koji-builder
 mock
 module-build
 perl-Fedora-Rebuild
 python3-dlrn
 python3-dnf-plugin-local
 python3-dnf-plugin-modulesync
 python3-rpmfluff
 retrace-server

== Contingency Plan ==
* Contingency mechanism: If we cannot finish the changes in time there
would be no action necessary. We can even ship only part of the
changes as they don't depend on each other. In case there are
additional unexpected issues with the changes we can revert them in
createrepo_c.
* Contingency deadline: 2023-08-01
* Blocks release? No

== Documentation ==
Createrepo_c documentation will be updated accordingly.

== Release Notes ==



-- 
Ben Cotton
He / Him / His
Fedora Program Manager
Red Hat
TZ=America/Indiana/Indianapolis