https://fedoraproject.org/wiki/Changes/createrepo_c_1.0.0
This document represents a proposed Change. As part of the Changes process, proposals are publicly announced in order to receive community feedback. This proposal will only be implemented if approved by the Fedora Engineering Steering Committee.
== Summary == Update createrepo_c to 1.0.0, new release will include change of default compression to zstd, no longer generating metadata in sqlite database format by default and simplified comps xml type in repodata.
== Owner == * Name: [[User:Amatej | Aleš Matěj]] * Email: amatej@redhat.com
== Detailed Description == - Switch default compression from gz to zstd. DNF is able to decompress zstd via libsolv since Fedora 30 and RHEL 8.4. It will still be possible to create gz compressed repodata by manually specifying gz compression. Further information: https://github.com/rpm-software-management/createrepo_c/issues/82, https://bugzilla.redhat.com/show_bug.cgi?id=1914876
- Stop generating metadata in sqlite database format by default. It will still be possible to create them using a --database switch or via sqliterepo_c tool. (It looks like the only user left is mdapi: https://github.com/fedora-infra/mdapi/issues/97). Further information: https://github.com/rpm-software-management/createrepo_c/issues/338 and https://pagure.io/releng/issue/10745
- When adding groups.xml to repodata createrepo_c currently adds two variants to repomd.xml. The specified file as is, uncompressed, with the type "group" and also a compressed variant with type "group_XX", where XX is compression suffix. This is atypical and unexpected. We propose to include just one variant of groups.xml using specified compression and repomd.xml type "group". This is not compatible with yum in RHEL 7. If required users will still be able to create repositories with the old layout using modifyrepo_c. Further information: https://bugzilla.redhat.com/show_bug.cgi?id=2056318
- To reflect API and overall stability increase major version above 0.
== Feedback ==
== Benefit to Fedora == * Using zstd compression provides smaller metadata that are faster to decompress. * Generating sqlite metadata is expensive, its slowing createrepo_c runs and it takes up space on mirrors. (For F37 updates repo: primary_db: 12.8 mb, filelists_db: 22.0 mb, other_db: 4.2 mb.) * More consistent metadata, groups use the same logic as other metadata types. Fixes: https://bugzilla.redhat.com/show_bug.cgi?id=1904360
== Scope == * Proposal owners: Implement specified changes, mostly changes of default. * Other developers: There are no requirements on other developers.
* Release engineering: * Policies and guidelines: N/A (not needed for this Change) * Trademark approval: N/A (not needed for this Change) * Alignment with Objectives:
== Upgrade/compatibility impact == Repodata created with createrepo_c's new defaults will not be compatible with fedora < 30 and RHEL < 8.4 (including RHEL 7, 6). This is mainly due to zstd compression but old YUM also cannot handle the changed mdtype of groups (it is compatible only with uncompressed - "group" and compressed - "group_XX" not compressed - "group").
Regarding the sqlite databases if some other users would require them they can always download the xml metadata and use sqliterepo_c to generate them locally.
== How To Test == Repositories generated with updated createrepo_c have to work with dnf. Specifically we should ensure that groups are still loaded.
== User Experience == The only noticeable effect should be smaller metadata thanks to zstd compression resulting in faster downloads (when zchunk is not used). The (de)compression speed up is not very noticeable.
For example for F37 fedora repo: * primary.xml.gz: 18M -> primary.xml.zst: 14M * filelists.xml.gz: 52M -> filelists.xml.zst: 41M * other.xml.gz: 6.9M -> other.xml.zst: 5.4M
== Dependencies == Updating createrepo_c does not require any other change requests to complete first.
Metadata generated by the following tools could be affected by the described changes if they use createrepo_c's default configuration. It is hard to tell what effect will this have on the tools. If they create metadata for dnf it will work fine. In case there are any problems it is possible to manually setup the old behavior (as described above) or revert the changes.
bodhi-server copr-backend libguestfs modulemd-tools prunerepo pungi retrace-server swid-tools anaconda-install-env-deps cobbler ddiskit koji-builder mock module-build perl-Fedora-Rebuild python3-dlrn python3-dnf-plugin-local python3-dnf-plugin-modulesync python3-rpmfluff retrace-server
== Contingency Plan == * Contingency mechanism: If we cannot finish the changes in time there would be no action necessary. We can even ship only part of the changes as they don't depend on each other. In case there are additional unexpected issues with the changes we can revert them in createrepo_c. * Contingency deadline: 2023-08-01 * Blocks release? No
== Documentation == Createrepo_c documentation will be updated accordingly.
== Release Notes ==
On Thu, Mar 23, 2023 at 02:52:50PM -0400, Ben Cotton wrote:
== Summary == Update createrepo_c to 1.0.0, new release will include change of default compression to zstd, no longer generating metadata in sqlite database format by default and simplified comps xml type in repodata.
LGTM. Thank you for taking the time to figure out all the dependencies.
While at it, would it be possible to narrow down the check for which file paths are included in primary filelists to match the packaging docs?
Currently [1] the pattern is implemented as '(^/etc|^/usr/lib/sendmail$|bin/)' (note that "bin/" can apply anywhere in the string). The comment says that this is for yum compat [2]. Our guidelines only mention files under /usr/bin, /usr/sbin, /etc [3].
I think that the number of files affected is small, but it is a potential source of confusion and a hard-to-explain discrepancy between what the guidelines say and actual dnf behaviour. Yum is long gone and we clearly intended to clean this up many years ago but never got round to it.
[1] https://github.com/rpm-software-management/createrepo_c/blob/master/src/misc... [2] https://github.com/rpm-software-management/createrepo_c/blob/master/src/misc... [3] https://docs.fedoraproject.org/en-US/packaging-guidelines/#_file_and_directo...
Zbyszek
On Fri, Mar 24, 2023 at 3:42 AM Zbigniew Jędrzejewski-Szmek zbyszek@in.waw.pl wrote:
On Thu, Mar 23, 2023 at 02:52:50PM -0400, Ben Cotton wrote:
== Summary == Update createrepo_c to 1.0.0, new release will include change of default compression to zstd, no longer generating metadata in sqlite database format by default and simplified comps xml type in repodata.
LGTM. Thank you for taking the time to figure out all the dependencies.
While at it, would it be possible to narrow down the check for which file paths are included in primary filelists to match the packaging docs?
Currently [1] the pattern is implemented as '(^/etc|^/usr/lib/sendmail$|bin/)' (note that "bin/" can apply anywhere in the string). The comment says that this is for yum compat [2]. Our guidelines only mention files under /usr/bin, /usr/sbin, /etc [3].
I think that the number of files affected is small, but it is a potential source of confusion and a hard-to-explain discrepancy between what the guidelines say and actual dnf behaviour. Yum is long gone and we clearly intended to clean this up many years ago but never got round to it.
[1] https://github.com/rpm-software-management/createrepo_c/blob/master/src/misc... [2] https://github.com/rpm-software-management/createrepo_c/blob/master/src/misc... [3] https://docs.fedoraproject.org/en-US/packaging-guidelines/#_file_and_directo...
I'd rather it be bin/|lib*/|^/etc so that binaries and libraries of all primary paths (the majority of file deps) are covered in primary.
We should not assume that everything is in /usr because createrepo_c is used for non-Linux repos that use different prefixes (hence why "bin/").
On Sat, Mar 25, 2023 at 06:29:14AM -0400, Neal Gompa wrote:
On Fri, Mar 24, 2023 at 3:42 AM Zbigniew Jędrzejewski-Szmek zbyszek@in.waw.pl wrote:
On Thu, Mar 23, 2023 at 02:52:50PM -0400, Ben Cotton wrote:
== Summary == Update createrepo_c to 1.0.0, new release will include change of default compression to zstd, no longer generating metadata in sqlite database format by default and simplified comps xml type in repodata.
LGTM. Thank you for taking the time to figure out all the dependencies.
While at it, would it be possible to narrow down the check for which file paths are included in primary filelists to match the packaging docs?
Currently [1] the pattern is implemented as '(^/etc|^/usr/lib/sendmail$|bin/)' (note that "bin/" can apply anywhere in the string). The comment says that this is for yum compat [2]. Our guidelines only mention files under /usr/bin, /usr/sbin, /etc [3].
I think that the number of files affected is small, but it is a potential source of confusion and a hard-to-explain discrepancy between what the guidelines say and actual dnf behaviour. Yum is long gone and we clearly intended to clean this up many years ago but never got round to it.
[1] https://github.com/rpm-software-management/createrepo_c/blob/master/src/misc... [2] https://github.com/rpm-software-management/createrepo_c/blob/master/src/misc... [3] https://docs.fedoraproject.org/en-US/packaging-guidelines/#_file_and_directo...
I'd rather it be bin/|lib*/|^/etc so that binaries and libraries of all primary paths (the majority of file deps) are covered in primary.
Sorry, but that'd be a (big) step backwards. `lib*/` would hugely expand the list of files in primary, while the goal is to make it smaller. Also, for libraries we already have a better mechanism: provides based on SONAME and symbol versions.
We should not assume that everything is in /usr because createrepo_c is used for non-Linux repos that use different prefixes (hence why "bin/").
We should probably make it ^(/etc/|/bin/|/sbin/|/usr/bin/|/usr/sbin/). Split-usr distros are a blast from the past, but adding the extra paths wouldn't change anything for us, since we don't use them.
Zbyszek
On Sun, Mar 26, 2023 at 5:06 PM Zbigniew Jędrzejewski-Szmek < zbyszek@in.waw.pl> wrote:
We should probably make it ^(/etc/|/bin/|/sbin/|/usr/bin/|/usr/sbin/). Split-usr distros are a blast from the past, but adding the extra paths wouldn't change anything for us, since we don't use them.
Please add /app/bin/ and /app/sbin/ as well to that list for flatpak rpm builds.
On Sun, Mar 26, 2023 at 12:27 PM Kalev Lember kalevlember@gmail.com wrote:
On Sun, Mar 26, 2023 at 5:06 PM Zbigniew Jędrzejewski-Szmek zbyszek@in.waw.pl wrote:
We should probably make it ^(/etc/|/bin/|/sbin/|/usr/bin/|/usr/sbin/). Split-usr distros are a blast from the past, but adding the extra paths wouldn't change anything for us, since we don't use them.
Please add /app/bin/ and /app/sbin/ as well to that list for flatpak rpm builds.
This shouldn't be needed because the buildsystem can happily pull filelists.xml. That said, this is part of the reason why we don't mandate a prefix in createrepo_c. We can't know what the prefix is going to be for RPMs for projects.
On Mon, Mar 27, 2023 at 2:10 PM Neal Gompa ngompa13@gmail.com wrote:
On Sun, Mar 26, 2023 at 12:27 PM Kalev Lember kalevlember@gmail.com wrote:
On Sun, Mar 26, 2023 at 5:06 PM Zbigniew Jędrzejewski-Szmek zbyszek@in.waw.pl wrote:
We should probably make it ^(/etc/|/bin/|/sbin/|/usr/bin/|/usr/sbin/). Split-usr distros are a blast from the past, but adding the extra paths wouldn't change anything for us, since we don't use them.
Please add /app/bin/ and /app/sbin/ as well to that list for flatpak rpm builds.
This shouldn't be needed because the buildsystem can happily pull filelists.xml. That said, this is part of the reason why we don't mandate a prefix in createrepo_c. We can't know what the prefix is going to be for RPMs for projects.
If this is a difficult topic because different distros have different expectations here, would it be possible to make this configurable (either at runtime or at build time)? That way createrepo_c wouldn't need to be opinionated ...
Fabio
On Mon, Mar 27, 2023 at 11:24 AM Fabio Valentini decathorpe@gmail.com wrote:
On Mon, Mar 27, 2023 at 2:10 PM Neal Gompa ngompa13@gmail.com wrote:
On Sun, Mar 26, 2023 at 12:27 PM Kalev Lember kalevlember@gmail.com wrote:
On Sun, Mar 26, 2023 at 5:06 PM Zbigniew Jędrzejewski-Szmek zbyszek@in.waw.pl wrote:
We should probably make it ^(/etc/|/bin/|/sbin/|/usr/bin/|/usr/sbin/). Split-usr distros are a blast from the past, but adding the extra paths wouldn't change anything for us, since we don't use them.
Please add /app/bin/ and /app/sbin/ as well to that list for flatpak rpm builds.
This shouldn't be needed because the buildsystem can happily pull filelists.xml. That said, this is part of the reason why we don't mandate a prefix in createrepo_c. We can't know what the prefix is going to be for RPMs for projects.
If this is a difficult topic because different distros have different expectations here, would it be possible to make this configurable (either at runtime or at build time)? That way createrepo_c wouldn't need to be opinionated ...
Just don't anchor the regex with "^" the beginning. Then it works for anything.
That is: "/bin/|/lib(|32|64|exec)/|/etc/"
-- 真実はいつも一つ!/ Always, there's only one truth!
On Mon, Mar 27, 2023 at 11:46:49AM -0400, Neal Gompa wrote:
On Mon, Mar 27, 2023 at 11:24 AM Fabio Valentini decathorpe@gmail.com wrote:
On Mon, Mar 27, 2023 at 2:10 PM Neal Gompa ngompa13@gmail.com wrote:
On Sun, Mar 26, 2023 at 12:27 PM Kalev Lember kalevlember@gmail.com wrote:
On Sun, Mar 26, 2023 at 5:06 PM Zbigniew Jędrzejewski-Szmek zbyszek@in.waw.pl wrote:
We should probably make it ^(/etc/|/bin/|/sbin/|/usr/bin/|/usr/sbin/). Split-usr distros are a blast from the past, but adding the extra paths wouldn't change anything for us, since we don't use them.
Please add /app/bin/ and /app/sbin/ as well to that list for flatpak rpm builds.
This shouldn't be needed because the buildsystem can happily pull filelists.xml. That said, this is part of the reason why we don't mandate a prefix in createrepo_c. We can't know what the prefix is going to be for RPMs for projects.
If this is a difficult topic because different distros have different expectations here, would it be possible to make this configurable (either at runtime or at build time)? That way createrepo_c wouldn't need to be opinionated ...
Just don't anchor the regex with "^" the beginning. Then it works for anything.
That is: "/bin/|/lib(|32|64|exec)/|/etc/"
I want to keep it clear that _my_ proposal is to make this narrower, to match the packaging guidelines. The guidelines say that packages SHOULD NOT use paths outside of /usr/bin|/usr/sbin|/etc, so any other paths that we put in primary just make things less efficient.
And as mentioned previously, adding libs would blow up the number of paths in primary, even though they are not useful because we have a dependency generator.
Making this configurable is probably easier than discussing the "correct" path endlessly. Then the setting for Fedora can easily be made to match the guidelines. What other distros do will be up to them.
Zbyszek
V Mon, Mar 27, 2023 at 11:46:49AM -0400, Neal Gompa napsal(a):
On Mon, Mar 27, 2023 at 11:24 AM Fabio Valentini decathorpe@gmail.com wrote:
On Mon, Mar 27, 2023 at 2:10 PM Neal Gompa ngompa13@gmail.com wrote:
On Sun, Mar 26, 2023 at 12:27 PM Kalev Lember kalevlember@gmail.com wrote:
On Sun, Mar 26, 2023 at 5:06 PM Zbigniew Jędrzejewski-Szmek zbyszek@in.waw.pl wrote:
We should probably make it ^(/etc/|/bin/|/sbin/|/usr/bin/|/usr/sbin/). Split-usr distros are a blast from the past, but adding the extra paths wouldn't change anything for us, since we don't use them.
Please add /app/bin/ and /app/sbin/ as well to that list for flatpak rpm builds.
This shouldn't be needed because the buildsystem can happily pull filelists.xml. That said, this is part of the reason why we don't mandate a prefix in createrepo_c. We can't know what the prefix is going to be for RPMs for projects.
If this is a difficult topic because different distros have different expectations here, would it be possible to make this configurable (either at runtime or at build time)? That way createrepo_c wouldn't need to be opinionated ...
Just don't anchor the regex with "^" the beginning. Then it works for anything.
That is: "/bin/|/lib(|32|64|exec)/|/etc/"
I don't think it's wise to match /usr/libexec which is defined as https://refspecs.linuxfoundation.org/FHS_3.0/fhs/ch04s07.html:
/usr/libexec includes internal binaries that are not intended to be executed directly by users or shell scripts.
-- Petr
Il 23/03/23 19:52, Ben Cotton ha scritto:
A quick look about this change impact on bodhi-server, as I understand from the relevant module at https://github.com/fedora-infra/bodhi/blob/develop/bodhi-server/bodhi/server... :
== Detailed Description ==
- Switch default compression from gz to zstd. DNF is able to
decompress zstd via libsolv since Fedora 30 and RHEL 8.4. It will still be possible to create gz compressed repodata by manually specifying gz compression.
As I understand, bodhi sets XZ compression on all repos but those with a release prefix set to 'FEDORA-EPEL' (EPEL7, EPEL8 and EPEL9) where it uses BZ2. Do we want to continue using XZ or switch to the new ZSTD default? Do we want to continue using BZ for EPEL9 or switch to ZSTD/XZ? (EPEL Next, both 8 and 9, are already using XZ if I understand correctly)
- Stop generating metadata in sqlite database format by default. It
will still be possible to create them using a --database switch or via sqliterepo_c tool. (It looks like the only user left is mdapi: https://github.com/fedora-infra/mdapi/issues/97).
I think this haven't got impact on bodhi-server.
- When adding groups.xml to repodata createrepo_c currently adds two
variants to repomd.xml. The specified file as is, uncompressed, with the type "group" and also a compressed variant with type "group_XX", where XX is compression suffix. This is atypical and unexpected. We propose to include just one variant of groups.xml using specified compression and repomd.xml type "group". This is not compatible with yum in RHEL 7. If required users will still be able to create repositories with the old layout using modifyrepo_c.
I'm not sure if this has impact on bodhi-server. We're composing repodata for EPEL7 and EPEL8 which still use yum, but I don't think we're adding groups metadata?
Mattia
- Stop generating metadata in sqlite database format by default. It
will still be possible to create them using a --database switch or via sqliterepo_c tool. (It looks like the only user left is mdapi: https://github.com/fedora-infra/mdapi/issues/97).
I think this haven't got impact on bodhi-server.
Correction: bodhi-server generates sqlite database format. Do we want to continue distributing that?
Mattia
On Fri, Mar 24, 2023 at 05:25:17PM +0000, Mattia Verga via devel wrote:
Il 23/03/23 19:52, Ben Cotton ha scritto:
A quick look about this change impact on bodhi-server, as I understand from the relevant module at https://github.com/fedora-infra/bodhi/blob/develop/bodhi-server/bodhi/server...
This is the part of bodhi that inserts metadata into the repodata. ie, 'this is a security update' etc. ie, updateinfo.xml Right?
== Detailed Description ==
- Switch default compression from gz to zstd. DNF is able to
decompress zstd via libsolv since Fedora 30 and RHEL 8.4. It will still be possible to create gz compressed repodata by manually specifying gz compression.
As I understand, bodhi sets XZ compression on all repos but those with a release prefix set to 'FEDORA-EPEL' (EPEL7, EPEL8 and EPEL9) where it uses BZ2. Do we want to continue using XZ or switch to the new ZSTD default? Do we want to continue using BZ for EPEL9 or switch to ZSTD/XZ? (EPEL Next, both 8 and 9, are already using XZ if I understand correctly)
Do note that bodhi just calls pungi to make the updates repos, so it's really about pungi config here right? Of course the questions are still needing answers if we want to change/adjust the pungi config.
This should be: https://pagure.io/fedora-infra/ansible/blob/main/f/roles/bodhi2/backend/temp... and https://pagure.io/fedora-infra/ansible/blob/main/f/roles/bodhi2/backend/temp... for modules.
So from that it looks like epel8 is xz specifically, but the rest are default.
- Stop generating metadata in sqlite database format by default. It
will still be possible to create them using a --database switch or via sqliterepo_c tool. (It looks like the only user left is mdapi: https://github.com/fedora-infra/mdapi/issues/97).
I think this haven't got impact on bodhi-server.
- When adding groups.xml to repodata createrepo_c currently adds two
variants to repomd.xml. The specified file as is, uncompressed, with the type "group" and also a compressed variant with type "group_XX", where XX is compression suffix. This is atypical and unexpected. We propose to include just one variant of groups.xml using specified compression and repomd.xml type "group". This is not compatible with yum in RHEL 7. If required users will still be able to create repositories with the old layout using modifyrepo_c.
I'm not sure if this has impact on bodhi-server. We're composing repodata for EPEL7 and EPEL8 which still use yum, but I don't think we're adding groups metadata?
We are I am pretty sure. ;) This is from fedora-comps, we gather that and pungi does some processing on it, but then inserts it in the end as group data.
I have a few additional questions:
* How does this impact zchunk? or does it?
* Kind of unrealted to this change, but I'll ask here as createrepo_c folks might know off hand: Is it possible to decouple drpms from a repo? ie, if we had a updates repo and a completely seperate updates-drpms repo with just drpms in it, what would it take to use that? Changes in dnf I guess? but also changes in createrepo_c to generate repodata for a repo that was only drpms?
Thanks for working on this!
kevin
Il 25/03/23 21:34, Kevin Fenzi ha scritto:
On Fri, Mar 24, 2023 at 05:25:17PM +0000, Mattia Verga via devel wrote:
Il 23/03/23 19:52, Ben Cotton ha scritto:
A quick look about this change impact on bodhi-server, as I understand from the relevant module at https://github.com/fedora-infra/bodhi/blob/develop/bodhi-server/bodhi/server...
This is the part of bodhi that inserts metadata into the repodata. ie, 'this is a security update' etc. ie, updateinfo.xml Right?
Ah, yes, it is relevant for updateinfo.xml only.
https://github.com/fedora-infra/bodhi/blob/1a2b192929fa1836ed04e7a0660c74365...
== Detailed Description ==
- Switch default compression from gz to zstd. DNF is able to
decompress zstd via libsolv since Fedora 30 and RHEL 8.4. It will still be possible to create gz compressed repodata by manually specifying gz compression.
As I understand, bodhi sets XZ compression on all repos but those with a release prefix set to 'FEDORA-EPEL' (EPEL7, EPEL8 and EPEL9) where it uses BZ2. Do we want to continue using XZ or switch to the new ZSTD default? Do we want to continue using BZ for EPEL9 or switch to ZSTD/XZ? (EPEL Next, both 8 and 9, are already using XZ if I understand correctly)
Do note that bodhi just calls pungi to make the updates repos, so it's really about pungi config here right? Of course the questions are still needing answers if we want to change/adjust the pungi config.
This should be: https://pagure.io/fedora-infra/ansible/blob/main/f/roles/bodhi2/backend/temp... and https://pagure.io/fedora-infra/ansible/blob/main/f/roles/bodhi2/backend/temp... for modules.
So from that it looks like epel8 is xz specifically, but the rest are default.
So, we're using different settings for pungi config and for updateinfo.xml generated by Bodhi.
I don't think there's a (easy) way to have bodhi inherit configuration from pungi config file.
I would first suggest to "fix" Bodhi metadata generation to use the same compression setting (which is now XZ) for everything except EPEL7 and EPEL8 (with the current conditional it's using BZ2 for EPEL9 also). Then we can discuss about switching from XZ to ZSTD, but I think the choice should be mirrored on pungi side.
This would be a kind of hard coded change in bodhi code. If there's a rationale we could modify bodhi to have those parameters set in database for each Release object, but does it worth the work?
- Stop generating metadata in sqlite database format by default. It
will still be possible to create them using a --database switch or via sqliterepo_c tool. (It looks like the only user left is mdapi: https://github.com/fedora-infra/mdapi/issues/97).
I think this haven't got impact on bodhi-server.
- When adding groups.xml to repodata createrepo_c currently adds two
variants to repomd.xml. The specified file as is, uncompressed, with the type "group" and also a compressed variant with type "group_XX", where XX is compression suffix. This is atypical and unexpected. We propose to include just one variant of groups.xml using specified compression and repomd.xml type "group". This is not compatible with yum in RHEL 7. If required users will still be able to create repositories with the old layout using modifyrepo_c.
I'm not sure if this has impact on bodhi-server. We're composing repodata for EPEL7 and EPEL8 which still use yum, but I don't think we're adding groups metadata?
We are I am pretty sure. ;) This is from fedora-comps, we gather that and pungi does some processing on it, but then inserts it in the end as group data.
I have a few additional questions:
- How does this impact zchunk? or does it?
AFAIK, zchunck is not enabled for EPEL, so it should not have impact.
Mattia
On Sun, Mar 26, 2023 at 04:54:37PM +0000, Mattia Verga via devel wrote:
So, we're using different settings for pungi config and for updateinfo.xml generated by Bodhi.
I don't think there's a (easy) way to have bodhi inherit configuration from pungi config file.
I would first suggest to "fix" Bodhi metadata generation to use the same compression setting (which is now XZ) for everything except EPEL7 and EPEL8 (with the current conditional it's using BZ2 for EPEL9 also). Then we can discuss about switching from XZ to ZSTD, but I think the choice should be mirrored on pungi side.
Sounds reasonable.
This would be a kind of hard coded change in bodhi code. If there's a rationale we could modify bodhi to have those parameters set in database for each Release object, but does it worth the work?
I'm not sure. I suppose down the road fedora 45 or something might change again, but we could deal with the hard coded change then, or implement the release object thing.
kevin
On Fri, Mar 24, 2023 at 05:25:17PM +0000, Mattia Verga via devel wrote: I have a few additional questions:
- How does this impact zchunk? or does it?
I also think there shouldn't be any impact on zchunk.
- Kind of unrealted to this change, but I'll ask here as createrepo_c
folks might know off hand: Is it possible to decouple drpms from a repo? ie, if we had a updates repo and a completely seperate updates-drpms repo with just drpms in it, what would it take to use that? Changes in dnf I guess? but also changes in createrepo_c to generate repodata for a repo that was only drpms?
Such repos can be created by createrepo_c and modifyrepo_c but it is a bit awkward. Basically first create normal (coupled) repo and then extract the prestodelta.xml and drpms into a new repo. Additionally createrepo_c API could be used to make it more straight forward. However you are right dnf can't handle that. It looks for deltas in the same repo where it finds the normal update package. It would require changes in dnf and libdnf.
On Mon, Mar 27, 2023 at 11:57:04AM -0000, Aleš Matěj wrote:
On Fri, Mar 24, 2023 at 05:25:17PM +0000, Mattia Verga via devel wrote: I have a few additional questions:
- How does this impact zchunk? or does it?
I also think there shouldn't be any impact on zchunk.
ok. Great to hear.
- Kind of unrealted to this change, but I'll ask here as createrepo_c
folks might know off hand: Is it possible to decouple drpms from a repo? ie, if we had a updates repo and a completely seperate updates-drpms repo with just drpms in it, what would it take to use that? Changes in dnf I guess? but also changes in createrepo_c to generate repodata for a repo that was only drpms?
Such repos can be created by createrepo_c and modifyrepo_c but it is a bit awkward. Basically first create normal (coupled) repo and then extract the prestodelta.xml and drpms into a new repo. Additionally createrepo_c API could be used to make it more straight forward.
Well, I was thinking of a completely disconnected process. ie, the drpms are created compltely seperately from the main repo and a prestodelta.xml is created there in the seperate repo.
However you are right dnf can't handle that. It looks for deltas in the same repo where it finds the normal update package. It would require changes in dnf and libdnf.
ok. Thanks for the info.
If we want to bring drpms back to useful, I think this would probibly be the best way to go. Then we could have some app that creates them async and dnf could use them if available.
kevin
On Mon, 27 Mar 2023 at 10:29, Kevin Fenzi kevin@scrye.com wrote:
On Mon, Mar 27, 2023 at 11:57:04AM -0000, Aleš Matěj wrote:
On Fri, Mar 24, 2023 at 05:25:17PM +0000, Mattia Verga via devel wrote: I have a few additional questions:
- How does this impact zchunk? or does it?
I also think there shouldn't be any impact on zchunk.
ok. Great to hear.
- Kind of unrealted to this change, but I'll ask here as createrepo_c
folks might know off hand: Is it possible to decouple drpms from a
repo?
ie, if we had a updates repo and a completely seperate updates-drpms repo with just drpms in it, what would it take to use that? Changes in dnf I guess? but also changes in createrepo_c to generate repodata for
a
repo that was only drpms?
Such repos can be created by createrepo_c and modifyrepo_c but it is a
bit awkward.
Basically first create normal (coupled) repo and then extract the
prestodelta.xml and drpms
into a new repo. Additionally createrepo_c API could be used to make it
more straight forward.
Well, I was thinking of a completely disconnected process. ie, the drpms are created compltely seperately from the main repo and a prestodelta.xml is created there in the seperate repo.
Or hear me out here for this wild thought: We stop offering drpms.
On Mon, Mar 27, 2023 at 10:32:13AM -0400, Stephen Smoogen wrote:
Or hear me out here for this wild thought: We stop offering drpms.
yes, yes, I just wanted to know if there was a plan forward for people interested in doing the work to support them. I am not committing to doing that work. ;)
kevin
On Mon Mar 27, 2023 at 07:27 -0700, Kevin Fenzi wrote:
On Mon, Mar 27, 2023 at 11:57:04AM -0000, Aleš Matěj wrote:
<snip>
However you are right dnf can't handle that. It looks for deltas in the same repo where it finds the normal update package. It would require changes in dnf and libdnf.
ok. Thanks for the info.
If we want to bring drpms back to useful, I think this would probibly be the best way to go. Then we could have some app that creates them async and dnf could use them if available.
kevin
Can you or someone else more familiar with DRPMs than I file a dnf5 RFE?
https://github.com/rpm-software-management/dnf5/issues/new
-- Maxwell G (@gotmax23) Pronouns: He/They
On Mon, Mar 27, 2023 at 03:19:00PM +0000, Maxwell G wrote:
On Mon Mar 27, 2023 at 07:27 -0700, Kevin Fenzi wrote:
On Mon, Mar 27, 2023 at 11:57:04AM -0000, Aleš Matěj wrote:
<snip> > > However you are right dnf can't handle that. It looks for deltas in the same repo where it finds > > the normal update package. It would require changes in dnf and libdnf. > > ok. Thanks for the info. > > If we want to bring drpms back to useful, I think this would probibly be the > best way to go. Then we could have some app that creates them async and > dnf could use them if available. > > kevin
Can you or someone else more familiar with DRPMs than I file a dnf5 RFE?
Well, I don't know that I have any time to drive this, and I am not sure if it should be on dnf5 or libdnf or librepo or all of them?
Anyhow, filed https://github.com/rpm-software-management/dnf5/issues/408
kevin
- When adding groups.xml to repodata createrepo_c currently adds two
variants to repomd.xml. The specified file as is, uncompressed, with the type "group" and also a compressed variant with type "group_XX", where XX is compression suffix. This is atypical and unexpected. We propose to include just one variant of groups.xml using specified compression and repomd.xml type "group". This is not compatible with yum in RHEL 7. If required users will still be able to create repositories with the old layout using modifyrepo_c. Further information: https://bugzilla.redhat.com/show_bug.cgi?id=2056318
If we're planning to drop by-default compatibility with EL7 anyway and bump to 1.0, we could go a little bit further.
* "createrepo_c" doesn't need to support sqlite metadata generation at all, so long as the repository producer can run "sqliterepo_c". If I recall correctly, yum on EL7 doesn't actually need the sqlite metadata to be generated in any case, it can produce that metadata itself so long as it has the XML. Making it available only avoids a local processing step. But if "modifyrepo_c" will be necessary to work with most EL7 repos, then making "sqliterepo_c" necessary for a minor optimization isn't a big leap.
* createrepo_c currently has two different switches for compression types, "compress-type" and "general-compress-type", where the latter applies to everything and the former only applies to files that are *not* "primary", "filelists", and "other". there should be one compression type option and it should be applied uniformly to all metadata produced by the "createrepo_c" tool. Yum would have choked on this because of the special handling around "comps", but I believe it should no longer be a concern?
If we're planning to drop by-default compatibility with EL7 anyway and bump to 1.0, we could go a little bit further.
- "createrepo_c" doesn't need to support sqlite metadata generation at all,
so long as the repository producer can run "sqliterepo_c". If I recall correctly, yum on EL7 doesn't actually need the sqlite metadata to be generated in any case, it can produce that metadata itself so long as it has the XML. Making it available only avoids a local processing step. But if "modifyrepo_c" will be necessary to work with most EL7 repos, then making "sqliterepo_c" necessary for a minor optimization isn't a big leap.
We were also discussing adding some "mode" options that could be set to EL7 to generate compatible metadata for EL7 with just createrepo_c.
- createrepo_c currently has two different switches for compression types,
"compress-type" and "general-compress-type", where the latter applies to everything and the former only applies to files that are *not* "primary", "filelists", and "other". there should be one compression type option and it should be applied uniformly to all metadata produced by the "createrepo_c" tool. Yum would have choked on this because of the special handling around "comps", but I believe it should no longer be a concern?
Yes that is a good suggestion. There are also other command line options that could use some (re)work. For example the unique --xz option which bypasses the mentioned compression options. Unfortunately I am not sure if we will be able to prioritize all the improvements.