I was asked to weigh in on https://pagure.io/releng/issue/7215 as a priority. Last time we talked about this we didn't really get anywhere...
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org/...
... and that ticket hasn't moved, because fixing it isn't trivial.
What we're doing now — as has been the case for several years, already noted in the previous discussion — has very little end-user value. Also as noted in that thread (as in the ticket)... that's unfortunate, because it did bring some real benefits (and could possibly do even more.)
But, I think it's time to move on. We have ostree and various container-delta approaches. We should focus on those — and give DeltaRPMs a sad, fond farewell.
On Tue, Feb 21, 2023 at 9:48 PM Matthew Miller mattdm@fedoraproject.org wrote:
I was asked to weigh in on https://pagure.io/releng/issue/7215 as a priority. Last time we talked about this we didn't really get anywhere...
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org/...
... and that ticket hasn't moved, because fixing it isn't trivial.
What we're doing now — as has been the case for several years, already noted in the previous discussion — has very little end-user value. Also as noted in that thread (as in the ticket)... that's unfortunate, because it did bring some real benefits (and could possibly do even more.)
But, I think it's time to move on. We have ostree and various container-delta approaches. We should focus on those — and give DeltaRPMs a sad, fond farewell.
It's been a long, long, time since any dnf transaction I ran printed positive news about deltarpms ... most of the time there either aren't any, or using them actually increased data download.
So, I agree. It's time to pull the plug.
Fabio
On 2/21/23 15:53, Fabio Valentini wrote:
It's been a long, long, time since any dnf transaction I ran printed positive news about deltarpms ... most of the time there either aren't any, or using them actually increased data download.
A recent transaction went so poorly that it prompted me to disable deltarpms completely on my system.
Got it.
Leslie Satenstein
On Tuesday, February 21, 2023 at 03:48:23 p.m. EST, Matthew Miller mattdm@fedoraproject.org wrote:
I was asked to weigh in on https://pagure.io/releng/issue/7215 as a priority. Last time we talked about this we didn't really get anywhere...
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org/...
... and that ticket hasn't moved, because fixing it isn't trivial.
What we're doing now — as has been the case for several years, already noted in the previous discussion — has very little end-user value. Also as noted in that thread (as in the ticket)... that's unfortunate, because it did bring some real benefits (and could possibly do even more.)
But, I think it's time to move on. We have ostree and various container-delta approaches. We should focus on those — and give DeltaRPMs a sad, fond farewell.
On Tue, 21 Feb 2023 at 15:48, Matthew Miller mattdm@fedoraproject.org wrote:
I was asked to weigh in on https://pagure.io/releng/issue/7215 as a priority. Last time we talked about this we didn't really get anywhere...
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org/...
... and that ticket hasn't moved, because fixing it isn't trivial.
So do we kill it for:
F39/F40 and beyond? F38 and beyond? X-date and all releases?
What we're doing now — as has been the case for several years, already noted in the previous discussion — has very little end-user value. Also as noted in that thread (as in the ticket)... that's unfortunate, because it did bring some real benefits (and could possibly do even more.)
But, I think it's time to move on. We have ostree and various container-delta approaches. We should focus on those — and give DeltaRPMs a sad, fond farewell.
-- Matthew Miller mattdm@fedoraproject.org Fedora Project Leader _______________________________________________ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-leave@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org Do not reply to spam, report it: https://pagure.io/fedora-infrastructure/new_issue
On Tue, Feb 21, 2023 at 04:44:31PM -0500, Stephen Smoogen wrote:
So do we kill it for:
F39/F40 and beyond? F38 and beyond? X-date and all releases?
My normal response would be... well, I missed the F38 change deadline by a wide margin, so F39+.
But, I think we could stop producing deltarpms for current releases without really affecting those releases (there would just not be more created, which as previously explored, would not really have a strong effect). So... I wouldn't leave the other options out of the question.
What do infra / releng folks think?
How easy is it to shut things off?
If shut off, can it be turned back on again (in case of Regrets)?
Once shut off, is decomissioning infrastructure around it a Project, or just more shutting off?
What risks are there?
And... how much would be saved in:
* compute resources * storage * delays * ongoing maintenance work * other?
On 2/21/23 17:13, Matthew Miller wrote:
On Tue, Feb 21, 2023 at 04:44:31PM -0500, Stephen Smoogen wrote:
So do we kill it for:
F39/F40 and beyond? F38 and beyond? X-date and all releases?
My normal response would be... well, I missed the F38 change deadline by a wide margin, so F39+.
But, I think we could stop producing deltarpms for current releases without really affecting those releases (there would just not be more created, which as previously explored, would not really have a strong effect). So... I wouldn't leave the other options out of the question.
Can we also disable deltarpms in the F38 repo files? It’s a 1-line change, trivially revertable, and it measurably reduces the attack surface of DNF. If no deltarpms are being generated, there is no point in DNF looking for them.
On Tue, Feb 21, 2023 at 05:13:26PM -0500, Matthew Miller wrote:
On Tue, Feb 21, 2023 at 04:44:31PM -0500, Stephen Smoogen wrote:
So do we kill it for:
F39/F40 and beyond? F38 and beyond? X-date and all releases?
My normal response would be... well, I missed the F38 change deadline by a wide margin, so F39+.
Just FYI, we do not produce drpms for rawhide/branched at all (since 2017 ish).
But, I think we could stop producing deltarpms for current releases without really affecting those releases (there would just not be more created, which as previously explored, would not really have a strong effect). So... I wouldn't leave the other options out of the question.
What do infra / releng folks think?
How easy is it to shut things off?
Its one line in bodhi pungi config: createrepo_deltas = False
If shut off, can it be turned back on again (in case of Regrets)?
Just change the one line back to True (well, it's more complex because we only actually do drpms for the 'Everything' repo, not others, but it's one line).
Once shut off, is decomissioning infrastructure around it a Project, or just more shutting off?
As far as I can think, nothing else needs to be done. They will just disappear.
What risks are there?
None that leap to mind.
And... how much would be saved in:
- compute resources
Some, but not a lot, as we only delta against the previous update composes and thus don't do too many.
- storage
100's of MB... not much at all.
- delays
- ongoing maintenance work
- other?
Nothing comes to mind.
kevin
On Wed, Feb 22, 2023 at 10:33:27AM -0800, Kevin Fenzi wrote:
Just FYI, we do not produce drpms for rawhide/branched at all (since 2017 ish).
[...]
Its one line in bodhi pungi config: createrepo_deltas = False
If shut off, can it be turned back on again (in case of Regrets)?
Just change the one line back to True (well, it's more complex because we only actually do drpms for the 'Everything' repo, not others, but it's one line).
So, it sounds like "remove the step in the release SOP to turn them _on_ for the branch at release time" would be the easiest way to go. And the default config could be changed in DNF at any time for F38+.
It doesn't sound like it's really worthwhile to bother changing F36 or F37.
On 2/23/23 12:52, Matthew Miller wrote:
On Wed, Feb 22, 2023 at 10:33:27AM -0800, Kevin Fenzi wrote:
Just FYI, we do not produce drpms for rawhide/branched at all (since 2017 ish).
[...]
Its one line in bodhi pungi config: createrepo_deltas = False
If shut off, can it be turned back on again (in case of Regrets)?
Just change the one line back to True (well, it's more complex because we only actually do drpms for the 'Everything' repo, not others, but it's one line).
So, it sounds like "remove the step in the release SOP to turn them _on_ for the branch at release time" would be the easiest way to go. And the default config could be changed in DNF at any time for F38+.
I would like to see the DNF config changed in F38, for attack surface reduction reasons.
On 2/21/23 16:44, Stephen Smoogen wrote:
On Tue, 21 Feb 2023 at 15:48, Matthew Miller mattdm@fedoraproject.org wrote:
I was asked to weigh in on https://pagure.io/releng/issue/7215 as a priority. Last time we talked about this we didn't really get anywhere...
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org/...
... and that ticket hasn't moved, because fixing it isn't trivial.
So do we kill it for:
F39/F40 and beyond? F38 and beyond? X-date and all releases?
F38+? Also maybe disable deltarpms in dnf.conf, to reduce attack surface.
On Tue, Feb 21, 2023 at 03:48:06PM -0500, Matthew Miller wrote:
I was asked to weigh in on https://pagure.io/releng/issue/7215 as a priority. Last time we talked about this we didn't really get anywhere...
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org/...
... and that ticket hasn't moved, because fixing it isn't trivial.
Yeah. ;(
I fear the only way to fix it is to get pungi to merge entire repos, and I don't think thats anything that pungi wants to be on the hook for doing.
What we're doing now — as has been the case for several years, already noted in the previous discussion — has very little end-user value. Also as noted in that thread (as in the ticket)... that's unfortunate, because it did bring some real benefits (and could possibly do even more.)
But, I think it's time to move on. We have ostree and various container-delta approaches. We should focus on those — and give DeltaRPMs a sad, fond farewell.
yeah, I agree.
kevin
Kevin Fenzi wrote:
I fear the only way to fix it is to get pungi to merge entire repos, and I don't think thats anything that pungi wants to be on the hook for doing.
But there are more things than just DeltaRPMs it could be useful for. E.g., I remember that in ancient times (pre Core-Extras Merge), some Fedora repository (IIRC, the old Fedora Extras) used to ship not only the latest build for each package, but the TWO latest builds, so that if the latest build was broken, you could easily downgrade to the previous one. That should really be done in all the rolling repositories (updates, updates- testing, Rawhide).
Kevin Kofler
On Wed, Feb 22, 2023 at 05:38:23AM +0100, Kevin Kofler via devel wrote:
Kevin Fenzi wrote:
I fear the only way to fix it is to get pungi to merge entire repos, and I don't think thats anything that pungi wants to be on the hook for doing.
But there are more things than just DeltaRPMs it could be useful for. E.g., I remember that in ancient times (pre Core-Extras Merge), some Fedora repository (IIRC, the old Fedora Extras) used to ship not only the latest build for each package, but the TWO latest builds, so that if the latest build was broken, you could easily downgrade to the previous one. That should really be done in all the rolling repositories (updates, updates- testing, Rawhide).
It does have advantages for sure.
Pros: * can dnf downgrade easily. * can choose not to upgrade something thats a big change you aren't ready for.
Cons: * It takes a bunch more mirror space. * It would make composes take longer. * It will allow for more easily tricking people into downgrading to a version that has serious security problems so they could be exploited.
kevin
Once upon a time, Kevin Fenzi kevin@scrye.com said:
Pros:
- can dnf downgrade easily.
This assumes the N-1 release is working though, which is not always valid when there's an issue (sometimes release N doesn't work, so there's release N+1 that still has a problem, but then release N-1 would be gone).
Cons:
Wouldn't it also increase the repodata size significantly, which is already a problem for dnf resource usage, especially on small devices like Raspberry Pi?
On Wed, Feb 22, 2023 at 12:51:54PM -0600, Chris Adams wrote:
Once upon a time, Kevin Fenzi kevin@scrye.com said:
Pros:
- can dnf downgrade easily.
This assumes the N-1 release is working though, which is not always valid when there's an issue (sometimes release N doesn't work, so there's release N+1 that still has a problem, but then release N-1 would be gone).
Right. Also, for things that incompatibly upgrade your data, rolling back will not be as easy as a dnf downgrade. ;( Luckily those things are rare.
Cons:
Wouldn't it also increase the repodata size significantly, which is already a problem for dnf resource usage, especially on small devices like Raspberry Pi?
Yep. Thats another con.
kevin
On Wed, Feb 22, 2023 at 12:49 PM Kevin Fenzi kevin@scrye.com wrote:
It does have advantages for sure.
Pros:
- can dnf downgrade easily.
- can choose not to upgrade something thats a big change you aren't
ready for.
I don't know enough about RPM packaging or DNF CoW to say either way, but I wonder if these enhancements might also contribute to getting DNF CoW's "reusable extents" feature working? If I understood the FOSDEM presentation correctly, DNF CoW would not only save the installer from having to rewrite the file to the user's disk on update when the file has not changed, it would also not download the file in the first place if it hasn't changed. Presumably this depends on the same functionality that deltarpm would need to identify unchanged files in RPM packages? So if these enhancements to deltarpm would also contribute to DNF CoW, I would consider that another plus.
Here is the DNF CoW FOSDEM presentation if anyone is interested in it. https://www.youtube.com/watch?v=-qYgkcs6Uxk
Just my two cents.
On 2023-02-22 10:48, Kevin Fenzi wrote:
On Wed, Feb 22, 2023 at 05:38:23AM +0100, Kevin Kofler via devel wrote:
I remember that in ancient times (pre Core-Extras Merge), some Fedora repository (IIRC, the old Fedora Extras) used to ship not only the latest build for each package, but the TWO latest builds,
Cons:
- It will allow for more easily tricking people into downgrading to a
version that has serious security problems so they could be exploited.
Contrary-wise: Because Fedora updates only contains the latest built, once a build marked as a security fix is obsoleted by another build, there is no longer any indication that a security issue existed in any version, at which point "dnf update --security" no longer works.
That might be a problem only for systems that are updated less frequently than the window between a security update and a later build, I still think it's a flaw that should be fixed.
On 2023-02-23 10:05, Gordon Messmer wrote:
Contrary-wise: Because Fedora updates only contains the latest built, once a build marked as a security fix is obsoleted by another build, there is no longer any indication that a security issue existed in any version, at which point "dnf update --security" no longer works.
For example, https://bodhi.fedoraproject.org/updates/FEDORA-2022-839fd408a5 is no longer an indication of a problem in a default package:
$ podman run --rm -it fedora:37 [root@d1c2aa7da870 /]# rpm -qa vim* vim-data-9.0.475-1.fc37.noarch vim-minimal-9.0.475-1.fc37.x86_64 [root@d1c2aa7da870 /]# dnf update --security vim* No security updates needed for "vim*", but 2 updates available Dependencies resolved. Nothing to do. Complete!
That might be a problem only for systems that are updated less frequently than the window between a security update and a later build, I still think it's a flaw that should be fixed.
(And I probably shouldn't have phrased this as if it's very limited. Anything installed from the installation media or "fedora" repo without full updates would definitely have security issues that weren't reflected in the package set selected by "dnf update --security")
On Thu, Feb 23, 2023 at 10:15:42AM -0800, Gordon Messmer wrote:
On 2023-02-23 10:05, Gordon Messmer wrote:
Contrary-wise: Because Fedora updates only contains the latest built, once a build marked as a security fix is obsoleted by another build, there is no longer any indication that a security issue existed in any version, at which point "dnf update --security" no longer works.
For example, https://bodhi.fedoraproject.org/updates/FEDORA-2022-839fd408a5 is no longer an indication of a problem in a default package:
$ podman run --rm -it fedora:37 [root@d1c2aa7da870 /]# rpm -qa vim* vim-data-9.0.475-1.fc37.noarch vim-minimal-9.0.475-1.fc37.x86_64 [root@d1c2aa7da870 /]# dnf update --security vim* No security updates needed for "vim*", but 2 updates available Dependencies resolved. Nothing to do. Complete!
That might be a problem only for systems that are updated less frequently than the window between a security update and a later build, I still think it's a flaw that should be fixed.
(And I probably shouldn't have phrased this as if it's very limited. Anything installed from the installation media or "fedora" repo without full updates would definitely have security issues that weren't reflected in the package set selected by "dnf update --security")
For this reason, bodhi used to mark such packages for the rest of the release. Ie, if you mark foo-1.0-1.fc37 a security update, forever after that foo package gets 'security' in the updateinfo. I think this was dropped because it confused too many people and it also didn't really express the actual problem here.
I'm not sure what a solution could be. Keep every update in updateinfo so dnf could tell you that there's 2 updates and 1 is security and the other bugfix? but then we would need to also keep those updates around to update to.
kevin
Kevin Fenzi wrote:
I'm not sure what a solution could be. Keep every update in updateinfo so dnf could tell you that there's 2 updates and 1 is security and the other bugfix? but then we would need to also keep those updates around to update to.
Add a repodata field "last security update EVR" that would be filled in by Bodhi for any non-security update. Then DNF would just need to check whether the installed EVR is less than the value of that field and treat the update as a security update in that case.
Kevin Kofler
On Fri, Feb 24, 2023 at 06:03:39AM +0100, Kevin Kofler via devel wrote:
Kevin Fenzi wrote:
I'm not sure what a solution could be. Keep every update in updateinfo so dnf could tell you that there's 2 updates and 1 is security and the other bugfix? but then we would need to also keep those updates around to update to.
Add a repodata field "last security update EVR" that would be filled in by Bodhi for any non-security update. Then DNF would just need to check whether the installed EVR is less than the value of that field and treat the update as a security update in that case.
That might be workable. Someone willing to file a bodhi RFE/PR? ;)
Then we would need buyin from dnf folks...
kevin
Are you saying that DNF does an exact version match instead of making the assumption that packages with version >= X contain a fix for a security bug which the updateinfo declares to be fixed in X? Or that the updateinfo itself gets purged of advisories that don't apply to the latest versions available.
On Fri, Feb 24, 2023 at 05:56:01AM -0000, Daniel Alley wrote:
Are you saying that DNF does an exact version match instead of making the assumption that packages with version >= X contain a fix for a security bug which the updateinfo declares to be fixed in X? Or that the updateinfo itself gets purged of advisories that don't apply to the latest versions available.
updateinfo is created by bodhi on every push with the current data.
So consider:
You have foo-1.0-1.fc37 in the base repo foo-1.1-1.fc37 comes out as an update and it fixes a security bug. later foo-1.2-1.fc37 comes out and it's an enhancement.
Users that updated to 1.1-1.fc37 will just see the enhancement update.
Users that just installed or haven't updated to 1.1-1.fc37 will see just 'an enhancement update to 1.2-1.fc37' and --security will not update the package.
kevin
Gordon Messmer wrote:
Contrary-wise: Because Fedora updates only contains the latest built, once a build marked as a security fix is obsoleted by another build, there is no longer any indication that a security issue existed in any version, at which point "dnf update --security" no longer works.
There are also other dangers with installing only security fixes. If a bugfix is released and packaged, and later it's discovered that the bug had security implications, then no security update will be made because the fix is already packaged. It might be possible to set a security flag on the update after the fact, but nobody will bother with that.
I would therefore advise against using --security. If one can't install all the updates continuously, then one should use a more stable distribution than Fedora.
Björn Persson
On Fri, Feb 24 2023 at 12:00:40 AM +0100, Björn Persson Bjorn@xn--rombobjrn-67a.se wrote:
There are also other dangers with installing only security fixes. If a bugfix is released and packaged, and later it's discovered that the bug had security implications, then no security update will be made because the fix is already packaged. It might be possible to set a security flag on the update after the fact, but nobody will bother with that.
I would therefore advise against using --security. If one can't install all the updates continuously, then one should use a more stable distribution than Fedora.
tbh I'd go so far as to propose that this functionality should not be reimplemented in dnf5 at all.
Michael Catanzaro wrote:
On Fri, Feb 24 2023 at 12:00:40 AM +0100, Björn Persson Bjorn@xn--rombobjrn-67a.se wrote:
There are also other dangers with installing only security fixes. If a bugfix is released and packaged, and later it's discovered that the bug had security implications, then no security update will be made because the fix is already packaged. It might be possible to set a security flag on the update after the fact, but nobody will bother with that.
I would therefore advise against using --security. If one can't install all the updates continuously, then one should use a more stable distribution than Fedora.
tbh I'd go so far as to propose that this functionality should not be reimplemented in dnf5 at all.
I would personally not miss --security, but what is really useful is the --advisory=… flag that used to be implemented by the same plugin. (They are now both built in in DNF.) That is invaluable to test individual updates from updates-testing. I would consider the absence of --advisory=… a dealbreaker.
Kevin Kofler
Kevin Kofler via devel wrote:
I would personally not miss --security, but what is really useful is the --advisory=… flag that used to be implemented by the same plugin. (They are now both built in in DNF.) That is invaluable to test individual updates from updates-testing. I would consider the absence of --advisory=… a dealbreaker.
PS: Note that --advisory=… actually works on *any* update, not just security updates. But the functionality is often lumped together with --security, which is why I am mentioning it here.
Kevin Kofler
On 2/23/23 8:04 PM, Michael Catanzaro wrote:
On Fri, Feb 24 2023 at 12:00:40 AM +0100, Björn Persson Bjorn@xn--rombobjrn-67a.se wrote:
There are also other dangers with installing only security fixes. If a bugfix is released and packaged, and later it's discovered that the bug had security implications, then no security update will be made because the fix is already packaged. It might be possible to set a security flag on the update after the fact, but nobody will bother with that.
I would therefore advise against using --security. If one can't install all the updates continuously, then one should use a more stable distribution than Fedora.
tbh I'd go so far as to propose that this functionality should not be reimplemented in dnf5 at all.
Does DNF on RHEL for example do something different when --security is involved? Because the RHEL documentation talks about it as a feature to use. Is a lack of metadata for previous updates the problem or the implementation?
devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-leave@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org Do not reply to spam, report it: https://pagure.io/fedora-infrastructure/new_issue
Once upon a time, Robert Marcano via devel devel@lists.fedoraproject.org said:
Does DNF on RHEL for example do something different when --security is involved? Because the RHEL documentation talks about it as a feature to use. Is a lack of metadata for previous updates the problem or the implementation?
Just a guess, but... updates in RHEL are different from updates in Fedora because of policy. In RHEL, updates outside of a point release are much more targeted - mostly security and significant bug fixes. Since there are fewer updates, the security updates stick around for a while and stand out more.
In Fedora, essentially anything can be updated at any time for any reason, whenever the packager(s) want. It could be a minor bugfix, a new upstream release, etc. So the update "churn" tends to be higher. There could be a security update today to a package (maybe just by applying a quick patch), and then maybe upstream incorporates the patch next week (along with other changes) and the Fedora packager updates to that release. From the Fedora point of view, the second new package is not addressing any security issue, because the first new package did.
Neither are wrong, they're just different polices.
On 2/24/23 12:01 PM, Chris Adams wrote:
Once upon a time, Robert Marcano via devel devel@lists.fedoraproject.org said:
Does DNF on RHEL for example do something different when --security is involved? Because the RHEL documentation talks about it as a feature to use. Is a lack of metadata for previous updates the problem or the implementation?
Just a guess, but... updates in RHEL are different from updates in Fedora because of policy. In RHEL, updates outside of a point release are much more targeted - mostly security and significant bug fixes. Since there are fewer updates, the security updates stick around for a while and stand out more.
In Fedora, essentially anything can be updated at any time for any reason, whenever the packager(s) want. It could be a minor bugfix, a new upstream release, etc. So the update "churn" tends to be higher. There could be a security update today to a package (maybe just by applying a quick patch), and then maybe upstream incorporates the patch next week (along with other changes) and the Fedora packager updates to that release. From the Fedora point of view, the second new package is not addressing any security issue, because the first new package did.
Neither are wrong, they're just different polices.
Right, but does a security update replaced by a non security update will hide the first security update on RHEL like happens on Fedora?
Because if the problem is how DNF process --security and not how Fedora and RHEL push security updates metadata, The Red Hat documenting how to use dnf-automatic to only install security updates is probably not at all secure. Just wondering where is the problem, metadata or implementation.
V Fri, Feb 24, 2023 at 12:08:28PM -0400, Robert Marcano via devel napsal(a):
On 2/24/23 12:01 PM, Chris Adams wrote:
Once upon a time, Robert Marcano via devel devel@lists.fedoraproject.org said:
Does DNF on RHEL for example do something different when --security is involved? Because the RHEL documentation talks about it as a feature to use. Is a lack of metadata for previous updates the problem or the implementation?
Just a guess, but... updates in RHEL are different from updates in Fedora because of policy. In RHEL, updates outside of a point release are much more targeted - mostly security and significant bug fixes. Since there are fewer updates, the security updates stick around for a while and stand out more.
In Fedora, essentially anything can be updated at any time for any reason, whenever the packager(s) want. It could be a minor bugfix, a new upstream release, etc. So the update "churn" tends to be higher. There could be a security update today to a package (maybe just by applying a quick patch), and then maybe upstream incorporates the patch next week (along with other changes) and the Fedora packager updates to that release. From the Fedora point of view, the second new package is not addressing any security issue, because the first new package did.
Neither are wrong, they're just different polices.
Right, but does a security update replaced by a non security update will hide the first security update on RHEL like happens on Fedora?
Because if the problem is how DNF process --security and not how Fedora and RHEL push security updates metadata, The Red Hat documenting how to use dnf-automatic to only install security updates is probably not at all secure. Just wondering where is the problem, metadata or implementation.
I think that DNF works same both in Fedora dnf in RHEL. The main difference is that RHEL repositories contain all historical updates. Therefore DNF can see security updateinfo data even for packages whose latest update is a non-security.
-- Petr
I am not sure whether by "all historical updates" you are only referring to all updates being listed in updateinfo.xml, or all history generally (including old packages). But in the latter case, note that keeping all updates massively inflates the storage requirements for maintaining a copy of the repo, which many (or even most) corporate users do. This is not a huge problem, generally, but it's also not ideal, and probably isn't the right tradeoff for Fedora.
Here[0] for example is RHEL 8 baseos and appstream, for which the difference between storing "only the latest package" and "all the packages listed" is 20x and 10x, respectively. Metadata size would likewise be larger, meaning DNF clients have more to download.
[0]
[dalley@thinkpad repos]$ rpmrepo details el8-baseos ... Number of packages: 12910 Number of unique packages (latest version): 1798 Number of packages (latest 3 versions): 4459 Packages total size: 23.82 GB Packages total size (latest version): 1.4 GB Packages total size (latest 3 versions): 4.03 GB
[dalley@thinkpad repos]$ rpmrepo details el8-appstream ... Number of packages: 29103 Number of unique packages (latest version): 5902 Number of packages (latest 3 versions): 12988 Packages total size: 92.91 GB Packages total size (latest version): 9.12 GB Packages total size (latest 3 versions): 23.91 GB
V Tue, Feb 28, 2023 at 02:47:03AM -0000, Daniel Alley napsal(a):
I am not sure whether by "all historical updates" you are only referring to all updates being listed in updateinfo.xml, or all history generally (including old packages).
The latter.
But in the latter case, note that keeping all updates massively inflates the storage requirements for maintaining a copy of the repo, which many (or even most) corporate users do. This is not a huge problem, generally, but it's also not ideal, and probably isn't the right tradeoff for Fedora.
I know. I only replied the question.
Here[0] for example is RHEL 8 baseos and appstream, for which the difference between storing "only the latest package" and "all the packages listed" is 20x and 10x, respectively. Metadata size would likewise be larger, meaning DNF clients have more to download.
I don't say Fedora needs to do it the same way. Fedora could only accumulate updateinfos while only retaining the latest package. Would it inflate metadata? Definitely. But if you want to deliver the data to the clients, you have to store them somewhere. Would that affect all clients? No. updateinfo.xml can only be downloaded by clients requesting that data. People doing "dnf upgrade" can safely skip it.
Or Fedora could reverse it: Fedora would run a network service which clients would send a list of installed packages and the service would return a list of affected packages. At the end, ostree od debuginfod services work like that.
-- Petr
Petr Pisar wrote:
Would that affect all clients? No. updateinfo.xml can only be downloaded by clients requesting that data. People doing "dnf upgrade" can safely skip it.
Keep in mind that updateinfo.xml is also used by GUI updaters such as dnfdragora or plasma-pk-updates to display details about the updates, not just for CLI filtering purposes (dnf --security, dnf --advisory=…).
Kevin Kofler
On 2/28/23 05:06, Petr Pisar wrote:
V Tue, Feb 28, 2023 at 02:47:03AM -0000, Daniel Alley napsal(a):
I am not sure whether by "all historical updates" you are only referring to all updates being listed in updateinfo.xml, or all history generally (including old packages).
The latter.
But in the latter case, note that keeping all updates massively inflates the storage requirements for maintaining a copy of the repo, which many (or even most) corporate users do. This is not a huge problem, generally, but it's also not ideal, and probably isn't the right tradeoff for Fedora.
I know. I only replied the question.
Here[0] for example is RHEL 8 baseos and appstream, for which the difference between storing "only the latest package" and "all the packages listed" is 20x and 10x, respectively. Metadata size would likewise be larger, meaning DNF clients have more to download.
I don't say Fedora needs to do it the same way. Fedora could only accumulate updateinfos while only retaining the latest package. Would it inflate metadata? Definitely. But if you want to deliver the data to the clients, you have to store them somewhere. Would that affect all clients? No. updateinfo.xml can only be downloaded by clients requesting that data. People doing "dnf upgrade" can safely skip it.
Or Fedora could reverse it: Fedora would run a network service which clients would send a list of installed packages and the service would return a list of affected packages. At the end, ostree od debuginfod services work like that.
debuginfod clients should be checking the downloaded data against a hash included in the binary being debugged.
On 2023-02-24 07:42, Robert Marcano via devel wrote:
Does DNF on RHEL for example do something different when --security is involved? Because the RHEL documentation talks about it as a feature to use. Is a lack of metadata for previous updates the problem or the implementation?
I don't have the log, but I checked this about a month ago:
I can set up an 8.3 system (I used a UBI container, to be specific) and subscribe to Red Hat's repositories. Since 8.3, there has been a security update to bash, but the most recent bash package is not a security fix. If I run |dnf update --security bash|, the system will offer the most recent bash package, even though it is not a security fix. Naturally, if I run |dnf update bash|, I get the same offer.
So on RHEL, dnf will evidently offer to update a package to the current version if any of the available update candidates are marked as a security update. And there are multiple update candidates in RHEL repositories, as well as CentOS Stream repositories, unlike Fedora.
So, from my point of view the biggest problem with "dnf update --security" on Fedora is that rpm doesn't track minor-version dependencies of libraries without versioned symbols, which means that "dnf update --security" can easily break the system by updating a leaf package but not updating dependencies that have added new interfaces that it requires. (But I'm working on fixing that.)
On 2/24/23 12:35 PM, Gordon Messmer wrote:
On 2023-02-24 07:42, Robert Marcano via devel wrote:
Does DNF on RHEL for example do something different when --security is involved? Because the RHEL documentation talks about it as a feature to use. Is a lack of metadata for previous updates the problem or the implementation?
I don't have the log, but I checked this about a month ago:
I can set up an 8.3 system (I used a UBI container, to be specific) and subscribe to Red Hat's repositories. Since 8.3, there has been a security update to bash, but the most recent bash package is not a security fix. If I run |dnf update --security bash|, the system will offer the most recent bash package, even though it is not a security fix. Naturally, if I run |dnf update bash|, I get the same offer.
So on RHEL, dnf will evidently offer to update a package to the current version if any of the available update candidates are marked as a security update. And there are multiple update candidates in RHEL repositories, as well as CentOS Stream repositories, unlike Fedora.
Thanks for testing it. So, if there is a desire to "fix" this in Fedora, having the repository includes the latest package marked as a security update and the latest one. I am not sure that will solve much because it still depends, as other posts said, Fedora has more open rules for updates, than enterprise distributions and therea are entire version jumps without tracking errata metadata.
So, from my point of view the biggest problem with "dnf update --security" on Fedora is that rpm doesn't track minor-version dependencies of libraries without versioned symbols, which means that "dnf update --security" can easily break the system by updating a leaf package but not updating dependencies that have added new interfaces that it requires. (But I'm working on fixing that.) _______________________________________________ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-leave@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org Do not reply to spam, report it: https://pagure.io/fedora-infrastructure/new_issue
On Fri, Feb 24 2023 at 11:42:17 AM -0400, Robert Marcano via devel devel@lists.fedoraproject.org wrote:
Does DNF on RHEL for example do something different when --security is involved? Because the RHEL documentation talks about it as a feature to use. Is a lack of metadata for previous updates the problem or the implementation?
So if you are missing a non-security update, then any security update built against it will break your system, exactly the same as in the scenarios proposed in the very first comment in this thread. A scenario like libsoup depending on an nghttp update could easily happen in your RHEL updates just the same as it could in Fedora.
Perhaps --security might work sometimes or even most of the time, but it cannot work safely in general.
Michael
On 2/23/23 18:00, Björn Persson wrote:
Gordon Messmer wrote:
Contrary-wise: Because Fedora updates only contains the latest built, once a build marked as a security fix is obsoleted by another build, there is no longer any indication that a security issue existed in any version, at which point "dnf update --security" no longer works.
There are also other dangers with installing only security fixes. If a bugfix is released and packaged, and later it's discovered that the bug had security implications, then no security update will be made because the fix is already packaged. It might be possible to set a security flag on the update after the fact, but nobody will bother with that.
I would therefore advise against using --security. If one can't install all the updates continuously, then one should use a more stable distribution than Fedora.
Björn Persson
I actually use --security for the *opposite* purpose: to get security updates from updates-testing. Only problem I can remember having is broken syntax highlighting from a somewhat recent vim update.
Dne 22. 02. 23 v 5:38 Kevin Kofler via devel napsal(a):
Kevin Fenzi wrote:
I fear the only way to fix it is to get pungi to merge entire repos, and I don't think thats anything that pungi wants to be on the hook for doing.
But there are more things than just DeltaRPMs it could be useful for. E.g., I remember that in ancient times (pre Core-Extras Merge), some Fedora repository (IIRC, the old Fedora Extras) used to ship not only the latest build for each package, but the TWO latest builds, so that if the latest build was broken, you could easily downgrade to the previous one. That should really be done in all the rolling repositories (updates, updates- testing, Rawhide).
Once there was this DNF RFE:
https://bugzilla.redhat.com/show_bug.cgi?id=1397174
Vít
Kevin Kofler
devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-leave@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org Do not reply to spam, report it: https://pagure.io/fedora-infrastructure/new_issue
On Thu, Feb 23, 2023 at 10:46:03AM +0100, Vít Ondruch wrote:
Dne 22. 02. 23 v 5:38 Kevin Kofler via devel napsal(a):
Kevin Fenzi wrote:
I fear the only way to fix it is to get pungi to merge entire repos, and I don't think thats anything that pungi wants to be on the hook for doing.
But there are more things than just DeltaRPMs it could be useful for. E.g., I remember that in ancient times (pre Core-Extras Merge), some Fedora repository (IIRC, the old Fedora Extras) used to ship not only the latest build for each package, but the TWO latest builds, so that if the latest build was broken, you could easily downgrade to the previous one. That should really be done in all the rolling repositories (updates, updates- testing, Rawhide).
Once there was this DNF RFE:
yeah. We actually do have a repo with all updates for stable releases, which we had to setup for a corner fedora coreos corner case. I suppose we could enable that more globally if the desire is for easy downgrades in stable releases. I think they would be much more common in rawhide/branched though.
kevin
It's also been a long time since I've seen any benefit.
Can anyone summarise quickly why delta RPMs don't work? It seems like they "obviously" should be possible, because there must be a lot of commonality in the content of adjacent RPM versions ...
Rich.
On Wed, Feb 22, 2023 at 3:37 AM Richard W.M. Jones rjones@redhat.com wrote:
It's also been a long time since I've seen any benefit.
Can anyone summarise quickly why delta RPMs don't work? It seems like they "obviously" should be possible, because there must be a lot of commonality in the content of adjacent RPM versions ...
They don't work because we compose updates wrong. Instead of building on top of previous updates, we throw them away and rebuild from the latest. Since we don't merge previous composes into a new compose, we are missing too much stuff for DeltaRPMs to be continuously useful.
On Wed, Feb 22, 2023 at 08:37:00AM +0000, Richard W.M. Jones wrote:
It's also been a long time since I've seen any benefit.
Can anyone summarise quickly why delta RPMs don't work? It seems like they "obviously" should be possible, because there must be a lot of commonality in the content of adjacent RPM versions ...
pungi creates composes. It does this by asking koji for 'whats the latest rpm tagged into this tag'. It then operates on those packages. It has little concept of other repos/things other than koji.
So, in order to do what might be expected it would have to:
* query koji/download things as it does now. * copy in the current repos * possibly prune based on something (only 2 copies, only X days old) * then go on with it's repo creation, etc.
Last I checked pungi developers weren't too excited by pungi growing that ability/code. I can try asking again.
kevin
Il 22/02/23 19:46, Kevin Fenzi ha scritto:
On Wed, Feb 22, 2023 at 08:37:00AM +0000, Richard W.M. Jones wrote:
It's also been a long time since I've seen any benefit.
Can anyone summarise quickly why delta RPMs don't work? It seems like they "obviously" should be possible, because there must be a lot of commonality in the content of adjacent RPM versions ...
pungi creates composes. It does this by asking koji for 'whats the latest rpm tagged into this tag'. It then operates on those packages. It has little concept of other repos/things other than koji.
So, in order to do what might be expected it would have to:
- query koji/download things as it does now.
- copy in the current repos
- possibly prune based on something (only 2 copies, only X days old)
- then go on with it's repo creation, etc.
So, does that mean that every time a compose is performed pungi downloads all RPMs tagged with a specific tag in a directory and then it operates on those to create repository metadata? Or it just downloads the rpms headers, or whatever?
Can we have a persistent compose root directory as base, then download the latest builds (if newer), do the pruning "based on something", create the repo and save the state as the base for the next run? Or will that eat way too much disk space? On the other hand we could save bandwith because we would not download all the rpms which were not updated from the previous run... what's more valuable for our infrastructure, disk space or network bandwith usage?
Mattia
Well, regarding the "based on something", you can hand off a list of packages to createrepo_c with --pkglist, and avoid the need to download files with --update + --skip-stat. Unfortunately that doesn't help you with the package file management. In a vacuum --baseurl would help here because you could have one root directory, however in reality it breaks repository mirroring because any mirror be telling clients to fetch the packages from the source-of-truth.
I'm not 100% sure how --basedir works, the description is a bit vague.
Another option is to use something like Pulp which stores all the information required for metadata generation inside Postgresql and thus can do so without ever touching the packages / headers again. That approach isn't necessarily free of downsides either, but it does abstract the whole file management problem.
On Wed, 22 Feb 2023 at 15:56, Daniel Alley dalley@redhat.com wrote:
Well, regarding the "based on something", you can hand off a list of packages to createrepo_c with --pkglist, and avoid the need to download files with --update + --skip-stat. Unfortunately that doesn't help you with the package file management. In a vacuum --baseurl would help here because you could have one root directory, however in reality it breaks repository mirroring because any mirror be telling clients to fetch the packages from the source-of-truth.
I'm not 100% sure how --basedir works, the description is a bit vague.
Another option is to use something like Pulp which stores all the information required for metadata generation inside Postgresql and thus can do so without ever touching the packages / headers again. That approach isn't necessarily free of downsides either, but it does abstract the whole file management problem.
I think we need to begin by figuring out what happens in at least the 'pungi' part of the daily 'let's make updates and rawhide'. There are a LOT of things which are going on which interrelate with each other and are prone to cascading breakage when something is 'added', 'removed', 'fixed', or 'changed'. There are also some hard resource limitations on how much CPU/disk/memory is available, how close things must be to 'work', and places where things break a lot but trying to remove/fix/change would require longer downtimes than the project has allowed in the past.
I say this from having done all of the above at one point or another and caused all kinds of chaos in doing so. I have probably used up more of Kevin's patience on those than was right.
On Wed, Feb 22, 2023 at 08:12:01PM +0000, Mattia Verga via devel wrote:
So, does that mean that every time a compose is performed pungi downloads all RPMs tagged with a specific tag in a directory and then it operates on those to create repository metadata? Or it just downloads the rpms headers, or whatever?
Yes. It downloads them all, because it needs the ones signed with the specific keys in it's config. It can optionally however look at a previous config and decide that nothing changed in koji so it can reuse them, but in practice I think this doesn't happen much.
Can we have a persistent compose root directory as base, then download the latest builds (if newer), do the pruning "based on something", create the repo and save the state as the base for the next run? Or will that eat way too much disk space? On the other hand we could save bandwith because we would not download all the rpms which were not updated from the previous run... what's more valuable for our infrastructure, disk space or network bandwith usage?
Definitely disk space is more important right now.
kevin
On Tue, Feb 21, 2023 at 8:48 PM Matthew Miller mattdm@fedoraproject.org wrote:
What we're doing now — as has been the case for several years, already noted in the previous discussion — has very little end-user value.
While occasionally I have seen a small decrease in the size of the files transferred (which certainly can benefit some people some of the time), the total elapsed time of the transaction has always ended up being higher as the recreation of the original rpm exceeds the time that it would have taken me to just download the full new rpm (with an admittedly reasonably high speed network provider in my environment).
However, that is an end user experience. What about the mirror providers? Even a small percentage of bandwidth savings might be useful for them (depending on their cost model) at the scale(s) they may be operating. Has anyone asked those providers, or do all (or most) now have a network cost structure such that it does not matter?
On Tue, Feb 21, 2023 at 9:48 PM Matthew Miller mattdm@fedoraproject.org wrote:
But, I think it's time to move on. We have ostree and various container-delta approaches. We should focus on those — and give DeltaRPMs a sad, fond farewell.
+1 from me. It will speed up the compose, and I haven't seen a positive impact from deltarpms in a long time. They are rarely available and the saved bandwidth is usually negligible (and reassembling of the rpm is usually quite slow).
Yeah I'd say +1 from here too. Was just wondering about this yesterday.
Le mer. 22 févr. 2023, 05 h 52, Kamil Paral kparal@redhat.com a écrit :
On Tue, Feb 21, 2023 at 9:48 PM Matthew Miller mattdm@fedoraproject.org wrote:
But, I think it's time to move on. We have ostree and various container-delta approaches. We should focus on those — and give DeltaRPMs a sad, fond farewell.
+1 from me. It will speed up the compose, and I haven't seen a positive impact from deltarpms in a long time. They are rarely available and the saved bandwidth is usually negligible (and reassembling of the rpm is usually quite slow).
devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-leave@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org Do not reply to spam, report it: https://pagure.io/fedora-infrastructure/new_issue
Dislike -1
On 2/22/23, Steve Cossette farchord@gmail.com wrote:
Yeah I'd say +1 from here too. Was just wondering about this yesterday.
Le mer. 22 févr. 2023, 05 h 52, Kamil Paral kparal@redhat.com a écrit :
On Tue, Feb 21, 2023 at 9:48 PM Matthew Miller mattdm@fedoraproject.org wrote:
But, I think it's time to move on. We have ostree and various container-delta approaches. We should focus on those — and give DeltaRPMs a sad, fond farewell.
+1 from me. It will speed up the compose, and I haven't seen a positive impact from deltarpms in a long time. They are rarely available and the saved bandwidth is usually negligible (and reassembling of the rpm is usually quite slow).
devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-leave@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org Do not reply to spam, report it: https://pagure.io/fedora-infrastructure/new_issue
On Wed, Feb 22, 2023 at 01:41:43PM +0100, Luna Jernberg wrote:
Dislike -1
This kind of "vote" isn't really very helpful. Of course not everyone is going to like everything. As I said initially, it's unfortunate that we aren't in a better place here. But we're in the place we're in. If you have a constructive, alternate proposal, let's hear that, please.
On Tue, Feb 21, 2023 at 3:48 PM Matthew Miller mattdm@fedoraproject.org wrote:
I was asked to weigh in on https://pagure.io/releng/issue/7215 as a priority. Last time we talked about this we didn't really get anywhere...
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org/...
... and that ticket hasn't moved, because fixing it isn't trivial.
What we're doing now — as has been the case for several years, already noted in the previous discussion — has very little end-user value. Also as noted in that thread (as in the ticket)... that's unfortunate, because it did bring some real benefits (and could possibly do even more.)
Our tooling has been broken for a long time and contributions to that tooling is just not going to happen since nobody can run this stuff outside of Fedora infrastructure. It's a sad state of affairs indeed.
But, I think it's time to move on. We have ostree and various container-delta approaches. We should focus on those — and give DeltaRPMs a sad, fond farewell.
Please don't try to equate those things to DeltaRPMs, unless you're trying to equate their general uselessness. OSTree and "container-delta" stuff is not generally useful or applicable for Fedora users and they won't be for a very long time.
And they might never be useful because OSTree's approach requires us to use OSTree remotes (which we're killing for OCI remotes), and there's no standard for "container-delta" since the baseline OCI format isn't amenable to delta fetching.
-- 真実はいつも一つ！/ Always, there's only one truth!
On Wed, Feb 22, 2023 at 06:18:23AM -0500, Neal Gompa wrote:
Please don't try to equate those things to DeltaRPMs, unless you're trying to equate their general uselessness. OSTree and
Neal, hyperbole like "general uselessness" is not appropriate in talking about other people's work. In any case, CoreOS is our fourth-most-common variant at this point, so clearly some people are finding it useful.
"container-delta" stuff is not generally useful or applicable for Fedora users and they won't be for a very long time.
Maybe, depending on your definition of "very long time". There is work in the right direction, and supporting that can only help.
On Wed, Feb 22, 2023 at 06:18:23AM -0500, Neal Gompa wrote:
On Tue, Feb 21, 2023 at 3:48 PM Matthew Miller mattdm@fedoraproject.org wrote:
I was asked to weigh in on https://pagure.io/releng/issue/7215 as a priority. Last time we talked about this we didn't really get anywhere...
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org/...
... and that ticket hasn't moved, because fixing it isn't trivial.
What we're doing now — as has been the case for several years, already noted in the previous discussion — has very little end-user value. Also as noted in that thread (as in the ticket)... that's unfortunate, because it did bring some real benefits (and could possibly do even more.)
Our tooling has been broken for a long time and contributions to that tooling is just not going to happen since nobody can run this stuff outside of Fedora infrastructure. It's a sad state of affairs indeed.
It's not "broken" just because it doesn't do what you would like it to do. Please can we not disparage other peoples work?
The reason it behaves this way is because koji has no concept of 'newer version' or 'older version'. It has only when a valid build was tagged into a tag, and pungi operates on that with 'give me the latest tagged packages in these tags'.
We could merge current repos into the newly created one from pungi, but it would be a vast amount of work. It would mean all repodata would need to be regenerated. That said, it could be done, but no one has had the cycles to work on it.
kevin
On Wed, 2023-02-22 at 10:40 -0800, Kevin Fenzi wrote:
On Wed, Feb 22, 2023 at 06:18:23AM -0500, Neal Gompa wrote:
On Tue, Feb 21, 2023 at 3:48 PM Matthew Miller mattdm@fedoraproject.org wrote:
I was asked to weigh in on https://pagure.io/releng/issue/7215 as a priority. Last time we talked about this we didn't really get anywhere...
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org/...
... and that ticket hasn't moved, because fixing it isn't trivial.
What we're doing now — as has been the case for several years, already noted in the previous discussion — has very little end-user value. Also as noted in that thread (as in the ticket)... that's unfortunate, because it did bring some real benefits (and could possibly do even more.)
Our tooling has been broken for a long time and contributions to that tooling is just not going to happen since nobody can run this stuff outside of Fedora infrastructure. It's a sad state of affairs indeed.
It's not "broken" just because it doesn't do what you would like it to do. Please can we not disparage other peoples work?
The reason it behaves this way is because koji has no concept of 'newer version' or 'older version'. It has only when a valid build was tagged into a tag, and pungi operates on that with 'give me the latest tagged packages in these tags'.
We could merge current repos into the newly created one from pungi, but it would be a vast amount of work. It would mean all repodata would need to be regenerated. That said, it could be done, but no one has had the cycles to work on it.
I'm guessing this would also make composes slower and less reliable?
On Wed, Feb 22, 2023 at 12:43:20PM -0800, Adam Williamson wrote:
It's not "broken" just because it doesn't do what you would like it to do. Please can we not disparage other peoples work?
The reason it behaves this way is because koji has no concept of 'newer version' or 'older version'. It has only when a valid build was tagged into a tag, and pungi operates on that with 'give me the latest tagged packages in these tags'.
We could merge current repos into the newly created one from pungi, but it would be a vast amount of work. It would mean all repodata would need to be regenerated. That said, it could be done, but no one has had the cycles to work on it.
I'm guessing this would also make composes slower and less reliable?
yep.
kevin
On Wed, Feb 22, 2023 at 1:40 PM Kevin Fenzi kevin@scrye.com wrote:
The reason it behaves this way is because koji has no concept of 'newer version' or 'older version'. It has only when a valid build was tagged into a tag, and pungi operates on that with 'give me the latest tagged packages in these tags'.
We have compose metadata for every single compose that has succeeded and failed. In that metadata, we know what NVRs from Koji were pulled in, and we can reconstitute any historical update push with that data.
We could merge current repos into the newly created one from pungi, but it would be a vast amount of work. It would mean all repodata would need to be regenerated. That said, it could be done, but no one has had the cycles to work on it.
Maybe we should just publish into Pulp and have them do that bit for us automatically. I wouldn't personally be comfortable with that until Pulp is available in Fedora for people to easily deploy, but I know the COPR folks are investigating it for their tooling. And the AlmaLinux folks use Pulp for their system for similar reasons.
Pulp is designed to handle super-huge repositories like that.
-- 真実はいつも一つ！/ Always, there's only one truth!
On Wed, Feb 22, 2023 at 04:44:29PM -0500, Neal Gompa wrote:
We could merge current repos into the newly created one from pungi, but it would be a vast amount of work. It would mean all repodata would need to be regenerated. That said, it could be done, but no one has had the cycles to work on it.
Maybe we should just publish into Pulp and have them do that bit for us automatically. I wouldn't personally be comfortable with that until Pulp is available in Fedora for people to easily deploy, but I know the COPR folks are investigating it for their tooling. And the AlmaLinux folks use Pulp for their system for similar reasons.
Pulp is designed to handle super-huge repositories like that.
I haven't looked at pulp in years... but back when I did it seemed pretty complex and heavy, but sure we could look at it again. I suspect some of the things we do wouldn't be to compatible, like bodhi injecting security/bugfix/enhancement data into repodata after it's made.
In any case we don't have it now, so I still think dropping drpms for now is the way to go.
kevin
On Wednesday, February 22, 2023, Kevin Fenzi kevin@scrye.com wrote:
On Wed, Feb 22, 2023 at 06:18:23AM -0500, Neal Gompa wrote:
On Tue, Feb 21, 2023 at 3:48 PM Matthew Miller mattdm@fedoraproject.org
wrote:
I was asked to weigh in on https://pagure.io/releng/issue/7215 as a priority. Last time we talked about this we didn't really get
anywhere...
fedoraproject.org/thread/JYKVELSBJQMEK6KEFXG354ZDZDDX4C5G/# RLEUYSWOUVUS53YAP7WQQNN7HNEBIC4E
... and that ticket hasn't moved, because fixing it isn't trivial.
What we're doing now — as has been the case for several years, already
noted
in the previous discussion — has very little end-user value. Also as
noted
in that thread (as in the ticket)... that's unfortunate, because it did bring some real benefits (and could possibly do even more.)
Our tooling has been broken for a long time and contributions to that tooling is just not going to happen since nobody can run this stuff outside of Fedora infrastructure. It's a sad state of affairs indeed.
It's not "broken" just because it doesn't do what you would like it to do. Please can we not disparage other peoples work?
Well everything is someone's work. Being that defensive makes discussion on improving things unnecessary hard. I doubt people mean this comments as personal attacks on the people who did this work.
Back on topic: using fedora without having an adequate internet connection is not really "fun" even with delta roms due to the amount and frequency of updates. From that pov the can go if the costs of keeping them is not justified.
On Tue, Feb 21, 2023 at 3:48 PM Matthew Miller mattdm@fedoraproject.org wrote:
What we're doing now — as has been the case for several years, already noted in the previous discussion — has very little end-user value. Also as noted in that thread (as in the ticket)... that's unfortunate, because it did bring some real benefits (and could possibly do even more.)
The fact that the value of deltas requires frequent updates means that most people don't get the benefit. And since delta RPMs trade bandwidth for CPU, it probably makes things worse for folks in developing countries. So I agree, it's probably not worth keeping deltas as the default.
But, I think it's time to move on. We have ostree and various container-delta approaches. We should focus on those — and give DeltaRPMs a sad, fond farewell.
Could we do this as a two-step approach? First change the default to not use deltas but still allow people to opt-in to it. Then (assuming we can track this, which maybe we can't) see how much they're used before we decide to pull the plug on producing them.
-- Ben Cotton He / Him / His Fedora Program Manager Red Hat TZ=America/Indiana/Indianapolis
I agree with the idea that Ben suggested of not enabling deltas by default and giving users the option until a certain time where it can be phased out fully.
I remember in my personal experience that delta slowed down update durations.
On Wed, 22 Feb 2023, 15:40 Ben Cotton, bcotton@redhat.com wrote:
On Tue, Feb 21, 2023 at 3:48 PM Matthew Miller mattdm@fedoraproject.org wrote:
What we're doing now — as has been the case for several years, already
noted
in the previous discussion — has very little end-user value. Also as
noted
in that thread (as in the ticket)... that's unfortunate, because it did bring some real benefits (and could possibly do even more.)
The fact that the value of deltas requires frequent updates means that most people don't get the benefit. And since delta RPMs trade bandwidth for CPU, it probably makes things worse for folks in developing countries. So I agree, it's probably not worth keeping deltas as the default.
But, I think it's time to move on. We have ostree and various container-delta approaches. We should focus on those — and give
DeltaRPMs a
sad, fond farewell.
Could we do this as a two-step approach? First change the default to not use deltas but still allow people to opt-in to it. Then (assuming we can track this, which maybe we can't) see how much they're used before we decide to pull the plug on producing them.
-- Ben Cotton He / Him / His Fedora Program Manager Red Hat TZ=America/Indiana/Indianapolis _______________________________________________ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-leave@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org Do not reply to spam, report it: https://pagure.io/fedora-infrastructure/new_issue
On 2/22/23 10:39, Ben Cotton wrote:
On Tue, Feb 21, 2023 at 3:48 PM Matthew Miller mattdm@fedoraproject.org wrote:
What we're doing now — as has been the case for several years, already noted in the previous discussion — has very little end-user value. Also as noted in that thread (as in the ticket)... that's unfortunate, because it did bring some real benefits (and could possibly do even more.)
The fact that the value of deltas requires frequent updates means that most people don't get the benefit. And since delta RPMs trade bandwidth for CPU, it probably makes things worse for folks in developing countries. So I agree, it's probably not worth keeping deltas as the default.
But, I think it's time to move on. We have ostree and various container-delta approaches. We should focus on those — and give DeltaRPMs a sad, fond farewell.
Could we do this as a two-step approach? First change the default to not use deltas but still allow people to opt-in to it. Then (assuming we can track this, which maybe we can't) see how much they're used before we decide to pull the plug on producing them.
That would be absolutely awesome.
On Wed, Feb 22, 2023 at 10:56:53AM -0500, Demi Marie Obenour wrote:
Could we do this as a two-step approach? First change the default to not use deltas but still allow people to opt-in to it. Then (assuming we can track this, which maybe we can't) see how much they're used before we decide to pull the plug on producing them.
That would be absolutely awesome.
I don't think we can actually tell easily. Additionally, we can't actually tell the important thing, which was "how useful were they really?" — if we have a million people using them but getting an average 0.01% size benefit... that probably doesn't outweigh the costs.
But, we _can_ tell (again, see previous discussion) that what we're currently providing is really unlikely to be very useful. So, I'm not actually sure this approach buys us anything.
On Tue, Feb 21, 2023 at 2:48 PM Matthew Miller mattdm@fedoraproject.org wrote:
I was asked to weigh in on https://pagure.io/releng/issue/7215 as a priority. Last time we talked about this we didn't really get anywhere...
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org/...
... and that ticket hasn't moved, because fixing it isn't trivial.
What we're doing now — as has been the case for several years, already noted in the previous discussion — has very little end-user value. Also as noted in that thread (as in the ticket)... that's unfortunate, because it did bring some real benefits (and could possibly do even more.)
But, I think it's time to move on. We have ostree and various container-delta approaches. We should focus on those — and give DeltaRPMs a sad, fond farewell.
The last updates just now on three different machines gave me Delta RPMs reduced 284.9 MB of updates to 281.0 MB (1.4% saved) Delta RPMs reduced 14.3 MB of updates to 3.3 MB (76.9% saved) and the third had no Delta RPMs
Outside of specific instances, the first or last results are typical. I think it is time to say goodbye. Times are very different from what they were when we added support.
Dennis
On 2/23/23 8:24 PM, Dennis Gilmore via devel wrote:
On Tue, Feb 21, 2023 at 2:48 PM Matthew Miller <mattdm@fedoraproject.org mailto:mattdm@fedoraproject.org> wrote:
I was asked to weigh in on https://pagure.io/releng/issue/7215 <https://pagure.io/releng/issue/7215> as a priority. Last time we talked about this we didn't really get anywhere... https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org/thread/JYKVELSBJQMEK6KEFXG354ZDZDDX4C5G/#RLEUYSWOUVUS53YAP7WQQNN7HNEBIC4E <https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org/thread/JYKVELSBJQMEK6KEFXG354ZDZDDX4C5G/#RLEUYSWOUVUS53YAP7WQQNN7HNEBIC4E> ... and that ticket hasn't moved, because fixing it isn't trivial. What we're doing now — as has been the case for several years, already noted in the previous discussion — has very little end-user value. Also as noted in that thread (as in the ticket)... that's unfortunate, because it did bring some real benefits (and could possibly do even more.) But, I think it's time to move on. We have ostree and various container-delta approaches. We should focus on those — and give DeltaRPMs a sad, fond farewell.The last updates just now on three different machines gave me Delta RPMs reduced 284.9 MB of updates to 281.0 MB (1.4% saved) Delta RPMs reduced 14.3 MB of updates to 3.3 MB (76.9% saved) and the third had no Delta RPMs
Outside of specific instances, the first or last results are typical. I think it is time to say goodbye. Times are very different from what they were when we added support.
Results may be better for applications that require a lot of data outside the projects binaries, like games related packages as wesnoth-data, built from the wesnoth main spec file, but not xonotic-data (1.1G) that is a split from the binaries spec files.
Disabling deltarpms may not change much for a lot of people, but it may affect a few ones using these games. maybe packagers will be forces to split their data to different specs, how easy it is depends on the build process of those games.
Dennis
devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-leave@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org Do not reply to spam, report it: https://pagure.io/fedora-infrastructure/new_issue
On 2023-02-21 12:48, Matthew Miller wrote:
I was asked to weigh in onhttps://pagure.io/releng/issue/7215 as a priority. Last time we talked about this we didn't really get anywhere...
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org/...
... and that ticket hasn't moved, because fixing it isn't trivial.
In that thread, Kevin noted:
* only can make drpms at createrepo_c run time, this means you can't do them later/on the side/keep previous ones around/etc. * pungi doesn't have access to older rpms at the right time to generate more and has no way to keep them accross pushes.
If someone were interested in working on this, to fix either delta rpms, or the vanishing security advisory problem, or both: Are there any tools, repos, collections of scripts, etc that they would need to update in addition to those two?
On Fri, Feb 24, 2023 at 12:12:41PM -0800, Gordon Messmer wrote:
On 2023-02-21 12:48, Matthew Miller wrote:
I was asked to weigh in onhttps://pagure.io/releng/issue/7215 as a priority. Last time we talked about this we didn't really get anywhere...
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org/...
... and that ticket hasn't moved, because fixing it isn't trivial.
In that thread, Kevin noted:
- only can make drpms at createrepo_c run time, this means you can't do
them later/on the side/keep previous ones around/etc.
- pungi doesn't have access to older rpms at the right time to generate
more and has no way to keep them accross pushes.
If someone were interested in working on this, to fix either delta rpms, or the vanishing security advisory problem, or both: Are there any tools, repos, collections of scripts, etc that they would need to update in addition to those two?
Well, thats a pretty wide scope... a lot of it depends on _how_ the fix would work.
I think Kevin's (the other Kevin) suggestion to enhance bodhi to inject into repodata a 'last security fix for this package' for every package in the current compose might at least help to fix that part of the missing security update problem (combined with a dnf change to read and use this data). So that would need bodhi and dnf work.
It's not too clear what the best solution to drpms would be to me. If we moved drpm creation out of composes, the standlone app would still need to know what to make drpms against. Against GA content would be somewhat easy, but against previous updates... it would need to perhaps query bodhi for that? I suspect it would need to download all the rpms involved, so thats a ton of disk space, or it would need to have some other way to access them all (perhaps if we ran it in infra it could have ro access to the koji data). So that would need a new application made, dnf may need changes to look for the drpms in another repo, or perhaps we could just add a drpms repo (I don't know how dnf detects drpms off hand, but I know currently they are in the main repodata).
I think it's probibly time to just let drpms go though. Bandwith vs cpu/disk space isn't as much of a good tradeoff these days.
kevin
Testing F38 KDE spin today, I see this: Delta RPMs reduced 235.2 MB to 23.8 MB (89.9%)
So, it seems we do still have some cornercases when deltaRPM shows its value.
But, ok, In agree that in most cases it's irrelevant this days. +1 for dropping deltarpm
geraldosimiao
Em ter., 21 de fev. de 2023 às 17:48, Matthew Miller < mattdm@fedoraproject.org> escreveu:
I was asked to weigh in on https://pagure.io/releng/issue/7215 as a priority. Last time we talked about this we didn't really get anywhere...
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org/...
... and that ticket hasn't moved, because fixing it isn't trivial.
What we're doing now — as has been the case for several years, already noted in the previous discussion — has very little end-user value. Also as noted in that thread (as in the ticket)... that's unfortunate, because it did bring some real benefits (and could possibly do even more.)
But, I think it's time to move on. We have ostree and various container-delta approaches. We should focus on those — and give DeltaRPMs a sad, fond farewell.
-- Matthew Miller mattdm@fedoraproject.org Fedora Project Leader _______________________________________________ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-leave@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org Do not reply to spam, report it: https://pagure.io/fedora-infrastructure/new_issue
On Fri, Mar 3, 2023 at 7:06 AM Geraldo Simião Kutz geraldo.simiao.kutz@gmail.com wrote:
Testing F38 KDE spin today, I see this: Delta RPMs reduced 235.2 MB to 23.8 MB (89.9%)
So, it seems we do still have some cornercases when deltaRPM shows its value.
Not really. That's local dis, but the aggregate burden of regularly downloading additional repodata exceeds that pretty quickly in environments that do daily update audits.
But, ok, In agree that in most cases it's irrelevant this days. +1 for dropping deltarpm
geraldosimiao
It's a complex feature to support, we're not scrambling for slivers of disk space these days, and maintaining all the deltas grows very quickly for environments that, in theory, no longer have point releases to use as references for deltarpm.
On Mon, Jun 12, 2023 at 09:46:26AM -0700, Kevin Fenzi wrote:
I don't think there's a formal change filed yet.
Matthew: Did you want to do that? Or would you like me or someone else to do so?
I would love someone else to do so, but if no one else wants to, I can. :)
On Thu, Jun 22, 2023 at 10:32 AM Matthew Miller mattdm@fedoraproject.org wrote:
On Mon, Jun 12, 2023 at 09:46:26AM -0700, Kevin Fenzi wrote:
I don't think there's a formal change filed yet.
Matthew: Did you want to do that? Or would you like me or someone else to do so?
I would love someone else to do so, but if no one else wants to, I can. :)
Well ... has anybody filed a change proposal yet, or should I do that?
Fabio
On 7/4/23 11:49, Fabio Valentini wrote:
On Thu, Jun 22, 2023 at 10:32 AM Matthew Miller mattdm@fedoraproject.org wrote:
On Mon, Jun 12, 2023 at 09:46:26AM -0700, Kevin Fenzi wrote:
I don't think there's a formal change filed yet.
Matthew: Did you want to do that? Or would you like me or someone else to do so?
I would love someone else to do so, but if no one else wants to, I can. :)
Well ... has anybody filed a change proposal yet, or should I do that?
Fabio
Do it! Also include deltarpm=False by default for attack surface reduction.
-
Adam Williamson -
Ahmed Almeleh -
Ben Cotton -
Björn Persson -
Chris Adams -
Daniel Alley -
Demi Marie Obenour -
Dennis Gilmore -
drago01 -
Fabio Valentini -
Gary Buhrmaster -
Geraldo Simião Kutz -
Gordon Messmer -
Gregory Bartholomew -
Kamil Paral -
Kevin Fenzi -
Kevin Kofler -
Kevin P. Fleming -
Leslie Satenstein -
Luna Jernberg -
Matthew Miller -
Mattia Verga -
Michael Catanzaro -
Neal Gompa -
Nico Kadel-Garcia -
Petr Pisar -
Reon Beon -
Richard W.M. Jones -
Robert Marcano -
Stephen Smoogen -
Steve Cossette -
Vít Ondruch