DNF Sytem Upgrade requirements for an F37 → F38 upgrade

[Alexander Ploumistos]

Hello,

TL;DR:
DNF memory usage during upgrades from F37 to F38 on a couple of Fedora
Cloud images (with 2 GB of RAM each) led to oomd kicking in and
killing the upgrade process. It might be worth looking into before the
final (also beta?) release.


The longer story:
I have a couple of hosted VMs, which started their lives as Fedora
Cloud 27 and which I have been upgrading to the next Fedora release
via dnf-plugin-system-upgrade. These VMs run very few things, mainly
some PHP applications on Apache and a good chunk of the PHP stack,
some scheduled network scripts, a private rpm repository, OpenVPN and
Wireguard nodes and when I am far from my main computer and koji or
copr happen to be busy, I test things in mock on them. These VMs are
near clones of each other, so usually only one of them is up at any
moment.

I've never had an issue with memory use up to now and when the systems
aren't building packages, they seldom require more than 200 MB of RAM.
What's more, there's the default swap-on-zram, so they should be more
than happy memory-wise with 2 GB of RAM.

I tried to upgrade one VM to F38, to see what problems I might face
and after the packages had been downloaded, I lost the ssh connection
as DNF was running the transaction test, right after a successful
transaction check. I repeated the whole thing a couple more times,
until I remembered that oomd is there and sure enough, DNF had been
consuming 1-1.3 GB until oomd decided to kill it. Well, actually it
killed my entire session and not just DNF. I tried running the upgrade
inside a screen (the GNU one) session and I came back to an empty
session. The transaction involved the upgrade of 1620 packages, with
a total size of 1.2 GB. I think the difference between that and the
previous upgrade F36 to F37) is around 100 MB and 30 or 40 packages.
In each of the failed attempts, the memory pressure was 55 to 78% for
more than 20 seconds, according to systemd-oomd. The exact same issues
appeared when upgrading the second VM.

Eventually I did the upgrade by doubling the available RAM to 4 GB
(and then set it back to 2), but perhaps this could be a problem for
others, especially those who pay for cloud-based resources. Is DNF
supposed to be a legitimate target for systemd-oomd? Conversely, is
DNF supposed to use up so much memory?

[Fabio Valentini]

On Mon, Mar 6, 2023 at 4:09 AM Alexander Ploumistos
<alex.ploumistos(a)gmail.com&gt; wrote:
>  Hello,
>
>  TL;DR:
>  DNF memory usage during upgrades from F37 to F38 on a couple of Fedora
>  Cloud images (with 2 GB of RAM each) led to oomd kicking in and
>  killing the upgrade process. It might be worth looking into before the
>  final (also beta?) release.
>
>
>  The longer story:
>  I have a couple of hosted VMs, which started their lives as Fedora
>  Cloud 27 and which I have been upgrading to the next Fedora release
>  via dnf-plugin-system-upgrade. These VMs run very few things, mainly
>  some PHP applications on Apache and a good chunk of the PHP stack,
>  some scheduled network scripts, a private rpm repository, OpenVPN and
>  Wireguard nodes and when I am far from my main computer and koji or
>  copr happen to be busy, I test things in mock on them. These VMs are
>  near clones of each other, so usually only one of them is up at any
>  moment.
>
>  I've never had an issue with memory use up to now and when the systems
>  aren't building packages, they seldom require more than 200 MB of RAM.
>  What's more, there's the default swap-on-zram, so they should be more
>  than happy memory-wise with 2 GB of RAM.
>
>  I tried to upgrade one VM to F38, to see what problems I might face
>  and after the packages had been downloaded, I lost the ssh connection
>  as DNF was running the transaction test, right after a successful
>  transaction check. I repeated the whole thing a couple more times,
>  until I remembered that oomd is there and sure enough, DNF had been
>  consuming 1-1.3 GB until oomd decided to kill it. Well, actually it
>  killed my entire session and not just DNF. I tried running the upgrade
>  inside a screen (the GNU one) session and I came back to an empty
>  session. The transaction involved the upgrade of 1620 packages, with
>  a total size of 1.2 GB. I think the difference between that and the
>  previous upgrade F36 to F37) is around 100 MB and 30 or 40 packages.
>  In each of the failed attempts, the memory pressure was 55 to 78% for
>  more than 20 seconds, according to systemd-oomd. The exact same issues
>  appeared when upgrading the second VM.
>
>  Eventually I did the upgrade by doubling the available RAM to 4 GB
>  (and then set it back to 2), but perhaps this could be a problem for
>  others, especially those who pay for cloud-based resources. Is DNF
>  supposed to be a legitimate target for systemd-oomd? Conversely, is
>  DNF supposed to use up so much memory? 
I know I'm late to the party, but I think this is still worth taking a look at.
I have two small Fedora servers that used to run just fine with 1 GB
of RAM, but I needed to up that to 2 GB for the upgrade to Fedora 36
(thanks to DNF running out of memory otherwise).
Reading the original post, it sounds like I'll need to up that again
soon, either for the upgrade to Fedora 37 or 38?
I don't really want to throw money out the window just because DNF
eats up all the memory it can :(

Additionally, the smallest offerings of popular VPS providers have
just 1 or 2 GB of RAM, is Fedora Server no longer supported on systems
like these?
Do we need to update the documentation for system requirements? Ping
cloud hosting providers that they shouldn't offer installing Fedora on
configurations with too little RAM?
There's also lots of SBCs with limited RAM - are Fedora users on these
devices supposed to reinstall Fedora every six months, since in-place
upgrades might no longer be possible?

Fabio

[Alexander Ploumistos]

On Thu, Mar 30, 2023 at 12:09 AM Fabio Valentini <decathorpe(a)gmail.com&gt; wrote:
>  Additionally, the smallest offerings of popular VPS providers have
>  just 1 or 2 GB of RAM, is Fedora Server no longer supported on systems
>  like these?
>  Do we need to update the documentation for system requirements? Ping
>  cloud hosting providers that they shouldn't offer installing Fedora on
>  configurations with too little RAM?
>  There's also lots of SBCs with limited RAM - are Fedora users on these
>  devices supposed to reinstall Fedora every six months, since in-place
>  upgrades might no longer be possible? 
Another solution could be to tweak systemd-oomd not to touch dnf when
running a system upgrade (perhaps through
python3-dnf-plugin-system-upgrade). When I repeated the upgrade
process with the second VM and I kept an eye out for memory usage, I
saw that there were more than 300 MB available when oomd stepped in.
Even installations requiring much larger transactions should be fine
as long as there's a swap.

[Miroslav Suchý]

Dne 30. 03. 23 v 0:08 Fabio Valentini napsal(a):
>  Additionally, the smallest offerings of popular VPS providers have
>  just 1 or 2 GB of RAM, is Fedora Server no longer supported on systems
>  like these? 
At first place - upgrades in cloud environment should be discouraged. The standard should
be 1) describe your system in 
Ansible playbook 2) terminate machine 3) change in your playbook the name of new image 4)
run your playbook. This is how 
we upgrade machines in cloud in fedora-infra.

And not sure if it worth a work (likely not easy) as the upgrades from F39 to F40 will be
done by DNF5 with totally 
different (smaller I hope) footprint.

Miroslav

[Daniel P. Berrangé]

On Thu, Mar 30, 2023 at 10:49:33AM +0200, Miroslav Suchý wrote:
>  Dne 30. 03. 23 v 0:08 Fabio Valentini napsal(a):
>  > Additionally, the smallest offerings of popular VPS providers have
>  > just 1 or 2 GB of RAM, is Fedora Server no longer supported on systems
>  > like these?
>  
>  At first place - upgrades in cloud environment should be discouraged. The
>  standard should be 1) describe your system in Ansible playbook 2) terminate
>  machine 3) change in your playbook the name of new image 4) run your
>  playbook. This is how we upgrade machines in cloud in fedora-infra. 
I don't think it is realistic to expect users to follow such steps
in the general case as plenty of cloud VMs are considered precious
and not simply able to be thrown away and replaced by freshly built
system that way.

>  And not sure if it worth a work (likely not easy) as the upgrades
> from F39
>  to F40 will be done by DNF5 with totally different (smaller I hope)
>  footprint. 
With regards,
Daniel
-- 
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|

[Colin Walters]

On Wed, Mar 29, 2023, at 6:08 PM, Fabio Valentini wrote:
>  
>  I don't really want to throw money out the window just because DNF
>  eats up all the memory it can :( 
Everyone needs to internalize:  This has nothing to do with DNF, really.

It's about the *size of the repository metadata*.

Every single time someone adds a new package into the single "Fedora" rpm-md
repo, that makes clients going OOM a bit more likely.  Every single time there's a new
package, it makes it more likely that a client will hit a timeout downloading the repodata
and fail to get that critical kernel security update.

Even doing the obvious thing and splitting off e.g. a `fedora-devel` rpm-md repo (as is
done kind of done in RHEL with CRB) that had all the -devel and Go/Rust etc. packages
would likely create a huge amount of savings.

Also as I said in an earlier thread, relating to SBCs but also small cloud instances  - we
also have image-based editions; these are Fedora systems too.  rpm-ostree will not
download the rpm-md by default, so is immune to this.  And particularly after
https://fedoraproject.org/wiki/Changes/OstreeNativeContainerStable - the UX for managing
these is the same as podman/docker style application containers.  You don't pay the
cost of downloading the rpm-md on each client.

[Zbigniew Jędrzejewski-Szmek]

On Thu, Mar 30, 2023 at 12:08:03AM +0200, Fabio Valentini wrote:
>  On Mon, Mar 6, 2023 at 4:09 AM Alexander Ploumistos
>  <alex.ploumistos(a)gmail.com&gt; wrote:
>  >
>  > Hello,
>  >
>  > TL;DR:
>  > DNF memory usage during upgrades from F37 to F38 on a couple of Fedora
>  > Cloud images (with 2 GB of RAM each) led to oomd kicking in and
>  > killing the upgrade process. It might be worth looking into before the
>  > final (also beta?) release. 
Please try with systemd-253.2-1.fc38 / systemd-253.2-1.fc39.

Zbyszek

[Alexander Ploumistos]

On Thu, Mar 30, 2023 at 2:10 PM Zbigniew Jędrzejewski-Szmek
<zbyszek(a)in.waw.pl&gt; wrote:
>  On Thu, Mar 30, 2023 at 12:08:03AM +0200, Fabio Valentini wrote:
>  > On Mon, Mar 6, 2023 at 4:09 AM Alexander Ploumistos
>  > <alex.ploumistos(a)gmail.com&gt; wrote:
>  > >
>  > > Hello,
>  > >
>  > > TL;DR:
>  > > DNF memory usage during upgrades from F37 to F38 on a couple of Fedora
>  > > Cloud images (with 2 GB of RAM each) led to oomd kicking in and
>  > > killing the upgrade process. It might be worth looking into before the
>  > > final (also beta?) release.
>
>  Please try with systemd-253.2-1.fc38 / systemd-253.2-1.fc39. 

Hi Zbigniew, is F37 getting that version too or are you talking about
a future F38 → F39 upgrade?

[Zbigniew Jędrzejewski-Szmek]

On Thu, Mar 30, 2023 at 02:25:33PM +0200, Alexander Ploumistos wrote:
>  On Thu, Mar 30, 2023 at 2:10 PM Zbigniew Jędrzejewski-Szmek
>  <zbyszek(a)in.waw.pl&gt; wrote:
>  >
>  > On Thu, Mar 30, 2023 at 12:08:03AM +0200, Fabio Valentini wrote:
>  > > On Mon, Mar 6, 2023 at 4:09 AM Alexander Ploumistos
>  > > <alex.ploumistos(a)gmail.com&gt; wrote:
>  > > >
>  > > > Hello,
>  > > >
>  > > > TL;DR:
>  > > > DNF memory usage during upgrades from F37 to F38 on a couple of Fedora
>  > > > Cloud images (with 2 GB of RAM each) led to oomd kicking in and
>  > > > killing the upgrade process. It might be worth looking into before the
>  > > > final (also beta?) release.
>  >
>  > Please try with systemd-253.2-1.fc38 / systemd-253.2-1.fc39.
>  
>  
>  Hi Zbigniew, is F37 getting that version too or are you talking about
>  a future F38 → F39 upgrade? 
That version — no. The patch for oomd config — yes (if it proves to be
beneficial in F38/rawhide).

Zbyszek

[Alexander Ploumistos]

On Thu, Mar 30, 2023 at 2:33 PM Zbigniew Jędrzejewski-Szmek
<zbyszek(a)in.waw.pl&gt; wrote:
>  On Thu, Mar 30, 2023 at 02:25:33PM +0200, Alexander Ploumistos wrote:
>  > On Thu, Mar 30, 2023 at 2:10 PM Zbigniew Jędrzejewski-Szmek
>  > <zbyszek(a)in.waw.pl&gt; wrote:
>  > >
>  > > On Thu, Mar 30, 2023 at 12:08:03AM +0200, Fabio Valentini wrote:
>  > > > On Mon, Mar 6, 2023 at 4:09 AM Alexander Ploumistos
>  > > > <alex.ploumistos(a)gmail.com&gt; wrote:
>  > > > >
>  > > > > Hello,
>  > > > >
>  > > > > TL;DR:
>  > > > > DNF memory usage during upgrades from F37 to F38 on a couple of
> Fedora
>  > > > > Cloud images (with 2 GB of RAM each) led to oomd kicking in and
>  > > > > killing the upgrade process. It might be worth looking into before
> the
>  > > > > final (also beta?) release.
>  > >
>  > > Please try with systemd-253.2-1.fc38 / systemd-253.2-1.fc39.
>  >
>  >
>  > Hi Zbigniew, is F37 getting that version too or are you talking about
>  > a future F38 → F39 upgrade?
>
>  That version — no. The patch for oomd config — yes (if it proves to be
>  beneficial in F38/rawhide). 

Could you please bump this thread when it's done?

[Zbigniew Jędrzejewski-Szmek]

On Thu, Mar 30, 2023 at 02:37:16PM +0200, Alexander Ploumistos wrote:
>  On Thu, Mar 30, 2023 at 2:33 PM Zbigniew Jędrzejewski-Szmek
>  <zbyszek(a)in.waw.pl&gt; wrote:
>  >
>  > On Thu, Mar 30, 2023 at 02:25:33PM +0200, Alexander Ploumistos wrote:
>  > > On Thu, Mar 30, 2023 at 2:10 PM Zbigniew Jędrzejewski-Szmek
>  > > <zbyszek(a)in.waw.pl&gt; wrote:
>  > > >
>  > > > On Thu, Mar 30, 2023 at 12:08:03AM +0200, Fabio Valentini wrote:
>  > > > > On Mon, Mar 6, 2023 at 4:09 AM Alexander Ploumistos
>  > > > > <alex.ploumistos(a)gmail.com&gt; wrote:
>  > > > > >
>  > > > > > Hello,
>  > > > > >
>  > > > > > TL;DR:
>  > > > > > DNF memory usage during upgrades from F37 to F38 on a couple of
> Fedora
>  > > > > > Cloud images (with 2 GB of RAM each) led to oomd kicking in and
>  > > > > > killing the upgrade process. It might be worth looking into
> before the
>  > > > > > final (also beta?) release.
>  > > >
>  > > > Please try with systemd-253.2-1.fc38 / systemd-253.2-1.fc39.
>  > >
>  > >
>  > > Hi Zbigniew, is F37 getting that version too or are you talking about
>  > > a future F38 → F39 upgrade?
>  >
>  > That version — no. The patch for oomd config — yes (if it proves to be
>  > beneficial in F38/rawhide).
>  
>  Could you please bump this thread when it's done? 
I will submit systemd-tests-251.14-2.fc37 with the patch.
I doubt that it solves the issue completely, but it shouldn't make
it worse, so let's at least do this for now.

Zbyszek

[Alexander Ploumistos]

On Fri, Mar 31, 2023 at 7:09 PM Zbigniew Jędrzejewski-Szmek
<zbyszek(a)in.waw.pl&gt; wrote:
>  I will submit systemd-tests-251.14-2.fc37 with the patch.
>  I doubt that it solves the issue completely, but it shouldn't make
>  it worse, so let's at least do this for now. 
Thank you, I see it's still being built in koji. I will spin a VM with
the same configuration over the weekend and try the upgrade again.

[Alexander Ploumistos]

On Fri, Mar 31, 2023 at 7:25 PM Alexander Ploumistos
<alex.ploumistos(a)gmail.com&gt; wrote:
>  On Fri, Mar 31, 2023 at 7:09 PM Zbigniew Jędrzejewski-Szmek
>  <zbyszek(a)in.waw.pl&gt; wrote:
>  >
>  > I will submit systemd-tests-251.14-2.fc37 with the patch.
>  > I doubt that it solves the issue completely, but it shouldn't make
>  > it worse, so let's at least do this for now.
>
>  Thank you, I see it's still being built in koji. I will spin a VM with
>  the same configuration over the weekend and try the upgrade again. 
Something is glitching on my provider's side and it's impossible to
create new VMs. I've opened a ticket, but I doubt anyone is going to
deal with it before Monday.

[Alexander Ploumistos]

On Sat, Apr 1, 2023 at 3:44 PM Alexander Ploumistos
<alex.ploumistos(a)gmail.com&gt; wrote:
>  Something is glitching on my provider's side and it's impossible to
>  create new VMs. I've opened a ticket, but I doubt anyone is going to
>  deal with it before Monday. 
It has taken them over a month to (sort of) resolve the problem, but
in any case, I was able to set up another VM to test.
I can confirm that with the backported fix in 251.14-2.fc37, 2GB of
RAM is more than enough to perform the upgrade.
Sorry for the delay.