Heads-up / for discussion: dnf not working with 1G of RAM or less

[Adam Williamson]

Hey folks! I apologize for the wide distribution, but this seemed like
a bug it'd be appropriate to get a wide range of input on.

There's a bug that was proposed as an F37 Beta blocker:
https://bugzilla.redhat.com/show_bug.cgi?id=1907030

it's quite an old bug, but up until recently, the summary was
apparently accurate - dnf would run out of memory with 512M of RAM, but
was OK with 1G. However, as of quite recently, on F36 at least (not
sure if anyone's explicitly tested F37), dnf operations are commonly
failing on VMs/containers with 1G of RAM due to running out of RAM and
getting OOM-killed.

There's some discussion in the bug about what might be causing this and
potential ways to resolve it, and please do dig into/contribute to that
if you can, but the other question here I guess is: how much do we care
about this? How bad is it that you can't reliably run dnf operations on
top of a minimal Fedora environment with 1G of RAM?

This obviously has some overlap with our stated hardware requirements,
so here they are for the record:

https://docs.fedoraproject.org/en-US/fedora/latest/release-notes/welcome/...

that specifies 2GB as the minimum memory for "the default
installation", by which I think it's referring to a default Workstation
install, though this should be clarified. But then there's a "Low
memory installations" boxout, which suggests that "users with less than
768MB of system memory may have better results performing a minimal
install and adding to it afterward", which kinda is recommending that
people do exactly the thing that doesn't work (do a minimal install
then use dnf on it), and implying it'll work.

After some consideration I don't think it makes sense to take this bug
as an F37 blocker, since it already affects F36, and that's what I'll
be suggesting at the next blocker review meeting. However, it does seem
a perfect candidate for prioritized bug status, and I've nominated it
for that.

I guess if folks can chime in with thoughts here and/or in the bug
report, maybe a consensus will emerge on just how big of an issue this
is (and how likely it is to get fixed). There will presumably be a
FESCo ticket related to prioritized bug status too.

Thanks folks!
-- 
Adam Williamson
Fedora QA
IRC: adamw | Twitter: adamw_ha
https://www.happyassassin.net

[Zbigniew Jędrzejewski-Szmek]

On Mon, Aug 22, 2022 at 05:44:26PM -0700, Adam Williamson wrote:
>  Hey folks! I apologize for the wide distribution, but this seemed
> like
>  a bug it'd be appropriate to get a wide range of input on.
>  
>  There's a bug that was proposed as an F37 Beta blocker:
>  https://bugzilla.redhat.com/show_bug.cgi?id=1907030
>  
>  it's quite an old bug, but up until recently, the summary was
>  apparently accurate - dnf would run out of memory with 512M of RAM, but
>  was OK with 1G. However, as of quite recently, on F36 at least (not
>  sure if anyone's explicitly tested F37), dnf operations are commonly
>  failing on VMs/containers with 1G of RAM due to running out of RAM and
>  getting OOM-killed. 
The discussion in the bug indicates that this memory growth is related to
loading of the full filepath dataset. We have been discussing splitting
out the non-primary-filepath-data (i.e. paths that are not /etc, /usr/bin,
/usr/sbin), out into a separate lazilly-loaded file. If we manage to do
that, we'll kill two birds with one stone:

- initial download of repo metadata on every freakin' dnf operation can
  go down from 80 to 20 MB
- the peak memory use will go down

Apparently DNF5 makes this possible.
My vote is: yes, this is an issue. No, we shouldn't block F37 on this.
Apparently the only reasonable way to tackle this is with a major rework
of DNF, so let's get it right with DNF5.

Zbyszek

[fedora-arm] Heads-up / for discussion: dnf not working with 1G of RAM or less

[Hans de Goede]

Hi,

On 8/23/22 02:44, Adam Williamson wrote:
>  Hey folks! I apologize for the wide distribution, but this seemed
> like
>  a bug it'd be appropriate to get a wide range of input on.
>  
>  There's a bug that was proposed as an F37 Beta blocker:
>  https://bugzilla.redhat.com/show_bug.cgi?id=1907030
>  
>  it's quite an old bug, but up until recently, the summary was
>  apparently accurate - dnf would run out of memory with 512M of RAM, but
>  was OK with 1G. However, as of quite recently, on F36 at least (not
>  sure if anyone's explicitly tested F37), dnf operations are commonly
>  failing on VMs/containers with 1G of RAM due to running out of RAM and
>  getting OOM-killed.
>  
>  There's some discussion in the bug about what might be causing this and
>  potential ways to resolve it, and please do dig into/contribute to that
>  if you can, but the other question here I guess is: how much do we care
>  about this? How bad is it that you can't reliably run dnf operations on
>  top of a minimal Fedora environment with 1G of RAM?
>  
>  This obviously has some overlap with our stated hardware requirements,
>  so here they are for the record:
>  
>
> https://docs.fedoraproject.org/en-US/fedora/latest/release-notes/welcome/...
>  
>  that specifies 2GB as the minimum memory for "the default
>  installation", by which I think it's referring to a default Workstation
>  install, though this should be clarified. But then there's a "Low
>  memory installations" boxout, which suggests that "users with less than
>  768MB of system memory may have better results performing a minimal
>  install and adding to it afterward", which kinda is recommending that
>  people do exactly the thing that doesn't work (do a minimal install
>  then use dnf on it), and implying it'll work.
>  
>  After some consideration I don't think it makes sense to take this bug
>  as an F37 blocker, since it already affects F36, and that's what I'll
>  be suggesting at the next blocker review meeting. However, it does seem
>  a perfect candidate for prioritized bug status, and I've nominated it
>  for that.
>  
>  I guess if folks can chime in with thoughts here and/or in the bug
>  report, maybe a consensus will emerge on just how big of an issue this
>  is (and how likely it is to get fixed). There will presumably be a
>  FESCo ticket related to prioritized bug status too. 
I still have quite a few x86_64 tablets with only 1G RAM, so I personally
definitely care about this.

With that said I do regularly run dnf on them without issues,
although I think most of the 1G models I have are still at F35...

Still I wonder why this is an issue in some cases:

1. ZRAM helps a lot here, but I guess with containers when limiting them
to 1G you really only get 1G since ZRAM works on the system level not
the container level. In the VM case though ZRAM should help, is ZRAM
enabled ?  ZRAM is enabled by default for Workstation installs, but
maybe not in other installs ?

2. Maybe there are some other processes also taking up more RAM
then expected causing extra memory pressure ?

Regards,

Hans

[Dusty Mabe]

On 8/22/22 20:44, Adam Williamson wrote:
>  Hey folks! I apologize for the wide distribution, but this seemed
> like
>  a bug it'd be appropriate to get a wide range of input on.
>  
>  There's a bug that was proposed as an F37 Beta blocker:
>  https://bugzilla.redhat.com/show_bug.cgi?id=1907030
>  
>  it's quite an old bug, but up until recently, the summary was
>  apparently accurate - dnf would run out of memory with 512M of RAM, but
>  was OK with 1G. However, as of quite recently, on F36 at least (not
>  sure if anyone's explicitly tested F37), dnf operations are commonly
>  failing on VMs/containers with 1G of RAM due to running out of RAM and
>  getting OOM-killed.
>  
>  There's some discussion in the bug about what might be causing this and
>  potential ways to resolve it, and please do dig into/contribute to that
>  if you can, but the other question here I guess is: how much do we care
>  about this? How bad is it that you can't reliably run dnf operations on
>  top of a minimal Fedora environment with 1G of RAM? 
I think we need to stop thinking about this as just a single computer
problem.

In a containerized world you could have numerous containers running dnf
operations on a machine at any given time. Making the dnf operations take
less memory means we'll hit many fewer issues as single machines are
able to run many dnf operations at a given moment.

> 
> This obviously has some overlap with our stated hardware requirements,
> so here they are for the record:
> 
>
https://docs.fedoraproject.org/en-US/fedora/latest/release-notes/welcome/...
> 
> that specifies 2GB as the minimum memory for "the default
> installation", by which I think it's referring to a default Workstation
> install, though this should be clarified. But then there's a "Low
> memory installations" boxout, which suggests that "users with less than
> 768MB of system memory may have better results performing a minimal
> install and adding to it afterward", which kinda is recommending that
> people do exactly the thing that doesn't work (do a minimal install
> then use dnf on it), and implying it'll work.
> 
> After some consideration I don't think it makes sense to take this bug
> as an F37 blocker, since it already affects F36, and that's what I'll
> be suggesting at the next blocker review meeting. However, it does seem
> a perfect candidate for prioritized bug status, and I've nominated it
> for that.
> 
> I guess if folks can chime in with thoughts here and/or in the bug
> report, maybe a consensus will emerge on just how big of an issue this
> is (and how likely it is to get fixed). There will presumably be a
> FESCo ticket related to prioritized bug status too.
> 
> Thanks folks!