Turns out that with the squashfs.img fetched from network we need a lot more RAM. PATCH 1/4 adds a code that makes check_memory aware of that fact, PATCH 2/4 turns on zRAM up to 2 GB RAM because with squashfs.img from network it is necessary or really useful up to 2 GB. PATCH 3/4 fixes an issue I've discovered when fiding memory limits where anaconda-yum got oom-killed and the anaconda process didn't care and failed on missing authconfig in the installed system where nothing was actually installed. Finally PATCH 4/4 is just a cosmetic change removing useless numeric operations potentially leading to errors.
Vratislav Podzimek (4): RAM requirements depend on squashfs.img's origin Use zRAM swap up to 2 GB of RAM Raise exception if reading lines from a killed process Do not multiply/divide RAM sizes by 1024 back and forth
anaconda | 14 +++++++++++--- pyanaconda/isys/__init__.py | 9 +++++---- pyanaconda/iutil.py | 16 ++++++++++++++++ scripts/zramswapon | 4 +++- 4 files changed, 35 insertions(+), 8 deletions(-)
We have much higher memory requirements if the squashfs.img was fetched from somewhere and has to be kept in RAM. With not enough RAM, the loop1 process freaks out, takes 100 % CPU and eventually oom_kill kills the anaconda-yum process.
Signed-off-by: Vratislav Podzimek vpodzime@redhat.com --- anaconda | 8 ++++++++ pyanaconda/isys/__init__.py | 1 + pyanaconda/iutil.py | 14 ++++++++++++++ 3 files changed, 23 insertions(+)
diff --git a/anaconda b/anaconda index 9374d1c..9f1dbb4 100755 --- a/anaconda +++ b/anaconda @@ -578,6 +578,9 @@ def gtk_warning(title, reason):
# pylint: disable=redefined-outer-name def check_memory(anaconda, options, display_mode=None): + from pyanaconda import isys + from pyanaconda.iutil import persistent_root_image + reason_strict = _("%(product_name)s requires %(needed_ram)s MB of memory to " "install, but you only have %(total_ram)s MB on this machine.\n") reason_graphical = _("The %(product_name)s graphical installer requires %(needed_ram)s " @@ -602,6 +605,11 @@ def check_memory(anaconda, options, display_mode=None): needed_ram = int(isys.MIN_RAM / 1024) graphical_ram = int(isys.MIN_GUI_RAM / 1024)
+ # count the squashfs.img in if it is kept in RAM + if not persistent_root_image(): + needed_ram += isys.SQUASHFS_RAM + graphical_ram += isys.SQUASHFS_RAM + log.info("check_memory(): total:%s, needed:%s, graphical:%s", total_ram, needed_ram, graphical_ram)
diff --git a/pyanaconda/isys/__init__.py b/pyanaconda/isys/__init__.py index def8e96..06291f6 100644 --- a/pyanaconda/isys/__init__.py +++ b/pyanaconda/isys/__init__.py @@ -53,6 +53,7 @@ else: GUI_INSTALL_EXTRA_RAM = 90 * 1024
MIN_GUI_RAM = MIN_RAM + GUI_INSTALL_EXTRA_RAM +SQUASHFS_EXTRA_RAM = 750 * 1024
## Flush filesystem buffers. def sync (): diff --git a/pyanaconda/iutil.py b/pyanaconda/iutil.py index 642e7b0..3f0cc0e 100644 --- a/pyanaconda/iutil.py +++ b/pyanaconda/iutil.py @@ -29,6 +29,7 @@ import subprocess import unicodedata import string import types +import re from threading import Thread from Queue import Queue, Empty from urllib import quote, unquote @@ -878,3 +879,16 @@ def xprogressive_delay(): while True: yield 0.25*(2**counter) counter += 1 + +def persistent_root_image(): + """:returns: whether we are running from a persistent (not in RAM) root.img""" + + for line in execReadlines("losetup", ["--list"]): + # if there is an active loop device for a curl-fetched file that has + # been deleted, it means we run from a non-persistent root image + # EXAMPLE line: + # /dev/loop0 0 0 0 1 /tmp/curl_fetch_url0/my_comps_squashfs.img (deleted) + if re.match(r'.*curl_fetch_url.*(deleted)\s*$', line): + return False + + return True
If squashfs.img is in RAM, zRAM swap is required and really useful up to 2 GB of RAM. If it is mounted from somewhere directly, zRAM doesn't hurt and speeds up installation a bit. With more than 2 GB of RAM it just presents CPU overhead for no benefit.
Signed-off-by: Vratislav Podzimek vpodzime@redhat.com --- scripts/zramswapon | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-)
diff --git a/scripts/zramswapon b/scripts/zramswapon index f37764a..161ea80 100755 --- a/scripts/zramswapon +++ b/scripts/zramswapon @@ -24,13 +24,15 @@
set -x
+MAX_RAM_ON=2097152 + # get the amount of memory in the machine mem_total_kb=$(grep MemTotal /proc/meminfo | grep -E --only-matching '[[:digit:]]+') mem_total=$((mem_total_kb * 1024))
grep -E 'inst.zram=(on|1)' /proc/cmdline > /dev/null force=$? -if [ ( $mem_total_kb -gt "1048576" ) -a ( $force != "0" ) ]; then +if [ ( $mem_total_kb -gt $MAX_RAM_ON ) -a ( $force != "0" ) ]; then # more than 1 GB of RAM and not forced, do nothing exit 0 fi
If process is killed and we try to read lines from its output, the caller should be informed. Otherwise it simply continues with a notion that everything went well and the output just finished.
Signed-off-by: Vratislav Podzimek vpodzime@redhat.com --- pyanaconda/iutil.py | 2 ++ 1 file changed, 2 insertions(+)
diff --git a/pyanaconda/iutil.py b/pyanaconda/iutil.py index 3f0cc0e..f783de3 100644 --- a/pyanaconda/iutil.py +++ b/pyanaconda/iutil.py @@ -268,6 +268,8 @@ def execReadlines(command, argv, stdin=None, root='/', env_prune=None): q.task_done() except Empty: if proc.poll() is not None: + if os.WIFSIGNALED(proc.returncode): + raise OSError("process '%s' was killed" % argv) break q.join()
It doesn't make sense to multiply the value by 1024 if its only usage divides it back as the first thing.
Signed-off-by: Vratislav Podzimek vpodzime@redhat.com --- anaconda | 10 +++++----- pyanaconda/isys/__init__.py | 10 +++++----- 2 files changed, 10 insertions(+), 10 deletions(-)
diff --git a/anaconda b/anaconda index 9f1dbb4..4d4676e 100755 --- a/anaconda +++ b/anaconda @@ -601,14 +601,14 @@ def check_memory(anaconda, options, display_mode=None): display_mode = anaconda.displayMode
reason = reason_strict - total_ram = int(isys.total_memory() / 1024) - needed_ram = int(isys.MIN_RAM / 1024) - graphical_ram = int(isys.MIN_GUI_RAM / 1024) + total_ram = int(isys.total_memory()) + needed_ram = int(isys.MIN_RAM) + graphical_ram = int(isys.MIN_GUI_RAM)
# count the squashfs.img in if it is kept in RAM if not persistent_root_image(): - needed_ram += isys.SQUASHFS_RAM - graphical_ram += isys.SQUASHFS_RAM + needed_ram += isys.SQUASHFS_EXTRA_RAM + graphical_ram += isys.SQUASHFS_EXTRA_RAM
log.info("check_memory(): total:%s, needed:%s, graphical:%s", total_ram, needed_ram, graphical_ram) diff --git a/pyanaconda/isys/__init__.py b/pyanaconda/isys/__init__.py index 06291f6..29e9b24 100644 --- a/pyanaconda/isys/__init__.py +++ b/pyanaconda/isys/__init__.py @@ -46,14 +46,14 @@ import logging log = logging.getLogger("anaconda")
if blivet.arch.getArch() in ["ppc64", "ppc64le"]: - MIN_RAM = 768 * 1024 - GUI_INSTALL_EXTRA_RAM = 512 * 1024 + MIN_RAM = 768 + GUI_INSTALL_EXTRA_RAM = 512 else: - MIN_RAM = 320 * 1024 - GUI_INSTALL_EXTRA_RAM = 90 * 1024 + MIN_RAM = 320 + GUI_INSTALL_EXTRA_RAM = 90
MIN_GUI_RAM = MIN_RAM + GUI_INSTALL_EXTRA_RAM -SQUASHFS_EXTRA_RAM = 750 * 1024 +SQUASHFS_EXTRA_RAM = 750
## Flush filesystem buffers. def sync ():
On Fri, Jul 25, 2014 at 12:36:58PM +0200, Vratislav Podzimek wrote:
Turns out that with the squashfs.img fetched from network we need a lot more RAM. PATCH 1/4 adds a code that makes check_memory aware of that fact, PATCH 2/4 turns on zRAM up to 2 GB RAM because with squashfs.img from network it is necessary or really useful up to 2 GB. PATCH 3/4 fixes an issue I've discovered when fiding memory limits where anaconda-yum got oom-killed and the anaconda process didn't care and failed on missing authconfig in the installed system where nothing was actually installed. Finally PATCH 4/4 is just a cosmetic change removing useless numeric operations potentially leading to errors.
Vratislav Podzimek (4): RAM requirements depend on squashfs.img's origin Use zRAM swap up to 2 GB of RAM Raise exception if reading lines from a killed process Do not multiply/divide RAM sizes by 1024 back and forth
anaconda | 14 +++++++++++--- pyanaconda/isys/__init__.py | 9 +++++---- pyanaconda/iutil.py | 16 ++++++++++++++++ scripts/zramswapon | 4 +++- 4 files changed, 35 insertions(+), 8 deletions(-)
-- 1.9.3
These all look good to me.
anaconda-patches@lists.fedorahosted.org
-
Brian C. Lane -
Vratislav Podzimek