[PATCH] crashkernel: optimize arm64 reserved size if PAGE_SIZE=4k

[Pingfan Liu]

The arm64 platform supports either 64KB or 4KB page size. In the case of
4KB page size, the reserved memory size can be aligned to that on
x86_64.

Introducing a new formula for 4KB on arm64, which bases on x86_64 plus
extra 64MB.

Signed-off-by: Pingfan Liu 
---
gitlab: https://gitlab.com/coxu/fedora-kexec-tools/-/merge_requests/8
 kdump-lib.sh | 8 +++++++-
 1 file changed, 7 insertions(+), 1 deletion(-)

diff --git a/kdump-lib.sh b/kdump-lib.sh
index b137c89..ef9bd64 100755
--- a/kdump-lib.sh
+++ b/kdump-lib.sh
@@ -853,7 +853,13 @@ kdump_get_arch_recommend_crashkernel()
    if [[ $_arch == "x86_64" ]] || [[ $_arch == "s390x" ]]; then
    	_ck_cmdline="1G-4G:192M,4G-64G:256M,64G-:512M"
    elif [[ $_arch == "aarch64" ]]; then
-		_ck_cmdline="2G-:448M"
+		# 64K page
+		if [[ $(getconf PAGE_SIZE) -eq 65536 ]]; then
+			_ck_cmdline="2G-:448M"
+		else
+			# based on x86 plus extra = 64M
+			_ck_cmdline="1G-4G:256M,4G-64G:320M,64G-:576M"
+		fi
    elif [[ $_arch == "ppc64le" ]]; then
    	if [[ $_dump_mode == "fadump" ]]; then
    		_ck_cmdline="4G-16G:768M,16G-64G:1G,64G-128G:2G,128G-1T:4G,1T-2T:6G,2T-4T:12G,4T-8T:20G,8T-16T:36G,16T-32T:64G,32T-64T:128G,64T-:180G"

[Baoquan He]

Hi Pingfan,

On 05/23/22 at 04:30pm, Pingfan Liu wrote:
> The arm64 platform supports either 64KB or 4KB page size. In the case of
> 4KB page size, the reserved memory size can be aligned to that on
> x86_64.

Does arm64 support 64KB page size on fedora or RHEL9 currently?

I didn't find the 64KB page size setting in fedora or rhel9, I might
miss something.
> Introducing a new formula for 4KB on arm64, which bases on x86_64 plus
> extra 64MB.
> Signed-off-by: Pingfan Liu
>
> gitlab: https://gitlab.com/coxu/fedora-kexec-tools/-/merge_requests/8
>  kdump-lib.sh | 8 +++++++-
>  1 file changed, 7 insertions(+), 1 deletion(-)
>
> diff --git a/kdump-lib.sh b/kdump-lib.sh
> index b137c89..ef9bd64 100755
> --- a/kdump-lib.sh
> +++ b/kdump-lib.sh
> @@ -853,7 +853,13 @@ kdump_get_arch_recommend_crashkernel()
>    if [[ $_arch == "x86_64" ]] || [[ $_arch == "s390x" ]]; then
>    	_ck_cmdline="1G-4G:192M,4G-64G:256M,64G-:512M"
>    elif [[ $_arch == "aarch64" ]]; then
> _ck_cmdline="2G-:448M"
> # 64K page
> if [[ $(getconf PAGE_SIZE) -eq 65536 ]]; then
> 	_ck_cmdline="2G-:448M"
> else
> 	# based on x86 plus extra = 64M
> 	_ck_cmdline="1G-4G:256M,4G-64G:320M,64G-:576M"
> fi
> elif [[ $_arch == "ppc64le" ]]; then
> if [[ $_dump_mode == "fadump" ]]; then
> 	_ck_cmdline="4G-16G:768M,16G-64G:1G,64G-128G:2G,128G-1T:4G,1T-2T:6G,2T-4T:12G,4T-8T:20G,8T-16T:36G,16T-32T:64G,32T-64T:128G,64T-:180G"
>
> -- 
> 2.31.1
> _______________________________________________
> kexec mailing list -- [e-mail]
> To unsubscribe send an email to [e-mail]
> Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/
> List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
> List Archives: https://lists.fedoraproject.org/archives/list/[e-mail]
> Do not reply to spam on the list, report it: https://pagure.io/fedora-infrastructure

[Pingfan Liu]

Hi Baoquan,

Thanks for your review. Please see the comment inline.

On Tue, May 31, 2022 at 10:46 AM Baoquan He  wrote:
> Hi Pingfan,
>
> On 05/23/22 at 04:30pm, Pingfan Liu wrote:
> > The arm64 platform supports either 64KB or 4KB page size. In the case of
> > 4KB page size, the reserved memory size can be aligned to that on
> > x86_64.
>
> Does arm64 support 64KB page size on fedora or RHEL9 currently?
>
> I didn't find the 64KB page size setting in fedora or rhel9, I might
> miss something.
> > Introducing a new formula for 4KB on arm64, which bases on x86_64 plus
> > extra 64MB.
> > Signed-off-by: Pingfan Liu
> >
> > gitlab: https://gitlab.com/coxu/fedora-kexec-tools/-/merge_requests/8
> >  kdump-lib.sh | 8 +++++++-
> >  1 file changed, 7 insertions(+), 1 deletion(-)
> >
> > diff --git a/kdump-lib.sh b/kdump-lib.sh
> > index b137c89..ef9bd64 100755
> > --- a/kdump-lib.sh
> > +++ b/kdump-lib.sh
> > @@ -853,7 +853,13 @@ kdump_get_arch_recommend_crashkernel()
> >       if [[ $_arch == "x86_64" ]] || [[ $_arch == "s390x" ]]; then
> >               _ck_cmdline="1G-4G:192M,4G-64G:256M,64G-:512M"
> >       elif [[ $_arch == "aarch64" ]]; then
> >         _ck_cmdline="2G-:448M"

This formula comes from 64KB page size.
> >         # 64K page
> >         if [[ $(getconf PAGE_SIZE) -eq 65536 ]]; then
> >                 _ck_cmdline="2G-:448M"

So here keeps the formula for 64KB page size in case it comes back again.

Thanks,

Pingfan
> >         else
> >                 # based on x86 plus extra = 64M
> >                 _ck_cmdline="1G-4G:256M,4G-64G:320M,64G-:576M"
> >         fi
> > elif [[ $_arch == "ppc64le" ]]; then
> >         if [[ $_dump_mode == "fadump" ]]; then
> >                 _ck_cmdline="4G-16G:768M,16G-64G:1G,64G-128G:2G,128G-1T:4G,1T-2T:6G,2T-4T:12G,4T-8T:20G,8T-16T:36G,16T-32T:64G,32T-64T:128G,64T-:180G"
> >
> > --
> > 2.31.1
> > _______________________________________________
> > kexec mailing list -- [e-mail]
> > To unsubscribe send an email to [e-mail]
> > Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/
> > List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
> > List Archives: https://lists.fedoraproject.org/archives/list/[e-mail]
> > Do not reply to spam on the list, report it: https://pagure.io/fedora-infrastructure

[Pingfan Liu]

Coiby has raised a concern about the "getconf PAGESIZE", and worries
it can not be portable on some cloud environments.

I traced down the implementation. It should be archived in glibc by
__getpagesize (void)
{
  assert (GLRO(dl_pagesize) != 0);
  return GLRO(dl_pagesize);
}

And the call stack looks like:
Breakpoint 7, __GI___getpagesize () at
../sysdeps/unix/sysv/linux/getpagesize.c:28
28        assert (GLRO(dl_pagesize) != 0);
(gdb) bt
#0  __GI___getpagesize () at ../sysdeps/unix/sysv/linux/getpagesize.c:28
#1  0x0000fffff7ece5bc in posix_sysconf (name=30) at
../sysdeps/posix/sysconf.c:217
#2  linux_sysconf (name=30) at ../sysdeps/unix/sysv/linux/sysconf.c:121
#3  __GI___sysconf (name=name@entry=30) at
../sysdeps/unix/sysv/linux/aarch64/sysconf.c:49
#4  0x0000fffff7e4322c in _nl_load_locale_from_archive
(category=category@entry=0, namep=namep@entry=0xffffffffebd0) at
loadarchive.c:153
#5  0x0000fffff7e422f0 in _nl_find_locale (locale_path=0x0,
locale_path_len=0, category=category@entry=0,
name=name@entry=0xffffffffebd0) at findlocale.c:151
#6  0x0000fffff7e41e6c in __GI_setlocale (category=category@entry=0,
locale=locale@entry=0xaaaaaaaa34e8 "") at setlocale.c:414
#7  0x0000aaaaaaaa2af4 in main (argc=2, argv=0xffffffffee18) at getconf.c:476

The variable "dl_pagesize" is initialized to EXEC_PAGESIZE during the
compiling time.
And the macro EXEC_PAGESIZE is exported by kernel development header.

So iff glibc can work on the target environment, "getconf PAGESIZE" is
no problem.

Thanks,

Pingfan

On Tue, May 24, 2022 at 10:14 AM Pingfan Liu  wrote:
> The arm64 platform supports either 64KB or 4KB page size. In the case of
> 4KB page size, the reserved memory size can be aligned to that on
> x86_64.
>
> Introducing a new formula for 4KB on arm64, which bases on x86_64 plus
> extra 64MB.
> Signed-off-by: Pingfan Liu
>
> gitlab: https://gitlab.com/coxu/fedora-kexec-tools/-/merge_requests/8
>  kdump-lib.sh | 8 +++++++-
>  1 file changed, 7 insertions(+), 1 deletion(-)
>
> diff --git a/kdump-lib.sh b/kdump-lib.sh
> index b137c89..ef9bd64 100755
> --- a/kdump-lib.sh
> +++ b/kdump-lib.sh
> @@ -853,7 +853,13 @@ kdump_get_arch_recommend_crashkernel()
>         if [[ $_arch == "x86_64" ]] || [[ $_arch == "s390x" ]]; then
>                 _ck_cmdline="1G-4G:192M,4G-64G:256M,64G-:512M"
>         elif [[ $_arch == "aarch64" ]]; then
>           _ck_cmdline="2G-:448M"
>           # 64K page
>           if [[ $(getconf PAGE_SIZE) -eq 65536 ]]; then
>                   _ck_cmdline="2G-:448M"
>           else
>                   # based on x86 plus extra = 64M
>                   _ck_cmdline="1G-4G:256M,4G-64G:320M,64G-:576M"
>           fi
>   elif [[ $_arch == "ppc64le" ]]; then
>           if [[ $_dump_mode == "fadump" ]]; then
>                   _ck_cmdline="4G-16G:768M,16G-64G:1G,64G-128G:2G,128G-1T:4G,1T-2T:6G,2T-4T:12G,4T-8T:20G,8T-16T:36G,16T-32T:64G,32T-64T:128G,64T-:180G"
>
> --
> 2.31.1
> _______________________________________________
> kexec mailing list -- [e-mail]
> To unsubscribe send an email to [e-mail]
> Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/
> List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
> List Archives: https://lists.fedoraproject.org/archives/list/[e-mail]
> Do not reply to spam on the list, report it: https://pagure.io/fedora-infrastructure

[Coiby Xu]

Hi Pingfan,

It seems pagesize is determined by the running kernel. I installed a RHEL9
kernel on RHEL8.5 and different values were returned despite the same
glibc,

[root@aarch64-hv01-guest03 t]# uname -r
4.18.0-348.23.1.el8_5.aarch64
[root@localhost ~]# getconf PAGESIZE
65536

[root@aarch64-hv01-guest03 ~]# uname -r
5.14.0-105.el9.aarch64
[root@aarch64-hv01-guest03 ~]# getconf PAGESIZE
4096

One potential impact is when upgrading RHEL9's kexec-tools using RHEL8's
virt-customize, "getconf PAGESIZE" won't wok as expected. For example,
the above two commands write the same page size i.e. 4096 to
/root/pagesize when using F37's virt-customize,

LIBGUESTFS_BACKEND=direct virt-customize  -a rhel-guest-image-9.0-20220420.0.aarch64.qcow2  --run-command "getconf PAGESIZE > /root/pagesize"
LIBGUESTFS_BACKEND=direct virt-customize  -a rhel-guest-image-8.5-1174.aarch64.qcow2         --run-command "getconf PAGESIZE > /root/pagesize"

I'll ask virt-customize developers how likely is rhel8 virt-customize used
to upgrade rhel9 cloud images.

On Wed, Jun 08, 2022 at 10:24:46AM +0800, Pingfan Liu wrote:
> Coiby has raised a concern about the "getconf PAGESIZE", and worries
> it can not be portable on some cloud environments.
>
> I traced down the implementation. It should be archived in glibc by
> __getpagesize (void)
> {
>  assert (GLRO(dl_pagesize) != 0);
>  return GLRO(dl_pagesize);
> }
>
> And the call stack looks like:
> Breakpoint 7, __GI___getpagesize () at
> ../sysdeps/unix/sysv/linux/getpagesize.c:28
> 28        assert (GLRO(dl_pagesize) != 0);
> (gdb) bt
> #0  __GI___getpagesize () at ../sysdeps/unix/sysv/linux/getpagesize.c:28
> #1  0x0000fffff7ece5bc in posix_sysconf (name=30) at
> ../sysdeps/posix/sysconf.c:217
> #2  linux_sysconf (name=30) at ../sysdeps/unix/sysv/linux/sysconf.c:121
> #3  __GI___sysconf (name=name@entry=30) at
> ../sysdeps/unix/sysv/linux/aarch64/sysconf.c:49
> #4  0x0000fffff7e4322c in _nl_load_locale_from_archive
> (category=category@entry=0, namep=namep@entry=0xffffffffebd0) at
> loadarchive.c:153
> #5  0x0000fffff7e422f0 in _nl_find_locale (locale_path=0x0,
> locale_path_len=0, category=category@entry=0,
> name=name@entry=0xffffffffebd0) at findlocale.c:151
> #6  0x0000fffff7e41e6c in __GI_setlocale (category=category@entry=0,
> locale=locale@entry=0xaaaaaaaa34e8 "") at setlocale.c:414
> #7  0x0000aaaaaaaa2af4 in main (argc=2, argv=0xffffffffee18) at getconf.c:476
>
> The variable "dl_pagesize" is initialized to EXEC_PAGESIZE during the
> compiling time.
> And the macro EXEC_PAGESIZE is exported by kernel development header.
>
> So iff glibc can work on the target environment, "getconf PAGESIZE" is
> no problem.
>
> Thanks,
>
> Pingfan
>
> On Tue, May 24, 2022 at 10:14 AM Pingfan Liu  wrote:
> > The arm64 platform supports either 64KB or 4KB page size. In the case of
> > 4KB page size, the reserved memory size can be aligned to that on
> > x86_64.
> >
> > Introducing a new formula for 4KB on arm64, which bases on x86_64 plus
> > extra 64MB.
> > Signed-off-by: Pingfan Liu
> >
> > gitlab: https://gitlab.com/coxu/fedora-kexec-tools/-/merge_requests/8
> >  kdump-lib.sh | 8 +++++++-
> >  1 file changed, 7 insertions(+), 1 deletion(-)
> >
> > diff --git a/kdump-lib.sh b/kdump-lib.sh
> > index b137c89..ef9bd64 100755
> > --- a/kdump-lib.sh
> > +++ b/kdump-lib.sh
> > @@ -853,7 +853,13 @@ kdump_get_arch_recommend_crashkernel()
> >         if [[ $_arch == "x86_64" ]] || [[ $_arch == "s390x" ]]; then
> >                 _ck_cmdline="1G-4G:192M,4G-64G:256M,64G-:512M"
> >         elif [[ $_arch == "aarch64" ]]; then
> >           _ck_cmdline="2G-:448M"
> >           # 64K page
> >           if [[ $(getconf PAGE_SIZE) -eq 65536 ]]; then
> >                   _ck_cmdline="2G-:448M"
> >           else
> >                   # based on x86 plus extra = 64M
> >                   _ck_cmdline="1G-4G:256M,4G-64G:320M,64G-:576M"
> >           fi
> >   elif [[ $_arch == "ppc64le" ]]; then
> >           if [[ $_dump_mode == "fadump" ]]; then
> >                   _ck_cmdline="4G-16G:768M,16G-64G:1G,64G-128G:2G,128G-1T:4G,1T-2T:6G,2T-4T:12G,4T-8T:20G,8T-16T:36G,16T-32T:64G,32T-64T:128G,64T-:180G"
> >
> > --
> > 2.31.1
> > _______________________________________________
> > kexec mailing list -- [e-mail]
> > To unsubscribe send an email to [e-mail]
> > Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/
> > List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
> > List Archives: https://lists.fedoraproject.org/archives/list/[e-mail]
> > Do not reply to spam on the list, report it: https://pagure.io/fedora-infrastructure

[Coiby Xu]

Hi Laszlo,

We meet a problem that when running "getconf PAGESIZE" with
virt-customize, the returned value is not the page size of the guest
image and it's likely the value of the VM started by virt-customize.
Since you the expert on virt-customize, do you know what's the
recommended way to get the page size of a guest image and how do we tell
if we are inside a virt-customize environment? If there is no good way,
how likely is rhel8 virt-customize used to modify rhel9 cloud images? If
this is not likely to happen, we may simply ignore this case.

On Wed, Jun 08, 2022 at 04:08:14PM +0800, Coiby Xu wrote:
> Hi Pingfan,
>
> It seems pagesize is determined by the running kernel. I installed a RHEL9
> kernel on RHEL8.5 and different values were returned despite the same
> glibc,
>
> [root@aarch64-hv01-guest03 t]# uname -r
> 4.18.0-348.23.1.el8_5.aarch64
> [root@localhost ~]# getconf PAGESIZE
> 65536
>
> [root@aarch64-hv01-guest03 ~]# uname -r
> 5.14.0-105.el9.aarch64
> [root@aarch64-hv01-guest03 ~]# getconf PAGESIZE
> 4096
>
> One potential impact is when upgrading RHEL9's kexec-tools using RHEL8's
> virt-customize, "getconf PAGESIZE" won't wok as expected. For example,
> the above two commands write the same page size i.e. 4096 to
> /root/pagesize when using F37's virt-customize,
>
> LIBGUESTFS_BACKEND=direct virt-customize  -a rhel-guest-image-9.0-20220420.0.aarch64.qcow2  --run-command "getconf PAGESIZE > /root/pagesize"
> LIBGUESTFS_BACKEND=direct virt-customize  -a rhel-guest-image-8.5-1174.aarch64.qcow2         --run-command "getconf PAGESIZE > /root/pagesize"
>
> I'll ask virt-customize developers how likely is rhel8 virt-customize used
> to upgrade rhel9 cloud images.
>
> On Wed, Jun 08, 2022 at 10:24:46AM +0800, Pingfan Liu wrote:
> > Coiby has raised a concern about the "getconf PAGESIZE", and worries
> > it can not be portable on some cloud environments.
> >
> > I traced down the implementation. It should be archived in glibc by
> > __getpagesize (void)
> > {
> > assert (GLRO(dl_pagesize) != 0);
> > return GLRO(dl_pagesize);
> > }
> >
> > And the call stack looks like:
> > Breakpoint 7, __GI___getpagesize () at
> > ../sysdeps/unix/sysv/linux/getpagesize.c:28
> > 28        assert (GLRO(dl_pagesize) != 0);
> > (gdb) bt
> > #0  __GI___getpagesize () at ../sysdeps/unix/sysv/linux/getpagesize.c:28
> > #1  0x0000fffff7ece5bc in posix_sysconf (name=30) at
> > ../sysdeps/posix/sysconf.c:217
> > #2  linux_sysconf (name=30) at ../sysdeps/unix/sysv/linux/sysconf.c:121
> > #3  __GI___sysconf (name=name@entry=30) at
> > ../sysdeps/unix/sysv/linux/aarch64/sysconf.c:49
> > #4  0x0000fffff7e4322c in _nl_load_locale_from_archive
> > (category=category@entry=0, namep=namep@entry=0xffffffffebd0) at
> > loadarchive.c:153
> > #5  0x0000fffff7e422f0 in _nl_find_locale (locale_path=0x0,
> > locale_path_len=0, category=category@entry=0,
> > name=name@entry=0xffffffffebd0) at findlocale.c:151
> > #6  0x0000fffff7e41e6c in __GI_setlocale (category=category@entry=0,
> > locale=locale@entry=0xaaaaaaaa34e8 "") at setlocale.c:414
> > #7  0x0000aaaaaaaa2af4 in main (argc=2, argv=0xffffffffee18) at getconf.c:476
> >
> > The variable "dl_pagesize" is initialized to EXEC_PAGESIZE during the
> > compiling time.
> > And the macro EXEC_PAGESIZE is exported by kernel development header.
> >
> > So iff glibc can work on the target environment, "getconf PAGESIZE" is
> > no problem.
> >
> > Thanks,
> >
> > Pingfan
> >
> > On Tue, May 24, 2022 at 10:14 AM Pingfan Liu  wrote:
> > > The arm64 platform supports either 64KB or 4KB page size. In the case of
> > > 4KB page size, the reserved memory size can be aligned to that on
> > > x86_64.
> > >
> > > Introducing a new formula for 4KB on arm64, which bases on x86_64 plus
> > > extra 64MB.
> > > Signed-off-by: Pingfan Liu
> > >
> > > gitlab: https://gitlab.com/coxu/fedora-kexec-tools/-/merge_requests/8
> > > kdump-lib.sh | 8 +++++++-
> > > 1 file changed, 7 insertions(+), 1 deletion(-)
> > >
> > > diff --git a/kdump-lib.sh b/kdump-lib.sh
> > > index b137c89..ef9bd64 100755
> > > --- a/kdump-lib.sh
> > > +++ b/kdump-lib.sh
> > > @@ -853,7 +853,13 @@ kdump_get_arch_recommend_crashkernel()
> > >        if [[ $_arch == "x86_64" ]] || [[ $_arch == "s390x" ]]; then
> > >                _ck_cmdline="1G-4G:192M,4G-64G:256M,64G-:512M"
> > >        elif [[ $_arch == "aarch64" ]]; then
> > >           _ck_cmdline="2G-:448M"
> > >           # 64K page
> > >           if [[ $(getconf PAGE_SIZE) -eq 65536 ]]; then
> > >                   _ck_cmdline="2G-:448M"
> > >           else
> > >                   # based on x86 plus extra = 64M
> > >                   _ck_cmdline="1G-4G:256M,4G-64G:320M,64G-:576M"
> > >           fi
> > >  elif [[ $_arch == "ppc64le" ]]; then
> > >          if [[ $_dump_mode == "fadump" ]]; then
> > >                  _ck_cmdline="4G-16G:768M,16G-64G:1G,64G-128G:2G,128G-1T:4G,1T-2T:6G,2T-4T:12G,4T-8T:20G,8T-16T:36G,16T-32T:64G,32T-64T:128G,64T-:180G"
> > >
> > > --
> > > 2.31.1
> > > _______________________________________________
> > > kexec mailing list -- [e-mail]
> > > To unsubscribe send an email to [e-mail]
> > > Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/
> > > List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
> > > List Archives: https://lists.fedoraproject.org/archives/list/[e-mail]
> > > Do not reply to spam on the list, report it: https://pagure.io/fedora-infrastructure
>
> -- 
> Best regards,
> Coiby

[Laszlo Ersek]

+Rich

On 06/08/22 11:49, Coiby Xu wrote:
> Hi Laszlo,
>
> We meet a problem that when running "getconf PAGESIZE" with
> virt-customize, the returned value is not the page size of the guest
> image and it's likely the value of the VM started by virt-customize.
> Since you the expert on virt-customize, do you know what's the
> recommended way to get the page size of a guest image and how do we tell
> if we are inside a virt-customize environment? If there is no good way,
> how likely is rhel8 virt-customize used to modify rhel9 cloud images? If
> this is not likely to happen, we may simply ignore this case.

I don't understand (after skimming the below too) how virt-customize
enters the picture here. "page size" is not really a guest filesystem
dependent concept, but a guest *kernel* dependent concept. And
libguestfs (and the dependent virt tools, such as virt-customize) use a
separate kernel, initrd, and small root filesystem (which we together
call "appliance") for manipulating guest disk images. The appliance is
usually collected from host-side packages; so by default you get the
host-side kernel in the appliance when you fire up virt-customize. That
kernel may use a different page size from any kernel that might be
installed on the guest disk.

libguestfs-based tools do attempt to query some information out of the
guest-installed kernels, for example virt-v2v looks at various CONFIG
options to see if various virtio devices are supported by the installed
guest kernel(s). The page size is not something that we look at (to my
knowledgge) -- why do you care about guest pagesize, and why do you
think that libguestfs and/or virt-customize is a good way to determine
it? What is your end goal?

Libguestfs documentation about running commands in general:

https://libguestfs.org/guestfs.3.html#running-commands

Thanks
Laszlo
> On Wed, Jun 08, 2022 at 04:08:14PM +0800, Coiby Xu wrote:
> > Hi Pingfan,
> >
> > It seems pagesize is determined by the running kernel. I installed a
> > RHEL9
> > kernel on RHEL8.5 and different values were returned despite the same
> > glibc,
> >
> > [root@aarch64-hv01-guest03 t]# uname -r
> > 4.18.0-348.23.1.el8_5.aarch64
> > [root@localhost ~]# getconf PAGESIZE
> > 65536
> >
> > [root@aarch64-hv01-guest03 ~]# uname -r
> > 5.14.0-105.el9.aarch64
> > [root@aarch64-hv01-guest03 ~]# getconf PAGESIZE
> > 4096
> >
> > One potential impact is when upgrading RHEL9's kexec-tools using RHEL8's
> > virt-customize, "getconf PAGESIZE" won't wok as expected. For example,
> > the above two commands write the same page size i.e. 4096 to
> > /root/pagesize when using F37's virt-customize,
> >
> > LIBGUESTFS_BACKEND=direct virt-customize  -a
> > rhel-guest-image-9.0-20220420.0.aarch64.qcow2  --run-command "getconf
> > PAGESIZE > /root/pagesize"
> > LIBGUESTFS_BACKEND=direct virt-customize  -a
> > rhel-guest-image-8.5-1174.aarch64.qcow2         --run-command "getconf
> > PAGESIZE > /root/pagesize"
> >
> > I'll ask virt-customize developers how likely is rhel8 virt-customize
> > used
> > to upgrade rhel9 cloud images.
> >
> > On Wed, Jun 08, 2022 at 10:24:46AM +0800, Pingfan Liu wrote:
> > > Coiby has raised a concern about the "getconf PAGESIZE", and worries
> > > it can not be portable on some cloud environments.
> > >
> > > I traced down the implementation. It should be archived in glibc by
> > > __getpagesize (void)
> > > {
> > > assert (GLRO(dl_pagesize) != 0);
> > > return GLRO(dl_pagesize);
> > > }
> > >
> > > And the call stack looks like:
> > > Breakpoint 7, __GI___getpagesize () at
> > > ../sysdeps/unix/sysv/linux/getpagesize.c:28
> > > 28        assert (GLRO(dl_pagesize) != 0);
> > > (gdb) bt
> > > #0  __GI___getpagesize () at ../sysdeps/unix/sysv/linux/getpagesize.c:28
> > > #1  0x0000fffff7ece5bc in posix_sysconf (name=30) at
> > > ../sysdeps/posix/sysconf.c:217
> > > #2  linux_sysconf (name=30) at ../sysdeps/unix/sysv/linux/sysconf.c:121
> > > #3  __GI___sysconf (name=name@entry=30) at
> > > ../sysdeps/unix/sysv/linux/aarch64/sysconf.c:49
> > > #4  0x0000fffff7e4322c in _nl_load_locale_from_archive
> > > (category=category@entry=0, namep=namep@entry=0xffffffffebd0) at
> > > loadarchive.c:153
> > > #5  0x0000fffff7e422f0 in _nl_find_locale (locale_path=0x0,
> > > locale_path_len=0, category=category@entry=0,
> > > name=name@entry=0xffffffffebd0) at findlocale.c:151
> > > #6  0x0000fffff7e41e6c in __GI_setlocale (category=category@entry=0,
> > > locale=locale@entry=0xaaaaaaaa34e8 "") at setlocale.c:414
> > > #7  0x0000aaaaaaaa2af4 in main (argc=2, argv=0xffffffffee18) at
> > > getconf.c:476
> > >
> > > The variable "dl_pagesize" is initialized to EXEC_PAGESIZE during the
> > > compiling time.
> > > And the macro EXEC_PAGESIZE is exported by kernel development header.
> > >
> > > So iff glibc can work on the target environment, "getconf PAGESIZE" is
> > > no problem.
> > >
> > > Thanks,
> > >
> > > Pingfan
> > >
> > > On Tue, May 24, 2022 at 10:14 AM Pingfan Liu  wrote:
> > > > The arm64 platform supports either 64KB or 4KB page size. In the
> > > > case of
> > > > 4KB page size, the reserved memory size can be aligned to that on
> > > > x86_64.
> > > >
> > > > Introducing a new formula for 4KB on arm64, which bases on x86_64 plus
> > > > extra 64MB.
> > > > Signed-off-by: Pingfan Liu
> > > >
> > > > gitlab: https://gitlab.com/coxu/fedora-kexec-tools/-/merge_requests/8
> > > > kdump-lib.sh | 8 +++++++-
> > > > 1 file changed, 7 insertions(+), 1 deletion(-)
> > > >
> > > > diff --git a/kdump-lib.sh b/kdump-lib.sh
> > > > index b137c89..ef9bd64 100755
> > > > --- a/kdump-lib.sh
> > > > +++ b/kdump-lib.sh
> > > > @@ -853,7 +853,13 @@ kdump_get_arch_recommend_crashkernel()
> > > >        if [[ $_arch == "x86_64" ]] || [[ $_arch == "s390x" ]]; then
> > > >                _ck_cmdline="1G-4G:192M,4G-64G:256M,64G-:512M"
> > > >        elif [[ $_arch == "aarch64" ]]; then
> > > > -               _ck_cmdline="2G-:448M"
> > > > +               # 64K page
> > > > +               if [[ $(getconf PAGE_SIZE) -eq 65536 ]]; then
> > > > +                       _ck_cmdline="2G-:448M"
> > > > +               else
> > > > +                       # based on x86 plus extra = 64M
> > > > +                       _ck_cmdline="1G-4G:256M,4G-64G:320M,64G-:576M"
> > > > +               fi
> > > >        elif [[ $_arch == "ppc64le" ]]; then
> > > >                if [[ $_dump_mode == "fadump" ]]; then
> > > >                       
> > > > _ck_cmdline="4G-16G:768M,16G-64G:1G,64G-128G:2G,128G-1T:4G,1T-2T:6G,2T-4T:12G,4T-8T:20G,8T-16T:36G,16T-32T:64G,32T-64T:128G,64T-:180G"
> > > >
> > > > -- 
> > > > 2.31.1
> > > > _______________________________________________
> > > > kexec mailing list -- [e-mail]
> > > > To unsubscribe send an email to [e-mail]
> > > > Fedora Code of Conduct:
> > > > https://docs.fedoraproject.org/en-US/project/code-of-conduct/
> > > > List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
> > > > List Archives:
> > > > https://lists.fedoraproject.org/archives/list/[e-mail]
> > > >
> > > > Do not reply to spam on the list, report it:
> > > > https://pagure.io/fedora-infrastructure
> >
> > -- 
> > Best regards,
> > Coiby

[Richard W.M. Jones]

On Wed, Jun 08, 2022 at 04:34:47PM +0200, Laszlo Ersek wrote:
> +Rich
>
> On 06/08/22 11:49, Coiby Xu wrote:
> > Hi Laszlo,
> >
> > We meet a problem that when running "getconf PAGESIZE" with
> > virt-customize, the returned value is not the page size of the guest
> > image and it's likely the value of the VM started by virt-customize.
> > Since you the expert on virt-customize, do you know what's the
> > recommended way to get the page size of a guest image and how do we tell
> > if we are inside a virt-customize environment? If there is no good way,
> > how likely is rhel8 virt-customize used to modify rhel9 cloud images? If
> > this is not likely to happen, we may simply ignore this case.
>
> I don't understand (after skimming the below too) how virt-customize
> enters the picture here. "page size" is not really a guest filesystem
> dependent concept, but a guest *kernel* dependent concept. And
> libguestfs (and the dependent virt tools, such as virt-customize) use a
> separate kernel, initrd, and small root filesystem (which we together
> call "appliance") for manipulating guest disk images. The appliance is
> usually collected from host-side packages; so by default you get the
> host-side kernel in the appliance when you fire up virt-customize. That
> kernel may use a different page size from any kernel that might be
> installed on the guest disk.
>
> libguestfs-based tools do attempt to query some information out of the
> guest-installed kernels, for example virt-v2v looks at various CONFIG
> options to see if various virtio devices are supported by the installed
> guest kernel(s). The page size is not something that we look at (to my
> knowledgge) -- why do you care about guest pagesize, and why do you
> think that libguestfs and/or virt-customize is a good way to determine
> it? What is your end goal?
>
> Libguestfs documentation about running commands in general:
>
> https://libguestfs.org/guestfs.3.html#running-commands

This isn't something that virt-customize can really handle.  It boots
an appliance using the host kernel and so getconf PAGESIZE running in
that appliance (say, in an RPM %post script) is going to return the
host kernel page size.  Getting the page size from the guest kernel
when it isn't running is painful.  You could in theory look into
/boot/config*, but it might not be present, or might not correspond to
the kernel which is finally installed.

I couldn't find where exactly you're calling getconf.  It doesn't seem
to be present in the Rawhide RPM:
https://src.fedoraproject.org/rpms/kexec-tools/tree/rawhide

In general though, there are lots of ways that RPMs are installed
which don't use the final kernel during installation, so you're going
to hit this problem in many contexts -- eg in containers or clouds
where it's very common to build an image which is used as a template
for many deployments.

Probably best to do the test at runtime right before you need it.

Rich.

[Coiby Xu]

Hi Laszlo and Rich,

Thanks for the quick and detailed explanation! kexec-tools is
responsible for setting the kernel command line parameter crashkernel.
And when there is an update of the default crashkernel value, it will
try to reset crashkernel to the new default value during package update.
A customer could use virt-customize to update kexec-tools and this is
how virt-customize enters the picture.

Pingfan has decided to drop getconf in next version since RHEL9 arm use
4k page size. So there will not be a problem for running virt-customize
to update kexec-tools.

Note I've also left some inline comments.

On Wed, Jun 08, 2022 at 06:09:01PM +0100, Richard W.M. Jones wrote:
> On Wed, Jun 08, 2022 at 04:34:47PM +0200, Laszlo Ersek wrote:
> > +Rich
> >
> > On 06/08/22 11:49, Coiby Xu wrote:
> > > Hi Laszlo,
> > >
> > > We meet a problem that when running "getconf PAGESIZE" with
> > > virt-customize, the returned value is not the page size of the guest
> > > image and it's likely the value of the VM started by virt-customize.
> > > Since you the expert on virt-customize, do you know what's the
> > > recommended way to get the page size of a guest image and how do we tell
> > > if we are inside a virt-customize environment? If there is no good way,
> > > how likely is rhel8 virt-customize used to modify rhel9 cloud images? If
> > > this is not likely to happen, we may simply ignore this case.
> >
> > I don't understand (after skimming the below too) how virt-customize
> > enters the picture here. "page size" is not really a guest filesystem
> > dependent concept, but a guest *kernel* dependent concept. And
> > libguestfs (and the dependent virt tools, such as virt-customize) use a
> > separate kernel, initrd, and small root filesystem (which we together
> > call "appliance") for manipulating guest disk images. The appliance is
> > usually collected from host-side packages; so by default you get the
> > host-side kernel in the appliance when you fire up virt-customize. That
> > kernel may use a different page size from any kernel that might be
> > installed on the guest disk.

Thanks for giving me a big picture of how virt-customize works!
> > libguestfs-based tools do attempt to query some information out of the
> > guest-installed kernels, for example virt-v2v looks at various CONFIG
> > options to see if various virtio devices are supported by the installed
> > guest kernel(s). The page size is not something that we look at (to my
> > knowledgge) -- why do you care about guest pagesize, and why do you
> > think that libguestfs and/or virt-customize is a good way to determine
> > it? What is your end goal?
> >
> > Libguestfs documentation about running commands in general:
> >
> > https://libguestfs.org/guestfs.3.html#running-commands

Thanks for the documentation!
> This isn't something that virt-customize can really handle.  It boots
> an appliance using the host kernel and so getconf PAGESIZE running in
> that appliance (say, in an RPM %post script) is going to return the
> host kernel page size.  Getting the page size from the guest kernel
> when it isn't running is painful.  You could in theory look into
> /boot/config*, but it might not be present, or might not correspond to
> the kernel which is finally installed.
>
> I couldn't find where exactly you're calling getconf.  It doesn't seem
> to be present in the Rawhide RPM:
> https://src.fedoraproject.org/rpms/kexec-tools/tree/rawhide

This patch hasn't been merged due to this concern about getconf. With
this patch, getconf is supposed to be called in this way: kexec-tools
package update -> RPM scriptlet -> kdumpctl get-default-crashkernel ->
getconf.
> In general though, there are lots of ways that RPMs are installed
> which don't use the final kernel during installation, so you're going
> to hit this problem in many contexts -- eg in containers or clouds
> where it's very common to build an image which is used as a template
> for many deployments.

Yes, take osbuild as an example, osbuild exposes the
container=bwrap-osbuild environment variable to tell if we inside the
its container. I'm curious if guest-customize has provided a similar way
in case we need to treat guest-customize specially in the future.
> Probably best to do the test at runtime right before you need it.

Thanks for the suggestion! As said before, Pingfan plans to drop getconf
in next version.
> Rich.
>
> -- 
> Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones
> Read my programming and virtualization blog: http://rwmj.wordpress.com
> Fedora Windows cross-compiler. Compile Windows programs, test, and
> build Windows installers. Over 100 libraries supported.
> http://fedoraproject.org/wiki/MinGW

[Laszlo Ersek]

On 06/09/22 15:59, Coiby Xu wrote:
> This patch hasn't been merged due to this concern about getconf. With
> this patch, getconf is supposed to be called in this way: kexec-tools
> package update -> RPM scriptlet -> kdumpctl get-default-crashkernel ->
> getconf.

You could try replacing the --update option of virt-customize with

--firstboot-command 'sleep 30; dnf -y --best upgrade kexec-tools'

which would make your RPM script run at the next boot of the guest (and
so getconf would query the guest's own kernel).

... The "sleep 30" is a work-around for the network not being up and
running in the guest, that early during the boot process, causing DNF to
fail. (We're just experimenting with nm-online for this, and it's not
working, alas.)

So in the end, with --firstboot-*, you'd trade one set of problems for
another :/

Thanks
Laszlo

[Richard W.M. Jones]

On Thu, Jun 09, 2022 at 09:59:54PM +0800, Coiby Xu wrote:
> Yes, take osbuild as an example, osbuild exposes the
> container=bwrap-osbuild environment variable to tell if we inside the
> its container. I'm curious if guest-customize has provided a similar way
> in case we need to treat guest-customize specially in the future.

I don't think we do, although it's possible we should.

Rich.

[Baoquan He]

On 05/23/22 at 04:30pm, Pingfan Liu wrote:
> The arm64 platform supports either 64KB or 4KB page size. In the case of
> 4KB page size, the reserved memory size can be aligned to that on
> x86_64.
>
> Introducing a new formula for 4KB on arm64, which bases on x86_64 plus
> extra 64MB.

LGTM,

Acked-by: Baoquan He
> Signed-off-by: Pingfan Liu
>
> gitlab: https://gitlab.com/coxu/fedora-kexec-tools/-/merge_requests/8
>  kdump-lib.sh | 8 +++++++-
>  1 file changed, 7 insertions(+), 1 deletion(-)
>
> diff --git a/kdump-lib.sh b/kdump-lib.sh
> index b137c89..ef9bd64 100755
> --- a/kdump-lib.sh
> +++ b/kdump-lib.sh
> @@ -853,7 +853,13 @@ kdump_get_arch_recommend_crashkernel()
>    if [[ $_arch == "x86_64" ]] || [[ $_arch == "s390x" ]]; then
>    	_ck_cmdline="1G-4G:192M,4G-64G:256M,64G-:512M"
>    elif [[ $_arch == "aarch64" ]]; then
> _ck_cmdline="2G-:448M"
> # 64K page
> if [[ $(getconf PAGE_SIZE) -eq 65536 ]]; then
> 	_ck_cmdline="2G-:448M"
> else
> 	# based on x86 plus extra = 64M
> 	_ck_cmdline="1G-4G:256M,4G-64G:320M,64G-:576M"
> fi
> elif [[ $_arch == "ppc64le" ]]; then
> if [[ $_dump_mode == "fadump" ]]; then
> 	_ck_cmdline="4G-16G:768M,16G-64G:1G,64G-128G:2G,128G-1T:4G,1T-2T:6G,2T-4T:12G,4T-8T:20G,8T-16T:36G,16T-32T:64G,32T-64T:128G,64T-:180G"
>
> -- 
> 2.31.1
> _______________________________________________
> kexec mailing list -- [e-mail]
> To unsubscribe send an email to [e-mail]
> Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/
> List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
> List Archives: https://lists.fedoraproject.org/archives/list/[e-mail]
> Do not reply to spam on the list, report it: https://pagure.io/fedora-infrastructure