1 p.m.
In FADump mode, when the dump targer is '/home', regular boot is failing
with logs like below:
Starting Reload Configuration from the Real Root...
[ 3.289716] EXT4-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null)
[FAILED] Failed to mount /kdumproot/home.
See 'systemctl status kdumproot-home.mount' for details.
[DEPEND] Dependency failed for Local File Systems.
[ OK ] Stopped dracut pre-udev hook.
[ OK ] Stopped dracut initqueue hook.
Starting Kdump Emergency...
[ OK ] Stopped dracut cmdline hook.
[ OK ] Stopped dracut ask for additional cmdline parameters.
Starting Setup Virtual Console...
Starting Create Volatile Files and Directories...
[ OK ] Started Cleanup squashfs mounts when switch root.
[ OK ] Started Kdump Emergency.
and eventually ending up in emergency mode. 'kdumproot/home' seems to
be mounted successfully though. 'Reload from real root' is altering the
dependency graph leading to this mess. Updated kdump-capture.service
unit file to run before initrd-parse-etc.service to avoid running into
such problems. While here, dropped the deprecated 'OnFailureIsolate'
in favour of 'OnFailureJobMode'. Tested this change successfully in both
KDump and FADump modes with different dump methods (local, ssh, nfs).
Signed-off-by: Hari Bathini <hbathini(a)linux.ibm.com>
---
dracut-kdump-capture.service | 7 +++----
1 file changed, 3 insertions(+), 4 deletions(-)
diff --git a/dracut-kdump-capture.service b/dracut-kdump-capture.service
index 57139c9..df4a437 100644
--- a/dracut-kdump-capture.service
+++ b/dracut-kdump-capture.service
@@ -7,12 +7,11 @@
[Unit]
Description=Kdump Vmcore Save Service
-After=initrd.target initrd-parse-etc.service sysroot.mount
-After=dracut-initqueue.service dracut-pre-mount.service dracut-mount.service
dracut-pre-pivot.service
-Before=initrd-cleanup.service
+After=initrd-root-fs.target remote-fs.target
+Before=initrd-parse-etc.service
ConditionPathExists=/etc/initrd-release
OnFailure=emergency.target
-OnFailureIsolate=yes
+OnFailureJobMode=isolate
[Service]
Environment=DRACUT_SYSTEMD=1
1:03 a.m.
Hi Hari,
On 06/11/19 at 11:30pm, Hari Bathini wrote:
In FADump mode, when the dump targer is '/home', regular boot
is failing
with logs like below:
Starting Reload Configuration from the Real Root...
[ 3.289716] EXT4-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null)
[FAILED] Failed to mount /kdumproot/home.
See 'systemctl status kdumproot-home.mount' for details.
[DEPEND] Dependency failed for Local File Systems.
[ OK ] Stopped dracut pre-udev hook.
[ OK ] Stopped dracut initqueue hook.
Starting Kdump Emergency...
[ OK ] Stopped dracut cmdline hook.
[ OK ] Stopped dracut ask for additional cmdline parameters.
Starting Setup Virtual Console...
Starting Create Volatile Files and Directories...
[ OK ] Started Cleanup squashfs mounts when switch root.
[ OK ] Started Kdump Emergency.
and eventually ending up in emergency mode. 'kdumproot/home' seems to
be mounted successfully though. 'Reload from real root' is altering the
dependency graph leading to this mess. Updated kdump-capture.service
unit file to run before initrd-parse-etc.service to avoid running into
such problems. While here, dropped the deprecated 'OnFailureIsolate'
in favour of 'OnFailureJobMode'. Tested this change successfully in both
KDump and FADump modes with different dump methods (local, ssh, nfs).
Signed-off-by: Hari Bathini <hbathini(a)linux.ibm.com>
---
dracut-kdump-capture.service | 7 +++----
1 file changed, 3 insertions(+), 4 deletions(-)
diff --git a/dracut-kdump-capture.service b/dracut-kdump-capture.service
index 57139c9..df4a437 100644
--- a/dracut-kdump-capture.service
+++ b/dracut-kdump-capture.service
@@ -7,12 +7,11 @@
[Unit]
Description=Kdump Vmcore Save Service
-After=initrd.target initrd-parse-etc.service sysroot.mount
-After=dracut-initqueue.service dracut-pre-mount.service dracut-mount.service
dracut-pre-pivot.service
-Before=initrd-cleanup.service
+After=initrd-root-fs.target remote-fs.target
+Before=initrd-parse-etc.service
Still not get why this can solve the problem, but with above change it
could cause kdump fail:
| sysroot.mount
| |
| v
| initrd-root-fs.target
| |
| v
v initrd-parse-etc.service
(custom initrd |
services...) v
| (sysroot-usr.mount and
| various mounts marked
| with fstab option
| x-initrd.mount...)
| |
| v
| initrd-fs.target
\______________________ |
\|
v
initrd.target
|
v
initrd-cleanup.service
We can see there is sysroot-usr.mount and other mounts after
initrd-parse-etc.service, kdump needs to be after those mounts.
Actually we do not care about rootfs, current kdump scripts can work
without root, if dump to root fs we still regard it as a normal mount.
ConditionPathExists=/etc/initrd-release
OnFailure=emergency.target
-OnFailureIsolate=yes
+OnFailureJobMode=isolate
Thanks for updating this, would you mind to repost as a separate patch
for above OnFailureJobMode?
>
> [Service]
> Environment=DRACUT_SYSTEMD=1
>
2:23 a.m.
Hi Dave,
Thanks for the review..
On 12/06/19 11:33 AM, Dave Young wrote:
Hi Hari,
On 06/11/19 at 11:30pm, Hari Bathini wrote:
> In FADump mode, when the dump targer is '/home', regular boot is failing
> with logs like below:
>
> Starting Reload Configuration from the Real Root...
> [ 3.289716] EXT4-fs (dm-1): mounted filesystem with ordered data mode. Opts:
(null)
> [FAILED] Failed to mount /kdumproot/home.
> See 'systemctl status kdumproot-home.mount' for details.
> [DEPEND] Dependency failed for Local File Systems.
> [ OK ] Stopped dracut pre-udev hook.
> [ OK ] Stopped dracut initqueue hook.
> Starting Kdump Emergency...
> [ OK ] Stopped dracut cmdline hook.
> [ OK ] Stopped dracut ask for additional cmdline parameters.
> Starting Setup Virtual Console...
> Starting Create Volatile Files and Directories...
> [ OK ] Started Cleanup squashfs mounts when switch root.
> [ OK ] Started Kdump Emergency.
>
> and eventually ending up in emergency mode. 'kdumproot/home' seems to
> be mounted successfully though. 'Reload from real root' is altering the
> dependency graph leading to this mess. Updated kdump-capture.service
> unit file to run before initrd-parse-etc.service to avoid running into
> such problems. While here, dropped the deprecated 'OnFailureIsolate'
> in favour of 'OnFailureJobMode'. Tested this change successfully in both
> KDump and FADump modes with different dump methods (local, ssh, nfs).
>
> Signed-off-by: Hari Bathini <hbathini(a)linux.ibm.com>
> ---
> dracut-kdump-capture.service | 7 +++----
> 1 file changed, 3 insertions(+), 4 deletions(-)
>
> diff --git a/dracut-kdump-capture.service b/dracut-kdump-capture.service
> index 57139c9..df4a437 100644
> --- a/dracut-kdump-capture.service
> +++ b/dracut-kdump-capture.service
> @@ -7,12 +7,11 @@
>
> [Unit]
> Description=Kdump Vmcore Save Service
> -After=initrd.target initrd-parse-etc.service sysroot.mount
> -After=dracut-initqueue.service dracut-pre-mount.service dracut-mount.service
dracut-pre-pivot.service
> -Before=initrd-cleanup.service
> +After=initrd-root-fs.target remote-fs.target
> +Before=initrd-parse-etc.service
Still not get why this can solve the problem, but with above change it
could cause kdump fail:
| sysroot.mount
| |
| v
| initrd-root-fs.target
| |
| v
v initrd-parse-etc.service
(custom initrd |
services...) v
| (sysroot-usr.mount and
| various mounts marked
| with fstab option
| x-initrd.mount...)
| |
| v
| initrd-fs.target
\______________________ |
\|
v
initrd.target
|
v
initrd-cleanup.service
We can see there is sysroot-usr.mount and other mounts after
initrd-parse-etc.service, kdump needs to be after those mounts.
Actually we do not care about rootfs, current kdump scripts can work
without root, if dump to root fs we still regard it as a normal mount.
We are covered by systemd-fstab-generator. It generates mount units, for all entries
in /etc/fstab, which finish before local-fs.target. Same goes for sysroot.mount
as well with an entry for /sysroot available in /etc/fstab in KDump/FADump case.
So, the mount(s) dump capture scripts would be looking for should be ready well
before initrd-root-fs.target (local-fs.target/sysroot.mount)..
> ConditionPathExists=/etc/initrd-release
> OnFailure=emergency.target
> -OnFailureIsolate=yes
> +OnFailureJobMode=isolate
Thanks for updating this, would you mind to repost as a separate patch
for above OnFailureJobMode?
Sure. Will do that..
- Hari
3:40 a.m.
Hi Hari,
On 06/13/19 at 12:53pm, Hari Bathini wrote:
Hi Dave,
Thanks for the review..
On 12/06/19 11:33 AM, Dave Young wrote:
> Hi Hari,
> On 06/11/19 at 11:30pm, Hari Bathini wrote:
>> In FADump mode, when the dump targer is '/home', regular boot is
failing
>> with logs like below:
>>
>> Starting Reload Configuration from the Real Root...
>> [ 3.289716] EXT4-fs (dm-1): mounted filesystem with ordered data mode.
Opts: (null)
>> [FAILED] Failed to mount /kdumproot/home.
>> See 'systemctl status kdumproot-home.mount' for details.
>> [DEPEND] Dependency failed for Local File Systems.
>> [ OK ] Stopped dracut pre-udev hook.
>> [ OK ] Stopped dracut initqueue hook.
>> Starting Kdump Emergency...
>> [ OK ] Stopped dracut cmdline hook.
>> [ OK ] Stopped dracut ask for additional cmdline parameters.
>> Starting Setup Virtual Console...
>> Starting Create Volatile Files and Directories...
>> [ OK ] Started Cleanup squashfs mounts when switch root.
>> [ OK ] Started Kdump Emergency.
>>
>> and eventually ending up in emergency mode. 'kdumproot/home' seems to
>> be mounted successfully though. 'Reload from real root' is altering the
>> dependency graph leading to this mess. Updated kdump-capture.service
>> unit file to run before initrd-parse-etc.service to avoid running into
>> such problems. While here, dropped the deprecated 'OnFailureIsolate'
>> in favour of 'OnFailureJobMode'. Tested this change successfully in
both
>> KDump and FADump modes with different dump methods (local, ssh, nfs).
>>
>> Signed-off-by: Hari Bathini <hbathini(a)linux.ibm.com>
>> ---
>> dracut-kdump-capture.service | 7 +++----
>> 1 file changed, 3 insertions(+), 4 deletions(-)
>>
>> diff --git a/dracut-kdump-capture.service b/dracut-kdump-capture.service
>> index 57139c9..df4a437 100644
>> --- a/dracut-kdump-capture.service
>> +++ b/dracut-kdump-capture.service
>> @@ -7,12 +7,11 @@
>>
>> [Unit]
>> Description=Kdump Vmcore Save Service
>> -After=initrd.target initrd-parse-etc.service sysroot.mount
>> -After=dracut-initqueue.service dracut-pre-mount.service dracut-mount.service
dracut-pre-pivot.service
>> -Before=initrd-cleanup.service
>> +After=initrd-root-fs.target remote-fs.target
>> +Before=initrd-parse-etc.service
> Still not get why this can solve the problem, but with above change it
> could cause kdump fail:
> | sysroot.mount
> | |
> | v
> | initrd-root-fs.target
> | |
> | v
> v initrd-parse-etc.service
> (custom initrd |
> services...) v
> | (sysroot-usr.mount and
> | various mounts marked
> | with fstab option
> | x-initrd.mount...)
> | |
> | v
> | initrd-fs.target
> \______________________ |
> \|
> v
> initrd.target
> |
> v
> initrd-cleanup.service
>
>
> We can see there is sysroot-usr.mount and other mounts after
> initrd-parse-etc.service, kdump needs to be after those mounts.
>
> Actually we do not care about rootfs, current kdump scripts can work
> without root, if dump to root fs we still regard it as a normal mount.
We are covered by systemd-fstab-generator. It generates mount units, for all entries
in /etc/fstab, which finish before local-fs.target. Same goes for sysroot.mount
as well with an entry for /sysroot available in /etc/fstab in KDump/FADump case.
So, the mount(s) dump capture scripts would be looking for should be ready well
before initrd-root-fs.target (local-fs.target/sysroot.mount)..
I'm still not sure I can get the whole process clearly, but since in
the boot process flow, it saids about "x-initrd.mount", we appended the
flag in mkdumprd like below:
# "x-initrd.mount" mount failure will trigger isolate emergency service
# W/o this, systemd won't isolate, thus we won't get to emergency.
# This is not applicable to remote fs mount, because if we use
# "x-initrd.mount", remote mount will become required by
# "initrd-root-fs.target", instead of "remote-fs.target".
That's how it is
# handled within systemd internal. We need remote mount to be required
# "remote-fs.target", because we need to bring up network before any remote
# mount and "remote-fs.target" can be a checkpoint of that.
# If remote mount fails, dracut-initqueue will still start and once
# dracut-initqueue finishes, kdump service will start. Because remote mount
# failed, kdump service will fail and it will lead to kdump error handler.
if ! is_nfs_dump_target; then
_options="$_options,x-initrd.mount"
fi
Another thing is we still not sure why this can fix the bug, more
digging should be neede, let's see if Kairui and Pingfan have comments.
>
>> ConditionPathExists=/etc/initrd-release
>> OnFailure=emergency.target
>> -OnFailureIsolate=yes
>> +OnFailureJobMode=isolate
> Thanks for updating this, would you mind to repost as a separate patch
> for above OnFailureJobMode?
Sure. Will do that..
Thanks! I noticed Pingfan sent a patch for that already, so let's just
go with Pingfan's patch for this issue.
>
> - Hari
>
6:06 a.m.
On 13/06/19 2:10 PM, Dave Young wrote:
Hi Hari,
On 06/13/19 at 12:53pm, Hari Bathini wrote:
> Hi Dave,
>
> Thanks for the review..
>
> On 12/06/19 11:33 AM, Dave Young wrote:
>> Hi Hari,
>> On 06/11/19 at 11:30pm, Hari Bathini wrote:
>>> In FADump mode, when the dump targer is '/home', regular boot is
failing
>>> with logs like below:
>>>
>>> Starting Reload Configuration from the Real Root...
>>> [ 3.289716] EXT4-fs (dm-1): mounted filesystem with ordered data mode.
Opts: (null)
>>> [FAILED] Failed to mount /kdumproot/home.
>>> See 'systemctl status kdumproot-home.mount' for details.
>>> [DEPEND] Dependency failed for Local File Systems.
>>> [ OK ] Stopped dracut pre-udev hook.
>>> [ OK ] Stopped dracut initqueue hook.
>>> Starting Kdump Emergency...
>>> [ OK ] Stopped dracut cmdline hook.
>>> [ OK ] Stopped dracut ask for additional cmdline parameters.
>>> Starting Setup Virtual Console...
>>> Starting Create Volatile Files and Directories...
>>> [ OK ] Started Cleanup squashfs mounts when switch root.
>>> [ OK ] Started Kdump Emergency.
>>>
>>> and eventually ending up in emergency mode. 'kdumproot/home' seems
to
>>> be mounted successfully though. 'Reload from real root' is altering
the
>>> dependency graph leading to this mess. Updated kdump-capture.service
>>> unit file to run before initrd-parse-etc.service to avoid running into
>>> such problems. While here, dropped the deprecated 'OnFailureIsolate'
>>> in favour of 'OnFailureJobMode'. Tested this change successfully in
both
>>> KDump and FADump modes with different dump methods (local, ssh, nfs).
>>>
>>> Signed-off-by: Hari Bathini <hbathini(a)linux.ibm.com>
>>> ---
>>> dracut-kdump-capture.service | 7 +++----
>>> 1 file changed, 3 insertions(+), 4 deletions(-)
>>>
>>> diff --git a/dracut-kdump-capture.service b/dracut-kdump-capture.service
>>> index 57139c9..df4a437 100644
>>> --- a/dracut-kdump-capture.service
>>> +++ b/dracut-kdump-capture.service
>>> @@ -7,12 +7,11 @@
>>>
>>> [Unit]
>>> Description=Kdump Vmcore Save Service
>>> -After=initrd.target initrd-parse-etc.service sysroot.mount
>>> -After=dracut-initqueue.service dracut-pre-mount.service dracut-mount.service
dracut-pre-pivot.service
>>> -Before=initrd-cleanup.service
>>> +After=initrd-root-fs.target remote-fs.target
>>> +Before=initrd-parse-etc.service
>> Still not get why this can solve the problem, but with above change it
>> could cause kdump fail:
>> | sysroot.mount
>> | |
>> | v
>> | initrd-root-fs.target
>> | |
>> | v
>> v initrd-parse-etc.service
>> (custom initrd |
>> services...) v
>> | (sysroot-usr.mount and
>> | various mounts marked
>> | with fstab option
>> | x-initrd.mount...)
>> | |
>> | v
>> | initrd-fs.target
>> \______________________ |
>> \|
>> v
>> initrd.target
>> |
>> v
>> initrd-cleanup.service
>>
>>
>> We can see there is sysroot-usr.mount and other mounts after
>> initrd-parse-etc.service, kdump needs to be after those mounts.
>>
>> Actually we do not care about rootfs, current kdump scripts can work
>> without root, if dump to root fs we still regard it as a normal mount.
> We are covered by systemd-fstab-generator. It generates mount units, for all entries
> in /etc/fstab, which finish before local-fs.target. Same goes for sysroot.mount
> as well with an entry for /sysroot available in /etc/fstab in KDump/FADump case.
> So, the mount(s) dump capture scripts would be looking for should be ready well
> before initrd-root-fs.target (local-fs.target/sysroot.mount)..
I'm still not sure I can get the whole process clearly, but since in
the boot process flow, it saids about "x-initrd.mount", we appended the
flag in mkdumprd like below:
# "x-initrd.mount" mount failure will trigger isolate emergency service
# W/o this, systemd won't isolate, thus we won't get to emergency.
# This is not applicable to remote fs mount, because if we use
# "x-initrd.mount", remote mount will become required by
# "initrd-root-fs.target", instead of "remote-fs.target".
That's how it is
# handled within systemd internal. We need remote mount to be required
# "remote-fs.target", because we need to bring up network before any
remote
# mount and "remote-fs.target" can be a checkpoint of that.
# If remote mount fails, dracut-initqueue will still start and once
# dracut-initqueue finishes, kdump service will start. Because remote mount
# failed, kdump service will fail and it will lead to kdump error handler.
if ! is_nfs_dump_target; then
_options="$_options,x-initrd.mount"
fi
With fstab-generator, it is no longer relevant which was nicely summarized in
commit 8fd9f017b235 (mkdumprd: remove useless "x-initrd.mount")
Another thing is we still not sure why this can fix the bug, more
digging should be neede, let's see if Kairui and Pingfan have comments.
Niether do I. Someone said we can't trust "init" to do the sane thing
anymore for a reason :)
I am sure of two things though. One, initrd-parse-etc.service can change the dependency
graph
and it has something to do with why we are failing even after the filesystem mount did
succeed. Two, stopping systemd from getting to that point till kdump-capture.service is
done with is helping...
- Hari
7 a.m.
On 06/13/19 at 04:36pm, Hari Bathini wrote:
On 13/06/19 2:10 PM, Dave Young wrote:
> Hi Hari,
> On 06/13/19 at 12:53pm, Hari Bathini wrote:
>> Hi Dave,
>>
>> Thanks for the review..
>>
>> On 12/06/19 11:33 AM, Dave Young wrote:
>>> Hi Hari,
>>> On 06/11/19 at 11:30pm, Hari Bathini wrote:
>>>> In FADump mode, when the dump targer is '/home', regular boot is
failing
>>>> with logs like below:
>>>>
>>>> Starting Reload Configuration from the Real Root...
>>>> [ 3.289716] EXT4-fs (dm-1): mounted filesystem with ordered data
mode. Opts: (null)
>>>> [FAILED] Failed to mount /kdumproot/home.
>>>> See 'systemctl status kdumproot-home.mount' for details.
>>>> [DEPEND] Dependency failed for Local File Systems.
>>>> [ OK ] Stopped dracut pre-udev hook.
>>>> [ OK ] Stopped dracut initqueue hook.
>>>> Starting Kdump Emergency...
>>>> [ OK ] Stopped dracut cmdline hook.
>>>> [ OK ] Stopped dracut ask for additional cmdline parameters.
>>>> Starting Setup Virtual Console...
>>>> Starting Create Volatile Files and Directories...
>>>> [ OK ] Started Cleanup squashfs mounts when switch root.
>>>> [ OK ] Started Kdump Emergency.
>>>>
>>>> and eventually ending up in emergency mode. 'kdumproot/home'
seems to
>>>> be mounted successfully though. 'Reload from real root' is
altering the
>>>> dependency graph leading to this mess. Updated kdump-capture.service
>>>> unit file to run before initrd-parse-etc.service to avoid running into
>>>> such problems. While here, dropped the deprecated
'OnFailureIsolate'
>>>> in favour of 'OnFailureJobMode'. Tested this change successfully
in both
>>>> KDump and FADump modes with different dump methods (local, ssh, nfs).
>>>>
>>>> Signed-off-by: Hari Bathini <hbathini(a)linux.ibm.com>
>>>> ---
>>>> dracut-kdump-capture.service | 7 +++----
>>>> 1 file changed, 3 insertions(+), 4 deletions(-)
>>>>
>>>> diff --git a/dracut-kdump-capture.service
b/dracut-kdump-capture.service
>>>> index 57139c9..df4a437 100644
>>>> --- a/dracut-kdump-capture.service
>>>> +++ b/dracut-kdump-capture.service
>>>> @@ -7,12 +7,11 @@
>>>>
>>>> [Unit]
>>>> Description=Kdump Vmcore Save Service
>>>> -After=initrd.target initrd-parse-etc.service sysroot.mount
>>>> -After=dracut-initqueue.service dracut-pre-mount.service
dracut-mount.service dracut-pre-pivot.service
>>>> -Before=initrd-cleanup.service
>>>> +After=initrd-root-fs.target remote-fs.target
>>>> +Before=initrd-parse-etc.service
>>> Still not get why this can solve the problem, but with above change it
>>> could cause kdump fail:
>>> | sysroot.mount
>>> | |
>>> | v
>>> | initrd-root-fs.target
>>> | |
>>> | v
>>> v initrd-parse-etc.service
>>> (custom initrd |
>>> services...) v
>>> | (sysroot-usr.mount and
>>> | various mounts marked
>>> | with fstab option
>>> | x-initrd.mount...)
>>> | |
>>> | v
>>> | initrd-fs.target
>>> \______________________ |
>>> \|
>>> v
>>> initrd.target
>>> |
>>> v
>>> initrd-cleanup.service
>>>
>>>
>>> We can see there is sysroot-usr.mount and other mounts after
>>> initrd-parse-etc.service, kdump needs to be after those mounts.
>>>
>>> Actually we do not care about rootfs, current kdump scripts can work
>>> without root, if dump to root fs we still regard it as a normal mount.
>> We are covered by systemd-fstab-generator. It generates mount units, for all
entries
>> in /etc/fstab, which finish before local-fs.target. Same goes for sysroot.mount
>> as well with an entry for /sysroot available in /etc/fstab in KDump/FADump case.
>> So, the mount(s) dump capture scripts would be looking for should be ready well
>> before initrd-root-fs.target (local-fs.target/sysroot.mount)..
> I'm still not sure I can get the whole process clearly, but since in
> the boot process flow, it saids about "x-initrd.mount", we appended the
> flag in mkdumprd like below:
> # "x-initrd.mount" mount failure will trigger isolate emergency
service
> # W/o this, systemd won't isolate, thus we won't get to emergency.
> # This is not applicable to remote fs mount, because if we use
> # "x-initrd.mount", remote mount will become required by
> # "initrd-root-fs.target", instead of "remote-fs.target".
That's how it is
> # handled within systemd internal. We need remote mount to be required
> # "remote-fs.target", because we need to bring up network before any
remote
> # mount and "remote-fs.target" can be a checkpoint of that.
> # If remote mount fails, dracut-initqueue will still start and once
> # dracut-initqueue finishes, kdump service will start. Because remote mount
> # failed, kdump service will fail and it will lead to kdump error handler.
> if ! is_nfs_dump_target; then
> _options="$_options,x-initrd.mount"
> fi
With fstab-generator, it is no longer relevant which was nicely summarized in
commit 8fd9f017b235 (mkdumprd: remove useless "x-initrd.mount")
> Another thing is we still not sure why this can fix the bug, more
> digging should be neede, let's see if Kairui and Pingfan have comments.
Niether do I. Someone said we can't trust "init" to do the sane thing
anymore for a reason :)
I am sure of two things though. One, initrd-parse-etc.service can change the dependency
graph
and it has something to do with why we are failing even after the filesystem mount did
succeed. Two, stopping systemd from getting to that point till kdump-capture.service is
done with is helping...
Oops, I may read some old mkdumprd code, indeed the x-initrd code was
removed in latest kexec-tools.
About the issue itself, still thinking about it, will reply later, also let's
see if Kairui, Pingfan can provide some comments.
Thanks
Dave
2:46 a.m.
On Wed, Jun 12, 2019 at 2:01 AM Hari Bathini <hbathini(a)linux.ibm.com> wrote:
In FADump mode, when the dump targer is '/home', regular boot is failing
with logs like below:
Starting Reload Configuration from the Real Root...
[ 3.289716] EXT4-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null)
[FAILED] Failed to mount /kdumproot/home.
See 'systemctl status kdumproot-home.mount' for details.
[DEPEND] Dependency failed for Local File Systems.
[ OK ] Stopped dracut pre-udev hook.
[ OK ] Stopped dracut initqueue hook.
Starting Kdump Emergency...
[ OK ] Stopped dracut cmdline hook.
[ OK ] Stopped dracut ask for additional cmdline parameters.
Starting Setup Virtual Console...
Starting Create Volatile Files and Directories...
[ OK ] Started Cleanup squashfs mounts when switch root.
[ OK ] Started Kdump Emergency.
and eventually ending up in emergency mode. 'kdumproot/home' seems to
be mounted successfully though. 'Reload from real root' is altering the
dependency graph leading to this mess. Updated kdump-capture.service
unit file to run before initrd-parse-etc.service to avoid running into
such problems. While here, dropped the deprecated 'OnFailureIsolate'
in favour of 'OnFailureJobMode'. Tested this change successfully in both
KDump and FADump modes with different dump methods (local, ssh, nfs).
Signed-off-by: Hari Bathini <hbathini(a)linux.ibm.com>
---
dracut-kdump-capture.service | 7 +++----
1 file changed, 3 insertions(+), 4 deletions(-)
diff --git a/dracut-kdump-capture.service b/dracut-kdump-capture.service
index 57139c9..df4a437 100644
--- a/dracut-kdump-capture.service
+++ b/dracut-kdump-capture.service
@@ -7,12 +7,11 @@
[Unit]
Description=Kdump Vmcore Save Service
-After=initrd.target initrd-parse-etc.service sysroot.mount
-After=dracut-initqueue.service dracut-pre-mount.service dracut-mount.service
dracut-pre-pivot.service
-Before=initrd-cleanup.service
+After=initrd-root-fs.target remote-fs.target
+Before=initrd-parse-etc.service
ConditionPathExists=/etc/initrd-release
OnFailure=emergency.target
-OnFailureIsolate=yes
+OnFailureJobMode=isolate
[Service]
Environment=DRACUT_SYSTEMD=1
Hi Hari,
I still don't understand how this fixed the issue, and I can't
repdocue it eigher (by generating a fadump initrd on my VM).
The VM I use is a default install which use LVM, not sure if this is
related. /home is on a dedicated disk. I did found some other issue
with fadump initrd with /home as target:
[FAILED] Failed unmounting /kdumproot/home.
[ OK ] Stopped File System Check on /dev/mapper/rhel-swap.
[ OK ] Stopped udev Kernel Device Manager.
[ OK ] Stopped dracut pre-udev hook.
[ OK ] Stopped dracut cmdline hook.
But then everything goes on well.
It give more log with "systemd.log_target=console
systemd.journald.forward_to_console=1" appended to kernel cmdline:
kdumproot-home.mount: Failed to execute command: No such file or
directory
kdumproot-home.mount: Failed at step EXEC spawning /usr/bin/umount: No
such file or directory
And could be fixed by:
diff --git a/dracut-module-setup.sh b/dracut-module-setup.sh
index 2998c72..5222040 100755
--- a/dracut-module-setup.sh
+++ b/dracut-module-setup.sh
@@ -30,7 +30,7 @@ depends() {
done
}
- if is_squash_available; then
+ if is_squash_available && ! is_fadump_capable; then
_dep="$_dep squash"
else
dwarning "Required modules to build a squashed kdump image is
missing!"
I'll post a patch to disable squash for fadump in case any similiar
issue, and can you post more logs with "systemd.log_target=console
systemd.journald.forward_to_console=1" so we can know what actually
caused the failure?
--
Best Regards,
Kairui Song
6:13 a.m.
On 14/06/19 1:16 PM, Kairui Song wrote:
On Wed, Jun 12, 2019 at 2:01 AM Hari Bathini
<hbathini(a)linux.ibm.com> wrote:
> In FADump mode, when the dump targer is '/home', regular boot is failing
> with logs like below:
>
> Starting Reload Configuration from the Real Root...
> [ 3.289716] EXT4-fs (dm-1): mounted filesystem with ordered data mode. Opts:
(null)
> [FAILED] Failed to mount /kdumproot/home.
> See 'systemctl status kdumproot-home.mount' for details.
> [DEPEND] Dependency failed for Local File Systems.
> [ OK ] Stopped dracut pre-udev hook.
> [ OK ] Stopped dracut initqueue hook.
> Starting Kdump Emergency...
> [ OK ] Stopped dracut cmdline hook.
> [ OK ] Stopped dracut ask for additional cmdline parameters.
> Starting Setup Virtual Console...
> Starting Create Volatile Files and Directories...
> [ OK ] Started Cleanup squashfs mounts when switch root.
> [ OK ] Started Kdump Emergency.
>
> and eventually ending up in emergency mode. 'kdumproot/home' seems to
> be mounted successfully though. 'Reload from real root' is altering the
> dependency graph leading to this mess. Updated kdump-capture.service
> unit file to run before initrd-parse-etc.service to avoid running into
> such problems. While here, dropped the deprecated 'OnFailureIsolate'
> in favour of 'OnFailureJobMode'. Tested this change successfully in both
> KDump and FADump modes with different dump methods (local, ssh, nfs).
>
> Signed-off-by: Hari Bathini <hbathini(a)linux.ibm.com>
> ---
> dracut-kdump-capture.service | 7 +++----
> 1 file changed, 3 insertions(+), 4 deletions(-)
>
> diff --git a/dracut-kdump-capture.service b/dracut-kdump-capture.service
> index 57139c9..df4a437 100644
> --- a/dracut-kdump-capture.service
> +++ b/dracut-kdump-capture.service
> @@ -7,12 +7,11 @@
>
> [Unit]
> Description=Kdump Vmcore Save Service
> -After=initrd.target initrd-parse-etc.service sysroot.mount
> -After=dracut-initqueue.service dracut-pre-mount.service dracut-mount.service
dracut-pre-pivot.service
> -Before=initrd-cleanup.service
> +After=initrd-root-fs.target remote-fs.target
> +Before=initrd-parse-etc.service
> ConditionPathExists=/etc/initrd-release
> OnFailure=emergency.target
> -OnFailureIsolate=yes
> +OnFailureJobMode=isolate
>
> [Service]
> Environment=DRACUT_SYSTEMD=1
>
Hi Hari,
Hi Kairui,
Thanks for the review.
I still don't understand how this fixed the issue, and I
can't
repdocue it eigher (by generating a fadump initrd on my VM).
All I can say is, I faced issues with mount and initrd-parse-etc.service previously as
well.
FWIW, I have seen failure on a system with the below configuration:
# lsblk
NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT
sda 8:0 0 100G 0 disk
├─sda1 8:1 0 4M 0 part
├─sda2 8:2 0 1G 0 part /boot
└─sda3 8:3 0 99G 0 part
├─rhel_ltczep2--lp1-root 253:0 0 50G 0 lvm /
└─rhel_ltczep2--lp1-home 253:2 0 158.4G 0 lvm /home
sdb 8:16 0 50G 0 disk
└─sdb1 8:17 0 50G 0 part
└─rhel_ltczep2--lp1-home 253:2 0 158.4G 0 lvm /home
sdc 8:32 0 50G 0 disk
└─sdc1 8:33 0 50G 0 part
└─rhel_ltczep2--lp1-home 253:2 0 158.4G 0 lvm /home
sdd 8:48 1 14.4G 0 disk
└─sdd1 8:49 1 14.4G 0 part
├─rhel_ltczep2--lp1-swap 253:1 0 5G 0 lvm [SWAP]
└─rhel_ltczep2--lp1-home 253:2 0 158.4G 0 lvm /home
System is hung when "systemd.journald.forward_to_console=1" parameter is passed
to the kernel. So, attaching the console log on the failing system with parameters
"systemd.log_target=console systemd.log_level=debug" passed to the kernel...
The VM I use is a default install which use LVM, not sure if this is
related. /home is on a dedicated disk. I did found some other issue
with fadump initrd with /home as target:
[FAILED] Failed unmounting /kdumproot/home.
[ OK ] Stopped File System Check on /dev/mapper/rhel-swap.
[ OK ] Stopped udev Kernel Device Manager.
[ OK ] Stopped dracut pre-udev hook.
[ OK ] Stopped dracut cmdline hook.
But then everything goes on well.
It give more log with "systemd.log_target=console
systemd.journald.forward_to_console=1" appended to kernel cmdline:
kdumproot-home.mount: Failed to execute command: No such file or
directory
kdumproot-home.mount: Failed at step EXEC spawning /usr/bin/umount: No
such file or directory
And could be fixed by:
diff --git a/dracut-module-setup.sh b/dracut-module-setup.sh
index 2998c72..5222040 100755
--- a/dracut-module-setup.sh
+++ b/dracut-module-setup.sh
@@ -30,7 +30,7 @@ depends() {
done
}
- if is_squash_available; then
+ if is_squash_available && ! is_fadump_capable; then
_dep="$_dep squash"
else
dwarning "Required modules to build a squashed kdump image is
missing!"
This patch helps with the unmount issue (seen even with the patch I proposed)..
- Hari
7:16 a.m.
On Mon, Jun 17, 2019 at 7:13 PM Hari Bathini <hbathini(a)linux.ibm.com> wrote:
On 14/06/19 1:16 PM, Kairui Song wrote:
On Wed, Jun 12, 2019 at 2:01 AM Hari Bathini <hbathini(a)linux.ibm.com> wrote:
In FADump mode, when the dump targer is '/home', regular boot is failing
with logs like below:
Starting Reload Configuration from the Real Root...
[ 3.289716] EXT4-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null)
[FAILED] Failed to mount /kdumproot/home.
See 'systemctl status kdumproot-home.mount' for details.
[DEPEND] Dependency failed for Local File Systems.
[ OK ] Stopped dracut pre-udev hook.
[ OK ] Stopped dracut initqueue hook.
Starting Kdump Emergency...
[ OK ] Stopped dracut cmdline hook.
[ OK ] Stopped dracut ask for additional cmdline parameters.
Starting Setup Virtual Console...
Starting Create Volatile Files and Directories...
[ OK ] Started Cleanup squashfs mounts when switch root.
[ OK ] Started Kdump Emergency.
and eventually ending up in emergency mode. 'kdumproot/home' seems to
be mounted successfully though. 'Reload from real root' is altering the
dependency graph leading to this mess. Updated kdump-capture.service
unit file to run before initrd-parse-etc.service to avoid running into
such problems. While here, dropped the deprecated 'OnFailureIsolate'
in favour of 'OnFailureJobMode'. Tested this change successfully in both
KDump and FADump modes with different dump methods (local, ssh, nfs).
Signed-off-by: Hari Bathini <hbathini(a)linux.ibm.com>
---
dracut-kdump-capture.service | 7 +++----
1 file changed, 3 insertions(+), 4 deletions(-)
diff --git a/dracut-kdump-capture.service b/dracut-kdump-capture.service
index 57139c9..df4a437 100644
--- a/dracut-kdump-capture.service
+++ b/dracut-kdump-capture.service
@@ -7,12 +7,11 @@
[Unit]
Description=Kdump Vmcore Save Service
-After=initrd.target initrd-parse-etc.service sysroot.mount
-After=dracut-initqueue.service dracut-pre-mount.service dracut-mount.service
dracut-pre-pivot.service
-Before=initrd-cleanup.service
+After=initrd-root-fs.target remote-fs.target
+Before=initrd-parse-etc.service
ConditionPathExists=/etc/initrd-release
OnFailure=emergency.target
-OnFailureIsolate=yes
+OnFailureJobMode=isolate
[Service]
Environment=DRACUT_SYSTEMD=1
Hi Hari,
Hi Kairui,
Thanks for the review.
I still don't understand how this fixed the issue, and I can't
repdocue it eigher (by generating a fadump initrd on my VM).
All I can say is, I faced issues with mount and initrd-parse-etc.service previously as
well.
FWIW, I have seen failure on a system with the below configuration:
# lsblk
NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT
sda 8:0 0 100G 0 disk
├─sda1 8:1 0 4M 0 part
├─sda2 8:2 0 1G 0 part /boot
└─sda3 8:3 0 99G 0 part
├─rhel_ltczep2--lp1-root 253:0 0 50G 0 lvm /
└─rhel_ltczep2--lp1-home 253:2 0 158.4G 0 lvm /home
sdb 8:16 0 50G 0 disk
└─sdb1 8:17 0 50G 0 part
└─rhel_ltczep2--lp1-home 253:2 0 158.4G 0 lvm /home
sdc 8:32 0 50G 0 disk
└─sdc1 8:33 0 50G 0 part
└─rhel_ltczep2--lp1-home 253:2 0 158.4G 0 lvm /home
sdd 8:48 1 14.4G 0 disk
└─sdd1 8:49 1 14.4G 0 part
├─rhel_ltczep2--lp1-swap 253:1 0 5G 0 lvm [SWAP]
└─rhel_ltczep2--lp1-home 253:2 0 158.4G 0 lvm /home
System is hung when "systemd.journald.forward_to_console=1" parameter is
passed
to the kernel. So, attaching the console log on the failing system with parameters
"systemd.log_target=console systemd.log_level=debug" passed to the kernel...
The VM I use is a default install which use LVM, not sure if this is
related. /home is on a dedicated disk. I did found some other issue
with fadump initrd with /home as target:
[FAILED] Failed unmounting /kdumproot/home.
[ OK ] Stopped File System Check on /dev/mapper/rhel-swap.
[ OK ] Stopped udev Kernel Device Manager.
[ OK ] Stopped dracut pre-udev hook.
[ OK ] Stopped dracut cmdline hook.
But then everything goes on well.
It give more log with "systemd.log_target=console
systemd.journald.forward_to_console=1" appended to kernel cmdline:
kdumproot-home.mount: Failed to execute command: No such file or
directory
kdumproot-home.mount: Failed at step EXEC spawning /usr/bin/umount: No
such file or directory
And could be fixed by:
diff --git a/dracut-module-setup.sh b/dracut-module-setup.sh
index 2998c72..5222040 100755
--- a/dracut-module-setup.sh
+++ b/dracut-module-setup.sh
@@ -30,7 +30,7 @@ depends() {
done
}
- if is_squash_available; then
+ if is_squash_available && ! is_fadump_capable; then
_dep="$_dep squash"
else
dwarning "Required modules to build a squashed kdump image is
missing!"
This patch helps with the unmount issue (seen even with the patch I proposed)..
- Hari
Thanks for the update! If I get it right, this seems a known systemd bug:
https://github.com/systemd/systemd/issues/10872
As I did see following logs in the attachment:
dev-mapper-rhel_ltczep2\x2d\x2dlp1\x2dhome.device: Failed to send unit
change signal for dev-mapper-rhel_ltczep2\x2d\x2dlp1\x2dhome.device:
Connection reset by peer
Child 981 (mount) died (code=exited, status=0/SUCCESS)
kdumproot-home.mount: Child 981 belongs to kdumproot-home.mount.
kdumproot-home.mount: Mount process exited, code=exited status=0
kdumproot-home.mount: Mount process finished, but there is no mount.
kdumproot-home.mount: Failed with result 'protocol'.
kdumproot-home.mount: Changed mounting -> failed
kdumproot-home.mount: Job kdumproot-home.mount/start finished, result=failed
I'll try again to see if I can reproduce it...
--
Best Regards,
Kairui Song
1772
days inactive
1778
days old
8 comments
3 participants
participants (3)
-
Dave Young -
Hari Bathini -
Kairui Song