[PATCH] Run kdump capture service before initrd-parse-etc.service

[Hari Bathini]

In FADump mode, when the dump targer is '/home', regular boot is failing
with logs like below:

Starting Reload Configuration from the Real Root...
  [    3.289716] EXT4-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null)
  [FAILED] Failed to mount /kdumproot/home.
  See 'systemctl status kdumproot-home.mount' for details.
  [DEPEND] Dependency failed for Local File Systems.
  [  OK  ] Stopped dracut pre-udev hook.
  [  OK  ] Stopped dracut initqueue hook.
           Starting Kdump Emergency...
  [  OK  ] Stopped dracut cmdline hook.
  [  OK  ] Stopped dracut ask for additional cmdline parameters.
           Starting Setup Virtual Console...
           Starting Create Volatile Files and Directories...
  [  OK  ] Started Cleanup squashfs mounts when switch root.
  [  OK  ] Started Kdump Emergency.

and eventually ending up in emergency mode. 'kdumproot/home' seems to
be mounted successfully though. 'Reload from real root' is altering the
dependency graph leading to this mess. Updated kdump-capture.service
unit file to run before initrd-parse-etc.service to avoid running into
such problems. While here, dropped the deprecated 'OnFailureIsolate'
in favour of 'OnFailureJobMode'. Tested this change successfully in both
KDump and FADump modes with different dump methods (local, ssh, nfs).

Signed-off-by: Hari Bathini 
---
 dracut-kdump-capture.service |    7 +++----
 1 file changed, 3 insertions(+), 4 deletions(-)

diff --git a/dracut-kdump-capture.service b/dracut-kdump-capture.service
index 57139c9..df4a437 100644
--- a/dracut-kdump-capture.service
+++ b/dracut-kdump-capture.service
@@ -7,12 +7,11 @@

[Unit]
 Description=Kdump Vmcore Save Service
-After=initrd.target initrd-parse-etc.service sysroot.mount
-After=dracut-initqueue.service dracut-pre-mount.service dracut-mount.service dracut-pre-pivot.service
-Before=initrd-cleanup.service
+After=initrd-root-fs.target remote-fs.target
+Before=initrd-parse-etc.service
 ConditionPathExists=/etc/initrd-release
 OnFailure=emergency.target
-OnFailureIsolate=yes
+OnFailureJobMode=isolate

[Service]
 Environment=DRACUT_SYSTEMD=1

[Dave Young]

Hi Hari,
On 06/11/19 at 11:30pm, Hari Bathini wrote:
> In FADump mode, when the dump targer is '/home', regular boot is failing
> with logs like below:
>        Starting Reload Configuration from the Real Root...
>
> [    3.289716] EXT4-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null)
>   [FAILED] Failed to mount /kdumproot/home.
>   See 'systemctl status kdumproot-home.mount' for details.
>   [DEPEND] Dependency failed for Local File Systems.
>   [  OK  ] Stopped dracut pre-udev hook.
>   [  OK  ] Stopped dracut initqueue hook.
>            Starting Kdump Emergency...
>   [  OK  ] Stopped dracut cmdline hook.
>   [  OK  ] Stopped dracut ask for additional cmdline parameters.
>            Starting Setup Virtual Console...
>            Starting Create Volatile Files and Directories...
>   [  OK  ] Started Cleanup squashfs mounts when switch root.
>   [  OK  ] Started Kdump Emergency.
>
> and eventually ending up in emergency mode. 'kdumproot/home' seems to
> be mounted successfully though. 'Reload from real root' is altering the
> dependency graph leading to this mess. Updated kdump-capture.service
> unit file to run before initrd-parse-etc.service to avoid running into
> such problems. While here, dropped the deprecated 'OnFailureIsolate'
> in favour of 'OnFailureJobMode'. Tested this change successfully in both
> KDump and FADump modes with different dump methods (local, ssh, nfs).
> Signed-off-by: Hari Bathini
>
> dracut-kdump-capture.service |    7 +++----
>  1 file changed, 3 insertions(+), 4 deletions(-)
>
> diff --git a/dracut-kdump-capture.service b/dracut-kdump-capture.service
> index 57139c9..df4a437 100644
> --- a/dracut-kdump-capture.service
> +++ b/dracut-kdump-capture.service
> @@ -7,12 +7,11 @@
>
> [Unit]
>  Description=Kdump Vmcore Save Service
> -After=initrd.target initrd-parse-etc.service sysroot.mount
> -After=dracut-initqueue.service dracut-pre-mount.service dracut-mount.service dracut-pre-pivot.service
> -Before=initrd-cleanup.service
> +After=initrd-root-fs.target remote-fs.target
> +Before=initrd-parse-etc.service

Still not get why this can solve the problem, but with above change it
could cause kdump fail:
                                  |                  sysroot.mount
                                  |                       |
                                  |                       v
                                  |             initrd-root-fs.target
                                  |                       |
                                  |                       v
                                  v            initrd-parse-etc.service
                           (custom initrd                 |
                            services...)                  v
                                  |            (sysroot-usr.mount and
                                  |             various mounts marked
                                  |               with fstab option
                                  |              x-initrd.mount...)
                                  |                       |
                                  |                       v
                                  |                initrd-fs.target
                                  ______________________ |
                                                         |
                                                          v
                                                     initrd.target
                                                          |
                                                          v
                                                initrd-cleanup.service

We can see there is sysroot-usr.mount and other mounts after
initrd-parse-etc.service,  kdump needs to be after those mounts.

Actually we do not care about rootfs, current kdump scripts can work
without root, if dump to root fs we still regard it as a normal mount.
> ConditionPathExists=/etc/initrd-release
>  OnFailure=emergency.target
> -OnFailureIsolate=yes
> +OnFailureJobMode=isolate

Thanks for updating this, would you mind to repost as a separate patch
for above OnFailureJobMode?
> [Service]
>  Environment=DRACUT_SYSTEMD=1

[Hari Bathini]

Hi Dave,

Thanks for the review..

On 12/06/19 11:33 AM, Dave Young wrote:
> Hi Hari,
> On 06/11/19 at 11:30pm, Hari Bathini wrote:
> > In FADump mode, when the dump targer is '/home', regular boot is failing
> > with logs like below:
> >        Starting Reload Configuration from the Real Root...
> >
> > [    3.289716] EXT4-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null)
> >   [FAILED] Failed to mount /kdumproot/home.
> >   See 'systemctl status kdumproot-home.mount' for details.
> >   [DEPEND] Dependency failed for Local File Systems.
> >   [  OK  ] Stopped dracut pre-udev hook.
> >   [  OK  ] Stopped dracut initqueue hook.
> >            Starting Kdump Emergency...
> >   [  OK  ] Stopped dracut cmdline hook.
> >   [  OK  ] Stopped dracut ask for additional cmdline parameters.
> >            Starting Setup Virtual Console...
> >            Starting Create Volatile Files and Directories...
> >   [  OK  ] Started Cleanup squashfs mounts when switch root.
> >   [  OK  ] Started Kdump Emergency.
> >
> > and eventually ending up in emergency mode. 'kdumproot/home' seems to
> > be mounted successfully though. 'Reload from real root' is altering the
> > dependency graph leading to this mess. Updated kdump-capture.service
> > unit file to run before initrd-parse-etc.service to avoid running into
> > such problems. While here, dropped the deprecated 'OnFailureIsolate'
> > in favour of 'OnFailureJobMode'. Tested this change successfully in both
> > KDump and FADump modes with different dump methods (local, ssh, nfs).
> > Signed-off-by: Hari Bathini
> >
> > dracut-kdump-capture.service |    7 +++----
> >  1 file changed, 3 insertions(+), 4 deletions(-)
> >
> > diff --git a/dracut-kdump-capture.service b/dracut-kdump-capture.service
> > index 57139c9..df4a437 100644
> > --- a/dracut-kdump-capture.service
> > +++ b/dracut-kdump-capture.service
> > @@ -7,12 +7,11 @@
> >
> > [Unit]
> >  Description=Kdump Vmcore Save Service
> > -After=initrd.target initrd-parse-etc.service sysroot.mount
> > -After=dracut-initqueue.service dracut-pre-mount.service dracut-mount.service dracut-pre-pivot.service
> > -Before=initrd-cleanup.service
> > +After=initrd-root-fs.target remote-fs.target
> > +Before=initrd-parse-etc.service
>
> Still not get why this can solve the problem, but with above change it
> could cause kdump fail:
>                                   |                  sysroot.mount
>                                   |                       |
>                                   |                       v
>                                   |             initrd-root-fs.target
>                                   |                       |
>                                   |                       v
>                                   v            initrd-parse-etc.service
>                            (custom initrd                 |
>                             services...)                  v
>                                   |            (sysroot-usr.mount and
>                                   |             various mounts marked
>                                   |               with fstab option
>                                   |              x-initrd.mount...)
>                                   |                       |
>                                   |                       v
>                                   |                initrd-fs.target
>                                   ______________________ |
>                                                          |
>                                                           v
>                                                      initrd.target
>                                                           |
>                                                           v
>                                                 initrd-cleanup.service
>
> We can see there is sysroot-usr.mount and other mounts after
> initrd-parse-etc.service,  kdump needs to be after those mounts.
>
> Actually we do not care about rootfs, current kdump scripts can work
> without root, if dump to root fs we still regard it as a normal mount.

We are covered by systemd-fstab-generator. It generates mount units, for all entries
in /etc/fstab, which finish before local-fs.target. Same goes for sysroot.mount
as well with an entry for /sysroot available in /etc/fstab in KDump/FADump case. 
So, the mount(s) dump capture scripts would be looking for should be ready well
before initrd-root-fs.target (local-fs.target/sysroot.mount)..
> > ConditionPathExists=/etc/initrd-release
> >  OnFailure=emergency.target
> > -OnFailureIsolate=yes
> > +OnFailureJobMode=isolate
>
> Thanks for updating this, would you mind to repost as a separate patch
> for above OnFailureJobMode?

Sure. Will do that..

- Hari

[Dave Young]

Hi Hari,
On 06/13/19 at 12:53pm, Hari Bathini wrote:
> Hi Dave,
>
> Thanks for the review..
>
> On 12/06/19 11:33 AM, Dave Young wrote:
> > Hi Hari,
> > On 06/11/19 at 11:30pm, Hari Bathini wrote:
> > > In FADump mode, when the dump targer is '/home', regular boot is failing
> > > with logs like below:
> > >        Starting Reload Configuration from the Real Root...
> > >
> > > [    3.289716] EXT4-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null)
> > >   [FAILED] Failed to mount /kdumproot/home.
> > >   See 'systemctl status kdumproot-home.mount' for details.
> > >   [DEPEND] Dependency failed for Local File Systems.
> > >   [  OK  ] Stopped dracut pre-udev hook.
> > >   [  OK  ] Stopped dracut initqueue hook.
> > >            Starting Kdump Emergency...
> > >   [  OK  ] Stopped dracut cmdline hook.
> > >   [  OK  ] Stopped dracut ask for additional cmdline parameters.
> > >            Starting Setup Virtual Console...
> > >            Starting Create Volatile Files and Directories...
> > >   [  OK  ] Started Cleanup squashfs mounts when switch root.
> > >   [  OK  ] Started Kdump Emergency.
> > >
> > > and eventually ending up in emergency mode. 'kdumproot/home' seems to
> > > be mounted successfully though. 'Reload from real root' is altering the
> > > dependency graph leading to this mess. Updated kdump-capture.service
> > > unit file to run before initrd-parse-etc.service to avoid running into
> > > such problems. While here, dropped the deprecated 'OnFailureIsolate'
> > > in favour of 'OnFailureJobMode'. Tested this change successfully in both
> > > KDump and FADump modes with different dump methods (local, ssh, nfs).
> > > Signed-off-by: Hari Bathini
> > >
> > > dracut-kdump-capture.service |    7 +++----
> > >  1 file changed, 3 insertions(+), 4 deletions(-)
> > >
> > > diff --git a/dracut-kdump-capture.service b/dracut-kdump-capture.service
> > > index 57139c9..df4a437 100644
> > > --- a/dracut-kdump-capture.service
> > > +++ b/dracut-kdump-capture.service
> > > @@ -7,12 +7,11 @@
> > >
> > > [Unit]
> > >  Description=Kdump Vmcore Save Service
> > > -After=initrd.target initrd-parse-etc.service sysroot.mount
> > > -After=dracut-initqueue.service dracut-pre-mount.service dracut-mount.service dracut-pre-pivot.service
> > > -Before=initrd-cleanup.service
> > > +After=initrd-root-fs.target remote-fs.target
> > > +Before=initrd-parse-etc.service
> >
> > Still not get why this can solve the problem, but with above change it
> > could cause kdump fail:
> >                                   |                  sysroot.mount
> >                                   |                       |
> >                                   |                       v
> >                                   |             initrd-root-fs.target
> >                                   |                       |
> >                                   |                       v
> >                                   v            initrd-parse-etc.service
> >                            (custom initrd                 |
> >                             services...)                  v
> >                                   |            (sysroot-usr.mount and
> >                                   |             various mounts marked
> >                                   |               with fstab option
> >                                   |              x-initrd.mount...)
> >                                   |                       |
> >                                   |                       v
> >                                   |                initrd-fs.target
> >                                   ______________________ |
> >                                                          |
> >                                                           v
> >                                                      initrd.target
> >                                                           |
> >                                                           v
> >                                                 initrd-cleanup.service
> >
> > We can see there is sysroot-usr.mount and other mounts after
> > initrd-parse-etc.service,  kdump needs to be after those mounts.
> >
> > Actually we do not care about rootfs, current kdump scripts can work
> > without root, if dump to root fs we still regard it as a normal mount.
>
> We are covered by systemd-fstab-generator. It generates mount units, for all entries
> in /etc/fstab, which finish before local-fs.target. Same goes for sysroot.mount
> as well with an entry for /sysroot available in /etc/fstab in KDump/FADump case. 
> So, the mount(s) dump capture scripts would be looking for should be ready well
> before initrd-root-fs.target (local-fs.target/sysroot.mount)..

I'm still not sure I can get the whole process clearly,  but since in
the boot process flow, it saids about "x-initrd.mount", we appended the
flag in mkdumprd like below:
    # "x-initrd.mount" mount failure will trigger isolate emergency service
    # W/o this, systemd won't isolate, thus we won't get to emergency.
    # This is not applicable to remote fs mount, because if we use
    # "x-initrd.mount", remote mount will become required by
    # "initrd-root-fs.target", instead of "remote-fs.target". That's how it is
    # handled within systemd internal. We need remote mount to be required
    # "remote-fs.target", because we need to bring up network before any remote
    # mount and "remote-fs.target" can be a checkpoint of that.
    # If remote mount fails, dracut-initqueue will still start and once
    # dracut-initqueue finishes, kdump service will start. Because remote mount
    # failed, kdump service will fail and it will lead to kdump error handler.
    if ! is_nfs_dump_target; then
       _options="$_options,x-initrd.mount"
    fi

Another thing is we still not sure why this can fix the bug,  more
digging should be neede, let's see if Kairui and Pingfan have comments.
> > > ConditionPathExists=/etc/initrd-release
> > >  OnFailure=emergency.target
> > > -OnFailureIsolate=yes
> > > +OnFailureJobMode=isolate
> >
> > Thanks for updating this, would you mind to repost as a separate patch
> > for above OnFailureJobMode?
>
> Sure. Will do that..

Thanks!  I noticed Pingfan sent a patch for that already, so let's just
go with Pingfan's patch for this issue.
> Hari

[Hari Bathini]

On 13/06/19 2:10 PM, Dave Young wrote:
> Hi Hari,
> On 06/13/19 at 12:53pm, Hari Bathini wrote:
> > Hi Dave,
> >
> > Thanks for the review..
> >
> > On 12/06/19 11:33 AM, Dave Young wrote:
> > > Hi Hari,
> > > On 06/11/19 at 11:30pm, Hari Bathini wrote:
> > > > In FADump mode, when the dump targer is '/home', regular boot is failing
> > > > with logs like below:
> > > >        Starting Reload Configuration from the Real Root...
> > > >
> > > > [    3.289716] EXT4-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null)
> > > >   [FAILED] Failed to mount /kdumproot/home.
> > > >   See 'systemctl status kdumproot-home.mount' for details.
> > > >   [DEPEND] Dependency failed for Local File Systems.
> > > >   [  OK  ] Stopped dracut pre-udev hook.
> > > >   [  OK  ] Stopped dracut initqueue hook.
> > > >            Starting Kdump Emergency...
> > > >   [  OK  ] Stopped dracut cmdline hook.
> > > >   [  OK  ] Stopped dracut ask for additional cmdline parameters.
> > > >            Starting Setup Virtual Console...
> > > >            Starting Create Volatile Files and Directories...
> > > >   [  OK  ] Started Cleanup squashfs mounts when switch root.
> > > >   [  OK  ] Started Kdump Emergency.
> > > >
> > > > and eventually ending up in emergency mode. 'kdumproot/home' seems to
> > > > be mounted successfully though. 'Reload from real root' is altering the
> > > > dependency graph leading to this mess. Updated kdump-capture.service
> > > > unit file to run before initrd-parse-etc.service to avoid running into
> > > > such problems. While here, dropped the deprecated 'OnFailureIsolate'
> > > > in favour of 'OnFailureJobMode'. Tested this change successfully in both
> > > > KDump and FADump modes with different dump methods (local, ssh, nfs).
> > > > Signed-off-by: Hari Bathini
> > > >
> > > > dracut-kdump-capture.service |    7 +++----
> > > >  1 file changed, 3 insertions(+), 4 deletions(-)
> > > >
> > > > diff --git a/dracut-kdump-capture.service b/dracut-kdump-capture.service
> > > > index 57139c9..df4a437 100644
> > > > --- a/dracut-kdump-capture.service
> > > > +++ b/dracut-kdump-capture.service
> > > > @@ -7,12 +7,11 @@
> > > >
> > > > [Unit]
> > > >  Description=Kdump Vmcore Save Service
> > > > -After=initrd.target initrd-parse-etc.service sysroot.mount
> > > > -After=dracut-initqueue.service dracut-pre-mount.service dracut-mount.service dracut-pre-pivot.service
> > > > -Before=initrd-cleanup.service
> > > > +After=initrd-root-fs.target remote-fs.target
> > > > +Before=initrd-parse-etc.service
> > >
> > > Still not get why this can solve the problem, but with above change it
> > > could cause kdump fail:
> > >                                   |                  sysroot.mount
> > >                                   |                       |
> > >                                   |                       v
> > >                                   |             initrd-root-fs.target
> > >                                   |                       |
> > >                                   |                       v
> > >                                   v            initrd-parse-etc.service
> > >                            (custom initrd                 |
> > >                             services...)                  v
> > >                                   |            (sysroot-usr.mount and
> > >                                   |             various mounts marked
> > >                                   |               with fstab option
> > >                                   |              x-initrd.mount...)
> > >                                   |                       |
> > >                                   |                       v
> > >                                   |                initrd-fs.target
> > >                                   ______________________ |
> > >                                                          |
> > >                                                           v
> > >                                                      initrd.target
> > >                                                           |
> > >                                                           v
> > >                                                 initrd-cleanup.service
> > >
> > > We can see there is sysroot-usr.mount and other mounts after
> > > initrd-parse-etc.service,  kdump needs to be after those mounts.
> > >
> > > Actually we do not care about rootfs, current kdump scripts can work
> > > without root, if dump to root fs we still regard it as a normal mount.
> >
> > We are covered by systemd-fstab-generator. It generates mount units, for all entries
> > in /etc/fstab, which finish before local-fs.target. Same goes for sysroot.mount
> > as well with an entry for /sysroot available in /etc/fstab in KDump/FADump case. 
> > So, the mount(s) dump capture scripts would be looking for should be ready well
> > before initrd-root-fs.target (local-fs.target/sysroot.mount)..
>
> I'm still not sure I can get the whole process clearly,  but since in
> the boot process flow, it saids about "x-initrd.mount", we appended the
> flag in mkdumprd like below:
>     # "x-initrd.mount" mount failure will trigger isolate emergency service
>     # W/o this, systemd won't isolate, thus we won't get to emergency.
>     # This is not applicable to remote fs mount, because if we use
>     # "x-initrd.mount", remote mount will become required by
>     # "initrd-root-fs.target", instead of "remote-fs.target". That's how it is
>     # handled within systemd internal. We need remote mount to be required
>     # "remote-fs.target", because we need to bring up network before any remote
>     # mount and "remote-fs.target" can be a checkpoint of that.
>     # If remote mount fails, dracut-initqueue will still start and once
>     # dracut-initqueue finishes, kdump service will start. Because remote mount
>     # failed, kdump service will fail and it will lead to kdump error handler.
>     if ! is_nfs_dump_target; then
>        _options="$_options,x-initrd.mount"
>     fi

With fstab-generator, it is no longer relevant which was nicely summarized in
commit 8fd9f017b235 (mkdumprd: remove useless "x-initrd.mount")
> Another thing is we still not sure why this can fix the bug,  more
> digging should be neede, let's see if Kairui and Pingfan have comments.

Niether do I. Someone said we can't trust "init" to do the sane thing anymore for a reason :)
I am sure of two things though. One, initrd-parse-etc.service can change the dependency graph
and it has something to do with why we are failing even after the filesystem mount did
succeed. Two, stopping systemd from getting to that point till kdump-capture.service is
done with is helping...

- Hari

[Dave Young]

On 06/13/19 at 04:36pm, Hari Bathini wrote:
> On 13/06/19 2:10 PM, Dave Young wrote:
> > Hi Hari,
> > On 06/13/19 at 12:53pm, Hari Bathini wrote:
> > > Hi Dave,
> > >
> > > Thanks for the review..
> > >
> > > On 12/06/19 11:33 AM, Dave Young wrote:
> > > > Hi Hari,
> > > > On 06/11/19 at 11:30pm, Hari Bathini wrote:
> > > > > In FADump mode, when the dump targer is '/home', regular boot is failing
> > > > > with logs like below:
> > > > >        Starting Reload Configuration from the Real Root...
> > > > >
> > > > > [    3.289716] EXT4-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null)
> > > > >   [FAILED] Failed to mount /kdumproot/home.
> > > > >   See 'systemctl status kdumproot-home.mount' for details.
> > > > >   [DEPEND] Dependency failed for Local File Systems.
> > > > >   [  OK  ] Stopped dracut pre-udev hook.
> > > > >   [  OK  ] Stopped dracut initqueue hook.
> > > > >            Starting Kdump Emergency...
> > > > >   [  OK  ] Stopped dracut cmdline hook.
> > > > >   [  OK  ] Stopped dracut ask for additional cmdline parameters.
> > > > >            Starting Setup Virtual Console...
> > > > >            Starting Create Volatile Files and Directories...
> > > > >   [  OK  ] Started Cleanup squashfs mounts when switch root.
> > > > >   [  OK  ] Started Kdump Emergency.
> > > > >
> > > > > and eventually ending up in emergency mode. 'kdumproot/home' seems to
> > > > > be mounted successfully though. 'Reload from real root' is altering the
> > > > > dependency graph leading to this mess. Updated kdump-capture.service
> > > > > unit file to run before initrd-parse-etc.service to avoid running into
> > > > > such problems. While here, dropped the deprecated 'OnFailureIsolate'
> > > > > in favour of 'OnFailureJobMode'. Tested this change successfully in both
> > > > > KDump and FADump modes with different dump methods (local, ssh, nfs).
> > > > > Signed-off-by: Hari Bathini
> > > > >
> > > > > dracut-kdump-capture.service |    7 +++----
> > > > >  1 file changed, 3 insertions(+), 4 deletions(-)
> > > > >
> > > > > diff --git a/dracut-kdump-capture.service b/dracut-kdump-capture.service
> > > > > index 57139c9..df4a437 100644
> > > > > --- a/dracut-kdump-capture.service
> > > > > +++ b/dracut-kdump-capture.service
> > > > > @@ -7,12 +7,11 @@
> > > > >
> > > > > [Unit]
> > > > >  Description=Kdump Vmcore Save Service
> > > > > -After=initrd.target initrd-parse-etc.service sysroot.mount
> > > > > -After=dracut-initqueue.service dracut-pre-mount.service dracut-mount.service dracut-pre-pivot.service
> > > > > -Before=initrd-cleanup.service
> > > > > +After=initrd-root-fs.target remote-fs.target
> > > > > +Before=initrd-parse-etc.service
> > > >
> > > > Still not get why this can solve the problem, but with above change it
> > > > could cause kdump fail:
> > > >                                   |                  sysroot.mount
> > > >                                   |                       |
> > > >                                   |                       v
> > > >                                   |             initrd-root-fs.target
> > > >                                   |                       |
> > > >                                   |                       v
> > > >                                   v            initrd-parse-etc.service
> > > >                            (custom initrd                 |
> > > >                             services...)                  v
> > > >                                   |            (sysroot-usr.mount and
> > > >                                   |             various mounts marked
> > > >                                   |               with fstab option
> > > >                                   |              x-initrd.mount...)
> > > >                                   |                       |
> > > >                                   |                       v
> > > >                                   |                initrd-fs.target
> > > >                                   ______________________ |
> > > >                                                          |
> > > >                                                           v
> > > >                                                      initrd.target
> > > >                                                           |
> > > >                                                           v
> > > >                                                 initrd-cleanup.service
> > > >
> > > > We can see there is sysroot-usr.mount and other mounts after
> > > > initrd-parse-etc.service,  kdump needs to be after those mounts.
> > > >
> > > > Actually we do not care about rootfs, current kdump scripts can work
> > > > without root, if dump to root fs we still regard it as a normal mount.
> > >
> > > We are covered by systemd-fstab-generator. It generates mount units, for all entries
> > > in /etc/fstab, which finish before local-fs.target. Same goes for sysroot.mount
> > > as well with an entry for /sysroot available in /etc/fstab in KDump/FADump case. 
> > > So, the mount(s) dump capture scripts would be looking for should be ready well
> > > before initrd-root-fs.target (local-fs.target/sysroot.mount)..
> >
> > I'm still not sure I can get the whole process clearly,  but since in
> > the boot process flow, it saids about "x-initrd.mount", we appended the
> > flag in mkdumprd like below:
> >     # "x-initrd.mount" mount failure will trigger isolate emergency service
> >     # W/o this, systemd won't isolate, thus we won't get to emergency.
> >     # This is not applicable to remote fs mount, because if we use
> >     # "x-initrd.mount", remote mount will become required by
> >     # "initrd-root-fs.target", instead of "remote-fs.target". That's how it is
> >     # handled within systemd internal. We need remote mount to be required
> >     # "remote-fs.target", because we need to bring up network before any remote
> >     # mount and "remote-fs.target" can be a checkpoint of that.
> >     # If remote mount fails, dracut-initqueue will still start and once
> >     # dracut-initqueue finishes, kdump service will start. Because remote mount
> >     # failed, kdump service will fail and it will lead to kdump error handler.
> >     if ! is_nfs_dump_target; then
> >        _options="$_options,x-initrd.mount"
> >     fi
>
> With fstab-generator, it is no longer relevant which was nicely summarized in
> commit 8fd9f017b235 (mkdumprd: remove useless "x-initrd.mount")
> > Another thing is we still not sure why this can fix the bug,  more
> > digging should be neede, let's see if Kairui and Pingfan have comments.
>
> Niether do I. Someone said we can't trust "init" to do the sane thing anymore for a reason :)
> I am sure of two things though. One, initrd-parse-etc.service can change the dependency graph
> and it has something to do with why we are failing even after the filesystem mount did
> succeed. Two, stopping systemd from getting to that point till kdump-capture.service is
> done with is helping...

Oops, I may read some old mkdumprd code, indeed the x-initrd code was
removed in latest kexec-tools.

About the issue itself, still thinking about it, will reply later, also let's
see if Kairui, Pingfan can provide some comments.

Thanks
Dave

[Kairui Song]

On Wed, Jun 12, 2019 at 2:01 AM Hari Bathini  wrote:
> In FADump mode, when the dump targer is '/home', regular boot is failing
> with logs like below:
>        Starting Reload Configuration from the Real Root...
>
> [    3.289716] EXT4-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null)
>   [FAILED] Failed to mount /kdumproot/home.
>   See 'systemctl status kdumproot-home.mount' for details.
>   [DEPEND] Dependency failed for Local File Systems.
>   [  OK  ] Stopped dracut pre-udev hook.
>   [  OK  ] Stopped dracut initqueue hook.
>            Starting Kdump Emergency...
>   [  OK  ] Stopped dracut cmdline hook.
>   [  OK  ] Stopped dracut ask for additional cmdline parameters.
>            Starting Setup Virtual Console...
>            Starting Create Volatile Files and Directories...
>   [  OK  ] Started Cleanup squashfs mounts when switch root.
>   [  OK  ] Started Kdump Emergency.
>
> and eventually ending up in emergency mode. 'kdumproot/home' seems to
> be mounted successfully though. 'Reload from real root' is altering the
> dependency graph leading to this mess. Updated kdump-capture.service
> unit file to run before initrd-parse-etc.service to avoid running into
> such problems. While here, dropped the deprecated 'OnFailureIsolate'
> in favour of 'OnFailureJobMode'. Tested this change successfully in both
> KDump and FADump modes with different dump methods (local, ssh, nfs).
> Signed-off-by: Hari Bathini
>
> dracut-kdump-capture.service |    7 +++----
>  1 file changed, 3 insertions(+), 4 deletions(-)
>
> diff --git a/dracut-kdump-capture.service b/dracut-kdump-capture.service
> index 57139c9..df4a437 100644
> --- a/dracut-kdump-capture.service
> +++ b/dracut-kdump-capture.service
> @@ -7,12 +7,11 @@
>
> [Unit]
>  Description=Kdump Vmcore Save Service
> -After=initrd.target initrd-parse-etc.service sysroot.mount
> -After=dracut-initqueue.service dracut-pre-mount.service dracut-mount.service dracut-pre-pivot.service
> -Before=initrd-cleanup.service
> +After=initrd-root-fs.target remote-fs.target
> +Before=initrd-parse-etc.service
>  ConditionPathExists=/etc/initrd-release
>  OnFailure=emergency.target
> -OnFailureIsolate=yes
> +OnFailureJobMode=isolate
>
> [Service]
>  Environment=DRACUT_SYSTEMD=1

Hi Hari,

I still don't understand how this fixed the issue, and I can't
repdocue it eigher (by generating a fadump initrd on my VM).
The VM I use is a default install which use LVM, not sure if this is
related. /home is on a dedicated disk. I did found some other issue
with fadump initrd with /home as target:

[FAILED] Failed unmounting /kdumproot/home.
[  OK  ] Stopped File System Check on /dev/mapper/rhel-swap.
[  OK  ] Stopped udev Kernel Device Manager.
[  OK  ] Stopped dracut pre-udev hook.
[  OK  ] Stopped dracut cmdline hook.

But then everything goes on well.
It give more log with "systemd.log_target=console
systemd.journald.forward_to_console=1" appended to kernel cmdline:
kdumproot-home.mount: Failed to execute command: No such file or
directory
kdumproot-home.mount: Failed at step EXEC spawning /usr/bin/umount: No
such file or directory

And could be fixed by:
diff --git a/dracut-module-setup.sh b/dracut-module-setup.sh
index 2998c72..5222040 100755
--- a/dracut-module-setup.sh
+++ b/dracut-module-setup.sh
@@ -30,7 +30,7 @@ depends() {
         done
     }

-    if is_squash_available; then
+    if is_squash_available && ! is_fadump_capable; then
         _dep="$_dep squash"
     else
         dwarning "Required modules to build a squashed kdump image is missing!"

I'll post a patch to disable squash for fadump in case any similiar
issue, and can you post more logs with "systemd.log_target=console
systemd.journald.forward_to_console=1" so we can know what actually
caused the failure?

[Hari Bathini]

On 14/06/19 1:16 PM, Kairui Song wrote:
> On Wed, Jun 12, 2019 at 2:01 AM Hari Bathini  wrote:
> > In FADump mode, when the dump targer is '/home', regular boot is failing
> > with logs like below:
> >        Starting Reload Configuration from the Real Root...
> >
> > [    3.289716] EXT4-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null)
> >   [FAILED] Failed to mount /kdumproot/home.
> >   See 'systemctl status kdumproot-home.mount' for details.
> >   [DEPEND] Dependency failed for Local File Systems.
> >   [  OK  ] Stopped dracut pre-udev hook.
> >   [  OK  ] Stopped dracut initqueue hook.
> >            Starting Kdump Emergency...
> >   [  OK  ] Stopped dracut cmdline hook.
> >   [  OK  ] Stopped dracut ask for additional cmdline parameters.
> >            Starting Setup Virtual Console...
> >            Starting Create Volatile Files and Directories...
> >   [  OK  ] Started Cleanup squashfs mounts when switch root.
> >   [  OK  ] Started Kdump Emergency.
> >
> > and eventually ending up in emergency mode. 'kdumproot/home' seems to
> > be mounted successfully though. 'Reload from real root' is altering the
> > dependency graph leading to this mess. Updated kdump-capture.service
> > unit file to run before initrd-parse-etc.service to avoid running into
> > such problems. While here, dropped the deprecated 'OnFailureIsolate'
> > in favour of 'OnFailureJobMode'. Tested this change successfully in both
> > KDump and FADump modes with different dump methods (local, ssh, nfs).
> > Signed-off-by: Hari Bathini
> >
> > dracut-kdump-capture.service |    7 +++----
> >  1 file changed, 3 insertions(+), 4 deletions(-)
> >
> > diff --git a/dracut-kdump-capture.service b/dracut-kdump-capture.service
> > index 57139c9..df4a437 100644
> > --- a/dracut-kdump-capture.service
> > +++ b/dracut-kdump-capture.service
> > @@ -7,12 +7,11 @@
> >
> > [Unit]
> >  Description=Kdump Vmcore Save Service
> > -After=initrd.target initrd-parse-etc.service sysroot.mount
> > -After=dracut-initqueue.service dracut-pre-mount.service dracut-mount.service dracut-pre-pivot.service
> > -Before=initrd-cleanup.service
> > +After=initrd-root-fs.target remote-fs.target
> > +Before=initrd-parse-etc.service
> >  ConditionPathExists=/etc/initrd-release
> >  OnFailure=emergency.target
> > -OnFailureIsolate=yes
> > +OnFailureJobMode=isolate
> >
> > [Service]
> >  Environment=DRACUT_SYSTEMD=1
>
> Hi Hari,

Hi Kairui,

Thanks for the review.
> I still don't understand how this fixed the issue, and I can't
> repdocue it eigher (by generating a fadump initrd on my VM).

All I can say is, I faced issues with mount and initrd-parse-etc.service previously as well.
FWIW, I have seen failure on a system with the below configuration:

# lsblk
    NAME                       MAJ:MIN RM   SIZE RO TYPE MOUNTPOINT
    sda                          8:0    0   100G  0 disk 
    ├─sda1                       8:1    0     4M  0 part 
    ├─sda2                       8:2    0     1G  0 part /boot
    └─sda3                       8:3    0    99G  0 part 
      ├─rhel_ltczep2--lp1-root 253:0    0    50G  0 lvm  /
      └─rhel_ltczep2--lp1-home 253:2    0 158.4G  0 lvm  /home
    sdb                          8:16   0    50G  0 disk 
    └─sdb1                       8:17   0    50G  0 part 
      └─rhel_ltczep2--lp1-home 253:2    0 158.4G  0 lvm  /home
    sdc                          8:32   0    50G  0 disk 
    └─sdc1                       8:33   0    50G  0 part 
      └─rhel_ltczep2--lp1-home 253:2    0 158.4G  0 lvm  /home
    sdd                          8:48   1  14.4G  0 disk 
    └─sdd1                       8:49   1  14.4G  0 part 
      ├─rhel_ltczep2--lp1-swap 253:1    0     5G  0 lvm  [SWAP]
      └─rhel_ltczep2--lp1-home 253:2    0 158.4G  0 lvm  /home

System is hung when "systemd.journald.forward_to_console=1" parameter is passed
to the kernel. So, attaching the console log on the failing system with parameters 
"systemd.log_target=console systemd.log_level=debug" passed to the kernel...
> The VM I use is a default install which use LVM, not sure if this is
> related. /home is on a dedicated disk. I did found some other issue
> with fadump initrd with /home as target:
>
> [FAILED] Failed unmounting /kdumproot/home.
> [  OK  ] Stopped File System Check on /dev/mapper/rhel-swap.
> [  OK  ] Stopped udev Kernel Device Manager.
> [  OK  ] Stopped dracut pre-udev hook.
> [  OK  ] Stopped dracut cmdline hook.
>
> But then everything goes on well.
> It give more log with "systemd.log_target=console
> systemd.journald.forward_to_console=1" appended to kernel cmdline:
> kdumproot-home.mount: Failed to execute command: No such file or
> directory
> kdumproot-home.mount: Failed at step EXEC spawning /usr/bin/umount: No
> such file or directory
>
> And could be fixed by:
> diff --git a/dracut-module-setup.sh b/dracut-module-setup.sh
> index 2998c72..5222040 100755
> --- a/dracut-module-setup.sh
> +++ b/dracut-module-setup.sh
> @@ -30,7 +30,7 @@ depends() {
>          done
>      }
> if is_squash_available; then
> if is_squash_available && ! is_fadump_capable; then
>     _dep="$_dep squash"
> else
>     dwarning "Required modules to build a squashed kdump image is missing!"

This patch helps with the unmount issue (seen even with the patch I proposed)..

- Hari

[Kairui Song]

On Mon, Jun 17, 2019 at 7:13 PM Hari Bathini  wrote:
> On 14/06/19 1:16 PM, Kairui Song wrote:
>
> On Wed, Jun 12, 2019 at 2:01 AM Hari Bathini  wrote:
>
> In FADump mode, when the dump targer is '/home', regular boot is failing
> with logs like below:
>        Starting Reload Configuration from the Real Root...
>
> [    3.289716] EXT4-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null)
>   [FAILED] Failed to mount /kdumproot/home.
>   See 'systemctl status kdumproot-home.mount' for details.
>   [DEPEND] Dependency failed for Local File Systems.
>   [  OK  ] Stopped dracut pre-udev hook.
>   [  OK  ] Stopped dracut initqueue hook.
>            Starting Kdump Emergency...
>   [  OK  ] Stopped dracut cmdline hook.
>   [  OK  ] Stopped dracut ask for additional cmdline parameters.
>            Starting Setup Virtual Console...
>            Starting Create Volatile Files and Directories...
>   [  OK  ] Started Cleanup squashfs mounts when switch root.
>   [  OK  ] Started Kdump Emergency.
>
> and eventually ending up in emergency mode. 'kdumproot/home' seems to
> be mounted successfully though. 'Reload from real root' is altering the
> dependency graph leading to this mess. Updated kdump-capture.service
> unit file to run before initrd-parse-etc.service to avoid running into
> such problems. While here, dropped the deprecated 'OnFailureIsolate'
> in favour of 'OnFailureJobMode'. Tested this change successfully in both
> KDump and FADump modes with different dump methods (local, ssh, nfs).
> Signed-off-by: Hari Bathini
>
> dracut-kdump-capture.service |    7 +++----
>  1 file changed, 3 insertions(+), 4 deletions(-)
>
> diff --git a/dracut-kdump-capture.service b/dracut-kdump-capture.service
> index 57139c9..df4a437 100644
> --- a/dracut-kdump-capture.service
> +++ b/dracut-kdump-capture.service
> @@ -7,12 +7,11 @@
>
> [Unit]
>  Description=Kdump Vmcore Save Service
> -After=initrd.target initrd-parse-etc.service sysroot.mount
> -After=dracut-initqueue.service dracut-pre-mount.service dracut-mount.service dracut-pre-pivot.service
> -Before=initrd-cleanup.service
> +After=initrd-root-fs.target remote-fs.target
> +Before=initrd-parse-etc.service
>  ConditionPathExists=/etc/initrd-release
>  OnFailure=emergency.target
> -OnFailureIsolate=yes
> +OnFailureJobMode=isolate
>
> [Service]
>  Environment=DRACUT_SYSTEMD=1
>
> Hi Hari,
>
> Hi Kairui,
>
> Thanks for the review.
>
> I still don't understand how this fixed the issue, and I can't
> repdocue it eigher (by generating a fadump initrd on my VM).
>
> All I can say is, I faced issues with mount and initrd-parse-etc.service previously as well.
> FWIW, I have seen failure on a system with the below configuration:
>
> # lsblk
> NAME                       MAJ:MIN RM   SIZE RO TYPE MOUNTPOINT
> sda                          8:0    0   100G  0 disk
> ├─sda1                       8:1    0     4M  0 part
> ├─sda2                       8:2    0     1G  0 part /boot
> └─sda3                       8:3    0    99G  0 part
>  ├─rhel_ltczep2--lp1-root 253:0    0    50G  0 lvm  /
>  └─rhel_ltczep2--lp1-home 253:2    0 158.4G  0 lvm  /home
> sdb                          8:16   0    50G  0 disk
> └─sdb1                       8:17   0    50G  0 part
>  └─rhel_ltczep2--lp1-home 253:2    0 158.4G  0 lvm  /home
> sdc                          8:32   0    50G  0 disk
> └─sdc1                       8:33   0    50G  0 part
>  └─rhel_ltczep2--lp1-home 253:2    0 158.4G  0 lvm  /home
> sdd                          8:48   1  14.4G  0 disk
> └─sdd1                       8:49   1  14.4G  0 part
>  ├─rhel_ltczep2--lp1-swap 253:1    0     5G  0 lvm  [SWAP]
>  └─rhel_ltczep2--lp1-home 253:2    0 158.4G  0 lvm  /home
>
> System is hung when "systemd.journald.forward_to_console=1" parameter is passed
> to the kernel. So, attaching the console log on the failing system with parameters
> "systemd.log_target=console systemd.log_level=debug" passed to the kernel...
>
> The VM I use is a default install which use LVM, not sure if this is
> related. /home is on a dedicated disk. I did found some other issue
> with fadump initrd with /home as target:
>
> [FAILED] Failed unmounting /kdumproot/home.
> [  OK  ] Stopped File System Check on /dev/mapper/rhel-swap.
> [  OK  ] Stopped udev Kernel Device Manager.
> [  OK  ] Stopped dracut pre-udev hook.
> [  OK  ] Stopped dracut cmdline hook.
>
> But then everything goes on well.
> It give more log with "systemd.log_target=console
> systemd.journald.forward_to_console=1" appended to kernel cmdline:
> kdumproot-home.mount: Failed to execute command: No such file or
> directory
> kdumproot-home.mount: Failed at step EXEC spawning /usr/bin/umount: No
> such file or directory
>
> And could be fixed by:
> diff --git a/dracut-module-setup.sh b/dracut-module-setup.sh
> index 2998c72..5222040 100755
> --- a/dracut-module-setup.sh
> +++ b/dracut-module-setup.sh
> @@ -30,7 +30,7 @@ depends() {
>          done
>      }
> if is_squash_available; then
> if is_squash_available && ! is_fadump_capable; then
>     _dep="$_dep squash"
> else
>     dwarning "Required modules to build a squashed kdump image is missing!"
>
> This patch helps with the unmount issue (seen even with the patch I proposed)..
> Hari

Thanks for the update! If I get it right, this seems a known systemd bug:
https://github.com/systemd/systemd/issues/10872

As I did see following logs in the attachment:
dev-mapper-rhel_ltczep2\x2d\x2dlp1\x2dhome.device: Failed to send unit
change signal for dev-mapper-rhel_ltczep2\x2d\x2dlp1\x2dhome.device:
Connection reset by peer
Child 981 (mount) died (code=exited, status=0/SUCCESS)
kdumproot-home.mount: Child 981 belongs to kdumproot-home.mount.
kdumproot-home.mount: Mount process exited, code=exited status=0
kdumproot-home.mount: Mount process finished, but there is no mount.
kdumproot-home.mount: Failed with result 'protocol'.
kdumproot-home.mount: Changed mounting -> failed
kdumproot-home.mount: Job kdumproot-home.mount/start finished, result=failed

I'll try again to see if I can reproduce it...