On 07/28/14 at 08:56am, Vivek Goyal wrote:
On Mon, Jul 28, 2014 at 05:13:52PM +0800, WANG Chao wrote:
On 07/25/14 at 08:01am, Vivek Goyal wrote:
On Fri, Jul 25, 2014 at 01:44:40PM +0800, WANG Chao wrote:
On 07/24/14 at 01:35pm, Vivek Goyal wrote:
On Fri, Jul 25, 2014 at 12:13:45AM +0800, WANG Chao wrote:
Now when mount in /etc/fstab fails, systemd would not consider it as critical and it would continue to boot. In fact, emergency service is triggered, but not in a isolation mode, and it results in the emergency service getting shutdown at some point later of the boot process. We need isolation otherwise we won't see any emergency service.
That is because in kdump initramfs, mount units specified in /etc/fstab are required by "local-fs.target". When any of these mounts fails, local-fs.target fails.
For kdump initramfs, we need to isolate to emergency service on any of the mount failure, that said, every service should be stopped and onlu emergency service would run. But local-fs.target won't trigger that on its failure. That means in case of mount failure, local-fs.target also enters failure state, but all the service will continue without any interruption.
After digging looking into source code of systemd-fstab-generator. I find "x-initrd.mount" using in initramfs mount, will make the mount units required by "initrd-root-fs.target" rather than it's used to be "local-fs.target".
"initrd-root-fs.target" is suitable to us because if it fails, it will isolate to emergency service. That means in case of any mount failure, the emergeny service will start and everything else will stop. We want this effect because we need to take kdump fail-safe action when there's a mount failure.
>From systemd unit point of view, "initrd-root-fs.target" has OnFailureIsolate=yes, but "local-fs.target" doesn't. From systemd.unit(5):
OnFailureIsolate= Takes a boolean argument. If true, the unit listed in OnFailure= will be enqueued in isolation mode, i.e. all units that are not its dependency will be stopped. If this is set, only a single unit may be listed in OnFailure=. Defaults to false.
NOTE: Harald who contributed "x-initrd.mount" in systemd, confirmed that this feature will stay.
Signed-off-by: WANG Chao chaowang@redhat.com
mkdumprd | 6 ++++++ 1 file changed, 6 insertions(+)
diff --git a/mkdumprd b/mkdumprd index ba35800..9ee2f0b 100644 --- a/mkdumprd +++ b/mkdumprd @@ -106,6 +106,12 @@ to_mount() { _fstype=$(findmnt -k -f -n -r -o FSTYPE $_dev) _options=$(findmnt -k -f -n -r -o OPTIONS $_dev) _options=${_options/#ro/rw} #mount fs target as rw in 2nd kernel
- # "x-initrd.mount" mount failure will trigger isolate emergency service
- # W/o this, systemd won't isolate, thus we won't get to emergency
- # This only applicable for local fs mount
- if ! is_nfs_target; then
_options="$_options,x-initrd.mount"- fi
So why is it applicable to local fs mounts only. What will happen in case of remote mount. Do we already isolate to emergency service in that case?
If we use "x-initrd.mount", remote mount will become required by "initrd-root-fs.target", instead of "remote-fs.target". That's how it is handled within systemd internal.
We need remote mount to be required "remote-fs.target", because we need to bring up network before any remote mount and "remote-fs.target" can be a checkpoint of that.
So first of all this explanation should be part of the comment near the code.
Sure. Will do.
Secondly, so what will happen if nfs mount fails and remote-fs.target does not reach. Will we isolate to kdump error handler or not? So handling of remote-fs.target is different from local-fs.target when it comes to emergency shell? I think all this needs to be part of changelog/comment so that reader can understand why are we treating remote mounts and local mounts differently.
In case of nfs mount failure, we won't isolate kdump error handler right away. But will do in the later point in kdump.sh fails because the dump target isn't mounted.
And why do we treat those two cases differently? One problem with local mounts was that we will never hit the kdump.sh because some intermediate targets have not reached. That's why we had to isolate to emergency shell immediately.
So in case of remote mounts we don't have the same issue? We will still hit kdump.sh, no matter what.
In fact, "remote-fs-pre.target" plays a role as a checkpoint for network is brought up.
Say we use "initrd-root-fs.target" for nfs mount, systemd could start to mount nfs before network is OK. nfs mount would fail.
Now we use "remote-fs.target", systemd will start mount nfs after "remote-fs-pre.target". The call graph looks like:
remote-fs-pre.target -> bring up network -> nfs.mount -> remote-fs.target
The reason we treat nfs and local mount diffferent because nfs mount must be started after network is ok but local mount doesn't need network.
Also what happens in case of raw disk? What happens in case of ssh dump when network interface does not show up?
In dracut-initqueue hook, both network and disk will be brought up, if needed. And kdump.sh runs after dracut-initqueue, so network and disk will be there before running kdump.sh.
So if dracut-initqueue fails/timesout, will we isolate to kdump emergency shell?
No. dracut-initqueue is configured by dracut and it's emergency handler is dracut-emergency.service.
But I thought we disabled dracut-emergency.service. So are you saying that if failure happens early, then dracut emergency handler will run. But that will put us on shell instead of rebooting?
We've disabled dracut-emergency.service, so failure of dracut-initqueue is ignored and systemd would just continue to run kdump.sh
Rethinking another round, I think it's better to enable dracut-emergency service but replace it with our own error handler. So that we can fail early and run kdump error handler, instead of ignoring the failure.
Thanks WANG Chao
In case of dracut-initqueue failure, I think we can still proceed kdump.sh and gets failed there.
Not sure what does this mean.
Thanks Vivek