systemd: root device on iscsi

[Jeremy Sanders]

Hi - is an iscsi root device a supported configuration setup with F16 and 
systemd? I asked on #systemd, and got pointed here.

We're using dracut to boot diskless systems with root devices on iscsi. This 
works fine for booting using Fedora 16 and systemd.

Unfortunately, on shutdown the system hangs. It stops the network interface 
before mounting the root device read-only or stopping the iscsi service 
(leaving the fedora iscsi target in a confused state too). On Fedora 14, 
this worked fine with the old fashioned init system.

In the old init system you used to be able to put a "_netdev" flag in fstab, 
which would fix this problem. This doesn't seem to work now.

Is this a simple bug or a bigger issue?

I could imagine we could write some sort of evil systemd service which would 
run busybox (perhaps copied to tmpfs) to kill processes, remount root, kill 
off iscsi and run halt or reboot depending on the run level, before the 
network was stopped. That would be an awful hack (if it even worked).

Jeremy

[Bill Nottingham]

Jeremy Sanders ([e-mail]) said:
> Hi - is an iscsi root device a supported configuration setup with F16 and 
> systemd? I asked on #systemd, and got pointed here.
>
> We're using dracut to boot diskless systems with root devices on iscsi. This 
> works fine for booting using Fedora 16 and systemd.
>
> Unfortunately, on shutdown the system hangs. It stops the network interface 
> before mounting the root device read-only or stopping the iscsi service 
> (leaving the fedora iscsi target in a confused state too). On Fedora 14, 
> this worked fine with the old fashioned init system.
>
> In the old init system you used to be able to put a "_netdev" flag in fstab, 
> which would fix this problem. This doesn't seem to work now.

What are you using for networking - the old network service or
NetworkManager?

Bill

[Jeremy Sanders]

Bill Nottingham wrote:
> Jeremy Sanders ([e-mail]) said:
> > Hi - is an iscsi root device a supported configuration setup with F16 and
> > systemd? I asked on #systemd, and got pointed here.
> >
> > We're using dracut to boot diskless systems with root devices on iscsi.
> > This works fine for booting using Fedora 16 and systemd.
> >
> > Unfortunately, on shutdown the system hangs. It stops the network
> > interface before mounting the root device read-only or stopping the iscsi
> > service (leaving the fedora iscsi target in a confused state too). On
> > Fedora 14, this worked fine with the old fashioned init system.
> >
> > In the old init system you used to be able to put a "_netdev" flag in
> > fstab, which would fix this problem. This doesn't seem to work now.
>
> What are you using for networking - the old network service or
> NetworkManager?

The old network service. The root device is cloned from a non-iscsi 
installation, however, as kickstart didn't like to install onto a non-
partitioned iscsi device: https://bugzilla.redhat.com/show_bug.cgi?id=757106
I thought that the _netdev option would be sufficient to get the ordering 
correct.

Jeremy

[Bill Nottingham]

Jeremy Sanders ([e-mail]) said:
> > What are you using for networking - the old network service or
> > NetworkManager?
>
> The old network service. The root device is cloned from a non-iscsi 
> installation, however, as kickstart didn't like to install onto a non-
> partitioned iscsi device: https://bugzilla.redhat.com/show_bug.cgi?id=757106
> I thought that the _netdev option would be sufficient to get the ordering 
> correct.

Ah, crud. The issue is that we're checking /etc/mtab for _netdev, which is
now /proc/mounts, which doesn't transfer over random options such as these
into its data store.

So, we either need it to start doing that (cc: kzak), or we need to have th
code cross-reference devices against /etc/fstab.

Bill

[Karel Zak]

On Tue, Nov 29, 2011 at 12:18:03PM -0500, Bill Nottingham wrote:
> Jeremy Sanders ([e-mail]) said:
> > > What are you using for networking - the old network service or
> > > NetworkManager?
> >
> > The old network service. The root device is cloned from a non-iscsi 
> > installation, however, as kickstart didn't like to install onto a non-
> > partitioned iscsi device: https://bugzilla.redhat.com/show_bug.cgi?id=757106
> > I thought that the _netdev option would be sufficient to get the ordering 
> > correct.
>
> Ah, crud. The issue is that we're checking /etc/mtab for _netdev, which is
> now /proc/mounts, which doesn't transfer over random options such as these
> into its data store.
>
> So, we either need it to start doing that (cc: kzak), or we need to have th
> code cross-reference devices against /etc/fstab.

libmount manages the option _netdev in userspace (in /run/mount/utab -- but this
private libmount file is not part of any API, so don't read it), use findmnt(8):

$ grep _netdev /etc/fstab 
/dev/sdb1   /mnt   ext3    defaults,_netdev        0

$ findmnt --mtab /mnt
TARGET SOURCE    FSTYPE OPTIONS
/mnt   /dev/sdb1 ext3   rw,relatime,errors=continue,barrier=0,data=ordered,_netdev

$ mount | grep /mnt
/dev/sdb1 on /mnt type ext3(rw,relatime,errors=continue,barrier=0,data=ordered,_netdev)

Note that things like _netdev are ugly hacks, it would be nice to found a 
better way how mark/detect some specific devices.

Karel

[Bill Nottingham]

Karel Zak ([e-mail]) said:
> On Tue, Nov 29, 2011 at 12:18:03PM -0500, Bill Nottingham wrote:
> > Jeremy Sanders ([e-mail]) said:
> > > > What are you using for networking - the old network service or
> > > > NetworkManager?
> > >
> > > The old network service. The root device is cloned from a non-iscsi 
> > > installation, however, as kickstart didn't like to install onto a non-
> > > partitioned iscsi device: https://bugzilla.redhat.com/show_bug.cgi?id=757106
> > > I thought that the _netdev option would be sufficient to get the ordering 
> > > correct.
> >
> > Ah, crud. The issue is that we're checking /etc/mtab for _netdev, which is
> > now /proc/mounts, which doesn't transfer over random options such as these
> > into its data store.
> >
> > So, we either need it to start doing that (cc: kzak), or we need to have th
> > code cross-reference devices against /etc/fstab.
>
> libmount manages the option _netdev in userspace (in /run/mount/utab -- but this
> private libmount file is not part of any API, so don't read it), use findmnt(8):
>
> $ grep _netdev /etc/fstab 
> /dev/sdb1   /mnt   ext3    defaults,_netdev        0
>
> $ findmnt --mtab /mnt
> TARGET SOURCE    FSTYPE OPTIONS
> /mnt   /dev/sdb1 ext3   rw,relatime,errors=continue,barrier=0,data=ordered,_netdev
>
> $ mount | grep /mnt
> /dev/sdb1 on /mnt type ext3(rw,relatime,errors=continue,barrier=0,data=ordered,_netdev)
>
> Note that things like _netdev are ugly hacks, it would be nice to found a 
> better way how mark/detect some specific devices.

It would be nice, yes. In the meantime, Jeremy, can you try the attached
patch?

Bill

[Jeremy Sanders]

Bill Nottingham wrote:
> It would be nice, yes. In the meantime, Jeremy, can you try the attached
> patch?

Thanks very much - it doesn't seem to work in our setup. Dracut is mounting 
the root iscsi device using the dhcp root-dev option, but we have _netdev 
set in fstab.

-bash-4.2# mount | grep "on / "
/dev/sda on / type ext4 (rw,relatime,user_xattr,acl,barrier=0,data=ordered)

-bash-4.2# grep _netdev /etc/fstab
/dev/sda                        /               ext4    
_netdev,barrier=0,relatime 1 1

-bash-4.2#  mount | grep "/ "
/dev/sda on / type ext4 (rw,relatime,user_xattr,acl,barrier=0,data=ordered)

Your patch does

-bash-4.2# findmnt -m -n -o fstype,options /
ext4   rw,relatime,user_xattr,acl,barrier=0,data=ordered

Which doesn't show the _netdev option. If I change this to

-bash-4.2# findmnt -s -n -o fstype,options /
ext4   _netdev,barrier=0,relatime

This works. I don't know whether you want to change "-m" to "-s"...

Jeremy

[Jeremy Sanders]

Jeremy Sanders wrote:
> This works. I don't know whether you want to change "-m" to "-s"...

I tried changing -m to -s.

The network still tries to get shut down, despite the if statement being run 
and the exit 1 command being run. This is because the exit command only 
exits from the shell looping over the read, not the shell the script is 
executing in (I hate shells!).

I had to modify the patch to exit after the loop if the loop had exited 
internally.

This fixed version appears to work (attached).

Jeremy

[Bill Nottingham]

Jeremy Sanders ([e-mail]) said:
> Your patch does
>
> -bash-4.2# findmnt -m -n -o fstype,options /
> ext4   rw,relatime,user_xattr,acl,barrier=0,data=ordered
>
> Which doesn't show the _netdev option. If I change this to
>
> -bash-4.2# findmnt -s -n -o fstype,options /
> ext4   _netdev,barrier=0,relatime
>
> This works. I don't know whether you want to change "-m" to "-s"...

Well, -s requires everything to be in fstab. -m would work for things
mounted with plain "mount -o _netdev", assuming /run/mount/utab is
properly populated. I'm assuming on your system it's not?

Bill

[Jeremy Sanders]

Bill Nottingham wrote:
> Jeremy Sanders ([e-mail]) said:
> > Your patch does
> >
> > -bash-4.2# findmnt -m -n -o fstype,options /
> > ext4   rw,relatime,user_xattr,acl,barrier=0,data=ordered
> >
> > Which doesn't show the _netdev option. If I change this to
> >
> > -bash-4.2# findmnt -s -n -o fstype,options /
> > ext4   _netdev,barrier=0,relatime
> >
> > This works. I don't know whether you want to change "-m" to "-s"...
>
> Well, -s requires everything to be in fstab. -m would work for things
> mounted with plain "mount -o _netdev", assuming /run/mount/utab is
> properly populated. I'm assuming on your system it's not?

/run/mount/utab is empty. We're using tftpboot and dracut's network initrd 
support to mount the root, so perhaps that's not properly populating it. 
Perhaps this is a dracut bug too. Presumably I should file a bug for 
something so this can be worked on offline.

Jeremy

[Bill Nottingham]

Jeremy Sanders ([e-mail]) said:
> > Jeremy Sanders ([e-mail]) said:
> > > Your patch does
> > >
> > > -bash-4.2# findmnt -m -n -o fstype,options /
> > > ext4   rw,relatime,user_xattr,acl,barrier=0,data=ordered
> > >
> > > Which doesn't show the _netdev option. If I change this to
> > >
> > > -bash-4.2# findmnt -s -n -o fstype,options /
> > > ext4   _netdev,barrier=0,relatime
> > >
> > > This works. I don't know whether you want to change "-m" to "-s"...
> >
> > Well, -s requires everything to be in fstab. -m would work for things
> > mounted with plain "mount -o _netdev", assuming /run/mount/utab is
> > properly populated. I'm assuming on your system it's not?
>
> /run/mount/utab is empty. We're using tftpboot and dracut's network initrd 
> support to mount the root, so perhaps that's not properly populating it. 
> Perhaps this is a dracut bug too. Presumably I should file a bug for 
> something so this can be worked on offline.

Sure, start with initscripts and we can go from there.

Bill