mount to folder below root fails

[Heiko Onnebrink]

I am testing migration of CoreOS with Ignition 2.2 (named here current env) to to preview
FedoraCoreOS with Ignition 3.0 (named here new env)
Tests are performed on OpenStack Rocky on prem.

In current env we mount /dev/vdb to a folder /dockerdata and /dev/vdc to /var/lib/docker
which works fine using Ignition 2.0.

I have transferred config to Ignition 3 and test now in new env.
Mount to /var/lib/docker is fine, mount to /dockerdata fails :

..
[    9.931065] ignition[696]:     "filesystems": [
[    9.931075] ignition[696]:       {
[    9.931087] ignition[696]:         "device": "/dev/vdc",
[    9.931097] ignition[696]:         "format": "ext4",
[    9.931107] ignition[696]:         "label": "docker",
[    9.932644] ignition[696]:         "path": "/var/lib/docker",
[    9.932660] ignition[696]:         "wipeFilesystem": true
[    9.932670] ignition[696]:       },
[    9.932680] ignition[696]:       {
[    9.932689] ignition[696]:         "device": "/dev/vdb",
[    9.932699] ignition[696]:         "format": "ext4",
[    9.932714] ignition[696]:         "label": "dockerdata",
[    9.932724] ignition[696]:         "path": "/dockerdata",
[    9.932734] ignition[696]:         "wipeFilesystem": true
[    9.932743] ignition[696]:       }
[    9.932753] ignition[696]:     ]
..
..
[    9.935087] ignition[696]: }CRITICAL : Ignition failed: mkdir /sysroot/dockerdata:
operation not permitted
[    9.936383] umount[704]: umount: /sysroot/var: not mounted.
[    9.936548] systemd[1]: ignition-mount.service: Main process exited, code=exited,
status=1/FAILURE
[    9.936696] systemd[1]: ignition-mount.service: Failed with result
'exit-code'.
[    9.936730] systemd[1]: Failed to start Ignition (mount).

Also tried to explicitly add a create directory /dockerdata using Ignition
"directories .. /dockerdata"  but same error.
[    9.537684] ignition[696]: }CRITICAL : Ignition failed: mkdir /sysroot/dockerdata:
operation not permitted

Did I do something in the past that is not ok but was "tolerated" and now fails
due to more strict checking?

Thanks for any feedback
cheers
Heiko

[Dusty Mabe]

On 9/8/19 5:07 AM, Heiko Onnebrink wrote:

Hi Heiko!

>  I am testing migration of CoreOS with Ignition 2.2 (named here
> current env) to to preview FedoraCoreOS with Ignition 3.0 (named here new env)
>  Tests are performed on OpenStack Rocky on prem. 
Thanks for trying out Fedora CoreOS!

>  
>  In current env we mount /dev/vdb to a folder /dockerdata and /dev/vdc to /var/lib/docker
> which works fine using Ignition 2.0.
>  
>  I have transferred config to Ignition 3 and test now in new env.
>  Mount to /var/lib/docker is fine, mount to /dockerdata fails :
>  
>  ..
>  [    9.931065] ignition[696]:     "filesystems": [
>  [    9.931075] ignition[696]:       {
>  [    9.931087] ignition[696]:         "device": "/dev/vdc",
>  [    9.931097] ignition[696]:         "format": "ext4",
>  [    9.931107] ignition[696]:         "label": "docker",
>  [    9.932644] ignition[696]:         "path": "/var/lib/docker",
>  [    9.932660] ignition[696]:         "wipeFilesystem": true
>  [    9.932670] ignition[696]:       },
>  [    9.932680] ignition[696]:       {
>  [    9.932689] ignition[696]:         "device": "/dev/vdb",
>  [    9.932699] ignition[696]:         "format": "ext4",
>  [    9.932714] ignition[696]:         "label": "dockerdata",
>  [    9.932724] ignition[696]:         "path": "/dockerdata",
>  [    9.932734] ignition[696]:         "wipeFilesystem": true
>  [    9.932743] ignition[696]:       }
>  [    9.932753] ignition[696]:     ]
>  ..
>  ..
>  [    9.935087] ignition[696]: }CRITICAL : Ignition failed: mkdir /sysroot/dockerdata:
> operation not permitted
>  [    9.936383] umount[704]: umount: /sysroot/var: not mounted.
>  [    9.936548] systemd[1]: ignition-mount.service: Main process exited, code=exited,
> status=1/FAILURE
>  [    9.936696] systemd[1]: ignition-mount.service: Failed with result
> 'exit-code'.
>  [    9.936730] systemd[1]: Failed to start Ignition (mount).
>  
>  Also tried to explicitly add a create directory /dockerdata using Ignition
> "directories .. /dockerdata"  but same error.
>  [    9.537684] ignition[696]: }CRITICAL : Ignition failed: mkdir /sysroot/dockerdata:
> operation not permitted
>  
>  Did I do something in the past that is not ok but was "tolerated" and now fails
> due to more strict checking? 
I believe you are hitting this problem because of the filesystem structure/layout of
OSTree (the underlying technology
behind Fedora CoreOS). With OSTree there are certain filesystems that are read-write and
certain ones that are read-only.
It just so happens that the root of the filesystem tree is marked as immutable, which
means you can't create files or
directories under `/`. You can see that by running `lsattr -d /` on a booted system:

```
[root@localhost ~]# lsattr -d /
----i-------------- /
```

There is an open issue discussing this upstream as well:
https://github.com/projectatomic/rpm-ostree/issues/337

Is it possible for you to use a directory under a writable filesystem (like
`/var/lib/dockerdata`) instead?

[Heiko Onnebrink]

Thanks for fast reply and guiding me to the discussion link,, I posted my comment there.

To answer your question:

If I would be on a green-field I would just go for /var/lib/dockerdata.. 

But unfortunately I am on a brown field and have to lock how to stay backward compatible.
I would prefer an immutable directory creation option in the ignition config that allows
me to create a folder under root (with immutable = true) that is just used as mount dir..

[Dusty Mabe]

On 9/9/19 8:58 AM, Heiko Onnebrink wrote:
>  Thanks for fast reply and guiding me to the discussion link,, I
> posted my comment there.
>  
>  To answer your question:
>  
>  If I would be on a green-field I would just go for /var/lib/dockerdata.. 
>  
>  But unfortunately I am on a brown field and have to lock how to stay backward compatible.
I understand. You might be able to get creative with symlinks, but yeah, not ideal.

>  I would prefer an immutable directory creation option in the ignition
> config that allows me to create a folder under root (with immutable = true) that is just
> used as mount dir..  
Yeah I would say maybe open an issue at github.com/coreos/fedora-coreos-tracker where
we can have a discussion on the feasibility of that.

Thanks for the feedback!
Dusty

[Heiko Onnebrink]

Opened issue here https://github.com/coreos/fedora-coreos-tracker/issues/270

Hope for a fruitful discussion ;-)