Filesystem which stays on reinstall on same physical device as the installation

[Joel Kessler]

Hello there,

Question related to the file system setup of fcos.

I defined a partition within ignition on the install device.

disks:
    - device: /dev/nvme0n1
      wipe_table: false
      partitions:
        - label: DATA 
          start_mib: 0
          number: 0
          size_mib: 10240

As well as the filesystem:

filesystems:
    - device: /dev/disk/by-partlabel/DATA
      format: ext4 
      path: /mnt/data
      wipe_filesystem: false
      label: data
      
My goal would be that after the setup my disk looks like this:
The reason for putting my own partition in front of the system partitions, would be that if I re execute ignition on this device
the defined partition should not be erased and possible to expand with a new ignition file. The sysroot partition should get all the remaining space.

nvme0n1     259:0    0 238.5G  0 disk
|-nvme0n1p1 259:5    0    10G  0 part /mnt/data
|-nvme0n1p2 259:1    0   384M  0 part /boot
|-nvme0n1p3 259:2    0   127M  0 part /boot/efi
|-nvme0n1p4 259:3    0     1M  0 part
|-nvme0n1p5 259:4    0 225.8G  0 part /sysroot

But because of some  it currently looks like this. When adding the filesystems as defined above my ignition will fail on first boot.

nvme0n1     259:0    0 238.5G  0 disk
|-nvme0n1p1 259:1    0   384M  0 part /boot
|-nvme0n1p2 259:2    0   127M  0 part /boot/efi
|-nvme0n1p3 259:3    0     1M  0 part
|-nvme0n1p4 259:4    0   2.1G  0 part /sysroot
`-nvme0n1p5 259:5    0    10G  0 part

If I would run it with just the disks defined I can run the ignition but the file system will be erased each time.

I tested it by manually adding the filesystem: sudo mount -t auto -v /dev/nvme0n1p5 /mnt/data
this will return an error and expects me to redo: mkfs.ext4 /dev/nvme0n1p5

Is there any way to define a partition which won't get lost after a re execution of ignition?

[Benjamin Gilbert]

On Fri, Feb 21, 2020 at 10:57 AM Joel Kessler  wrote:
> The reason for putting my own partition in front of the system partitions,
> would be that if I re execute ignition on this device
> the defined partition should not be erased and possible to expand with a
> new ignition file. The sysroot partition should get all the remaining space.

There's currently no way to put a new partition before the partitions
shipped with the OS.  Work is in progress to allow moving the root
partition, but I don't expect that we'll ever support moving /boot or
/boot/efi.

In general, Ignition is intended to run exactly once per system, and
running it more than once is not supported.  (There is a limited exception
for live ISO/PXE systems, where, arguably, every boot is the first boot.)
Why do you want to run Ignition more than once?

--Benjamin Gilbert

[James Cassell]

On Fri, Feb 21, 2020, at 4:46 PM, Benjamin Gilbert wrote:
> On Fri, Feb 21, 2020 at 10:57 AM Joel Kessler  wrote:
> > The reason for putting my own partition in front of the system partitions, would be that if I re execute ignition on this device
> >  the defined partition should not be erased and possible to expand with a new ignition file. The sysroot partition should get all the remaining space.
>
> There's currently no way to put a new partition before the partitions 
> shipped with the OS. Work is in progress to allow moving the root 
> partition, but I don't expect that we'll ever support moving /boot or 
> /boot/efi.
>
> In general, Ignition is intended to run exactly once per system, and 
> running it more than once is not supported. (There is a limited 
> exception for live ISO/PXE systems, where, arguably, every boot is the 
> first boot.) Why do you want to run Ignition more than once?

How does one reprovision an existing machine without losing its data? How can one update a machine's config when the canonical config is the ignition?

For the latter, I hope to write an ansible role that takes an .fcc as input and enforces it on a host. Obviously doesn't work with arbitrary scripts, but anything declarative should work in concept...

V/r,
James Cassell

[Benjamin Gilbert]

On Fri, Feb 21, 2020 at 6:00 PM James Cassell 
wrote:
> On Fri, Feb 21, 2020, at 4:46 PM, Benjamin Gilbert wrote:
> > In general, Ignition is intended to run exactly once per system, and
> > running it more than once is not supported. (There is a limited
> > exception for live ISO/PXE systems, where, arguably, every boot is the
> > first boot.) Why do you want to run Ignition more than once?
>
> How does one reprovision an existing machine without losing its data? How
> can one update a machine's config when the canonical config is the ignition?

Ah, that makes sense.  We don't usually think of reprovisioning as
"re-executing Ignition", but your use case matches the live ISO/PXE one I
mentioned.

Your config is basically right.  With wipe_filesystem: false, Ignition
should reuse an existing filesystem if it finds one with the specified
properties.  If you want to allow room for expansion of the root
filesystem, you can set the start_mib of the DATA partition to a suitable
value.  What error are you getting from Ignition when you apply your config?

If you put the data partition on a second disk, this approach should work
today.  It won't work for the boot disk right now because reprovisioning
with coreos-installer will wipe the entire disk.  We should fix that
https://github.com/coreos/coreos-installer/issues/170.

For the latter, I hope to write an ansible role that takes an .fcc as input
> and enforces it on a host. Obviously doesn't work with arbitrary scripts,
> but anything declarative should work in concept...

In principle this can work.  (Though you'd want to transpile the FCC to
Ignition and apply the Ignition config to the host.)  In general, though,
our recommendation is to reprovision the machine, rather than trying to
keep it in sync after the fact.

--Benjamin Gilbert

[Joel Kessler]

Good Morning,

Thanks for your response.

My usecase is similar to the one described above.
In my case, I'd like to wipe / override the whole os with a newly distributed ISO version.

In some cases where the user changes an os based configuration like a static IP Address, the ignition will be reapplied and setup from the data partition (offline scenario). 
-> On this partition the .fcc / .iso / and currently required .raw file (I assume the raw file will no longer be required at some point because the live image already contains the whole OS, if my assumtion is wrong please correct me c: )
-> As well as some user sepecific config files

As for the error I got:

disks:
    - device: /dev/nvme0n1
      wipe_table: false
      partitions:
        - label: DATA 
          number: 0
          size_mib: 10240
  filesystems:
    - device: /dev/disk/by-partlabel/DATA
      format: ext4 
      path: /mnt/data
      wipe_filesystem: false
      label: data
      with_mount_unit: true

I was not yet able to locate the error on the installation output, but will eventually come back if I found something.

[Joel Kessler]

When I run the following command, on my bare metal installation (meaning coreos has been installed and is currently up and running): sudo coreos-installer install /dev/nvme0n1 --image-file /mnt/insdev/fedora_coreos.raw.xz --ignition  /mnt/insdev/ignition.ign --insecure

The following error will occour, is there any way to resolve this, or is this not supported at all? (I have to run it on the running os)

Error: checking for exclusive access to /dev/nvme0n1
Caused by: couldn't reread partition table: device is in use
Caused by: EBUSY: Device or resource busy

[Benjamin Gilbert]

The installer intentionally refuses to run on a disk with mounted
partitions.  This is a safety check to make sure you don't wipe your
running system.

I'm not sure I understand the reinstallation flow you're proposing.  You
have a running Fedora CoreOS system, with a data partition containing an
FCC and an ISO live image.  When you want to reprovision the machine, how
do you plan to boot into the ISO?  How are you running FCCT to convert the
FCC to an Ignition config?

--Benjamin Gilbert

On Mon, Feb 24, 2020 at 3:41 AM Joel Kessler  wrote:
> When I run the following command, on my bare metal installation (meaning
> coreos has been installed and is currently up and running): sudo
> coreos-installer install /dev/nvme0n1 --image-file
> /mnt/insdev/fedora_coreos.raw.xz --ignition  /mnt/insdev/ignition.ign
> --insecure
>
> The following error will occour, is there any way to resolve this, or is
> this not supported at all? (I have to run it on the running os)
>
> Error: checking for exclusive access to /dev/nvme0n1
> Caused by: couldn't reread partition table: device is in use
> Caused by: EBUSY: Device or resource busy
> _______________________________________________
> CoreOS mailing list -- [e-mail]
> To unsubscribe send an email to [e-mail]
> Fedora Code of Conduct:
> https://docs.fedoraproject.org/en-US/project/code-of-conduct/
> List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
> List Archives:
> https://lists.fedoraproject.org/archives/list/[e-mail]

[Joel Kessler]

I planed to have it like this:

nvme0n1 259:0 0 238.5G 0 disk
|-nvme0n1p1 259:5 0 10G 0 part /mnt/data
|-nvme0n1p2 259:1 0 384M 0 part /boot
|-nvme0n1p3 259:2 0 127M 0 part /boot/efi
|-nvme0n1p4 259:3 0 1M 0 part
|-nvme0n1p5 259:4 0 225.8G 0 part /sysroot

For a reprovision I would now like to run this on the active fcos

1. docker run -i --rm quay.io/coreos/fcct:v0.2.0 -pretty -strict < /mnt/data/ignition.fcc > /mnt/data/transpiled_config.ign
2. coreos-installer install /dev/nvme0n1 --image-file /mnt/data/fedora_coreos.raw.xz --ignition  /mnt/data/transpiled_config.ign --insecure

Which then should override the currently running os, except for the data partition.

[Benjamin Gilbert]

On Tue, Feb 25, 2020 at 1:52 AM Joel Kessler  wrote:
> I planed to have it like this:
>
> nvme0n1 259:0 0 238.5G 0 disk
> |-nvme0n1p1 259:5 0 10G 0 part /mnt/data
> |-nvme0n1p2 259:1 0 384M 0 part /boot
> |-nvme0n1p3 259:2 0 127M 0 part /boot/efi
> |-nvme0n1p4 259:3 0 1M 0 part
> |-nvme0n1p5 259:4 0 225.8G 0 part /sysroot
>
> For a reprovision I would now like to run this on the active fcos
> docker run -i --rm quay.io/coreos/fcct:v0.2.0 -pretty -strict <
>
> /mnt/data/ignition.fcc > /mnt/data/transpiled_config.ign
> 2. coreos-installer install /dev/nvme0n1 --image-file
> /mnt/data/fedora_coreos.raw.xz --ignition  /mnt/data/transpiled_config.ign
> --insecure
>
> Which then should override the currently running os, except for the data
> partition.

Step 1 looks fine.  Step 2 won't work; you can't overwrite the root
partition while booted from it.  A couple possible alternatives:

- When you want to reprovision, boot from a PXE server on the local network
and run coreos-installer from the resulting live system.  Then reboot into
the installed system.
- Boot and run from PXE every time, and skip the install to disk
altogether.  You can still use the local disk for your data partition.
Note that there are no automatic updates in this mode; updates can only
happen by updating the image on the PXE server and rebooting nodes into the
new OS version.

--Benjamin Gilbert

[Benjamin Gilbert]

https://github.com/coreos/coreos-installer/pull/172 stops coreos-installer
from wiping the entire disk, allowing data partitions to be recreated with
contents preserved.
https://github.com/coreos/fedora-coreos-tracker/issues/399 proposes adding
a factory reset mechanism to aid reprovisioning of single-node bare metal
installs.

--Benjamin Gilbert