8:52 a.m.
Hey Fedora Cloud WG!
Firstly, thanks for providing Fedora for the popular cloud providers!
Since Fedora 35, the Fedora Cloud images use btrfs by default [1].
For my deployments, I would like to use ext4 or xfs, so my question
are:
1) Is it possible to change the root file system at deploy time?
2) If not, how could one change the cloud images to use a different
root file system?
Thanks and regards,
Tadej Janež
[1] https://fedoraproject.org/wiki/Changes/FedoraCloudBtrfsByDefault
9 a.m.
New subject: Changing file system for Cloud images to ext4, xfs or something custom
On Tue, Apr 9, 2024 at 9:53 AM Tadej Janež <tadej.j(a)nez.si> wrote:
Hey Fedora Cloud WG!
Firstly, thanks for providing Fedora for the popular cloud providers!
Since Fedora 35, the Fedora Cloud images use btrfs by default [1].
For my deployments, I would like to use ext4 or xfs, so my question
are:
1) Is it possible to change the root file system at deploy time?
2) If not, how could one change the cloud images to use a different
root file system?
It is not possible to change at deployment time, you would need to
build your own custom images.
Our images are now defined here:
https://pagure.io/fedora-kiwi-descriptions/blob/rawhide/f/teams/cloud/clo...
If you want to use something else, you'd want to have your own version
of the definitions and modify that file to use the filesystem of your
choice.
Is there a particular reason you want to use ext4 or xfs for your
rootfs? Typically the pattern we see is that people attach a secondary
volume or use S3 and put their data on that instead of the rootfs.
--
真実はいつも一つ！/ Always, there's only one truth!
10:04 a.m.
On 4/9/24 10:00, Neal Gompa wrote:
On Tue, Apr 9, 2024 at 9:53 AM Tadej Janež <tadej.j(a)nez.si>
wrote:
>
> Hey Fedora Cloud WG!
>
> Firstly, thanks for providing Fedora for the popular cloud providers!
>
> Since Fedora 35, the Fedora Cloud images use btrfs by default [1].
> For my deployments, I would like to use ext4 or xfs, so my question
> are:
> 1) Is it possible to change the root file system at deploy time?
> 2) If not, how could one change the cloud images to use a different
> root file system?
>
It is not possible to change at deployment time, you would need to
build your own custom images.
We do have Fedora CoreOS, which may or may not fit your use case (i.e.
run software in containers, automatic updates, rpm-ostree as the
update mechanism), which can re-provision the root filesystem on first
boot and does provide images for various clouds:
- https://fedoraproject.org/coreos/download?stream=stable
-
https://docs.fedoraproject.org/en-US/fedora-coreos/storage/#_reconfigurin...
This may or may not be an option for you, but figured I'd mention.
>
> Our images are now defined here:
> https://pagure.io/fedora-kiwi-descriptions/blob/rawhide/f/teams/cloud/clo...
>
> If you want to use something else, you'd want to have your own version
> of the definitions and modify that file to use the filesystem of your
> choice.
>
> Is there a particular reason you want to use ext4 or xfs for your
> rootfs? Typically the pattern we see is that people attach a secondary
> volume or use S3 and put their data on that instead of the rootfs.
>
>
>
4:25 a.m.
On Tue, 2024-04-09 at 11:04 -0400, Dusty Mabe wrote:
On 4/9/24 10:00, Neal Gompa wrote:
> On Tue, Apr 9, 2024 at 9:53 AM Tadej Janež <tadej.j(a)nez.si> wrote:
> >
> > Hey Fedora Cloud WG!
> >
> > Firstly, thanks for providing Fedora for the popular cloud
> > providers!
> >
> > Since Fedora 35, the Fedora Cloud images use btrfs by default
> > [1].
> > For my deployments, I would like to use ext4 or xfs, so my
> > question
> > are:
> > 1) Is it possible to change the root file system at deploy time?
> > 2) If not, how could one change the cloud images to use a
> > different
> > root file system?
> >
>
> It is not possible to change at deployment time, you would need to
> build your own custom images.
We do have Fedora CoreOS, which may or may not fit your use case
(i.e.
run software in containers, automatic updates, rpm-ostree as the
update mechanism), which can re-provision the root filesystem on
first
boot and does provide images for various clouds:
- https://fedoraproject.org/coreos/download?stream=stable
-
https://docs.fedoraproject.org/en-US/fedora-coreos/storage/#_reconfigurin...
This may or may not be an option for you, but figured I'd mention.
Thanks for mentioning this, this looks very nice!
Does it copy the whole rootfs to RAM, erases and reformats it with a
new file system and copies the contents from RAM back to it?
On the subject if this fits my use case, the answer is no. The systems
don't run applications in containers. They are set up to run services
via classical systemd units.
7:52 a.m.
New subject: Changing file system for Cloud images to ext4, xfs or something custom
On Thu, Apr 11, 2024 at 5:25 AM Tadej Janež <tadej.j(a)nez.si> wrote:
On Tue, 2024-04-09 at 11:04 -0400, Dusty Mabe wrote:
>
>
> On 4/9/24 10:00, Neal Gompa wrote:
> > On Tue, Apr 9, 2024 at 9:53 AM Tadej Janež <tadej.j(a)nez.si> wrote:
> > >
> > > Hey Fedora Cloud WG!
> > >
> > > Firstly, thanks for providing Fedora for the popular cloud
> > > providers!
> > >
> > > Since Fedora 35, the Fedora Cloud images use btrfs by default
> > > [1].
> > > For my deployments, I would like to use ext4 or xfs, so my
> > > question
> > > are:
> > > 1) Is it possible to change the root file system at deploy time?
> > > 2) If not, how could one change the cloud images to use a
> > > different
> > > root file system?
> > >
> >
> > It is not possible to change at deployment time, you would need to
> > build your own custom images.
>
> We do have Fedora CoreOS, which may or may not fit your use case
> (i.e.
> run software in containers, automatic updates, rpm-ostree as the
> update mechanism), which can re-provision the root filesystem on
> first
> boot and does provide images for various clouds:
>
> - https://fedoraproject.org/coreos/download?stream=stable
> -
>
https://docs.fedoraproject.org/en-US/fedora-coreos/storage/#_reconfigurin...
>
> This may or may not be an option for you, but figured I'd mention.
Thanks for mentioning this, this looks very nice!
Does it copy the whole rootfs to RAM, erases and reformats it with a
new file system and copies the contents from RAM back to it?
This is basically what ignition does, yes.
--
真実はいつも一つ！/ Always, there's only one truth!
9:54 a.m.
On Tue, 2024-04-09 at 10:00 -0400, Neal Gompa wrote:
On Tue, Apr 9, 2024 at 9:53 AM Tadej Janež <tadej.j(a)nez.si>
wrote:
>
> Hey Fedora Cloud WG!
>
> Firstly, thanks for providing Fedora for the popular cloud
> providers!
>
> Since Fedora 35, the Fedora Cloud images use btrfs by default [1].
> For my deployments, I would like to use ext4 or xfs, so my question
> are:
> 1) Is it possible to change the root file system at deploy time?
> 2) If not, how could one change the cloud images to use a different
> root file system?
>
It is not possible to change at deployment time, you would need to
build your own custom images.
Ok, thanks for clarifying!
Our images are now defined here:
https://pagure.io/fedora-kiwi-descriptions/blob/rawhide/f/teams/cloud/clo...
If you want to use something else, you'd want to have your own
version
of the definitions and modify that file to use the filesystem of your
choice.
I see, one would need to modify the "filesystem" key of the selected
Cloud-Base-<provider> image.
Are there instructions on how to build a custom cloud image?
However, even if I can easily build a custom cloud image, the overhead
of maintaining custom cloud images is very high.
For every Fedora release and cloud provider, I would need to build the
image and then with IaC (e.g. Terraform), handle uploading the custom
image and using it...
Is there a particular reason you want to use ext4 or xfs for your
rootfs? Typically the pattern we see is that people attach a
secondary
volume or use S3 and put their data on that instead of the rootfs.
Yes, I use that pattern as well. Usually, /srv or /var would be on a
separate block storage and formatted with the file system of choice.
The reasons why I would want to use, e.g. ext4 for the rootfs, would
be:
1. Familiarity. I've mainly been using ext4 or LVM+ext4 (with LUKS
underneath) for the last 2 decades. I know the tools and I know what
"care" such filesystems need.
2. Maturity. Ext2/3/4 have been round for quite longer than btrfs and
there are very little "unknowns" or "surprises" with it.
3. Simplicity. When provisioning machines with a cloud provider, I
actually don't need the LVM+ext4 combination because the cloud provider
would typically handle the things LVM would handle for a non-cloud
machine, e.g. increasing the block storage size, snapshotting, ...
Please, don't read this as a critique against btrfs, just me trying to
explain why I would find it nicer to just use ext4 for the rootfs as
well.
Regards,
Tadej
10:22 a.m.
New subject: Changing file system for Cloud images to ext4, xfs or something custom
On Wed, Apr 10, 2024 at 10:54 AM Tadej Janež <tadej.j(a)nez.si> wrote:
On Tue, 2024-04-09 at 10:00 -0400, Neal Gompa wrote:
> On Tue, Apr 9, 2024 at 9:53 AM Tadej Janež <tadej.j(a)nez.si> wrote:
> >
> > Hey Fedora Cloud WG!
> >
> > Firstly, thanks for providing Fedora for the popular cloud
> > providers!
> >
> > Since Fedora 35, the Fedora Cloud images use btrfs by default [1].
> > For my deployments, I would like to use ext4 or xfs, so my question
> > are:
> > 1) Is it possible to change the root file system at deploy time?
> > 2) If not, how could one change the cloud images to use a different
> > root file system?
> >
>
> It is not possible to change at deployment time, you would need to
> build your own custom images.
>
Ok, thanks for clarifying!
> Our images are now defined here:
> https://pagure.io/fedora-kiwi-descriptions/blob/rawhide/f/teams/cloud/clo...
>
> If you want to use something else, you'd want to have your own
> version
> of the definitions and modify that file to use the filesystem of your
> choice.
I see, one would need to modify the "filesystem" key of the selected
Cloud-Base-<provider> image.
Are there instructions on how to build a custom cloud image?
The repository README includes a quickstart section for building your
own images.
However, even if I can easily build a custom cloud image, the
overhead
of maintaining custom cloud images is very high.
For every Fedora release and cloud provider, I would need to build the
image and then with IaC (e.g. Terraform), handle uploading the custom
image and using it...
Yeah, it is more overhead, for sure.
>
> Is there a particular reason you want to use ext4 or xfs for your
> rootfs? Typically the pattern we see is that people attach a
> secondary
> volume or use S3 and put their data on that instead of the rootfs.
>
Yes, I use that pattern as well. Usually, /srv or /var would be on a
separate block storage and formatted with the file system of choice.
The reasons why I would want to use, e.g. ext4 for the rootfs, would
be:
1. Familiarity. I've mainly been using ext4 or LVM+ext4 (with LUKS
underneath) for the last 2 decades. I know the tools and I know what
"care" such filesystems need.
2. Maturity. Ext2/3/4 have been round for quite longer than btrfs and
there are very little "unknowns" or "surprises" with it.
3. Simplicity. When provisioning machines with a cloud provider, I
actually don't need the LVM+ext4 combination because the cloud provider
would typically handle the things LVM would handle for a non-cloud
machine, e.g. increasing the block storage size, snapshotting, ...
Please, don't read this as a critique against btrfs, just me trying to
explain why I would find it nicer to just use ext4 for the rootfs as
well.
That's totally fair. I will point out that one of the big reasons we
use Btrfs is actually to enable space efficiency (which can lead to
cost savings across the board). The used storage is roughly 40% less
than on ext4 due to leveraging both transparent compression and
reflinks.
The other reason is to make it easy to do replication for backups and
other purposes.
You may also find it to be beneficial for easily avoiding certain
kinds of failure cases that can happen on ext4 (such as inode
exhaustion with containers), since btrfs dynamically allocates inodes
per subvolume.
You can also grow (and shrink!) the volume live while the system is
mounted and online safely.
--
真実はいつも一つ！/ Always, there's only one truth!
4:58 a.m.
On Wed, 2024-04-10 at 11:22 -0400, Neal Gompa wrote:
On Wed, Apr 10, 2024 at 10:54 AM Tadej Janež <tadej.j(a)nez.si>
wrote:
>
> On Tue, 2024-04-09 at 10:00 -0400, Neal Gompa wrote:
> > On Tue, Apr 9, 2024 at 9:53 AM Tadej Janež <tadej.j(a)nez.si>
> > wrote:
> > >
> > > Hey Fedora Cloud WG!
> > >
> > > Firstly, thanks for providing Fedora for the popular cloud
> > > providers!
> > >
> > > Since Fedora 35, the Fedora Cloud images use btrfs by default
> > > [1].
> > > For my deployments, I would like to use ext4 or xfs, so my
> > > question
> > > are:
> > > 1) Is it possible to change the root file system at deploy
> > > time?
> > > 2) If not, how could one change the cloud images to use a
> > > different
> > > root file system?
> > >
> >
> > It is not possible to change at deployment time, you would need
> > to
> > build your own custom images.
> >
>
> Ok, thanks for clarifying!
>
> > Our images are now defined here:
> >
https://pagure.io/fedora-kiwi-descriptions/blob/rawhide/f/teams/cloud/clo...
> >
> > If you want to use something else, you'd want to have your own
> > version
> > of the definitions and modify that file to use the filesystem of
> > your
> > choice.
>
> I see, one would need to modify the "filesystem" key of the
> selected
> Cloud-Base-<provider> image.
>
> Are there instructions on how to build a custom cloud image?
>
The repository README includes a quickstart section for building your
own images.
> However, even if I can easily build a custom cloud image, the
> overhead
> of maintaining custom cloud images is very high.
>
> For every Fedora release and cloud provider, I would need to build
> the
> image and then with IaC (e.g. Terraform), handle uploading the
> custom
> image and using it...
>
Yeah, it is more overhead, for sure.
Thinking out loud... the modification in my case would not be a custom
tailored image for my use case but more similar to how Fedora project
builds different spins if people want to use a non-GNOME desktop
environment or ALT images for people using alternative architectures.
Is there a possibility to reuse the existing image building
infrastructure and add tasks for building images with e.g. ext4 as the
rootfs?
> >
> > Is there a particular reason you want to use ext4 or xfs for your
> > rootfs? Typically the pattern we see is that people attach a
> > secondary
> > volume or use S3 and put their data on that instead of the
> > rootfs.
> >
>
> Yes, I use that pattern as well. Usually, /srv or /var would be on
> a
> separate block storage and formatted with the file system of
> choice.
>
> The reasons why I would want to use, e.g. ext4 for the rootfs,
> would
> be:
>
> 1. Familiarity. I've mainly been using ext4 or LVM+ext4 (with LUKS
> underneath) for the last 2 decades. I know the tools and I know
> what
> "care" such filesystems need.
>
> 2. Maturity. Ext2/3/4 have been round for quite longer than btrfs
> and
> there are very little "unknowns" or "surprises" with it.
>
> 3. Simplicity. When provisioning machines with a cloud provider, I
> actually don't need the LVM+ext4 combination because the cloud
> provider
> would typically handle the things LVM would handle for a non-cloud
> machine, e.g. increasing the block storage size, snapshotting, ...
>
> Please, don't read this as a critique against btrfs, just me trying
> to
> explain why I would find it nicer to just use ext4 for the rootfs
> as
> well.
>
That's totally fair. I will point out that one of the big reasons we
use Btrfs is actually to enable space efficiency (which can lead to
cost savings across the board). The used storage is roughly 40% less
than on ext4 due to leveraging both transparent compression and
reflinks.
The other reason is to make it easy to do replication for backups and
other purposes.
You may also find it to be beneficial for easily avoiding certain
kinds of failure cases that can happen on ext4 (such as inode
exhaustion with containers), since btrfs dynamically allocates inodes
per subvolume.
You can also grow (and shrink!) the volume live while the system is
mounted and online safely.
Thanks for enumerating these benefits. And I agree that these benefits
alone justify the reason for switching to btrfs by default.
I might be tempted to check out the btrfs status page again and start
playing with it.
7:51 a.m.
New subject: Changing file system for Cloud images to ext4, xfs or something custom
On Thu, Apr 11, 2024 at 5:58 AM Tadej Janež <tadej.j(a)nez.si> wrote:
On Wed, 2024-04-10 at 11:22 -0400, Neal Gompa wrote:
> On Wed, Apr 10, 2024 at 10:54 AM Tadej Janež <tadej.j(a)nez.si> wrote:
> >
> > On Tue, 2024-04-09 at 10:00 -0400, Neal Gompa wrote:
> > > On Tue, Apr 9, 2024 at 9:53 AM Tadej Janež <tadej.j(a)nez.si>
> > > wrote:
> > > >
> > > > Hey Fedora Cloud WG!
> > > >
> > > > Firstly, thanks for providing Fedora for the popular cloud
> > > > providers!
> > > >
> > > > Since Fedora 35, the Fedora Cloud images use btrfs by default
> > > > [1].
> > > > For my deployments, I would like to use ext4 or xfs, so my
> > > > question
> > > > are:
> > > > 1) Is it possible to change the root file system at deploy
> > > > time?
> > > > 2) If not, how could one change the cloud images to use a
> > > > different
> > > > root file system?
> > > >
> > >
> > > It is not possible to change at deployment time, you would need
> > > to
> > > build your own custom images.
> > >
> >
> > Ok, thanks for clarifying!
> >
> > > Our images are now defined here:
> > >
https://pagure.io/fedora-kiwi-descriptions/blob/rawhide/f/teams/cloud/clo...
> > >
> > > If you want to use something else, you'd want to have your own
> > > version
> > > of the definitions and modify that file to use the filesystem of
> > > your
> > > choice.
> >
> > I see, one would need to modify the "filesystem" key of the
> > selected
> > Cloud-Base-<provider> image.
> >
> > Are there instructions on how to build a custom cloud image?
> >
>
> The repository README includes a quickstart section for building your
> own images.
>
> > However, even if I can easily build a custom cloud image, the
> > overhead
> > of maintaining custom cloud images is very high.
> >
> > For every Fedora release and cloud provider, I would need to build
> > the
> > image and then with IaC (e.g. Terraform), handle uploading the
> > custom
> > image and using it...
> >
>
> Yeah, it is more overhead, for sure.
>
Thinking out loud... the modification in my case would not be a custom
tailored image for my use case but more similar to how Fedora project
builds different spins if people want to use a non-GNOME desktop
environment or ALT images for people using alternative architectures.
Is there a possibility to reuse the existing image building
infrastructure and add tasks for building images with e.g. ext4 as the
rootfs?
I think we could consider adding some overlay samples that show
different ways to leverage the descriptions for building custom stuff
on top of ours in the pagure.io/fedora-cloud namespace. I know David
and I were already talking about doing that after we got done with the
F40 cycle.
> > >
> > > Is there a particular reason you want to use ext4 or xfs for your
> > > rootfs? Typically the pattern we see is that people attach a
> > > secondary
> > > volume or use S3 and put their data on that instead of the
> > > rootfs.
> > >
> >
> > Yes, I use that pattern as well. Usually, /srv or /var would be on
> > a
> > separate block storage and formatted with the file system of
> > choice.
> >
> > The reasons why I would want to use, e.g. ext4 for the rootfs,
> > would
> > be:
> >
> > 1. Familiarity. I've mainly been using ext4 or LVM+ext4 (with LUKS
> > underneath) for the last 2 decades. I know the tools and I know
> > what
> > "care" such filesystems need.
> >
> > 2. Maturity. Ext2/3/4 have been round for quite longer than btrfs
> > and
> > there are very little "unknowns" or "surprises" with it.
> >
> > 3. Simplicity. When provisioning machines with a cloud provider, I
> > actually don't need the LVM+ext4 combination because the cloud
> > provider
> > would typically handle the things LVM would handle for a non-cloud
> > machine, e.g. increasing the block storage size, snapshotting, ...
> >
> > Please, don't read this as a critique against btrfs, just me trying
> > to
> > explain why I would find it nicer to just use ext4 for the rootfs
> > as
> > well.
> >
>
> That's totally fair. I will point out that one of the big reasons we
> use Btrfs is actually to enable space efficiency (which can lead to
> cost savings across the board). The used storage is roughly 40% less
> than on ext4 due to leveraging both transparent compression and
> reflinks.
>
> The other reason is to make it easy to do replication for backups and
> other purposes.
>
> You may also find it to be beneficial for easily avoiding certain
> kinds of failure cases that can happen on ext4 (such as inode
> exhaustion with containers), since btrfs dynamically allocates inodes
> per subvolume.
>
> You can also grow (and shrink!) the volume live while the system is
> mounted and online safely.
>
Thanks for enumerating these benefits. And I agree that these benefits
alone justify the reason for switching to btrfs by default.
I might be tempted to check out the btrfs status page again and start
playing with it.
Give it a shot, I think you'll be pleasantly surprised! :)
--
真実はいつも一つ！/ Always, there's only one truth!
36
days inactive
38
days old
8 comments
3 participants
participants (3)
-
Dusty Mabe -
Neal Gompa -
Tadej Janež