https://fedoraproject.org/wiki/Changes/FedoraCloudBtrfsByDefault
== Summary ==
For cloud installs of Fedora, we want to provide advanced file system features to users in a transparent fashion. Thus, we are changing the file system for the Cloud Edition to Btrfs so we can leverage its features and capabilities to improve the quality of experience for Cloud users.
== Owners ==
* Name: [[User:Davdunc|David Duncan]], [[User:Chrismurphy|Chris Murphy]], [[User:Josef|Josef Bacik]], [[User:Salimma|Michel Alexandre Salim]], [[User:Dcavalca|Davide Cavalca]], [[User:Ngompa|Neal Gompa]], [[User:Dustymabe|Dusty Mabe]], [[User:Malmond|Matthew Almond]] * Email: davdunc@amazon.com, chrismurphy@fedoraproject.org, josef@toxicpanda.com, michel@michel-slm.name, dcavalca@fb.com, ngompa13@gmail.com, dusty@dustymabe.com, malmond@fb.com * Products: Fedora Cloud Edition * Responsible WGs: Fedora Cloud WG
== Detailed Description ==
Fedora Cloud Edition will switch to using Btrfs for its images. The configuration for the Cloud Edition will match the setup used on the desktop variants, as this has been very well-tested with production deployments across multiple Fedora Linux releases now.
This includes the same subvolume layout that is used on the desktop variants [[Changes/BtrfsByDefault|as introduced in Fedora Linux 33]], as well as transparent Zstd compression [[Changes/BtrfsTransparentCompression|as introduced in Fedora Linux 34]].
== Feedback ==
== Benefit to Fedora ==
The benefits are similar to [[Changes/BtrfsByDefault#Benefit_to_Fedora|the ones for Fedora desktop variants]]; however, there are specific benefits for Fedora Cloud:
* Adds support to Fedora Cloud for [[Changes/RPMCoW|the Change to introduce support for Copy-on-Write enhancements to improve performance to package management]] * Adds the ability to logically separate contents of the volume without dividing up the available space ** Transparent compression: significantly reduces write amplification and improves effective I/O throughput ** Reflinks and snapshots improve efficiency for use cases like containers (CRI-O, containerd, and Podman support both) * Storage devices can be flaky, resulting in data corruption; Btrfs can help mitigate this ** Everything is checksummed and verified on every read ** Corrupt data results in EIO (input/output error), instead of resulting in application confusion, and isn’t replicated into backups and archives * Improves system responsiveness under pressure ** Btrfs has been tested in production to have proper IO isolation capability via cgroups2 ** Completes the resource control picture: memory, cpu, IO isolation * File system resize ** Online shrink and grow are cornerstones of the Btrfs design * Complex storage setups are… complicated ** Simple and comprehensive command interface. One master command ** Simpler to boot, all code is in the kernel, no initramfs complexities ** Simple and efficient file system replication, including incremental backups, with <code>btrfs send</code> and <code>btrfs receive</code>
== Scope ==
* Proposal owners: ** Submit PRs for Cloud Edition kickstarts to produce disk images using Btrfs. * Release engineering: [https://pagure.io/releng/issue/10129 #10129] * Policies and guidelines: N/A * Trademark approval: N/A
== Upgrade/compatibility impact ==
Change will not affect upgrades.
== How To Test ==
Once the change lands in Rawhide, spin up the images in AWS, GCP, and KVM/OpenStack to test to see systems boot and run.
== User Experience ==
* Mostly transparent. * Space savings and extend hardware life, via compression. * Utilities for used and free space are expected to behave the same. No special commands are required. * More detailed information can be revealed by <code>btrfs</code> specific commands. * <code>cp</code> command will create reflink copies [https://src.fedoraproject.org/rpms/coreutils/c/5d08d14b/ by default.]
== Dependencies ==
None.
== Contingency Plan ==
* Contingency mechanism: Owner will revert changes back to the previous ext4 configuration * Contingency deadline: Beta freeze * Blocks release? Yes * Blocks product? Cloud
== Documentation ==
Strictly speaking, no extra documentation is required reading for users.
== Release Notes ==
The default file system on the cloud is now Btrfs, following the desktop change in Fedora Linux 33. Fedora Server, IoT, and CoreOS are still specifically excluded.
Am 25.05.2021 um 19:03 schrieb Ben Cotton <bcotton@redhat.com mailto:bcotton@redhat.com>:
https://fedoraproject.org/wiki/Changes/FedoraCloudBtrfsByDefault
== Summary ==
For cloud installs of Fedora, we want to provide advanced file system features to users in a transparent fashion. Thus, we are changing the file system for the Cloud Edition to Btrfs so we can leverage its features and capabilities to improve the quality of experience for Cloud users.
... == Detailed Description ==
Fedora Cloud Edition will switch to using Btrfs for its images. The configuration for the Cloud Edition will match the setup used on the desktop variants, as this has been very well-tested with production deployments across multiple Fedora Linux releases now.
This includes the same subvolume layout that is used on the desktop variants [[Changes/BtrfsByDefault|as introduced in Fedora Linux 33]], as well as transparent Zstd compression [[Changes/BtrfsTransparentCompression|as introduced in Fedora Linux 34]].
Wonder why a cloud image would just have a special affinity for a desktop system. Who runs their desktop in the cloud?
And there is a reason why the partition layout of server and desktop differs.
== Release Notes ==
The default file system on the cloud is now Btrfs, following the desktop change in Fedora Linux 33. Fedora Server, IoT, and CoreOS are still specifically excluded.
It wasn't long ago that the plan was to better align cloud and server.
On Tue, May 25, 2021 at 3:52 PM Peter Boy pboy@uni-bremen.de wrote:
Am 25.05.2021 um 19:03 schrieb Ben Cotton bcotton@redhat.com:
https://fedoraproject.org/wiki/Changes/FedoraCloudBtrfsByDefault
== Summary ==
For cloud installs of Fedora, we want to provide advanced file system features to users in a transparent fashion. Thus, we are changing the file system for the Cloud Edition to Btrfs so we can leverage its features and capabilities to improve the quality of experience for Cloud users.
... == Detailed Description ==
Fedora Cloud Edition will switch to using Btrfs for its images. The configuration for the Cloud Edition will match the setup used on the desktop variants, as this has been very well-tested with production deployments across multiple Fedora Linux releases now.
This includes the same subvolume layout that is used on the desktop variants [[Changes/BtrfsByDefault|as introduced in Fedora Linux 33]], as well as transparent Zstd compression [[Changes/BtrfsTransparentCompression|as introduced in Fedora Linux 34]].
Wonder why a cloud image would just have a special affinity for a desktop system. Who runs their desktop in the cloud?
And there is a reason why the partition layout of server and desktop differs.
The partition layout differed specifically to maximize available space. With subvolumes, this problem doesn't exist, since space allocation is flexible by default across all subvolumes in a partition. So the same model works totally fine for both desktop and server.
== Release Notes ==
The default file system on the cloud is now Btrfs, following the desktop change in Fedora Linux 33. Fedora Server, IoT, and CoreOS are still specifically excluded.
It wasn't long ago that the plan was to better align cloud and server.
That's still a goal, but Server Edition has more complex needs to address before we could do it there by default. Since the storage needs for Cloud Edition are tremendously simpler, we can do it here first and iterate on getting things to the point that we could propose it for Server Edition.
Am 25.05.2021 um 21:56 schrieb Neal Gompa ngompa13@gmail.com:
On Tue, May 25, 2021 at 3:52 PM Peter Boy pboy@uni-bremen.de wrote:
Am 25.05.2021 um 19:03 schrieb Ben Cotton bcotton@redhat.com:
https://fedoraproject.org/wiki/Changes/FedoraCloudBtrfsByDefault
This includes the same subvolume layout that is used on the desktop variants [[Changes/BtrfsByDefault|as introduced in Fedora Linux 33]], as well as transparent Zstd compression [[Changes/BtrfsTransparentCompression|as introduced in Fedora Linux 34]].
Wonder why a cloud image would just have a special affinity for a desktop system. Who runs their desktop in the cloud?
And there is a reason why the partition layout of server and desktop differs.
The partition layout differed specifically to maximize available space.
As to my knowledge it’s not about space (there is no difference in this respect), it's about strikt separation of system and user data, containment in the event of a file system failure, and opportunities and simplification of recovery.
...
So the same model works totally fine for both desktop and server.
I myself lack the exact technical knowledge, but (all?) server and file system experts I hear consider just that a gross misconception.
== Release Notes ==
The default file system on the cloud is now Btrfs, following the desktop change in Fedora Linux 33. Fedora Server, IoT, and CoreOS are still specifically excluded.
It wasn't long ago that the plan was to better align cloud and server.
That's still a goal, but Server Edition has more complex needs to address before we could do it there by default. Since the storage needs for Cloud Edition are tremendously simpler, we can do it here first and iterate on getting things to the point that we could propose it for Server Edition.
That’s fine for me. Practically, this means putting that plan on hold for the next 10 or so Fedora releases.
For this time, we should come up with something else to easily set up a Fedora Server VM in Fedora Server as a transitional measure. Maybe the ARM disk image for SBC's would be a good starting point. This disk image already offers an almost perfect implementation of the structure and concept of Fedora Server Edition, at least much closer than the current Cloud Base disk image. Their composition script might be a good starting point if you leave off the hardware specific parts.
On Tue, May 25, 2021 at 5:05 PM Peter Boy pboy@uni-bremen.de wrote:
Am 25.05.2021 um 21:56 schrieb Neal Gompa ngompa13@gmail.com:
On Tue, May 25, 2021 at 3:52 PM Peter Boy pboy@uni-bremen.de wrote:
Am 25.05.2021 um 19:03 schrieb Ben Cotton bcotton@redhat.com:
https://fedoraproject.org/wiki/Changes/FedoraCloudBtrfsByDefault
This includes the same subvolume layout that is used on the desktop variants [[Changes/BtrfsByDefault|as introduced in Fedora Linux 33]], as well as transparent Zstd compression [[Changes/BtrfsTransparentCompression|as introduced in Fedora Linux 34]].
Wonder why a cloud image would just have a special affinity for a desktop system. Who runs their desktop in the cloud?
And there is a reason why the partition layout of server and desktop differs.
The partition layout differed specifically to maximize available space.
As to my knowledge it’s not about space (there is no difference in this respect), it's about strikt separation of system and user data, containment in the event of a file system failure, and opportunities and simplification of recovery.
Cloud images never had such separation. It was always one big ext4 partition. With Btrfs, we can introduce subvolumes so user data can be trivially managed separately without losing the benefits of a single pool of storage.
...
So the same model works totally fine for both desktop and server.
I myself lack the exact technical knowledge, but (all?) server and file system experts I hear consider just that a gross misconception.
In the cloud use-case, it is relatively rare for important data to be on the OS partition. Either a secondary volume is attached and formatted, or people are using some kind of distributed storage (such as Ceph, Gluster, S3, etc.).
The net effect of the subvolume setup is that people who had individual user data on the OS volume could now cleanly isolate that from the rest of the data on the OS volume without worrying about space contention.
== Release Notes ==
The default file system on the cloud is now Btrfs, following the desktop change in Fedora Linux 33. Fedora Server, IoT, and CoreOS are still specifically excluded.
It wasn't long ago that the plan was to better align cloud and server.
That's still a goal, but Server Edition has more complex needs to address before we could do it there by default. Since the storage needs for Cloud Edition are tremendously simpler, we can do it here first and iterate on getting things to the point that we could propose it for Server Edition.
That’s fine for me. Practically, this means putting that plan on hold for the next 10 or so Fedora releases.
Why would we wait 5 years? That seems like a weird overreaction here. We literally talked about how this would go in the last Fedora Server meeting[1] with us agreeing that Fedora Cloud would be a starting point for Btrfs for server use-cases. Starting with the simpler cases (from a storage perspective) is a good way to see how things go.
[1]: https://meetbot.fedoraproject.org/teams/fedora-server/fedora-server.2021-05-...
For this time, we should come up with something else to easily set up a Fedora Server VM in Fedora Server as a transitional measure. Maybe the ARM disk image for SBC's would be a good starting point. This disk image already offers an almost perfect implementation of the structure and concept of Fedora Server Edition, at least much closer than the current Cloud Base disk image. Their composition script might be a good starting point if you leave off the hardware specific parts.
Fedora Cloud was never set up as a "Fedora Server" lookalike in the first place. That's why it's a distinct Edition.
Some examples of differences that are already present:
* We used single ext4 instead of xfs+lvm with logical volumes for root and home * Set up with cloud-init instead of standard boot * No cockpit software
If you want to do an "easy" Fedora Server as a VM on classical KVM hosts, you're probably better off with using virt-install[2]. Setting up Fedora Cloud images on regular KVM hosts is a pain and not the intended use-case. Working with cloud-init or ignition in classical VM models is painful and probably not what people want to do for that type of setup.
[2]: https://www.mankier.com/1/virt-install
Am 25.05.2021 um 23:20 schrieb Neal Gompa ngompa13@gmail.com:
On Tue, May 25, 2021 at 5:05 PM Peter Boy pboy@uni-bremen.de wrote:
As to my knowledge it’s not about space (there is no difference in this respect), it's about strikt separation of system and user data, containment in the event of a file system failure, and opportunities and simplification of recovery.
Cloud images never had such separation. It was always one big ext4 partition. With Btrfs, we can introduce subvolumes so user data can be trivially managed separately without losing the benefits of a single pool of storage.
That’s the difference: As a server sysadmin I would rather consider potential risks of a single pool of storage. :-)
The net effect of the subvolume setup is that people who had individual user data on the OS volume could now cleanly isolate that from the rest of the data on the OS volume without worrying about space contention.
What I get from the controversial discussion about BTRFS is that there is doubt about how really clean that "cleanly isolate" is, at least compared to LVM. But if important data is (supposed to be) on additional external storage, that's perfect.
That’s fine for me. Practically, this means putting that plan on hold for the next 10 or so Fedora releases.
Why would we wait 5 years? That seems like a weird overreaction here.
No overreaction, just a little sense of reality. I see a lot of concerns about stability and reliability of BTRFS and a widespread reluctance to move servers to it. In any case, nothing in the near future.
Fedora Cloud was never set up as a "Fedora Server" lookalike in the first place. That's why it's a distinct Edition. . . . If you want to do an "easy" Fedora Server as a VM on classical KVM hosts, you're probably better off with using virt-install[2].
It is not me. It was Matthew (and others) who introduced the concept of Fedora Server Edition and Cloud Base Image as a kind of two sides of a coin. And related to this is the idea of uniting Fedora Server and Cloud image working groups. Stimulated by this, I evaluated a bit the practicability of this concept and wrote an article in Fedora Magazine about Cloud Base Image as VM in Fedora Server, by the way.
Regarding that idea, the switch to BTRFS tends to be a step in the wrong direction.
On Tue, May 25, 2021 at 5:52 PM Peter Boy pboy@uni-bremen.de wrote:
Am 25.05.2021 um 23:20 schrieb Neal Gompa ngompa13@gmail.com: Cloud images never had such separation. It was always one big ext4 partition. With Btrfs, we can introduce subvolumes so user data can be trivially managed separately without losing the benefits of a single pool of storage.
That’s the difference: As a server sysadmin I would rather consider potential risks of a single pool of storage. :-)
OK, let's consider 2-3 potential risks.
Hard drive mechanical failure (spindle, motor, actuator, rw head), no matter what file system and partitioning you use, you are out of luck. It's a 100% fail. But if it's a media defect, torn or misdirected write, it's quite different. Btrfs is the only linux filesystem that has a chance of surviving such cases while remaining in normal operation. That's because by default on hard drives, it keeps two copies of file system metadata in different locations on the drive, and can self-heal so long as there's one good copy of a block. If the problem results in corruption of data, only Btrfs checksums data, so it unambiguously knows if the data lacks integrity, and will refuse to propagate corrupt data to user space.
In the case of SSDs and virtual block device backed by multiple physical drives, duplicate metadata may not help anyway. But since data is by far the biggest portion of a file system volume other than free space, it's a much bigger target for any problems that occur with the storage stack. Btrfs is the best early detection system for such problems, which can only get worse. Early detection is the benefit. Btrfs is like a canary in a coal mine, even if it can't self-heal or stop the impending doom.
What are the potential risks you are concerned about in the cloud and server use cases? Please be specific.
The net effect of the subvolume setup is that people who had individual user data on the OS volume could now cleanly isolate that from the rest of the data on the OS volume without worrying about space contention.
What I get from the controversial discussion about BTRFS is that there is doubt about how really clean that "cleanly isolate" is, at least compared to LVM. But if important data is (supposed to be) on additional external storage, that's perfect.
What controversial discussion is being referenced? Currently before us is a proposal to switch to Btrfs for Cloud edition. There isn't a proposal for Btrfs for Server edition, nor is there a proposal to use LVM in Cloud edition. So I'm kinda confused about what you want to discuss.
If the case for Btrfs for Cloud isn't strong enough in the proposal; it's most helpful if you could directly criticize what's wrong, missing, or unpersuasive about it.
That’s fine for me. Practically, this means putting that plan on hold for the next 10 or so Fedora releases.
Why would we wait 5 years? That seems like a weird overreaction here.
No overreaction, just a little sense of reality. I see a lot of concerns about stability and reliability of BTRFS and a widespread reluctance to move servers to it. In any case, nothing in the near future.
Could you point out some of these real concerns about stability and reliability of Btrfs? Btrfs isn't perfect, but I'm very frequently the person of first contact in Fedora for Btrfs issues, and we're really not seeing many problems even when including problems resulting from hardware defects like bad RAM and drive firmware dropping writes. And all of the reported problems get a followup.
I'm not sure what qualifies as widespread reluctance, or what the metric is. However, I am absolutely confident that if you are asserting Fedora should avoid technology that no one else is using, the community should push back on such a misapplication of logic.
Regarding that idea, the switch to BTRFS tends to be a step in the wrong direction.
Cloud and Server editions have had different file system and partitioning strategies since day 1. Today is the first I've heard that there should be, could be, would be, some kind of realignment where Cloud uses LVM or Server uses ext4, or some other combination. But you are now asserting that this alignment should happen? And are you asserting that this is a central question needing an answer as a prerequisite for evaluating this Fedora 35 change proposal?
Am 26.05.2021 um 08:51 schrieb Chris Murphy lists@colorremedies.com:
On Tue, May 25, 2021 at 5:52 PM Peter Boy pboy@uni-bremen.de wrote:
Am 25.05.2021 um 23:20 schrieb Neal Gompa ngompa13@gmail.com: Cloud images never had such separation. It was always one big ext4 partition. With Btrfs, we can introduce subvolumes so user data can be trivially managed separately without losing the benefits of a single pool of storage.
That’s the difference: As a server sysadmin I would rather consider potential risks of a single pool of storage. :-)
OK, let's consider 2-3 potential risks.
Hard drive mechanical failure (spindle, motor, actuator, rw head), no matter what file system and partitioning you use, you are out of luck. ...
What are the potential risks you are concerned about in the cloud and server use cases? Please be specific.
That sounds very promising.
But I wanted to emphasize something else: The criteria and the point of view are different. For logical reasons, an argument from one context has no assertiveness in the other context. The same detail is evaluated differently in a different context.
What I get from the controversial discussion about BTRFS is that there is doubt about how really clean that "cleanly isolate" is, at least compared to LVM. But if important data is (supposed to be) on additional external storage, that's perfect.
What controversial discussion is being referenced? Currently before us is a proposal to switch to Btrfs for Cloud edition.
I’m quite sure you know that discussion. Part to it was Red Hat’s decision to drop BTRFS in RHEL 8 (to my dismay, by the way, because I used it for some file systems on our servers). And because there is that decision and that discussion, I expect it will take a longer time to switch e.g. server to BTRFS as system wide default. So my casual 10-year forecast is not an overreaction as Neal put it (at most some pessimism).
I think we have a misunderstanding here. My argument refers to expected hurdles of a possible changeover process, not to technical features.
Regarding that idea, the switch to BTRFS tends to be a step in the wrong direction.
Cloud and Server editions have had different file system and partitioning strategies since day 1. Today is the first I've heard that there should be, could be, would be, some kind of realignment where Cloud uses LVM or Server uses ext4, or some other combination. But you are now asserting that this alignment should happen? And are you asserting that this is a central question needing an answer as a prerequisite for evaluating this Fedora 35 change proposal?
Again, it’s not about technical properties. We have (or probably had?) an agreement to align (or try to align) Server Edition and Cloud. That was 2 or 3 months ago. Regarding to that agreement, it is a step into the wrong direction.
But maybe in the meantime we had another trend coming up. Neal came up with some arguments about a general difference between both so an alignment is seen as not possible or not useful.
I’m not a stakeholder for any decision about that. And I have no intention of missionizing for any of the options. I’m just the bookkeeper and try to ensure that nothing is forgotten and everything follows a solid and consistent logic. And because I am not a native English speaker, some wording may be misleading, much to my chagrin.
So we may drop it from the table. I’ve put it on the agenda of todays Server IRC meeting.
Best Peter
On Wed, May 26, 2021 at 5:30 AM Peter Boy pboy@uni-bremen.de wrote:
Am 26.05.2021 um 08:51 schrieb Chris Murphy lists@colorremedies.com: What controversial discussion is being referenced? Currently before us is a proposal to switch to Btrfs for Cloud edition.
I’m quite sure you know that discussion. Part to it was Red Hat’s decision to drop BTRFS in RHEL 8 (to my dismay, by the way, because I used it for some file systems on our servers). And because there is that decision and that discussion, I expect it will take a longer time to switch e.g. server to BTRFS as system wide default. So my casual 10-year forecast is not an overreaction as Neal put it (at most some pessimism).
I understand you think it will take longer. What I don't understand is why you think this, but I guess we can just skip over that.
But let's keep the Red Hat aspect of the storyline in context though. And not extrapolate beyond the available facts. https://news.ycombinator.com/item?id=14907771
Whereupon Server SIG/WG perform an evaluation of Btrfs for their use cases, and decide Btrfs should be the default in a compelling manner, FESCo will approve it. And this plausibly could still happen for Fedora 35, if folks really want it to happen. But I think it's neither urgent nor requires a long delay. Server SIG can do anything they want. Red Hat is doing the same.
I think we have a misunderstanding here. My argument refers to expected hurdles of a possible changeover process, not to technical features.
My opinion is to not worry about the process in advance of arriving at the hurdle. You jump over the hurdle at the proper time. The vast majority of the process is about technical features liabilities.
For example, I expect Server folks probably prefer LVM LV's for backing virtual machines, rather than raw or qcow2 no matter the file system. Even if this can be approximated by fallocate (raw and qcow2 support it, as as well as ext4, xfs, btrfs) to preallocate the backing file, it's likely a lot of Server folks will just prefer using LVM for this case. That's fine.
A good question is to what degree can Server edition, via Anaconda kickstart, divvy up a drive in boolean fashion? I know there's some thresholds like, if the drive is below a certain size, don't create /home, and so on. Could Server default installations default to a ~20G Btrfs system volume; and either leave the rest unallocated (not partitioned)? Or make all remaining space an LVM PV, added to a single VG? And then just not create any LVs? I think I'd like to see each: unallocated, PV only, PV + VG, in Cockpit's UI and get some feedback from those folks because it'd be ideal if the overview of the initially installed system can clearly show the layout, whatever it is.
Not often but sometimes folks ask "where has all the space gone?" following a Server installation. They're not expecting or maybe not discovering, that quite a lot is held in reserve in the VG.
Again, it’s not about technical properties. We have (or probably had?) an agreement to align (or try to align) Server Edition and Cloud. That was 2 or 3 months ago. Regarding to that agreement, it is a step into the wrong direction.
Is there a Server or Cloud meeting with minutes that this discussion happened in? Or email thread you can point to?
-- Chris Murphy
On Wed, 2021-05-26 at 16:59 -0600, Chris Murphy wrote:
On Wed, May 26, 2021 at 5:30 AM Peter Boy pboy@uni-bremen.de wrote:
Am 26.05.2021 um 08:51 schrieb Chris Murphy < lists@colorremedies.com>: What controversial discussion is being referenced? Currently before us is a proposal to switch to Btrfs for Cloud edition.
I’m quite sure you know that discussion. Part to it was Red Hat’s decision to drop BTRFS in RHEL 8 (to my dismay, by the way, because I used it for some file systems on our servers). And because there is that decision and that discussion, I expect it will take a longer time to switch e.g. server to BTRFS as system wide default. So my casual 10-year forecast is not an overreaction as Neal put it (at most some pessimism).
I understand you think it will take longer. What I don't understand is why you think this, but I guess we can just skip over that.
But let's keep the Red Hat aspect of the storyline in context though. And not extrapolate beyond the available facts. https://news.ycombinator.com/item?id=14907771
Whereupon Server SIG/WG perform an evaluation of Btrfs for their use cases, and decide Btrfs should be the default in a compelling manner, FESCo will approve it. And this plausibly could still happen for Fedora 35, if folks really want it to happen. But I think it's neither urgent nor requires a long delay. Server SIG can do anything they want. Red Hat is doing the same.
I think we have a misunderstanding here. My argument refers to expected hurdles of a possible changeover process, not to technical features.
My opinion is to not worry about the process in advance of arriving at the hurdle. You jump over the hurdle at the proper time. The vast majority of the process is about technical features liabilities.
For example, I expect Server folks probably prefer LVM LV's for backing virtual machines, rather than raw or qcow2 no matter the file system. Even if this can be approximated by fallocate (raw and qcow2 support it, as as well as ext4, xfs, btrfs) to preallocate the backing file, it's likely a lot of Server folks will just prefer using LVM for this case. That's fine.
A good question is to what degree can Server edition, via Anaconda kickstart, divvy up a drive in boolean fashion? I know there's some thresholds like, if the drive is below a certain size, don't create /home, and so on. Could Server default installations default to a ~20G Btrfs system volume; and either leave the rest unallocated (not partitioned)? Or make all remaining space an LVM PV, added to a single VG? And then just not create any LVs? I think I'd like to see each: unallocated, PV only, PV + VG, in Cockpit's UI and get some feedback from those folks because it'd be ideal if the overview of the initially installed system can clearly show the layout, whatever it is.
Not often but sometimes folks ask "where has all the space gone?" following a Server installation. They're not expecting or maybe not discovering, that quite a lot is held in reserve in the VG.
Again, it’s not about technical properties. We have (or probably had?) an agreement to align (or try to align) Server Edition and Cloud. That was 2 or 3 months ago. Regarding to that agreement, it is a step into the wrong direction.
Is there a Server or Cloud meeting with minutes that this discussion happened in? Or email thread you can point to?
As I recall it was very preliminary, and the suggestion is that the working groups can be merged at some point in the future.
- mattdm brought it up back in December
https://meetbot.fedoraproject.org/teams/serversig/serversig.2020-12-16-18.00... (timestamp: 18:52:23)
- there's discussion about talking to Cloud WG on April 7
https://meetbot.fedoraproject.org/teams/fedora-server/fedora-server.2021-04-... (timestamp: 17:44:55)
I don't think there was ever a discussion of making the two products technically aligned (nor do I really see a need, even if the working groups end up being merged).
There has not been an official proposal; to my recollection it has not been brought up at a Cloud WG meeting.
Best regards,
Am 27.05.2021 um 00:59 schrieb Chris Murphy lists@colorremedies.com:
On Wed, May 26, 2021 at 5:30 AM Peter Boy pboy@uni-bremen.de wrote: ... Whereupon Server SIG/WG perform an evaluation of Btrfs for their use cases, and decide Btrfs should be the default in a compelling manner, FESCo will approve it. And this plausibly could still happen for Fedora 35, if folks really want it to happen.
Theoretically, it could. But practically, I don't see it happen. The need for discussion is too great. And not everyone is as convinced as you are that BTRFS is the non-plus-ultra for all possible use cases.
Server SIG can do anything they want. Red Hat is doing the same.
Nevertheless, coordination and cooperation is at least very desirable (in fact, indispensable). And is is not just about who is paying the bills. Beyond this crude economic dimension, Fedora benefits from the reputation of being upstream for RHEL (and vice versa, for sure). A defiant "we can do as we like" is not helpful.
I think we have a misunderstanding here. My argument refers to expected hurdles of a possible changeover process, not to technical features.
My opinion is to not worry about the process in advance of arriving at the hurdle. You jump over the hurdle at the proper time. The vast majority of the process is about technical features liabilities.
...
And when we address discussion and evidence:
Not often but sometimes folks ask "where has all the space gone?" following a Server installation. They're not expecting or maybe not discovering, that quite a lot is held in reserve in the VG.
As said before, I agree with that, at least for the most part. I use BTRFS myself in LVs to use specific capabilities. Still, I'm against converting "with a flick of the wrist," so to speak. It needs careful preparation. And one possible outcome is also, not to switch to BTRFS. I don't think it is a given that a switch is right in any case. That is perhaps the difference between us.
Again, it’s not about technical properties. We have (or probably had?) an agreement to align (or try to align) Server Edition and Cloud. That was 2 or 3 months ago. Regarding to that agreement, it is a step into the wrong direction.
Is there a Server or Cloud meeting with minutes that this discussion happened in? Or email thread you can point to?
Michel gave you the link. And there were several brief comments about that in previous meetings and also before that reboot event.
And when we address discussion and evidence: What I miss is a prior detailed discussion of this change in cloud WG and coordination with other possible affected areas, e.g. server or CoreOS. Cloud Working Group did not happened for years, then there were a few short, sparsely-attended and content-dry meetings. A range of existing problems, starting with lack of documentation. A hesitancy to make any change currently to the cloud artifacts (expressed by Dusty Mabe at that March meeting, 3). And then out of nowhere the file system conversion, a very central element. To me, it seems like a playground for missionaries to gain ground, certainly not like a considerate and methodical long-term design.
On Fri, May 28, 2021 at 3:45 AM Peter Boy pboy@uni-bremen.de wrote:
Am 27.05.2021 um 00:59 schrieb Chris Murphy lists@colorremedies.com:
On Wed, May 26, 2021 at 5:30 AM Peter Boy pboy@uni-bremen.de wrote: ... Whereupon Server SIG/WG perform an evaluation of Btrfs for their use cases, and decide Btrfs should be the default in a compelling manner, FESCo will approve it. And this plausibly could still happen for Fedora 35, if folks really want it to happen.
Theoretically, it could. But practically, I don't see it happen. The need for discussion is too great. And not everyone is as convinced as you are that BTRFS is the non-plus-ultra for all possible use cases.
Server SIG can do anything they want. Red Hat is doing the same.
Nevertheless, coordination and cooperation is at least very desirable (in fact, indispensable). And is is not just about who is paying the bills. Beyond this crude economic dimension, Fedora benefits from the reputation of being upstream for RHEL (and vice versa, for sure). A defiant "we can do as we like" is not helpful.
Being upstream for RHEL also means pushing forward and demonstrating that things *can* work better in a different way. If we didn't, there's no way for RHEL to change. We bring no value as upstream if we don't actually *do* things.
I think we have a misunderstanding here. My argument refers to expected hurdles of a possible changeover process, not to technical features.
My opinion is to not worry about the process in advance of arriving at the hurdle. You jump over the hurdle at the proper time. The vast majority of the process is about technical features liabilities.
...
And when we address discussion and evidence:
Not often but sometimes folks ask "where has all the space gone?" following a Server installation. They're not expecting or maybe not discovering, that quite a lot is held in reserve in the VG.
As said before, I agree with that, at least for the most part. I use BTRFS myself in LVs to use specific capabilities. Still, I'm against converting "with a flick of the wrist," so to speak. It needs careful preparation. And one possible outcome is also, not to switch to BTRFS. I don't think it is a given that a switch is right in any case. That is perhaps the difference between us.
As I'll say further down, this was definitely not a "flick of the wrist" or a "snap" decision.
Again, it’s not about technical properties. We have (or probably had?) an agreement to align (or try to align) Server Edition and Cloud. That was 2 or 3 months ago. Regarding to that agreement, it is a step into the wrong direction.
Is there a Server or Cloud meeting with minutes that this discussion happened in? Or email thread you can point to?
Michel gave you the link. And there were several brief comments about that in previous meetings and also before that reboot event.
And when we address discussion and evidence: What I miss is a prior detailed discussion of this change in cloud WG and coordination with other possible affected areas, e.g. server or CoreOS.
Part of the point of the different working groups was to handle the different use-cases *well* at their own pace. The CoreOS Working Group is *explicitly* excluded and frankly unlikely to ever switch because Colin believes that Btrfs is only suitable for "pet" workloads[1] despite the evidence to the contrary[2][3][4].
[1]: https://blog.verbum.org/2020/07/14/on-btrfs/ [2]: https://facebookmicrosites.github.io/btrfs/docs/btrfs-facebook.html [3]: https://www.youtube.com/watch?v=U7gXR2L05IU [4]: https://lwn.net/Articles/824855/
Cloud Working Group did not happened for years, then there were a few short, sparsely-attended and content-dry meetings. A range of existing problems, starting with lack of documentation. A hesitancy to make any change currently to the cloud artifacts (expressed by Dusty Mabe at that March meeting, 3). And then out of nowhere the file system conversion, a very central element. To me, it seems like a playground for missionaries to gain ground, certainly not like a considerate and methodical long-term design.
Actually, the Cloud WG members had been talking about this ad-hoc since all the desktop variants switched last year. The ticket[4] was filed around the same time the desktop change was proposed. Dusty, Joe, James, and myself had been talking about it outside of meetings since. David became interested after the Nest talk about Btrfs[5]. That conversation started again after my talk on Btrfs at DevConf.cz (the recording has not been published still, sadly). So this has been a long time coming.
[4]: https://pagure.io/cloud-sig/issue/308 [5]: https://www.youtube.com/watch?v=iHjhouSxIrc
-- 真実はいつも一つ!/ Always, there's only one truth!
On Fri, May 28, 2021, at 5:43 AM, Neal Gompa wrote:
Part of the point of the different working groups was to handle the different use-cases *well* at their own pace. The CoreOS Working Group is *explicitly* excluded and frankly unlikely to ever switch because Colin believes
I am not CoreOS, I'm an engineer working on it.
that Btrfs is only suitable for "pet" workloads[1]
That's not what my blog said. One of the sub-headers is "BTRFS is good for "pet" systems". There's no word "only" there - you inserted that.
On this topic though, if Fedora CoreOS didn't exist, this proposal to change Cloud would be significantly more consequential. The defaults *really matter* here in particular, even more than Workstation. But, I think because CoreOS does exist, this change matters less.
One big advantage CoreOS has is Ignition, which allows provisioning filesystems in the initramfs, including the root partition. It works today to boot up a stock Fedora CoreOS AMI, OpenStack qcow2 etc. and provide an Ignition config that changes it: https://docs.fedoraproject.org/en-US/fedora-coreos/storage/#_reconfiguring_t... And it works to use btrfs too! Just s/ext4/btrfs/ in the first example. (And that *exact same* Ignition config also works for bare metal)
That's not true of cloud-init based systems - instead of e.g. you wanted to use xfs/ext4 for a high performance database-like workload, in cloud you'd need to attach a separate instance volume or so for `/var/lib/$whatever` (That said separate volumes can actually be a better approach anyways, it depends). Some traditional Cloud image users won't notice this or care (just like Workstation users) but others definitely will.
On Fri, May 28, 2021 at 9:04 AM Colin Walters walters@verbum.org wrote:
On Fri, May 28, 2021, at 5:43 AM, Neal Gompa wrote:
Part of the point of the different working groups was to handle the different use-cases *well* at their own pace. The CoreOS Working Group is *explicitly* excluded and frankly unlikely to ever switch because Colin believes
I am not CoreOS, I'm an engineer working on it.
that Btrfs is only suitable for "pet" workloads[1]
That's not what my blog said. One of the sub-headers is "BTRFS is good for "pet" systems". There's no word "only" there - you inserted that.
On this topic though, if Fedora CoreOS didn't exist, this proposal to change Cloud would be significantly more consequential. The defaults *really matter* here in particular, even more than Workstation. But, I think because CoreOS does exist, this change matters less.
One big advantage CoreOS has is Ignition, which allows provisioning filesystems in the initramfs, including the root partition. It works today to boot up a stock Fedora CoreOS AMI, OpenStack qcow2 etc. and provide an Ignition config that changes it:
No need for "ignition". This has worked since grub and anaconda were created by using '%pre' scripts in anaconda to pre-partition the file-system and potentially insert other pre-configuration steps in it. If ignition has new integration features rather than the manual scripting I used to do, great. But ignition is not required for this.
On Fri, May 28, 2021 at 4:37 PM Nico Kadel-Garcia nkadel@gmail.com wrote:
On Fri, May 28, 2021 at 9:04 AM Colin Walters walters@verbum.org wrote:
On Fri, May 28, 2021, at 5:43 AM, Neal Gompa wrote:
Part of the point of the different working groups was to handle the different use-cases *well* at their own pace. The CoreOS Working Group is *explicitly* excluded and frankly unlikely to ever switch because Colin believes
I am not CoreOS, I'm an engineer working on it.
that Btrfs is only suitable for "pet" workloads[1]
That's not what my blog said. One of the sub-headers is "BTRFS is good for "pet" systems". There's no word "only" there - you inserted that.
On this topic though, if Fedora CoreOS didn't exist, this proposal to change Cloud would be significantly more consequential. The defaults *really matter* here in particular, even more than Workstation. But, I think because CoreOS does exist, this change matters less.
One big advantage CoreOS has is Ignition, which allows provisioning filesystems in the initramfs, including the root partition. It works today to boot up a stock Fedora CoreOS AMI, OpenStack qcow2 etc. and provide an Ignition config that changes it:
No need for "ignition". This has worked since grub and anaconda were created by using '%pre' scripts in anaconda to pre-partition the file-system and potentially insert other pre-configuration steps in it. If ignition has new integration features rather than the manual scripting I used to do, great. But ignition is not required for this.
Fedora Cloud does not use Anaconda for provisioning, so that whole strategy does not work.
On Fri, May 28, 2021 at 7:04 AM Colin Walters walters@verbum.org wrote:
On this topic though, if Fedora CoreOS didn't exist, this proposal to change Cloud would be significantly more consequential. The defaults *really matter* here in particular, even more than Workstation. But, I think because CoreOS does exist, this change matters less.
Well it's sorta ancient history now, but CoreOS started out on Btrfs by default because of the feature set, including the early native btrfs driver support taking advantage of cheap snapshots. CoreOS moved off of Btrfs because of ENOSPC bugs, and performance impact of the various work arounds (all predating ticketed enospc infrastructure).
"Source of truth" related features that I think could be relevant for cloud use cases:
* (immutable) read-only subvolumes, root can't write to them * (immutable) read-only Btrfs seed device images, truly resettable just by dropping the writable device(s) * checksums for metadata and data: crc32c, xxhash, blake2, sha256 * currently in-development: fsverity support * currently in-development: fscrypt support * currently in-development: authenticated (keyed) checksumming support
And generally usable, whether image used for provisioning, or installed sysroot, or workload data: native compression.
And out of band deduplication.
One big advantage CoreOS has is Ignition, which allows provisioning filesystems in the initramfs, including the root partition. It works today to boot up a stock Fedora CoreOS AMI, OpenStack qcow2 etc. and provide an Ignition config that changes it: https://docs.fedoraproject.org/en-US/fedora-coreos/storage/#_reconfiguring_t... And it works to use btrfs too! Just s/ext4/btrfs/ in the first example. (And that *exact same* Ignition config also works for bare metal)
That's not true of cloud-init based systems - instead of e.g. you wanted to use xfs/ext4 for a high performance database-like workload, in cloud you'd need to attach a separate instance volume or so for `/var/lib/$whatever` (That said separate volumes can actually be a better approach anyways, it depends). Some traditional Cloud image users won't notice this or care (just like Workstation users) but others definitely will.
Not all databases are cow-unfriendly. RocksDB and sqlite with WAL enabled do fine on Btrfs, with datacow. There's also been a bunch of database+fsync related performance enhancements in ever kernel release for the past year since Btrfs because the default file system on the desktop. Last I checked (a year ago) postgreSQL did have some performance issues on Btrfs, but I don't know the significance, in particular with today's kernel. Interested parties could create a micro-SIG, I'm happy to help coordinate, and do some testing and document best practices.
-- Chris Murphy
Am 28.05.2021 um 11:43 schrieb Neal Gompa ngompa13@gmail.com:
On Fri, May 28, 2021 at 3:45 AM Peter Boy pboy@uni-bremen.de wrote:
Nevertheless, coordination and cooperation is at least very desirable (in fact, indispensable). And is is not just about who is paying the bills. Beyond this crude economic dimension, Fedora benefits from the reputation of being upstream for RHEL (and vice versa, for sure). A defiant "we can do as we like" is not helpful
Being upstream for RHEL also means pushing forward and demonstrating that things *can* work better in a different way. If we didn't, there's no way for RHEL to change. We bring no value as upstream if we don't actually *do* things.
Agreed. And yet it requires coordination and discussion, in a transparent and comprehensible manner.
Cloud Working Group did not happened for years, then there were a few short, sparsely-attended and content-dry meetings. A range of existing problems, starting with lack of documentation. A hesitancy to make any change currently to the cloud artifacts (expressed by Dusty Mabe at that March meeting, 3). And then out of nowhere the file system conversion, a very central element. To me, it seems like a playground for missionaries to gain ground, certainly not like a considerate and methodical long-term design.
Actually, the Cloud WG members had been talking about this ad-hoc since all the desktop variants switched last year. The ticket[4] was filed around the same time the desktop change was proposed. Dusty, Joe, James, and myself had been talking about it outside of meetings since. David became interested after the Nest talk about Btrfs[5]. That conversation started again after my talk on Btrfs at DevConf.cz (the recording has not been published still, sadly). So this has been a long time coming.
I would have expected at least a public announcement on the cloud mailing list including an invitation for discussion at an IRC meeting and - for such a far-reaching matter - a subsequent voting.
But I'm not involved in the Cloud SIG. So let's close the discussion.
On Fri, May 28, 2021 at 1:45 AM Peter Boy pboy@uni-bremen.de wrote:
Am 27.05.2021 um 00:59 schrieb Chris Murphy lists@colorremedies.com:
Whereupon Server SIG/WG perform an evaluation of Btrfs for their use cases, and decide Btrfs should be the default in a compelling manner, FESCo will approve it. And this plausibly could still happen for Fedora 35, if folks really want it to happen.
Theoretically, it could. But practically, I don't see it happen. The need for discussion is too great. And not everyone is as convinced as you are that BTRFS is the non-plus-ultra for all possible use cases.
All I mean by this is to push back on the idea that the proposal for Cloud translates into delaying the decision for Server by 5 or 10 years. Not that Server folks should escalate their discussion.
Also, I don't think Btrfs is the end all for all use cases; I gave an example to the contrary in the previous email. There are always trade offs. The conversation should focus on what those tradeoffs are and how much each SIG values them.
Server SIG can do anything they want. Red Hat is doing the same.
Nevertheless, coordination and cooperation is at least very desirable (in fact, indispensable). And is is not just about who is paying the bills. Beyond this crude economic dimension, Fedora benefits from the reputation of being upstream for RHEL (and vice versa, for sure). A defiant "we can do as we like" is not helpful.
It isn't defiance, it's conviction consistent with Fedora's mission and the four foundations. My working assumption is substantive public discussion, to reveal the pros and cons of the proposal, in order to come to a decision. The proposal is not the decision.
My opinion is to not worry about the process in advance of arriving at the hurdle. You jump over the hurdle at the proper time. The vast majority of the process is about technical features liabilities.
...
And when we address discussion and evidence:
Not often but sometimes folks ask "where has all the space gone?" following a Server installation. They're not expecting or maybe not discovering, that quite a lot is held in reserve in the VG.
As said before, I agree with that, at least for the most part. I use BTRFS myself in LVs to use specific capabilities. Still, I'm against converting "with a flick of the wrist," so to speak. It needs careful preparation. And one possible outcome is also, not to switch to BTRFS. I don't think it is a given that a switch is right in any case. That is perhaps the difference between us.
I agree with all of these things. But from my point of view they are obvious, to the degree that since you're stating them, it makes me wonder whether you think something has happened abruptly, frivolously, or without sufficient care and preparation. In your view should something have occurred before the proposal was submitted that didn't?
And when we address discussion and evidence: What I miss is a prior detailed discussion of this change in cloud WG and coordination with other possible affected areas, e.g. server or CoreOS. Cloud Working Group did not happened for years, then there were a few short, sparsely-attended and content-dry meetings. A range of existing problems, starting with lack of documentation. A hesitancy to make any change currently to the cloud artifacts (expressed by Dusty Mabe at that March meeting, 3). And then out of nowhere the file system conversion, a very central element. To me, it seems like a playground for missionaries to gain ground, certainly not like a considerate and methodical long-term design.
I don't agree it happened out of nowhere. It's been floated by various folks over the years, even before Workstation edition switched to Btrfs by default. Fedora has quite a lot of sprawl, it's a diverse community, not all conversations happen on devel@ so it can be easy to draw a conclusion that it's sudden. But that is the whole point of the change proposal process, is to make a broad and grand announcement on the primary development list, expressly because we don't want folks missing big changes. Now is exactly the time to dig into the drawbacks, liabilities, risks of proposals, and weigh them against the proponents' typically strong take in favor of the change or else they probably wouldn't have submitted the proposal in the first place.
Take it from me, I really like the adversarial process. I don't mean this in the negative connotation, but rather the legal denotation. We should have a debate. That time is right now, in this thread. And I welcome it.
Am 28.05.2021 um 23:08 schrieb Chris Murphy lists@colorremedies.com:
……
All I mean by this is to push back on the idea that the proposal for Cloud translates into delaying the decision for Server by 5 or 10 years. Not that Server folks should escalate their discussion.
What I originally meant was the idea to discuss / explore opportunities to coordinate or join the efforts of Server WG and Cloud WG and to better aline both.
But that idea seems to me got silently out of attention over time. We discussed that an the beginning of March, Server WG picked it up again at the end of March, and there was a brief mention in Cloud WG in early April. Since then there has been silence.
My working assumption is substantive public discussion, to reveal the pros and cons of the proposal, in order to come to a decision. The proposal is not the decision.
We fully agree on that.
And when we address discussion and evidence: What I miss is a prior detailed discussion of this change in cloud WG and coordination with other possible affected areas, e.g. server or CoreOS. Cloud Working Group did not happened for years, then there were a few short, sparsely-attended and content-dry meetings. A range of existing problems, starting with lack of documentation. A hesitancy to make any change currently to the cloud artifacts (expressed by Dusty Mabe at that March meeting, 3). And then out of nowhere the file system conversion, a very central element. To me, it seems like a playground for missionaries to gain ground, certainly not like a considerate and methodical long-term design.
I don't agree it happened out of nowhere. It's been floated by various folks over the years, even before Workstation edition switched to Btrfs by default. Fedora has quite a lot of sprawl, it's a diverse community, not all conversations happen on devel@ so it can be easy to draw a conclusion that it's sudden. But that is the whole point of the change proposal process, is to make a broad and grand announcement on the primary development list, expressly because we don't want folks missing big changes. Now is exactly the time to dig into the drawbacks, liabilities, risks of proposals, and weigh them against the proponents' typically strong take in favor of the change or else they probably wouldn't have submitted the proposal in the first place.
Take it from me, I really like the adversarial process. I don't mean this in the negative connotation, but rather the legal denotation. We should have a debate. That time is right now, in this thread. And I welcome it.
The mail came from Ben as the FPM and not as part of an invitation to cloud irc meeting nor as part of a broader programmatic discussion in Cloud WG about the continued long-term evolution of cloud images. That is what irritated me and what I missed.
I see the invigorating effect of an adversarial process. However, I worry that the long-term programmatic aspects will be neglected. And it emphasizes very much the differentiating and the separating. That’s fine in a legal process, but not in a community process.
And part of such a programmatic discussion would also be the exploration of common features of server and cloud, of possible adaptations and synergy effects, which was discussed at the beginning of the year and planned after Dusty's return. All that is gone now as a result of that action, virtually in an "overnight coup d'état". And this was (and is) my argument, not various single technical properties, advantages or disadvantages of BTRFS. And that is what I regret.
On Mon, 2021-05-31 at 15:19 +0200, Peter Boy wrote:
Am 28.05.2021 um 23:08 schrieb Chris Murphy < lists@colorremedies.com>:
……
All I mean by this is to push back on the idea that the proposal for Cloud translates into delaying the decision for Server by 5 or 10 years. Not that Server folks should escalate their discussion.
What I originally meant was the idea to discuss / explore opportunities to coordinate or join the efforts of Server WG and Cloud WG and to better aline both.
Until that happens (and if the Server WG cares about this maybe that should be formally proposed to the Cloud WG?) - it probably does not make sense to expect that Cloud development be held back by anything the Server WG might want to do.
But that idea seems to me got silently out of attention over time. We discussed that an the beginning of March, Server WG picked it up again at the end of March, and there was a brief mention in Cloud WG in early April. Since then there has been silence.
Do you have a reference for that? Would love to follow that discussion
I see the invigorating effect of an adversarial process. However, I worry that the long-term programmatic aspects will be neglected. And it emphasizes very much the differentiating and the separating. That’s fine in a legal process, but not in a community process.
Could you clarify what you mean here? There's a lot of open source communities that develop over time using something similar to the Fedora Change Process. e.g. Python with PEP, Rust with RFC and MCP (Major Change Proposals), even Swift with Evolutions.
And part of such a programmatic discussion would also be the exploration of common features of server and cloud, of possible adaptations and synergy effects, which was discussed at the beginning of the year and planned after Dusty's return. All that is gone now as a result of that action, virtually in an "overnight coup d'état". And this was (and is) my argument, not various single technical properties, advantages or disadvantages of BTRFS. And that is what I regret.
Such a high level conversation would be nice, but again, until that happens (on devel/server/cloud mailing lists), and there's a consensus about how Server and Cloud would work together going forward, how is a Change Proposal targeted at Cloud usage after months of conversation an "overnight coup d'état"?
Best regards,
Am 02.06.2021 um 03:09 schrieb Michel Alexandre Salim via devel devel@lists.fedoraproject.org:
On Mon, 2021-05-31 at 15:19 +0200, Peter Boy wrote:
Am 28.05.2021 um 23:08 schrieb Chris Murphy < lists@colorremedies.com>:
……
All I mean by this is to push back on the idea that the proposal for Cloud translates into delaying the decision for Server by 5 or 10 years. Not that Server folks should escalate their discussion.
What I originally meant was the idea to discuss / explore opportunities to coordinate or join the efforts of Server WG and Cloud WG and to better aline both.
Until that happens (and if the Server WG cares about this maybe that should be formally proposed to the Cloud WG?) - it probably does not make sense to expect that Cloud development be held back by anything the Server WG might want to do.
Indeed, I see it differently. If I seriously want to discuss cooperation and product alignment with another group, then I refrain from making any major changes beforehand, unless an emergency arises. This applies all the more to matters that are known to be controversial and that increase existing differences.
But that idea seems to me got silently out of attention over time. We discussed that an the beginning of March, Server WG picked it up again at the end of March, and there was a brief mention in Cloud WG in early April. Since then there has been silence.
Do you have a reference for that? Would love to follow that discussion
https://meetbot.fedoraproject.org/teams/fedora-server/fedora-server.2021-03-... 17:55:53 <pboyHB> Just as an info about our recent cloud coop discussion Fedora Cloud had had a meeting yesterday .... 17:56:07 <pboyHB> And we can feel honored: As Dusty put it: „the server WG has approached us again about possible collaboration/merging. There seems to be more interest this time.“
https://meetbot.fedoraproject.org/fedora-meeting/2021-04-07/fedora-server.20... 7:45:52 <michel_slm> "so... last week it's apparent the Cloud WG knows we want to reach out. should we start engaging more formally? e.g. start a thread in our mailing lists?“ :-) and the follow up.
And part of such a programmatic discussion would also be the exploration of common features of server and cloud, of possible adaptations and synergy effects, which was discussed at the beginning of the year and planned after Dusty's return. All that is gone now as a result of that action, virtually in an "overnight coup d'état". And this was (and is) my argument, not various single technical properties, advantages or disadvantages of BTRFS. And that is what I regret.
Such a high level conversation would be nice, but again, until that happens (on devel/server/cloud mailing lists), and there's a consensus about how Server and Cloud would work together going forward, how is a Change Proposal targeted at Cloud usage after months of conversation an "overnight coup d'état"?
very nicely put. According to Matthew and others cloud wg did virtually not exist for more than a year, no meetings, silence on mailing list beyond bi-weekly announcement of a „standing“ meeting nobody attended to, there was discussion to „kill“ it, great reluctance on the side of Dusty in our meeting March 3 to make any changes on the artefacts at all, new start of 2-weekly meetings March 30, „months of conversation“ which spell down to 4 meetings in 2 months, and then such a very fundamental change. Well, the wording may be a bit offhand, but by no means farfetched.
And lest there be any misconception, Cloud WG may decide what or about what they deem useful. That is not my business. But regarding the planned discussion about cooperation and alignment of server and Cloud WG and artifacts, I am "not amused" and can't really take it seriously anymore. There are more serious and urgent tasks waiting for us.
On 6/2/21 5:28 AM, Peter Boy wrote:
very nicely put. According to Matthew and others cloud wg did virtually not exist for more than a year, no meetings, silence on mailing list beyond bi-weekly announcement of a „standing“ meeting nobody attended to, there was discussion to „kill“ it, great reluctance on the side of Dusty in our meeting March 3 to make any changes on the artefacts at all, new start of 2-weekly meetings March 30, „months of conversation“ which spell down to 4 meetings in 2 months, and then such a very fundamental change. Well, the wording may be a bit offhand, but by no means farfetched.
We typically run meetings every two weeks, but that did stop happening when I was away earlier this year. There is some truth in here though. It's very true that the cloud WG needs a leader. I'm mostly just trying to keep the lights on. I don't have significant amounts of time to put into it. When people show up with ideas and are willing to execute on them I try not to get in the way assuming the change isn't vastly fundamental.
And lest there be any misconception, Cloud WG may decide what or about what they deem useful. That is not my business. But regarding the planned discussion about cooperation and alignment of server and Cloud WG and artifacts, I am "not amused" and can't really take it seriously anymore. There are more serious and urgent tasks waiting for us.
Bear with me and try to assume good faith. When this was first brought up in the cloud WG meeting recently I did ask of the implications on our pending talks to merge with Server. I thought it had been discussed in the Server WG first.
https://meetbot-raw.fedoraproject.org/teams/fedora_cloud_meeting/fedora_clou... 16:18:15 <dustymabe> There is also the discussion of merging with the server WG - if we change to BTRFS does that inhibit that potential path for us (i.e. should we get server to buy in on the FS change too, or maybe they have already)?
But.. at the end of the day I'm mostly just trying to do the best I can with limited time. Maybe we can get everyone together to discuss all the various options and the best path forward?
Dusty
On Thu, Jun 03, 2021 at 09:35:24AM -0400, Dusty Mabe wrote:
Bear with me and try to assume good faith. When this was first brought up in the cloud WG meeting recently I did ask of the implications on our pending talks to merge with Server. I thought it had been discussed in the Server WG first.
We had a long talk about it several years ago, but that was effectively the previous Server WG not the current one.
But.. at the end of the day I'm mostly just trying to do the best I can with limited time. Maybe we can get everyone together to discuss all the various options and the best path forward?
+1
Am 03.06.2021 um 15:35 schrieb Dusty Mabe dusty@dustymabe.com:
On 6/2/21 5:28 AM, Peter Boy wrote:
very nicely put. According to Matthew and others cloud wg did virtually not exist for more than a year, no meetings, silence on mailing list beyond bi-weekly announcement of a „standing“ meeting nobody attended to, there was discussion to „kill“ it, great reluctance on the side of Dusty in our meeting March 3 to make any changes on the artefacts at all, new start of 2-weekly meetings March 30, „months of conversation“ which spell down to 4 meetings in 2 months, and then such a very fundamental change. Well, the wording may be a bit offhand, but by no means farfetched.
We typically run meetings every two weeks, but that did stop happening when I was away earlier this year. There is some truth in here though.
To clear up any misunderstanding: I am in no way criticizing or disliking the way the Cloud WG works. It is about rebutting the attempt of a subsequent legitimization ("months of discussion") by the facts as they are.
https://meetbot-raw.fedoraproject.org/teams/fedora_cloud_meeting/fedora_clou... 16:18:15 <dustymabe> There is also the discussion of merging with the server WG - if we change to BTRFS does that inhibit that potential path for us (i.e. should we get server to buy in on the FS change too, or maybe they have already)?
I do not want any misunderstanding here either. This passage has nothing to do with you. It is about the fact that all Cloud WG members have known from the very beginning that there are considerations about a cooperation. No one can pretend not to have known (some Cloud members asked me for proof of the existence of this discussion). Everybody knew pretty well what he was doing.
Maybe we can get everyone together to discuss all the various options and the best path forward?
I would still very much welcome and advocate that. However, some Server WG members are already a bit annoyed and feel little inclination to let the server WG eventually get infected, too. In our discussion yesterday, I would have preferred to keep all options open and carefully weigh all arguments and options before making any commitment to one option.
On 5/25/21 5:04 PM, Peter Boy wrote:
So the same model works totally fine for both desktop and server.
I myself lack the exact technical knowledge, but (all?) server and file system experts I hear consider just that a gross misconception.
I think you and Neal talk about two different things. The BTRFS proponents claim that the subvolume technology allows you to have flexible partitioning, so that you can dynamically create whatever partition layout is appropriate for your situation. Nobody is claiming that desktop partition scheme ('model') should be forced onto servers.
Am 25.05.2021 um 23:33 schrieb przemek klosowski via devel devel@lists.fedoraproject.org:
On 5/25/21 5:04 PM, Peter Boy wrote:
So the same model works totally fine for both desktop and server.
I myself lack the exact technical knowledge, but (all?) server and file system experts I hear consider just that a gross misconception.
I think you and Neal talk about two different things. The BTRFS proponents claim that the subvolume technology allows you to have flexible partitioning, so that you can dynamically create whatever partition layout is appropriate for your situation. Nobody is claiming that desktop partition scheme ('model') should be forced onto servers.
Yes, you are right. It’s not about the konkrete partition scheme but the rationale behind it. As a server admin, I'm all about stability, taking precautions in case of errors, recovery, less about flexibility. On my workstation, flexibility is a high priority.
On Tue, May 25, 2021 at 12:04 PM Ben Cotton bcotton@redhat.com wrote:
https://fedoraproject.org/wiki/Changes/FedoraCloudBtrfsByDefault
== Summary ==
For cloud installs of Fedora, we want to provide advanced file system features to users in a transparent fashion. Thus, we are changing the file system for the Cloud Edition to Btrfs so we can leverage its features and capabilities to improve the quality of experience for Cloud users.
== Owners ==
- Name: [[User:Davdunc|David Duncan]], [[User:Chrismurphy|Chris
Murphy]], [[User:Josef|Josef Bacik]], [[User:Salimma|Michel Alexandre Salim]], [[User:Dcavalca|Davide Cavalca]], [[User:Ngompa|Neal Gompa]], [[User:Dustymabe|Dusty Mabe]], [[User:Malmond|Matthew Almond]]
- Email: davdunc@amazon.com, chrismurphy@fedoraproject.org,
josef@toxicpanda.com, michel@michel-slm.name, dcavalca@fb.com, ngompa13@gmail.com, dusty@dustymabe.com, malmond@fb.com
- Products: Fedora Cloud Edition
- Responsible WGs: Fedora Cloud WG
== Detailed Description ==
Fedora Cloud Edition will switch to using Btrfs for its images. The configuration for the Cloud Edition will match the setup used on the desktop variants, as this has been very well-tested with production deployments across multiple Fedora Linux releases now.
This includes the same subvolume layout that is used on the desktop variants [[Changes/BtrfsByDefault|as introduced in Fedora Linux 33]], as well as transparent Zstd compression [[Changes/BtrfsTransparentCompression|as introduced in Fedora Linux 34]].
== Feedback ==
== Benefit to Fedora ==
The benefits are similar to [[Changes/BtrfsByDefault#Benefit_to_Fedora|the ones for Fedora desktop variants]]; however, there are specific benefits for Fedora Cloud:
- Adds support to Fedora Cloud for [[Changes/RPMCoW|the Change to
introduce support for Copy-on-Write enhancements to improve performance to package management]]
- Adds the ability to logically separate contents of the volume
without dividing up the available space ** Transparent compression: significantly reduces write amplification and improves effective I/O throughput ** Reflinks and snapshots improve efficiency for use cases like containers (CRI-O, containerd, and Podman support both)
- Storage devices can be flaky, resulting in data corruption; Btrfs
can help mitigate this ** Everything is checksummed and verified on every read ** Corrupt data results in EIO (input/output error), instead of resulting in application confusion, and isn’t replicated into backups and archives
- Improves system responsiveness under pressure
** Btrfs has been tested in production to have proper IO isolation capability via cgroups2 ** Completes the resource control picture: memory, cpu, IO isolation
- File system resize
** Online shrink and grow are cornerstones of the Btrfs design
- Complex storage setups are… complicated
** Simple and comprehensive command interface. One master command ** Simpler to boot, all code is in the kernel, no initramfs complexities ** Simple and efficient file system replication, including incremental backups, with <code>btrfs send</code> and <code>btrfs receive</code>
== Scope ==
- Proposal owners:
** Submit PRs for Cloud Edition kickstarts to produce disk images using Btrfs.
- Release engineering: [https://pagure.io/releng/issue/10129 #10129]
- Policies and guidelines: N/A
- Trademark approval: N/A
== Upgrade/compatibility impact ==
Change will not affect upgrades.
== How To Test ==
Once the change lands in Rawhide, spin up the images in AWS, GCP, and KVM/OpenStack to test to see systems boot and run.
== User Experience ==
- Mostly transparent.
- Space savings and extend hardware life, via compression.
- Utilities for used and free space are expected to behave the same.
No special commands are required.
- More detailed information can be revealed by <code>btrfs</code>
specific commands.
- <code>cp</code> command will create reflink copies
[https://src.fedoraproject.org/rpms/coreutils/c/5d08d14b/ by default.]
== Dependencies ==
None.
== Contingency Plan ==
- Contingency mechanism: Owner will revert changes back to the
previous ext4 configuration
- Contingency deadline: Beta freeze
- Blocks release? Yes
- Blocks product? Cloud
== Documentation ==
Strictly speaking, no extra documentation is required reading for users.
== Release Notes ==
The default file system on the cloud is now Btrfs, following the desktop change in Fedora Linux 33. Fedora Server, IoT, and CoreOS are still specifically excluded.
Just for the formality, from a kernel standpoint, btrfs is treated like it always has been. The decision as to which default filesystem any edition or spin chooses is entirely up to the SIG or spin maintainer of that edition/spin.
Justin
Bumping this for technical discussion. We are planning to put this in action if there are no technical objections.
On Wed, Jun 9, 2021, 4:13 AM Zbigniew Jędrzejewski-Szmek zbyszek@in.waw.pl wrote:
On Wed, Jun 09, 2021 at 06:52:40AM -0000, David Duncan wrote:
Bumping this for technical discussion. We are planning to put this in
action if there are no technical objections.
Please wait until the FESCo has approved the Change.
I made that sound like an imperative. That was a mistake. I wanted to redirect the thread to technical review instead of program concerns. Of course no actual implementation will be done without approval.
_______________________________________________
devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-leave@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org Do not reply to spam on the list, report it: https://pagure.io/fedora-infrastructure
On Wed, Jun 9, 2021 at 10:19 AM David Duncan davdunc@gmail.com wrote:
On Wed, Jun 9, 2021, 4:13 AM Zbigniew Jędrzejewski-Szmek zbyszek@in.waw.pl wrote:
On Wed, Jun 09, 2021 at 06:52:40AM -0000, David Duncan wrote:
Bumping this for technical discussion. We are planning to put this in action if there are no technical objections.
Please wait until the FESCo has approved the Change.
I made that sound like an imperative. That was a mistake. I wanted to redirect the thread to technical review instead of program concerns. Of course no actual implementation will be done without approval.
So, I suppose I should mention here for discussion what I put in the FESCo ticket. My main concern with this from a cloud image standpoint, is cloud images are run on a number of hosts. Many of those hosts are RHEL or CentOS based. As RHEL does not enable the btrfs filesystem at all, and has no plans to that I am aware of, this means that users on those hosts will no longer be able to mount their images to debug issues or modify in any way. Libguestfs for RHEL is not a workaround at this point, because it doesn't support btrfs on RHEL either. It would also mean that these images could not be used as container images in that environment. Of course the images can still be booted, and will work as expected as long as they are running in a virtualized environment with their own kernel. I am not sure how important this is for people, but it needs to be considered.
Justin
On Wed, Jun 09, 2021 at 10:29:50AM -0500, Justin Forbes wrote:
On Wed, Jun 9, 2021 at 10:19 AM David Duncan davdunc@gmail.com wrote:
On Wed, Jun 9, 2021, 4:13 AM Zbigniew Jędrzejewski-Szmek zbyszek@in.waw.pl wrote:
On Wed, Jun 09, 2021 at 06:52:40AM -0000, David Duncan wrote:
Bumping this for technical discussion. We are planning to put this in action if there are no technical objections.
Please wait until the FESCo has approved the Change.
I made that sound like an imperative. That was a mistake. I wanted to redirect the thread to technical review instead of program concerns. Of course no actual implementation will be done without approval.
So, I suppose I should mention here for discussion what I put in the FESCo ticket. My main concern with this from a cloud image standpoint, is cloud images are run on a number of hosts. Many of those hosts are RHEL or CentOS based. As RHEL does not enable the btrfs filesystem at all, and has no plans to that I am aware of, this means that users on those hosts will no longer be able to mount their images to debug issues or modify in any way. Libguestfs for RHEL is not a workaround at this point, because it doesn't support btrfs on RHEL either. It would also mean that these images could not be used as container images in that environment. Of course the images can still be booted, and will work as expected as long as they are running in a virtualized environment with their own kernel. I am not sure how important this is for people, but it needs to be considered.
Yeah, I think we have to accept that there won't be any kernel support in RHEL in a timeframe that matters for Fedora, and the RHEL host will not be able to mount the images natively. So the questions for me are: 1. I this a problem in practice? I.e. how often do people need to use Fedora images for containers on RHEL hosts? 2. What workaround can we put in place? Something in EPEL or dkms module with btrfs in a copr repo?
Zbyszek
On Wed, 2021-06-09 at 19:53 +0000, Zbigniew Jędrzejewski-Szmek wrote:
Yeah, I think we have to accept that there won't be any kernel support in RHEL in a timeframe that matters for Fedora, and the RHEL host will not be able to mount the images natively. So the questions for me are:
- I this a problem in practice? I.e. how often do people need to use
Fedora images for containers on RHEL hosts? 2. What workaround can we put in place? Something in EPEL or dkms module with btrfs in a copr repo?
I've put together a Fedora-based libguestfs container that should address some of these issues and allow accessing and manipulating btrfs filesystems even on RHEL hosts that do not support it. It's currently in review at https://bugzilla.redhat.com/show_bug.cgi?id=1970071 and I'd appreciate any feedback you might have there. In addition to this, we're exploring the possiblity of developing a userspace btrfs-fuse implementation by leveraging the existing logic in brtfsprogs, which could also provide an alternative access option.
On the CentOS side, the Hyperscale SIG is working on a btrfs-enabled kernel for CentOS Stream: https://pagure.io/centos-sig-hyperscale/sig/issue/7 . There's also a kmods SIG currently being proposed (https://wiki.centos.org/SpecialInterestGroup/Kmods) that could be a place for distributing a btrfs module for RHEL / CentOS stock kernels for the time being.
Cheers Davide
On 6/9/21 15:53, Zbigniew Jędrzejewski-Szmek wrote:
On Wed, Jun 09, 2021 at 10:29:50AM -0500, Justin Forbes wrote:
On Wed, Jun 9, 2021 at 10:19 AM David Duncan davdunc@gmail.com wrote:
On Wed, Jun 9, 2021, 4:13 AM Zbigniew Jędrzejewski-Szmek zbyszek@in.waw.pl wrote:
On Wed, Jun 09, 2021 at 06:52:40AM -0000, David Duncan wrote:
Bumping this for technical discussion. We are planning to put this in action if there are no technical objections.
Please wait until the FESCo has approved the Change.
I made that sound like an imperative. That was a mistake. I wanted to redirect the thread to technical review instead of program concerns. Of course no actual implementation will be done without approval.
So, I suppose I should mention here for discussion what I put in the FESCo ticket. My main concern with this from a cloud image standpoint, is cloud images are run on a number of hosts. Many of those hosts are RHEL or CentOS based. As RHEL does not enable the btrfs filesystem at all, and has no plans to that I am aware of, this means that users on those hosts will no longer be able to mount their images to debug issues or modify in any way. Libguestfs for RHEL is not a workaround at this point, because it doesn't support btrfs on RHEL either. It would also mean that these images could not be used as container images in that environment. Of course the images can still be booted, and will work as expected as long as they are running in a virtualized environment with their own kernel. I am not sure how important this is for people, but it needs to be considered.
Yeah, I think we have to accept that there won't be any kernel support in RHEL in a timeframe that matters for Fedora, and the RHEL host will not be able to mount the images natively. So the questions for me are:
- I this a problem in practice? I.e. how often do people need to use Fedora images for containers on RHEL hosts?
- What workaround can we put in place? Something in EPEL or dkms module with btrfs in a copr repo?
Zbyszek
Container images have nothing to do with this. Container images are stored as tar balls and are constructed on the underlying OS. This is purely an issue with soemthing like libguestfs
devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-leave@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org Do not reply to spam on the list, report it: https://pagure.io/fedora-infrastructure
On Wed, Jun 09, 2021 at 07:53:04PM +0000, Zbigniew Jędrzejewski-Szmek wrote:
On Wed, Jun 09, 2021 at 10:29:50AM -0500, Justin Forbes wrote:
On Wed, Jun 9, 2021 at 10:19 AM David Duncan davdunc@gmail.com wrote:
On Wed, Jun 9, 2021, 4:13 AM Zbigniew Jędrzejewski-Szmek zbyszek@in.waw.pl wrote:
On Wed, Jun 09, 2021 at 06:52:40AM -0000, David Duncan wrote:
Bumping this for technical discussion. We are planning to put this in action if there are no technical objections.
Please wait until the FESCo has approved the Change.
I made that sound like an imperative. That was a mistake. I wanted to redirect the thread to technical review instead of program concerns. Of course no actual implementation will be done without approval.
So, I suppose I should mention here for discussion what I put in the FESCo ticket. My main concern with this from a cloud image standpoint, is cloud images are run on a number of hosts. Many of those hosts are RHEL or CentOS based. As RHEL does not enable the btrfs filesystem at all, and has no plans to that I am aware of, this means that users on those hosts will no longer be able to mount their images to debug issues or modify in any way. Libguestfs for RHEL is not a workaround at this point, because it doesn't support btrfs on RHEL either. It would also mean that these images could not be used as container images in that environment. Of course the images can still be booted, and will work as expected as long as they are running in a virtualized environment with their own kernel. I am not sure how important this is for people, but it needs to be considered.
Yeah, I think we have to accept that there won't be any kernel support in RHEL in a timeframe that matters for Fedora, and the RHEL host will not be able to mount the images natively. So the questions for me are:
- I this a problem in practice? I.e. how often do people need to use Fedora images for containers on RHEL hosts?
The case I am wondering about is openstack and other private cloud's. I know in the past when we ran an openstack cloud, it could resize images to the flavor you picked and it did so outside of cloud-init, it was actual openstack putting the image on a backing store and resizing it. Perhaps it doesn't do that anymore, or perhaps it would work with btfs, but it would be nice to know. :(
kevin