Hi,
This is in regards to this Fedora 34 change: https://fedoraproject.org/wiki/Changes/BtrfsTransparentCompression#Scope
The gist is to do 'mount -o compress=zstd:1' anytime Btrfs is used, whether Destination Installation>Automatic or Custom or Advanced-Custom. Apply it during the installation, and add it to /etc/fstab.
Somehow I got confused thinking that autopart supports --fsoptions, and that would be the way to do this. But (a) --fsoptions isn't supported with autopart, and (b) it wouldn't be a universal approach anyway. And we want this to be consistent. Now I'm thinking it needs to go somewhere in:
https://github.com/storaged-project/blivet/blob/3.4-devel/blivet/devices/btr...
Am I on the right track, or does it need to go somewhere else, or in addition to that?
Thanks,
Hi,
On Mon, 2021-02-08 at 21:02 -0700, Chris Murphy wrote:
Hi,
This is in regards to this Fedora 34 change: https://fedoraproject.org/wiki/Changes/BtrfsTransparentCompression#Scope
The gist is to do 'mount -o compress=zstd:1' anytime Btrfs is used, whether Destination Installation>Automatic or Custom or Advanced-Custom. Apply it during the installation, and add it to /etc/fstab.
Somehow I got confused thinking that autopart supports --fsoptions, and that would be the way to do this. But (a) --fsoptions isn't supported with autopart, and (b) it wouldn't be a universal approach anyway. And we want this to be consistent. Now I'm thinking it needs to go somewhere in:
https://github.com/storaged-project/blivet/blob/3.4-devel/blivet/devices/btr...
Am I on the right track, or does it need to go somewhere else, or in addition to that?
(I'm one of the change owners, and trying to figure this out with Chris).
Ideally we have a solution that is configurable - i.e. this is exposed via a kickstart command (and probably entailing changes in Anaconda and pykickstart), but if that is not possible, or require a lot of rework (which we can try to work on), if we are willing to carry a patch for blivet or some other component to override the behavior just for Fedora 34, what's the best way of achieving that?
Btrfs is probably the only filesystem with built-in compression that we potentially care about, so designing an interface to expose this functionality might be an overkill -- but we're not sure.
Thanks,
On Tue, Feb 9, 2021 at 1:09 AM Michel Alexandre Salim michel@michel-slm.name wrote:
Hi,
On Mon, 2021-02-08 at 21:02 -0700, Chris Murphy wrote:
Hi,
This is in regards to this Fedora 34 change: https://fedoraproject.org/wiki/Changes/BtrfsTransparentCompression#Scope
The gist is to do 'mount -o compress=zstd:1' anytime Btrfs is used, whether Destination Installation>Automatic or Custom or Advanced-Custom. Apply it during the installation, and add it to /etc/fstab.
Somehow I got confused thinking that autopart supports --fsoptions, and that would be the way to do this. But (a) --fsoptions isn't supported with autopart, and (b) it wouldn't be a universal approach anyway. And we want this to be consistent. Now I'm thinking it needs to go somewhere in:
https://github.com/storaged-project/blivet/blob/3.4-devel/blivet/devices/btr...
Am I on the right track, or does it need to go somewhere else, or in addition to that?
(I'm one of the change owners, and trying to figure this out with Chris).
Ideally we have a solution that is configurable - i.e. this is exposed via a kickstart command (and probably entailing changes in Anaconda and pykickstart), but if that is not possible, or require a lot of rework (which we can try to work on), if we are willing to carry a patch for blivet or some other component to override the behavior just for Fedora 34, what's the best way of achieving that?
Btrfs is probably the only filesystem with built-in compression that we potentially care about, so designing an interface to expose this functionality might be an overkill -- but we're not sure.
Eventually, VDO might get integrated into the mainline tree, so having the interface which could be used for LVM compression through VDO wouldn't be too bad.
Hi everyone,
I see a few options for this. First is to add this directly to blivet library as you pointed out already. However, I don't think blivet developers would be happy about that because they are trying to be as much as possible general purpose library and this change is really just about Anaconda.
Another option seems to be to modify Anaconda code directly. Unfortunately, I can't tell from top of my mind how hard that would be but still seems like the easiest solution.
However, the best approach which I can think of is to add something like that to our configuration files. Benefit would be that another modifications like that could be done easily in the future. That is of course for a discussion for the Anaconda team because it could be pretty hard to implemented this in some common form.
Vendy, Vojta, do you have some better ideas or could you tell us something more about my ideas above?
Best Regards, Jirka
On Tue, 2021-02-09 at 01:18 -0500, Neal Gompa wrote:
On Tue, Feb 9, 2021 at 1:09 AM Michel Alexandre Salim michel@michel-slm.name wrote:
Hi,
On Mon, 2021-02-08 at 21:02 -0700, Chris Murphy wrote:
Hi,
This is in regards to this Fedora 34 change: https://fedoraproject.org/wiki/Changes/BtrfsTransparentCompression#Scope
The gist is to do 'mount -o compress=zstd:1' anytime Btrfs is used, whether Destination Installation>Automatic or Custom or Advanced-Custom. Apply it during the installation, and add it to /etc/fstab.
Somehow I got confused thinking that autopart supports -- fsoptions, and that would be the way to do this. But (a) --fsoptions isn't supported with autopart, and (b) it wouldn't be a universal approach anyway. And we want this to be consistent. Now I'm thinking it needs to go somewhere in:
https://github.com/storaged-project/blivet/blob/3.4-devel/blivet/devices/btr...
Am I on the right track, or does it need to go somewhere else, or in addition to that?
(I'm one of the change owners, and trying to figure this out with Chris).
Ideally we have a solution that is configurable - i.e. this is exposed via a kickstart command (and probably entailing changes in Anaconda and pykickstart), but if that is not possible, or require a lot of rework (which we can try to work on), if we are willing to carry a patch for blivet or some other component to override the behavior just for Fedora 34, what's the best way of achieving that?
Btrfs is probably the only filesystem with built-in compression that we potentially care about, so designing an interface to expose this functionality might be an overkill -- but we're not sure.
Eventually, VDO might get integrated into the mainline tree, so having the interface which could be used for LVM compression through VDO wouldn't be too bad.
On Tue, Feb 9, 2021 at 10:42 AM jkonecny@redhat.com wrote:
Hi everyone,
I see a few options for this. First is to add this directly to blivet library as you pointed out already. However, I don't think blivet developers would be happy about that because they are trying to be as much as possible general purpose library and this change is really just about Anaconda.
Another option seems to be to modify Anaconda code directly. Unfortunately, I can't tell from top of my mind how hard that would be but still seems like the easiest solution.
However, the best approach which I can think of is to add something like that to our configuration files. Benefit would be that another modifications like that could be done easily in the future. That is of course for a discussion for the Anaconda team because it could be pretty hard to implemented this in some common form.
Vendy, Vojta, do you have some better ideas or could you tell us something more about my ideas above?
Best Regards, Jirka
On Tue, 2021-02-09 at 01:18 -0500, Neal Gompa wrote:
On Tue, Feb 9, 2021 at 1:09 AM Michel Alexandre Salim michel@michel-slm.name wrote:
Hi,
On Mon, 2021-02-08 at 21:02 -0700, Chris Murphy wrote:
Hi,
This is in regards to this Fedora 34 change: https://fedoraproject.org/wiki/Changes/BtrfsTransparentCompression#Scope
The gist is to do 'mount -o compress=zstd:1' anytime Btrfs is used, whether Destination Installation>Automatic or Custom or Advanced-Custom. Apply it during the installation, and add it to /etc/fstab.
Somehow I got confused thinking that autopart supports -- fsoptions, and that would be the way to do this. But (a) --fsoptions isn't supported with autopart, and (b) it wouldn't be a universal approach anyway. And we want this to be consistent. Now I'm thinking it needs to go somewhere in:
https://github.com/storaged-project/blivet/blob/3.4-devel/blivet/devices/btr...
Am I on the right track, or does it need to go somewhere else, or in addition to that?
(I'm one of the change owners, and trying to figure this out with Chris).
Ideally we have a solution that is configurable - i.e. this is exposed via a kickstart command (and probably entailing changes in Anaconda and pykickstart), but if that is not possible, or require a lot of rework (which we can try to work on), if we are willing to carry a patch for blivet or some other component to override the behavior just for Fedora 34, what's the best way of achieving that?
Btrfs is probably the only filesystem with built-in compression that we potentially care about, so designing an interface to expose this functionality might be an overkill -- but we're not sure.
Eventually, VDO might get integrated into the mainline tree, so having the interface which could be used for LVM compression through VDO wouldn't be too bad.
Hello!
The change says: "On variants using btrfs as the default filesystem, enable transparent compression using zstd." It means that the compression has to be configurable per product, so we need to add a new configuration option to the Anaconda configuration files (something like btrfs_compression_enabled = True). I guess you planned to use interactive-defaults.ks, but we are going to replace this file with the configuration files anyway.
I have checked the Blivet's code and we might be able to redefine the _mount_class attribute of the BTRFS class. The mount class defines default mount options. That should be enough to fix all partitioning methods. What do you think, Vojta?
Vendy
Hi Jiří, Vendula, all,
On Tue, 2021-02-09 at 12:12 +0100, Vendula Poncova wrote:
On Tue, Feb 9, 2021 at 10:42 AM jkonecny@redhat.com wrote:
Hi everyone,
I see a few options for this. First is to add this directly to blivet library as you pointed out already. However, I don't think blivet developers would be happy about that because they are trying to be as much as possible general purpose library and this change is really just about Anaconda.
Another option seems to be to modify Anaconda code directly. Unfortunately, I can't tell from top of my mind how hard that would be but still seems like the easiest solution.
However, the best approach which I can think of is to add something like that to our configuration files. Benefit would be that another modifications like that could be done easily in the future. That is of course for a discussion for the Anaconda team because it could be pretty hard to implemented this in some common form.
Vendy, Vojta, do you have some better ideas or could you tell us something more about my ideas above?
Best Regards, Jirka
On Tue, 2021-02-09 at 01:18 -0500, Neal Gompa wrote:
On Tue, Feb 9, 2021 at 1:09 AM Michel Alexandre Salim michel@michel-slm.name wrote:
Hi,
On Mon, 2021-02-08 at 21:02 -0700, Chris Murphy wrote:
Hi,
This is in regards to this Fedora 34 change: https://fedoraproject.org/wiki/Changes/BtrfsTransparentCompression#Scope
The gist is to do 'mount -o compress=zstd:1' anytime Btrfs is used, whether Destination Installation>Automatic or Custom or Advanced-Custom. Apply it during the installation, and add it to /etc/fstab.
Somehow I got confused thinking that autopart supports -- fsoptions, and that would be the way to do this. But (a) --fsoptions isn't supported with autopart, and (b) it wouldn't be a universal approach anyway. And we want this to be consistent. Now I'm thinking it needs to go somewhere in:
https://github.com/storaged-project/blivet/blob/3.4-devel/blivet/devices/btr...
Am I on the right track, or does it need to go somewhere else, or in addition to that?
(I'm one of the change owners, and trying to figure this out with Chris).
Ideally we have a solution that is configurable - i.e. this is exposed via a kickstart command (and probably entailing changes in Anaconda and pykickstart), but if that is not possible, or require a lot of rework (which we can try to work on), if we are willing to carry a patch for blivet or some other component to override the behavior just for Fedora 34, what's the best way of achieving that?
Btrfs is probably the only filesystem with built-in compression that we potentially care about, so designing an interface to expose this functionality might be an overkill -- but we're not sure.
Eventually, VDO might get integrated into the mainline tree, so having the interface which could be used for LVM compression through VDO wouldn't be too bad.
Hello!
The change says: "On variants using btrfs as the default filesystem, enable transparent compression using zstd." It means that the compression has to be configurable per product, so we need to add a new configuration option to the Anaconda configuration files (something like btrfs_compression_enabled = True). I guess you planned to use interactive-defaults.ks, but we are going to replace this file with the configuration files anyway.
I have checked the Blivet's code and we might be able to redefine the _mount_class attribute of the BTRFS class. The mount class defines default mount options. That should be enough to fix all partitioning methods. What do you think, Vojta?
Vendy
Anaconda-devel-list mailing list Anaconda-devel-list@redhat.com https://www.redhat.com/mailman/listinfo/anaconda-devel-list
I took an initial stab at this -- would this be the way to achieve this by overriding `_mount_class` in Blivet only?
https://github.com/storaged-project/blivet/pull/930 (for 3.4-devel) https://github.com/storaged-project/blivet/pull/931 (for 3.3-devel)
Presumably this might be too specific to get merged upstream, but could be something we can apply as patches against Fedora's python-blivet RPM while we work out a more generic solution.
Vendula: is the more generic configuration mechanism supposed to work like this?
- specify in an Anaconda config file that we want to override some options (which file would this be?) - if this flag is set, when Anaconda tells Blivet it wants a Btrfs filesystem, override `_mount_class` (so the code for the new class can live in Anaconda but subclasses a Blivet class)
There's a further complication: Chris just informed me that on BIOS systems, space is more of a constraint so the zstd module is not part of GRUB ... meaning we have to handle these two scenarios:
- if /boot is a directory on a / btrfs file system, we need to override the compression property for it before any files get written there, *or* defragment the files in %post - if /boot is a separate btrfs file system, we need to make sure we don't compress it so we might need to either use a separate BTRFSBootMount class, or use the standard FSMount, but also somehow prevent BTRFSMount to be used if the mount point is /boot
Oh what a tangled web we weave... thoughts on all this? And thanks for the useful pointers,
On Tue, Feb 9, 2021 at 9:15 AM Michel Alexandre Salim michel@michel-slm.name wrote:
There's a further complication: Chris just informed me that on BIOS systems, space is more of a constraint so the zstd module is not part of GRUB ... meaning we have to handle these two scenarios:
It's available, but it's the 2nd largest GRUB module at ~100KiB. And on BIOS we're space limit right now to 1 MiB MBR gap or BIOS Boot partition.
GRUB works fine with a zstd compressed /boot/ but obviously there's next to no benefit compared to the added complexity since the kernel and initramfs are already compressed.
So if it's not difficult to exclude compression on /boot/ that's probably preferred?
On Tue, 2021-02-09 at 09:50 -0700, Chris Murphy wrote:
On Tue, Feb 9, 2021 at 9:15 AM Michel Alexandre Salim michel@michel-slm.name wrote:
There's a further complication: Chris just informed me that on BIOS systems, space is more of a constraint so the zstd module is not part of GRUB ... meaning we have to handle these two scenarios:
It's available, but it's the 2nd largest GRUB module at ~100KiB. And on BIOS we're space limit right now to 1 MiB MBR gap or BIOS Boot partition.
GRUB works fine with a zstd compressed /boot/ but obviously there's next to no benefit compared to the added complexity since the kernel and initramfs are already compressed.
So if it's not difficult to exclude compression on /boot/ that's probably preferred?
Are we talking about custom partitioning layouts ?
Because I've just booted a current Rawhide installer and it creates /boot on a separate standard partition formatted with EXT4 by default, with / and /home on BTRFS.
On Tue, 2021-02-09 at 19:40 +0100, Martin Kolman wrote:
On Tue, 2021-02-09 at 09:50 -0700, Chris Murphy wrote:
On Tue, Feb 9, 2021 at 9:15 AM Michel Alexandre Salim michel@michel-slm.name wrote:
There's a further complication: Chris just informed me that on BIOS systems, space is more of a constraint so the zstd module is not part of GRUB ... meaning we have to handle these two scenarios:
It's available, but it's the 2nd largest GRUB module at ~100KiB. And on BIOS we're space limit right now to 1 MiB MBR gap or BIOS Boot partition.
GRUB works fine with a zstd compressed /boot/ but obviously there's next to no benefit compared to the added complexity since the kernel and initramfs are already compressed.
So if it's not difficult to exclude compression on /boot/ that's probably preferred?
Are we talking about custom partitioning layouts ?
Because I've just booted a current Rawhide installer and it creates /boot on a separate standard partition formatted with EXT4 by default, with / and /home on BTRFS.
Yup, the "being able to keep /boot uncompressed" is mostly to make sure people who do custom layouts (and are using BIOS instead of UEFI) don't get in a bad state.
Thanks,
On Tue, Feb 9, 2021 at 11:40 AM Martin Kolman mkolman@redhat.com wrote:
On Tue, 2021-02-09 at 09:50 -0700, Chris Murphy wrote:
On Tue, Feb 9, 2021 at 9:15 AM Michel Alexandre Salim michel@michel-slm.name wrote:
There's a further complication: Chris just informed me that on BIOS systems, space is more of a constraint so the zstd module is not part of GRUB ... meaning we have to handle these two scenarios:
It's available, but it's the 2nd largest GRUB module at ~100KiB. And on BIOS we're space limit right now to 1 MiB MBR gap or BIOS Boot partition.
GRUB works fine with a zstd compressed /boot/ but obviously there's next to no benefit compared to the added complexity since the kernel and initramfs are already compressed.
So if it's not difficult to exclude compression on /boot/ that's probably preferred?
Are we talking about custom partitioning layouts ?
For Fedora 34, yes. There's an idea/pre-plan for Fedora 35/36 to put /boot/ on Btrfs by default; but it's contingent on GRUB changes needed upstream to better support grubenv by putting it on a dedicated partition. [1]
Because I've just booted a current Rawhide installer and it creates /boot on a separate standard partition formatted with EXT4 by default, with / and /home on BTRFS.
Yes.
On Tue, Feb 9, 2021 at 12:26 PM Chris Murphy lists@colorremedies.com wrote:
On Tue, Feb 9, 2021 at 11:40 AM Martin Kolman mkolman@redhat.com wrote:
On Tue, 2021-02-09 at 09:50 -0700, Chris Murphy wrote:
On Tue, Feb 9, 2021 at 9:15 AM Michel Alexandre Salim michel@michel-slm.name wrote:
There's a further complication: Chris just informed me that on BIOS systems, space is more of a constraint so the zstd module is not part of GRUB ... meaning we have to handle these two scenarios:
It's available, but it's the 2nd largest GRUB module at ~100KiB. And on BIOS we're space limit right now to 1 MiB MBR gap or BIOS Boot partition.
GRUB works fine with a zstd compressed /boot/ but obviously there's next to no benefit compared to the added complexity since the kernel and initramfs are already compressed.
So if it's not difficult to exclude compression on /boot/ that's probably preferred?
Are we talking about custom partitioning layouts ?
For Fedora 34, yes. There's an idea/pre-plan for Fedora 35/36 to put /boot/ on Btrfs by default; but it's contingent on GRUB changes needed upstream to better support grubenv by putting it on a dedicated partition. [1]
:D
[1] GRUB disallows writes to it from the pre-boot environment when grubenv is on mdadm, luks, btrfs, zfs; while GRUB allows writes to grubenv when on ext[234] and XFS the fs developers really frown on anything other than kernel code writing to a file system owned partition, even if that write is just a data block overwrite. So the idea is to stop doing it and give grubenv it's own partition sorta like BIOS Boot.
Hi Jiří, Vendula, all,
On Tue, 2021-02-09 at 12:12 +0100, Vendula Poncova wrote:
On Tue, Feb 9, 2021 at 10:42 AM jkonecny@redhat.com wrote:
Hi everyone,
I see a few options for this. First is to add this directly to blivet library as you pointed out already. However, I don't think blivet developers would be happy about that because they are trying to be as much as possible general purpose library and this change is really just about Anaconda.
Another option seems to be to modify Anaconda code directly. Unfortunately, I can't tell from top of my mind how hard that would be but still seems like the easiest solution.
However, the best approach which I can think of is to add something like that to our configuration files. Benefit would be that another modifications like that could be done easily in the future. That is of course for a discussion for the Anaconda team because it could be pretty hard to implemented this in some common form.
Vendy, Vojta, do you have some better ideas or could you tell us something more about my ideas above?
Best Regards, Jirka
On Tue, 2021-02-09 at 01:18 -0500, Neal Gompa wrote:
On Tue, Feb 9, 2021 at 1:09 AM Michel Alexandre Salim michel@michel-slm.name wrote:
Hi,
On Mon, 2021-02-08 at 21:02 -0700, Chris Murphy wrote:
Hi,
This is in regards to this Fedora 34 change: https://fedoraproject.org/wiki/Changes/BtrfsTransparentCompression#Scope
The gist is to do 'mount -o compress=zstd:1' anytime Btrfs is used, whether Destination Installation>Automatic or Custom or Advanced-Custom. Apply it during the installation, and add it to /etc/fstab.
Somehow I got confused thinking that autopart supports -- fsoptions, and that would be the way to do this. But (a) --fsoptions isn't supported with autopart, and (b) it wouldn't be a universal approach anyway. And we want this to be consistent. Now I'm thinking it needs to go somewhere in:
https://github.com/storaged-project/blivet/blob/3.4-devel/blivet/devices/btr...
Am I on the right track, or does it need to go somewhere else, or in addition to that?
(I'm one of the change owners, and trying to figure this out with Chris).
Ideally we have a solution that is configurable - i.e. this is exposed via a kickstart command (and probably entailing changes in Anaconda and pykickstart), but if that is not possible, or require a lot of rework (which we can try to work on), if we are willing to carry a patch for blivet or some other component to override the behavior just for Fedora 34, what's the best way of achieving that?
Btrfs is probably the only filesystem with built-in compression that we potentially care about, so designing an interface to expose this functionality might be an overkill -- but we're not sure.
Eventually, VDO might get integrated into the mainline tree, so having the interface which could be used for LVM compression through VDO wouldn't be too bad.
Hello!
The change says: "On variants using btrfs as the default filesystem, enable transparent compression using zstd." It means that the compression has to be configurable per product, so we need to add a new configuration option to the Anaconda configuration files (something like btrfs_compression_enabled = True). I guess you planned to use interactive-defaults.ks, but we are going to replace this file with the configuration files anyway.
I have checked the Blivet's code and we might be able to redefine the _mount_class attribute of the BTRFS class. The mount class defines default mount options. That should be enough to fix all partitioning methods. What do you think, Vojta?
I'm trying to figure this out, and am currently a bit stuck:
- patching devicetree/fsset.py's `fstab` to ensure fstab contains compress=zstd:1 works, but the change only affects fstab on the installed system, so during installation compression is not enabled - patching devicetree/handler.py's `mount_options` to inject compress=zstd:1 does not seem to do anything https://github.com/rhinstaller/anaconda/compare/master...michel-slm:f34-34.2...
- patching devicetree/handler.py's `get_device_mount_options` to ensure btrfs filesystems area always mounted with compress=zstd:1 does not work as expected (patching `set_device_mount_options` does not do anything either): https://github.com/rhinstaller/anaconda/compare/master...michel-slm:f34-34.2...
Since I don't really want to hardcode this long-term (hardcoding in a proof-of-concept, or for the F34 release only, is not too bad) -- is Blivet the place to do this? What I had in mind is:
- Some product definition in data/product.d specifies compression should be enabled - When that is enabled, Anaconda overrides the _mount_class of the BTRFS class it gets from Blivet
^ where in Anaconda's codebase can I do this? Trying to follow this in the code but it's not easy especially since there does not seem to be an example of another filesystem conditionally overriding the default mount options.
I'll try and be on IRC early enough to get hold of one of the developers tomorrow. The timezone difference is not making this easy!
Best regards,
On Tue, Feb 16, 2021 at 3:16 AM Michel Alexandre Salim michel@michel-slm.name wrote:
Hi Jiří, Vendula, all,
On Tue, 2021-02-09 at 12:12 +0100, Vendula Poncova wrote:
On Tue, Feb 9, 2021 at 10:42 AM jkonecny@redhat.com wrote:
Hi everyone,
I see a few options for this. First is to add this directly to blivet library as you pointed out already. However, I don't think blivet developers would be happy about that because they are trying to be as much as possible general purpose library and this change is really just about Anaconda.
Another option seems to be to modify Anaconda code directly. Unfortunately, I can't tell from top of my mind how hard that would be but still seems like the easiest solution.
However, the best approach which I can think of is to add something like that to our configuration files. Benefit would be that another modifications like that could be done easily in the future. That is of course for a discussion for the Anaconda team because it could be pretty hard to implemented this in some common form.
Vendy, Vojta, do you have some better ideas or could you tell us something more about my ideas above?
Best Regards, Jirka
On Tue, 2021-02-09 at 01:18 -0500, Neal Gompa wrote:
On Tue, Feb 9, 2021 at 1:09 AM Michel Alexandre Salim michel@michel-slm.name wrote:
Hi,
On Mon, 2021-02-08 at 21:02 -0700, Chris Murphy wrote:
Hi,
This is in regards to this Fedora 34 change: https://fedoraproject.org/wiki/Changes/BtrfsTransparentCompression#Scope
The gist is to do 'mount -o compress=zstd:1' anytime Btrfs is used, whether Destination Installation>Automatic or Custom or Advanced-Custom. Apply it during the installation, and add it to /etc/fstab.
Somehow I got confused thinking that autopart supports -- fsoptions, and that would be the way to do this. But (a) --fsoptions isn't supported with autopart, and (b) it wouldn't be a universal approach anyway. And we want this to be consistent. Now I'm thinking it needs to go somewhere in:
https://github.com/storaged-project/blivet/blob/3.4-devel/blivet/devices/btr...
Am I on the right track, or does it need to go somewhere else, or in addition to that?
(I'm one of the change owners, and trying to figure this out with Chris).
Ideally we have a solution that is configurable - i.e. this is exposed via a kickstart command (and probably entailing changes in Anaconda and pykickstart), but if that is not possible, or require a lot of rework (which we can try to work on), if we are willing to carry a patch for blivet or some other component to override the behavior just for Fedora 34, what's the best way of achieving that?
Btrfs is probably the only filesystem with built-in compression that we potentially care about, so designing an interface to expose this functionality might be an overkill -- but we're not sure.
Eventually, VDO might get integrated into the mainline tree, so having the interface which could be used for LVM compression through VDO wouldn't be too bad.
Hello!
The change says: "On variants using btrfs as the default filesystem, enable transparent compression using zstd." It means that the compression has to be configurable per product, so we need to add a new configuration option to the Anaconda configuration files (something like btrfs_compression_enabled = True). I guess you planned to use interactive-defaults.ks, but we are going to replace this file with the configuration files anyway.
I have checked the Blivet's code and we might be able to redefine the _mount_class attribute of the BTRFS class. The mount class defines default mount options. That should be enough to fix all partitioning methods. What do you think, Vojta?
I'm trying to figure this out, and am currently a bit stuck:
- patching devicetree/fsset.py's `fstab` to ensure fstab contains
compress=zstd:1 works, but the change only affects fstab on the installed system, so during installation compression is not enabled
- patching devicetree/handler.py's `mount_options` to inject
compress=zstd:1 does not seem to do anything https://github.com/rhinstaller/anaconda/compare/master...michel-slm:f34-34.2...
- patching devicetree/handler.py's `get_device_mount_options` to ensure
btrfs filesystems area always mounted with compress=zstd:1 does not work as expected (patching `set_device_mount_options` does not do anything either): https://github.com/rhinstaller/anaconda/compare/master...michel-slm:f34-34.2...
Since I don't really want to hardcode this long-term (hardcoding in a proof-of-concept, or for the F34 release only, is not too bad) -- is Blivet the place to do this? What I had in mind is:
- Some product definition in data/product.d specifies compression
should be enabled
- When that is enabled, Anaconda overrides the _mount_class of the
BTRFS class it gets from Blivet
^ where in Anaconda's codebase can I do this? Trying to follow this in the code but it's not easy especially since there does not seem to be an example of another filesystem conditionally overriding the default mount options.
Hello,
I have talked to Vojta and we suggest the following solution:
1. Open a pull request with [0] in the upstream. 2. Add a new flag to blivet.flags for enabling the compression of btrfs. It should be False by default. Modify [0] to respect the flag. 3. Open a pull request that sets up the new flag in Anaconda. Anaconda sets the Blivet's flags in [1]. 4. If you want, add a new option to the [Storage] section of the Anaconda configuration file that will control the new flag. It should be False by default. Set this option in the relevant product configuration files located at [2].
[0] https://src.fedoraproject.org/rpms/python-blivet/pull-request/8 [1] https://github.com/rhinstaller/anaconda/blob/master/pyanaconda/modules/stora... [2] https://github.com/rhinstaller/anaconda/tree/master/data/product.d
Vendy
I'll try and be on IRC early enough to get hold of one of the developers tomorrow. The timezone difference is not making this easy!
Best regards,
-- Michel Alexandre Salim profile: https://keyoxide.org/michel@michel-slm.name chat via email: https://delta.chat/ GPG key: 5DCE 2E7E 9C3B 1CFF D335 C1D7 8B22 9D2F 7CCC 04F2 _______________________________________________ Anaconda-devel-list mailing list Anaconda-devel-list@redhat.com https://www.redhat.com/mailman/listinfo/anaconda-devel-list
On Tue, 2021-02-16 at 15:05 +0100, Vendula Poncova wrote:
On Tue, Feb 16, 2021 at 3:16 AM Michel Alexandre Salim michel@michel-slm.name wrote:
Hi Jiří, Vendula, all,
On Tue, 2021-02-09 at 12:12 +0100, Vendula Poncova wrote:
On Tue, Feb 9, 2021 at 10:42 AM jkonecny@redhat.com wrote:
Hi everyone,
I see a few options for this. First is to add this directly to blivet library as you pointed out already. However, I don't think blivet developers would be happy about that because they are trying to be as much as possible general purpose library and this change is really just about Anaconda.
Another option seems to be to modify Anaconda code directly. Unfortunately, I can't tell from top of my mind how hard that would be but still seems like the easiest solution.
However, the best approach which I can think of is to add something like that to our configuration files. Benefit would be that another modifications like that could be done easily in the future. That is of course for a discussion for the Anaconda team because it could be pretty hard to implemented this in some common form.
Vendy, Vojta, do you have some better ideas or could you tell us something more about my ideas above?
Best Regards, Jirka
On Tue, 2021-02-09 at 01:18 -0500, Neal Gompa wrote:
On Tue, Feb 9, 2021 at 1:09 AM Michel Alexandre Salim michel@michel-slm.name wrote:
Hi,
On Mon, 2021-02-08 at 21:02 -0700, Chris Murphy wrote: > Hi, > > This is in regards to this Fedora 34 change: > https://fedoraproject.org/wiki/Changes/BtrfsTransparentCompression#Scope > > The gist is to do 'mount -o compress=zstd:1' anytime > Btrfs is > used, > whether Destination Installation>Automatic or Custom or > Advanced-Custom. Apply it during the installation, and > add it > to > /etc/fstab. > > Somehow I got confused thinking that autopart supports -- > fsoptions, > and that would be the way to do this. But (a) --fsoptions > isn't > supported with autopart, and (b) it wouldn't be a > universal > approach > anyway. And we want this to be consistent. Now I'm > thinking it > needs > to go somewhere in: > > https://github.com/storaged-project/blivet/blob/3.4-devel/blivet/devices/btr... > > Am I on the right track, or does it need to go somewhere > else, > or > in > addition to that? > (I'm one of the change owners, and trying to figure this out with Chris).
Ideally we have a solution that is configurable - i.e. this is exposed via a kickstart command (and probably entailing changes in Anaconda and pykickstart), but if that is not possible, or require a lot of rework (which we can try to work on), if we are willing to carry a patch for blivet or some other component to override the behavior just for Fedora 34, what's the best way of achieving that?
Btrfs is probably the only filesystem with built-in compression that we potentially care about, so designing an interface to expose this functionality might be an overkill -- but we're not sure.
Eventually, VDO might get integrated into the mainline tree, so having the interface which could be used for LVM compression through VDO wouldn't be too bad.
Hello!
The change says: "On variants using btrfs as the default filesystem, enable transparent compression using zstd." It means that the compression has to be configurable per product, so we need to add a new configuration option to the Anaconda configuration files (something like btrfs_compression_enabled = True). I guess you planned to use interactive-defaults.ks, but we are going to replace this file with the configuration files anyway.
I have checked the Blivet's code and we might be able to redefine the _mount_class attribute of the BTRFS class. The mount class defines default mount options. That should be enough to fix all partitioning methods. What do you think, Vojta?
I'm trying to figure this out, and am currently a bit stuck:
- patching devicetree/fsset.py's `fstab` to ensure fstab contains
compress=zstd:1 works, but the change only affects fstab on the installed system, so during installation compression is not enabled
- patching devicetree/handler.py's `mount_options` to inject
compress=zstd:1 does not seem to do anything https://github.com/rhinstaller/anaconda/compare/master...michel-slm:f34-34.2...
- patching devicetree/handler.py's `get_device_mount_options` to
ensure btrfs filesystems area always mounted with compress=zstd:1 does not work as expected (patching `set_device_mount_options` does not do anything either): https://github.com/rhinstaller/anaconda/compare/master...michel-slm:f34-34.2...
Since I don't really want to hardcode this long-term (hardcoding in a proof-of-concept, or for the F34 release only, is not too bad) -- is Blivet the place to do this? What I had in mind is:
- Some product definition in data/product.d specifies compression
should be enabled
- When that is enabled, Anaconda overrides the _mount_class of the
BTRFS class it gets from Blivet
^ where in Anaconda's codebase can I do this? Trying to follow this in the code but it's not easy especially since there does not seem to be an example of another filesystem conditionally overriding the default mount options.
Hello,
I have talked to Vojta and we suggest the following solution:
- Open a pull request with [0] in the upstream.
- Add a new flag to blivet.flags for enabling the compression of
btrfs. It should be False by default. Modify [0] to respect the flag. 3. Open a pull request that sets up the new flag in Anaconda. Anaconda sets the Blivet's flags in [1]. 4. If you want, add a new option to the [Storage] section of the Anaconda configuration file that will control the new flag. It should be False by default. Set this option in the relevant product configuration files located at [2].
[0] https://src.fedoraproject.org/rpms/python-blivet/pull-request/8 [1] https://github.com/rhinstaller/anaconda/blob/master/pyanaconda/modules/stora... [2] https://github.com/rhinstaller/anaconda/tree/master/data/product.d
Vendy
Thanks Vendy!
Per the discussion in IRC, going to follow the plan you outlined: 1. I'll modify my patch to make it accept a flag - btrfs_compression, which can be either False (or a falsy value) or a string with the compression option 2. I'll work on an Anaconda patch to set that flag 3. Test, if it works, submit both as PRs (hopefully today so it can be reviewed tomorrow CET) 4. Agreed that exposing this in Anaconda configuration can be done later, after the first two patches are in first. Will try this as a stretch goal if we have time
Thanks all,
On Tue, 2021-02-16 at 15:05 +0100, Vendula Poncova wrote:
On Tue, Feb 16, 2021 at 3:16 AM Michel Alexandre Salim michel@michel-slm.name wrote:
Hi Jiří, Vendula, all,
On Tue, 2021-02-09 at 12:12 +0100, Vendula Poncova wrote:
On Tue, Feb 9, 2021 at 10:42 AM jkonecny@redhat.com wrote:
Hi everyone,
I see a few options for this. First is to add this directly to blivet library as you pointed out already. However, I don't think blivet developers would be happy about that because they are trying to be as much as possible general purpose library and this change is really just about Anaconda.
Another option seems to be to modify Anaconda code directly. Unfortunately, I can't tell from top of my mind how hard that would be but still seems like the easiest solution.
However, the best approach which I can think of is to add something like that to our configuration files. Benefit would be that another modifications like that could be done easily in the future. That is of course for a discussion for the Anaconda team because it could be pretty hard to implemented this in some common form.
Vendy, Vojta, do you have some better ideas or could you tell us something more about my ideas above?
Best Regards, Jirka
On Tue, 2021-02-09 at 01:18 -0500, Neal Gompa wrote:
On Tue, Feb 9, 2021 at 1:09 AM Michel Alexandre Salim michel@michel-slm.name wrote:
Hi,
On Mon, 2021-02-08 at 21:02 -0700, Chris Murphy wrote: > Hi, > > This is in regards to this Fedora 34 change: > https://fedoraproject.org/wiki/Changes/BtrfsTransparentCompression#Scope > > The gist is to do 'mount -o compress=zstd:1' anytime > Btrfs is > used, > whether Destination Installation>Automatic or Custom or > Advanced-Custom. Apply it during the installation, and > add it > to > /etc/fstab. > > Somehow I got confused thinking that autopart supports -- > fsoptions, > and that would be the way to do this. But (a) --fsoptions > isn't > supported with autopart, and (b) it wouldn't be a > universal > approach > anyway. And we want this to be consistent. Now I'm > thinking it > needs > to go somewhere in: > > https://github.com/storaged-project/blivet/blob/3.4-devel/blivet/devices/btr... > > Am I on the right track, or does it need to go somewhere > else, > or > in > addition to that? > (I'm one of the change owners, and trying to figure this out with Chris).
Ideally we have a solution that is configurable - i.e. this is exposed via a kickstart command (and probably entailing changes in Anaconda and pykickstart), but if that is not possible, or require a lot of rework (which we can try to work on), if we are willing to carry a patch for blivet or some other component to override the behavior just for Fedora 34, what's the best way of achieving that?
Btrfs is probably the only filesystem with built-in compression that we potentially care about, so designing an interface to expose this functionality might be an overkill -- but we're not sure.
Eventually, VDO might get integrated into the mainline tree, so having the interface which could be used for LVM compression through VDO wouldn't be too bad.
Hello!
The change says: "On variants using btrfs as the default filesystem, enable transparent compression using zstd." It means that the compression has to be configurable per product, so we need to add a new configuration option to the Anaconda configuration files (something like btrfs_compression_enabled = True). I guess you planned to use interactive-defaults.ks, but we are going to replace this file with the configuration files anyway.
I have checked the Blivet's code and we might be able to redefine the _mount_class attribute of the BTRFS class. The mount class defines default mount options. That should be enough to fix all partitioning methods. What do you think, Vojta?
I'm trying to figure this out, and am currently a bit stuck:
- patching devicetree/fsset.py's `fstab` to ensure fstab contains
compress=zstd:1 works, but the change only affects fstab on the installed system, so during installation compression is not enabled
- patching devicetree/handler.py's `mount_options` to inject
compress=zstd:1 does not seem to do anything https://github.com/rhinstaller/anaconda/compare/master...michel-slm:f34-34.2...
- patching devicetree/handler.py's `get_device_mount_options` to
ensure btrfs filesystems area always mounted with compress=zstd:1 does not work as expected (patching `set_device_mount_options` does not do anything either): https://github.com/rhinstaller/anaconda/compare/master...michel-slm:f34-34.2...
Since I don't really want to hardcode this long-term (hardcoding in a proof-of-concept, or for the F34 release only, is not too bad) -- is Blivet the place to do this? What I had in mind is:
- Some product definition in data/product.d specifies compression
should be enabled
- When that is enabled, Anaconda overrides the _mount_class of the
BTRFS class it gets from Blivet
^ where in Anaconda's codebase can I do this? Trying to follow this in the code but it's not easy especially since there does not seem to be an example of another filesystem conditionally overriding the default mount options.
Hello,
I have talked to Vojta and we suggest the following solution:
- Open a pull request with [0] in the upstream.
- Add a new flag to blivet.flags for enabling the compression of
btrfs. It should be False by default. Modify [0] to respect the flag. 3. Open a pull request that sets up the new flag in Anaconda. Anaconda sets the Blivet's flags in [1]. 4. If you want, add a new option to the [Storage] section of the Anaconda configuration file that will control the new flag. It should be False by default. Set this option in the relevant product configuration files located at [2].
[0] https://src.fedoraproject.org/rpms/python-blivet/pull-request/8 [1] https://github.com/rhinstaller/anaconda/blob/master/pyanaconda/modules/stora... [2] https://github.com/rhinstaller/anaconda/tree/master/data/product.d
Vendy
Thanks Vendy and Vojta! This turns out to be way more straightforward than I expected.
https://github.com/storaged-project/blivet/pull/932 for the new Blivet PR - I tested that with just this change, a normal installation still works https://github.com/rhinstaller/anaconda/pull/3177 for the Anaconda PR - with both blivet and Anaconda patched, on Fedora I ended up with a Btrfs filesystem that's compressed both when the filesystem is mounted by Anaconda and when it's booted (the mount options end up in fstab).
Thanks all! I wish I could send everyone beers.
On Tue, 2021-02-16 at 20:16 -0800, Michel Alexandre Salim wrote:
On Tue, 2021-02-16 at 15:05 +0100, Vendula Poncova wrote:
On Tue, Feb 16, 2021 at 3:16 AM Michel Alexandre Salim michel@michel-slm.name wrote:
Hi Jiří, Vendula, all,
On Tue, 2021-02-09 at 12:12 +0100, Vendula Poncova wrote:
On Tue, Feb 9, 2021 at 10:42 AM jkonecny@redhat.com wrote:
Hi everyone,
I see a few options for this. First is to add this directly to blivet library as you pointed out already. However, I don't think blivet developers would be happy about that because they are trying to be as much as possible general purpose library and this change is really just about Anaconda.
Another option seems to be to modify Anaconda code directly. Unfortunately, I can't tell from top of my mind how hard that would be but still seems like the easiest solution.
However, the best approach which I can think of is to add something like that to our configuration files. Benefit would be that another modifications like that could be done easily in the future. That is of course for a discussion for the Anaconda team because it could be pretty hard to implemented this in some common form.
Vendy, Vojta, do you have some better ideas or could you tell us something more about my ideas above?
Best Regards, Jirka
On Tue, 2021-02-09 at 01:18 -0500, Neal Gompa wrote:
On Tue, Feb 9, 2021 at 1:09 AM Michel Alexandre Salim michel@michel-slm.name wrote: > Hi, > > On Mon, 2021-02-08 at 21:02 -0700, Chris Murphy wrote: > > Hi, > > > > This is in regards to this Fedora 34 change: > > https://fedoraproject.org/wiki/Changes/BtrfsTransparentCompression#Scope > > > > The gist is to do 'mount -o compress=zstd:1' anytime > > Btrfs is > > used, > > whether Destination Installation>Automatic or Custom or > > Advanced-Custom. Apply it during the installation, and > > add it > > to > > /etc/fstab. > > > > Somehow I got confused thinking that autopart supports -- > > fsoptions, > > and that would be the way to do this. But (a) --fsoptions > > isn't > > supported with autopart, and (b) it wouldn't be a > > universal > > approach > > anyway. And we want this to be consistent. Now I'm > > thinking it > > needs > > to go somewhere in: > > > > https://github.com/storaged-project/blivet/blob/3.4-devel/blivet/devices/btr... > > > > Am I on the right track, or does it need to go somewhere > > else, > > or > > in > > addition to that? > > > (I'm one of the change owners, and trying to figure this > out with > Chris). > > Ideally we have a solution that is configurable - i.e. this > is > exposed > via a kickstart command (and probably entailing changes in > Anaconda > and > pykickstart), but if that is not possible, or require a lot > of > rework > (which we can try to work on), if we are willing to carry a > patch > for > blivet or some other component to override the behavior > just for > Fedora > 34, what's the best way of achieving that? > > Btrfs is probably the only filesystem with built-in > compression > that we > potentially care about, so designing an interface to expose > this > functionality might be an overkill -- but we're not sure. >
Eventually, VDO might get integrated into the mainline tree, so having the interface which could be used for LVM compression through VDO wouldn't be too bad.
Hello!
The change says: "On variants using btrfs as the default filesystem, enable transparent compression using zstd." It means that the compression has to be configurable per product, so we need to add a new configuration option to the Anaconda configuration files (something like btrfs_compression_enabled = True). I guess you planned to use interactive-defaults.ks, but we are going to replace this file with the configuration files anyway.
I have checked the Blivet's code and we might be able to redefine the _mount_class attribute of the BTRFS class. The mount class defines default mount options. That should be enough to fix all partitioning methods. What do you think, Vojta?
I'm trying to figure this out, and am currently a bit stuck:
- patching devicetree/fsset.py's `fstab` to ensure fstab contains
compress=zstd:1 works, but the change only affects fstab on the installed system, so during installation compression is not enabled
- patching devicetree/handler.py's `mount_options` to inject
compress=zstd:1 does not seem to do anything https://github.com/rhinstaller/anaconda/compare/master...michel-slm:f34-34.2...
- patching devicetree/handler.py's `get_device_mount_options` to
ensure btrfs filesystems area always mounted with compress=zstd:1 does not work as expected (patching `set_device_mount_options` does not do anything either): https://github.com/rhinstaller/anaconda/compare/master...michel-slm:f34-34.2...
Since I don't really want to hardcode this long-term (hardcoding in a proof-of-concept, or for the F34 release only, is not too bad) -- is Blivet the place to do this? What I had in mind is:
- Some product definition in data/product.d specifies compression
should be enabled
- When that is enabled, Anaconda overrides the _mount_class of the
BTRFS class it gets from Blivet
^ where in Anaconda's codebase can I do this? Trying to follow this in the code but it's not easy especially since there does not seem to be an example of another filesystem conditionally overriding the default mount options.
Hello,
I have talked to Vojta and we suggest the following solution:
- Open a pull request with [0] in the upstream.
- Add a new flag to blivet.flags for enabling the compression of
btrfs. It should be False by default. Modify [0] to respect the flag. 3. Open a pull request that sets up the new flag in Anaconda. Anaconda sets the Blivet's flags in [1]. 4. If you want, add a new option to the [Storage] section of the Anaconda configuration file that will control the new flag. It should be False by default. Set this option in the relevant product configuration files located at [2].
[0] https://src.fedoraproject.org/rpms/python-blivet/pull-request/8 [1] https://github.com/rhinstaller/anaconda/blob/master/pyanaconda/modules/stora... [2] https://github.com/rhinstaller/anaconda/tree/master/data/product.d
Vendy
Thanks Vendy and Vojta! This turns out to be way more straightforward than I expected.
https://github.com/storaged-project/blivet/pull/932 for the new Blivet PR - I tested that with just this change, a normal installation still works https://github.com/rhinstaller/anaconda/pull/3177 for the Anaconda PR - with both blivet and Anaconda patched, on Fedora I ended up with a Btrfs filesystem that's compressed both when the filesystem is mounted by Anaconda and when it's booted (the mount options end up in fstab).
Thanks all! I wish I could send everyone beers.
Anaconda and Blivet with the needed changes have been built for F34 (and Rawhide): https://koji.fedoraproject.org/koji/buildinfo?buildID=1711886 https://koji.fedoraproject.org/koji/buildinfo?buildID=1711853
Both should be hitting Fedora composes near you shortly. :)
Anaconda-devel-list mailing list Anaconda-devel-list@listman.redhat.com https://listman.redhat.com/mailman/listinfo/anaconda-devel-list
On Fri, 2021-02-19 at 14:02 +0100, Martin Kolman wrote:
On Tue, 2021-02-16 at 20:16 -0800, Michel Alexandre Salim wrote:
On Tue, 2021-02-16 at 15:05 +0100, Vendula Poncova wrote:
On Tue, Feb 16, 2021 at 3:16 AM Michel Alexandre Salim michel@michel-slm.name wrote:
Hi Jiří, Vendula, all,
On Tue, 2021-02-09 at 12:12 +0100, Vendula Poncova wrote:
On Tue, Feb 9, 2021 at 10:42 AM jkonecny@redhat.com wrote:
Hi everyone,
I see a few options for this. First is to add this directly to blivet library as you pointed out already. However, I don't think blivet developers would be happy about that because they are trying to be as much as possible general purpose library and this change is really just about Anaconda.
Another option seems to be to modify Anaconda code directly. Unfortunately, I can't tell from top of my mind how hard that would be but still seems like the easiest solution.
However, the best approach which I can think of is to add something like that to our configuration files. Benefit would be that another modifications like that could be done easily in the future. That is of course for a discussion for the Anaconda team because it could be pretty hard to implemented this in some common form.
Vendy, Vojta, do you have some better ideas or could you tell us something more about my ideas above?
Best Regards, Jirka
On Tue, 2021-02-09 at 01:18 -0500, Neal Gompa wrote: > On Tue, Feb 9, 2021 at 1:09 AM Michel Alexandre Salim > michel@michel-slm.name wrote: > > Hi, > > > > On Mon, 2021-02-08 at 21:02 -0700, Chris Murphy wrote: > > > Hi, > > > > > > This is in regards to this Fedora 34 change: > > > https://fedoraproject.org/wiki/Changes/BtrfsTransparentCompression#Scope > > > > > > The gist is to do 'mount -o compress=zstd:1' anytime > > > Btrfs is > > > used, > > > whether Destination Installation>Automatic or Custom or > > > Advanced-Custom. Apply it during the installation, and > > > add it > > > to > > > /etc/fstab. > > > > > > Somehow I got confused thinking that autopart supports > > > -- > > > fsoptions, > > > and that would be the way to do this. But (a) -- > > > fsoptions > > > isn't > > > supported with autopart, and (b) it wouldn't be a > > > universal > > > approach > > > anyway. And we want this to be consistent. Now I'm > > > thinking it > > > needs > > > to go somewhere in: > > > > > > https://github.com/storaged-project/blivet/blob/3.4-devel/blivet/devices/btr... > > > > > > Am I on the right track, or does it need to go > > > somewhere > > > else, > > > or > > > in > > > addition to that? > > > > > (I'm one of the change owners, and trying to figure this > > out with > > Chris). > > > > Ideally we have a solution that is configurable - i.e. > > this > > is > > exposed > > via a kickstart command (and probably entailing changes > > in > > Anaconda > > and > > pykickstart), but if that is not possible, or require a > > lot > > of > > rework > > (which we can try to work on), if we are willing to carry > > a > > patch > > for > > blivet or some other component to override the behavior > > just for > > Fedora > > 34, what's the best way of achieving that? > > > > Btrfs is probably the only filesystem with built-in > > compression > > that we > > potentially care about, so designing an interface to > > expose > > this > > functionality might be an overkill -- but we're not sure. > > > > Eventually, VDO might get integrated into the mainline > tree, > so > having > the interface which could be used for LVM compression > through > VDO > wouldn't be too bad. > > >
Hello!
The change says: "On variants using btrfs as the default filesystem, enable transparent compression using zstd." It means that the compression has to be configurable per product, so we need to add a new configuration option to the Anaconda configuration files (something like btrfs_compression_enabled = True). I guess you planned to use interactive-defaults.ks, but we are going to replace this file with the configuration files anyway.
I have checked the Blivet's code and we might be able to redefine the _mount_class attribute of the BTRFS class. The mount class defines default mount options. That should be enough to fix all partitioning methods. What do you think, Vojta?
I'm trying to figure this out, and am currently a bit stuck:
- patching devicetree/fsset.py's `fstab` to ensure fstab contains
compress=zstd:1 works, but the change only affects fstab on the installed system, so during installation compression is not enabled
- patching devicetree/handler.py's `mount_options` to inject
compress=zstd:1 does not seem to do anything https://github.com/rhinstaller/anaconda/compare/master...michel-slm:f34-34.2...
- patching devicetree/handler.py's `get_device_mount_options` to
ensure btrfs filesystems area always mounted with compress=zstd:1 does not work as expected (patching `set_device_mount_options` does not do anything either): https://github.com/rhinstaller/anaconda/compare/master...michel-slm:f34-34.2...
Since I don't really want to hardcode this long-term (hardcoding in a proof-of-concept, or for the F34 release only, is not too bad) -- is Blivet the place to do this? What I had in mind is:
- Some product definition in data/product.d specifies compression
should be enabled
- When that is enabled, Anaconda overrides the _mount_class of
the BTRFS class it gets from Blivet
^ where in Anaconda's codebase can I do this? Trying to follow this in the code but it's not easy especially since there does not seem to be an example of another filesystem conditionally overriding the default mount options.
Hello,
I have talked to Vojta and we suggest the following solution:
- Open a pull request with [0] in the upstream.
- Add a new flag to blivet.flags for enabling the compression of
btrfs. It should be False by default. Modify [0] to respect the flag. 3. Open a pull request that sets up the new flag in Anaconda. Anaconda sets the Blivet's flags in [1]. 4. If you want, add a new option to the [Storage] section of the Anaconda configuration file that will control the new flag. It should be False by default. Set this option in the relevant product configuration files located at [2].
[0] https://src.fedoraproject.org/rpms/python-blivet/pull-request/8 [1] https://github.com/rhinstaller/anaconda/blob/master/pyanaconda/modules/stora... [2] https://github.com/rhinstaller/anaconda/tree/master/data/product.d
Vendy
Thanks Vendy and Vojta! This turns out to be way more straightforward than I expected.
https://github.com/storaged-project/blivet/pull/932%C2%A0for the new Blivet PR - I tested that with just this change, a normal installation still works https://github.com/rhinstaller/anaconda/pull/3177%C2%A0for the Anaconda PR
with both blivet and Anaconda patched, on Fedora I ended up with a Btrfs filesystem that's compressed both when the filesystem is mounted by Anaconda and when it's booted (the mount options end up in fstab).
Thanks all! I wish I could send everyone beers.
Anaconda and Blivet with the needed changes have been built for F34 (and Rawhide): https://koji.fedoraproject.org/koji/buildinfo?buildID=1711886 https://koji.fedoraproject.org/koji/buildinfo?buildID=1711853
Both should be hitting Fedora composes near you shortly. :)
Thank you! Sorry for the belated reply, life happened[^1] so I'm really glad these got sorted just in time!
[^1]: https://michel-slm.name/posts/2021-02-26-software-wetware/
Best regards,
On Sat, 2021-02-27 at 11:41 -0800, Michel Alexandre Salim wrote:
On Fri, 2021-02-19 at 14:02 +0100, Martin Kolman wrote:
On Tue, 2021-02-16 at 20:16 -0800, Michel Alexandre Salim wrote:
On Tue, 2021-02-16 at 15:05 +0100, Vendula Poncova wrote:
On Tue, Feb 16, 2021 at 3:16 AM Michel Alexandre Salim michel@michel-slm.name wrote:
Hi Jiří, Vendula, all,
On Tue, 2021-02-09 at 12:12 +0100, Vendula Poncova wrote:
On Tue, Feb 9, 2021 at 10:42 AM jkonecny@redhat.com wrote: > Hi everyone, > > I see a few options for this. First is to add this directly > to > blivet > library as you pointed out already. However, I don't think > blivet > developers would be happy about that because they are trying > to > be as > much as possible general purpose library and this change is > really > just > about Anaconda. > > Another option seems to be to modify Anaconda code directly. > Unfortunately, I can't tell from top of my mind how hard that > would > be > but still seems like the easiest solution. > > However, the best approach which I can think of is to add > something > like that to our configuration files. Benefit would be that > another > modifications like that could be done easily in the future. > That is > of > course for a discussion for the Anaconda team because it > could > be > pretty hard to implemented this in some common form. > > Vendy, Vojta, do you have some better ideas or could you tell > us > something more about my ideas above? > > Best Regards, > Jirka > > On Tue, 2021-02-09 at 01:18 -0500, Neal Gompa wrote: > > On Tue, Feb 9, 2021 at 1:09 AM Michel Alexandre Salim > > michel@michel-slm.name wrote: > > > Hi, > > > > > > On Mon, 2021-02-08 at 21:02 -0700, Chris Murphy wrote: > > > > Hi, > > > > > > > > This is in regards to this Fedora 34 change: > > > > https://fedoraproject.org/wiki/Changes/BtrfsTransparentCompression#Scope > > > > > > > > The gist is to do 'mount -o compress=zstd:1' anytime > > > > Btrfs is > > > > used, > > > > whether Destination Installation>Automatic or Custom or > > > > Advanced-Custom. Apply it during the installation, and > > > > add it > > > > to > > > > /etc/fstab. > > > > > > > > Somehow I got confused thinking that autopart supports > > > > -- > > > > fsoptions, > > > > and that would be the way to do this. But (a) -- > > > > fsoptions > > > > isn't > > > > supported with autopart, and (b) it wouldn't be a > > > > universal > > > > approach > > > > anyway. And we want this to be consistent. Now I'm > > > > thinking it > > > > needs > > > > to go somewhere in: > > > > > > > > https://github.com/storaged-project/blivet/blob/3.4-devel/blivet/devices/btr... > > > > > > > > Am I on the right track, or does it need to go > > > > somewhere > > > > else, > > > > or > > > > in > > > > addition to that? > > > > > > > (I'm one of the change owners, and trying to figure this > > > out with > > > Chris). > > > > > > Ideally we have a solution that is configurable - i.e. > > > this > > > is > > > exposed > > > via a kickstart command (and probably entailing changes > > > in > > > Anaconda > > > and > > > pykickstart), but if that is not possible, or require a > > > lot > > > of > > > rework > > > (which we can try to work on), if we are willing to carry > > > a > > > patch > > > for > > > blivet or some other component to override the behavior > > > just for > > > Fedora > > > 34, what's the best way of achieving that? > > > > > > Btrfs is probably the only filesystem with built-in > > > compression > > > that we > > > potentially care about, so designing an interface to > > > expose > > > this > > > functionality might be an overkill -- but we're not sure. > > > > > > > Eventually, VDO might get integrated into the mainline > > tree, > > so > > having > > the interface which could be used for LVM compression > > through > > VDO > > wouldn't be too bad. > > > > > >
Hello!
The change says: "On variants using btrfs as the default filesystem, enable transparent compression using zstd." It means that the compression has to be configurable per product, so we need to add a new configuration option to the Anaconda configuration files (something like btrfs_compression_enabled = True). I guess you planned to use interactive-defaults.ks, but we are going to replace this file with the configuration files anyway.
I have checked the Blivet's code and we might be able to redefine the _mount_class attribute of the BTRFS class. The mount class defines default mount options. That should be enough to fix all partitioning methods. What do you think, Vojta?
I'm trying to figure this out, and am currently a bit stuck:
- patching devicetree/fsset.py's `fstab` to ensure fstab contains
compress=zstd:1 works, but the change only affects fstab on the installed system, so during installation compression is not enabled
- patching devicetree/handler.py's `mount_options` to inject
compress=zstd:1 does not seem to do anything https://github.com/rhinstaller/anaconda/compare/master...michel-slm:f34-34.2...
- patching devicetree/handler.py's `get_device_mount_options` to
ensure btrfs filesystems area always mounted with compress=zstd:1 does not work as expected (patching `set_device_mount_options` does not do anything either): https://github.com/rhinstaller/anaconda/compare/master...michel-slm:f34-34.2...
Since I don't really want to hardcode this long-term (hardcoding in a proof-of-concept, or for the F34 release only, is not too bad) -- is Blivet the place to do this? What I had in mind is:
- Some product definition in data/product.d specifies compression
should be enabled
- When that is enabled, Anaconda overrides the _mount_class of
the BTRFS class it gets from Blivet
^ where in Anaconda's codebase can I do this? Trying to follow this in the code but it's not easy especially since there does not seem to be an example of another filesystem conditionally overriding the default mount options.
Hello,
I have talked to Vojta and we suggest the following solution:
- Open a pull request with [0] in the upstream.
- Add a new flag to blivet.flags for enabling the compression of
btrfs. It should be False by default. Modify [0] to respect the flag. 3. Open a pull request that sets up the new flag in Anaconda. Anaconda sets the Blivet's flags in [1]. 4. If you want, add a new option to the [Storage] section of the Anaconda configuration file that will control the new flag. It should be False by default. Set this option in the relevant product configuration files located at [2].
[0] https://src.fedoraproject.org/rpms/python-blivet/pull-request/8 [1] https://github.com/rhinstaller/anaconda/blob/master/pyanaconda/modules/stora... [2] https://github.com/rhinstaller/anaconda/tree/master/data/product.d
Vendy
Thanks Vendy and Vojta! This turns out to be way more straightforward than I expected.
https://github.com/storaged-project/blivet/pull/932 for the new Blivet PR - I tested that with just this change, a normal installation still works https://github.com/rhinstaller/anaconda/pull/3177 for the Anaconda PR
with both blivet and Anaconda patched, on Fedora I ended up with a Btrfs filesystem that's compressed both when the filesystem is mounted by Anaconda and when it's booted (the mount options end up in fstab).
Thanks all! I wish I could send everyone beers.
Anaconda and Blivet with the needed changes have been built for F34 (and Rawhide): https://koji.fedoraproject.org/koji/buildinfo?buildID=1711886 https://koji.fedoraproject.org/koji/buildinfo?buildID=1711853
Both should be hitting Fedora composes near you shortly. :)
Thank you! Sorry for the belated reply, life happened[^1] so I'm really glad these got sorted just in time!
Nice, all the best! :)
Best regards,
Anaconda-devel-list mailing list Anaconda-devel-list@redhat.com https://listman.redhat.com/mailman/listinfo/anaconda-devel-list
anaconda-devel@lists.fedoraproject.org
-
Chris Murphy -
jkonecny＠redhat.com -
Martin Kolman -
Michel Alexandre Salim -
Neal Gompa -
Patrick Riehecky -
Vendula Poncova