Scoping out the change needed for Anaconda to support resizing Btrfs filesystems

[Michel Alexandre Salim]

Dear Anaconda developers,

There's this 7 year old bug about Anaconda not being able to resize
Btrfs filesystems, despite Btrfs itself supporting resizing.

https://bugzilla.redhat.com/show_bug.cgi?id=962143

Now that Fedora defaults to Btrfs on desktop variants, since Facebook
uses Fedora as its preferred desktop Linux distribution we might be
able to dedicate some manpower to working on this, but need help to
scope out this work.

Would any change needed be limited to python-blivet, or would other
components need to be touched?

Also, what's the best way to test this, build Anaconda from the master
branch and use it for our installation?

Thanks,

[Chris Murphy]

On Thu, Nov 12, 2020 at 8:18 PM Michel Alexandre Salim
 wrote:
> Dear Anaconda developers,
>
> There's this 7 year old bug about Anaconda not being able to resize
> Btrfs filesystems, despite Btrfs itself supporting resizing.
>
> https://bugzilla.redhat.com/show_bug.cgi?id=962143
>
> Now that Fedora defaults to Btrfs on desktop variants, since Facebook
> uses Fedora as its preferred desktop Linux distribution we might be
> able to dedicate some manpower to working on this, but need help to
> scope out this work.
>
> Would any change needed be limited to python-blivet, or would other
> components need to be touched?

I just created this to track the various issues around Btrfs shrink
and utilities that maybe/probably should support it.

https://pagure.io/fedora-btrfs/project/issue/35

I think the main thing that Anaconda needs is already in libblockdev.
But it's missing a minimum size check. That issue is referenced in the
fedora-btrfs#35 issue. I'm not sure where it belongs, but there
probably needs to be a check for number of devices and only support
shrink on single device Btrfs.

Hello Michel,

It's great to see community help for the BTRFS features! We will try to
help you to get the work done as smooth as possible.

First advice. If you need help with debbuging or questions about code
just come to our #anaconda on freenode IRC and you should be able to
get answers soon (GMT +1, Czech Republic timezone).

For the rest see my replies below.

On Thu, 2020-11-12 at 17:11 -0800, Michel Alexandre Salim wrote:
> Dear Anaconda developers,
>
> There's this 7 year old bug about Anaconda not being able to resize
> Btrfs filesystems, despite Btrfs itself supporting resizing.
>
> https://bugzilla.redhat.com/show_bug.cgi?id=962143
>
> Now that Fedora defaults to Btrfs on desktop variants, since Facebook
> uses Fedora as its preferred desktop Linux distribution we might be
> able to dedicate some manpower to working on this, but need help to
> scope out this work.
>
> Would any change needed be limited to python-blivet, or would other
> components need to be touched?

I didn't looked on the code but I expect that you have to take care of
both blivet and Anaconda to fix the RFC.
> Also, what's the best way to test this, build Anaconda from the
> master
> branch and use it for our installation?

Building is too unpleasant experience for testing. Ideally you should
be using updates image feature[1]. Updates image is just an image (or
tar) which is upacked and files in it will replace files in the
installation environment. Until you need to touch Dracut code (not
expected here) you should be fine to generate updates image, upload to
a HTTP server and boot with inst.updates. You can also replace blivet
by this. See my blog post about Anaconda debugging for more details[2].
My recommended way is to use the make-updates wrapper[3] mentioned as
the last part in blog post. Also for any other questions please feel
free to ask :).

Aside of this. This is a great candidate for Pure Community Feature[4].
If possible, could you or someone else please maintain this feature in
Anaconda so there will be contact on someone who can fix the feature if
it breaks?

[1]:
https://anaconda-installer.readthedocs.io/en/latest/boot-options.html#inst-u...
[2]:
https://rhinstaller.wordpress.com/2019/10/11/anaconda-debugging-and-testing-...
[3]:
https://github.com/rhinstaller/devel-tools/tree/master/anaconda_updates
[4]:
https://anaconda-installer.readthedocs.io/en/latest/contributing.html#pure-c...

Best Regards,
Jirka

[Vendula Poncova]

On Fri, Nov 13, 2020 at 2:22 PM  wrote:
> Hello Michel,
>
> It's great to see community help for the BTRFS features! We will try to
> help you to get the work done as smooth as possible.
>
> First advice. If you need help with debbuging or questions about code
> just come to our #anaconda on freenode IRC and you should be able to
> get answers soon (GMT +1, Czech Republic timezone).
>
> For the rest see my replies below.
>
> On Thu, 2020-11-12 at 17:11 -0800, Michel Alexandre Salim wrote:
> > Dear Anaconda developers,
> >
> > There's this 7 year old bug about Anaconda not being able to resize
> > Btrfs filesystems, despite Btrfs itself supporting resizing.
> >
> > https://bugzilla.redhat.com/show_bug.cgi?id=962143
> >
> > Now that Fedora defaults to Btrfs on desktop variants, since Facebook
> > uses Fedora as its preferred desktop Linux distribution we might be
> > able to dedicate some manpower to working on this, but need help to
> > scope out this work.
> >
> > Would any change needed be limited to python-blivet, or would other
> > components need to be touched?
>
> I didn't looked on the code but I expect that you have to take care of
> both blivet and Anaconda to fix the RFC.

Hello,

this change should affect only the "Reclaim Disk Space" dialog. Is it
correct? In that case, its scope depends on the Blivet's
implementation. If the btrfs partition can be resizable the same way
as any other resizable partition, no changes should be needed in
Anaconda. Otherwise, it might be necessary to modify our
implementation, extend the DBus API or change the user interface.

I'm adding Vojta Trefny to CC.

Vendy
> > Also, what's the best way to test this, build Anaconda from the
> > master
> > branch and use it for our installation?
>
> Building is too unpleasant experience for testing. Ideally you should
> be using updates image feature[1]. Updates image is just an image (or
> tar) which is upacked and files in it will replace files in the
> installation environment. Until you need to touch Dracut code (not
> expected here) you should be fine to generate updates image, upload to
> a HTTP server and boot with inst.updates. You can also replace blivet
> by this. See my blog post about Anaconda debugging for more details[2].
> My recommended way is to use the make-updates wrapper[3] mentioned as
> the last part in blog post. Also for any other questions please feel
> free to ask :).
>
> Aside of this. This is a great candidate for Pure Community Feature[4].
> If possible, could you or someone else please maintain this feature in
> Anaconda so there will be contact on someone who can fix the feature if
> it breaks?
>
> Best Regards,
> Jirka
>
> Anaconda-devel-list mailing list
> [e-mail]
> https://www.redhat.com/mailman/listinfo/anaconda-devel-list

[Vojtěch Trefný]

On 11/18/20 11:19 AM, Vendula Poncova wrote:
> On Fri, Nov 13, 2020 at 2:22 PM  wrote:
> > Hello Michel,
> >
> > It's great to see community help for the BTRFS features! We will try to
> > help you to get the work done as smooth as possible.
> >
> > First advice. If you need help with debbuging or questions about code
> > just come to our #anaconda on freenode IRC and you should be able to
> > get answers soon (GMT +1, Czech Republic timezone).
> >
> > For the rest see my replies below.
> >
> > On Thu, 2020-11-12 at 17:11 -0800, Michel Alexandre Salim wrote:
> > > Dear Anaconda developers,
> > >
> > > There's this 7 year old bug about Anaconda not being able to resize
> > > Btrfs filesystems, despite Btrfs itself supporting resizing.
> > >
> > > https://bugzilla.redhat.com/show_bug.cgi?id=962143
> > >
> > > Now that Fedora defaults to Btrfs on desktop variants, since Facebook
> > > uses Fedora as its preferred desktop Linux distribution we might be
> > > able to dedicate some manpower to working on this, but need help to
> > > scope out this work.
> > >
> > > Would any change needed be limited to python-blivet, or would other
> > > components need to be touched?
> >
> > I didn't looked on the code but I expect that you have to take care of
> > both blivet and Anaconda to fix the RFC.
>
> Hello,
>
> this change should affect only the "Reclaim Disk Space" dialog. Is it
> correct? In that case, its scope depends on the Blivet's
> implementation. If the btrfs partition can be resizable the same way
> as any other resizable partition, no changes should be needed in
> Anaconda. Otherwise, it might be necessary to modify our
> implementation, extend the DBus API or change the user interface.
>
> I'm adding Vojta Trefny to CC.
>
> Vendy

Hi, I'm afraid we might need some change in Anaconda too. The problem is
the way how blivet works with btrfs -- internally we don't really see
btrfs as a filesystem but only as a volume manager with primary focus on
subvolumes.

This how a simple btrfs device with two subvolumes looks in blivet:

existing 718 MiB partition sdc1 (323) with existing btrfs filesystem
  existing 718 MiB btrfs volume btrfs.332 (332) with existing btrfs
filesystem
    existing 718 MiB btrfs subvolume subvol1 (339) with existing btrfs
filesystem
    existing 718 MiB btrfs subvolume subvol2 (343) with existing btrfs
filesystem

and we'll need to be able to resize all of this. In the end it's just
one "btrfs filesystem resize" but we'll need to make sure the size of
these "virtual" devices is correct when running the resize from Anaconda
and we'll probably need a special code for that in Anaconda too.

What changes will be needed:

libblockdev: Chris already mentioned the minimal size, also the resize
function should be smarter, we currently run only "btrfs filesystem
resize <size> <mountpoint>" without any checks.
(Note: We are currently working on a new major version of libblockdev
and part of it should be separating the "single filesystem"
functionality like mkfs and fsck and possibly also resize and the
"volume manager" functionality. Part of this should also be rewriting
the btrfs plugin usingo libbtrfsutil[1].)

blivet: The biggest change should be here, the BTRFS filesystem[2] or
the BTRFSDevice[3] (or both) must be made resizable and min size must
also be set correctly.
Unfortunately the blivet code is a little complicated. But if you (or
someone else from Facebook) is willing to work on the resize support,
I'll be happy to help you with that.

anaconda: Hopefully only a small change in the reclaim space dialog.

[1] https://github.com/storaged-project/libblockdev/issues/552
[2]
https://github.com/storaged-project/blivet/blob/3.3-devel/blivet/formats/fs....
[3]
https://github.com/storaged-project/blivet/blob/3.3-devel/blivet/blivet.py
> > > Also, what's the best way to test this, build Anaconda from the
> > > master
> > > branch and use it for our installation?
> >
> > Building is too unpleasant experience for testing. Ideally you should
> > be using updates image feature[1]. Updates image is just an image (or
> > tar) which is upacked and files in it will replace files in the
> > installation environment. Until you need to touch Dracut code (not
> > expected here) you should be fine to generate updates image, upload to
> > a HTTP server and boot with inst.updates. You can also replace blivet
> > by this. See my blog post about Anaconda debugging for more details[2].
> > My recommended way is to use the make-updates wrapper[3] mentioned as
> > the last part in blog post. Also for any other questions please feel
> > free to ask :).
> >
> > Aside of this. This is a great candidate for Pure Community Feature[4].
> > If possible, could you or someone else please maintain this feature in
> > Anaconda so there will be contact on someone who can fix the feature if
> > it breaks?
> >
> > Best Regards,
> > Jirka
> >
> > Anaconda-devel-list mailing list
> > [e-mail]
> > https://www.redhat.com/mailman/listinfo/anaconda-devel-list