F25 Self Contained Change: Replace UDisks2 by Storaged

[Jan Kurik]

= Proposed Self Contained Change: Replace UDisks2 by Storaged =
https://fedoraproject.org/wiki/Changes/Replace_UDisks2_by_Storaged

Change owner(s):
* Peter Hatina <phatina redhat com>
* Tomáš Smetana <tsmetana redhat com >

Storaged extends UDisks2 API by exporting several enterprise features
(in form of plugins), such as LVM2 and iSCSI. This project is a
drop-in replacement for UDisks2, either from D-Bus or binary point of
view. The main motivation of this change is to provide the unified
D-Bus API for all the clients who are willing to manage LVM2, iSCSI,
Btrfs, BCache, LSM and ZRam.

== Detailed Description ==
Aim of Storaged is to provide unified higher level management
interface for various clients who are willing to query and manage
storage bits of the system. We plan to replace UDisks2 by Storaged,
since the Storaged itself is the fork of UDisks2 and these 2 projects
in its core haven't diverged so much (Storaged got some improvements
which popped up while using it).

== Scope ==
Proposal owners: To implement this Change

Other developers: N/A (not a System Wide Change)

Release engineering: N/A (not a System Wide Change)

List of deliverables: N/A (not a System Wide Change)

Policies and guidelines: N/A (not a System Wide Change)

Trademark approval: N/A (not needed for this Change)

[Bastien Nocera]

Hey,

----- Original Message -----
> = Proposed Self Contained Change: Replace UDisks2 by Storaged =
> https://fedoraproject.org/wiki/Changes/Replace_UDisks2_by_Storaged
>
> Change owner(s):
> Peter Hatina <phatina redhat com>
> Tomáš Smetana <tsmetana redhat com >
>
> Storaged extends UDisks2 API by exporting several enterprise features
> (in form of plugins), such as LVM2 and iSCSI. This project is a
> drop-in replacement for UDisks2, either from D-Bus or binary point of
> view. The main motivation of this change is to provide the unified
> D-Bus API for all the clients who are willing to manage LVM2, iSCSI,
> Btrfs, BCache, LSM and ZRam.
>
> == Detailed Description ==
> Aim of Storaged is to provide unified higher level management
> interface for various clients who are willing to query and manage
> storage bits of the system. We plan to replace UDisks2 by Storaged,
> since the Storaged itself is the fork of UDisks2 and these 2 projects
> in its core haven't diverged so much (Storaged got some improvements
> which popped up while using it).

What are the additional dependencies compared to Udisks2?

Would gnome-disk-utility, gvfs, etc. work as well as they used to without regressions when
dropping in storaged, either on a running system, or when compiling against it?

Will bug fixes and enhancements to the common part between storaged and
udisks2 be backported to udisks2?

I'm fairly certain we don't want iSCSI binaries in the Workstation installation
(we've been trying to get rid of the ones that Anaconda brings in already).

I also don't see why ZRam is something 1) you'd want to have to configure,
2) that has its place in a storage API.

Cheers

[Martin Kolman]

On Fri, 2016-04-15 at 05:09 -0400, Bastien Nocera wrote:
> Hey,
>
> ----- Original Message -----
> > = Proposed Self Contained Change: Replace UDisks2 by Storaged =
> > https://fedoraproject.org/wiki/Changes/Replace_UDisks2_by_Storaged
> >
> > Change owner(s):
> > Peter Hatina <phatina redhat com>
> > Tomáš Smetana <tsmetana redhat com >
> >
> > Storaged extends UDisks2 API by exporting several enterprise
> > features
> > (in form of plugins), such as LVM2 and iSCSI. This project is a
> > drop-in replacement for UDisks2, either from D-Bus or binary point
> > of
> > view. The main motivation of this change is to provide the unified
> > D-Bus API for all the clients who are willing to manage LVM2,
> > iSCSI,
> > Btrfs, BCache, LSM and ZRam.
> >
> > == Detailed Description ==
> > Aim of Storaged is to provide unified higher level management
> > interface for various clients who are willing to query and manage
> > storage bits of the system. We plan to replace UDisks2 by Storaged,
> > since the Storaged itself is the fork of UDisks2 and these 2
> > projects
> > in its core haven't diverged so much (Storaged got some
> > improvements
> > which popped up while using it).
>
> What are the additional dependencies compared to Udisks2?
>
> Would gnome-disk-utility, gvfs, etc. work as well as they used to
> without regressions when
> dropping in storaged, either on a running system, or when compiling
> against it?
>
> Will bug fixes and enhancements to the common part between storaged
> and
> udisks2 be backported to udisks2?
>
> I'm fairly certain we don't want iSCSI binaries in the Workstation
> installation
> (we've been trying to get rid of the ones that Anaconda brings in
> already).
>
> I also don't see why ZRam is something 1) you'd want to have to
> configure,

Well, its a compressed bloc device residing in RAM - I can see various
use cases, not just the most frequent one (zram used as a swap device).
Some people for example use tmpfs for running builds, I guess zram
could be used for similar purposes, possibly enabling bigger builds
that take a bit longer than on tmpfs.
> that has its place in a storage API.

Why is an API providing some additional functionality you are not going
to use in your specific usecase a problem ?

I'm sure glibc and kernel provide many API calls not directly used by
the default apps on the workstation and we are still using them...
> Cheers
>
> devel mailing list
> [e-mail]
> http://lists.fedoraproject.org/admin/lists/[e-mail].
> org

[Bastien Nocera]

----- Original Message -----
<snip>
> > that has its place in a storage API.
>
> Why is an API providing some additional functionality you are not going
> to use in your specific usecase a problem ?
>
> I'm sure glibc and kernel provide many API calls not directly used by
> the default apps on the workstation and we are still using them...

If I ever saw an example of logical fallacy, that is one.

Why do I care? Because if I didn't whine on this very list some time ago,
storaged wouldn't be API-compatible with udisks2, and I have to wonder about
the architecture of something we would have to support when it makes
decisions like supporting features for which you don't have a clear use-case.

[Tomáš Smetana]

On Fri, 15 Apr 2016 05:09:00 -0400 (EDT)
Bastien Nocera  wrote:
> Hey,

...
> What are the additional dependencies compared to Udisks2?

None AFAIK: storaged itself should not pull anything else than udisks2 in
the current Rawhide.
> Would gnome-disk-utility, gvfs, etc. work as well as they used to without
> regressions when dropping in storaged, either on a running system, or when
> compiling against it?

Yes, such is the plan.
> Will bug fixes and enhancements to the common part between storaged and
> udisks2 be backported to udisks2?

I assume this is a question for the udisks2 developers.
> I'm fairly certain we don't want iSCSI binaries in the Workstation
> installation (we've been trying to get rid of the ones that Anaconda brings
> in already).
>
> I also don't see why ZRam is something 1) you'd want to have to configure,
> 2) that has its place in a storage API.

It's been put to the API because Blivet would like to use it from storaged
eventually. However you're right: this is something that the user may not want
to install and therefore the storaged-zram is a separate package. Same as
LSM, LVM2, iSCSI, bcache and BTRFS plugins. Of course the plugins have their
own additional dependencies.

Regards,

[Bastien Nocera]

----- Original Message -----
> On Fri, 15 Apr 2016 05:09:00 -0400 (EDT)
> Bastien Nocera  wrote:
> > Hey,
>
> ...
> > What are the additional dependencies compared to Udisks2?
>
> None AFAIK: storaged itself should not pull anything else than udisks2 in
> the current Rawhide.
> > Would gnome-disk-utility, gvfs, etc. work as well as they used to without
> > regressions when dropping in storaged, either on a running system, or when
> > compiling against it?
>
> Yes, such is the plan.
> > Will bug fixes and enhancements to the common part between storaged and
> > udisks2 be backported to udisks2?
>
> I assume this is a question for the udisks2 developers.

It's not. I don't think that Workstation wants to stray from upstream GNOME,
which uses udisks2, so providing maintenance would be useful.
> > I'm fairly certain we don't want iSCSI binaries in the Workstation
> > installation (we've been trying to get rid of the ones that Anaconda brings
> > in already).
> >
> > I also don't see why ZRam is something 1) you'd want to have to configure,
> > 2) that has its place in a storage API.
>
> It's been put to the API because Blivet would like to use it from storaged
> eventually.

Still, I don't understand the use-case for such an interface.
> However you're right: this is something that the user may not
> want
> to install and therefore the storaged-zram is a separate package. Same as
> LSM, LVM2, iSCSI, bcache and BTRFS plugins. Of course the plugins have their
> own additional dependencies.

Would that mean that we don't need to ship with the iscsi initiator in
Workstation because there would be no direct or indirect dependencies in Anaconda,
for example?

[Chris Murphy]

On Fri, Apr 15, 2016 at 10:14 AM, Bastien Nocera  wrote:
> ----- Original Message -----
> > On Fri, 15 Apr 2016 05:09:00 -0400 (EDT)
> > Bastien Nocera  wrote:
> > > Hey,
> >
> > ...
> > > What are the additional dependencies compared to Udisks2?
> >
> > None AFAIK: storaged itself should not pull anything else than udisks2 in
> > the current Rawhide.
> > > Would gnome-disk-utility, gvfs, etc. work as well as they used to without
> > > regressions when dropping in storaged, either on a running system, or when
> > > compiling against it?
> >
> > Yes, such is the plan.
> > > Will bug fixes and enhancements to the common part between storaged and
> > > udisks2 be backported to udisks2?
> >
> > I assume this is a question for the udisks2 developers.
>
> It's not. I don't think that Workstation wants to stray from upstream GNOME,
> which uses udisks2, so providing maintenance would be useful.

What about GNOME transitioning to use storaged directly? There are
things missing for some time, like filesystem resize, LVM thin, almost
anything related to Btrfs, etc.
> > > I'm fairly certain we don't want iSCSI binaries in the Workstation
> > > installation (we've been trying to get rid of the ones that Anaconda brings
> > > in already).
> > >
> > > I also don't see why ZRam is something 1) you'd want to have to configure,
> > > 2) that has its place in a storage API.
> >
> > It's been put to the API because Blivet would like to use it from storaged
> > eventually.
>
> Still, I don't understand the use-case for such an interface.

It's the creation of a block device that needs configuration: size,
max size, compression algorithm, and that config presumably needs to
be stored somewhere to create zram and swap at boot since it's
volatile. And since it's a alternative for swap on disk, it makes
sense to have an API that the installer can use, and other things can
modify/manage post-install rather than each thing having to reinvent
that wheel.

[Stephen Gallagher]

On 04/14/2016 10:50 AM, Jan Kurik wrote:
> = Proposed Self Contained Change: Replace UDisks2 by Storaged =
> https://fedoraproject.org/wiki/Changes/Replace_UDisks2_by_Storaged
>
> Change owner(s):
> Peter Hatina <phatina redhat com>
> Tomáš Smetana <tsmetana redhat com >
>
> Storaged extends UDisks2 API by exporting several enterprise features
> (in form of plugins), such as LVM2 and iSCSI. This project is a
> drop-in replacement for UDisks2, either from D-Bus or binary point of
> view. The main motivation of this change is to provide the unified
> D-Bus API for all the clients who are willing to manage LVM2, iSCSI,
> Btrfs, BCache, LSM and ZRam.
>
> == Detailed Description ==
> Aim of Storaged is to provide unified higher level management
> interface for various clients who are willing to query and manage
> storage bits of the system. We plan to replace UDisks2 by Storaged,
> since the Storaged itself is the fork of UDisks2 and these 2 projects
> in its core haven't diverged so much (Storaged got some improvements
> which popped up while using it).
>
> == Scope ==
> Proposal owners: To implement this Change
>
> Other developers: N/A (not a System Wide Change)
>
> Release engineering: N/A (not a System Wide Change)
>
> List of deliverables: N/A (not a System Wide Change)
>
> Policies and guidelines: N/A (not a System Wide Change)
>
> Trademark approval: N/A (not needed for this Change)

CCing the desktop list, but please keep replies on the Fedora devel list.

At today's FESCo meeting[1], we raised several questions that we would like to 
have answered about this change.

1) Have the GNOME, Anaconda/Blivet and Cockpit folks agreed to replacing Udisks2?
2) Is it expected to be a complete cutover (where we drop Udisks2 from the 
default install and/or from the distro entirely

Additionally (from my own stockpile of questions): how is such a replacement 
going to be done? Are you planning to have storaged take ownership of 
org.freedesktop.Udisks2 or are you expecting to convert all consumers of that 
interface over to using org.storaged.Storaged? I know that, except for the 
interface name, storaged is providing a superset of the functionality offered by 
storaged, so I would guess that the simplest approach would be to have it take 
over org.freedesktop.Udisks2, but that hasn't been documented anywhere that I 
can find.

Please include answers to these questions on the Change Proposal.

[1] 
https://meetbot.fedoraproject.org/fedora-meeting/2016-05-13/fesco.2016-05-13...

[Michael Catanzaro]

On Fri, 2016-05-13 at 13:01 -0400, Stephen Gallagher wrote:
> CCing the desktop list, but please keep replies on the Fedora devel
> list.
>
> At today's FESCo meeting[1], we raised several questions that we
> would like to 
> have answered about this change.
> Have the GNOME, Anaconda/Blivet and Cockpit folks agreed to
>
> replacing Udisks2?
> 2) Is it expected to be a complete cutover (where we drop Udisks2
> from the 
> default install and/or from the distro entirely

udisks upstream is inactive and has recently expressed some interest in
allowing the storaged developers to take over maintenance of udisks
going forward. It would be better to first try merging the storaged
changes into udisks instead; that way, we share the same code across
all distros, without having to evangelize storaged vs. udisks.

Michael

[Stephen Gallagher]

On 05/13/2016 03:34 PM, Michael Catanzaro wrote:
> On Fri, 2016-05-13 at 13:01 -0400, Stephen Gallagher wrote:
> > CCing the desktop list, but please keep replies on the Fedora devel
> > list.
> >
> > At today's FESCo meeting[1], we raised several questions that we
> > would like to
> > have answered about this change.
> > Have the GNOME, Anaconda/Blivet and Cockpit folks agreed to
> >
> > replacing Udisks2?
> > 2) Is it expected to be a complete cutover (where we drop Udisks2
> > from the
> > default install and/or from the distro entirely
>
> udisks upstream is inactive and has recently expressed some interest in
> allowing the storaged developers to take over maintenance of udisks
> going forward. It would be better to first try merging the storaged
> changes into udisks instead; that way, we share the same code across
> all distros, without having to evangelize storaged vs. udisks.

With respect, the reason storaged exists is because Udisks2 upstream refused to 
accept patches to support LVM. I'm not saying we shouldn't try to merge them 
back together, but the fork happened because upstream was obstinate.

To the best of my knowledge (Tomas Smetana can correct me), storaged is a proper 
superset (and has a more encompassing name), so my suggestion would be to just 
make storaged the official upstream and migrate away from the older Udisks2. 
(With a plan to deprecate the org.freedesktop.Udisks2 interface in 2-3 years in 
favor of the org.storaged.Storaged interface).

[Michael Catanzaro]

On Fri, 2016-05-13 at 15:49 -0400, Stephen Gallagher wrote:
> With respect, the reason storaged exists is because Udisks2 upstream
> refused to 
> accept patches to support LVM. I'm not saying we shouldn't try to
> merge them 
> back together, but the fork happened because upstream was obstinate.

The old maintainer quit a few years ago, unfortunately. The new
maintainer, Martin Pitt, is just keeping the lights on in the absence
of an active maintainer, and has recently indicated that he's open to
the storaged folks taking over udisks. It would be good if the storaged
folks would work with Martin and seriously consider this option before
moving ahead.

Michael

[Tomáš Smetana]

Hello.

On Fri, 13 May 2016 15:49:45 -0400
Stephen Gallagher  wrote:
> On 05/13/2016 03:34 PM, Michael Catanzaro wrote:
> > On Fri, 2016-05-13 at 13:01 -0400, Stephen Gallagher wrote:
> > > CCing the desktop list, but please keep replies on the Fedora devel
> > > list.
> > >
> > > At today's FESCo meeting[1], we raised several questions that we
> > > would like to
> > > have answered about this change.
> > > Have the GNOME, Anaconda/Blivet and Cockpit folks agreed to
> > >
> > > replacing Udisks2?

As for Gnome: I think this has been sorted out -- as long as we don't break
anything, nobody seems to have strong opinions about udisks2 vs storaged.

Anaconda/Blivet has never been the udisks2 API consumer: the storaged
dependency has been added for iSCSI support that was never present in udisks2.
> > > Is it expected to be a complete cutover (where we drop Udisks2
> > >
> > > from the
> > > default install and/or from the distro entirely

The plan for storaged is to provide complete udisks2 API drop-in replacement:
the dependent components should not notice. Once there is nothing requiring
udisks2 itself there is no need to keep it in the distribution...
> > udisks upstream is inactive and has recently expressed some interest in
> > allowing the storaged developers to take over maintenance of udisks
> > going forward. It would be better to first try merging the storaged
> > changes into udisks instead; that way, we share the same code across
> > all distros, without having to evangelize storaged vs. udisks.
>
> With respect, the reason storaged exists is because Udisks2 upstream
> refused to accept patches to support LVM. I'm not saying we shouldn't try
> to merge them back together, but the fork happened because upstream was
> obstinate.

Merging the code and joining forces would be of course the best way forward.
I wouldn't care about the name of the result.
> To the best of my knowledge (Tomas Smetana can correct me), storaged is a
> proper superset (and has a more encompassing name), so my suggestion would
> be to just make storaged the official upstream and migrate away from the
> older Udisks2. (With a plan to deprecate the org.freedesktop.Udisks2
> interface in 2-3 years in favor of the org.storaged.Storaged interface).

There is no plan to deprecate the org.freedesktop.Udisks2 interface yet. And
if the two projects merge again it's perhaps better to keep the old naming so
the API consumers don't have to suffer unnecessary changes. The more
important tasks for storaged would be improving scalability, adding more
automated tests (especially for the new stuff), packaging for Debian/Ubuntu,
etc...

Regards,

[Jan Kurik]

On Mon, May 16, 2016 at 1:05 PM, Tomáš Smetana  wrote:
> Hello.
>
> On Fri, 13 May 2016 15:49:45 -0400
> Stephen Gallagher  wrote:
> > On 05/13/2016 03:34 PM, Michael Catanzaro wrote:
> > > On Fri, 2016-05-13 at 13:01 -0400, Stephen Gallagher wrote:
> > > > CCing the desktop list, but please keep replies on the Fedora devel
> > > > list.
> > > >
> > > > At today's FESCo meeting[1], we raised several questions that we
> > > > would like to
> > > > have answered about this change.
> > > > Have the GNOME, Anaconda/Blivet and Cockpit folks agreed to
> > > >
> > > > replacing Udisks2?
>
> As for Gnome: I think this has been sorted out -- as long as we don't break
> anything, nobody seems to have strong opinions about udisks2 vs storaged.
>
> Anaconda/Blivet has never been the udisks2 API consumer: the storaged
> dependency has been added for iSCSI support that was never present in udisks2.
> > > > Is it expected to be a complete cutover (where we drop Udisks2
> > > >
> > > > from the
> > > > default install and/or from the distro entirely
>
> The plan for storaged is to provide complete udisks2 API drop-in replacement:
> the dependent components should not notice. Once there is nothing requiring
> udisks2 itself there is no need to keep it in the distribution...
> > > udisks upstream is inactive and has recently expressed some interest in
> > > allowing the storaged developers to take over maintenance of udisks
> > > going forward. It would be better to first try merging the storaged
> > > changes into udisks instead; that way, we share the same code across
> > > all distros, without having to evangelize storaged vs. udisks.
> >
> > With respect, the reason storaged exists is because Udisks2 upstream
> > refused to accept patches to support LVM. I'm not saying we shouldn't try
> > to merge them back together, but the fork happened because upstream was
> > obstinate.
>
> Merging the code and joining forces would be of course the best way forward.
> I wouldn't care about the name of the result.
> > To the best of my knowledge (Tomas Smetana can correct me), storaged is a
> > proper superset (and has a more encompassing name), so my suggestion would
> > be to just make storaged the official upstream and migrate away from the
> > older Udisks2. (With a plan to deprecate the org.freedesktop.Udisks2
> > interface in 2-3 years in favor of the org.storaged.Storaged interface).
>
> There is no plan to deprecate the org.freedesktop.Udisks2 interface yet. And
> if the two projects merge again it's perhaps better to keep the old naming so
> the API consumers don't have to suffer unnecessary changes. The more
> important tasks for storaged would be improving scalability, adding more
> automated tests (especially for the new stuff), packaging for Debian/Ubuntu,
> etc...
>
> Regards,

This Change has been turned to System-Wide Change [1].

[1] https://fedorahosted.org/fesco/ticket/1568#comment:13

Regards,
Jan
> --
> Tomáš Smetana
> Platform Engineering, Red Hat
> --
> devel mailing list
> [e-mail]
> http://lists.fedoraproject.org/admin/lists/[e-mail]