FYI https://fedoraproject.org/wiki/Changes/CGroupsV2

[Daniel Walsh]

I have opened a Change Request to change the defaults for Fedora 31 to
Cgroups V2.  I am looking for what packages will be affected by this
change.   Basically any package that adjusts Cgroups via the CgroupFS,
my understanding is working the the systemd APIs, you should be fine.

Packages That I know will be affected:

Container Tools

* Runc, Podman, Buildah, Kubernetes, Docker, Moby-engine

Virtualization tools

* Libvirt

Java

* jvm

Install

* Anaconda

Anyone know of other packages?

Hi Dan,

How the Anaconda will be affected? I'm not aware of any cgroups control
from Anaconda side or do we need change something to the installed
system to enable v2 cgroups?

Thanks for answers,
Jirka

On Thu, 2019-02-14 at 08:10 -0500, Daniel Walsh wrote:
> I have opened a Change Request to change the defaults for Fedora 31
> to
> Cgroups V2.  I am looking for what packages will be affected by this
> change.   Basically any package that adjusts Cgroups via the
> CgroupFS,
> my understanding is working the the systemd APIs, you should be fine.
>
> Packages That I know will be affected:
>
> Container Tools
> Runc, Podman, Buildah, Kubernetes, Docker, Moby-engine
>
> Virtualization tools
> Libvirt
>
> Java
> jvm
>
> Install
> Anaconda
>
> Anyone know of other packages?
>
> devel mailing list -- [e-mail]
> To unsubscribe send an email to [e-mail]
> Fedora Code of Conduct: https://getfedora.org/code-of-conduct.html
> List Guidelines: 
> https://fedoraproject.org/wiki/Mailing_list_guidelines
> List Archives: 
> https://lists.fedoraproject.org/archives/list/[e-mail]

[Lennart Poettering]

On Do, 14.02.19 15:55, [e-mail] ([e-mail]) wrote:
> Hi Dan,
>
> How the Anaconda will be affected? I'm not aware of any cgroups control
> from Anaconda side or do we need change something to the installed
> system to enable v2 cgroups?

The feature page suggests the installer would switch from cgroupsv1 to
cgroupsv2.

My suggestion would instead be to simply update the systemd RPM to
pass slightly different params to meson, to make cgroupsv2 the new
default.

Lennart

--
Lennart Poettering, Red Hat

[Daniel Walsh]

On 2/14/19 9:55 AM, [e-mail] wrote:
> Hi Dan,
>
> How the Anaconda will be affected? I'm not aware of any cgroups control
> from Anaconda side or do we need change something to the installed
> system to enable v2 cgroups?
>
> Thanks for answers,
> Jirka

It will need to change the default kernel option to enable it.
> On Thu, 2019-02-14 at 08:10 -0500, Daniel Walsh wrote:
> > I have opened a Change Request to change the defaults for Fedora 31
> > to
> > Cgroups V2.  I am looking for what packages will be affected by this
> > change.   Basically any package that adjusts Cgroups via the
> > CgroupFS,
> > my understanding is working the the systemd APIs, you should be fine.
> >
> > Packages That I know will be affected:
> >
> > Container Tools
> > Runc, Podman, Buildah, Kubernetes, Docker, Moby-engine
> >
> > Virtualization tools
> > Libvirt
> >
> > Java
> > jvm
> >
> > Install
> > Anaconda
> >
> > Anyone know of other packages?
> >
> > devel mailing list -- [e-mail]
> > To unsubscribe send an email to [e-mail]
> > Fedora Code of Conduct: https://getfedora.org/code-of-conduct.html
> > List Guidelines: 
> > https://fedoraproject.org/wiki/Mailing_list_guidelines
> > List Archives: 
> > https://lists.fedoraproject.org/archives/list/[e-mail]

[Daniel Walsh]

Ok I guess this will not effect anaconda, just affect systemd.

On 2/14/19 1:58 PM, Daniel Walsh wrote:
> On 2/14/19 9:55 AM, [e-mail] wrote:
> > Hi Dan,
> >
> > How the Anaconda will be affected? I'm not aware of any cgroups control
> > from Anaconda side or do we need change something to the installed
> > system to enable v2 cgroups?
> >
> > Thanks for answers,
> > Jirka
>
> It will need to change the default kernel option to enable it.
> > On Thu, 2019-02-14 at 08:10 -0500, Daniel Walsh wrote:
> > > I have opened a Change Request to change the defaults for Fedora 31
> > > to
> > > Cgroups V2.  I am looking for what packages will be affected by this
> > > change.   Basically any package that adjusts Cgroups via the
> > > CgroupFS,
> > > my understanding is working the the systemd APIs, you should be fine.
> > >
> > > Packages That I know will be affected:
> > >
> > > Container Tools
> > > Runc, Podman, Buildah, Kubernetes, Docker, Moby-engine
> > >
> > > Virtualization tools
> > > Libvirt
> > >
> > > Java
> > > jvm
> > >
> > > Install
> > > Anaconda
> > >
> > > Anyone know of other packages?
> > >
> > > devel mailing list -- [e-mail]
> > > To unsubscribe send an email to [e-mail]
> > > Fedora Code of Conduct: https://getfedora.org/code-of-conduct.html
> > > List Guidelines: 
> > > https://fedoraproject.org/wiki/Mailing_list_guidelines
> > > List Archives: 
> > > https://lists.fedoraproject.org/archives/list/[e-mail]
>
> devel mailing list -- [e-mail]
> To unsubscribe send an email to [e-mail]
> Fedora Code of Conduct: https://getfedora.org/code-of-conduct.html
> List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
> List Archives: https://lists.fedoraproject.org/archives/list/[e-mail]

OK, thanks Dan and Lennart for explanation.

I also think it would be better to make this happen in the systemd RPM.
It feel more suitable place for these kind of settings.

Jirka

On Thu, 2019-02-14 at 13:59 -0500, Daniel Walsh wrote:
> Ok I guess this will not effect anaconda, just affect systemd.
>
> On 2/14/19 1:58 PM, Daniel Walsh wrote:
> > On 2/14/19 9:55 AM, [e-mail] wrote:
> > > Hi Dan,
> > >
> > > How the Anaconda will be affected? I'm not aware of any cgroups
> > > control
> > > from Anaconda side or do we need change something to the
> > > installed
> > > system to enable v2 cgroups?
> > >
> > > Thanks for answers,
> > > Jirka
> >
> > It will need to change the default kernel option to enable it.
> > > On Thu, 2019-02-14 at 08:10 -0500, Daniel Walsh wrote:
> > > > I have opened a Change Request to change the defaults for
> > > > Fedora 31
> > > > to
> > > > Cgroups V2.  I am looking for what packages will be affected by
> > > > this
> > > > change.   Basically any package that adjusts Cgroups via the
> > > > CgroupFS,
> > > > my understanding is working the the systemd APIs, you should be
> > > > fine.
> > > >
> > > > Packages That I know will be affected:
> > > >
> > > > Container Tools
> > > > Runc, Podman, Buildah, Kubernetes, Docker, Moby-engine
> > > >
> > > > Virtualization tools
> > > > Libvirt
> > > >
> > > > Java
> > > > jvm
> > > >
> > > > Install
> > > > Anaconda
> > > >
> > > > Anyone know of other packages?
> > > >
> > > > devel mailing list -- [e-mail]
> > > > To unsubscribe send an email to 
> > > > [e-mail]
> > > > Fedora Code of Conduct: 
> > > > https://getfedora.org/code-of-conduct.html
> > > > List Guidelines: 
> > > > https://fedoraproject.org/wiki/Mailing_list_guidelines
> > > > List Archives: 
> > > > https://lists.fedoraproject.org/archives/list/[e-mail]
> >
> > devel mailing list -- [e-mail]
> > To unsubscribe send an email to [e-mail]
> > Fedora Code of Conduct: https://getfedora.org/code-of-conduct.html
> > List Guidelines: 
> > https://fedoraproject.org/wiki/Mailing_list_guidelines
> > List Archives: 
> > https://lists.fedoraproject.org/archives/list/[e-mail]
>
> devel mailing list -- [e-mail]
> To unsubscribe send an email to [e-mail]
> Fedora Code of Conduct: https://getfedora.org/code-of-conduct.html
> List Guidelines: 
> https://fedoraproject.org/wiki/Mailing_list_guidelines
> List Archives: 
> https://lists.fedoraproject.org/archives/list/[e-mail]

[Zbigniew Jędrzejewski-Szmek]

On Thu, Feb 14, 2019 at 08:10:09AM -0500, Daniel Walsh wrote:
> I have opened a Change Request to change the defaults for Fedora 31 to
> Cgroups V2.  I am looking for what packages will be affected by this
> change.   Basically any package that adjusts Cgroups via the CgroupFS,
> my understanding is working the the systemd APIs, you should be fine.
>
> Packages That I know will be affected:
>
> Container Tools
> Runc, Podman, Buildah, Kubernetes, Docker, Moby-engine
>
> Virtualization tools
> Libvirt
>
> Java
> jvm
>
> Install
> Anaconda
>
> Anyone know of other packages?

Great news!

Can you briefly summarize the status / plans / timelines for the
affected components?
(Apart from anaconda, which doesn't need changes, we'll just flip the
default in systemd, as discussed in the other subthread.)

Zbyszek

[Neal Gompa]

On Fri, Feb 15, 2019 at 4:11 AM Zbigniew Jędrzejewski-Szmek
 wrote:
> On Thu, Feb 14, 2019 at 08:10:09AM -0500, Daniel Walsh wrote:
> > I have opened a Change Request to change the defaults for Fedora 31 to
> > Cgroups V2.  I am looking for what packages will be affected by this
> > change.   Basically any package that adjusts Cgroups via the CgroupFS,
> > my understanding is working the the systemd APIs, you should be fine.
> >
> > Packages That I know will be affected:
> >
> > Container Tools
> > Runc, Podman, Buildah, Kubernetes, Docker, Moby-engine
> >
> > Virtualization tools
> > Libvirt
> >
> > Java
> > jvm
> >
> > Install
> > Anaconda
> >
> > Anyone know of other packages?
>
> Great news!
>
> Can you briefly summarize the status / plans / timelines for the
> affected components?
> (Apart from anaconda, which doesn't need changes, we'll just flip the
> default in systemd, as discussed in the other subthread.)

snapd will break in cgroupv2 mode:
https://bugzilla.redhat.com/show_bug.cgi?id=1438079

I don't know of anyone working on fixing that...

[Zygmunt Krynicki]

> Wiadomość napisana przez Neal Gompa  w dniu 15.02.2019, o godz. 12:51:
>
> On Fri, Feb 15, 2019 at 4:11 AM Zbigniew Jędrzejewski-Szmek
> <[e-mail] > wrote:
> > On Thu, Feb 14, 2019 at 08:10:09AM -0500, Daniel Walsh wrote:
> > > I have opened a Change Request to change the defaults for Fedora 31 to
> > > Cgroups V2.  I am looking for what packages will be affected by this
> > > change.   Basically any package that adjusts Cgroups via the CgroupFS,
> > > my understanding is working the the systemd APIs, you should be fine.
> > >
> > > Packages That I know will be affected:
> > >
> > > Container Tools
> > > Runc, Podman, Buildah, Kubernetes, Docker, Moby-engine
> > >
> > > Virtualization tools
> > > Libvirt
> > >
> > > Java
> > > jvm
> > >
> > > Install
> > > Anaconda
> > >
> > > Anyone know of other packages?
> >
> > Great news!
> >
> > Can you briefly summarize the status / plans / timelines for the
> > affected components?
> > (Apart from anaconda, which doesn't need changes, we'll just flip the
> > default in systemd, as discussed in the other subthread.)
>
> snapd will break in cgroupv2 mode:
> https://bugzilla.redhat.com/show_bug.cgi?id=1438079 https://bugzilla.redhat.com/show_bug.cgi?id=1438079
>
> I don't know of anyone working on fixing that…

I think the stance of the snapd team is that once v2 supports freezer we can have a look at adding the support.

There’s a snapd bug tracking this as well: https://bugs.launchpad.net/snapd/+bug/1801664 https://bugs.launchpad.net/snapd/+bug/1801664

I’m happy to work on this issue once it becomes „pressing” and once the prerequisites are available. If F30 disables v1 entirely and has a kernel where we can get device, freezer and pid controllers then it’s „just” a matter of coding.

Best regards
ZK

[Lennart Poettering]

On Fr, 15.02.19 12:55, Zygmunt Krynicki ([e-mail]) wrote:
> I’m happy to work on this issue once it becomes „pressing” and once
> the prerequisites are available. If F30 disables v1 entirely and has
> a kernel where we can get device, freezer and pid controllers then
> it’s „just” a matter of coding.

The kernel will support both modes for a long time I figure. And so
will systemd. You can switch between both modes at boot time through a
kernel cmdline option, and the change for F30 would be to just default
to a different setting.

Note that in cgroupsv2 mode systemd will not mount the cgroupv1
hierchies at all. (You can mount them yourself if you want to
though. For example, systemd-nspawn can optionally provide cgroupsv1
to container payloads on cgroupsv2 hosts, by simply mounting the
name=systemd hierarchy into the container anyway)

The "devices" cgroup controller is generally not available on
cgroupsv2. However, there's now a set of bpf hook-ups that you can use
instead and provide pretty much equivalent functionality. (systemd
supports them already).

The "freezer" cgroup controller is not available yet on cgroupsv2. But
this is likely going to change soon, but it will be core cgroupsv2
functionality, not a controller of its own. Until the freezer becomes
available it should be completely fine to simply use SIGSTOP instead,
semantics are not thaaaaat different.

The "pids" controllers has been available on cgroupsv2 since day one
basically, at the same day it was introduced for cgroupsv1.

Lennart

--
Lennart Poettering, Red Hat

[Zygmunt Krynicki]

> Wiadomość napisana przez Lennart Poettering  w dniu 15.02.2019, o godz. 13:02:
>
> On Fr, 15.02.19 12:55, Zygmunt Krynicki ([e-mail]) wrote:
> > I’m happy to work on this issue once it becomes „pressing” and once
> > the prerequisites are available. If F30 disables v1 entirely and has
> > a kernel where we can get device, freezer and pid controllers then
> > it’s „just” a matter of coding.
>
> The kernel will support both modes for a long time I figure. And so
> will systemd. You can switch between both modes at boot time through a
> kernel cmdline option, and the change for F30 would be to just default
> to a different setting.
>
> Note that in cgroupsv2 mode systemd will not mount the cgroupv1
> hierchies at all. (You can mount them yourself if you want to
> though. For example, systemd-nspawn can optionally provide cgroupsv1
> to container payloads on cgroupsv2 hosts, by simply mounting the
> name=systemd hierarchy into the container anyway)
>
> The "devices" cgroup controller is generally not available on
> cgroupsv2. However, there's now a set of bpf hook-ups that you can use
> instead and provide pretty much equivalent functionality. (systemd
> supports them already).

Indeed, the is a possible way out. It requires some thought on our side to integrate with our current use of v1 devices cgroup, udev rules, snapd side „hot plug” and live changes to running programs (which v1 devices allowed).
> The "freezer" cgroup controller is not available yet on cgroupsv2. But
> this is likely going to change soon, but it will be core cgroupsv2
> functionality, not a controller of its own. Until the freezer becomes
> available it should be completely fine to simply use SIGSTOP instead,
> semantics are not thaaaaat different.

We use the freezer for „snap scope” process enumeration (but there are other ways to do that) and to crucially, stop processes while we perform some mount namespace updates, so that there’s less risk of apps attacking the mount code with racing symlinks and what not. Using SIGSTOP for that is, I guess, okay, as long as we can „win” and stop all processes in a given snap reliably enough.
> The "pids" controllers has been available on cgroupsv2 since day one
> basically, at the same day it was introduced for cgroupsv1.

Yeah. I’m trying to move some of the burden from the freezer to pid so that snapd doesn’t rely on the freezer that much.
> Lennart
>
> --
> Lennart Poettering, Red Hat
> _______________________________________________
> devel mailing list -- [e-mail]
> To unsubscribe send an email to [e-mail]
> Fedora Code of Conduct: https://getfedora.org/code-of-conduct.html
> List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
> List Archives: https://lists.fedoraproject.org/archives/list/[e-mail]

[Lennart Poettering]

On Fr, 15.02.19 13:42, Zygmunt Krynicki ([e-mail]) wrote:
> > The "devices" cgroup controller is generally not available on
> > cgroupsv2. However, there's now a set of bpf hook-ups that you can use
> > instead and provide pretty much equivalent functionality. (systemd
> > supports them already).
>
> Indeed, the is a possible way out. It requires some thought on our
> side to integrate with our current use of v1 devices cgroup, udev
> rules, snapd side „hot plug” and live changes to running programs
> (which v1 devices allowed).

The bpf devices thing allows live changes too. In fact, in the bpf logic in
systemd it's implemented that way already.
> > The "freezer" cgroup controller is not available yet on cgroupsv2. But
> > this is likely going to change soon, but it will be core cgroupsv2
> > functionality, not a controller of its own. Until the freezer becomes
> > available it should be completely fine to simply use SIGSTOP instead,
> > semantics are not thaaaaat different.
>
> We use the freezer for „snap scope” process enumeration (but there
> are other ways to do that) and to crucially, stop processes while we
> perform some mount namespace updates, so that there’s less risk of
> apps attacking the mount code with racing symlinks and what
> not. Using SIGSTOP for that is, I guess, okay, as long as we can
> „win” and stop all processes in a given snap reliably enough.

Well, the SIGSTOP thing is racy: processes can fork() quicker than you
can pause them. Together with the pids controller you should be fine
though, as you can put a limit on forks.

Lennart

--
Lennart Poettering, Red Hat

[Daniel Walsh]

On 2/15/19 7:02 AM, Lennart Poettering wrote:
> On Fr, 15.02.19 12:55, Zygmunt Krynicki ([e-mail]) wrote:
> > I’m happy to work on this issue once it becomes „pressing” and once
> > the prerequisites are available. If F30 disables v1 entirely and has
> > a kernel where we can get device, freezer and pid controllers then
> > it’s „just” a matter of coding.
>
> The kernel will support both modes for a long time I figure. And so
> will systemd. You can switch between both modes at boot time through a
> kernel cmdline option, and the change for F30 would be to just default
> to a different setting.

This is for F31 NOT F30.
> Note that in cgroupsv2 mode systemd will not mount the cgroupv1
> hierchies at all. (You can mount them yourself if you want to
> though. For example, systemd-nspawn can optionally provide cgroupsv1
> to container payloads on cgroupsv2 hosts, by simply mounting the
> name=systemd hierarchy into the container anyway)
>
> The "devices" cgroup controller is generally not available on
> cgroupsv2. However, there's now a set of bpf hook-ups that you can use
> instead and provide pretty much equivalent functionality. (systemd
> supports them already).

Awesome.
> The "freezer" cgroup controller is not available yet on cgroupsv2. But
> this is likely going to change soon, but it will be core cgroupsv2
> functionality, not a controller of its own. Until the freezer becomes
> available it should be completely fine to simply use SIGSTOP instead,
> semantics are not thaaaaat different.

Yes I was told that it just missed getting into the 5.0 kernel, and
should be in the 5.1 kernel.
> The "pids" controllers has been available on cgroupsv2 since day one
> basically, at the same day it was introduced for cgroupsv1.
>
> Lennart
>
> --
> Lennart Poettering, Red Hat

[Daniel Walsh]

On 2/15/19 6:55 AM, Zygmunt Krynicki wrote:
> > Wiadomość napisana przez Neal Gompa <[e-mail]
> > > w dniu 15.02.2019, o godz. 12:51:
> >
> > On Fri, Feb 15, 2019 at 4:11 AM Zbigniew Jędrzejewski-Szmek
> > <[e-mail] > wrote:
> > > On Thu, Feb 14, 2019 at 08:10:09AM -0500, Daniel Walsh wrote:
> > > > I have opened a Change Request to change the defaults for Fedora 31 to
> > > > Cgroups V2.  I am looking for what packages will be affected by this
> > > > change.   Basically any package that adjusts Cgroups via the CgroupFS,
> > > > my understanding is working the the systemd APIs, you should be fine.
> > > >
> > > > Packages That I know will be affected:
> > > >
> > > > Container Tools
> > > > Runc, Podman, Buildah, Kubernetes, Docker, Moby-engine
> > > >
> > > > Virtualization tools
> > > > Libvirt
> > > >
> > > > Java
> > > > jvm
> > > >
> > > > Install
> > > > Anaconda
> > > >
> > > > Anyone know of other packages?
> > >
> > > Great news!
> > >
> > > Can you briefly summarize the status / plans / timelines for the
> > > affected components?
> > > (Apart from anaconda, which doesn't need changes, we'll just flip the
> > > default in systemd, as discussed in the other subthread.)
> >
> > snapd will break in cgroupv2 mode:
> > https://bugzilla.redhat.com/show_bug.cgi?id=1438079
> >
> > I don't know of anyone working on fixing that…
>
> I think the stance of the snapd team is that once v2 supports freezer
> we can have a look at adding the support.

Freezer is supposed to go into Kernel 5.1, I have been told.
> There’s a snapd bug tracking this as
> well: https://bugs.launchpad.net/snapd/+bug/1801664
>
> I’m happy to work on this issue once it becomes „pressing” and once
> the prerequisites are available. If F30 disables v1 entirely and has a
> kernel where we can get device, freezer and pid controllers then it’s
> „just” a matter of coding.

Device Cgroup is supposed to be done in eBPF, and systemd has an
implementation I believe.

This is for F31 NOT f30.  Also you will be able to boot the machine in
V1 mode. We are just talking about the default here.
> Best regards
> ZK
>
> devel mailing list -- [e-mail]
> To unsubscribe send an email to [e-mail]
> Fedora Code of Conduct: https://getfedora.org/code-of-conduct.html
> List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
> List Archives: https://lists.fedoraproject.org/archives/list/[e-mail]

[Daniel P. Berrangé]

On Thu, Feb 14, 2019 at 08:10:09AM -0500, Daniel Walsh wrote:
> I have opened a Change Request to change the defaults for Fedora 31 to
> Cgroups V2.  I am looking for what packages will be affected by this
> change.   Basically any package that adjusts Cgroups via the CgroupFS,
> my understanding is working the the systemd APIs, you should be fine.
>
> Packages That I know will be affected:
>
> Container Tools
> Runc, Podman, Buildah, Kubernetes, Docker, Moby-engine
>
> Virtualization tools
> Libvirt
>
> Java
> jvm
>
> Install
> Anaconda
>
> Anyone know of other packages?

What is the expectation of compatibility for applications running inside
containers ?

Historically any running containers will see a v1 hierarchy inside their
container. If the host is switched to v2, then presumably applications
inside the container will also see a v2 layout.

The wiki page says tools/scripts would be unaffected if they had used
the systemd interface to access cgroups. This won't apply to tools
or scripts inside the container though as the containers rarely run
systemd inside.

If this is correct, then even if all software in Fedora supports v2,
users might still see breakage of applications they are running in
containers on Fedora.

Admittedly the set of things likely to be affected in this way is probably
fairly small, but it feels like something that should be described in
the change page on the wiki.

Regards,
Daniel

[Daniel Walsh]

On 2/14/19 10:49 AM, Daniel P. Berrangé wrote:
> On Thu, Feb 14, 2019 at 08:10:09AM -0500, Daniel Walsh wrote:
> > I have opened a Change Request to change the defaults for Fedora 31 to
> > Cgroups V2.  I am looking for what packages will be affected by this
> > change.   Basically any package that adjusts Cgroups via the CgroupFS,
> > my understanding is working the the systemd APIs, you should be fine.
> >
> > Packages That I know will be affected:
> >
> > Container Tools
> > Runc, Podman, Buildah, Kubernetes, Docker, Moby-engine
> >
> > Virtualization tools
> > Libvirt
> >
> > Java
> > jvm
> >
> > Install
> > Anaconda
> >
> > Anyone know of other packages?
>
> What is the expectation of compatibility for applications running inside
> containers ?
>
> Historically any running containers will see a v1 hierarchy inside their
> container. If the host is switched to v2, then presumably applications
> inside the container will also see a v2 layout.
>
> The wiki page says tools/scripts would be unaffected if they had used
> the systemd interface to access cgroups. This won't apply to tools
> or scripts inside the container though as the containers rarely run
> systemd inside.
>
> If this is correct, then even if all software in Fedora supports v2,
> users might still see breakage of applications they are running in
> containers on Fedora.
>
> Admittedly the set of things likely to be affected in this way is probably
> fairly small, but it feels like something that should be described in
> the change page on the wiki.
>
> Regards,
> Daniel

Good point.

I will update the Change with a comment on this.