F37 Change Proposal: MAC Address Policy none (System-Wide Change)

[Vipul Siddharth]

This document represents a proposed Change. As part of the Changes
process, proposals are publicly announced in order to receive
community feedback. This proposal will only be implemented if approved
by the Fedora Engineering Steering Committee.


== Summary ==

The `systemd-udev` package installs
`"/usr/lib/systemd/network/99-default.link"`,
which sets `Link.MACAddressPolicy=persistent`. This proposal is to
change it to set `Link.MACAddressPolicy=none` to stop changing the MAC address.
This is particularly important for bridge and bond devices.

This change can either only apply to bridge/bond devices, or to
various software devices. That is to be discussed.

== Owner ==

* Name: [[User:thaller|Thomas Haller]] (NetworkManager)
* Email: <thaller(a)redhat.com&gt;
* Name: [[User:dustymabe| Dusty Mabe]] (Fedora CoreOS)
* Email: <dmabe(a)redhat.com&gt;


== Detailed Description ==

On Fedora, udev by default changes the MAC address of a wide range of
software devices.
This was introduced by systemd 242 in early 2019 (Fedora 31), when
`MACAddressPolicy=` was
extended to affect more types of devices.

With `MACAddressPolicy=persistent` udev's aim is to provide a stable
MAC address, otherwise
the kernel will assign a random one. However, that can cause problems:

Firstly, software devices are always created by some tool that has
plans for the device. The tool may not
expect that udev is going to change the MAC address and races against
that. The best solution
for the tool is to set the MAC address when creating an interface.
This will prevent
udev from changing the MAC address according to the MACAddressPolicy.
Otherwise, the tool should wait for udev to initialize the device to
avoid the race. In theory, a
tool is always advised to wait for udev to initialize the device.
However, if it were not for MACAddressPolicy,
in common scenarios udev doesn't do anything relevant for software
devices to make that necessary.

Secondly, for interface types bridge and bond, an unset MAC address
has a special meaning
to the kernel and the MAC address of the first port is used. If udev
changes the MAC
address, that no longer works.
Now the generated MAC address is not directly discoverable as it is
based on `/etc/machine-id`
([https://www.man7.org/linux/man-pages/man5/machine-id.5.html
machine-id(5)]), among
other data. Even if there were a tool to easily calculate the MAC
address, it could be cumbersome
to use it without logging into the machine first. The MAC address can
directly affect the
assigned IP address, for example when using DHCP. When booting a new
virtual machine, the user might
know the MAC address of the (virtual) "physical" interfaces. When
bonding/bridging those
interfaces, the bond/bridge would get one of the well known MAC
addresses. `MACAddressPolicy=persistent`
interferes with that.

The goal of persistent policy is to provide a stable MAC address. Note
that if the
tool or user who created the interface would want a certain MAC address, they
have all the means to set it already. That applies regardless whether
the tool is
iproute2, NetworkManager, systemd-networkd. Neither NetworkManager nor
systemd-networkd
rely on udev's MACAddressPolicy for setting the MAC address. This
behavior is mostly
useful for plain `ip link add`, but it's unclear which real world user
wants this behavior.

Of course, the user is welcome to configure the MAC address in any way
they want. Including,
dropping a link file that sets `MACAddressPolicy=persistent`. The
problem is once udev sets a MAC address,
it cannot be unset. Which makes this problematic to do by default.

While Fedora inherited this behavior from upstream systemd, RHEL-9
does not follow this behavior
([https://gitlab.com/redhat/centos-stream/rpms/systemd/-/blob/c8953519504bf...
centos9],
[https://bugzilla.redhat.com/show_bug.cgi?id=1921094 rh#1921094]). For
RHEL-8, this doesn't
apply because the systemd there is too old to change the MAC address
of most software devices.

This could be either implemented by patching
`/usr/lib/systemd/network/99-default.link`
to have a different policy, or by dropping a link file with higher
priority. In the latter
case, that override could be shipped either by udev or even by NetworkManager.

Another option is to change the scope of this proposal to only change to
`MACAddressPolicy=none` for the device types where this causes the most issues
(bridge/bond/team).


== Feedback ==

This was also discussed on upstream systemd mailing list
[https://lists.freedesktop.org/archives/systemd-devel/2022-May/047893.html
here].
The upstream systemd maintainers' opinion is that the current udev
behavior is desirable, but accepts that distributions (or network
stacks such as NetworkManager) may choose to change the default
depending on their needs
([https://lists.freedesktop.org/archives/systemd-devel/2022-May/047943.html
reference]).
The goal here is to find out what the Fedora community thinks is the
most appropriate default.

The RHEL-9 bug is [https://bugzilla.redhat.com/show_bug.cgi?id=1921094
[rh#1921094]].


== Benefit to Fedora ==

Pros:

- Consistent behavior with RHEL8 and RHEL9.

- Avoid race of udev and the tool that creates the interface.

- Bridge and bond interfaces can get the MAC addresses from their first port.

In the case of `MACAddressPolicy=none` for a bond (or bridge) the bond will
get a MAC related to one of its physical NIC devices. In the case of
provisioning
new systems (especially in a large datacenter) information about the server
can be used to configure the network environment (DHCP, Firewall, etc) before
the server is ever even powered on. This does mean that you may have to update
your network environment configuration if you replace a NIC (con), but that you
won't have to update your network environment configuration if you re-install
your server (pro), which is what you'd have to do now with
`MACAddressPolicy=persistent`.

Cons:

- Deviate from upstream systemd.

It is desirable that RHEL and Fedora behaves similar. A possible outcome
could be the current behavior stays and RHEL 10 would change behavior. On the
other hand, different distributions (or even Fedora spins) have different
uses and needs. Deviating might be fine. In the same vein, there is also
a desire to stay close to upstream systemd behavior. But the uses of systemd
project go beyond Fedora/RHEL, so deviating here may also be fine.


== Scope ==

* Proposal owners:
The main goal of this request is to generate productive discussion and
find the desired behavior.
The implementation/changes are either way very simple.

* Other developers:
Other projects that wish a certain MAC address are welcome to
set it for their devices. Including using udev's MACAddressPolicy.

* Release engineering:
Not needed for this change.

* Policies and guidelines: N/A (not needed for this Change)
* Trademark approval: N/A (not needed for this Change)
* Alignment with Objectives:

== Upgrade/compatibility impact ==

After the change, the MAC address for the affected device types changes.


== How To Test ==

1) Create a software device two times, for example `ip link add type
bridge`. Note
that the MAC address is either stable or random, depending on the
`MACAddressPolicy=`.

2) Note that if the software device has the MAC address set initially,
udev does not
change it (`ip link add address aa:aa:aa:aa:aa:aa type bridge`). That depends on
`/sys/class/net/$dev/addr_assign_type`.

3) Create a bridge/bond interface without setting the MAC address.
Note that if `MACAddressPolicy=none`,
the MAC address is random at first. Note that attaching the first port
will update the controller's MAC address.
On the other hand, with `MACAddressPolicy=persistent`, the MAC address
of the controller is fixed
and not inherited from the port.

4) Run

  ip monitor link &
  while : ; do
    ip link del xxx
    ip link add name xxx type dummy \
    && ip link set xxx addr aa:00:00:00:00:00 \
    && ip link show xxx | grep -q aa:00:00:00:00:00 \
    || break
  done

to reproduce the race between a simple tool and udev changing the MAC address.


== User Experience ==

Bond/bridge devices would again get assigned the MAC address of the
first NIC added to the device.
If we chose to not limit the scope of this change to just
bonds/bridges then all software devices would get randomly assigned
MAC addresses.


== Dependencies ==

None.


== Contingency Plan ==

If the change is rejected, nothing needs to be done. The change
itself will be simple to implement.
Contingency deadline: beta freeze
Blocks release? No


== Documentation ==

TODO.

== Release Notes ==

-- 
Vipul Siddharth
He/His/Him
FPgM team member

[Zbigniew Jędrzejewski-Szmek]

On Sun, Jun 26, 2022 at 12:36:14AM +0530, Vipul Siddharth wrote:
>  This document represents a proposed Change. As part of the Changes
>  process, proposals are publicly announced in order to receive
>  community feedback. This proposal will only be implemented if approved
>  by the Fedora Engineering Steering Committee.
>  
>  
>  == Summary ==
>  
>  The `systemd-udev` package installs
>  `"/usr/lib/systemd/network/99-default.link"`,
>  which sets `Link.MACAddressPolicy=persistent`. This proposal is to
>  change it to set `Link.MACAddressPolicy=none` to stop changing the MAC address.
>  This is particularly important for bridge and bond devices.
>  
>  This change can either only apply to bridge/bond devices, or to
>  various software devices. That is to be discussed. 
Hi!

I already participated in the upstream discussion, so what I write
here will be a restatement to some extent, but with a look from the
side of Fedora specifically.

The proposal has two variants: 1. just changing the policy to MACAddressPolicy=none
or 2. limiting the change to bridge and bond devices.

Re variant 1: MACAddressPolicy=persistent applies to all devices that don't have
a hardware address. The proposal as written (blanket MACAddressPolicy=none)
would change behaviour for all kinds of devices, incl. e.g. software
devices like veths, and cheap hardware devices without a fixed MAC.
The proposal doesn't provide any justification for this (except for
simplicity of implementation) and this variant seems pretty bad and
I'm strongly opposed.

Re variant 2: the proposal limited to brige/bond devices seems much more
reasonable. In particular, the case described below of a server (virtualized
or not) in a big datacenter is the one case where the benefits of
MACAddressPolicy=none are clearly visible. I still don't think it's worth
changing the default, but here the cost:benefit ratio is much closer.

>  == Benefit to Fedora ==
>  
>  Pros:
>  
>  - Consistent behavior with RHEL8 and RHEL9.
>  
>  - Avoid race of udev and the tool that creates the interface. 
The race will happen if the creation is done in a specific way. But at
the same time, even the Change proposal describes how to avoid the
race ('ip link add address aa:aa:aa:aa:aa:aa type bridge'). So the situation
can be summarized as "we have a bunch of 'big' tools that create
devices like NetworkManager or systemd-networkd, which already know or
can be easily fixed to avoid the race, and a manual tool which can be invoked
in a way that avoids the race". Instead of changing the default in udev we
could educate people how to invoke it better.

>  - Bridge and bond interfaces can get the MAC addresses from their
> first port.
>  
>  In the case of `MACAddressPolicy=none` for a bond (or bridge) the bond will
>  get a MAC related to one of its physical NIC devices. In the case of
>  provisioning
>  new systems (especially in a large datacenter) information about the server
>  can be used to configure the network environment (DHCP, Firewall, etc) before
>  the server is ever even powered on. This does mean that you may have to update
>  your network environment configuration if you replace a NIC (con), but that you
>  won't have to update your network environment configuration if you re-install
>  your server (pro), which is what you'd have to do now with
>  `MACAddressPolicy=persistent`. 
Yep. This is *the* case.

>  Cons:
>  
>  - Deviate from upstream systemd. 
It is also important to mention that Fedora will "deviate" from itself
(it's former self). We would be changing a default in place since ~2013 [1].

[1] https://github.com/systemd/systemd/commit/16b9b87aee

>  It is desirable that RHEL and Fedora behaves similar. A possible
> outcome
>  could be the current behavior stays and RHEL 10 would change behavior. On the
>  other hand, different distributions (or even Fedora spins) have different
>  uses and needs. Deviating might be fine. In the same vein, there is also
>  a desire to stay close to upstream systemd behavior. But the uses of systemd
>  project go beyond Fedora/RHEL, so deviating here may also be fine. 
So:
- Variant 1 is not good, variant 2 makes more sense.
- The motivating case for v.2 is the "big datacenter" case and race
  when 'ip' is invoked.
- We could improve the tools: 'ip link' could be taught to wait until
  udev has stopped processing the device, users can be taught to use
  better invocations.
- For "small" users (individual machine admins who just install some
  server and configure it after turning it on and/or swap network
  cards between machines), having stable MAC addresses that doesn't
  depend on the order of device discovery or removal of a single
  network card seems better.
- For "big" users (the datacenter case), changing the policy make sense,
  but at the same time, those folks can just insert a policy override,
  they're most likely using some ansible/puppet/cheffy thingy.
- RHEL is more of the "big" case, Fedora is more the "small" case.
  Sometimes RHEL and Fedora have different defaults, sometimes RHEL
  takes years to follow what Fedora does.

So overall, I think this proposal would make most sense when limited
to specific types of hardware (bridge and bond?), and also to specific
spins: it's a better fit for CoreOS than Workstation…

But I haven't seen discussion of other approaches: would making 'ip'
better be an option? 'udev' already locks some devices when processing
using a simple BSD lock. Should we do the same for network devices
and just teach 'ip' to do the same? (This would be much simpler and
maybe more appealing that teaching it to wait for udev to report
that it's finished).

Zbyszek

[Daniel P. Berrangé]

On Mon, Jun 27, 2022 at 10:10:31AM +0200, Zbigniew Jędrzejewski-Szmek wrote:
>  On Sun, Jun 26, 2022 at 12:36:14AM +0530, Vipul Siddharth wrote:
>  > This document represents a proposed Change. As part of the Changes
>  > process, proposals are publicly announced in order to receive
>  > community feedback. This proposal will only be implemented if approved
>  > by the Fedora Engineering Steering Committee.
>  > 
>  > 
>  > == Summary ==
>  > 
>  > The `systemd-udev` package installs
>  > `"/usr/lib/systemd/network/99-default.link"`,
>  > which sets `Link.MACAddressPolicy=persistent`. This proposal is to
>  > change it to set `Link.MACAddressPolicy=none` to stop changing the MAC address.
>  > This is particularly important for bridge and bond devices.
>  > 
>  > This change can either only apply to bridge/bond devices, or to
>  > various software devices. That is to be discussed.
>  
>  Hi!
>  
>  I already participated in the upstream discussion, so what I write
>  here will be a restatement to some extent, but with a look from the
>  side of Fedora specifically.
>  
>  The proposal has two variants: 1. just changing the policy to MACAddressPolicy=none
>  or 2. limiting the change to bridge and bond devices.
>  
>  Re variant 1: MACAddressPolicy=persistent applies to all devices that don't have
>  a hardware address. The proposal as written (blanket MACAddressPolicy=none)
>  would change behaviour for all kinds of devices, incl. e.g. software
>  devices like veths, and cheap hardware devices without a fixed MAC.
>  The proposal doesn't provide any justification for this (except for
>  simplicity of implementation) and this variant seems pretty bad and
>  I'm strongly opposed.
>  
>  Re variant 2: the proposal limited to brige/bond devices seems much more
>  reasonable. In particular, the case described below of a server (virtualized
>  or not) in a big datacenter is the one case where the benefits of
>  MACAddressPolicy=none are clearly visible. I still don't think it's worth
>  changing the default, but here the cost:benefit ratio is much closer.
>  
>  > == Benefit to Fedora ==
>  > 
>  > Pros:
>  > 
>  > - Consistent behavior with RHEL8 and RHEL9.
>  > 
>  > - Avoid race of udev and the tool that creates the interface.
>  
>  The race will happen if the creation is done in a specific way. But at
>  the same time, even the Change proposal describes how to avoid the
>  race ('ip link add address aa:aa:aa:aa:aa:aa type bridge'). So the situation
>  can be summarized as "we have a bunch of 'big' tools that create
>  devices like NetworkManager or systemd-networkd, which already know or
>  can be easily fixed to avoid the race, and a manual tool which can be invoked
>  in a way that avoids the race". Instead of changing the default in udev we
>  could educate people how to invoke it better. 
This comes across as blaming every networking tool out there for
having their previously correct & working behaviour be broken by
a systemd change imposing new requirements on them.

Are there any notable tools which actually follow the usage pattern
described, or are we in effect having to fix *everything*, including
an uncountable amount of documentation, blogs, examples, most of
which will prove impossible to fix in practice.

>  > - Bridge and bond interfaces can get the MAC addresses from
> their first port.
>  > 
>  > In the case of `MACAddressPolicy=none` for a bond (or bridge) the bond will
>  > get a MAC related to one of its physical NIC devices. In the case of
>  > provisioning
>  > new systems (especially in a large datacenter) information about the server
>  > can be used to configure the network environment (DHCP, Firewall, etc) before
>  > the server is ever even powered on. This does mean that you may have to update
>  > your network environment configuration if you replace a NIC (con), but that you
>  > won't have to update your network environment configuration if you re-install
>  > your server (pro), which is what you'd have to do now with
>  > `MACAddressPolicy=persistent`.
>  
>  Yep. This is *the* case. 
Having the bridge/bond get the MAC addr of tht first physical NIC is pretty
compelling from an automation POV, especially when the virtualziation host is
doing traffic filtering for the VM. When a mgmt app assigns a MAC & IP pair to
a guest, it is not uncommon to apply firewall rules providing anti-spoofing
protection for MAC/IP, such that the VM cannot send traffic with other MAC/IP
addresses.  Libvirt provides this functionality with its 'clean-traffic'
network filter, and other virt/cloud mgmt apps do similar.


>  > Cons:
>  > 
>  > - Deviate from upstream systemd. 
This is disappointing, as a great benefit of systemd is the level of
consistency it brought to distro behaviour, and I think this change
would be useful to all distros (in the context of bridge/bond devices)

>  It is also important to mention that Fedora will "deviate"
> from itself
>  (it's former self). We would be changing a default in place since ~2013 [1].
>
>  [1] https://github.com/systemd/systemd/commit/16b9b87aee 
Yes, that is quite unfortunate.

>  > It is desirable that RHEL and Fedora behaves similar. A possible
> outcome
>  > could be the current behavior stays and RHEL 10 would change behavior. On the
>  > other hand, different distributions (or even Fedora spins) have different
>  > uses and needs. Deviating might be fine. In the same vein, there is also
>  > a desire to stay close to upstream systemd behavior. But the uses of systemd
>  > project go beyond Fedora/RHEL, so deviating here may also be fine.
>  
>  So:
>  - Variant 1 is not good, variant 2 makes more sense.
>  - The motivating case for v.2 is the "big datacenter" case and race
>    when 'ip' is invoked. 
IMHO The more compelling reason is compliance with MAC address filtering
applied to VMs.

>  - We could improve the tools: 'ip link' could be taught to
> wait until
>    udev has stopped processing the device, users can be taught to use
>    better invocations. 
Teaching users is a impractical solution IMHO. The 'ip' tool already
has poor usability/learnability IME, such that getting users to learn
differences is pretty challenging, plus there's way too much docs out
there in the wild that will never be updatable.

>  - For "small" users (individual machine admins who just
> install some
>    server and configure it after turning it on and/or swap network
>    cards between machines), having stable MAC addresses that doesn't
>    depend on the order of device discovery or removal of a single
>    network card seems better.
>  - For "big" users (the datacenter case), changing the policy make sense,
>    but at the same time, those folks can just insert a policy override,
>    they're most likely using some ansible/puppet/cheffy thingy.
>  - RHEL is more of the "big" case, Fedora is more the "small" case.
>    Sometimes RHEL and Fedora have different defaults, sometimes RHEL
>    takes years to follow what Fedora does.
>  
>  So overall, I think this proposal would make most sense when limited
>  to specific types of hardware (bridge and bond?), and also to specific
>  spins: it's a better fit for CoreOS than Workstation… 
This doesn't feel like a spin-specific problem to me, given it appears
useful to both host & guest scenarios alike, and nested VM means
host+guest are often the same thing.

>  But I haven't seen discussion of other approaches: would making
> 'ip'
>  better be an option? 'udev' already locks some devices when processing
>  using a simple BSD lock. Should we do the same for network devices
>  and just teach 'ip' to do the same? (This would be much simpler and
>  maybe more appealing that teaching it to wait for udev to report
>  that it's finished). 

With regards,
Daniel
-- 
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|

[Zbigniew Jędrzejewski-Szmek]

On Mon, Jun 27, 2022 at 09:40:53AM +0100, Daniel P. Berrangé wrote:
>  On Mon, Jun 27, 2022 at 10:10:31AM +0200, Zbigniew Jędrzejewski-Szmek
> wrote:
>  > On Sun, Jun 26, 2022 at 12:36:14AM +0530, Vipul Siddharth wrote:
>  > > This document represents a proposed Change. As part of the Changes
>  > > process, proposals are publicly announced in order to receive
>  > > community feedback. This proposal will only be implemented if approved
>  > > by the Fedora Engineering Steering Committee.
>  > > 
>  > > 
>  > > == Summary ==
>  > > 
>  > > The `systemd-udev` package installs
>  > > `"/usr/lib/systemd/network/99-default.link"`,
>  > > which sets `Link.MACAddressPolicy=persistent`. This proposal is to
>  > > change it to set `Link.MACAddressPolicy=none` to stop changing the MAC
> address.
>  > > This is particularly important for bridge and bond devices.
>  > > 
>  > > This change can either only apply to bridge/bond devices, or to
>  > > various software devices. That is to be discussed.
>  > 
>  > Hi!
>  > 
>  > I already participated in the upstream discussion, so what I write
>  > here will be a restatement to some extent, but with a look from the
>  > side of Fedora specifically.
>  > 
>  > The proposal has two variants: 1. just changing the policy to MACAddressPolicy=none
>  > or 2. limiting the change to bridge and bond devices.
>  > 
>  > Re variant 1: MACAddressPolicy=persistent applies to all devices that don't
> have
>  > a hardware address. The proposal as written (blanket MACAddressPolicy=none)
>  > would change behaviour for all kinds of devices, incl. e.g. software
>  > devices like veths, and cheap hardware devices without a fixed MAC.
>  > The proposal doesn't provide any justification for this (except for
>  > simplicity of implementation) and this variant seems pretty bad and
>  > I'm strongly opposed.
>  > 
>  > Re variant 2: the proposal limited to brige/bond devices seems much more
>  > reasonable. In particular, the case described below of a server (virtualized
>  > or not) in a big datacenter is the one case where the benefits of
>  > MACAddressPolicy=none are clearly visible. I still don't think it's worth
>  > changing the default, but here the cost:benefit ratio is much closer.
>  > 
>  > > == Benefit to Fedora ==
>  > > 
>  > > Pros:
>  > > 
>  > > - Consistent behavior with RHEL8 and RHEL9.
>  > > 
>  > > - Avoid race of udev and the tool that creates the interface.
>  > 
>  > The race will happen if the creation is done in a specific way. But at
>  > the same time, even the Change proposal describes how to avoid the
>  > race ('ip link add address aa:aa:aa:aa:aa:aa type bridge'). So the
> situation
>  > can be summarized as "we have a bunch of 'big' tools that create
>  > devices like NetworkManager or systemd-networkd, which already know or
>  > can be easily fixed to avoid the race, and a manual tool which can be invoked
>  > in a way that avoids the race". Instead of changing the default in udev we
>  > could educate people how to invoke it better.
>  
>  This comes across as blaming every networking tool out there for
>  having their previously correct & working behaviour be broken by
>  a systemd change imposing new requirements on them. 
I was a bit sloppy in my phrasing. NetworkManager and systemd-networkd
already do the right thing. 'ip link' is the odd one out. Various other
tools (e.g. netplan or Debian's network scripts) don't implement this internally,
but call e.g. NetworkManager or 'ip'. So if we changed 'ip', this would
go
a long way to solving the problem (I'm may be wrong on some details here,
please correct me if necessary.)

I'm not "blaming" the tools, I completely understand that they were
written a long time ago. But in fact the issue is fairly generic: any
software which interacts with devices that udev also touches MUST wait
for udev to be done with the device. It's not just the MAC address
policy, but also other rules that users may configure locally, sysctl
configuration, etc. Without synchronization, one runs into races and
errors from the tools when they try to configure things in parallel.

>  Are there any notable tools which actually follow the usage pattern
>  described, or are we in effect having to fix *everything*, including
>  an uncountable amount of documentation, blogs, examples, most of
>  which will prove impossible to fix in practice.
>  
>  > > - Bridge and bond interfaces can get the MAC addresses from their first port.
>  > > 
>  > > In the case of `MACAddressPolicy=none` for a bond (or bridge) the bond will
>  > > get a MAC related to one of its physical NIC devices. In the case of
>  > > provisioning
>  > > new systems (especially in a large datacenter) information about the server
>  > > can be used to configure the network environment (DHCP, Firewall, etc) before
>  > > the server is ever even powered on. This does mean that you may have to update
>  > > your network environment configuration if you replace a NIC (con), but that
> you
>  > > won't have to update your network environment configuration if you
> re-install
>  > > your server (pro), which is what you'd have to do now with
>  > > `MACAddressPolicy=persistent`.
>  > 
>  > Yep. This is *the* case.
>  
>  Having the bridge/bond get the MAC addr of tht first physical NIC is pretty
>  compelling from an automation POV, especially when the virtualziation host is
>  doing traffic filtering for the VM. When a mgmt app assigns a MAC & IP pair to
>  a guest, it is not uncommon to apply firewall rules providing anti-spoofing
>  protection for MAC/IP, such that the VM cannot send traffic with other MAC/IP
>  addresses.  Libvirt provides this functionality with its 'clean-traffic'
>  network filter, and other virt/cloud mgmt apps do similar.
>
>  > > Cons:
>  > > 
>  > > - Deviate from upstream systemd.
>  
>  This is disappointing, as a great benefit of systemd is the level of
>  consistency it brought to distro behaviour, and I think this change
>  would be useful to all distros (in the context of bridge/bond devices) 
That is a good point. If this is done, I too would prefer to change it
upstream rather than just downstream in Fedora.
 
>  > It is also important to mention that Fedora will
> "deviate" from itself
>  > (it's former self). We would be changing a default in place since ~2013 [1].
>  >
>  > [1] https://github.com/systemd/systemd/commit/16b9b87aee
>  
>  Yes, that is quite unfortunate.
>  
>  > > It is desirable that RHEL and Fedora behaves similar. A possible outcome
>  > > could be the current behavior stays and RHEL 10 would change behavior. On the
>  > > other hand, different distributions (or even Fedora spins) have different
>  > > uses and needs. Deviating might be fine. In the same vein, there is also
>  > > a desire to stay close to upstream systemd behavior. But the uses of systemd
>  > > project go beyond Fedora/RHEL, so deviating here may also be fine.
>  > 
>  > So:
>  > - Variant 1 is not good, variant 2 makes more sense.
>  > - The motivating case for v.2 is the "big datacenter" case and race
>  >   when 'ip' is invoked.
>  
>  IMHO The more compelling reason is compliance with MAC address filtering
>  applied to VMs. 
FWIW, conditionalizing the policy on ConditionVirtualization=!vm would
be trivial to add…

>  > - We could improve the tools: 'ip link' could be taught
> to wait until
>  >   udev has stopped processing the device, users can be taught to use
>  >   better invocations.
>  
>  Teaching users is a impractical solution IMHO. The 'ip' tool already
>  has poor usability/learnability IME, such that getting users to learn
>  differences is pretty challenging, plus there's way too much docs out
>  there in the wild that will never be updatable.
>  
>  > - For "small" users (individual machine admins who just install some
>  >   server and configure it after turning it on and/or swap network
>  >   cards between machines), having stable MAC addresses that doesn't
>  >   depend on the order of device discovery or removal of a single
>  >   network card seems better.
>  > - For "big" users (the datacenter case), changing the policy make sense,
>  >   but at the same time, those folks can just insert a policy override,
>  >   they're most likely using some ansible/puppet/cheffy thingy.
>  > - RHEL is more of the "big" case, Fedora is more the "small"
> case.
>  >   Sometimes RHEL and Fedora have different defaults, sometimes RHEL
>  >   takes years to follow what Fedora does.
>  > 
>  > So overall, I think this proposal would make most sense when limited
>  > to specific types of hardware (bridge and bond?), and also to specific
>  > spins: it's a better fit for CoreOS than Workstation…
>  
>  This doesn't feel like a spin-specific problem to me, given it appears
>  useful to both host & guest scenarios alike, and nested VM means
>  host+guest are often the same thing.
>  
>  > But I haven't seen discussion of other approaches: would making 'ip'
>  > better be an option? 'udev' already locks some devices when processing
>  > using a simple BSD lock. Should we do the same for network devices
>  > and just teach 'ip' to do the same? (This would be much simpler and
>  > maybe more appealing that teaching it to wait for udev to report
>  > that it's finished). 
Zbyszek

[Dusty Mabe]

On 6/27/22 07:34, Zbigniew Jędrzejewski-Szmek wrote:
>  On Mon, Jun 27, 2022 at 09:40:53AM +0100, Daniel P. Berrangé wrote:
> > On Mon, Jun 27, 2022 at 10:10:31AM +0200, Zbigniew Jędrzejewski-Szmek wrote:
> >> On Sun, Jun 26, 2022 at 12:36:14AM +0530, Vipul Siddharth wrote:
> >>> This document represents a proposed Change. As part of the Changes
> >>> process, proposals are publicly announced in order to receive
> >>> community feedback. This proposal will only be implemented if approved
> >>> by the Fedora Engineering Steering Committee.
> >>>
> >>>
> >>> == Summary ==
> >>>
> >>> The `systemd-udev` package installs
> >>> `"/usr/lib/systemd/network/99-default.link"`,
> >>> which sets `Link.MACAddressPolicy=persistent`. This proposal is to
> >>> change it to set `Link.MACAddressPolicy=none` to stop changing the MAC
> address.
> >>> This is particularly important for bridge and bond devices.
> >>>
> >>> This change can either only apply to bridge/bond devices, or to
> >>> various software devices. That is to be discussed.
> >>
> >> Hi!
> >>
> >> I already participated in the upstream discussion, so what I write
> >> here will be a restatement to some extent, but with a look from the
> >> side of Fedora specifically.
> >>
> >> The proposal has two variants: 1. just changing the policy to
> MACAddressPolicy=none
> >> or 2. limiting the change to bridge and bond devices.
> >>
> >> Re variant 1: MACAddressPolicy=persistent applies to all devices that don't
> have
> >> a hardware address. The proposal as written (blanket MACAddressPolicy=none)
> >> would change behaviour for all kinds of devices, incl. e.g. software
> >> devices like veths, and cheap hardware devices without a fixed MAC.
> >> The proposal doesn't provide any justification for this (except for
> >> simplicity of implementation) and this variant seems pretty bad and
> >> I'm strongly opposed.
> >>
> >> Re variant 2: the proposal limited to brige/bond devices seems much more
> >> reasonable. In particular, the case described below of a server (virtualized
> >> or not) in a big datacenter is the one case where the benefits of
> >> MACAddressPolicy=none are clearly visible. I still don't think it's
> worth
> >> changing the default, but here the cost:benefit ratio is much closer.
> >>
> >>> == Benefit to Fedora ==
> >>>
> >>> Pros:
> >>>
> >>> - Consistent behavior with RHEL8 and RHEL9.
> >>>
> >>> - Avoid race of udev and the tool that creates the interface.
> >>
> >> The race will happen if the creation is done in a specific way. But at
> >> the same time, even the Change proposal describes how to avoid the
> >> race ('ip link add address aa:aa:aa:aa:aa:aa type bridge'). So the
> situation
> >> can be summarized as "we have a bunch of 'big' tools that create
> >> devices like NetworkManager or systemd-networkd, which already know or
> >> can be easily fixed to avoid the race, and a manual tool which can be invoked
> >> in a way that avoids the race". Instead of changing the default in udev we
> >> could educate people how to invoke it better.
> >
> > This comes across as blaming every networking tool out there for
> > having their previously correct & working behaviour be broken by
> > a systemd change imposing new requirements on them.
>  
>  I was a bit sloppy in my phrasing. NetworkManager and systemd-networkd
>  already do the right thing. 'ip link' is the odd one out. Various other
>  tools (e.g. netplan or Debian's network scripts) don't implement this
> internally,
>  but call e.g. NetworkManager or 'ip'. So if we changed 'ip', this would
> go
>  a long way to solving the problem (I'm may be wrong on some details here,
>  please correct me if necessary.)
>  
>  I'm not "blaming" the tools, I completely understand that they were
>  written a long time ago. But in fact the issue is fairly generic: any
>  software which interacts with devices that udev also touches MUST wait
>  for udev to be done with the device. It's not just the MAC address
>  policy, but also other rules that users may configure locally, sysctl
>  configuration, etc. Without synchronization, one runs into races and
>  errors from the tools when they try to configure things in parallel.
>  
> > Are there any notable tools which actually follow the usage pattern
> > described, or are we in effect having to fix *everything*, including
> > an uncountable amount of documentation, blogs, examples, most of
> > which will prove impossible to fix in practice.
> >
> >>> - Bridge and bond interfaces can get the MAC addresses from their first
> port.
> >>>
> >>> In the case of `MACAddressPolicy=none` for a bond (or bridge) the bond will
> >>> get a MAC related to one of its physical NIC devices. In the case of
> >>> provisioning
> >>> new systems (especially in a large datacenter) information about the server
> >>> can be used to configure the network environment (DHCP, Firewall, etc)
> before
> >>> the server is ever even powered on. This does mean that you may have to
> update
> >>> your network environment configuration if you replace a NIC (con), but that
> you
> >>> won't have to update your network environment configuration if you
> re-install
> >>> your server (pro), which is what you'd have to do now with
> >>> `MACAddressPolicy=persistent`.
> >>
> >> Yep. This is *the* case.
> >
> > Having the bridge/bond get the MAC addr of tht first physical NIC is pretty
> > compelling from an automation POV, especially when the virtualziation host is
> > doing traffic filtering for the VM. When a mgmt app assigns a MAC & IP pair to
> > a guest, it is not uncommon to apply firewall rules providing anti-spoofing
> > protection for MAC/IP, such that the VM cannot send traffic with other MAC/IP
> > addresses.  Libvirt provides this functionality with its 'clean-traffic'
> > network filter, and other virt/cloud mgmt apps do similar.
> >
> >>> Cons:
> >>>
> >>> - Deviate from upstream systemd.
> >
> > This is disappointing, as a great benefit of systemd is the level of
> > consistency it brought to distro behaviour, and I think this change
> > would be useful to all distros (in the context of bridge/bond devices)
>  
>  That is a good point. If this is done, I too would prefer to change it
>  upstream rather than just downstream in Fedora. 
Big +1 from me on getting upstream to change so that we are in alignment. I don't
know if it was made clear in the upstream discussion, but maybe upstream would be
more accepting for just changing the policy for bond/bridge/team devices.

Similarly, we might be able to get the next RHEL to adopt the approach of 
MACAddressPolicy=persistent for non bond/bridge/team and MACAddressPolicy=none
for bond/bridge/team. IOW we may be able to get upstream systemd, Fedora, and
downstream (RHEL/CentOS,Alma,Rocky) to all be in alignment.

Dusty

[Thomas Haller]

On Mon, 2022-06-27 at 13:34 +0200, Zbigniew Jędrzejewski-Szmek wrote:
>  
>  I'm not "blaming" the tools, I completely understand that they were
>  written a long time ago. But in fact the issue is fairly generic: any
>  software which interacts with devices that udev also touches MUST
>  wait
>  for udev to be done with the device. It's not just the MAC address
>  policy, but also other rules that users may configure locally, sysctl
>  configuration, etc. Without synchronization, one runs into races and
>  errors from the tools when they try to configure things in parallel. 
Yes, any software is well advised to wait for udev.

But if it were not for MACAddressPolicy=, default Fedora would not ship
much of relevant to make that necessary in practice.

For example, NetworkManager's udev rules set some attributes. But those
are mainly for NetworkManager itself (and NetworkManager knows to
wait). If you have a tool that creates a software device, then there is
no problem to not wait for those.

Also, "other rules that the user confiugres locally", are entirely
user's choice. Of course, the tools that this user runs, need to
interoperate with the user's environment.

>  > 
>  > IMHO The more compelling reason is compliance with MAC address
>  > filtering
>  > applied to VMs.
>  
>  FWIW, conditionalizing the policy on ConditionVirtualization=!vm
>  would
>  be trivial to add… 
Maybe having ConditionKernelCommandLine= could allow convenient
customization for users and could be set by CoreOS (or
suggested/documented).



Thomas

[Hans de Goede]

Hi,

On 6/27/22 10:10, Zbigniew Jędrzejewski-Szmek wrote:
>  On Sun, Jun 26, 2022 at 12:36:14AM +0530, Vipul Siddharth wrote:
> > This document represents a proposed Change. As part of the Changes
> > process, proposals are publicly announced in order to receive
> > community feedback. This proposal will only be implemented if approved
> > by the Fedora Engineering Steering Committee.
> >
> >
> > == Summary ==
> >
> > The `systemd-udev` package installs
> > `"/usr/lib/systemd/network/99-default.link"`,
> > which sets `Link.MACAddressPolicy=persistent`. This proposal is to
> > change it to set `Link.MACAddressPolicy=none` to stop changing the MAC address.
> > This is particularly important for bridge and bond devices.
> >
> > This change can either only apply to bridge/bond devices, or to
> > various software devices. That is to be discussed.
>  
>  Hi!
>  
>  I already participated in the upstream discussion, so what I write
>  here will be a restatement to some extent, but with a look from the
>  side of Fedora specifically.
>  
>  The proposal has two variants: 1. just changing the policy to MACAddressPolicy=none
>  or 2. limiting the change to bridge and bond devices.
>  
>  Re variant 1: MACAddressPolicy=persistent applies to all devices that don't have
>  a hardware address. The proposal as written (blanket MACAddressPolicy=none)
>  would change behaviour for all kinds of devices, incl. e.g. software
>  devices like veths, and cheap hardware devices without a fixed MAC.
>  The proposal doesn't provide any justification for this (except for
>  simplicity of implementation) and this variant seems pretty bad and
>  I'm strongly opposed. 
<a bit offtopic, sorry>

I did not know systemd/udev can save/restore MAC addresses for devices
where there is no MAC address. I have looking into a related issue on my
todo list. There are some cheap x86 devices with wifi which don't have
a nvram (eeprom) chip instead they load nvram from /lib/firmware. On most
devices there is enough otp inside the wifi chip that they do have a unique
MAC inside the wifi chip's otp memory so everything works well.

But on some devices that otp is either not there or not programmed, so
they take their MAC from the nvram in /lib/firmware which means that
all devices from the same model end up using the same MAC (which is
defined in the nvram in /lib/firmware).

Am I reading the above correctly that udev/systemd already is able
to deal with this and I just need to make it so that the device has
no MAC set at all and then the first time systemd sees this it will
generate a random MAC and use that on future boots ?

Any docs on this / code you can point me to how systemd determines
it needs to do this ?

<even further offtopic>

The same also applies to some bluetooth devices:

https://github.com/bluez/bluez/issues/107

I wonder if similar functionality for bluetooth would
be welcome in systemd ?

Regards,

Hans



>  
>  Re variant 2: the proposal limited to brige/bond devices seems much more
>  reasonable. In particular, the case described below of a server (virtualized
>  or not) in a big datacenter is the one case where the benefits of
>  MACAddressPolicy=none are clearly visible. I still don't think it's worth
>  changing the default, but here the cost:benefit ratio is much closer.
>  
> > == Benefit to Fedora ==
> >
> > Pros:
> >
> > - Consistent behavior with RHEL8 and RHEL9.
> >
> > - Avoid race of udev and the tool that creates the interface.
>  
>  The race will happen if the creation is done in a specific way. But at
>  the same time, even the Change proposal describes how to avoid the
>  race ('ip link add address aa:aa:aa:aa:aa:aa type bridge'). So the situation
>  can be summarized as "we have a bunch of 'big' tools that create
>  devices like NetworkManager or systemd-networkd, which already know or
>  can be easily fixed to avoid the race, and a manual tool which can be invoked
>  in a way that avoids the race". Instead of changing the default in udev we
>  could educate people how to invoke it better.
>  
> > - Bridge and bond interfaces can get the MAC addresses from their first port.
> >
> > In the case of `MACAddressPolicy=none` for a bond (or bridge) the bond will
> > get a MAC related to one of its physical NIC devices. In the case of
> > provisioning
> > new systems (especially in a large datacenter) information about the server
> > can be used to configure the network environment (DHCP, Firewall, etc) before
> > the server is ever even powered on. This does mean that you may have to update
> > your network environment configuration if you replace a NIC (con), but that you
> > won't have to update your network environment configuration if you re-install
> > your server (pro), which is what you'd have to do now with
> > `MACAddressPolicy=persistent`.
>  
>  Yep. This is *the* case.
>  
> > Cons:
> >
> > - Deviate from upstream systemd.
>  
>  It is also important to mention that Fedora will "deviate" from itself
>  (it's former self). We would be changing a default in place since ~2013 [1].
>  
>  [1] https://github.com/systemd/systemd/commit/16b9b87aee
>  
> > It is desirable that RHEL and Fedora behaves similar. A possible outcome
> > could be the current behavior stays and RHEL 10 would change behavior. On the
> > other hand, different distributions (or even Fedora spins) have different
> > uses and needs. Deviating might be fine. In the same vein, there is also
> > a desire to stay close to upstream systemd behavior. But the uses of systemd
> > project go beyond Fedora/RHEL, so deviating here may also be fine.
>  
>  So:
>  - Variant 1 is not good, variant 2 makes more sense.
>  - The motivating case for v.2 is the "big datacenter" case and race
>    when 'ip' is invoked.
>  - We could improve the tools: 'ip link' could be taught to wait until
>    udev has stopped processing the device, users can be taught to use
>    better invocations.
>  - For "small" users (individual machine admins who just install some
>    server and configure it after turning it on and/or swap network
>    cards between machines), having stable MAC addresses that doesn't
>    depend on the order of device discovery or removal of a single
>    network card seems better.
>  - For "big" users (the datacenter case), changing the policy make sense,
>    but at the same time, those folks can just insert a policy override,
>    they're most likely using some ansible/puppet/cheffy thingy.
>  - RHEL is more of the "big" case, Fedora is more the "small" case.
>    Sometimes RHEL and Fedora have different defaults, sometimes RHEL
>    takes years to follow what Fedora does.
>  
>  So overall, I think this proposal would make most sense when limited
>  to specific types of hardware (bridge and bond?), and also to specific
>  spins: it's a better fit for CoreOS than Workstation…
>  
>  But I haven't seen discussion of other approaches: would making 'ip'
>  better be an option? 'udev' already locks some devices when processing
>  using a simple BSD lock. Should we do the same for network devices
>  and just teach 'ip' to do the same? (This would be much simpler and
>  maybe more appealing that teaching it to wait for udev to report
>  that it's finished).
>  
>  Zbyszek
>  _______________________________________________
>  devel mailing list -- devel(a)lists.fedoraproject.org
>  To unsubscribe send an email to devel-leave(a)lists.fedoraproject.org
>  Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/
>  List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
>  List Archives:
> https://lists.fedoraproject.org/archives/list/[e-mail]
>  Do not reply to spam on the list, report it: https://pagure.io/fedora-infrastructure

[Zbigniew Jędrzejewski-Szmek]

On Mon, Jun 27, 2022 at 11:20:13AM +0200, Hans de Goede wrote:
>  Hi,
>  
>  On 6/27/22 10:10, Zbigniew Jędrzejewski-Szmek wrote:
>  > On Sun, Jun 26, 2022 at 12:36:14AM +0530, Vipul Siddharth wrote:
>  >> This document represents a proposed Change. As part of the Changes
>  >> process, proposals are publicly announced in order to receive
>  >> community feedback. This proposal will only be implemented if approved
>  >> by the Fedora Engineering Steering Committee.
>  >>
>  >>
>  >> == Summary ==
>  >>
>  >> The `systemd-udev` package installs
>  >> `"/usr/lib/systemd/network/99-default.link"`,
>  >> which sets `Link.MACAddressPolicy=persistent`. This proposal is to
>  >> change it to set `Link.MACAddressPolicy=none` to stop changing the MAC address.
>  >> This is particularly important for bridge and bond devices.
>  >>
>  >> This change can either only apply to bridge/bond devices, or to
>  >> various software devices. That is to be discussed.
>  > 
>  > Hi!
>  > 
>  > I already participated in the upstream discussion, so what I write
>  > here will be a restatement to some extent, but with a look from the
>  > side of Fedora specifically.
>  > 
>  > The proposal has two variants: 1. just changing the policy to MACAddressPolicy=none
>  > or 2. limiting the change to bridge and bond devices.
>  > 
>  > Re variant 1: MACAddressPolicy=persistent applies to all devices that don't
> have
>  > a hardware address. The proposal as written (blanket MACAddressPolicy=none)
>  > would change behaviour for all kinds of devices, incl. e.g. software
>  > devices like veths, and cheap hardware devices without a fixed MAC.
>  > The proposal doesn't provide any justification for this (except for
>  > simplicity of implementation) and this variant seems pretty bad and
>  > I'm strongly opposed.
>  
>  <a bit offtopic, sorry>
>  
>  I did not know systemd/udev can save/restore MAC addresses for devices
>  where there is no MAC address. I have looking into a related issue on my
>  todo list. There are some cheap x86 devices with wifi which don't have
>  a nvram (eeprom) chip instead they load nvram from /lib/firmware. On most
>  devices there is enough otp inside the wifi chip that they do have a unique
>  MAC inside the wifi chip's otp memory so everything works well.
>  
>  But on some devices that otp is either not there or not programmed, so
>  they take their MAC from the nvram in /lib/firmware which means that
>  all devices from the same model end up using the same MAC (which is
>  defined in the nvram in /lib/firmware).
>  
>  Am I reading the above correctly that udev/systemd already is able
>  to deal with this and I just need to make it so that the device has
>  no MAC set at all and then the first time systemd sees this it will
>  generate a random MAC and use that on future boots ? 
Yes. (With the following clarification: the MAC address is generated
from as some hash of machine-id + device name. So "first time" and
"any time" are the same — no state is ever saved, you're supposed to
get the same result for a specific device on a specific system.)

>  Any docs on this / code you can point me to how systemd determines
>  it needs to do this ? 
udev looks at /sys/class/net/hub0/addr_assign_type.
But the docs are not great :( The description is split between two man pages:
- https://www.freedesktop.org/software/systemd/man/systemd.link.html#MACAdd...
- https://www.freedesktop.org/software/systemd/man/systemd.net-naming-schem...

>  <even further offtopic>
>  
>  The same also applies to some bluetooth devices:
>  
>  https://github.com/bluez/bluez/issues/107
>  
>  I wonder if similar functionality for bluetooth would
>  be welcome in systemd ? 
I think so. From what I can see, we don't do much setup for bluetooth
devices, so I'm not sure if udevd is a better place for this than
e.g. bluez. But if yes, it's nice to do things uniformly, i.e. in this
case treat bluebooth interfaces more like other network interfaces…
The implementation should be simple, since it'd be a question of
extending existing code to cover one more case.

Zbyszek

[Hans de Goede]

Hi,

On 6/27/22 13:18, Zbigniew Jędrzejewski-Szmek wrote:
>  On Mon, Jun 27, 2022 at 11:20:13AM +0200, Hans de Goede wrote:
> > Hi,
> >
> > On 6/27/22 10:10, Zbigniew Jędrzejewski-Szmek wrote:
> >> On Sun, Jun 26, 2022 at 12:36:14AM +0530, Vipul Siddharth wrote:
> >>> This document represents a proposed Change. As part of the Changes
> >>> process, proposals are publicly announced in order to receive
> >>> community feedback. This proposal will only be implemented if approved
> >>> by the Fedora Engineering Steering Committee.
> >>>
> >>>
> >>> == Summary ==
> >>>
> >>> The `systemd-udev` package installs
> >>> `"/usr/lib/systemd/network/99-default.link"`,
> >>> which sets `Link.MACAddressPolicy=persistent`. This proposal is to
> >>> change it to set `Link.MACAddressPolicy=none` to stop changing the MAC
> address.
> >>> This is particularly important for bridge and bond devices.
> >>>
> >>> This change can either only apply to bridge/bond devices, or to
> >>> various software devices. That is to be discussed.
> >>
> >> Hi!
> >>
> >> I already participated in the upstream discussion, so what I write
> >> here will be a restatement to some extent, but with a look from the
> >> side of Fedora specifically.
> >>
> >> The proposal has two variants: 1. just changing the policy to
> MACAddressPolicy=none
> >> or 2. limiting the change to bridge and bond devices.
> >>
> >> Re variant 1: MACAddressPolicy=persistent applies to all devices that don't
> have
> >> a hardware address. The proposal as written (blanket MACAddressPolicy=none)
> >> would change behaviour for all kinds of devices, incl. e.g. software
> >> devices like veths, and cheap hardware devices without a fixed MAC.
> >> The proposal doesn't provide any justification for this (except for
> >> simplicity of implementation) and this variant seems pretty bad and
> >> I'm strongly opposed.
> >
> > <a bit offtopic, sorry>
> >
> > I did not know systemd/udev can save/restore MAC addresses for devices
> > where there is no MAC address. I have looking into a related issue on my
> > todo list. There are some cheap x86 devices with wifi which don't have
> > a nvram (eeprom) chip instead they load nvram from /lib/firmware. On most
> > devices there is enough otp inside the wifi chip that they do have a unique
> > MAC inside the wifi chip's otp memory so everything works well.
> >
> > But on some devices that otp is either not there or not programmed, so
> > they take their MAC from the nvram in /lib/firmware which means that
> > all devices from the same model end up using the same MAC (which is
> > defined in the nvram in /lib/firmware).
> >
> > Am I reading the above correctly that udev/systemd already is able
> > to deal with this and I just need to make it so that the device has
> > no MAC set at all and then the first time systemd sees this it will
> > generate a random MAC and use that on future boots ?
>  
>  Yes. (With the following clarification: the MAC address is generated
>  from as some hash of machine-id + device name. So "first time" and
>  "any time" are the same — no state is ever saved, you're supposed to
>  get the same result for a specific device on a specific system.)
>  
> > Any docs on this / code you can point me to how systemd determines
> > it needs to do this ?
>  
>  udev looks at /sys/class/net/hub0/addr_assign_type.
>  But the docs are not great :( The description is split between two man pages:
>  - https://www.freedesktop.org/software/systemd/man/systemd.link.html#MACAdd...
>  - https://www.freedesktop.org/software/systemd/man/systemd.net-naming-schem... 
Thanks this was useful. I've submitted a kernel patch upstream now 
which fixes the described issue by using a random MAC + setting
the addr_assign_type:

https://lore.kernel.org/linux-wireless/20220708133712.102179-2-hdegoede@r...

Regards,

Hans




>  
> > <even further offtopic>
> >
> > The same also applies to some bluetooth devices:
> >
> > https://github.com/bluez/bluez/issues/107
> >
> > I wonder if similar functionality for bluetooth would
> > be welcome in systemd ?
>  
>  I think so. From what I can see, we don't do much setup for bluetooth
>  devices, so I'm not sure if udevd is a better place for this than
>  e.g. bluez. But if yes, it's nice to do things uniformly, i.e. in this
>  case treat bluebooth interfaces more like other network interfaces…
>  The implementation should be simple, since it'd be a question of
>  extending existing code to cover one more case.
>  
>  Zbyszek
>

[Dusty Mabe]

On 6/27/22 04:10, Zbigniew Jędrzejewski-Szmek wrote:
>  
>  Re variant 2: the proposal limited to brige/bond devices seems much more
>  reasonable. In particular, the case described below of a server (virtualized
>  or not) in a big datacenter is the one case where the benefits of
>  MACAddressPolicy=none are clearly visible. I still don't think it's worth
>  changing the default, but here the cost:benefit ratio is much closer. 
I think that's reasonable. The proposal owners are willing to make this change
to the proposal so that it only applies to bond/bridge/team devices if that
is more acceptable.

[Thomas Haller]

On Mon, 2022-06-27 at 10:10 +0200, Zbigniew Jędrzejewski-Szmek wrote:
>  > 
>  > - Deviate from upstream systemd.
>  
>  It is also important to mention that Fedora will "deviate" from
>  itself
>  (it's former self). We would be changing a default in place since
>  ~2013 [1].
>  
>  [1] https://github.com/systemd/systemd/commit/16b9b87aee 

Hi,

Is that the case? I think in practice, it only changed in F31/2019 with
systemd 242:
https://github.com/systemd/systemd/blob/eaee50fee2d0baec9116ae76945f18fef...

Granted, 2019 is also a long time...


Thomas

[Zbigniew Jędrzejewski-Szmek]

On Mon, Jun 27, 2022 at 06:08:13PM +0200, Thomas Haller wrote:
>  On Mon, 2022-06-27 at 10:10 +0200, Zbigniew Jędrzejewski-Szmek
> wrote:
>  > > 
>  > > - Deviate from upstream systemd.
>  > 
>  > It is also important to mention that Fedora will "deviate" from
>  > itself
>  > (it's former self). We would be changing a default in place since
>  > ~2013 [1].
>  > 
>  > [1] https://github.com/systemd/systemd/commit/16b9b87aee
>  
>  
>  Hi,
>  
>  Is that the case? I think in practice, it only changed in F31/2019 with
>  systemd 242:
>
> https://github.com/systemd/systemd/blob/eaee50fee2d0baec9116ae76945f18fef...
>  
>  Granted, 2019 is also a long time... 
2013 — MACAddressPolicy=persistent mostly for hardware devices
2019 — MACAddressPolicy=persistent for bridge and bond and some other stuff

Zbyszek

[Thomas Haller]

On Mon, 2022-06-27 at 10:10 +0200, Zbigniew Jędrzejewski-Szmek wrote:
>  On Sun, Jun 26, 2022 at 12:36:14AM +0530, Vipul Siddharth wrote:
>  > This document represents a proposed Change. As part of the Changes
>  > process, proposals are publicly announced in order to receive
>  > community feedback. This proposal will only be implemented if
>  > approved
>  > by the Fedora Engineering Steering Committee.
>  > 
>  > 
>  > == Summary ==
>  > 
>  > The `systemd-udev` package installs
>  > `"/usr/lib/systemd/network/99-default.link"`,
>  > which sets `Link.MACAddressPolicy=persistent`. This proposal is to
>  > change it to set `Link.MACAddressPolicy=none` to stop changing the
>  > MAC address.
>  > This is particularly important for bridge and bond devices.
>  > 
>  > This change can either only apply to bridge/bond devices, or to
>  > various software devices. That is to be discussed.
>  
>  Hi!
>  
>  I already participated in the upstream discussion, so what I write
>  here will be a restatement to some extent, but with a look from the
>  side of Fedora specifically.
>  
>  The proposal has two variants: 1. just changing the policy to
>  MACAddressPolicy=none
>  or 2. limiting the change to bridge and bond devices.
>  
>  Re variant 1: MACAddressPolicy=persistent applies to all devices that
>  don't have
>  a hardware address. The proposal as written (blanket
>  MACAddressPolicy=none)
>  would change behaviour for all kinds of devices, incl. e.g. software
>  devices like veths, and cheap hardware devices without a fixed MAC. 
the proposal was only about software devices.

The difference is that software devices are explicitly added by some
tool (if we ignore module parameters like "max_bonds" -- which on
fedora/systemd is already 0 by default).

A hardware device gets automatically added when loading the kernel.
More importantly, udev already does renaming, so any user is well
advised to wait for udev to settles. At which point, they can also wait
for the MAC address policy.


>  The proposal doesn't provide any justification for this (except
> for
>  simplicity of implementation) and this variant seems pretty bad and
>  I'm strongly opposed. 
The first point: the race. And that udev touches interfaces created by
somebody, without being explicitly told to do so.

Yes, udev always does something to the device, like attaching
attributes such as ID_NET_NAMING_SCHEME. But aside the MAC address
policy, it doesn't do anything relevant where a race would matter.

>  
>  Re variant 2: the proposal limited to brige/bond devices seems much
>  more
>  reasonable. In particular, the case described below of a server
>  (virtualized
>  or not) in a big datacenter is the one case where the benefits of
>  MACAddressPolicy=none are clearly visible. I still don't think it's
>  worth
>  changing the default, but here the cost:benefit ratio is much closer.
>  
>  > == Benefit to Fedora ==
>  > 
>  > Pros:
>  > 
>  > - Consistent behavior with RHEL8 and RHEL9.
>  > 
>  > - Avoid race of udev and the tool that creates the interface.
>  
>  The race will happen if the creation is done in a specific way. But
>  at
>  the same time, even the Change proposal describes how to avoid the
>  race ('ip link add address aa:aa:aa:aa:aa:aa type bridge'). So the
>  situation
>  can be summarized as "we have a bunch of 'big' tools that create
>  devices like NetworkManager or systemd-networkd, which already know
>  or
>  can be easily fixed to avoid the race, and a manual tool which can be
>  invoked
>  in a way that avoids the race". Instead of changing the default in
>  udev we
>  could educate people how to invoke it better. 
I don't think that big projects matter much. They can learn this. For
example, docker had a problem with this ([1]), which presumably got
fixed long ago.

[1]
https://gitlab.freedesktop.org/NetworkManager/NetworkManager/-/merge_requ...

The problem are smaller projects and user scripts/tools.

For example, NetworkManager-ci contains hundreds of scripts for
testing, and -- while we should know better -- we don't wait for udev
after creating a veth interface for testing. That is no problem for us,
we just set MACAddressPolicy=none. But it makes me wonder, just how
many "naive" scripts are out there that don't get this right.

You say, "the race will happen if the creation is done in a specific
way". The problem is that the specific way is the most straight forward
one.

Also, to educate people how to use iproute2 (?) better, seems not
clearly best. Is there really a problem on the user's side when they
are creating an interface and don't expect udev to interfere?


>  > - Bridge and bond interfaces can get the MAC addresses from
> their
>  > first port.
>  > 
>  > In the case of `MACAddressPolicy=none` for a bond (or bridge) the
>  > bond will
>  > get a MAC related to one of its physical NIC devices. In the case
>  > of
>  > provisioning
>  > new systems (especially in a large datacenter) information about
>  > the server
>  > can be used to configure the network environment (DHCP, Firewall,
>  > etc) before
>  > the server is ever even powered on. This does mean that you may
>  > have to update
>  > your network environment configuration if you replace a NIC (con),
>  > but that you
>  > won't have to update your network environment configuration if you
>  > re-install
>  > your server (pro), which is what you'd have to do now with
>  > `MACAddressPolicy=persistent`.
>  
>  Yep. This is *the* case.
>  
>  > Cons:
>  > 
>  > - Deviate from upstream systemd.
>  
>  It is also important to mention that Fedora will "deviate" from
>  itself
>  (it's former self). We would be changing a default in place since
>  ~2013 [1].
>  
>  [1] https://github.com/systemd/systemd/commit/16b9b87aee
>  
>  > It is desirable that RHEL and Fedora behaves similar. A possible
>  > outcome
>  > could be the current behavior stays and RHEL 10 would change
>  > behavior. On the
>  > other hand, different distributions (or even Fedora spins) have
>  > different
>  > uses and needs. Deviating might be fine. In the same vein, there is
>  > also
>  > a desire to stay close to upstream systemd behavior. But the uses
>  > of systemd
>  > project go beyond Fedora/RHEL, so deviating here may also be fine.
>  
>  So:
>  - Variant 1 is not good, variant 2 makes more sense.
>  - The motivating case for v.2 is the "big datacenter" case and race
>    when 'ip' is invoked.
>  - We could improve the tools: 'ip link' could be taught to wait until
>    udev has stopped processing the device, users can be taught to use
>    better invocations. 
see below.

>  - For "small" users (individual machine admins who just
> install some
>    server and configure it after turning it on and/or swap network
>    cards between machines), having stable MAC addresses that doesn't
>    depend on the order of device discovery or removal of a single
>    network card seems better. 
That claim is most relevant, but seems not well justified. It comes
from a desire for order, where things are stable.

I am not so convinced though, given that the solution requires to
involve udev. A counter point would be that many users might not care
for this feature, and those who do, can enable it just fine.


>  - For "big" users (the datacenter case), changing the
> policy make
>  sense,
>    but at the same time, those folks can just insert a policy
>  override,
>    they're most likely using some ansible/puppet/cheffy thingy.
>  - RHEL is more of the "big" case, Fedora is more the "small" case.
>    Sometimes RHEL and Fedora have different defaults, sometimes RHEL
>    takes years to follow what Fedora does. 
I personally care mostly about RHEL here and to find out what to do. So
if we agree that RHEL is allowed to have a different downstream
behavior than Fedora, I'd be satisfied!


>  
>  So overall, I think this proposal would make most sense when limited
>  to specific types of hardware (bridge and bond?), and also to
>  specific
>  spins: it's a better fit for CoreOS than Workstation… 
Makes sense.

I think CoreOS would be tempted to set MACAddressPolicy=none. Thus, a
major Fedora user would go against the distro default. That raises a
question about why Workstation behaves different. But if CoreOS (or
other spins) are allowed to do that, I think Dusty would also be
satisfied.


>  
>  But I haven't seen discussion of other approaches: would making 'ip'
>  better be an option? 'udev' already locks some devices when
>  processing
>  using a simple BSD lock. Should we do the same for network devices
>  and just teach 'ip' to do the same? (This would be much simpler and
>  maybe more appealing that teaching it to wait for udev to report
>  that it's finished). 
iproute2 is the obvious candidate, but there can be any tools that use
the kernel API some other way (e.g. pyroute2). It wouldn't solve the
problem for them making the "solution" less universal and attractive.

I would be surprised if iproute2 would accept patches to integrate with
udev. It seems like a low-level tool, and I could understand if the
answer is to use the tool correctly. Unfortunately, the correct way is
not immediately obvious.



Thank you for all your feedback!!


Thomas

[Dusty Mabe]

On 6/27/22 13:03, Thomas Haller wrote:
>  On Mon, 2022-06-27 at 10:10 +0200, Zbigniew Jędrzejewski-Szmek wrote:
>  
> > - For "big" users (the datacenter case), changing the policy make
> > sense,
> >   but at the same time, those folks can just insert a policy
> > override,
> >   they're most likely using some ansible/puppet/cheffy thingy.
> > - RHEL is more of the "big" case, Fedora is more the "small"
> case.
> >   Sometimes RHEL and Fedora have different defaults, sometimes RHEL
> >   takes years to follow what Fedora does.
>  
>  I personally care mostly about RHEL here and to find out what to do. So
>  if we agree that RHEL is allowed to have a different downstream
>  behavior than Fedora, I'd be satisfied! 
RHEL and Fedora *can* deviate, but I think it's valuable for them to have
alignment whenever possible. i.e. if they deviate the benefit should outweigh
the cost. Optimally we find a default that works best in most cases that we
can share.

>  
>  
> >
> > So overall, I think this proposal would make most sense when limited
> > to specific types of hardware (bridge and bond?), and also to
> > specific
> > spins: it's a better fit for CoreOS than Workstation…
>  
>  Makes sense.
>  
>  I think CoreOS would be tempted to set MACAddressPolicy=none. Thus, a
>  major Fedora user would go against the distro default. That raises a
>  question about why Workstation behaves different. But if CoreOS (or
>  other spins) are allowed to do that, I think Dusty would also be
>  satisfied. 
While I agree it's nice to have the flexibility to deviate on just certain
variants, I'd much prefer to have the policy set Fedora wide if we can. Every
delta is a cost and our goal in Fedora CoreOS is to try to not deviate from
Fedora unless it really really benefits the users. The more we deviate over
time the more maintenance burden we accumulate.

If this change got rejected at the Fedora level I don't know if Fedora CoreOS
would pick it up individually. From one perspective it wouldn't make sense:
deviating from the rest of Fedora, but from another perspective it would make
sense: deviating from our downstream of RHCOS (based on RHEL).

Again, I'd really prefer to find a compromise here.

[Demi Marie Obenour]

On 6/27/22 13:03, Thomas Haller wrote:
>  On Mon, 2022-06-27 at 10:10 +0200, Zbigniew Jędrzejewski-Szmek
> wrote:
> > On Sun, Jun 26, 2022 at 12:36:14AM +0530, Vipul Siddharth wrote:
> >> This document represents a proposed Change. As part of the Changes
> >> process, proposals are publicly announced in order to receive
> >> community feedback. This proposal will only be implemented if
> >> approved
> >> by the Fedora Engineering Steering Committee.
> >>
> >>
> >> == Summary ==
> >>
> >> The `systemd-udev` package installs
> >> `"/usr/lib/systemd/network/99-default.link"`,
> >> which sets `Link.MACAddressPolicy=persistent`. This proposal is to
> >> change it to set `Link.MACAddressPolicy=none` to stop changing the
> >> MAC address.
> >> This is particularly important for bridge and bond devices.
> >>
> >> This change can either only apply to bridge/bond devices, or to
> >> various software devices. That is to be discussed.
> >
> > Hi!
> >
> > I already participated in the upstream discussion, so what I write
> > here will be a restatement to some extent, but with a look from the
> > side of Fedora specifically.
> >
> > The proposal has two variants: 1. just changing the policy to
> > MACAddressPolicy=none
> > or 2. limiting the change to bridge and bond devices.
> >
> > Re variant 1: MACAddressPolicy=persistent applies to all devices that
> > don't have
> > a hardware address. The proposal as written (blanket
> > MACAddressPolicy=none)
> > would change behaviour for all kinds of devices, incl. e.g. software
> > devices like veths, and cheap hardware devices without a fixed MAC.
>  
>  the proposal was only about software devices.
>  
>  The difference is that software devices are explicitly added by some
>  tool (if we ignore module parameters like "max_bonds" -- which on
>  fedora/systemd is already 0 by default).
>  
>  A hardware device gets automatically added when loading the kernel.
>  More importantly, udev already does renaming, so any user is well
>  advised to wait for udev to settles. At which point, they can also wait
>  for the MAC address policy.
>  
>  
> > The proposal doesn't provide any justification for this (except for
> > simplicity of implementation) and this variant seems pretty bad and
> > I'm strongly opposed.
>  
>  The first point: the race. And that udev touches interfaces created by
>  somebody, without being explicitly told to do so.
>  
>  Yes, udev always does something to the device, like attaching
>  attributes such as ID_NET_NAMING_SCHEME. But aside the MAC address
>  policy, it doesn't do anything relevant where a race would matter.
>  
> >
> > Re variant 2: the proposal limited to brige/bond devices seems much
> > more
> > reasonable. In particular, the case described below of a server
> > (virtualized
> > or not) in a big datacenter is the one case where the benefits of
> > MACAddressPolicy=none are clearly visible. I still don't think it's
> > worth
> > changing the default, but here the cost:benefit ratio is much closer.
> >
> >> == Benefit to Fedora ==
> >>
> >> Pros:
> >>
> >> - Consistent behavior with RHEL8 and RHEL9.
> >>
> >> - Avoid race of udev and the tool that creates the interface.
> >
> > The race will happen if the creation is done in a specific way. But
> > at
> > the same time, even the Change proposal describes how to avoid the
> > race ('ip link add address aa:aa:aa:aa:aa:aa type bridge'). So the
> > situation
> > can be summarized as "we have a bunch of 'big' tools that create
> > devices like NetworkManager or systemd-networkd, which already know
> > or
> > can be easily fixed to avoid the race, and a manual tool which can be
> > invoked
> > in a way that avoids the race". Instead of changing the default in
> > udev we
> > could educate people how to invoke it better.
>  
>  I don't think that big projects matter much. They can learn this. For
>  example, docker had a problem with this ([1]), which presumably got
>  fixed long ago.
>  
>  [1]
> https://gitlab.freedesktop.org/NetworkManager/NetworkManager/-/merge_requ...
>  
>  The problem are smaller projects and user scripts/tools.
>  
>  For example, NetworkManager-ci contains hundreds of scripts for
>  testing, and -- while we should know better -- we don't wait for udev
>  after creating a veth interface for testing. That is no problem for us,
>  we just set MACAddressPolicy=none. But it makes me wonder, just how
>  many "naive" scripts are out there that don't get this right. 
At least for block devices, waiting for udev to finish can actually
be a non-trivial overhead, especially when udev has been explicitly
told to *not* do anything with the devices in question.  It has a
measurable impact on how long LVM commands take to run, and LVM is
known to be a significant contributor to the amount of time Qubes OS
needs to start a VM.
-- 
Sincerely,
Demi Marie Obenour (she/her/hers)

F37 Change Proposal: MAC Address Policy none (System-Wide

[Gabriel L. Somlo]

On Sat, Jun 25, 2022 at 08:39:22PM +0000, devel-request(a)lists.fedoraproject.org wrote:
>  This document represents a proposed Change. As part of the Changes
>  process, proposals are publicly announced in order to receive
>  community feedback. This proposal will only be implemented if approved
>  by the Fedora Engineering Steering Committee.
>  
>  
>  == Summary ==
>  
>  The `systemd-udev` package installs
>  `"/usr/lib/systemd/network/99-default.link"`,
>  which sets `Link.MACAddressPolicy=persistent`. This proposal is to
>  change it to set `Link.MACAddressPolicy=none` to stop changing the MAC address.
>  This is particularly important for bridge and bond devices.
>  
>  This change can either only apply to bridge/bond devices, or to
>  various software devices. That is to be discussed.
>  
>  == Owner ==
>  
>  * Name: [[User:thaller|Thomas Haller]] (NetworkManager)
>  * Email: <thaller(a)redhat.com&gt;
>  * Name: [[User:dustymabe| Dusty Mabe]] (Fedora CoreOS)
>  * Email: <dmabe(a)redhat.com&gt;
>  
>  
>  == Detailed Description ==
>  
>  On Fedora, udev by default changes the MAC address of a wide range of
>  software devices.
>  This was introduced by systemd 242 in early 2019 (Fedora 31), when
>  `MACAddressPolicy=` was
>  extended to affect more types of devices.
>  
>  With `MACAddressPolicy=persistent` udev's aim is to provide a stable
>  MAC address, otherwise
>  the kernel will assign a random one. However, that can cause problems:
>  
>  Firstly, software devices are always created by some tool that has
>  plans for the device. The tool may not
>  expect that udev is going to change the MAC address and races against
>  that. The best solution
>  for the tool is to set the MAC address when creating an interface.
>  This will prevent
>  udev from changing the MAC address according to the MACAddressPolicy.
>  Otherwise, the tool should wait for udev to initialize the device to
>  avoid the race. In theory, a
>  tool is always advised to wait for udev to initialize the device.
>  However, if it were not for MACAddressPolicy,
>  in common scenarios udev doesn't do anything relevant for software
>  devices to make that necessary.
>  
>  Secondly, for interface types bridge and bond, an unset MAC address
>  has a special meaning
>  to the kernel and the MAC address of the first port is used. If udev
>  changes the MAC
>  address, that no longer works.
>  Now the generated MAC address is not directly discoverable as it is
>  based on `/etc/machine-id`
>  ([https://www.man7.org/linux/man-pages/man5/machine-id.5.html
>  machine-id(5)]), among
>  other data. Even if there were a tool to easily calculate the MAC
>  address, it could be cumbersome
>  to use it without logging into the machine first. The MAC address can
>  directly affect the
>  assigned IP address, for example when using DHCP. When booting a new
>  virtual machine, the user might
>  know the MAC address of the (virtual) "physical" interfaces. When
>  bonding/bridging those
>  interfaces, the bond/bridge would get one of the well known MAC
>  addresses. `MACAddressPolicy=persistent`
>  interferes with that.
>  
>  The goal of persistent policy is to provide a stable MAC address. Note
>  that if the
>  tool or user who created the interface would want a certain MAC address, they
>  have all the means to set it already. That applies regardless whether
>  the tool is
>  iproute2, NetworkManager, systemd-networkd. Neither NetworkManager nor
>  systemd-networkd
>  rely on udev's MACAddressPolicy for setting the MAC address. This
>  behavior is mostly
>  useful for plain `ip link add`, but it's unclear which real world user
>  wants this behavior.
>  
>  Of course, the user is welcome to configure the MAC address in any way
>  they want. Including,
>  dropping a link file that sets `MACAddressPolicy=persistent`. The
>  problem is once udev sets a MAC address,
>  it cannot be unset. Which makes this problematic to do by default.
>  
>  While Fedora inherited this behavior from upstream systemd, RHEL-9
>  does not follow this behavior
>
> ([https://gitlab.com/redhat/centos-stream/rpms/systemd/-/blob/c8953519504bf...
>  centos9],
>  [https://bugzilla.redhat.com/show_bug.cgi?id=1921094 rh#1921094]). For
>  RHEL-8, this doesn't
>  apply because the systemd there is too old to change the MAC address
>  of most software devices.
>  
>  This could be either implemented by patching
>  `/usr/lib/systemd/network/99-default.link`
>  to have a different policy, or by dropping a link file with higher
>  priority. In the latter
>  case, that override could be shipped either by udev or even by NetworkManager.
>  
>  Another option is to change the scope of this proposal to only change to
>  `MACAddressPolicy=none` for the device types where this causes the most issues
>  (bridge/bond/team). 
 
+1 (for very large values of 1 :)

"MACAddressPolicy=none" was the default, reliable, expected behavior for
literally *decades* until it got broken around the time of F32 (systemd 242):
https://bugzilla.redhat.com/show_bug.cgi?id=1834537

Unbreaking the behavior (at least for bridge/bond/team interfaces)
is welcome -- better late than never!

Thanks much,
--Gabriel
 
> == Feedback ==
> 
> This was also discussed on upstream systemd mailing list
> [https://lists.freedesktop.org/archives/systemd-devel/2022-May/047893.html
> here].
> The upstream systemd maintainers' opinion is that the current udev
> behavior is desirable, but accepts that distributions (or network
> stacks such as NetworkManager) may choose to change the default
> depending on their needs
> ([https://lists.freedesktop.org/archives/systemd-devel/2022-May/047943.html
> reference]).
> The goal here is to find out what the Fedora community thinks is the
> most appropriate default.
> 
> The RHEL-9 bug is [https://bugzilla.redhat.com/show_bug.cgi?id=1921094
> [rh#1921094]].
> 
> 
> == Benefit to Fedora ==
> 
> Pros:
> 
> - Consistent behavior with RHEL8 and RHEL9.
> 
> - Avoid race of udev and the tool that creates the interface.
> 
> - Bridge and bond interfaces can get the MAC addresses from their first port.
> 
> In the case of `MACAddressPolicy=none` for a bond (or bridge) the bond will
> get a MAC related to one of its physical NIC devices. In the case of
> provisioning
> new systems (especially in a large datacenter) information about the server
> can be used to configure the network environment (DHCP, Firewall, etc) before
> the server is ever even powered on. This does mean that you may have to update
> your network environment configuration if you replace a NIC (con), but that you
> won't have to update your network environment configuration if you re-install
> your server (pro), which is what you'd have to do now with
> `MACAddressPolicy=persistent`.
> 
> Cons:
> 
> - Deviate from upstream systemd.
> 
> It is desirable that RHEL and Fedora behaves similar. A possible outcome
> could be the current behavior stays and RHEL 10 would change behavior. On the
> other hand, different distributions (or even Fedora spins) have different
> uses and needs. Deviating might be fine. In the same vein, there is also
> a desire to stay close to upstream systemd behavior. But the uses of systemd
> project go beyond Fedora/RHEL, so deviating here may also be fine.
> 
> 
> == Scope ==
> 
> * Proposal owners:
> The main goal of this request is to generate productive discussion and
> find the desired behavior.
> The implementation/changes are either way very simple.
> 
> * Other developers:
> Other projects that wish a certain MAC address are welcome to
> set it for their devices. Including using udev's MACAddressPolicy.
> 
> * Release engineering:
> Not needed for this change.
> 
> * Policies and guidelines: N/A (not needed for this Change)
> * Trademark approval: N/A (not needed for this Change)
> * Alignment with Objectives:
> 
> == Upgrade/compatibility impact ==
> 
> After the change, the MAC address for the affected device types changes.
> 
> 
> == How To Test ==
> 
> 1) Create a software device two times, for example `ip link add type
> bridge`. Note
> that the MAC address is either stable or random, depending on the
> `MACAddressPolicy=`.
> 
> 2) Note that if the software device has the MAC address set initially,
> udev does not
> change it (`ip link add address aa:aa:aa:aa:aa:aa type bridge`). That depends on
> `/sys/class/net/$dev/addr_assign_type`.
> 
> 3) Create a bridge/bond interface without setting the MAC address.
> Note that if `MACAddressPolicy=none`,
> the MAC address is random at first. Note that attaching the first port
> will update the controller's MAC address.
> On the other hand, with `MACAddressPolicy=persistent`, the MAC address
> of the controller is fixed
> and not inherited from the port.
> 
> 4) Run
> 
>   ip monitor link &
>   while : ; do
>     ip link del xxx
>     ip link add name xxx type dummy \
>     && ip link set xxx addr aa:00:00:00:00:00 \
>     && ip link show xxx | grep -q aa:00:00:00:00:00 \
>     || break
>   done
> 
> to reproduce the race between a simple tool and udev changing the MAC address.
> 
> 
> == User Experience ==
> 
> Bond/bridge devices would again get assigned the MAC address of the
> first NIC added to the device.
> If we chose to not limit the scope of this change to just
> bonds/bridges then all software devices would get randomly assigned
> MAC addresses.
> 
> 
> == Dependencies ==
> 
> None.
> 
> 
> == Contingency Plan ==
> 
> If the change is rejected, nothing needs to be done. The change
> itself will be simple to implement.
> Contingency deadline: beta freeze
> Blocks release? No
> 
> 
> == Documentation ==
> 
> TODO.
> 
> == Release Notes ==
> 
> -- 
> Vipul Siddharth
> He/His/Him
> FPgM team member

[Tom Hughes]

As I said before I don't care that much what the policy is but I do
care very much that Fedora keeps changing it.

Twice now I have had to go and reconfigure my networks after a Fedora
upgrade has changed the MAC assignment policy.

My vote is therefore to leave it as it is so that I don't have to go
and change everything again!

Basically I don't care what the MAC of a machine is but I care very
much if it changes because I have to update DHCP so that it can
continue to issue the correct IPs and DNS for IPv6 addresses which
change when the MAC changes.

Tom

On 25/06/2022 20:06, Vipul Siddharth wrote:
>  This document represents a proposed Change. As part of the Changes
>  process, proposals are publicly announced in order to receive
>  community feedback. This proposal will only be implemented if approved
>  by the Fedora Engineering Steering Committee.
>  
>  
>  == Summary ==
>  
>  The `systemd-udev` package installs
>  `"/usr/lib/systemd/network/99-default.link"`,
>  which sets `Link.MACAddressPolicy=persistent`. This proposal is to
>  change it to set `Link.MACAddressPolicy=none` to stop changing the MAC address.
>  This is particularly important for bridge and bond devices.
>  
>  This change can either only apply to bridge/bond devices, or to
>  various software devices. That is to be discussed.
>  
>  == Owner ==
>  
>  * Name: [[User:thaller|Thomas Haller]] (NetworkManager)
>  * Email: <thaller(a)redhat.com&gt;
>  * Name: [[User:dustymabe| Dusty Mabe]] (Fedora CoreOS)
>  * Email: <dmabe(a)redhat.com&gt;
>  
>  
>  == Detailed Description ==
>  
>  On Fedora, udev by default changes the MAC address of a wide range of
>  software devices.
>  This was introduced by systemd 242 in early 2019 (Fedora 31), when
>  `MACAddressPolicy=` was
>  extended to affect more types of devices.
>  
>  With `MACAddressPolicy=persistent` udev's aim is to provide a stable
>  MAC address, otherwise
>  the kernel will assign a random one. However, that can cause problems:
>  
>  Firstly, software devices are always created by some tool that has
>  plans for the device. The tool may not
>  expect that udev is going to change the MAC address and races against
>  that. The best solution
>  for the tool is to set the MAC address when creating an interface.
>  This will prevent
>  udev from changing the MAC address according to the MACAddressPolicy.
>  Otherwise, the tool should wait for udev to initialize the device to
>  avoid the race. In theory, a
>  tool is always advised to wait for udev to initialize the device.
>  However, if it were not for MACAddressPolicy,
>  in common scenarios udev doesn't do anything relevant for software
>  devices to make that necessary.
>  
>  Secondly, for interface types bridge and bond, an unset MAC address
>  has a special meaning
>  to the kernel and the MAC address of the first port is used. If udev
>  changes the MAC
>  address, that no longer works.
>  Now the generated MAC address is not directly discoverable as it is
>  based on `/etc/machine-id`
>  ([https://www.man7.org/linux/man-pages/man5/machine-id.5.html
>  machine-id(5)]), among
>  other data. Even if there were a tool to easily calculate the MAC
>  address, it could be cumbersome
>  to use it without logging into the machine first. The MAC address can
>  directly affect the
>  assigned IP address, for example when using DHCP. When booting a new
>  virtual machine, the user might
>  know the MAC address of the (virtual) "physical" interfaces. When
>  bonding/bridging those
>  interfaces, the bond/bridge would get one of the well known MAC
>  addresses. `MACAddressPolicy=persistent`
>  interferes with that.
>  
>  The goal of persistent policy is to provide a stable MAC address. Note
>  that if the
>  tool or user who created the interface would want a certain MAC address, they
>  have all the means to set it already. That applies regardless whether
>  the tool is
>  iproute2, NetworkManager, systemd-networkd. Neither NetworkManager nor
>  systemd-networkd
>  rely on udev's MACAddressPolicy for setting the MAC address. This
>  behavior is mostly
>  useful for plain `ip link add`, but it's unclear which real world user
>  wants this behavior.
>  
>  Of course, the user is welcome to configure the MAC address in any way
>  they want. Including,
>  dropping a link file that sets `MACAddressPolicy=persistent`. The
>  problem is once udev sets a MAC address,
>  it cannot be unset. Which makes this problematic to do by default.
>  
>  While Fedora inherited this behavior from upstream systemd, RHEL-9
>  does not follow this behavior
>
> ([https://gitlab.com/redhat/centos-stream/rpms/systemd/-/blob/c8953519504bf...
>  centos9],
>  [https://bugzilla.redhat.com/show_bug.cgi?id=1921094 rh#1921094]). For
>  RHEL-8, this doesn't
>  apply because the systemd there is too old to change the MAC address
>  of most software devices.
>  
>  This could be either implemented by patching
>  `/usr/lib/systemd/network/99-default.link`
>  to have a different policy, or by dropping a link file with higher
>  priority. In the latter
>  case, that override could be shipped either by udev or even by NetworkManager.
>  
>  Another option is to change the scope of this proposal to only change to
>  `MACAddressPolicy=none` for the device types where this causes the most issues
>  (bridge/bond/team).
>  
>  
>  == Feedback ==
>  
>  This was also discussed on upstream systemd mailing list
>  [https://lists.freedesktop.org/archives/systemd-devel/2022-May/047893.html
>  here].
>  The upstream systemd maintainers' opinion is that the current udev
>  behavior is desirable, but accepts that distributions (or network
>  stacks such as NetworkManager) may choose to change the default
>  depending on their needs
>  ([https://lists.freedesktop.org/archives/systemd-devel/2022-May/047943.html
>  reference]).
>  The goal here is to find out what the Fedora community thinks is the
>  most appropriate default.
>  
>  The RHEL-9 bug is [https://bugzilla.redhat.com/show_bug.cgi?id=1921094
>  [rh#1921094]].
>  
>  
>  == Benefit to Fedora ==
>  
>  Pros:
>  
>  - Consistent behavior with RHEL8 and RHEL9.
>  
>  - Avoid race of udev and the tool that creates the interface.
>  
>  - Bridge and bond interfaces can get the MAC addresses from their first port.
>  
>  In the case of `MACAddressPolicy=none` for a bond (or bridge) the bond will
>  get a MAC related to one of its physical NIC devices. In the case of
>  provisioning
>  new systems (especially in a large datacenter) information about the server
>  can be used to configure the network environment (DHCP, Firewall, etc) before
>  the server is ever even powered on. This does mean that you may have to update
>  your network environment configuration if you replace a NIC (con), but that you
>  won't have to update your network environment configuration if you re-install
>  your server (pro), which is what you'd have to do now with
>  `MACAddressPolicy=persistent`.
>  
>  Cons:
>  
>  - Deviate from upstream systemd.
>  
>  It is desirable that RHEL and Fedora behaves similar. A possible outcome
>  could be the current behavior stays and RHEL 10 would change behavior. On the
>  other hand, different distributions (or even Fedora spins) have different
>  uses and needs. Deviating might be fine. In the same vein, there is also
>  a desire to stay close to upstream systemd behavior. But the uses of systemd
>  project go beyond Fedora/RHEL, so deviating here may also be fine.
>  
>  
>  == Scope ==
>  
>  * Proposal owners:
>  The main goal of this request is to generate productive discussion and
>  find the desired behavior.
>  The implementation/changes are either way very simple.
>  
>  * Other developers:
>  Other projects that wish a certain MAC address are welcome to
>  set it for their devices. Including using udev's MACAddressPolicy.
>  
>  * Release engineering:
>  Not needed for this change.
>  
>  * Policies and guidelines: N/A (not needed for this Change)
>  * Trademark approval: N/A (not needed for this Change)
>  * Alignment with Objectives:
>  
>  == Upgrade/compatibility impact ==
>  
>  After the change, the MAC address for the affected device types changes.
>  
>  
>  == How To Test ==
>  
>  1) Create a software device two times, for example `ip link add type
>  bridge`. Note
>  that the MAC address is either stable or random, depending on the
>  `MACAddressPolicy=`.
>  
>  2) Note that if the software device has the MAC address set initially,
>  udev does not
>  change it (`ip link add address aa:aa:aa:aa:aa:aa type bridge`). That depends on
>  `/sys/class/net/$dev/addr_assign_type`.
>  
>  3) Create a bridge/bond interface without setting the MAC address.
>  Note that if `MACAddressPolicy=none`,
>  the MAC address is random at first. Note that attaching the first port
>  will update the controller's MAC address.
>  On the other hand, with `MACAddressPolicy=persistent`, the MAC address
>  of the controller is fixed
>  and not inherited from the port.
>  
>  4) Run
>  
>     ip monitor link &
>     while : ; do
>       ip link del xxx
>       ip link add name xxx type dummy \
>       && ip link set xxx addr aa:00:00:00:00:00 \
>       && ip link show xxx | grep -q aa:00:00:00:00:00 \
>       || break
>     done
>  
>  to reproduce the race between a simple tool and udev changing the MAC address.
>  
>  
>  == User Experience ==
>  
>  Bond/bridge devices would again get assigned the MAC address of the
>  first NIC added to the device.
>  If we chose to not limit the scope of this change to just
>  bonds/bridges then all software devices would get randomly assigned
>  MAC addresses.
>  
>  
>  == Dependencies ==
>  
>  None.
>  
>  
>  == Contingency Plan ==
>  
>  If the change is rejected, nothing needs to be done. The change
>  itself will be simple to implement.
>  Contingency deadline: beta freeze
>  Blocks release? No
>  
>  
>  == Documentation ==
>  
>  TODO.
>  
>  == Release Notes ==
>   

-- 
Tom Hughes (tom(a)compton.nu)
http://compton.nu/

[Thomas Haller]

Hi,

On Mon, 2022-06-27 at 13:09 +0100, Tom Hughes via devel wrote:
>  
>  Twice now I have had to go and reconfigure my networks after a Fedora
>  upgrade has changed the MAC assignment policy. 
Interesting. Are you sure it was twice? I thought it changed "only"
once in F31 (2019).


Thomas

[Tom Hughes]

On 27/06/2022 17:05, Thomas Haller wrote:

>  On Mon, 2022-06-27 at 13:09 +0100, Tom Hughes via devel wrote:
> >
> > Twice now I have had to go and reconfigure my networks after a Fedora
> > upgrade has changed the MAC assignment policy.
>  
>  Interesting. Are you sure it was twice? I thought it changed "only"
>  once in F31 (2019). 
No it has just changed again in F36 at least for bridges, back
to what it was before the previous change I believe.

Tom

-- 
Tom Hughes (tom(a)compton.nu)
http://compton.nu/

[Tom Hughes]

On 27/06/2022 17:09, Tom Hughes via devel wrote:
>  On 27/06/2022 17:05, Thomas Haller wrote:
>  
> > On Mon, 2022-06-27 at 13:09 +0100, Tom Hughes via devel wrote:
> >>
> >> Twice now I have had to go and reconfigure my networks after a Fedora
> >> upgrade has changed the MAC assignment policy.
> >
> > Interesting. Are you sure it was twice? I thought it changed "only"
> > once in F31 (2019).
>  
>  No it has just changed again in F36 at least for bridges, back
>  to what it was before the previous change I believe. 
I've had a look through the git log for my DNS and taking one
machine as an example:

      initially - bridge had MAC of 00:b0:c2:02:4c:f3 from member
Aug 2015 (F22) - bridge changed to 1a:81:14:46:3c:c7
Jan 2020 (F31) - bridge changed to fe:e3:75:bd:6a:8c
May 2022 (F36) - bridge changed back to 1a:81:14:46:3c:c7

Only the first one corresponds to a member, so I think F22 is
where persistent addresses were first introduced. I'm not sure
what happened in F31 than then got reverted in F36.

Tom

-- 
Tom Hughes (tom(a)compton.nu)
http://compton.nu/

[Thomas Haller]

On Mon, 2022-06-27 at 17:26 +0100, Tom Hughes wrote:
>  On 27/06/2022 17:09, Tom Hughes via devel wrote:
>  > On 27/06/2022 17:05, Thomas Haller wrote:
>  > 
>  > > On Mon, 2022-06-27 at 13:09 +0100, Tom Hughes via devel wrote:
>  > > > 
>  > > > Twice now I have had to go and reconfigure my networks after a
>  > > > Fedora
>  > > > upgrade has changed the MAC assignment policy.
>  > > 
>  > > Interesting. Are you sure it was twice? I thought it changed
>  > > "only"
>  > > once in F31 (2019).
>  > 
>  > No it has just changed again in F36 at least for bridges, back
>  > to what it was before the previous change I believe.
>  
>  I've had a look through the git log for my DNS and taking one
>  machine as an example:
>  
>        initially - bridge had MAC of 00:b0:c2:02:4c:f3 from member
>  Aug 2015 (F22) - bridge changed to 1a:81:14:46:3c:c7
>  Jan 2020 (F31) - bridge changed to fe:e3:75:bd:6a:8c
>  May 2022 (F36) - bridge changed back to 1a:81:14:46:3c:c7
>  
>  Only the first one corresponds to a member, so I think F22 is
>  where persistent addresses were first introduced. I'm not sure
>  what happened in F31 than then got reverted in F36.
>  
>  Tom
>   

Hi Tom,


Thanks for checking.

I don't know why that would have happened. It also depends which tools
you are using for creating the bridge.

I agree, this is something to avoid! And maybe a the best reason to not
change anything on Fedora 37.


Thomas

[Kevin Kofler]

>  The `systemd-udev` package installs
>  `"/usr/lib/systemd/network/99-default.link"`,
>  which sets `Link.MACAddressPolicy=persistent`. This proposal is to
>  change it to set `Link.MACAddressPolicy=none` to stop changing the MAC
>  address. This is particularly important for bridge and bond devices.
>   
>  This change can either only apply to bridge/bond devices, or to
>  various software devices. That is to be discussed. 
IMHO, the default should be "none" for all devices. I do not see why systemd 
spoofs MAC addresses by default.

        Kevin Kofler

[Neal Gompa]

On Sat, Jun 25, 2022 at 3:06 PM Vipul Siddharth
<siddharthvipul1(a)gmail.com&gt; wrote:
>  This document represents a proposed Change. As part of the Changes
>  process, proposals are publicly announced in order to receive
>  community feedback. This proposal will only be implemented if approved
>  by the Fedora Engineering Steering Committee.
>
>
>  == Summary ==
>
>  The `systemd-udev` package installs
>  `"/usr/lib/systemd/network/99-default.link"`,
>  which sets `Link.MACAddressPolicy=persistent`. This proposal is to
>  change it to set `Link.MACAddressPolicy=none` to stop changing the MAC address.
>  This is particularly important for bridge and bond devices.
>
>  This change can either only apply to bridge/bond devices, or to
>  various software devices. That is to be discussed.
>
>  == Owner ==
>
>  * Name: [[User:thaller|Thomas Haller]] (NetworkManager)
>  * Email: <thaller(a)redhat.com&gt;
>  * Name: [[User:dustymabe| Dusty Mabe]] (Fedora CoreOS)
>  * Email: <dmabe(a)redhat.com&gt;
>
>
>  == Detailed Description ==
>
>  On Fedora, udev by default changes the MAC address of a wide range of
>  software devices.
>  This was introduced by systemd 242 in early 2019 (Fedora 31), when
>  `MACAddressPolicy=` was
>  extended to affect more types of devices.
>
>  With `MACAddressPolicy=persistent` udev's aim is to provide a stable
>  MAC address, otherwise
>  the kernel will assign a random one. However, that can cause problems:
>
>  Firstly, software devices are always created by some tool that has
>  plans for the device. The tool may not
>  expect that udev is going to change the MAC address and races against
>  that. The best solution
>  for the tool is to set the MAC address when creating an interface.
>  This will prevent
>  udev from changing the MAC address according to the MACAddressPolicy.
>  Otherwise, the tool should wait for udev to initialize the device to
>  avoid the race. In theory, a
>  tool is always advised to wait for udev to initialize the device.
>  However, if it were not for MACAddressPolicy,
>  in common scenarios udev doesn't do anything relevant for software
>  devices to make that necessary.
>
>  Secondly, for interface types bridge and bond, an unset MAC address
>  has a special meaning
>  to the kernel and the MAC address of the first port is used. If udev
>  changes the MAC
>  address, that no longer works.
>  Now the generated MAC address is not directly discoverable as it is
>  based on `/etc/machine-id`
>  ([https://www.man7.org/linux/man-pages/man5/machine-id.5.html
>  machine-id(5)]), among
>  other data. Even if there were a tool to easily calculate the MAC
>  address, it could be cumbersome
>  to use it without logging into the machine first. The MAC address can
>  directly affect the
>  assigned IP address, for example when using DHCP. When booting a new
>  virtual machine, the user might
>  know the MAC address of the (virtual) "physical" interfaces. When
>  bonding/bridging those
>  interfaces, the bond/bridge would get one of the well known MAC
>  addresses. `MACAddressPolicy=persistent`
>  interferes with that.
>
>  The goal of persistent policy is to provide a stable MAC address. Note
>  that if the
>  tool or user who created the interface would want a certain MAC address, they
>  have all the means to set it already. That applies regardless whether
>  the tool is
>  iproute2, NetworkManager, systemd-networkd. Neither NetworkManager nor
>  systemd-networkd
>  rely on udev's MACAddressPolicy for setting the MAC address. This
>  behavior is mostly
>  useful for plain `ip link add`, but it's unclear which real world user
>  wants this behavior.
>
>  Of course, the user is welcome to configure the MAC address in any way
>  they want. Including,
>  dropping a link file that sets `MACAddressPolicy=persistent`. The
>  problem is once udev sets a MAC address,
>  it cannot be unset. Which makes this problematic to do by default.
>
>  While Fedora inherited this behavior from upstream systemd, RHEL-9
>  does not follow this behavior
>
> ([https://gitlab.com/redhat/centos-stream/rpms/systemd/-/blob/c8953519504bf...
>  centos9],
>  [https://bugzilla.redhat.com/show_bug.cgi?id=1921094 rh#1921094]). For
>  RHEL-8, this doesn't
>  apply because the systemd there is too old to change the MAC address
>  of most software devices.
>
>  This could be either implemented by patching
>  `/usr/lib/systemd/network/99-default.link`
>  to have a different policy, or by dropping a link file with higher
>  priority. In the latter
>  case, that override could be shipped either by udev or even by NetworkManager.
>
>  Another option is to change the scope of this proposal to only change to
>  `MACAddressPolicy=none` for the device types where this causes the most issues
>  (bridge/bond/team).
>
>
>  == Feedback ==
>
>  This was also discussed on upstream systemd mailing list
>  [https://lists.freedesktop.org/archives/systemd-devel/2022-May/047893.html
>  here].
>  The upstream systemd maintainers' opinion is that the current udev
>  behavior is desirable, but accepts that distributions (or network
>  stacks such as NetworkManager) may choose to change the default
>  depending on their needs
>  ([https://lists.freedesktop.org/archives/systemd-devel/2022-May/047943.html
>  reference]).
>  The goal here is to find out what the Fedora community thinks is the
>  most appropriate default.
>
>  The RHEL-9 bug is [https://bugzilla.redhat.com/show_bug.cgi?id=1921094
>  [rh#1921094]].
>
>  
I'd be interested in seeing the justification for the revert in RHEL
9. From my perspective, it doesn't make sense to disable this because
it makes bonds/teams/etc. device dependent. This can be particularly
bad if you've got a bond and then you swap out the hardware over time.
The MAC address would change on the bond, which could break any layer
2 rules in place on the network.

The RHEL 9 bug seems to be private, though.


-- 
真実はいつも一つ！/ Always, there's only one truth!

[Chris Adams]

Once upon a time, Neal Gompa <ngompa13(a)gmail.com&gt; said:
>  I'd be interested in seeing the justification for the revert in
> RHEL
>  9. From my perspective, it doesn't make sense to disable this because
>  it makes bonds/teams/etc. device dependent. This can be particularly
>  bad if you've got a bond and then you swap out the hardware over time.
>  The MAC address would change on the bond, which could break any layer
>  2 rules in place on the network. 
The flip side of that is that converting a stand-alone interface to a
LAG changes MAC address on Linux, while it doesn't on most other things
like routers and switches.  Replacing a NIC is a more uncommon event
IMHO, which already would be disruptive to things that care about the
MAC in a non-LAG setup.

I don't care strongly either way myself; I've started copying the MAC
manually when I set up a LAG, but it would be nice to drop that step.

-- 
Chris Adams <linux(a)cmadams.net&gt;

[Dusty Mabe]

On 6/27/22 22:35, Neal Gompa wrote:
>  On Sat, Jun 25, 2022 at 3:06 PM Vipul Siddharth
>  <siddharthvipul1(a)gmail.com&gt; wrote:
> >
> > This document represents a proposed Change. As part of the Changes
> > process, proposals are publicly announced in order to receive
> > community feedback. This proposal will only be implemented if approved
> > by the Fedora Engineering Steering Committee.
> >
> >
> > == Summary ==
> >
> > The `systemd-udev` package installs
> > `"/usr/lib/systemd/network/99-default.link"`,
> > which sets `Link.MACAddressPolicy=persistent`. This proposal is to
> > change it to set `Link.MACAddressPolicy=none` to stop changing the MAC address.
> > This is particularly important for bridge and bond devices.
> >
> > This change can either only apply to bridge/bond devices, or to
> > various software devices. That is to be discussed.
> >
> > == Owner ==
> >
> > * Name: [[User:thaller|Thomas Haller]] (NetworkManager)
> > * Email: <thaller(a)redhat.com&gt;
> > * Name: [[User:dustymabe| Dusty Mabe]] (Fedora CoreOS)
> > * Email: <dmabe(a)redhat.com&gt;
> >
> >
> > == Detailed Description ==
> >
> > On Fedora, udev by default changes the MAC address of a wide range of
> > software devices.
> > This was introduced by systemd 242 in early 2019 (Fedora 31), when
> > `MACAddressPolicy=` was
> > extended to affect more types of devices.
> >
> > With `MACAddressPolicy=persistent` udev's aim is to provide a stable
> > MAC address, otherwise
> > the kernel will assign a random one. However, that can cause problems:
> >
> > Firstly, software devices are always created by some tool that has
> > plans for the device. The tool may not
> > expect that udev is going to change the MAC address and races against
> > that. The best solution
> > for the tool is to set the MAC address when creating an interface.
> > This will prevent
> > udev from changing the MAC address according to the MACAddressPolicy.
> > Otherwise, the tool should wait for udev to initialize the device to
> > avoid the race. In theory, a
> > tool is always advised to wait for udev to initialize the device.
> > However, if it were not for MACAddressPolicy,
> > in common scenarios udev doesn't do anything relevant for software
> > devices to make that necessary.
> >
> > Secondly, for interface types bridge and bond, an unset MAC address
> > has a special meaning
> > to the kernel and the MAC address of the first port is used. If udev
> > changes the MAC
> > address, that no longer works.
> > Now the generated MAC address is not directly discoverable as it is
> > based on `/etc/machine-id`
> > ([https://www.man7.org/linux/man-pages/man5/machine-id.5.html
> > machine-id(5)]), among
> > other data. Even if there were a tool to easily calculate the MAC
> > address, it could be cumbersome
> > to use it without logging into the machine first. The MAC address can
> > directly affect the
> > assigned IP address, for example when using DHCP. When booting a new
> > virtual machine, the user might
> > know the MAC address of the (virtual) "physical" interfaces. When
> > bonding/bridging those
> > interfaces, the bond/bridge would get one of the well known MAC
> > addresses. `MACAddressPolicy=persistent`
> > interferes with that.
> >
> > The goal of persistent policy is to provide a stable MAC address. Note
> > that if the
> > tool or user who created the interface would want a certain MAC address, they
> > have all the means to set it already. That applies regardless whether
> > the tool is
> > iproute2, NetworkManager, systemd-networkd. Neither NetworkManager nor
> > systemd-networkd
> > rely on udev's MACAddressPolicy for setting the MAC address. This
> > behavior is mostly
> > useful for plain `ip link add`, but it's unclear which real world user
> > wants this behavior.
> >
> > Of course, the user is welcome to configure the MAC address in any way
> > they want. Including,
> > dropping a link file that sets `MACAddressPolicy=persistent`. The
> > problem is once udev sets a MAC address,
> > it cannot be unset. Which makes this problematic to do by default.
> >
> > While Fedora inherited this behavior from upstream systemd, RHEL-9
> > does not follow this behavior
> >
> ([https://gitlab.com/redhat/centos-stream/rpms/systemd/-/blob/c8953519504bf...
> > centos9],
> > [https://bugzilla.redhat.com/show_bug.cgi?id=1921094 rh#1921094]). For
> > RHEL-8, this doesn't
> > apply because the systemd there is too old to change the MAC address
> > of most software devices.
> >
> > This could be either implemented by patching
> > `/usr/lib/systemd/network/99-default.link`
> > to have a different policy, or by dropping a link file with higher
> > priority. In the latter
> > case, that override could be shipped either by udev or even by NetworkManager.
> >
> > Another option is to change the scope of this proposal to only change to
> > `MACAddressPolicy=none` for the device types where this causes the most issues
> > (bridge/bond/team).
> >
> >
> > == Feedback ==
> >
> > This was also discussed on upstream systemd mailing list
> > [https://lists.freedesktop.org/archives/systemd-devel/2022-May/047893.html
> > here].
> > The upstream systemd maintainers' opinion is that the current udev
> > behavior is desirable, but accepts that distributions (or network
> > stacks such as NetworkManager) may choose to change the default
> > depending on their needs
> > ([https://lists.freedesktop.org/archives/systemd-devel/2022-May/047943.html
> > reference]).
> > The goal here is to find out what the Fedora community thinks is the
> > most appropriate default.
> >
> > The RHEL-9 bug is [https://bugzilla.redhat.com/show_bug.cgi?id=1921094
> > [rh#1921094]].
> >
> >
>  
>  I'd be interested in seeing the justification for the revert in RHEL
>  9. From my perspective, it doesn't make sense to disable this because
>  it makes bonds/teams/etc. device dependent. This can be particularly
>  bad if you've got a bond and then you swap out the hardware over time.
>  The MAC address would change on the bond, which could break any layer
>  2 rules in place on the network. 
Hmm. To me this feel more in line with a user's expectation. Let's draw
a parallel here to a single NIC (no bond) situation. Here are the two cases:

- single NIC
    - reinstalling your OS you don't need to update any external network env/config
    - the NIC fails
        - replace the NIC
        - update your external network env/configuration with the new MAC


- dual NIC (bond)
    - with MACAddressPolicy=persistent
         - reinstall OS you need to update external network env/config
         - a single NIC fails
             - replace the NIC
             - No need to update your external network env/config - no new MAC
    - with MACAddressPolicy=none
         - reinstall OS you DON'T need to update external network env/config
         - a single NIC fails
             - replace the NIC
             - update your external network env/configuration with the new MAC

To me the proposal to use MACAddressPolicy=none for bond/bridge/team gets us way more
inline with what users expect, which is that if they have to replace a NIC they have
to update something. I don't think users expect that if they re-install their OS
they'll need to update their network env/config.

I also suspect that an OS re-install is much much more common than a NIC failure
replacement.
 
> 
> The RHEL 9 bug seems to be private, though.
> 
>

[Dusty Mabe]

On 6/25/22 15:06, Vipul Siddharth wrote:
>  This document represents a proposed Change. As part of the Changes
>  process, proposals are publicly announced in order to receive
>  community feedback. This proposal will only be implemented if approved
>  by the Fedora Engineering Steering Committee.
>  
>  
>  == Summary ==
>  
>  The `systemd-udev` package installs
>  `"/usr/lib/systemd/network/99-default.link"`,
>  which sets `Link.MACAddressPolicy=persistent`. This proposal is to
>  change it to set `Link.MACAddressPolicy=none` to stop changing the MAC address.
>  This is particularly important for bridge and bond devices.
>  
>  This change can either only apply to bridge/bond devices, or to
>  various software devices. That is to be discussed. 
Based on the feedback here on the list we have modified the scope of this proposal
to now be limited to changing the MACAddressPolicy=none for bond/bridge/team devices
only.

New summary:

```
The systemd-udev package installs "/usr/lib/systemd/network/99-default.link",
which sets Link.MACAddressPolicy=persistent for all software NIC devices. This proposal is
to add to the policy so that we use Link.MACAddressPolicy=none for bond/bridge/team
devices. 
```

https://fedoraproject.org/wiki/Changes/MAC_Address_Policy_none

[Lennart Poettering]

On Di, 05.07.22 22:35, Dusty Mabe (dusty(a)dustymabe.com) wrote:

>  On 6/25/22 15:06, Vipul Siddharth wrote:
>  > This document represents a proposed Change. As part of the Changes
>  > process, proposals are publicly announced in order to receive
>  > community feedback. This proposal will only be implemented if approved
>  > by the Fedora Engineering Steering Committee.
>  >
>  >
>  > == Summary ==
>  >
>  > The `systemd-udev` package installs
>  > `"/usr/lib/systemd/network/99-default.link"`,
>  > which sets `Link.MACAddressPolicy=persistent`. This proposal is to
>  > change it to set `Link.MACAddressPolicy=none` to stop changing the MAC address.
>  > This is particularly important for bridge and bond devices.
>  >
>  > This change can either only apply to bridge/bond devices, or to
>  > various software devices. That is to be discussed.
>
>  Based on the feedback here on the list we have modified the scope of this proposal
>  to now be limited to changing the MACAddressPolicy=none for bond/bridge/team devices
> only.
>
>  New summary:
>
>  ```
>  The systemd-udev package installs "/usr/lib/systemd/network/99-default.link",
> which sets Link.MACAddressPolicy=persistent for all software NIC devices. This proposal is
> to add to the policy so that we use Link.MACAddressPolicy=none for bond/bridge/team
> devices.
>  ```
>
>  https://fedoraproject.org/wiki/Changes/MAC_Address_Policy_none 
So, with changing this back you'll also break all those setups where
it is assumed the bridge MAC just works and is stable and independent
from the devices added to it and the order in which they are added in.

What makes you so sure, that changing this *again* is so much better
than just leaving it the way it is now? This change isn't precisely
new, and changing stuff forth and back comes at quite a price.

Generally, if we change behaviour like this, then the pros must
seriously outweight the cons. Now you might argue that when the
original change was made that wasn't the case, that's a valid opinion,
though one I disagree with. But that has no effect on today, on the
question whether changing it again is worth it. I am pretty sure that
there are *also* numerous scripts that benefit from the predictability
and stability you get with the status quo, and which you'll break if
you revert to the old state – again.

I am not convinced we should flip flop on this all the time. Yes, it's
unfortunate that people tripped over this, but you are not really
making it better if you then break it *again*, given the benefit is
unclear, and the major software creating bridges/bonds/… doesn't care
anyway (i.e. NM, networkd, …).

I for one do not see where you'd get the crystal ball to look into to
know that by changing this *again* you'll make bazillions of people
happy, and only very few people sad, because you break their stuff. It
might very well be that you'll make more people sad because you break
their stuff again, than were happy before.

Also, let's not forget that allowing users to set the policy is a good
thing, we should let them. Given that the original change was already
made a lot of software that cannot work with such changes has already
been updated to override the default policy. That's a good thing,
since the overall system becomes more robust and people can more
safely change the policies locally, with less breakage to expect.

if you now revert to the status quo ante, then you basically also say:
fuck it, we don't care that software is fixed to work with changed
policies, let's keep things brittle that you cannot change policies
anymore effectively because we are sticking our head in the sand and
don't care that they are fixed.

So, I am not convinced.

I can accept that this wasn't handled particularly nicely
originally. My proposal for addressing this is via documentation, i.e
update the udev and iproute docs to explicitly point to the issue and
how to deal with it, with an example drop-in to make it easy to deal
with it.

Lennart

--
Lennart Poettering, Berlin

[Dusty Mabe]

On 7/6/22 04:37, Lennart Poettering wrote:
>  On Di, 05.07.22 22:35, Dusty Mabe (dusty(a)dustymabe.com) wrote:
>  
> > On 6/25/22 15:06, Vipul Siddharth wrote:
> >> This document represents a proposed Change. As part of the Changes
> >> process, proposals are publicly announced in order to receive
> >> community feedback. This proposal will only be implemented if approved
> >> by the Fedora Engineering Steering Committee.
> >>
> >>
> >> == Summary ==
> >>
> >> The `systemd-udev` package installs
> >> `"/usr/lib/systemd/network/99-default.link"`,
> >> which sets `Link.MACAddressPolicy=persistent`. This proposal is to
> >> change it to set `Link.MACAddressPolicy=none` to stop changing the MAC address.
> >> This is particularly important for bridge and bond devices.
> >>
> >> This change can either only apply to bridge/bond devices, or to
> >> various software devices. That is to be discussed.
> >
> > Based on the feedback here on the list we have modified the scope of this proposal
> > to now be limited to changing the MACAddressPolicy=none for bond/bridge/team devices
> only.
> >
> > New summary:
> >
> > ```
> > The systemd-udev package installs
> "/usr/lib/systemd/network/99-default.link", which sets
> Link.MACAddressPolicy=persistent for all software NIC devices. This proposal is to add to
> the policy so that we use Link.MACAddressPolicy=none for bond/bridge/team devices.
> > ```
> >
> > https://fedoraproject.org/wiki/Changes/MAC_Address_Policy_none
>  
>  So, with changing this back you'll also break all those setups where
>  it is assumed the bridge MAC just works and is stable and independent
>  from the devices added to it and the order in which they are added in. 
One thing we do need to think about here is the "upgrade" case. We could
consider leaving existing (upgrading) systems alone.

>  
>  What makes you so sure, that changing this *again* is so much better
>  than just leaving it the way it is now? This change isn't precisely
>  new, and changing stuff forth and back comes at quite a price. 
Indeed it does come at a price, which is why we're discussing the merits
here and also why we scaled back the proposal to just cover the use case
where it matters the most. For example, this kernel documentation is a good
place where a user is now confused because the kernel behaves one way for
bonds, but systemd delivers configuration to make it behave differently:
https://wiki.linuxfoundation.org/networking/bonding#where_does_a_bonding_...

If you look at the longer term landscape (i.e. years from now) the change
is worth it if it's the right change to make. I think the fact that RHEL9
didn't pick up the original change is an indication of it's value in enterprise
environments where bond/bridge/team devices are more common.

>  
>  Generally, if we change behaviour like this, then the pros must
>  seriously outweight the cons. Now you might argue that when the
>  original change was made that wasn't the case, that's a valid opinion,
>  though one I disagree with. But that has no effect on today, on the
>  question whether changing it again is worth it. I am pretty sure that
>  there are *also* numerous scripts that benefit from the predictability
>  and stability you get with the status quo, and which you'll break if
>  you revert to the old state – again. 
There is a lot that happens upstream systemd that we should more carefully
consider in Fedora. For example this probably should have been proposed to
Fedora as a change then (by systemd maintainer in Fedora I suspect) and the
merits examined at that point in time.

>  
>  I am not convinced we should flip flop on this all the time. Yes, it's
>  unfortunate that people tripped over this, but you are not really
>  making it better if you then break it *again*, given the benefit is
>  unclear, and the major software creating bridges/bonds/… doesn't care
>  anyway (i.e. NM, networkd, …). 
This definitely affects NetworkManager and people do care.. See
https://github.com/coreos/fedora-coreos-tracker/issues/919
https://github.com/systemd/systemd/issues/15208

>  
>  I for one do not see where you'd get the crystal ball to look into to
>  know that by changing this *again* you'll make bazillions of people
>  happy, and only very few people sad, because you break their stuff. It
>  might very well be that you'll make more people sad because you break
>  their stuff again, than were happy before. 
The discussion here is one way to gather feedback. There is also the collective
expertise of the members of FESCO, who make decisions like this all the time. 

>  
>  Also, let's not forget that allowing users to set the policy is a good
>  thing, we should let them. Given that the original change was already
>  made a lot of software that cannot work with such changes has already
>  been updated to override the default policy. That's a good thing,
>  since the overall system becomes more robust and people can more
>  safely change the policies locally, with less breakage to expect.
>  
>  if you now revert to the status quo ante, then you basically also say:
>  fuck it, we don't care that software is fixed to work with changed
>  policies, let's keep things brittle that you cannot change policies
>  anymore effectively because we are sticking our head in the sand and
>  don't care that they are fixed. 
I don't think that accurately reflects what we are saying here. I'm not
sure why making a change here makes things more brittle. Maybe it depends
on the implementation?

>  
>  So, I am not convinced.
>  
>  I can accept that this wasn't handled particularly nicely
>  originally. My proposal for addressing this is via documentation, i.e
>  update the udev and iproute docs to explicitly point to the issue and
>  how to deal with it, with an example drop-in to make it easy to deal
>  with it.

[Thomas Haller]

Hi,

After this change missed F37, it's now testable in F38:

https://src.fedoraproject.org/rpms/systemd/pull-request/100
systemd-253~rc2-3.fc38


Thanks Dusty, zbyszek!


Thomas