Re: OVS missing some interfaces on node reboot or NetworkManager retstart
by Fernando Fernandez Mancera
Hi Nobin,
just wanted to let you know that I backported the fix to nm-1-38,
nm-1-36 and nm-1-34 branches. You can check it here:
https://gitlab.freedesktop.org/NetworkManager/NetworkManager/-/tree/nm-1-34
Let me know if you need anything else.
Thanks,
Fernando.
On Mon, Aug 29, 2022 at 11:08 AM Nobin Mathew <nobin.mathew(a)ericsson.com> wrote:
>
> Thanks Fernando for the quick response...
>
> It will be great if you can provide a backported patch for 1.34, otherwise please share the patch which fixed this problem, I will try to backport.
>
> -Nobin
>
> -----Original Message-----
> From: Fernando Fernandez Mancera <ferferna(a)redhat.com>
> Sent: Monday, August 29, 2022 1:06 PM
> To: Nobin Mathew <nobin.mathew(a)ericsson.com>
> Cc: nmstate-devel(a)lists.fedorahosted.org
> Subject: Re: [nmstate-devel] OVS missing some interfaces on node reboot or NetworkManager retstart
>
> Hi!
>
> On Mon, Aug 29, 2022 at 7:36 AM Nobin Mathew <nobin.mathew(a)ericsson.com> wrote:
> >
> > Hi,
> >
> > We r running
> >
> > ######
> >
> > /home/ # NetworkManager -V
> >
> > 1.34.0
> >
> > /home/ # ovs-vsctl -V
> >
> > ovs-vsctl (Open vSwitch) 2.14.2
> >
> > DB Schema 8.2.0
> >
> >
> >
> > /home/ # uname -a
> >
> > Linux 5.3.18-150300.59.87-default SMP Thu Jul 21 14:31:28 UTC 2022
> > (cc90276) x86_64 x86_64 x86_64 GNU/Linux
> >
> >
> >
> > this is SLES15 sp3 machine.
> >
> >
> >
> > pool1-bm25c5:~> nmstatectl --version
> >
> > 1.2.0
> >
> > pool1-bm25c5:~>
> >
> >
> >
> > ######
> >
> > when we reboot node or restart NetworkManager(systemctl restart NetworkManager) we are loosing some of the ovs interfaces (randomly, not on all restarts).
> >
>
> I think this issue is related to this bug:
> https://bugzilla.redhat.com/show_bug.cgi?id=2080236
>
> It is a NetworkManager issue and it was fixed in NetworkManager-1.39.6, could you check it out? Would you need an upstream backport to 1.34?
>
> Thanks!
> Fernando.
>
> > logs are here:
> >
> > Missing interfaces:
> >
> > https://protect2.fireeye.com/v1/url?k=31323334-501d5122-313273af-45444
> > 5555731-93530804397fe1ff&q=1&e=f9884072-555f-4847-9f8b-466f9b806b2a&u=
> > https%3A%2F%2Fgist.github.com%2Fnmathew%2Fd7d472b938f389c4efbadd01ef78
> > d466%23file-missing-interfaces
> >
> > NetworkManager trace log for restart duration
> >
> > https://protect2.fireeye.com/v1/url?k=31323334-501d5122-313273af-45444
> > 5555731-cfdd31dcdea77648&q=1&e=f9884072-555f-4847-9f8b-466f9b806b2a&u=
> > https%3A%2F%2Fgist.github.com%2Fnmathew%2Fd7d472b938f389c4efbadd01ef78
> > d466%23file-networkmanager-logs-with-trace
> >
> > /etc/NetworkManager/system-connections
> >
> > https://protect2.fireeye.com/v1/url?k=31323334-501d5122-313273af-45444
> > 5555731-3474c8a3dfe835ee&q=1&e=f9884072-555f-4847-9f8b-466f9b806b2a&u=
> > https%3A%2F%2Fgist.github.com%2Fnmathew%2Fd7d472b938f389c4efbadd01ef78
> > d466%23file-connection-conf
> >
> > ovsdb-client-dump for non working case(or when interfaces are missing)
> >
> > https://protect2.fireeye.com/v1/url?k=31323334-501d5122-313273af-45444
> > 5555731-289bfcbe422b0355&q=1&e=f9884072-555f-4847-9f8b-466f9b806b2a&u=
> > https%3A%2F%2Fgist.github.com%2Fnmathew%2Fd7d472b938f389c4efbadd01ef78
> > d466%23file-non-working-case-ovsdb-client-dump
> >
> > ovsdb-client-dump for working case(or when all interfaces are present)
> >
> > https://protect2.fireeye.com/v1/url?k=31323334-501d5122-313273af-45444
> > 5555731-b1efb31f6496469e&q=1&e=f9884072-555f-4847-9f8b-466f9b806b2a&u=
> > https%3A%2F%2Fgist.github.com%2Fnmathew%2Fd7d472b938f389c4efbadd01ef78
> > d466%23file-on-working-ovsdb-client-dump
> >
> > ovs-vsctl-show when interfaces are missing
> >
> > https://protect2.fireeye.com/v1/url?k=31323334-501d5122-313273af-45444
> > 5555731-75a8fd4d20fe10f6&q=1&e=f9884072-555f-4847-9f8b-466f9b806b2a&u=
> > https%3A%2F%2Fgist.github.com%2Fnmathew%2Fd7d472b938f389c4efbadd01ef78
> > d466%23file-ovs-vsctl-show
> >
> > ovs-vswitchd-log for the restart duration
> >
> > https://protect2.fireeye.com/v1/url?k=31323334-501d5122-313273af-45444
> > 5555731-8c324322ae3a8e8f&q=1&e=f9884072-555f-4847-9f8b-466f9b806b2a&u=
> > https%3A%2F%2Fgist.github.com%2Fnmathew%2Fd7d472b938f389c4efbadd01ef78
> > d466%23file-ovs-vswitchd-log
> >
> > ovsdb-tool-show-log for the restart duration
> >
> > https://protect2.fireeye.com/v1/url?k=31323334-501d5122-313273af-45444
> > 5555731-c5b663cb0344f371&q=1&e=f9884072-555f-4847-9f8b-466f9b806b2a&u=
> > https%3A%2F%2Fgist.github.com%2Fnmathew%2Fd7d472b938f389c4efbadd01ef78
> > d466%23file-sudo-ovsdb-tool-mmmmmm-show-log
> >
> > In this particular instance of run, we lost two interfaces.
> >
> > we have a script which sets "autoconnect-slaves" to 0, which was introduced for an older version of NM. grep -lF -e type=ovs-bridge -e type=ovs-interface -e type=ovs-port /etc/NetworkManager/system-connections/*|xargs -I % nmcli connection modify filename % connection.autoconnect-slaves 0 this scripts also runs currently, that why all "autoconnect-slaves" set to zero.
> >
> > when we are losing interfaces that port additions are not coming to ovsdb:
> >
> > 2022-08-26T22:38:26.176Z|00179|bridge|INFO|bridge data_bridge: added
> > interface traffic5_iface on port 1
> >
> > 2022-08-26T22:38:26.348Z|00182|bridge|INFO|bridge data_bridge: added
> > interface traffic1_iface on port 2
> >
> > 2022-08-26T22:38:26.376Z|00183|bridge|INFO|bridge data_bridge: added
> > interface bm25c_ext_iface on port 3
> >
> > 2022-08-26T22:38:26.376Z|00184|bridge|INFO|bridge data_bridge: added
> > interface bond0 on port 4
> >
> > Only 4 interfaces are getting added back, two are missing, "ovsdb-tool -mmmmmm show-log " shows the same.
> >
> > Any pointers where it is going wrong?
> >
> > -Nobin
> >
> >
> >
> > _______________________________________________
> > nmstate-devel mailing list -- nmstate-devel(a)lists.fedorahosted.org
> > To unsubscribe send an email to
> > nmstate-devel-leave(a)lists.fedorahosted.org
> > Fedora Code of Conduct:
> > https://docs.fedoraproject.org/en-US/project/code-of-conduct/
> > List Guidelines:
> > https://fedoraproject.org/wiki/Mailing_list_guidelines
> > List Archives:
> > https://protect2.fireeye.com/v1/url?k=31323334-501d5122-313273af-45444
> > 5555731-09b7aeb775896a89&q=1&e=f9884072-555f-4847-9f8b-466f9b806b2a&u=
> > https%3A%2F%2Flists.fedorahosted.org%2Farchives%2Flist%2Fnmstate-devel
> > %40lists.fedorahosted.org Do not reply to spam, report it:
> > https://protect2.fireeye.com/v1/url?k=31323334-501d5122-313273af-45444
> > 5555731-1df4bd1e5f7cb906&q=1&e=f9884072-555f-4847-9f8b-466f9b806b2a&u=
> > https%3A%2F%2Fpagure.io%2Ffedora-infrastructure%2Fnew_issue
