Stacked ifaces teardown results in kernel locking

[Mariusz Mazur]

I have three team interfaces:
- sw1 and sw2, both are three-NIC lacps
- and an activebackup called team0 (I'm open to better naming suggestions
:) on top of sw1+sw2.

Here is how vlan interface teardown looks like using the unstacked sw1
interface:

[root@vc1n3 ~]# systemctl start teamd@sw1
[root@vc1n3 ~]# ip l s sw1 up
[root@vc1n3 ~]# ip link add link sw1 name lan.246 type vlan id 246
[root@vc1n3 ~]# ip l s lan.246 up
[root@vc1n3 ~]# systemctl stop teamd@sw1
[root@vc1n3 ~]#

Everything went fine, both sw1 and lan.246@sw1 are now gone.

Now let's try this with stacked team interfaces:

[root@vc1n3 ~]# systemctl start teamd@sw1 teamd@sw2
[root@vc1n3 ~]# systemctl start teamd@team0
[root@vc1n3 ~]# ip l s team0 up
[root@vc1n3 ~]# ip link add link team0 name lan.246 type vlan id 246
[root@vc1n3 ~]# ip l s lan.246 up
[root@vc1n3 ~]# systemctl stop teamd@team0

… and it freezes. I can't tell you what the state of the network interfaces
is, because running 'ip link show' also freezes. The teamd process becomes
unkillable. Only thing left to do now is to hard reset the system, cause I
can't do a proper shutdown in any shape or form.


Some info:
- libteam git master from today
- kernel 4.7.0
- full teamd configs are at the very end of this email


Logs:
Aug 09 08:54:58 vc1n3 polkitd[10340]: Registered Authentication Agent for
unix-process:11455:5149256 (system bus name :1.369 [/usr/bin/pkttyagent
--notify-fd 5 --fallback], object path
/org/freedesktop/PolicyKit1/AuthenticationAgent, locale C.utf-8)
Aug 09 08:54:58 vc1n3 kernel: team0: Port device sw2 removed
Aug 09 08:54:58 vc1n3 systemd[1]: Stopping Team Daemon for device team0...
Aug 09 08:54:58 vc1n3 teamd[11418]: Got SIGINT, SIGQUIT or SIGTERM.
Aug 09 08:55:35 vc1n3 polkitd[10340]: Unregistered Authentication Agent for
unix-process:11455:5149256 (system bus name :1.369, object path
/org/freedesktop/PolicyKit1/AuthenticationAgent, locale C.utf-8)
(disconnected from bus)
Aug 09 08:56:01 vc1n3 systemd[1]: Started Session c163 of user root.
Aug 09 08:56:01 vc1n3 systemd[1]: Starting Session c163 of user root.
Aug 09 08:56:01 vc1n3 CROND[11489]: (root) CMD
(/usr/bin/update_axeos_certificates --quiet)
Aug 09 08:56:28 vc1n3 systemd[1]: teamd(a)team0.service: State 'stop-sigterm'
timed out. Killing.
Aug 09 08:57:58 vc1n3 systemd[1]: teamd(a)team0.service: Processes still
around after SIGKILL. Ignoring.
Aug 09 08:58:46 vc1n3 kernel: INFO: task snmpd:2217 blocked for more than
120 seconds.
Aug 09 08:58:46 vc1n3 kernel:       Tainted: G           O    4.7.0-aos0.1
#1
Aug 09 08:58:46 vc1n3 kernel: "echo 0 >
/proc/sys/kernel/hung_task_timeout_secs" disables this message.
Aug 09 08:58:46 vc1n3 kernel: snmpd           D ffff88107fc96800     0
2217      1 0x00000000
Aug 09 08:58:46 vc1n3 kernel:  ffff88085b0c2580 0000000000000020
ffff881052e62480 ffff88085ac5c000
Aug 09 08:58:46 vc1n3 kernel:  ffff88085ac5bdc0 ffffffff81889624
ffff88085b0c2580 0000000000000246
Aug 09 08:58:46 vc1n3 kernel:  00000000ffffffff ffffffff8149831c
ffffffff81889620 ffffffff814985b5
Aug 09 08:58:46 vc1n3 kernel: Call Trace:
Aug 09 08:58:46 vc1n3 kernel:  [<ffffffff8149831c>] ? schedule+0x2c/0x80
Aug 09 08:58:46 vc1n3 kernel:  [<ffffffff814985b5>] ?
schedule_preempt_disabled+0x5/0x10
Aug 09 08:58:46 vc1n3 kernel:  [<ffffffff8149a60c>] ?
__mutex_lock_slowpath+0x10c/0x300
Aug 09 08:58:46 vc1n3 kernel:  [<ffffffff8149a809>] ? mutex_lock+0x9/0x20
Aug 09 08:58:46 vc1n3 kernel:  [<ffffffff813d6697>] ? dev_ioctl+0x2e7/0x570
Aug 09 08:58:46 vc1n3 kernel:  [<ffffffff811971f4>] ? clear_inode+0x14/0x80
Aug 09 08:58:46 vc1n3 kernel:  [<ffffffff810e848a>] ?
cgroup_sk_alloc+0x2a/0x40
Aug 09 08:58:46 vc1n3 kernel:  [<ffffffff813a20e6>] ?
sock_do_ioctl+0x36/0x40
Aug 09 08:58:46 vc1n3 kernel:  [<ffffffff813a253e>] ? sock_ioctl+0x19e/0x240
Aug 09 08:58:46 vc1n3 kernel:  [<ffffffff8118fe94>] ?
do_vfs_ioctl+0x84/0x5a0
Aug 09 08:58:46 vc1n3 kernel:  [<ffffffff81181076>] ? alloc_file+0x16/0xa0
Aug 09 08:58:46 vc1n3 kernel:  [<ffffffff813a1fc5>] ?
sock_alloc_file+0x95/0x110
Aug 09 08:58:46 vc1n3 kernel:  [<ffffffff81190421>] ? SyS_ioctl+0x71/0x80
Aug 09 08:58:46 vc1n3 kernel:  [<ffffffff813a4653>] ? SyS_socket+0x63/0xa0
Aug 09 08:58:46 vc1n3 kernel:  [<ffffffff8149c576>] ?
entry_SYSCALL_64_fastpath+0x1e/0xa8
Aug 09 08:58:46 vc1n3 kernel: INFO: task kworker/3:0:11165 blocked for more
than 120 seconds.
Aug 09 08:58:46 vc1n3 kernel:       Tainted: G           O    4.7.0-aos0.1
#1
Aug 09 08:58:46 vc1n3 kernel: "echo 0 >
/proc/sys/kernel/hung_task_timeout_secs" disables this message.
Aug 09 08:58:46 vc1n3 kernel: kworker/3:0     D ffff88085fd96800     0
11165      2 0x00000000
Aug 09 08:58:46 vc1n3 kernel: Workqueue: ipv6_addrconf addrconf_verify_work
Aug 09 08:58:46 vc1n3 kernel:  ffff88085ade8000 0000000000000000
0000000000000000 ffff8808557d4000
Aug 09 08:58:46 vc1n3 kernel:  ffff8808557d3e18 ffffffff81889624
ffff88085ade8000 0000000000000246
Aug 09 08:58:46 vc1n3 kernel:  00000000ffffffff ffffffff8149831c
ffffffff81889620 ffffffff814985b5
Aug 09 08:58:46 vc1n3 kernel: Call Trace:
Aug 09 08:58:46 vc1n3 kernel:  [<ffffffff8149831c>] ? schedule+0x2c/0x80
Aug 09 08:58:46 vc1n3 kernel:  [<ffffffff814985b5>] ?
schedule_preempt_disabled+0x5/0x10
Aug 09 08:58:46 vc1n3 kernel:  [<ffffffff8149a60c>] ?
__mutex_lock_slowpath+0x10c/0x300
Aug 09 08:58:46 vc1n3 kernel:  [<ffffffff810286f4>] ?
__switch_to+0x244/0x570
Aug 09 08:58:46 vc1n3 kernel:  [<ffffffff8149a809>] ? mutex_lock+0x9/0x20
Aug 09 08:58:46 vc1n3 kernel:  [<ffffffff8145e9f5>] ?
addrconf_verify_work+0x5/0x10
Aug 09 08:58:46 vc1n3 kernel:  [<ffffffff81086de5>] ?
process_one_work+0x145/0x3d0
Aug 09 08:58:46 vc1n3 kernel:  [<ffffffff810870d0>] ?
worker_thread+0x60/0x4a0
Aug 09 08:58:46 vc1n3 kernel:  [<ffffffff81497e67>] ? __schedule+0x1e7/0x670
Aug 09 08:58:46 vc1n3 kernel:  [<ffffffff81087070>] ?
process_one_work+0x3d0/0x3d0
Aug 09 08:58:46 vc1n3 kernel:  [<ffffffff8108beea>] ? kthread+0xca/0xe0
Aug 09 08:58:46 vc1n3 kernel:  [<ffffffff8149c79f>] ?
ret_from_fork+0x1f/0x40
Aug 09 08:58:46 vc1n3 kernel:  [<ffffffff8108be20>] ?
kthread_create_on_node+0x180/0x180
Aug 09 08:58:46 vc1n3 kernel: INFO: task teamd:11385 blocked for more than
120 seconds.
Aug 09 08:58:46 vc1n3 kernel:       Tainted: G           O    4.7.0-aos0.1
#1
Aug 09 08:58:46 vc1n3 kernel: "echo 0 >
/proc/sys/kernel/hung_task_timeout_secs" disables this message.
Aug 09 08:58:46 vc1n3 kernel: teamd           D ffff88107fd16800     0
11385      1 0x00000000
Aug 09 08:58:46 vc1n3 kernel:  ffff88105aed3e80 ffff88085b8a1600
00000000000000db ffff8810545f4000
Aug 09 08:58:46 vc1n3 kernel:  ffff8810545f3c78 ffffffff81889624
ffff88105aed3e80 0000000000000246
Aug 09 08:58:46 vc1n3 kernel:  00000000ffffffff ffffffff8149831c
ffffffff81889620 ffffffff814985b5
Aug 09 08:58:46 vc1n3 kernel: Call Trace:
Aug 09 08:58:46 vc1n3 kernel:  [<ffffffff8149831c>] ? schedule+0x2c/0x80
Aug 09 08:58:46 vc1n3 kernel:  [<ffffffff814985b5>] ?
schedule_preempt_disabled+0x5/0x10
Aug 09 08:58:46 vc1n3 kernel:  [<ffffffff8149a60c>] ?
__mutex_lock_slowpath+0x10c/0x300
Aug 09 08:58:46 vc1n3 kernel:  [<ffffffff81190dc0>] ?
poll_select_copy_remaining+0x120/0x120
Aug 09 08:58:46 vc1n3 kernel:  [<ffffffff8149a809>] ? mutex_lock+0x9/0x20
Aug 09 08:58:46 vc1n3 kernel:  [<ffffffff813cfa00>] ?
rtnetlink_rcv+0x10/0x30
Aug 09 08:58:46 vc1n3 kernel:  [<ffffffff813ed17a>] ?
netlink_unicast+0x14a/0x1e0
Aug 09 08:58:46 vc1n3 kernel:  [<ffffffff813ed609>] ?
netlink_sendmsg+0x349/0x3a0
Aug 09 08:58:46 vc1n3 kernel:  [<ffffffff813a3b5b>] ? sock_sendmsg+0x2b/0x40
Aug 09 08:58:46 vc1n3 kernel:  [<ffffffff813a43dd>] ?
___sys_sendmsg+0x24d/0x260
Aug 09 08:58:46 vc1n3 kernel:  [<ffffffff813a4538>] ?
___sys_recvmsg+0x148/0x1c0
Aug 09 08:58:46 vc1n3 kernel:  [<ffffffff811bdc34>] ?
ep_send_events_proc+0x74/0x170
Aug 09 08:58:46 vc1n3 kernel:  [<ffffffff8149a3bb>] ?
__mutex_unlock_slowpath+0xbb/0x160
Aug 09 08:58:46 vc1n3 kernel:  [<ffffffff811be517>] ? ep_poll+0x127/0x350
Aug 09 08:58:46 vc1n3 kernel:  [<ffffffff813a519c>] ?
__sys_sendmsg+0x3c/0x70
Aug 09 08:58:46 vc1n3 kernel:  [<ffffffff8149c576>] ?
entry_SYSCALL_64_fastpath+0x1e/0xa8
Aug 09 08:58:46 vc1n3 kernel: INFO: task teamd:11386 blocked for more than
120 seconds.
Aug 09 08:58:46 vc1n3 kernel:       Tainted: G           O    4.7.0-aos0.1
#1
Aug 09 08:58:46 vc1n3 kernel: "echo 0 >
/proc/sys/kernel/hung_task_timeout_secs" disables this message.
Aug 09 08:58:46 vc1n3 kernel: teamd           D ffff88107fc96800     0
11386      1 0x00000000
Aug 09 08:58:46 vc1n3 kernel:  ffff88105aed7080 ffff88085abf1f00
00000000000000db ffff881057ff4000
Aug 09 08:58:46 vc1n3 kernel:  ffff881057ff3c78 ffffffff81889624
ffff88105aed7080 0000000000000246
Aug 09 08:58:46 vc1n3 kernel:  00000000ffffffff ffffffff8149831c
ffffffff81889620 ffffffff814985b5
Aug 09 08:58:46 vc1n3 kernel: Call Trace:
Aug 09 08:58:46 vc1n3 kernel:  [<ffffffff8149831c>] ? schedule+0x2c/0x80
Aug 09 08:58:46 vc1n3 kernel:  [<ffffffff814985b5>] ?
schedule_preempt_disabled+0x5/0x10
Aug 09 08:58:46 vc1n3 kernel:  [<ffffffff8149a60c>] ?
__mutex_lock_slowpath+0x10c/0x300
Aug 09 08:58:46 vc1n3 kernel:  [<ffffffff81190dc0>] ?
poll_select_copy_remaining+0x120/0x120
Aug 09 08:58:46 vc1n3 kernel:  [<ffffffff8149a809>] ? mutex_lock+0x9/0x20
Aug 09 08:58:46 vc1n3 kernel:  [<ffffffff813cfa00>] ?
rtnetlink_rcv+0x10/0x30
Aug 09 08:58:46 vc1n3 kernel:  [<ffffffff813ed17a>] ?
netlink_unicast+0x14a/0x1e0
Aug 09 08:58:46 vc1n3 kernel:  [<ffffffff813ed609>] ?
netlink_sendmsg+0x349/0x3a0
Aug 09 08:58:46 vc1n3 kernel:  [<ffffffff813a3b5b>] ? sock_sendmsg+0x2b/0x40
Aug 09 08:58:46 vc1n3 kernel:  [<ffffffff813a43dd>] ?
___sys_sendmsg+0x24d/0x260
Aug 09 08:58:46 vc1n3 kernel:  [<ffffffff813a4538>] ?
___sys_recvmsg+0x148/0x1c0
Aug 09 08:58:46 vc1n3 kernel:  [<ffffffff811bdc34>] ?
ep_send_events_proc+0x74/0x170
Aug 09 08:58:46 vc1n3 kernel:  [<ffffffff8149a3bb>] ?
__mutex_unlock_slowpath+0xbb/0x160
Aug 09 08:58:46 vc1n3 kernel:  [<ffffffff8109b627>] ? update_curr+0xa7/0x130
Aug 09 08:58:46 vc1n3 kernel:  [<ffffffff810a2e72>] ?
pick_next_task_fair+0x2c2/0x490
Aug 09 08:58:46 vc1n3 kernel:  [<ffffffff813a519c>] ?
__sys_sendmsg+0x3c/0x70
Aug 09 08:58:46 vc1n3 kernel:  [<ffffffff8149c576>] ?
entry_SYSCALL_64_fastpath+0x1e/0xa8
Aug 09 08:58:46 vc1n3 kernel: INFO: task teamd:11418 blocked for more than
120 seconds.
Aug 09 08:58:46 vc1n3 kernel:       Tainted: G           O    4.7.0-aos0.1
#1
Aug 09 08:58:46 vc1n3 kernel: "echo 0 >
/proc/sys/kernel/hung_task_timeout_secs" disables this message.
Aug 09 08:58:46 vc1n3 kernel: teamd           D ffff88107fc16800     0
11418      1 0x00000004
Aug 09 08:58:46 vc1n3 kernel:  ffff88105af10000 0000000000000001
0000000000000000 ffff881057fe4000
Aug 09 08:58:46 vc1n3 kernel:  ffff881057fe3750 ffff88085560f814
ffff88105af10000 0000000000000246
Aug 09 08:58:46 vc1n3 kernel:  00000000ffffffff ffffffff8149831c
ffff88085560f810 ffffffff814985b5
Aug 09 08:58:46 vc1n3 kernel: Call Trace:
Aug 09 08:58:46 vc1n3 kernel:  [<ffffffff8149831c>] ? schedule+0x2c/0x80
Aug 09 08:58:46 vc1n3 kernel:  [<ffffffff814985b5>] ?
schedule_preempt_disabled+0x5/0x10
Aug 09 08:58:46 vc1n3 kernel:  [<ffffffff8149a60c>] ?
__mutex_lock_slowpath+0x10c/0x300
Aug 09 08:58:46 vc1n3 kernel:  [<ffffffff81429f63>] ?
inetdev_event+0x23/0x4b0
Aug 09 08:58:46 vc1n3 kernel:  [<ffffffff8149a809>] ? mutex_lock+0x9/0x20
Aug 09 08:58:46 vc1n3 kernel:  [<ffffffffa0828e34>] ?
team_device_event+0xe4/0x110 [team]
Aug 09 08:58:46 vc1n3 kernel:  [<ffffffff8108cc00>] ?
notifier_call_chain+0x40/0x60
Aug 09 08:58:46 vc1n3 kernel:  [<ffffffff813c1ef1>] ?
netdev_update_features+0x21/0x30
Aug 09 08:58:46 vc1n3 kernel:  [<ffffffff813c1c72>] ?
__netdev_update_features+0x2e2/0x540
Aug 09 08:58:46 vc1n3 kernel:  [<ffffffff813bcda7>] ? netdev_info+0x57/0x60
Aug 09 08:58:46 vc1n3 kernel:  [<ffffffff813c204d>] ?
netdev_change_features+0xd/0x30
Aug 09 08:58:46 vc1n3 kernel:  [<ffffffffa0828cde>] ?
team_port_del+0x1ae/0x1e0 [team]
Aug 09 08:58:46 vc1n3 kernel:  [<ffffffffa0828d38>] ?
team_del_slave+0x28/0x40 [team]
Aug 09 08:58:46 vc1n3 kernel:  [<ffffffff813d0650>] ? do_setlink+0x320/0xb60
Aug 09 08:58:46 vc1n3 kernel:  [<ffffffff8149a3bb>] ?
__mutex_unlock_slowpath+0xbb/0x160
Aug 09 08:58:46 vc1n3 kernel:  [<ffffffff811be3bd>] ?
ep_scan_ready_list+0x1ad/0x1c0
Aug 09 08:58:46 vc1n3 kernel:  [<ffffffff811be3d0>] ?
ep_scan_ready_list+0x1c0/0x1c0
Aug 09 08:58:46 vc1n3 kernel:  [<ffffffff811bdfc2>] ?
ep_call_nested.constprop.10+0xe2/0x100
Aug 09 08:58:46 vc1n3 kernel:  [<ffffffff813d15a1>] ?
rtnl_newlink+0x621/0x870
Aug 09 08:58:46 vc1n3 kernel:  [<ffffffff813d1084>] ?
rtnl_newlink+0x104/0x870
Aug 09 08:58:46 vc1n3 kernel:  [<ffffffff813d1c01>] ?
rtnl_getlink+0x121/0x1a0
Aug 09 08:58:46 vc1n3 kernel:  [<ffffffff813cfaa8>] ?
rtnetlink_rcv_msg+0x88/0x210
Aug 09 08:58:46 vc1n3 kernel:  [<ffffffff81177b7b>] ?
__kmalloc_node_track_caller+0x2b/0x210
Aug 09 08:58:46 vc1n3 kernel:  [<ffffffff813cfa20>] ?
rtnetlink_rcv+0x30/0x30
Aug 09 08:58:46 vc1n3 kernel:  [<ffffffff813ed82c>] ?
netlink_rcv_skb+0x9c/0xc0
Aug 09 08:58:46 vc1n3 kernel:  [<ffffffff813cfa0f>] ?
rtnetlink_rcv+0x1f/0x30
Aug 09 08:58:46 vc1n3 kernel:  [<ffffffff813ed17a>] ?
netlink_unicast+0x14a/0x1e0
Aug 09 08:58:46 vc1n3 kernel:  [<ffffffff813ed609>] ?
netlink_sendmsg+0x349/0x3a0
Aug 09 08:58:46 vc1n3 kernel:  [<ffffffff813a3b5b>] ? sock_sendmsg+0x2b/0x40
Aug 09 08:58:46 vc1n3 kernel:  [<ffffffff813a43dd>] ?
___sys_sendmsg+0x24d/0x260
Aug 09 08:58:46 vc1n3 kernel:  [<ffffffff811978d4>] ?
generic_update_time+0x74/0xc0
Aug 09 08:58:46 vc1n3 kernel:  [<ffffffff81198f69>] ? touch_atime+0x99/0xa0
Aug 09 08:58:46 vc1n3 kernel:  [<ffffffff811871fa>] ? pipe_read+0x2ca/0x300
Aug 09 08:58:46 vc1n3 kernel:  [<ffffffff8117e4fa>] ? __vfs_read+0xba/0x110
Aug 09 08:58:46 vc1n3 kernel:  [<ffffffff813a519c>] ?
__sys_sendmsg+0x3c/0x70
Aug 09 08:58:46 vc1n3 kernel:  [<ffffffff8149c576>] ?
entry_SYSCALL_64_fastpath+0x1e/0xa8
Aug 09 08:59:28 vc1n3 systemd[1]: teamd(a)team0.service: State
'stop-final-sigterm' timed out. Killing.
Aug 09 09:00:01 vc1n3 systemd[1]: Started Session c164 of user root.
Aug 09 09:00:01 vc1n3 CROND[11542]: (root) CMD (exec /usr/lib/sa/sa1 600 6)
Aug 09 09:00:01 vc1n3 systemd[1]: Starting Session c164 of user root.
Aug 09 09:00:01 vc1n3 CROND[11543]: (root) CMD
(/usr/bin/update_axeos_certificates --quiet)
Aug 09 09:00:01 vc1n3 systemd[1]: Started Session c165 of user root.
Aug 09 09:00:01 vc1n3 systemd[1]: Starting Session c165 of user root.
Aug 09 09:00:46 vc1n3 kernel: INFO: task snmpd:2217 blocked for more than
120 seconds.
Aug 09 09:00:46 vc1n3 kernel:       Tainted: G           O    4.7.0-aos0.1
#1
Aug 09 09:00:46 vc1n3 kernel: "echo 0 >
/proc/sys/kernel/hung_task_timeout_secs" disables this message.
Aug 09 09:00:46 vc1n3 kernel: snmpd           D ffff88107fc96800     0
2217      1 0x00000000
Aug 09 09:00:46 vc1n3 kernel:  ffff88085b0c2580 0000000000000020
ffff881052e62480 ffff88085ac5c000
Aug 09 09:00:46 vc1n3 kernel:  ffff88085ac5bdc0 ffffffff81889624
ffff88085b0c2580 0000000000000246
Aug 09 09:00:46 vc1n3 kernel:  00000000ffffffff ffffffff8149831c
ffffffff81889620 ffffffff814985b5
Aug 09 09:00:46 vc1n3 kernel: Call Trace:
Aug 09 09:00:46 vc1n3 kernel:  [<ffffffff8149831c>] ? schedule+0x2c/0x80
Aug 09 09:00:46 vc1n3 kernel:  [<ffffffff814985b5>] ?
schedule_preempt_disabled+0x5/0x10
Aug 09 09:00:46 vc1n3 kernel:  [<ffffffff8149a60c>] ?
__mutex_lock_slowpath+0x10c/0x300
Aug 09 09:00:46 vc1n3 kernel:  [<ffffffff8149a809>] ? mutex_lock+0x9/0x20
Aug 09 09:00:46 vc1n3 kernel:  [<ffffffff813d6697>] ? dev_ioctl+0x2e7/0x570
Aug 09 09:00:46 vc1n3 kernel:  [<ffffffff811971f4>] ? clear_inode+0x14/0x80
Aug 09 09:00:46 vc1n3 kernel:  [<ffffffff810e848a>] ?
cgroup_sk_alloc+0x2a/0x40
Aug 09 09:00:46 vc1n3 kernel:  [<ffffffff813a20e6>] ?
sock_do_ioctl+0x36/0x40
Aug 09 09:00:46 vc1n3 kernel:  [<ffffffff813a253e>] ? sock_ioctl+0x19e/0x240
Aug 09 09:00:46 vc1n3 kernel:  [<ffffffff8118fe94>] ?
do_vfs_ioctl+0x84/0x5a0
Aug 09 09:00:46 vc1n3 kernel:  [<ffffffff81181076>] ? alloc_file+0x16/0xa0
Aug 09 09:00:46 vc1n3 kernel:  [<ffffffff813a1fc5>] ?
sock_alloc_file+0x95/0x110
Aug 09 09:00:46 vc1n3 kernel:  [<ffffffff81190421>] ? SyS_ioctl+0x71/0x80
Aug 09 09:00:46 vc1n3 kernel:  [<ffffffff813a4653>] ? SyS_socket+0x63/0xa0
Aug 09 09:00:46 vc1n3 kernel:  [<ffffffff8149c576>] ?
entry_SYSCALL_64_fastpath+0x1e/0xa8
Aug 09 09:00:46 vc1n3 kernel: INFO: task kworker/3:0:11165 blocked for more
than 120 seconds.
Aug 09 09:00:46 vc1n3 kernel:       Tainted: G           O    4.7.0-aos0.1
#1
Aug 09 09:00:46 vc1n3 kernel: "echo 0 >
/proc/sys/kernel/hung_task_timeout_secs" disables this message.
Aug 09 09:00:46 vc1n3 kernel: kworker/3:0     D ffff88085fd96800     0
11165      2 0x00000000
Aug 09 09:00:46 vc1n3 kernel: Workqueue: ipv6_addrconf addrconf_verify_work
Aug 09 09:00:46 vc1n3 kernel:  ffff88085ade8000 0000000000000000
0000000000000000 ffff8808557d4000
Aug 09 09:00:46 vc1n3 kernel:  ffff8808557d3e18 ffffffff81889624
ffff88085ade8000 0000000000000246
Aug 09 09:00:46 vc1n3 kernel:  00000000ffffffff ffffffff8149831c
ffffffff81889620 ffffffff814985b5
Aug 09 09:00:46 vc1n3 kernel: Call Trace:
Aug 09 09:00:46 vc1n3 kernel:  [<ffffffff8149831c>] ? schedule+0x2c/0x80
Aug 09 09:00:46 vc1n3 kernel:  [<ffffffff814985b5>] ?
schedule_preempt_disabled+0x5/0x10
Aug 09 09:00:46 vc1n3 kernel:  [<ffffffff8149a60c>] ?
__mutex_lock_slowpath+0x10c/0x300
Aug 09 09:00:46 vc1n3 kernel:  [<ffffffff810286f4>] ?
__switch_to+0x244/0x570
Aug 09 09:00:46 vc1n3 kernel:  [<ffffffff8149a809>] ? mutex_lock+0x9/0x20
Aug 09 09:00:46 vc1n3 kernel:  [<ffffffff8145e9f5>] ?
addrconf_verify_work+0x5/0x10
Aug 09 09:00:46 vc1n3 kernel:  [<ffffffff81086de5>] ?
process_one_work+0x145/0x3d0
Aug 09 09:00:46 vc1n3 kernel:  [<ffffffff810870d0>] ?
worker_thread+0x60/0x4a0
Aug 09 09:00:46 vc1n3 kernel:  [<ffffffff81497e67>] ? __schedule+0x1e7/0x670
Aug 09 09:00:46 vc1n3 kernel:  [<ffffffff81087070>] ?
process_one_work+0x3d0/0x3d0
Aug 09 09:00:46 vc1n3 kernel:  [<ffffffff8108beea>] ? kthread+0xca/0xe0
Aug 09 09:00:46 vc1n3 kernel:  [<ffffffff8149c79f>] ?
ret_from_fork+0x1f/0x40
Aug 09 09:00:46 vc1n3 kernel:  [<ffffffff8108be20>] ?
kthread_create_on_node+0x180/0x180
Aug 09 09:00:46 vc1n3 kernel: INFO: task teamd:11385 blocked for more than
120 seconds.
Aug 09 09:00:46 vc1n3 kernel:       Tainted: G           O    4.7.0-aos0.1
#1
Aug 09 09:00:46 vc1n3 kernel: "echo 0 >
/proc/sys/kernel/hung_task_timeout_secs" disables this message.
Aug 09 09:00:46 vc1n3 kernel: teamd           D ffff88107fd16800     0
11385      1 0x00000000
Aug 09 09:00:46 vc1n3 kernel:  ffff88105aed3e80 ffff88085b8a1600
00000000000000db ffff8810545f4000
Aug 09 09:00:46 vc1n3 kernel:  ffff8810545f3c78 ffffffff81889624
ffff88105aed3e80 0000000000000246
Aug 09 09:00:46 vc1n3 kernel:  00000000ffffffff ffffffff8149831c
ffffffff81889620 ffffffff814985b5
Aug 09 09:00:46 vc1n3 kernel: Call Trace:
Aug 09 09:00:46 vc1n3 kernel:  [<ffffffff8149831c>] ? schedule+0x2c/0x80
Aug 09 09:00:46 vc1n3 kernel:  [<ffffffff814985b5>] ?
schedule_preempt_disabled+0x5/0x10
Aug 09 09:00:46 vc1n3 kernel:  [<ffffffff8149a60c>] ?
__mutex_lock_slowpath+0x10c/0x300
Aug 09 09:00:46 vc1n3 kernel:  [<ffffffff81190dc0>] ?
poll_select_copy_remaining+0x120/0x120
Aug 09 09:00:46 vc1n3 kernel:  [<ffffffff8149a809>] ? mutex_lock+0x9/0x20
Aug 09 09:00:46 vc1n3 kernel:  [<ffffffff813cfa00>] ?
rtnetlink_rcv+0x10/0x30
Aug 09 09:00:46 vc1n3 kernel:  [<ffffffff813ed17a>] ?
netlink_unicast+0x14a/0x1e0
Aug 09 09:00:46 vc1n3 kernel:  [<ffffffff813ed609>] ?
netlink_sendmsg+0x349/0x3a0
Aug 09 09:00:46 vc1n3 kernel:  [<ffffffff813a3b5b>] ? sock_sendmsg+0x2b/0x40
Aug 09 09:00:46 vc1n3 kernel:  [<ffffffff813a43dd>] ?
___sys_sendmsg+0x24d/0x260
Aug 09 09:00:46 vc1n3 kernel:  [<ffffffff813a4538>] ?
___sys_recvmsg+0x148/0x1c0
Aug 09 09:00:46 vc1n3 kernel:  [<ffffffff811bdc34>] ?
ep_send_events_proc+0x74/0x170
Aug 09 09:00:46 vc1n3 kernel:  [<ffffffff8149a3bb>] ?
__mutex_unlock_slowpath+0xbb/0x160
Aug 09 09:00:46 vc1n3 kernel:  [<ffffffff811be517>] ? ep_poll+0x127/0x350
Aug 09 09:00:46 vc1n3 kernel:  [<ffffffff813a519c>] ?
__sys_sendmsg+0x3c/0x70
Aug 09 09:00:46 vc1n3 kernel:  [<ffffffff8149c576>] ?
entry_SYSCALL_64_fastpath+0x1e/0xa8
Aug 09 09:00:46 vc1n3 kernel: INFO: task teamd:11386 blocked for more than
120 seconds.
Aug 09 09:00:46 vc1n3 kernel:       Tainted: G           O    4.7.0-aos0.1
#1
Aug 09 09:00:46 vc1n3 kernel: "echo 0 >
/proc/sys/kernel/hung_task_timeout_secs" disables this message.
Aug 09 09:00:46 vc1n3 kernel: teamd           D ffff88107fc96800     0
11386      1 0x00000000
Aug 09 09:00:46 vc1n3 kernel:  ffff88105aed7080 ffff88085abf1f00
00000000000000db ffff881057ff4000
Aug 09 09:00:46 vc1n3 kernel:  ffff881057ff3c78 ffffffff81889624
ffff88105aed7080 0000000000000246
Aug 09 09:00:46 vc1n3 kernel:  00000000ffffffff ffffffff8149831c
ffffffff81889620 ffffffff814985b5
Aug 09 09:00:46 vc1n3 kernel: Call Trace:
Aug 09 09:00:46 vc1n3 kernel:  [<ffffffff8149831c>] ? schedule+0x2c/0x80
Aug 09 09:00:46 vc1n3 kernel:  [<ffffffff814985b5>] ?
schedule_preempt_disabled+0x5/0x10
Aug 09 09:00:46 vc1n3 kernel:  [<ffffffff8149a60c>] ?
__mutex_lock_slowpath+0x10c/0x300
Aug 09 09:00:46 vc1n3 kernel:  [<ffffffff81190dc0>] ?
poll_select_copy_remaining+0x120/0x120
Aug 09 09:00:46 vc1n3 kernel:  [<ffffffff8149a809>] ? mutex_lock+0x9/0x20
Aug 09 09:00:46 vc1n3 kernel:  [<ffffffff813cfa00>] ?
rtnetlink_rcv+0x10/0x30
Aug 09 09:00:46 vc1n3 kernel:  [<ffffffff813ed17a>] ?
netlink_unicast+0x14a/0x1e0
Aug 09 09:00:46 vc1n3 kernel:  [<ffffffff813ed609>] ?
netlink_sendmsg+0x349/0x3a0
Aug 09 09:00:46 vc1n3 kernel:  [<ffffffff813a3b5b>] ? sock_sendmsg+0x2b/0x40
Aug 09 09:00:46 vc1n3 kernel:  [<ffffffff813a43dd>] ?
___sys_sendmsg+0x24d/0x260
Aug 09 09:00:46 vc1n3 kernel:  [<ffffffff813a4538>] ?
___sys_recvmsg+0x148/0x1c0
Aug 09 09:00:46 vc1n3 kernel:  [<ffffffff811bdc34>] ?
ep_send_events_proc+0x74/0x170
Aug 09 09:00:46 vc1n3 kernel:  [<ffffffff8149a3bb>] ?
__mutex_unlock_slowpath+0xbb/0x160
Aug 09 09:00:46 vc1n3 kernel:  [<ffffffff8109b627>] ? update_curr+0xa7/0x130
Aug 09 09:00:46 vc1n3 kernel:  [<ffffffff810a2e72>] ?
pick_next_task_fair+0x2c2/0x490
Aug 09 09:00:46 vc1n3 kernel:  [<ffffffff813a519c>] ?
__sys_sendmsg+0x3c/0x70
Aug 09 09:00:46 vc1n3 kernel:  [<ffffffff8149c576>] ?
entry_SYSCALL_64_fastpath+0x1e/0xa8
Aug 09 09:00:46 vc1n3 kernel: INFO: task teamd:11418 blocked for more than
120 seconds.
Aug 09 09:00:46 vc1n3 kernel:       Tainted: G           O    4.7.0-aos0.1
#1
Aug 09 09:00:46 vc1n3 kernel: "echo 0 >
/proc/sys/kernel/hung_task_timeout_secs" disables this message.
Aug 09 09:00:46 vc1n3 kernel: teamd           D ffff88107fc16800     0
11418      1 0x00000004
Aug 09 09:00:46 vc1n3 kernel:  ffff88105af10000 0000000000000001
0000000000000000 ffff881057fe4000
Aug 09 09:00:46 vc1n3 kernel:  ffff881057fe3750 ffff88085560f814
ffff88105af10000 0000000000000246
Aug 09 09:00:46 vc1n3 kernel:  00000000ffffffff ffffffff8149831c
ffff88085560f810 ffffffff814985b5
Aug 09 09:00:46 vc1n3 kernel: Call Trace:
Aug 09 09:00:46 vc1n3 kernel:  [<ffffffff8149831c>] ? schedule+0x2c/0x80
Aug 09 09:00:46 vc1n3 kernel:  [<ffffffff814985b5>] ?
schedule_preempt_disabled+0x5/0x10
Aug 09 09:00:46 vc1n3 kernel:  [<ffffffff8149a60c>] ?
__mutex_lock_slowpath+0x10c/0x300
Aug 09 09:00:46 vc1n3 kernel:  [<ffffffff81429f63>] ?
inetdev_event+0x23/0x4b0
Aug 09 09:00:46 vc1n3 kernel:  [<ffffffff8149a809>] ? mutex_lock+0x9/0x20
Aug 09 09:00:46 vc1n3 kernel:  [<ffffffffa0828e34>] ?
team_device_event+0xe4/0x110 [team]
Aug 09 09:00:46 vc1n3 kernel:  [<ffffffff8108cc00>] ?
notifier_call_chain+0x40/0x60
Aug 09 09:00:46 vc1n3 kernel:  [<ffffffff813c1ef1>] ?
netdev_update_features+0x21/0x30
Aug 09 09:00:46 vc1n3 kernel:  [<ffffffff813c1c72>] ?
__netdev_update_features+0x2e2/0x540
Aug 09 09:00:46 vc1n3 kernel:  [<ffffffff813bcda7>] ? netdev_info+0x57/0x60
Aug 09 09:00:46 vc1n3 kernel:  [<ffffffff813c204d>] ?
netdev_change_features+0xd/0x30
Aug 09 09:00:46 vc1n3 kernel:  [<ffffffffa0828cde>] ?
team_port_del+0x1ae/0x1e0 [team]
Aug 09 09:00:46 vc1n3 kernel:  [<ffffffffa0828d38>] ?
team_del_slave+0x28/0x40 [team]
Aug 09 09:00:46 vc1n3 kernel:  [<ffffffff813d0650>] ? do_setlink+0x320/0xb60
Aug 09 09:00:46 vc1n3 kernel:  [<ffffffff8149a3bb>] ?
__mutex_unlock_slowpath+0xbb/0x160
Aug 09 09:00:46 vc1n3 kernel:  [<ffffffff811be3bd>] ?
ep_scan_ready_list+0x1ad/0x1c0
Aug 09 09:00:46 vc1n3 kernel:  [<ffffffff811be3d0>] ?
ep_scan_ready_list+0x1c0/0x1c0
Aug 09 09:00:46 vc1n3 kernel:  [<ffffffff811bdfc2>] ?
ep_call_nested.constprop.10+0xe2/0x100
Aug 09 09:00:46 vc1n3 kernel:  [<ffffffff813d15a1>] ?
rtnl_newlink+0x621/0x870
Aug 09 09:00:46 vc1n3 kernel:  [<ffffffff813d1084>] ?
rtnl_newlink+0x104/0x870
Aug 09 09:00:46 vc1n3 kernel:  [<ffffffff813d1c01>] ?
rtnl_getlink+0x121/0x1a0
Aug 09 09:00:46 vc1n3 kernel:  [<ffffffff813cfaa8>] ?
rtnetlink_rcv_msg+0x88/0x210
Aug 09 09:00:46 vc1n3 kernel:  [<ffffffff81177b7b>] ?
__kmalloc_node_track_caller+0x2b/0x210
Aug 09 09:00:46 vc1n3 kernel:  [<ffffffff813cfa20>] ?
rtnetlink_rcv+0x30/0x30
Aug 09 09:00:46 vc1n3 kernel:  [<ffffffff813ed82c>] ?
netlink_rcv_skb+0x9c/0xc0
Aug 09 09:00:46 vc1n3 kernel:  [<ffffffff813cfa0f>] ?
rtnetlink_rcv+0x1f/0x30
Aug 09 09:00:46 vc1n3 kernel:  [<ffffffff813ed17a>] ?
netlink_unicast+0x14a/0x1e0
Aug 09 09:00:46 vc1n3 kernel:  [<ffffffff813ed609>] ?
netlink_sendmsg+0x349/0x3a0
Aug 09 09:00:46 vc1n3 kernel:  [<ffffffff813a3b5b>] ? sock_sendmsg+0x2b/0x40
Aug 09 09:00:46 vc1n3 kernel:  [<ffffffff813a43dd>] ?
___sys_sendmsg+0x24d/0x260
Aug 09 09:00:46 vc1n3 kernel:  [<ffffffff811978d4>] ?
generic_update_time+0x74/0xc0
Aug 09 09:00:46 vc1n3 kernel:  [<ffffffff81198f69>] ? touch_atime+0x99/0xa0
Aug 09 09:00:46 vc1n3 kernel:  [<ffffffff811871fa>] ? pipe_read+0x2ca/0x300
Aug 09 09:00:46 vc1n3 kernel:  [<ffffffff8117e4fa>] ? __vfs_read+0xba/0x110
Aug 09 09:00:46 vc1n3 kernel:  [<ffffffff813a519c>] ?
__sys_sendmsg+0x3c/0x70
Aug 09 09:00:46 vc1n3 kernel:  [<ffffffff8149c576>] ?
entry_SYSCALL_64_fastpath+0x1e/0xa8
Aug 09 09:00:58 vc1n3 systemd[1]: teamd(a)team0.service: Processes still
around after final SIGKILL. Entering failed mode.
Aug 09 09:00:58 vc1n3 systemd[1]: Stopped Team Daemon for device team0.
Aug 09 09:00:58 vc1n3 systemd[1]: teamd(a)team0.service: Unit entered failed
state.
Aug 09 09:00:58 vc1n3 systemd[1]: teamd(a)team0.service: Failed with result
'timeout'.






Configs:
[root@vc1n3 interfaces]# cat team-sw1
{
        "device":               "sw1",
        "hwaddr":               "00:0a:f7:06:1e:68",
        "runner": {
                "name": "lacp",
                "active": true,
                "fast_rate": true,
                "tx_hash": ["eth", "ipv4",
"ipv6"]
        },
        "link_watch":           {"name": "ethtool"},
        "ports":                {"enp17s0f0": {}, "eno2":
{}, "eno3": {}}
}
[root@vc1n3 interfaces]# cat team-sw2
{
        "device":               "sw2",
        "hwaddr":               "00:0a:f7:06:1e:6a",
        "runner": {
                "name": "lacp",
                "active": true,
                "fast_rate": true,
                "tx_hash": ["eth", "ipv4",
"ipv6"]
        },
        "link_watch":           {"name": "ethtool"},
        "ports":                {"enp17s0f1": {}, "eno4":
{}, "eno5": {}}
}
[root@vc1n3 interfaces]# cat team-team0
{
    "device":   "team0",
    "runner":   {"name": "activebackup"},
    "link_watch":   {
        "name": "ethtool"
    },
    "ports":    {
        "sw1": {
            "prio": 100
        },
        "sw2": {
            "prio": 50
        }
    }
}

[Jiri Pirko]

Tue, Aug 09, 2016 at 11:04:48AM CEST, mariusz.g.mazur(a)gmail.com wrote:
> I have three team interfaces:
> - sw1 and sw2, both are three-NIC lacps
> - and an activebackup called team0 (I'm open to better naming suggestions
> :) on top of sw1+sw2.
>
> Here is how vlan interface teardown looks like using the unstacked sw1
> interface:
>
> [root@vc1n3 ~]# systemctl start teamd@sw1
> [root@vc1n3 ~]# ip l s sw1 up
> [root@vc1n3 ~]# ip link add link sw1 name lan.246 type vlan id 246
> [root@vc1n3 ~]# ip l s lan.246 up
> [root@vc1n3 ~]# systemctl stop teamd@sw1
> [root@vc1n3 ~]#
>
> Everything went fine, both sw1 and lan.246@sw1 are now gone.
>
> Now let's try this with stacked team interfaces:
>
> [root@vc1n3 ~]# systemctl start teamd@sw1 teamd@sw2
> [root@vc1n3 ~]# systemctl start teamd@team0
> [root@vc1n3 ~]# ip l s team0 up
> [root@vc1n3 ~]# ip link add link team0 name lan.246 type vlan id 246
> [root@vc1n3 ~]# ip l s lan.246 up
> [root@vc1n3 ~]# systemctl stop teamd@team0
>
> … and it freezes. I can't tell you what the state of the network interfaces
> is, because running 'ip link show' also freezes. The teamd process becomes
> unkillable. Only thing left to do now is to hard reset the system, cause I
> can't do a proper shutdown in any shape or form. 

Looks this might be fixed with the Xin Long's patch I just applied. Try
the current git please.

[Mariusz Mazur]

I'm not on red hat and don't use their networking scripts. For what I use,
I do have a workaround to guarantee that vlans are removed before teamd
tries to remove a stacked interface.

The bug report was for figuring out where the kernel lock is and maybe
fixing it.

2016-08-14 10:24 GMT+02:00 Jiri Pirko <jiri(a)resnulli.us&gt;:

>  Tue, Aug 09, 2016 at 11:04:48AM CEST, mariusz.g.mazur(a)gmail.com
> wrote:
>  >I have three team interfaces:
>  >- sw1 and sw2, both are three-NIC lacps
>  >- and an activebackup called team0 (I'm open to better naming suggestions
>  >:) on top of sw1+sw2.
>  >
>  >Here is how vlan interface teardown looks like using the unstacked sw1
>  >interface:
>  >
>  >[root@vc1n3 ~]# systemctl start teamd@sw1
>  >[root@vc1n3 ~]# ip l s sw1 up
>  >[root@vc1n3 ~]# ip link add link sw1 name lan.246 type vlan id 246
>  >[root@vc1n3 ~]# ip l s lan.246 up
>  >[root@vc1n3 ~]# systemctl stop teamd@sw1
>  >[root@vc1n3 ~]#
>  >
>  >Everything went fine, both sw1 and lan.246@sw1 are now gone.
>  >
>  >Now let's try this with stacked team interfaces:
>  >
>  >[root@vc1n3 ~]# systemctl start teamd@sw1 teamd@sw2
>  >[root@vc1n3 ~]# systemctl start teamd@team0
>  >[root@vc1n3 ~]# ip l s team0 up
>  >[root@vc1n3 ~]# ip link add link team0 name lan.246 type vlan id 246
>  >[root@vc1n3 ~]# ip l s lan.246 up
>  >[root@vc1n3 ~]# systemctl stop teamd@team0
>  >
>  >… and it freezes. I can't tell you what the state of the network
>  interfaces
>  >is, because running 'ip link show' also freezes. The teamd process becomes
>  >unkillable. Only thing left to do now is to hard reset the system, cause I
>  >can't do a proper shutdown in any shape or form.
>
>
>  Looks this might be fixed with the Xin Long's patch I just applied. Try
>  the current git please.
>

[Jiri Pirko]

Tue, Aug 16, 2016 at 10:15:53AM CEST, mariusz.g.mazur(a)gmail.com wrote:
> I'm not on red hat and don't use their networking scripts. For
> what I use,
> I do have a workaround to guarantee that vlans are removed before teamd
> tries to remove a stacked interface.
>
> The bug report was for figuring out where the kernel lock is and maybe
> fixing it. 

got it. Looks like a mutex deadlock to me. kernel issue. would be good
if you can debug it some more. the stacked teams usecase is quite
unusual.

btw why do you stack teams? why don't you just have one team with 6
devices?


>
>2016-08-14 10:24 GMT+02:00 Jiri Pirko <jiri(a)resnulli.us&gt;:
>
>> Tue, Aug 09, 2016 at 11:04:48AM CEST, mariusz.g.mazur(a)gmail.com wrote:
>> >I have three team interfaces:
>> >- sw1 and sw2, both are three-NIC lacps
>> >- and an activebackup called team0 (I'm open to better naming
suggestions
>> >:) on top of sw1+sw2.
>> >
>> >Here is how vlan interface teardown looks like using the unstacked sw1
>> >interface:
>> >
>> >[root@vc1n3 ~]# systemctl start teamd@sw1
>> >[root@vc1n3 ~]# ip l s sw1 up
>> >[root@vc1n3 ~]# ip link add link sw1 name lan.246 type vlan id 246
>> >[root@vc1n3 ~]# ip l s lan.246 up
>> >[root@vc1n3 ~]# systemctl stop teamd@sw1
>> >[root@vc1n3 ~]#
>> >
>> >Everything went fine, both sw1 and lan.246@sw1 are now gone.
>> >
>> >Now let's try this with stacked team interfaces:
>> >
>> >[root@vc1n3 ~]# systemctl start teamd@sw1 teamd@sw2
>> >[root@vc1n3 ~]# systemctl start teamd@team0
>> >[root@vc1n3 ~]# ip l s team0 up
>> >[root@vc1n3 ~]# ip link add link team0 name lan.246 type vlan id 246
>> >[root@vc1n3 ~]# ip l s lan.246 up
>> >[root@vc1n3 ~]# systemctl stop teamd@team0
>> >
>> >… and it freezes. I can't tell you what the state of the network
>> interfaces
>> >is, because running 'ip link show' also freezes. The teamd process
becomes
>> >unkillable. Only thing left to do now is to hard reset the system, cause I
>> >can't do a proper shutdown in any shape or form.
>>
>>
>> Looks this might be fixed with the Xin Long's patch I just applied. Try
>> the current git please.
>>

[Jamie Bainbridge]

On 16 August 2016 at 18:52, Jiri Pirko <jiri(a)resnulli.us&gt; wrote:
>  Tue, Aug 16, 2016 at 10:15:53AM CEST, mariusz.g.mazur(a)gmail.com
> wrote:
> >I'm not on red hat and don't use their networking scripts. For what I use,
> >I do have a workaround to guarantee that vlans are removed before teamd
> >tries to remove a stacked interface.
> >
> >The bug report was for figuring out where the kernel lock is and maybe
> >fixing it.
>
>
>  got it. Looks like a mutex deadlock to me. kernel issue. would be good
>  if you can debug it some more. the stacked teams usecase is quite
>  unusual.
>
>  btw why do you stack teams? why don't you just have one team with 6
>  devices? 
To elaborate, it is not required to stack teams in this way to achieve
failover with LACP.

Only one Aggregator is used at a time. If you put all 6 interfaces in
the one Team, three ports will negotiate one Aggregator ID, and the
other three ports will negotiate another Aggregator ID. If one
Aggregator goes down then the other Aggregator used. In this way, LACP
has active-backup failover built into the protocol.

I've never tried this on Team but it works well with bonding. Actually
this is the only way to do it with bonding, because you cannot stack
bonds.

Bonding offers aggregator selection based on bandwidth and number of
slaves with the "ad_select" parameter. Just skimming the wiki, I could
not see an equivalent with Team, but check the source.

Jamie


>>2016-08-14 10:24 GMT+02:00 Jiri Pirko <jiri(a)resnulli.us&gt;:
>>
>>> Tue, Aug 09, 2016 at 11:04:48AM CEST, mariusz.g.mazur(a)gmail.com wrote:
>>> >I have three team interfaces:
>>> >- sw1 and sw2, both are three-NIC lacps
>>> >- and an activebackup called team0 (I'm open to better naming
suggestions
>>> >:) on top of sw1+sw2.
>>> >
>>> >Here is how vlan interface teardown looks like using the unstacked sw1
>>> >interface:
>>> >
>>> >[root@vc1n3 ~]# systemctl start teamd@sw1
>>> >[root@vc1n3 ~]# ip l s sw1 up
>>> >[root@vc1n3 ~]# ip link add link sw1 name lan.246 type vlan id 246
>>> >[root@vc1n3 ~]# ip l s lan.246 up
>>> >[root@vc1n3 ~]# systemctl stop teamd@sw1
>>> >[root@vc1n3 ~]#
>>> >
>>> >Everything went fine, both sw1 and lan.246@sw1 are now gone.
>>> >
>>> >Now let's try this with stacked team interfaces:
>>> >
>>> >[root@vc1n3 ~]# systemctl start teamd@sw1 teamd@sw2
>>> >[root@vc1n3 ~]# systemctl start teamd@team0
>>> >[root@vc1n3 ~]# ip l s team0 up
>>> >[root@vc1n3 ~]# ip link add link team0 name lan.246 type vlan id 246
>>> >[root@vc1n3 ~]# ip l s lan.246 up
>>> >[root@vc1n3 ~]# systemctl stop teamd@team0
>>> >
>>> >… and it freezes. I can't tell you what the state of the network
>>> interfaces
>>> >is, because running 'ip link show' also freezes. The teamd
process becomes
>>> >unkillable. Only thing left to do now is to hard reset the system, cause
I
>>> >can't do a proper shutdown in any shape or form.
>>>
>>>
>>> Looks this might be fixed with the Xin Long's patch I just applied. Try
>>> the current git please.
>>>
> _______________________________________________
> libteam mailing list
> libteam(a)lists.fedorahosted.org
> https://lists.fedorahosted.org/admin/lists/[e-mail]

[Jiri Pirko]

Tue, Aug 16, 2016 at 11:01:43AM CEST, jamie.bainbridge(a)gmail.com wrote:
> On 16 August 2016 at 18:52, Jiri Pirko <jiri(a)resnulli.us&gt;
> wrote:
> > Tue, Aug 16, 2016 at 10:15:53AM CEST, mariusz.g.mazur(a)gmail.com wrote:
> >>I'm not on red hat and don't use their networking scripts. For what I
> use,
> >>I do have a workaround to guarantee that vlans are removed before teamd
> >>tries to remove a stacked interface.
> >>
> >>The bug report was for figuring out where the kernel lock is and maybe
> >>fixing it.
> >
> >
> > got it. Looks like a mutex deadlock to me. kernel issue. would be good
> > if you can debug it some more. the stacked teams usecase is quite
> > unusual.
> >
> > btw why do you stack teams? why don't you just have one team with 6
> > devices?
>
> To elaborate, it is not required to stack teams in this way to achieve
> failover with LACP.
>
> Only one Aggregator is used at a time. If you put all 6 interfaces in
> the one Team, three ports will negotiate one Aggregator ID, and the
> other three ports will negotiate another Aggregator ID. If one
> Aggregator goes down then the other Aggregator used. In this way, LACP
> has active-backup failover built into the protocol.
>
> I've never tried this on Team but it works well with bonding. Actually
> this is the only way to do it with bonding, because you cannot stack
> bonds. 
Actually you can:
test1:~/net-next$ sudo ip link add bondx1 type bond
test1:~/net-next$ sudo ip link add bondx2 type bond
test1:~/net-next$ sudo ip link set bondx1 master bondx2

I was thinking about disabling stacking for team in past. Anyway, it is
not really needed.

> Bonding offers aggregator selection based on bandwidth and number of
> slaves with the "ad_select" parameter. Just skimming the wiki, I could
> not see an equivalent with Team, but check the source. 
Sure you can do that and more with team. Just see man:

       runner.agg_select_policy (string)
              This selects the policy of how the aggregators will be selected. The
following are available:

              lacp_prio — Aggregator with highest priority according to LACP standard will
be selected. Aggregator priority is affected by per-port option lacp_prio.

              lacp_prio_stable — Same as previous one, except do not replace selected
aggregator if it is still usable.

              bandwidth — Select aggregator with highest total bandwidth.

              count — Select aggregator with highest number of ports.

              port_options  —  Aggregator  with  highest  priority  according  to per-port
options prio and sticky will be selected. This means that the aggregator containing the
port with the highest priority will be selected unless at least one of the ports in the
currently
              selected aggregator is sticky.

              Default: lacp_prio



>
>Jamie
>
>
>>>2016-08-14 10:24 GMT+02:00 Jiri Pirko <jiri(a)resnulli.us&gt;:
>>>
>>>> Tue, Aug 09, 2016 at 11:04:48AM CEST, mariusz.g.mazur(a)gmail.com wrote:
>>>> >I have three team interfaces:
>>>> >- sw1 and sw2, both are three-NIC lacps
>>>> >- and an activebackup called team0 (I'm open to better naming
suggestions
>>>> >:) on top of sw1+sw2.
>>>> >
>>>> >Here is how vlan interface teardown looks like using the unstacked
sw1
>>>> >interface:
>>>> >
>>>> >[root@vc1n3 ~]# systemctl start teamd@sw1
>>>> >[root@vc1n3 ~]# ip l s sw1 up
>>>> >[root@vc1n3 ~]# ip link add link sw1 name lan.246 type vlan id 246
>>>> >[root@vc1n3 ~]# ip l s lan.246 up
>>>> >[root@vc1n3 ~]# systemctl stop teamd@sw1
>>>> >[root@vc1n3 ~]#
>>>> >
>>>> >Everything went fine, both sw1 and lan.246@sw1 are now gone.
>>>> >
>>>> >Now let's try this with stacked team interfaces:
>>>> >
>>>> >[root@vc1n3 ~]# systemctl start teamd@sw1 teamd@sw2
>>>> >[root@vc1n3 ~]# systemctl start teamd@team0
>>>> >[root@vc1n3 ~]# ip l s team0 up
>>>> >[root@vc1n3 ~]# ip link add link team0 name lan.246 type vlan id 246
>>>> >[root@vc1n3 ~]# ip l s lan.246 up
>>>> >[root@vc1n3 ~]# systemctl stop teamd@team0
>>>> >
>>>> >… and it freezes. I can't tell you what the state of the network
>>>> interfaces
>>>> >is, because running 'ip link show' also freezes. The teamd
process becomes
>>>> >unkillable. Only thing left to do now is to hard reset the system,
cause I
>>>> >can't do a proper shutdown in any shape or form.
>>>>
>>>>
>>>> Looks this might be fixed with the Xin Long's patch I just applied.
Try
>>>> the current git please.
>>>>
>> _______________________________________________
>> libteam mailing list
>> libteam(a)lists.fedorahosted.org
>> https://lists.fedorahosted.org/admin/lists/[e-mail]

[Mariusz Mazur]

Well, a lot of the docs I've seen regarding teaming had device stacking as
an explicit feature[1], so it was the first thing that came to my mind for
an activebackup on top of two 3-link lacps. Guess with stacking being buggy
and not really tested, I'll drop it for the suggested lacp-only config.
Only thing missing here is that I had a 90second delay_up configured on the
stacked activebackup, though for real world use cases this should be mostly
equivalent to just putting that delay on all of the enoX ports.

Regarding the kernel locking issue, I can provide some more debug info, I
just need to know how to obtain it.

[1] Case in point: http://libteam.org/files/teamdev.pp.pdf page 6

2016-08-16 11:09 GMT+02:00 Jiri Pirko <jiri(a)resnulli.us&gt;:

>  Tue, Aug 16, 2016 at 11:01:43AM CEST, jamie.bainbridge(a)gmail.com
> wrote:
>  >On 16 August 2016 at 18:52, Jiri Pirko <jiri(a)resnulli.us&gt; wrote:
>  >> Tue, Aug 16, 2016 at 10:15:53AM CEST, mariusz.g.mazur(a)gmail.com wrote:
>  >>>I'm not on red hat and don't use their networking scripts. For what
> I
>  use,
>  >>>I do have a workaround to guarantee that vlans are removed before teamd
>  >>>tries to remove a stacked interface.
>  >>>
>  >>>The bug report was for figuring out where the kernel lock is and maybe
>  >>>fixing it.
>  >>
>  >>
>  >> got it. Looks like a mutex deadlock to me. kernel issue. would be good
>  >> if you can debug it some more. the stacked teams usecase is quite
>  >> unusual.
>  >>
>  >> btw why do you stack teams? why don't you just have one team with 6
>  >> devices?
>  >
>  >To elaborate, it is not required to stack teams in this way to achieve
>  >failover with LACP.
>  >
>  >Only one Aggregator is used at a time. If you put all 6 interfaces in
>  >the one Team, three ports will negotiate one Aggregator ID, and the
>  >other three ports will negotiate another Aggregator ID. If one
>  >Aggregator goes down then the other Aggregator used. In this way, LACP
>  >has active-backup failover built into the protocol.
>  >
>  >I've never tried this on Team but it works well with bonding. Actually
>  >this is the only way to do it with bonding, because you cannot stack
>  >bonds.
>
>  Actually you can:
>  test1:~/net-next$ sudo ip link add bondx1 type bond
>  test1:~/net-next$ sudo ip link add bondx2 type bond
>  test1:~/net-next$ sudo ip link set bondx1 master bondx2
>
>  I was thinking about disabling stacking for team in past. Anyway, it is
>  not really needed.
>
>  >
>  >Bonding offers aggregator selection based on bandwidth and number of
>  >slaves with the "ad_select" parameter. Just skimming the wiki, I could
>  >not see an equivalent with Team, but check the source.
>
>  Sure you can do that and more with team. Just see man:
>
>         runner.agg_select_policy (string)
>                This selects the policy of how the aggregators will be
>  selected. The following are available:
>
>                lacp_prio — Aggregator with highest priority according to
>  LACP standard will be selected. Aggregator priority is affected by per-port
>  option lacp_prio.
>
>                lacp_prio_stable — Same as previous one, except do not
>  replace selected aggregator if it is still usable.
>
>                bandwidth — Select aggregator with highest total bandwidth.
>
>                count — Select aggregator with highest number of ports.
>
>                port_options  —  Aggregator  with  highest  priority
>  according  to per-port options prio and sticky will be selected. This means
>  that the aggregator containing the port with the highest priority will be
>  selected unless at least one of the ports in the currently
>                selected aggregator is sticky.
>
>                Default: lacp_prio
>
>
>
>  >
>  >Jamie
>  >
>  >
>  >>>2016-08-14 10:24 GMT+02:00 Jiri Pirko <jiri(a)resnulli.us&gt;:
>  >>>
>  >>>> Tue, Aug 09, 2016 at 11:04:48AM CEST, mariusz.g.mazur(a)gmail.com
>  wrote:
>  >>>> >I have three team interfaces:
>  >>>> >- sw1 and sw2, both are three-NIC lacps
>  >>>> >- and an activebackup called team0 (I'm open to better naming
>  suggestions
>  >>>> >:) on top of sw1+sw2.
>  >>>> >
>  >>>> >Here is how vlan interface teardown looks like using the unstacked
>  sw1
>  >>>> >interface:
>  >>>> >
>  >>>> >[root@vc1n3 ~]# systemctl start teamd@sw1
>  >>>> >[root@vc1n3 ~]# ip l s sw1 up
>  >>>> >[root@vc1n3 ~]# ip link add link sw1 name lan.246 type vlan id 246
>  >>>> >[root@vc1n3 ~]# ip l s lan.246 up
>  >>>> >[root@vc1n3 ~]# systemctl stop teamd@sw1
>  >>>> >[root@vc1n3 ~]#
>  >>>> >
>  >>>> >Everything went fine, both sw1 and lan.246@sw1 are now gone.
>  >>>> >
>  >>>> >Now let's try this with stacked team interfaces:
>  >>>> >
>  >>>> >[root@vc1n3 ~]# systemctl start teamd@sw1 teamd@sw2
>  >>>> >[root@vc1n3 ~]# systemctl start teamd@team0
>  >>>> >[root@vc1n3 ~]# ip l s team0 up
>  >>>> >[root@vc1n3 ~]# ip link add link team0 name lan.246 type vlan id
> 246
>  >>>> >[root@vc1n3 ~]# ip l s lan.246 up
>  >>>> >[root@vc1n3 ~]# systemctl stop teamd@team0
>  >>>> >
>  >>>> >… and it freezes. I can't tell you what the state of the
> network
>  >>>> interfaces
>  >>>> >is, because running 'ip link show' also freezes. The teamd
> process
>  becomes
>  >>>> >unkillable. Only thing left to do now is to hard reset the system,
>  cause I
>  >>>> >can't do a proper shutdown in any shape or form.
>  >>>>
>  >>>>
>  >>>> Looks this might be fixed with the Xin Long's patch I just applied.
>  Try
>  >>>> the current git please.
>  >>>>
>  >> _______________________________________________
>  >> libteam mailing list
>  >> libteam(a)lists.fedorahosted.org
>  >> https://lists.fedorahosted.org/admin/lists/libteam@lists.
>  fedorahosted.org
>

[Jiri Pirko]

Tue, Aug 16, 2016 at 11:36:18AM CEST, mariusz.g.mazur(a)gmail.com wrote:
> Well, a lot of the docs I've seen regarding teaming had device
> stacking as
> an explicit feature[1], so it was the first thing that came to my mind for
> an activebackup on top of two 3-link lacps. Guess with stacking being buggy
> and not really tested, I'll drop it for the suggested lacp-only config.
> Only thing missing here is that I had a 90second delay_up configured on the
> stacked activebackup, though for real world use cases this should be mostly
> equivalent to just putting that delay on all of the enoX ports. 
I believe you don't need delay_up. It's purpose is to workaround buggy
drivers which reports link even in fact they still don't have it. If
lacp reports link is up, it is up. No delay needed.


> Regarding the kernel locking issue, I can provide some more debug info, I
> just need to know how to obtain it. 
If I find some time I'll try to reproduce that myself and fix it.




>
>[1] Case in point: http://libteam.org/files/teamdev.pp.pdf page 6
>
>2016-08-16 11:09 GMT+02:00 Jiri Pirko <jiri(a)resnulli.us&gt;:
>
>> Tue, Aug 16, 2016 at 11:01:43AM CEST, jamie.bainbridge(a)gmail.com wrote:
>> >On 16 August 2016 at 18:52, Jiri Pirko <jiri(a)resnulli.us&gt; wrote:
>> >> Tue, Aug 16, 2016 at 10:15:53AM CEST, mariusz.g.mazur(a)gmail.com wrote:
>> >>>I'm not on red hat and don't use their networking scripts.
For what I
>> use,
>> >>>I do have a workaround to guarantee that vlans are removed before
teamd
>> >>>tries to remove a stacked interface.
>> >>>
>> >>>The bug report was for figuring out where the kernel lock is and
maybe
>> >>>fixing it.
>> >>
>> >>
>> >> got it. Looks like a mutex deadlock to me. kernel issue. would be good
>> >> if you can debug it some more. the stacked teams usecase is quite
>> >> unusual.
>> >>
>> >> btw why do you stack teams? why don't you just have one team with 6
>> >> devices?
>> >
>> >To elaborate, it is not required to stack teams in this way to achieve
>> >failover with LACP.
>> >
>> >Only one Aggregator is used at a time. If you put all 6 interfaces in
>> >the one Team, three ports will negotiate one Aggregator ID, and the
>> >other three ports will negotiate another Aggregator ID. If one
>> >Aggregator goes down then the other Aggregator used. In this way, LACP
>> >has active-backup failover built into the protocol.
>> >
>> >I've never tried this on Team but it works well with bonding. Actually
>> >this is the only way to do it with bonding, because you cannot stack
>> >bonds.
>>
>> Actually you can:
>> test1:~/net-next$ sudo ip link add bondx1 type bond
>> test1:~/net-next$ sudo ip link add bondx2 type bond
>> test1:~/net-next$ sudo ip link set bondx1 master bondx2
>>
>> I was thinking about disabling stacking for team in past. Anyway, it is
>> not really needed.
>>
>> >
>> >Bonding offers aggregator selection based on bandwidth and number of
>> >slaves with the "ad_select" parameter. Just skimming the wiki, I
could
>> >not see an equivalent with Team, but check the source.
>>
>> Sure you can do that and more with team. Just see man:
>>
>>        runner.agg_select_policy (string)
>>               This selects the policy of how the aggregators will be
>> selected. The following are available:
>>
>>               lacp_prio — Aggregator with highest priority according to
>> LACP standard will be selected. Aggregator priority is affected by per-port
>> option lacp_prio.
>>
>>               lacp_prio_stable — Same as previous one, except do not
>> replace selected aggregator if it is still usable.
>>
>>               bandwidth — Select aggregator with highest total bandwidth.
>>
>>               count — Select aggregator with highest number of ports.
>>
>>               port_options  —  Aggregator  with  highest  priority
>> according  to per-port options prio and sticky will be selected. This means
>> that the aggregator containing the port with the highest priority will be
>> selected unless at least one of the ports in the currently
>>               selected aggregator is sticky.
>>
>>               Default: lacp_prio
>>
>>
>>
>> >
>> >Jamie
>> >
>> >
>> >>>2016-08-14 10:24 GMT+02:00 Jiri Pirko <jiri(a)resnulli.us&gt;:
>> >>>
>> >>>> Tue, Aug 09, 2016 at 11:04:48AM CEST, mariusz.g.mazur(a)gmail.com
>> wrote:
>> >>>> >I have three team interfaces:
>> >>>> >- sw1 and sw2, both are three-NIC lacps
>> >>>> >- and an activebackup called team0 (I'm open to better
naming
>> suggestions
>> >>>> >:) on top of sw1+sw2.
>> >>>> >
>> >>>> >Here is how vlan interface teardown looks like using the
unstacked
>> sw1
>> >>>> >interface:
>> >>>> >
>> >>>> >[root@vc1n3 ~]# systemctl start teamd@sw1
>> >>>> >[root@vc1n3 ~]# ip l s sw1 up
>> >>>> >[root@vc1n3 ~]# ip link add link sw1 name lan.246 type vlan
id 246
>> >>>> >[root@vc1n3 ~]# ip l s lan.246 up
>> >>>> >[root@vc1n3 ~]# systemctl stop teamd@sw1
>> >>>> >[root@vc1n3 ~]#
>> >>>> >
>> >>>> >Everything went fine, both sw1 and lan.246@sw1 are now
gone.
>> >>>> >
>> >>>> >Now let's try this with stacked team interfaces:
>> >>>> >
>> >>>> >[root@vc1n3 ~]# systemctl start teamd@sw1 teamd@sw2
>> >>>> >[root@vc1n3 ~]# systemctl start teamd@team0
>> >>>> >[root@vc1n3 ~]# ip l s team0 up
>> >>>> >[root@vc1n3 ~]# ip link add link team0 name lan.246 type
vlan id 246
>> >>>> >[root@vc1n3 ~]# ip l s lan.246 up
>> >>>> >[root@vc1n3 ~]# systemctl stop teamd@team0
>> >>>> >
>> >>>> >… and it freezes. I can't tell you what the state of the
network
>> >>>> interfaces
>> >>>> >is, because running 'ip link show' also freezes. The
teamd process
>> becomes
>> >>>> >unkillable. Only thing left to do now is to hard reset the
system,
>> cause I
>> >>>> >can't do a proper shutdown in any shape or form.
>> >>>>
>> >>>>
>> >>>> Looks this might be fixed with the Xin Long's patch I just
applied.
>> Try
>> >>>> the current git please.
>> >>>>
>> >> _______________________________________________
>> >> libteam mailing list
>> >> libteam(a)lists.fedorahosted.org
>> >> https://lists.fedorahosted.org/admin/lists/libteam@lists.
>> fedorahosted.org
>>

[Mariusz Mazur]

2016-08-16 11:44 GMT+02:00 Jiri Pirko <jiri(a)resnulli.us&gt;:

>  I believe you don't need delay_up. It's purpose is to
> workaround buggy
>  drivers which reports link even in fact they still don't have it. If
>  lacp reports link is up, it is up. No delay needed.
>  

Oh, no, that's not my primary use case.

For an activebackup two-link config (no lacp; that's what I use on my older
systems, just with bonding), I use the updelay, because when a link with a
higher priority comes up, it'd be suicide to instantly switch to it. The
switch hasn't settled yet (especially on my older switches, which don't do
Rapid STP, a switch after a reboot can be useless for almost a minute) and
if my machines automatically switched the active link, all of my clusters
would explode. This is a valid use case for up_delay with activebackup.

However for a multi-lacp config, using the up_delay for the same purpose is
an ugly hack. What there should be is a setting like
runner.agg_select_delay that guarantees a link with a higher prio gets time
to settle before actually being used.

[Jiri Pirko]

Tue, Aug 16, 2016 at 12:55:21PM CEST, mariusz.g.mazur(a)gmail.com wrote:
> 2016-08-16 11:44 GMT+02:00 Jiri Pirko <jiri(a)resnulli.us&gt;:
>
> > I believe you don't need delay_up. It's purpose is to workaround buggy
> > drivers which reports link even in fact they still don't have it. If
> > lacp reports link is up, it is up. No delay needed.
> >
>
>
> Oh, no, that's not my primary use case.
>
> For an activebackup two-link config (no lacp; that's what I use on my older
> systems, just with bonding), I use the updelay, because when a link with a
> higher priority comes up, it'd be suicide to instantly switch to it. The
> switch hasn't settled yet (especially on my older switches, which don't do
> Rapid STP, a switch after a reboot can be useless for almost a minute) and
> if my machines automatically switched the active link, all of my clusters
> would explode. This is a valid use case for up_delay with activebackup. 

But if switch assemles lacp aggregator, you can probably assume it
works.


> However for a multi-lacp config, using the up_delay for the same purpose is
> an ugly hack. What there should be is a setting like
> runner.agg_select_delay that guarantees a link with a higher prio gets time
> to settle before actually being used. 
That could be implemented. Feel free to send a patch.

[Mariusz Mazur]

2016-08-16 13:47 GMT+02:00 Jiri Pirko <jiri(a)resnulli.us&gt;:

>  But if switch assemles lacp aggregator, you can probably assume it
>  works.
>  
I'm not an expert, but I doubt that's a safe assumption. If you want to
start doing stp after boot, you really need all the links, including the
aggregated ones, to be up, otherwise what's the point (especially with
switch2switch links likely being aggregated).

[Mariusz Mazur]

2016-08-16 13:47 GMT+02:00 Jiri Pirko <jiri(a)resnulli.us&gt;:

>  >However for a multi-lacp config, using the up_delay for the same
> purpose
>  is
>  >an ugly hack. What there should be is a setting like
>  >runner.agg_select_delay that guarantees a link with a higher prio gets
>  time
>  >to settle before actually being used.
>
>  That could be implemented. Feel free to send a patch.
>  
Turns out I really do need to implement this or switch back to a stacked
config, because up_delay is basically ignored by the lacp runner and a port
switch happens as soon as the new port is up. Here's how this looks like in
the logs (up_delay set to 90s):

Aug 17 13:36:56 vc1n3 teamd[5175]: enp17s0f0: Changed port state:
"disabled" -> "expired"
Aug 17 13:36:56 vc1n3 kernel: bnx2 0000:11:00.0 enp17s0f0: NIC Copper Link
is Up, 1000 Mbps full duplex
Aug 17 13:36:56 vc1n3 kernel:
Aug 17 13:36:57 vc1n3 teamd[5175]: enp17s0f0: Changed port state: "expired"
-> "current"
Aug 17 13:36:59 vc1n3 kernel: igb 0000:06:00.0 eno2: igb: eno2 NIC Link is
Up 1000 Mbps Full Duplex, Flow Control: RX
Aug 17 13:36:59 vc1n3 teamd[5175]: eno2: Changed port state: "disabled" ->
"expired"
Aug 17 13:36:59 vc1n3 teamd[5175]: eno2: Changed port state: "expired" ->
"current"
Aug 17 13:36:59 vc1n3 kernel: igb 0000:06:00.1 eno3: igb: eno3 NIC Link is
Up 1000 Mbps Full Duplex, Flow Control: RX
Aug 17 13:36:59 vc1n3 teamd[5175]: eno3: Changed port state: "disabled" ->
"expired"
Aug 17 13:36:59 vc1n3 teamd[5175]: eno3: Changed port state: "expired" ->
"current"
Aug 17 13:37:07 vc1n3 teamd[5175]: eno2: Changed port state: "current" ->
"expired"
Aug 17 13:37:07 vc1n3 teamd[5175]: eno2: Changed port state: "expired" ->
"current"
Aug 17 13:37:07 vc1n3 teamd[5175]: eno3: Changed port state: "current" ->
"expired"
Aug 17 13:37:07 vc1n3 teamd[5175]: eno3: Changed port state: "expired" ->
"current"
Aug 17 13:38:26 vc1n3 teamd[5175]: enp17s0f0: ethtool-link went up.
Aug 17 13:38:29 vc1n3 teamd[5175]: eno2: ethtool-link went up.
Aug 17 13:38:29 vc1n3 teamd[5175]: eno3: ethtool-link went up.