Stange problems with CentOS 7 and roundrobin runner

[Ingo Brand]

Hello,

I am currently setting up a high available NFS server cluster with 2 
nodes using pacemaker + corosync + drbd + flashcache.
The two nodes (media1 and media2) have 4 1G copper nics each.
I formed 2 teams:
team0: LACP with em1 and em2 connected to a switch for NFS services
team1: roundrobin with em3 and em4 and MTU 9000 directly connected with 
copper patch cables for drbd sync
During my testing everything was working as expected. Then I started to 
tranfer about 20TB from an old server to the new cluster. The copy 
process was fine for about 32 hours. Then corosync and drbd started to 
report errors on the roundrobin team1 interface (see logs at the end of 
this email).
The drbd sync and corosync started to flap between states for several 
hours on team1.
I checked the error counters with ifconfig and found a huge amount off 
TX dropped packets on team1 of the drbd slave node media2.
There were NO other error counters raising but team1 on media2.
To make this a bit more clear:
- team1 on media2 (the drbd slave) showed raising TX dropped packets
- em3 and em4 on media2 (the physical interfaces of team1) did not show 
any raising error counters.
- team1 on media1 (the drbd master) did not show any raising error counters.
- em3 and em4 on media1 (the physical interfaces of team1) did not show 
any raising error counters.
I then thought maybe one of the two patch cables between the two 
machines was not 100% ok. So I removed em4 from team1 on both machines:
"teamdctl team1 port remove em4"
The error counters stopped raising.
I thought: "Yeah! That cable must be broken!"
But just to make sure I really found the problem I re-enabled em4 on 
both nodes:
"teamdctl team1 port add em4"
Ok, now both team1 interfaces had 2 links up again and the error counter 
restarted to raise again.
I then stopped em3 on both nodes to force all traffic through em4:
"teamdctl team1 port remove em3"
And guess what? The error counters on team1 stopped raising!
So in short:
If I remove either one of the two physical interfaces from team1 
everything is working without any errors. As soon as I enable both 
physical interfaces the error counters start to raise.
After these tests I rebooted the slave node (media2) and added both 
ports to team1 again (Have you tried turning it off and on again...).
After doing this, I now see TX dropped packets on the not yet rebooted 
media1 team1 interface.
Why do I only see these errors on the team1 interface and not on em3 and 
em4?

Currently the copy process of 20TB is still running. I think that if I 
reboot media1 everything will work as expected again for some time 
because it worked during my initial tests.
But I do think that I hit a bug in the teaming driver and the raising 
error counters will come back.

Could anybody help?

Kind regards
Ingo

This is the used network config of team1:
media1:
cat /etc/sysconfig/network-scripts/ifcfg-team1_slave_0
# Generated by parse-kickstart
NAME=team1 slave 0
TEAM_MASTER=team1
DEVICETYPE=TeamPort
DEVICE=em3
ONBOOT=yes
UUID=b7026c5b-e9cd-457c-93d2-a2799361ed90

cat /etc/sysconfig/network-scripts/ifcfg-team1_slave_1
# Generated by parse-kickstart
NAME=team1 slave 1
TEAM_MASTER=team1
DEVICETYPE=TeamPort
DEVICE=em4
ONBOOT=yes
UUID=fbe5bc56-95f5-4f9e-a5ea-ab6b6f5e5b50

cat /etc/sysconfig/network-scripts/ifcfg-team1
# Generated by parse-kickstart
UUID=4e0090cd-32d3-4c5b-be23-9f53083da4dd
NAME="Team connection team1"
TEAM_CONFIG="{"runner": {"name": "roundrobin"}}"
GATEWAY=
IPV6_AUTOCONF=yes
BOOTPROTO=none
DEVICE=team1
MTU=9000
TYPE=Team
ONBOOT=yes
IPV6INIT=yes
DEVICETYPE=Team
IPADDR0=192.168.101.31
PREFIX0=24
DEFROUTE=yes
IPV4_FAILURE_FATAL=no
IPV6_DEFROUTE=yes
IPV6_PEERDNS=yes
IPV6_PEERROUTES=yes
IPV6_FAILURE_FATAL=no

media2:
cat /etc/sysconfig/network-scripts/ifcfg-team1_slave_0
# Generated by parse-kickstart
NAME=team1 slave 0
TEAM_MASTER=team1
DEVICETYPE=TeamPort
DEVICE=em3
ONBOOT=yes
UUID=bfd8e77e-7eb8-47ea-ab7e-95286e073202

cat /etc/sysconfig/network-scripts/ifcfg-team1_slave_1
# Generated by parse-kickstart
NAME=team1 slave 1
TEAM_MASTER=team1
DEVICETYPE=TeamPort
DEVICE=em4
ONBOOT=yes
UUID=93c5dfd8-3b81-4d1e-8ba9-c46f8daceb1a

cat /etc/sysconfig/network-scripts/ifcfg-team1
# Generated by parse-kickstart
UUID=79c843db-996c-41e4-9ee5-2a8f3da244ed
NAME="Team connection team1"
TEAM_CONFIG="{"runner": {"name": "roundrobin"}}"
GATEWAY=
IPV6_AUTOCONF=yes
BOOTPROTO=none
DEVICE=team1
MTU=9000
TYPE=Team
ONBOOT=yes
IPV6INIT=yes
DEVICETYPE=Team
IPADDR0=192.168.101.32
PREFIX0=24
DEFROUTE=yes
IPV4_FAILURE_FATAL=no
IPV6_DEFROUTE=yes
IPV6_PEERDNS=yes
IPV6_PEERROUTES=yes
IPV6_FAILURE_FATAL=no

Here are some system logs from both machines:
============================================================
media1:
============================================================
Nov  6 02:31:27 media1 corosync[3657]: [TOTEM ] Incrementing problem 
counter for seqid 412041 iface 192.168.100.31 to [1 of 10]
Nov  6 02:31:29 media1 corosync[3657]: [TOTEM ] ring 0 active with no faults
Nov  6 02:31:29 media1 corosync[3657]: [TOTEM ] Incrementing problem 
counter for seqid 412047 iface 192.168.100.31 to [1 of 10]
Nov  6 02:31:31 media1 corosync[3657]: [TOTEM ] ring 0 active with no faults
Nov  6 02:32:51 media1 corosync[3657]: [TOTEM ] Incrementing problem 
counter for seqid 412315 iface 192.168.101.31 to [1 of 10]
Nov  6 02:32:53 media1 corosync[3657]: [TOTEM ] ring 1 active with no faults
Nov  6 02:32:53 media1 corosync[3657]: [TOTEM ] Incrementing problem 
counter for seqid 412319 iface 192.168.101.31 to [1 of 10]
Nov  6 02:32:54 media1 corosync[3657]: [TOTEM ] Incrementing problem 
counter for seqid 412321 iface 192.168.101.31 to [2 of 10]
Nov  6 02:32:55 media1 corosync[3657]: [TOTEM ] Decrementing problem 
counter for iface 192.168.101.31 to [1 of 10]
Nov  6 02:32:56 media1 corosync[3657]: [TOTEM ] Incrementing problem 
counter for seqid 412325 iface 192.168.101.31 to [2 of 10]
Nov  6 02:32:57 media1 corosync[3657]: [TOTEM ] Decrementing problem 
counter for iface 192.168.101.31 to [1 of 10]
Nov  6 02:32:58 media1 corosync[3657]: [TOTEM ] Incrementing problem 
counter for seqid 412327 iface 192.168.101.31 to [2 of 10]
Nov  6 02:32:59 media1 kernel: drbd r0: PingAck did not arrive in time.
Nov  6 02:32:59 media1 kernel: drbd r0: peer( Secondary -> Unknown ) 
conn( Connected -> NetworkFailure ) pdsk( UpToDate -> DUnknown )
Nov  6 02:32:59 media1 kernel: block drbd0: new current UUID 
0C0308FA4F94F811:59E43E5FD7B26A4F:1963884285B385E6:1962884285B385E6
Nov  6 02:32:59 media1 kernel: drbd r0: asender terminated
Nov  6 02:32:59 media1 kernel: drbd r0: Terminating drbd_a_r0
Nov  6 02:32:59 media1 kernel: drbd r0: Connection closed
Nov  6 02:32:59 media1 kernel: drbd r0: conn( NetworkFailure -> 
Unconnected )
Nov  6 02:32:59 media1 kernel: drbd r0: receiver terminated
Nov  6 02:32:59 media1 kernel: drbd r0: Restarting receiver thread
Nov  6 02:32:59 media1 kernel: drbd r0: receiver (re)started
Nov  6 02:32:59 media1 kernel: drbd r0: conn( Unconnected -> WFConnection )
Nov  6 02:32:59 media1 corosync[3657]: [TOTEM ] Decrementing problem 
counter for iface 192.168.101.31 to [1 of 10]
Nov  6 02:32:59 media1 corosync[3657]: [TOTEM ] Incrementing problem 
counter for seqid 412329 iface 192.168.101.31 to [2 of 10]
Nov  6 02:33:01 media1 corosync[3657]: [TOTEM ] Incrementing problem 
counter for seqid 412333 iface 192.168.101.31 to [3 of 10]
Nov  6 02:33:01 media1 corosync[3657]: [TOTEM ] Decrementing problem 
counter for iface 192.168.101.31 to [2 of 10]
Nov  6 02:33:01 media1 crmd[4960]: notice: do_state_transition: State 
transition S_IDLE -> S_POLICY_ENGINE [ input=I_PE_CALC 
cause=C_FSA_INTERNAL origin=abort_transition_graph ]

Nov  6 02:33:01 media1 pengine[4959]: notice: unpack_config: On loss of 
CCM Quorum: Ignore

Nov  6 02:33:01 media1 pengine[4959]: warning: unpack_rsc_op: Processing 
failed op monitor for res_nfsserver_mediafiles on media1: unknown error 
(1)
Nov  6 02:33:01 media1 pengine[4959]: warning: unpack_rsc_op: Processing 
failed op monitor for res_nfsserver_mediafiles on media2: unknown error 
(1)
Nov  6 02:33:01 media1 pengine[4959]: notice: process_pe_message: 
Calculated Transition 144: /var/lib/pacemaker/pengine/pe-input-227.bz2

Nov  6 02:33:01 media1 crmd[4960]: notice: run_graph: Transition 144 
(Complete=0, Pending=0, Fired=0, Skipped=0, Incomplete=0, 
Source=/var/lib/pacemaker/pengine/pe-input-227.bz2): Complete

Nov  6 02:33:01 media1 crmd[4960]: notice: do_state_transition: State 
transition S_TRANSITION_ENGINE -> S_IDLE [ input=I_TE_SUCCESS 
cause=C_FSA_INTERNAL origin=notify_crmd ]
Nov  6 02:33:01 media1 kernel: drbd r0: Handshake successful: Agreed 
network protocol version 101

Nov  6 02:33:01 media1 kernel: drbd r0: Agreed to support TRIM on 
protocol level

Nov  6 02:33:01 media1 kernel: drbd r0: Peer authenticated using 20 
bytes HMAC

Nov  6 02:33:01 media1 kernel: drbd r0: conn( WFConnection -> 
WFReportParams )

Nov  6 02:33:01 media1 kernel: drbd r0: Starting asender thread (from 
drbd_r_r0 [5999])

Nov  6 02:33:02 media1 corosync[3657]: [TOTEM ] Incrementing problem 
counter for seqid 412337 iface 192.168.101.31 to [3 of 10]

Nov  6 02:33:02 media1 kernel: block drbd0: drbd_sync_handshake:

Nov  6 02:33:02 media1 kernel: block drbd0: self 
0C0308FA4F94F811:59E43E5FD7B26A4F:1963884285B385E6:1962884285B385E6 
bits:192101 flags:0
Nov  6 02:33:02 media1 kernel: block drbd0: peer 
59E43E5FD7B26A4E:0000000000000000:1963884285B385E6:1962884285B385E6 
bits:0 flags:0
Nov  6 02:33:02 media1 kernel: block drbd0: uuid_compare()=1 by rule 70

Nov  6 02:33:02 media1 kernel: block drbd0: peer( Unknown -> Secondary ) 
conn( WFReportParams -> WFBitMapS ) pdsk( DUnknown -> Consistent )

Nov  6 02:33:02 media1 kernel: block drbd0: send bitmap stats 
[Bytes(packets)]: plain 0(0), RLE 185(1), total 185; compression: 100.0%

Nov  6 02:33:02 media1 corosync[3657]: [TOTEM ] Incrementing problem 
counter for seqid 412339 iface 192.168.101.31 to [4 of 10]

Nov  6 02:33:03 media1 kernel: block drbd0: receive bitmap stats 
[Bytes(packets)]: plain 0(0), RLE 185(1), total 185; compression: 100.0%

Nov  6 02:33:03 media1 kernel: block drbd0: helper command: 
/sbin/drbdadm before-resync-source minor-0

Nov  6 02:33:03 media1 kernel: block drbd0: helper command: 
/sbin/drbdadm before-resync-source minor-0 exit code 0 (0x0)

Nov  6 02:33:03 media1 kernel: block drbd0: conn( WFBitMapS -> 
SyncSource ) pdsk( Consistent -> Inconsistent )

Nov  6 02:33:03 media1 kernel: block drbd0: Began resync as SyncSource 
(will sync 919244 KB [229811 bits set]).

Nov  6 02:33:03 media1 kernel: block drbd0: updated sync UUID 
0C0308FA4F94F811:59E53E5FD7B26A4F:59E43E5FD7B26A4F:1963884285B385E6

Nov  6 02:33:03 media1 corosync[3657]: [TOTEM ] Decrementing problem 
counter for iface 192.168.101.31 to [3 of 10]

Nov  6 02:33:03 media1 corosync[3657]: [TOTEM ] Incrementing problem 
counter for seqid 412343 iface 192.168.101.31 to [4 of 10]

Nov  6 02:33:04 media1 corosync[3657]: [TOTEM ] Incrementing problem 
counter for seqid 412345 iface 192.168.101.31 to [5 of 10]

Nov  6 02:33:05 media1 corosync[3657]: [TOTEM ] Incrementing problem 
counter for seqid 412347 iface 192.168.101.31 to [6 of 10]

Nov  6 02:33:05 media1 corosync[3657]: [TOTEM ] Decrementing problem 
counter for iface 192.168.101.31 to [5 of 10]

Nov  6 02:33:05 media1 corosync[3657]: [TOTEM ] Incrementing problem 
counter for seqid 412349 iface 192.168.101.31 to [6 of 10]

Nov  6 02:33:06 media1 corosync[3657]: [TOTEM ] Incrementing problem 
counter for seqid 412351 iface 192.168.101.31 to [7 of 10]

Nov  6 02:33:07 media1 corosync[3657]: [TOTEM ] Incrementing problem 
counter for seqid 412355 iface 192.168.101.31 to [8 of 10]

Nov  6 02:33:07 media1 corosync[3657]: [TOTEM ] Decrementing problem 
counter for iface 192.168.101.31 to [7 of 10]

Nov  6 02:33:07 media1 corosync[3657]: [TOTEM ] Incrementing problem 
counter for seqid 412357 iface 192.168.101.31 to [8 of 10]

Nov  6 02:33:08 media1 corosync[3657]: [TOTEM ] Incrementing problem 
counter for seqid 412361 iface 192.168.101.31 to [9 of 10]

Nov  6 02:33:09 media1 corosync[3657]: [TOTEM ] Incrementing problem 
counter for seqid 412363 iface 192.168.101.31 to [10 of 10]

Nov  6 02:33:09 media1 corosync[3657]: [TOTEM ] Marking seqid 412363 
ringid 1 interface 192.168.101.31 FAULTY

Nov  6 02:33:11 media1 crmd[4960]: notice: do_state_transition: State 
transition S_IDLE -> S_POLICY_ENGINE [ input=I_PE_CALC 
cause=C_FSA_INTERNAL origin=abort_transition_graph ]

Nov  6 02:33:11 media1 pengine[4959]: notice: unpack_config: On loss of 
CCM Quorum: Ignore

Nov  6 02:33:11 media1 pengine[4959]: warning: unpack_rsc_op: Processing 
failed op monitor for res_nfsserver_mediafiles on media1: unknown error 
(1)
Nov  6 02:33:11 media1 pengine[4959]: warning: unpack_rsc_op: Processing 
failed op monitor for res_nfsserver_mediafiles on media2: unknown error 
(1)
Nov  6 02:33:11 media1 pengine[4959]: notice: process_pe_message: 
Calculated Transition 145: /var/lib/pacemaker/pengine/pe-input-228.bz2

Nov  6 02:33:11 media1 crmd[4960]: notice: run_graph: Transition 145 
(Complete=0, Pending=0, Fired=0, Skipped=0, Incomplete=0, 
Source=/var/lib/pacemaker/pengine/pe-input-228.bz2): Complete
Nov  6 02:33:11 media1 crmd[4960]: notice: do_state_transition: State 
transition S_TRANSITION_ENGINE -> S_IDLE [ input=I_TE_SUCCESS 
cause=C_FSA_INTERNAL origin=notify_crmd ]
Nov  6 02:33:17 media1 corosync[3657]: [TOTEM ] Automatically recovered 
ring 1

Nov  6 02:33:18 media1 corosync[3657]: [TOTEM ] Incrementing problem 
counter for seqid 412387 iface 192.168.101.31 to [1 of 10]

Nov  6 02:33:20 media1 corosync[3657]: [TOTEM ] Incrementing problem 
counter for seqid 412389 iface 192.168.101.31 to [2 of 10]

Nov  6 02:33:20 media1 corosync[3657]: [TOTEM ] Incrementing problem 
counter for seqid 412391 iface 192.168.101.31 to [3 of 10]

Nov  6 02:33:21 media1 corosync[3657]: [TOTEM ] Incrementing problem 
counter for seqid 412393 iface 192.168.101.31 to [4 of 10]

Nov  6 02:33:22 media1 corosync[3657]: [TOTEM ] Incrementing problem 
counter for seqid 412397 iface 192.168.101.31 to [5 of 10]

Nov  6 02:33:22 media1 corosync[3657]: [TOTEM ] Incrementing problem 
counter for seqid 412399 iface 192.168.101.31 to [6 of 10]

Nov  6 02:33:24 media1 corosync[3657]: [TOTEM ] Incrementing problem 
counter for seqid 412403 iface 192.168.101.31 to [7 of 10]

Nov  6 02:33:31 media1 corosync[3657]: [TOTEM ] Incrementing problem 
counter for seqid 412421 iface 192.168.101.31 to [8 of 10]

Nov  6 02:33:32 media1 corosync[3657]: [TOTEM ] Incrementing problem 
counter for seqid 412423 iface 192.168.101.31 to [9 of 10]

Nov  6 02:33:34 media1 corosync[3657]: [TOTEM ] Incrementing problem 
counter for seqid 412429 iface 192.168.101.31 to [10 of 10]

Nov  6 02:33:34 media1 corosync[3657]: [TOTEM ] Marking seqid 412429 
ringid 1 interface 192.168.101.31 FAULTY

Nov  6 02:33:36 media1 corosync[3657]: [TOTEM ] Automatically recovered 
ring 1

Nov  6 02:33:39 media1 corosync[3657]: [TOTEM ] Incrementing problem 
counter for seqid 412437 iface 192.168.101.31 to [1 of 10]

============================================================
media2:
============================================================
Nov  6 02:32:59 media2 kernel: drbd r0: sock was shut down by peer

Nov  6 02:32:59 media2 kernel: drbd r0: peer( Primary -> Unknown ) conn( 
Connected -> BrokenPipe ) pdsk( UpToDate -> DUnknown )

Nov  6 02:32:59 media2 kernel: drbd r0: short read (expected size 16)

Nov  6 02:32:59 media2 kernel: drbd r0: asender terminated

Nov  6 02:32:59 media2 kernel: drbd r0: Terminating drbd_a_r0

Nov  6 02:32:59 media2 kernel: drbd r0: Connection closed

Nov  6 02:32:59 media2 kernel: drbd r0: conn( BrokenPipe -> Unconnected 
)

Nov  6 02:32:59 media2 kernel: drbd r0: receiver terminated

Nov  6 02:32:59 media2 kernel: drbd r0: Restarting receiver thread

Nov  6 02:32:59 media2 kernel: drbd r0: receiver (re)started

Nov  6 02:32:59 media2 kernel: drbd r0: conn( Unconnected -> 
WFConnection )

Nov  6 02:33:01 media2 attrd[4940]: notice: attrd_trigger_update: 
Sending flush op to all hosts for: master-res_drbd_mediafiles (1000)

Nov  6 02:33:01 media2 attrd[4940]: notice: attrd_perform_update: Sent 
update 16: master-res_drbd_mediafiles=1000

Nov  6 02:33:01 media2 kernel: drbd r0: Handshake successful: Agreed 
network protocol version 101

Nov  6 02:33:01 media2 kernel: drbd r0: Agreed to support TRIM on 
protocol level

Nov  6 02:33:01 media2 kernel: drbd r0: Peer authenticated using 20 
bytes HMAC

Nov  6 02:33:01 media2 kernel: drbd r0: conn( WFConnection -> 
WFReportParams )

Nov  6 02:33:01 media2 kernel: drbd r0: Starting asender thread (from 
drbd_r_r0 [6367])

Nov  6 02:33:01 media2 kernel: block drbd0: drbd_sync_handshake:

Nov  6 02:33:01 media2 kernel: block drbd0: self 
59E43E5FD7B26A4E:0000000000000000:1963884285B385E6:1962884285B385E6 
bits:0 flags:0
Nov  6 02:33:01 media2 kernel: block drbd0: peer 
0C0308FA4F94F811:59E43E5FD7B26A4F:1963884285B385E6:1962884285B385E6 
bits:192101 flags:0
Nov  6 02:33:01 media2 kernel: block drbd0: uuid_compare()=-1 by rule 50

Nov  6 02:33:01 media2 kernel: block drbd0: peer( Unknown -> Primary ) 
conn( WFReportParams -> WFBitMapT ) disk( UpToDate -> Outdated ) pdsk( 
DUnknown -> UpToDate )
Nov  6 02:33:02 media2 kernel: block drbd0: receive bitmap stats 
[Bytes(packets)]: plain 0(0), RLE 185(1), total 185; compression: 100.0%

Nov  6 02:33:03 media2 kernel: block drbd0: send bitmap stats 
[Bytes(packets)]: plain 0(0), RLE 185(1), total 185; compression: 100.0%

Nov  6 02:33:03 media2 kernel: block drbd0: conn( WFBitMapT -> 
WFSyncUUID )

Nov  6 02:33:03 media2 kernel: block drbd0: updated sync uuid 
59E53E5FD7B26A4E:0000000000000000:1963884285B385E6:1962884285B385E6

Nov  6 02:33:03 media2 kernel: block drbd0: helper command: 
/sbin/drbdadm before-resync-target minor-0

Nov  6 02:33:03 media2 kernel: block drbd0: helper command: 
/sbin/drbdadm before-resync-target minor-0 exit code 0 (0x0)

Nov  6 02:33:03 media2 kernel: block drbd0: conn( WFSyncUUID -> 
SyncTarget ) disk( Outdated -> Inconsistent )

Nov  6 02:33:03 media2 kernel: block drbd0: Began resync as SyncTarget 
(will sync 919244 KB [229811 bits set]).

Nov  6 02:33:10 media2 corosync[3690]: [TOTEM ] Incrementing problem 
counter for seqid 412364 iface 192.168.101.32 to [1 of 10]

Nov  6 02:33:10 media2 corosync[3690]: [TOTEM ] Automatically recovered 
ring 1

Nov  6 02:33:10 media2 corosync[3690]: [TOTEM ] Incrementing problem 
counter for seqid 412366 iface 192.168.101.32 to [1 of 10]

Nov  6 02:33:11 media2 corosync[3690]: [TOTEM ] Automatically recovered 
ring 1

Nov  6 02:33:11 media2 attrd[4940]: notice: attrd_trigger_update: 
Sending flush op to all hosts for: master-res_drbd_mediafiles (10)

Nov  6 02:33:11 media2 attrd[4940]: notice: attrd_perform_update: Sent 
update 18: master-res_drbd_mediafiles=10

Nov  6 02:33:11 media2 corosync[3690]: [TOTEM ] Incrementing problem 
counter for seqid 412368 iface 192.168.101.32 to [1 of 10]

Nov  6 02:33:12 media2 corosync[3690]: [TOTEM ] ring 1 active with no 
faults

Nov  6 02:33:12 media2 corosync[3690]: [TOTEM ] Incrementing problem 
counter for seqid 412370 iface 192.168.101.32 to [1 of 10]

Nov  6 02:33:12 media2 corosync[3690]: [TOTEM ] Automatically recovered 
ring 1

Nov  6 02:33:12 media2 corosync[3690]: [TOTEM ] Incrementing problem 
counter for seqid 412372 iface 192.168.101.32 to [1 of 10]

Nov  6 02:33:13 media2 corosync[3690]: [TOTEM ] Incrementing problem 
counter for seqid 412374 iface 192.168.101.32 to [2 of 10]

Nov  6 02:33:14 media2 corosync[3690]: [TOTEM ] Decrementing problem 
counter for iface 192.168.101.32 to [1 of 10]

Nov  6 02:33:14 media2 corosync[3690]: [TOTEM ] Incrementing problem 
counter for seqid 412376 iface 192.168.101.32 to [2 of 10]

Nov  6 02:33:15 media2 corosync[3690]: [TOTEM ] Incrementing problem 
counter for seqid 412378 iface 192.168.101.32 to [3 of 10]

Nov  6 02:33:15 media2 corosync[3690]: [TOTEM ] Automatically recovered 
ring 1

Nov  6 02:33:15 media2 corosync[3690]: [TOTEM ] Incrementing problem 
counter for seqid 412380 iface 192.168.101.32 to [1 of 10]

Nov  6 02:33:16 media2 corosync[3690]: [TOTEM ] ring 1 active with no 
faults

Nov  6 02:33:16 media2 corosync[3690]: [TOTEM ] Incrementing problem 
counter for seqid 412382 iface 192.168.101.32 to [1 of 10]

Nov  6 02:33:16 media2 corosync[3690]: [TOTEM ] Automatically recovered 
ring 1

Nov  6 02:33:17 media2 corosync[3690]: [TOTEM ] Incrementing problem 
counter for seqid 412384 iface 192.168.101.32 to [1 of 10]

Nov  6 02:33:17 media2 corosync[3690]: [TOTEM ] Automatically recovered 
ring 1

Nov  6 02:33:17 media2 corosync[3690]: [TOTEM ] Incrementing problem 
counter for seqid 412386 iface 192.168.101.32 to [1 of 10]

Nov  6 02:33:18 media2 corosync[3690]: [TOTEM ] ring 1 active with no 
faults

Nov  6 02:33:19 media2 corosync[3690]: [TOTEM ] Incrementing problem 
counter for seqid 412388 iface 192.168.100.32 to [1 of 10]

Nov  6 02:33:20 media2 corosync[3690]: [TOTEM ] ring 0 active with no 
faults

Nov  6 02:33:35 media2 corosync[3690]: [TOTEM ] Automatically recovered 
ring 1

Nov  6 02:33:36 media2 corosync[3690]: [TOTEM ] Incrementing problem 
counter for seqid 412430 iface 192.168.101.32 to [1 of 10]

Nov  6 02:33:36 media2 corosync[3690]: [TOTEM ] Automatically recovered 
ring 1

Nov  6 02:33:37 media2 corosync[3690]: [TOTEM ] Incrementing problem 
counter for seqid 412432 iface 192.168.101.32 to [1 of 10]

Nov  6 02:33:38 media2 corosync[3690]: [TOTEM ] ring 1 active with no 
faults

[Jiri Pirko]

Thu, Nov 06, 2014 at 04:26:36PM CET, [e-mail] wrote:
> Hello,
>
> I am currently setting up a high available NFS server cluster with 2 nodes
> using pacemaker + corosync + drbd + flashcache.
> The two nodes (media1 and media2) have 4 1G copper nics each.
> I formed 2 teams:
> team0: LACP with em1 and em2 connected to a switch for NFS services
> team1: roundrobin with em3 and em4 and MTU 9000 directly connected with
> copper patch cables for drbd sync
> During my testing everything was working as expected. Then I started to
> tranfer about 20TB from an old server to the new cluster. The copy process
> was fine for about 32 hours. Then corosync and drbd started to report errors
> on the roundrobin team1 interface (see logs at the end of this email).
> The drbd sync and corosync started to flap between states for several hours
> on team1.
> I checked the error counters with ifconfig and found a huge amount off TX
> dropped packets on team1 of the drbd slave node media2.
> There were NO other error counters raising but team1 on media2.
> To make this a bit more clear:
> team1 on media2 (the drbd slave) showed raising TX dropped packets
> em3 and em4 on media2 (the physical interfaces of team1) did not show any
>
> raising error counters.
> team1 on media1 (the drbd master) did not show any raising error counters.
> em3 and em4 on media1 (the physical interfaces of team1) did not show any
>
> raising error counters.
> I then thought maybe one of the two patch cables between the two machines was
> not 100% ok. So I removed em4 from team1 on both machines:
> "teamdctl team1 port remove em4"
> The error counters stopped raising.
> I thought: "Yeah! That cable must be broken!"
> But just to make sure I really found the problem I re-enabled em4 on both
> nodes:
> "teamdctl team1 port add em4"
> Ok, now both team1 interfaces had 2 links up again and the error counter
> restarted to raise again.
> I then stopped em3 on both nodes to force all traffic through em4:
> "teamdctl team1 port remove em3"
> And guess what? The error counters on team1 stopped raising!
> So in short:
> If I remove either one of the two physical interfaces from team1 everything
> is working without any errors. As soon as I enable both physical interfaces
> the error counters start to raise.
> After these tests I rebooted the slave node (media2) and added both ports to
> team1 again (Have you tried turning it off and on again...).
> After doing this, I now see TX dropped packets on the not yet rebooted media1
> team1 interface.
> Why do I only see these errors on the team1 interface and not on em3 and em4?
>
> Currently the copy process of 20TB is still running. I think that if I reboot
> media1 everything will work as expected again for some time because it worked
> during my initial tests.
> But I do think that I hit a bug in the teaming driver and the raising error
> counters will come back.
>
> Could anybody help?

Interesting. Could you do "teamdctl team1 s d" when the issue happens
and post me the result?

Also you can try random runner instead of roundrobin. The resulting
behaviour is almost the same. Maybe the issue won't happen there.

Also, can you please post me output of:
ethtool -i em3
ethtool em3
ethtool -S em3
ethtool -i em4
ethtool em4
ethtool -S em4
Of course in time the problem happens.
> Kind regards
> Ingo
>
> This is the used network config of team1:
> media1:
> cat /etc/sysconfig/network-scripts/ifcfg-team1_slave_0
> # Generated by parse-kickstart
> NAME=team1 slave 0
> TEAM_MASTER=team1
> DEVICETYPE=TeamPort
> DEVICE=em3
> ONBOOT=yes
> UUID=b7026c5b-e9cd-457c-93d2-a2799361ed90
>
> cat /etc/sysconfig/network-scripts/ifcfg-team1_slave_1
> # Generated by parse-kickstart
> NAME=team1 slave 1
> TEAM_MASTER=team1
> DEVICETYPE=TeamPort
> DEVICE=em4
> ONBOOT=yes
> UUID=fbe5bc56-95f5-4f9e-a5ea-ab6b6f5e5b50
>
> cat /etc/sysconfig/network-scripts/ifcfg-team1
> # Generated by parse-kickstart
> UUID=4e0090cd-32d3-4c5b-be23-9f53083da4dd
> NAME="Team connection team1"
> TEAM_CONFIG="{"runner": {"name": "roundrobin"}}"
> GATEWAY=
> IPV6_AUTOCONF=yes
> BOOTPROTO=none
> DEVICE=team1
> MTU=9000
> TYPE=Team
> ONBOOT=yes
> IPV6INIT=yes
> DEVICETYPE=Team
> IPADDR0=192.168.101.31
> PREFIX0=24
> DEFROUTE=yes
> IPV4_FAILURE_FATAL=no
> IPV6_DEFROUTE=yes
> IPV6_PEERDNS=yes
> IPV6_PEERROUTES=yes
> IPV6_FAILURE_FATAL=no
>
> media2:
> cat /etc/sysconfig/network-scripts/ifcfg-team1_slave_0
> # Generated by parse-kickstart
> NAME=team1 slave 0
> TEAM_MASTER=team1
> DEVICETYPE=TeamPort
> DEVICE=em3
> ONBOOT=yes
> UUID=bfd8e77e-7eb8-47ea-ab7e-95286e073202
>
> cat /etc/sysconfig/network-scripts/ifcfg-team1_slave_1
> # Generated by parse-kickstart
> NAME=team1 slave 1
> TEAM_MASTER=team1
> DEVICETYPE=TeamPort
> DEVICE=em4
> ONBOOT=yes
> UUID=93c5dfd8-3b81-4d1e-8ba9-c46f8daceb1a
>
> cat /etc/sysconfig/network-scripts/ifcfg-team1
> # Generated by parse-kickstart
> UUID=79c843db-996c-41e4-9ee5-2a8f3da244ed
> NAME="Team connection team1"
> TEAM_CONFIG="{"runner": {"name": "roundrobin"}}"
> GATEWAY=
> IPV6_AUTOCONF=yes
> BOOTPROTO=none
> DEVICE=team1
> MTU=9000
> TYPE=Team
> ONBOOT=yes
> IPV6INIT=yes
> DEVICETYPE=Team
> IPADDR0=192.168.101.32
> PREFIX0=24
> DEFROUTE=yes
> IPV4_FAILURE_FATAL=no
> IPV6_DEFROUTE=yes
> IPV6_PEERDNS=yes
> IPV6_PEERROUTES=yes
> IPV6_FAILURE_FATAL=no
> Here are some system logs from both machines:
> media1:
>
> Nov  6 02:31:27 media1 corosync[3657]: [TOTEM ] Incrementing problem counter
> for seqid 412041 iface 192.168.100.31 to [1 of 10]
> Nov  6 02:31:29 media1 corosync[3657]: [TOTEM ] ring 0 active with no faults
> Nov  6 02:31:29 media1 corosync[3657]: [TOTEM ] Incrementing problem counter
> for seqid 412047 iface 192.168.100.31 to [1 of 10]
> Nov  6 02:31:31 media1 corosync[3657]: [TOTEM ] ring 0 active with no faults
> Nov  6 02:32:51 media1 corosync[3657]: [TOTEM ] Incrementing problem counter
> for seqid 412315 iface 192.168.101.31 to [1 of 10]
> Nov  6 02:32:53 media1 corosync[3657]: [TOTEM ] ring 1 active with no faults
> Nov  6 02:32:53 media1 corosync[3657]: [TOTEM ] Incrementing problem counter
> for seqid 412319 iface 192.168.101.31 to [1 of 10]
> Nov  6 02:32:54 media1 corosync[3657]: [TOTEM ] Incrementing problem counter
> for seqid 412321 iface 192.168.101.31 to [2 of 10]
> Nov  6 02:32:55 media1 corosync[3657]: [TOTEM ] Decrementing problem counter
> for iface 192.168.101.31 to [1 of 10]
> Nov  6 02:32:56 media1 corosync[3657]: [TOTEM ] Incrementing problem counter
> for seqid 412325 iface 192.168.101.31 to [2 of 10]
> Nov  6 02:32:57 media1 corosync[3657]: [TOTEM ] Decrementing problem counter
> for iface 192.168.101.31 to [1 of 10]
> Nov  6 02:32:58 media1 corosync[3657]: [TOTEM ] Incrementing problem counter
> for seqid 412327 iface 192.168.101.31 to [2 of 10]
> Nov  6 02:32:59 media1 kernel: drbd r0: PingAck did not arrive in time.
> Nov  6 02:32:59 media1 kernel: drbd r0: peer( Secondary -> Unknown ) conn(
> Connected -> NetworkFailure ) pdsk( UpToDate -> DUnknown )
> Nov  6 02:32:59 media1 kernel: block drbd0: new current UUID
> 0C0308FA4F94F811:59E43E5FD7B26A4F:1963884285B385E6:1962884285B385E6
> Nov  6 02:32:59 media1 kernel: drbd r0: asender terminated
> Nov  6 02:32:59 media1 kernel: drbd r0: Terminating drbd_a_r0
> Nov  6 02:32:59 media1 kernel: drbd r0: Connection closed
> Nov  6 02:32:59 media1 kernel: drbd r0: conn( NetworkFailure -> Unconnected )
> Nov  6 02:32:59 media1 kernel: drbd r0: receiver terminated
> Nov  6 02:32:59 media1 kernel: drbd r0: Restarting receiver thread
> Nov  6 02:32:59 media1 kernel: drbd r0: receiver (re)started
> Nov  6 02:32:59 media1 kernel: drbd r0: conn( Unconnected -> WFConnection )
> Nov  6 02:32:59 media1 corosync[3657]: [TOTEM ] Decrementing problem counter
> for iface 192.168.101.31 to [1 of 10]
> Nov  6 02:32:59 media1 corosync[3657]: [TOTEM ] Incrementing problem counter
> for seqid 412329 iface 192.168.101.31 to [2 of 10]
> Nov  6 02:33:01 media1 corosync[3657]: [TOTEM ] Incrementing problem counter
> for seqid 412333 iface 192.168.101.31 to [3 of 10]
> Nov  6 02:33:01 media1 corosync[3657]: [TOTEM ] Decrementing problem counter
> for iface 192.168.101.31 to [2 of 10]
> Nov  6 02:33:01 media1 crmd[4960]: notice: do_state_transition: State
> transition S_IDLE -> S_POLICY_ENGINE [ input=I_PE_CALC cause=C_FSA_INTERNAL
> origin=abort_transition_graph ]
>
> Nov  6 02:33:01 media1 pengine[4959]: notice: unpack_config: On loss of CCM
> Quorum: Ignore
>
> Nov  6 02:33:01 media1 pengine[4959]: warning: unpack_rsc_op: Processing
> failed op monitor for res_nfsserver_mediafiles on media1: unknown error (1)
> Nov  6 02:33:01 media1 pengine[4959]: warning: unpack_rsc_op: Processing
> failed op monitor for res_nfsserver_mediafiles on media2: unknown error (1)
> Nov  6 02:33:01 media1 pengine[4959]: notice: process_pe_message: Calculated
> Transition 144: /var/lib/pacemaker/pengine/pe-input-227.bz2
>
> Nov  6 02:33:01 media1 crmd[4960]: notice: run_graph: Transition 144
> (Complete=0, Pending=0, Fired=0, Skipped=0, Incomplete=0,
> Source=/var/lib/pacemaker/pengine/pe-input-227.bz2): Complete
>
> Nov  6 02:33:01 media1 crmd[4960]: notice: do_state_transition: State
> transition S_TRANSITION_ENGINE -> S_IDLE [ input=I_TE_SUCCESS
> cause=C_FSA_INTERNAL origin=notify_crmd ]
> Nov  6 02:33:01 media1 kernel: drbd r0: Handshake successful: Agreed network
> protocol version 101
>
> Nov  6 02:33:01 media1 kernel: drbd r0: Agreed to support TRIM on protocol
> level
>
> Nov  6 02:33:01 media1 kernel: drbd r0: Peer authenticated using 20 bytes
> HMAC
>
> Nov  6 02:33:01 media1 kernel: drbd r0: conn( WFConnection -> WFReportParams
> )
>
> Nov  6 02:33:01 media1 kernel: drbd r0: Starting asender thread (from
> drbd_r_r0 [5999])
>
> Nov  6 02:33:02 media1 corosync[3657]: [TOTEM ] Incrementing problem counter
> for seqid 412337 iface 192.168.101.31 to [3 of 10]
>
> Nov  6 02:33:02 media1 kernel: block drbd0: drbd_sync_handshake:
>
> Nov  6 02:33:02 media1 kernel: block drbd0: self
> 0C0308FA4F94F811:59E43E5FD7B26A4F:1963884285B385E6:1962884285B385E6
> bits:192101 flags:0
> Nov  6 02:33:02 media1 kernel: block drbd0: peer
> 59E43E5FD7B26A4E:0000000000000000:1963884285B385E6:1962884285B385E6 bits:0
> flags:0
> Nov  6 02:33:02 media1 kernel: block drbd0: uuid_compare()=1 by rule 70
>
> Nov  6 02:33:02 media1 kernel: block drbd0: peer( Unknown -> Secondary )
> conn( WFReportParams -> WFBitMapS ) pdsk( DUnknown -> Consistent )
>
> Nov  6 02:33:02 media1 kernel: block drbd0: send bitmap stats
> [Bytes(packets)]: plain 0(0), RLE 185(1), total 185; compression: 100.0%
>
> Nov  6 02:33:02 media1 corosync[3657]: [TOTEM ] Incrementing problem counter
> for seqid 412339 iface 192.168.101.31 to [4 of 10]
>
> Nov  6 02:33:03 media1 kernel: block drbd0: receive bitmap stats
> [Bytes(packets)]: plain 0(0), RLE 185(1), total 185; compression: 100.0%
>
> Nov  6 02:33:03 media1 kernel: block drbd0: helper command: /sbin/drbdadm
> before-resync-source minor-0
>
> Nov  6 02:33:03 media1 kernel: block drbd0: helper command: /sbin/drbdadm
> before-resync-source minor-0 exit code 0 (0x0)
>
> Nov  6 02:33:03 media1 kernel: block drbd0: conn( WFBitMapS -> SyncSource )
> pdsk( Consistent -> Inconsistent )
>
> Nov  6 02:33:03 media1 kernel: block drbd0: Began resync as SyncSource (will
> sync 919244 KB [229811 bits set]).
>
> Nov  6 02:33:03 media1 kernel: block drbd0: updated sync UUID
> 0C0308FA4F94F811:59E53E5FD7B26A4F:59E43E5FD7B26A4F:1963884285B385E6
>
> Nov  6 02:33:03 media1 corosync[3657]: [TOTEM ] Decrementing problem counter
> for iface 192.168.101.31 to [3 of 10]
>
> Nov  6 02:33:03 media1 corosync[3657]: [TOTEM ] Incrementing problem counter
> for seqid 412343 iface 192.168.101.31 to [4 of 10]
>
> Nov  6 02:33:04 media1 corosync[3657]: [TOTEM ] Incrementing problem counter
> for seqid 412345 iface 192.168.101.31 to [5 of 10]
>
> Nov  6 02:33:05 media1 corosync[3657]: [TOTEM ] Incrementing problem counter
> for seqid 412347 iface 192.168.101.31 to [6 of 10]
>
> Nov  6 02:33:05 media1 corosync[3657]: [TOTEM ] Decrementing problem counter
> for iface 192.168.101.31 to [5 of 10]
>
> Nov  6 02:33:05 media1 corosync[3657]: [TOTEM ] Incrementing problem counter
> for seqid 412349 iface 192.168.101.31 to [6 of 10]
>
> Nov  6 02:33:06 media1 corosync[3657]: [TOTEM ] Incrementing problem counter
> for seqid 412351 iface 192.168.101.31 to [7 of 10]
>
> Nov  6 02:33:07 media1 corosync[3657]: [TOTEM ] Incrementing problem counter
> for seqid 412355 iface 192.168.101.31 to [8 of 10]
>
> Nov  6 02:33:07 media1 corosync[3657]: [TOTEM ] Decrementing problem counter
> for iface 192.168.101.31 to [7 of 10]
>
> Nov  6 02:33:07 media1 corosync[3657]: [TOTEM ] Incrementing problem counter
> for seqid 412357 iface 192.168.101.31 to [8 of 10]
>
> Nov  6 02:33:08 media1 corosync[3657]: [TOTEM ] Incrementing problem counter
> for seqid 412361 iface 192.168.101.31 to [9 of 10]
>
> Nov  6 02:33:09 media1 corosync[3657]: [TOTEM ] Incrementing problem counter
> for seqid 412363 iface 192.168.101.31 to [10 of 10]
>
> Nov  6 02:33:09 media1 corosync[3657]: [TOTEM ] Marking seqid 412363 ringid 1
> interface 192.168.101.31 FAULTY
>
> Nov  6 02:33:11 media1 crmd[4960]: notice: do_state_transition: State
> transition S_IDLE -> S_POLICY_ENGINE [ input=I_PE_CALC cause=C_FSA_INTERNAL
> origin=abort_transition_graph ]
>
> Nov  6 02:33:11 media1 pengine[4959]: notice: unpack_config: On loss of CCM
> Quorum: Ignore
>
> Nov  6 02:33:11 media1 pengine[4959]: warning: unpack_rsc_op: Processing
> failed op monitor for res_nfsserver_mediafiles on media1: unknown error (1)
> Nov  6 02:33:11 media1 pengine[4959]: warning: unpack_rsc_op: Processing
> failed op monitor for res_nfsserver_mediafiles on media2: unknown error (1)
> Nov  6 02:33:11 media1 pengine[4959]: notice: process_pe_message: Calculated
> Transition 145: /var/lib/pacemaker/pengine/pe-input-228.bz2
>
> Nov  6 02:33:11 media1 crmd[4960]: notice: run_graph: Transition 145
> (Complete=0, Pending=0, Fired=0, Skipped=0, Incomplete=0,
> Source=/var/lib/pacemaker/pengine/pe-input-228.bz2): Complete
> Nov  6 02:33:11 media1 crmd[4960]: notice: do_state_transition: State
> transition S_TRANSITION_ENGINE -> S_IDLE [ input=I_TE_SUCCESS
> cause=C_FSA_INTERNAL origin=notify_crmd ]
> Nov  6 02:33:17 media1 corosync[3657]: [TOTEM ] Automatically recovered ring
> 1
>
> Nov  6 02:33:18 media1 corosync[3657]: [TOTEM ] Incrementing problem counter
> for seqid 412387 iface 192.168.101.31 to [1 of 10]
>
> Nov  6 02:33:20 media1 corosync[3657]: [TOTEM ] Incrementing problem counter
> for seqid 412389 iface 192.168.101.31 to [2 of 10]
>
> Nov  6 02:33:20 media1 corosync[3657]: [TOTEM ] Incrementing problem counter
> for seqid 412391 iface 192.168.101.31 to [3 of 10]
>
> Nov  6 02:33:21 media1 corosync[3657]: [TOTEM ] Incrementing problem counter
> for seqid 412393 iface 192.168.101.31 to [4 of 10]
>
> Nov  6 02:33:22 media1 corosync[3657]: [TOTEM ] Incrementing problem counter
> for seqid 412397 iface 192.168.101.31 to [5 of 10]
>
> Nov  6 02:33:22 media1 corosync[3657]: [TOTEM ] Incrementing problem counter
> for seqid 412399 iface 192.168.101.31 to [6 of 10]
>
> Nov  6 02:33:24 media1 corosync[3657]: [TOTEM ] Incrementing problem counter
> for seqid 412403 iface 192.168.101.31 to [7 of 10]
>
> Nov  6 02:33:31 media1 corosync[3657]: [TOTEM ] Incrementing problem counter
> for seqid 412421 iface 192.168.101.31 to [8 of 10]
>
> Nov  6 02:33:32 media1 corosync[3657]: [TOTEM ] Incrementing problem counter
> for seqid 412423 iface 192.168.101.31 to [9 of 10]
>
> Nov  6 02:33:34 media1 corosync[3657]: [TOTEM ] Incrementing problem counter
> for seqid 412429 iface 192.168.101.31 to [10 of 10]
>
> Nov  6 02:33:34 media1 corosync[3657]: [TOTEM ] Marking seqid 412429 ringid 1
> interface 192.168.101.31 FAULTY
>
> Nov  6 02:33:36 media1 corosync[3657]: [TOTEM ] Automatically recovered ring
> 1
>
> Nov  6 02:33:39 media1 corosync[3657]: [TOTEM ] Incrementing problem counter
> for seqid 412437 iface 192.168.101.31 to [1 of 10]
>
> ============================================================
> media2:
> ============================================================
> Nov  6 02:32:59 media2 kernel: drbd r0: sock was shut down by peer
>
> Nov  6 02:32:59 media2 kernel: drbd r0: peer( Primary -> Unknown ) conn(
> Connected -> BrokenPipe ) pdsk( UpToDate -> DUnknown )
>
> Nov  6 02:32:59 media2 kernel: drbd r0: short read (expected size 16)
>
> Nov  6 02:32:59 media2 kernel: drbd r0: asender terminated
>
> Nov  6 02:32:59 media2 kernel: drbd r0: Terminating drbd_a_r0
>
> Nov  6 02:32:59 media2 kernel: drbd r0: Connection closed
>
> Nov  6 02:32:59 media2 kernel: drbd r0: conn( BrokenPipe -> Unconnected )
>
> Nov  6 02:32:59 media2 kernel: drbd r0: receiver terminated
>
> Nov  6 02:32:59 media2 kernel: drbd r0: Restarting receiver thread
>
> Nov  6 02:32:59 media2 kernel: drbd r0: receiver (re)started
>
> Nov  6 02:32:59 media2 kernel: drbd r0: conn( Unconnected -> WFConnection )
>
> Nov  6 02:33:01 media2 attrd[4940]: notice: attrd_trigger_update: Sending
> flush op to all hosts for: master-res_drbd_mediafiles (1000)
>
> Nov  6 02:33:01 media2 attrd[4940]: notice: attrd_perform_update: Sent update
> 16: master-res_drbd_mediafiles=1000
>
> Nov  6 02:33:01 media2 kernel: drbd r0: Handshake successful: Agreed network
> protocol version 101
>
> Nov  6 02:33:01 media2 kernel: drbd r0: Agreed to support TRIM on protocol
> level
>
> Nov  6 02:33:01 media2 kernel: drbd r0: Peer authenticated using 20 bytes
> HMAC
>
> Nov  6 02:33:01 media2 kernel: drbd r0: conn( WFConnection -> WFReportParams
> )
>
> Nov  6 02:33:01 media2 kernel: drbd r0: Starting asender thread (from
> drbd_r_r0 [6367])
>
> Nov  6 02:33:01 media2 kernel: block drbd0: drbd_sync_handshake:
>
> Nov  6 02:33:01 media2 kernel: block drbd0: self
> 59E43E5FD7B26A4E:0000000000000000:1963884285B385E6:1962884285B385E6 bits:0
> flags:0
> Nov  6 02:33:01 media2 kernel: block drbd0: peer
> 0C0308FA4F94F811:59E43E5FD7B26A4F:1963884285B385E6:1962884285B385E6
> bits:192101 flags:0
> Nov  6 02:33:01 media2 kernel: block drbd0: uuid_compare()=-1 by rule 50
>
> Nov  6 02:33:01 media2 kernel: block drbd0: peer( Unknown -> Primary ) conn(
> WFReportParams -> WFBitMapT ) disk( UpToDate -> Outdated ) pdsk( DUnknown ->
> UpToDate )
> Nov  6 02:33:02 media2 kernel: block drbd0: receive bitmap stats
> [Bytes(packets)]: plain 0(0), RLE 185(1), total 185; compression: 100.0%
>
> Nov  6 02:33:03 media2 kernel: block drbd0: send bitmap stats
> [Bytes(packets)]: plain 0(0), RLE 185(1), total 185; compression: 100.0%
>
> Nov  6 02:33:03 media2 kernel: block drbd0: conn( WFBitMapT -> WFSyncUUID )
>
> Nov  6 02:33:03 media2 kernel: block drbd0: updated sync uuid
> 59E53E5FD7B26A4E:0000000000000000:1963884285B385E6:1962884285B385E6
>
> Nov  6 02:33:03 media2 kernel: block drbd0: helper command: /sbin/drbdadm
> before-resync-target minor-0
>
> Nov  6 02:33:03 media2 kernel: block drbd0: helper command: /sbin/drbdadm
> before-resync-target minor-0 exit code 0 (0x0)
>
> Nov  6 02:33:03 media2 kernel: block drbd0: conn( WFSyncUUID -> SyncTarget )
> disk( Outdated -> Inconsistent )
>
> Nov  6 02:33:03 media2 kernel: block drbd0: Began resync as SyncTarget (will
> sync 919244 KB [229811 bits set]).
>
> Nov  6 02:33:10 media2 corosync[3690]: [TOTEM ] Incrementing problem counter
> for seqid 412364 iface 192.168.101.32 to [1 of 10]
>
> Nov  6 02:33:10 media2 corosync[3690]: [TOTEM ] Automatically recovered ring
> 1
>
> Nov  6 02:33:10 media2 corosync[3690]: [TOTEM ] Incrementing problem counter
> for seqid 412366 iface 192.168.101.32 to [1 of 10]
>
> Nov  6 02:33:11 media2 corosync[3690]: [TOTEM ] Automatically recovered ring
> 1
>
> Nov  6 02:33:11 media2 attrd[4940]: notice: attrd_trigger_update: Sending
> flush op to all hosts for: master-res_drbd_mediafiles (10)
>
> Nov  6 02:33:11 media2 attrd[4940]: notice: attrd_perform_update: Sent update
> 18: master-res_drbd_mediafiles=10
>
> Nov  6 02:33:11 media2 corosync[3690]: [TOTEM ] Incrementing problem counter
> for seqid 412368 iface 192.168.101.32 to [1 of 10]
>
> Nov  6 02:33:12 media2 corosync[3690]: [TOTEM ] ring 1 active with no faults
>
> Nov  6 02:33:12 media2 corosync[3690]: [TOTEM ] Incrementing problem counter
> for seqid 412370 iface 192.168.101.32 to [1 of 10]
>
> Nov  6 02:33:12 media2 corosync[3690]: [TOTEM ] Automatically recovered ring
> 1
>
> Nov  6 02:33:12 media2 corosync[3690]: [TOTEM ] Incrementing problem counter
> for seqid 412372 iface 192.168.101.32 to [1 of 10]
>
> Nov  6 02:33:13 media2 corosync[3690]: [TOTEM ] Incrementing problem counter
> for seqid 412374 iface 192.168.101.32 to [2 of 10]
>
> Nov  6 02:33:14 media2 corosync[3690]: [TOTEM ] Decrementing problem counter
> for iface 192.168.101.32 to [1 of 10]
>
> Nov  6 02:33:14 media2 corosync[3690]: [TOTEM ] Incrementing problem counter
> for seqid 412376 iface 192.168.101.32 to [2 of 10]
>
> Nov  6 02:33:15 media2 corosync[3690]: [TOTEM ] Incrementing problem counter
> for seqid 412378 iface 192.168.101.32 to [3 of 10]
>
> Nov  6 02:33:15 media2 corosync[3690]: [TOTEM ] Automatically recovered ring
> 1
>
> Nov  6 02:33:15 media2 corosync[3690]: [TOTEM ] Incrementing problem counter
> for seqid 412380 iface 192.168.101.32 to [1 of 10]
>
> Nov  6 02:33:16 media2 corosync[3690]: [TOTEM ] ring 1 active with no faults
>
> Nov  6 02:33:16 media2 corosync[3690]: [TOTEM ] Incrementing problem counter
> for seqid 412382 iface 192.168.101.32 to [1 of 10]
>
> Nov  6 02:33:16 media2 corosync[3690]: [TOTEM ] Automatically recovered ring
> 1
>
> Nov  6 02:33:17 media2 corosync[3690]: [TOTEM ] Incrementing problem counter
> for seqid 412384 iface 192.168.101.32 to [1 of 10]
>
> Nov  6 02:33:17 media2 corosync[3690]: [TOTEM ] Automatically recovered ring
> 1
>
> Nov  6 02:33:17 media2 corosync[3690]: [TOTEM ] Incrementing problem counter
> for seqid 412386 iface 192.168.101.32 to [1 of 10]
>
> Nov  6 02:33:18 media2 corosync[3690]: [TOTEM ] ring 1 active with no faults
>
> Nov  6 02:33:19 media2 corosync[3690]: [TOTEM ] Incrementing problem counter
> for seqid 412388 iface 192.168.100.32 to [1 of 10]
>
> Nov  6 02:33:20 media2 corosync[3690]: [TOTEM ] ring 0 active with no faults
>
> Nov  6 02:33:35 media2 corosync[3690]: [TOTEM ] Automatically recovered ring
> 1
>
> Nov  6 02:33:36 media2 corosync[3690]: [TOTEM ] Incrementing problem counter
> for seqid 412430 iface 192.168.101.32 to [1 of 10]
>
> Nov  6 02:33:36 media2 corosync[3690]: [TOTEM ] Automatically recovered ring
> 1
>
> Nov  6 02:33:37 media2 corosync[3690]: [TOTEM ] Incrementing problem counter
> for seqid 412432 iface 192.168.101.32 to [1 of 10]
>
> Nov  6 02:33:38 media2 corosync[3690]: [TOTEM ] ring 1 active with no faults
>
> libteam mailing list
> [e-mail]
> https://lists.fedorahosted.org/mailman/listinfo/libteam

[Ingo Brand]

Am 06.11.2014 um 17:03 schrieb Jiri Pirko:
> Thu, Nov 06, 2014 at 04:26:36PM CET, [e-mail] wrote:
> > Hello,
> >
> > I am currently setting up a high available NFS server cluster with 2 nodes
> > using pacemaker + corosync + drbd + flashcache.
> > The two nodes (media1 and media2) have 4 1G copper nics each.
> > I formed 2 teams:
> > team0: LACP with em1 and em2 connected to a switch for NFS services
> > team1: roundrobin with em3 and em4 and MTU 9000 directly connected with
> > copper patch cables for drbd sync
> > During my testing everything was working as expected. Then I started to
> > tranfer about 20TB from an old server to the new cluster. The copy process
> > was fine for about 32 hours. Then corosync and drbd started to report errors
> > on the roundrobin team1 interface (see logs at the end of this email).
> > The drbd sync and corosync started to flap between states for several hours
> > on team1.
> > I checked the error counters with ifconfig and found a huge amount off TX
> > dropped packets on team1 of the drbd slave node media2.
> > There were NO other error counters raising but team1 on media2.
> > To make this a bit more clear:
> > team1 on media2 (the drbd slave) showed raising TX dropped packets
> > em3 and em4 on media2 (the physical interfaces of team1) did not show any
> >
> > raising error counters.
> > team1 on media1 (the drbd master) did not show any raising error counters.
> > em3 and em4 on media1 (the physical interfaces of team1) did not show any
> >
> > raising error counters.
> > I then thought maybe one of the two patch cables between the two machines was
> > not 100% ok. So I removed em4 from team1 on both machines:
> > "teamdctl team1 port remove em4"
> > The error counters stopped raising.
> > I thought: "Yeah! That cable must be broken!"
> > But just to make sure I really found the problem I re-enabled em4 on both
> > nodes:
> > "teamdctl team1 port add em4"
> > Ok, now both team1 interfaces had 2 links up again and the error counter
> > restarted to raise again.
> > I then stopped em3 on both nodes to force all traffic through em4:
> > "teamdctl team1 port remove em3"
> > And guess what? The error counters on team1 stopped raising!
> > So in short:
> > If I remove either one of the two physical interfaces from team1 everything
> > is working without any errors. As soon as I enable both physical interfaces
> > the error counters start to raise.
> > After these tests I rebooted the slave node (media2) and added both ports to
> > team1 again (Have you tried turning it off and on again...).
> > After doing this, I now see TX dropped packets on the not yet rebooted media1
> > team1 interface.
> > Why do I only see these errors on the team1 interface and not on em3 and em4?
> >
> > Currently the copy process of 20TB is still running. I think that if I reboot
> > media1 everything will work as expected again for some time because it worked
> > during my initial tests.
> > But I do think that I hit a bug in the teaming driver and the raising error
> > counters will come back.
> >
> > Could anybody help?
>
> Interesting. Could you do "teamdctl team1 s d" when the issue happens
> and post me the result?

I just re-enabled port em4 on both machines and the issue happens again. 
Here are your requested results:
media1 ~]# teamdctl team1 s d
{
     "ports": {
         "em3": {
             "ifinfo": {
                 "dev_addr": "b8:2a:72:d5:6c:e6",
                 "dev_addr_len": 6,
                 "ifindex": 4,
                 "ifname": "em3"
             },
             "link": {
                 "duplex": "full",
                 "speed": 1000,
                 "up": true
             },
             "link_watches": {
                 "list": {
                     "link_watch_0": {
                         "delay_down": 0,
                         "delay_up": 0,
                         "name": "ethtool",
                         "up": true
                     }
                 },
                 "up": true
             }
         },
         "em4": {
             "ifinfo": {
                 "dev_addr": "b8:2a:72:d5:6c:e6",
                 "dev_addr_len": 6,
                 "ifindex": 5,
                 "ifname": "em4"
             },
             "link": {
                 "duplex": "full",
                 "speed": 1000,
                 "up": true
             },
             "link_watches": {
                 "list": {
                     "link_watch_0": {
                         "delay_down": 0,
                         "delay_up": 0,
                         "name": "ethtool",
                         "up": true
                     }
                 },
                 "up": true
             }
         }
     },
     "setup": {
         "daemonized": false,
         "dbus_enabled": true,
         "debug_level": 0,
         "kernel_team_mode_name": "roundrobin",
         "pid": 3107,
         "pid_file": "/var/run/teamd/team1.pid",
         "runner_name": "roundrobin",
         "zmq_enabled": false
     },
     "team_device": {
         "ifinfo": {
             "dev_addr": "b8:2a:72:d5:6c:e6",
             "dev_addr_len": 6,
             "ifindex": 7,
             "ifname": "team1"
         }
     }
}

media2 ~]# teamdctl team1 s d
{
     "ports": {
         "em3": {
             "ifinfo": {
                 "dev_addr": "b8:2a:72:d5:97:18",
                 "dev_addr_len": 6,
                 "ifindex": 4,
                 "ifname": "em3"
             },
             "link": {
                 "duplex": "full",
                 "speed": 1000,
                 "up": true
             },
             "link_watches": {
                 "list": {
                     "link_watch_0": {
                         "delay_down": 0,
                         "delay_up": 0,
                         "name": "ethtool",
                         "up": true
                     }
                 },
                 "up": true
             }
         },
         "em4": {
             "ifinfo": {
                 "dev_addr": "b8:2a:72:d5:97:18",
                 "dev_addr_len": 6,
                 "ifindex": 5,
                 "ifname": "em4"
             },
             "link": {
                 "duplex": "full",
                 "speed": 1000,
                 "up": true
             },
             "link_watches": {
                 "list": {
                     "link_watch_0": {
                         "delay_down": 0,
                         "delay_up": 0,
                         "name": "ethtool",
                         "up": true
                     }
                 },
                 "up": true
             }
         }
     },
     "setup": {
         "daemonized": false,
         "dbus_enabled": true,
         "debug_level": 0,
         "kernel_team_mode_name": "roundrobin",
         "pid": 3084,
         "pid_file": "/var/run/teamd/team1.pid",
         "runner_name": "roundrobin",
         "zmq_enabled": false
     },
     "team_device": {
         "ifinfo": {
             "dev_addr": "b8:2a:72:d5:97:18",
             "dev_addr_len": 6,
             "ifindex": 7,
             "ifname": "team1"
         }
     }
}
> Also you can try random runner instead of roundrobin. The resulting
> behaviour is almost the same. Maybe the issue won't happen there.

I do believe that if I "restart" the roundrobin runner on media1, the 
problem is gone anyway. So for the moment I'll keep roundrobin just to 
keep the failure environment in place in case you need some more data.
> Also, can you please post me output of:
> ethtool -i em3

media1 ~]# ethtool -i em3
driver: tg3
version: 3.136
firmware-version: FFV7.8.16 bc 5720-v1.32
bus-info: 0000:02:00.0
supports-statistics: yes
supports-test: yes
supports-eeprom-access: yes
supports-register-dump: yes
supports-priv-flags: no

media2 ~]# ethtool -i em3
driver: tg3
version: 3.136
firmware-version: FFV7.8.16 bc 5720-v1.32
bus-info: 0000:02:00.0
supports-statistics: yes
supports-test: yes
supports-eeprom-access: yes
supports-register-dump: yes
supports-priv-flags: no
> ethtool em3

media1 ~]# ethtool em3
Settings for em3:
         Supported ports: [ TP ]
         Supported link modes:   10baseT/Half 10baseT/Full
                                 100baseT/Half 100baseT/Full
                                 1000baseT/Half 1000baseT/Full
         Supported pause frame use: No
         Supports auto-negotiation: Yes
         Advertised link modes:  10baseT/Half 10baseT/Full
                                 100baseT/Half 100baseT/Full
                                 1000baseT/Half 1000baseT/Full
         Advertised pause frame use: Symmetric
         Advertised auto-negotiation: Yes
         Link partner advertised link modes:  10baseT/Half 10baseT/Full
                                              100baseT/Half 100baseT/Full
                                              1000baseT/Half 1000baseT/Full
         Link partner advertised pause frame use: Symmetric
         Link partner advertised auto-negotiation: Yes
         Speed: 1000Mb/s
         Duplex: Full
         Port: Twisted Pair
         PHYAD: 1
         Transceiver: internal
         Auto-negotiation: on
         MDI-X: off
         Supports Wake-on: g
         Wake-on: d
         Current message level: 0x000000ff (255)
                                drv probe link timer ifdown ifup rx_err 
tx_err
         Link detected: yes

media2 ~]# ethtool em3
Settings for em3:
         Supported ports: [ TP ]
         Supported link modes:   10baseT/Half 10baseT/Full
                                 100baseT/Half 100baseT/Full
                                 1000baseT/Half 1000baseT/Full
         Supported pause frame use: No
         Supports auto-negotiation: Yes
         Advertised link modes:  10baseT/Half 10baseT/Full
                                 100baseT/Half 100baseT/Full
                                 1000baseT/Half 1000baseT/Full
         Advertised pause frame use: Symmetric
         Advertised auto-negotiation: Yes
         Link partner advertised link modes:  10baseT/Half 10baseT/Full
                                              100baseT/Half 100baseT/Full
                                              1000baseT/Half 1000baseT/Full
         Link partner advertised pause frame use: Symmetric
         Link partner advertised auto-negotiation: Yes
         Speed: 1000Mb/s
         Duplex: Full
         Port: Twisted Pair
         PHYAD: 1
         Transceiver: internal
         Auto-negotiation: on
         MDI-X: on
         Supports Wake-on: g
         Wake-on: d
         Current message level: 0x000000ff (255)
                                drv probe link timer ifdown ifup rx_err 
tx_err
         Link detected: yes
> ethtool -S em3

media1 ~]# ethtool -S em3
NIC statistics:
      rx_octets: 26853394715
      rx_fragments: 0
      rx_ucast_packets: 227879267
      rx_mcast_packets: 84
      rx_bcast_packets: 6
      rx_fcs_errors: 0
      rx_align_errors: 0
      rx_xon_pause_rcvd: 0
      rx_xoff_pause_rcvd: 0
      rx_mac_ctrl_rcvd: 0
      rx_xoff_entered: 0
      rx_frame_too_long_errors: 0
      rx_jabbers: 0
      rx_undersize_packets: 0
      rx_in_length_errors: 0
      rx_out_length_errors: 0
      rx_64_or_less_octet_packets: 0
      rx_65_to_127_octet_packets: 0
      rx_128_to_255_octet_packets: 0
      rx_256_to_511_octet_packets: 0
      rx_512_to_1023_octet_packets: 0
      rx_1024_to_1522_octet_packets: 0
      rx_1523_to_2047_octet_packets: 0
      rx_2048_to_4095_octet_packets: 0
      rx_4096_to_8191_octet_packets: 0
      rx_8192_to_9022_octet_packets: 0
      tx_octets: 1179249531811
      tx_collisions: 0
      tx_xon_sent: 0
      tx_xoff_sent: 0
      tx_flow_control: 0
      tx_mac_errors: 0
      tx_single_collisions: 0
      tx_mult_collisions: 0
      tx_deferred: 0
      tx_excessive_collisions: 0
      tx_late_collisions: 0
      tx_collide_2times: 0
      tx_collide_3times: 0
      tx_collide_4times: 0
      tx_collide_5times: 0
      tx_collide_6times: 0
      tx_collide_7times: 0
      tx_collide_8times: 0
      tx_collide_9times: 0
      tx_collide_10times: 0
      tx_collide_11times: 0
      tx_collide_12times: 0
      tx_collide_13times: 0
      tx_collide_14times: 0
      tx_collide_15times: 0
      tx_ucast_packets: 254815741
      tx_mcast_packets: 8
      tx_bcast_packets: 612
      tx_carrier_sense_errors: 0
      tx_discards: 0
      tx_errors: 0
      dma_writeq_full: 0
      dma_write_prioq_full: 0
      rxbds_empty: 0
      rx_discards: 0
      rx_errors: 0
      rx_threshold_hit: 0
      dma_readq_full: 0
      dma_read_prioq_full: 0
      tx_comp_queue_full: 0
      ring_set_send_prod_index: 0
      ring_status_update: 0
      nic_irqs: 0
      nic_avoided_irqs: 0
      nic_tx_threshold_hit: 0
      mbuf_lwm_thresh_hit: 0

media2 ~]# ethtool -S em3
NIC statistics:
      rx_octets: 782620145025
      rx_fragments: 0
      rx_ucast_packets: 173893766
      rx_mcast_packets: 0
      rx_bcast_packets: 0
      rx_fcs_errors: 0
      rx_align_errors: 0
      rx_xon_pause_rcvd: 0
      rx_xoff_pause_rcvd: 0
      rx_mac_ctrl_rcvd: 0
      rx_xoff_entered: 0
      rx_frame_too_long_errors: 0
      rx_jabbers: 0
      rx_undersize_packets: 0
      rx_in_length_errors: 0
      rx_out_length_errors: 0
      rx_64_or_less_octet_packets: 0
      rx_65_to_127_octet_packets: 0
      rx_128_to_255_octet_packets: 0
      rx_256_to_511_octet_packets: 0
      rx_512_to_1023_octet_packets: 0
      rx_1024_to_1522_octet_packets: 0
      rx_1523_to_2047_octet_packets: 0
      rx_2048_to_4095_octet_packets: 0
      rx_4096_to_8191_octet_packets: 0
      rx_8192_to_9022_octet_packets: 0
      tx_octets: 17210360858
      tx_collisions: 0
      tx_xon_sent: 0
      tx_xoff_sent: 0
      tx_flow_control: 0
      tx_mac_errors: 0
      tx_single_collisions: 0
      tx_mult_collisions: 0
      tx_deferred: 0
      tx_excessive_collisions: 0
      tx_late_collisions: 0
      tx_collide_2times: 0
      tx_collide_3times: 0
      tx_collide_4times: 0
      tx_collide_5times: 0
      tx_collide_6times: 0
      tx_collide_7times: 0
      tx_collide_8times: 0
      tx_collide_9times: 0
      tx_collide_10times: 0
      tx_collide_11times: 0
      tx_collide_12times: 0
      tx_collide_13times: 0
      tx_collide_14times: 0
      tx_collide_15times: 0
      tx_ucast_packets: 147864937
      tx_mcast_packets: 23
      tx_bcast_packets: 2
      tx_carrier_sense_errors: 0
      tx_discards: 0
      tx_errors: 0
      dma_writeq_full: 0
      dma_write_prioq_full: 0
      rxbds_empty: 0
      rx_discards: 0
      rx_errors: 0
      rx_threshold_hit: 0
      dma_readq_full: 0
      dma_read_prioq_full: 0
      tx_comp_queue_full: 0
      ring_set_send_prod_index: 0
      ring_status_update: 0
      nic_irqs: 0
      nic_avoided_irqs: 0
      nic_tx_threshold_hit: 0
      mbuf_lwm_thresh_hit: 0
> ethtool -i em4

media1 ~]# ethtool -i em4
driver: tg3
version: 3.136
firmware-version: FFV7.8.16 bc 5720-v1.32
bus-info: 0000:02:00.1
supports-statistics: yes
supports-test: yes
supports-eeprom-access: yes
supports-register-dump: yes
supports-priv-flags: no

media2 ~]# ethtool -i em4
driver: tg3
version: 3.136
firmware-version: FFV7.8.16 bc 5720-v1.32
bus-info: 0000:02:00.1
supports-statistics: yes
supports-test: yes
supports-eeprom-access: yes
supports-register-dump: yes
supports-priv-flags: no
> ethtool em4

media1 ~]# ethtool em4
Settings for em4:
         Supported ports: [ TP ]
         Supported link modes:   10baseT/Half 10baseT/Full
                                 100baseT/Half 100baseT/Full
                                 1000baseT/Half 1000baseT/Full
         Supported pause frame use: No
         Supports auto-negotiation: Yes
         Advertised link modes:  10baseT/Half 10baseT/Full
                                 100baseT/Half 100baseT/Full
                                 1000baseT/Half 1000baseT/Full
         Advertised pause frame use: Symmetric
         Advertised auto-negotiation: Yes
         Link partner advertised link modes:  10baseT/Half 10baseT/Full
                                              100baseT/Half 100baseT/Full
                                              1000baseT/Half 1000baseT/Full
         Link partner advertised pause frame use: Symmetric
         Link partner advertised auto-negotiation: Yes
         Speed: 1000Mb/s
         Duplex: Full
         Port: Twisted Pair
         PHYAD: 2
         Transceiver: internal
         Auto-negotiation: on
         MDI-X: off
         Supports Wake-on: g
         Wake-on: d
         Current message level: 0x000000ff (255)
                                drv probe link timer ifdown ifup rx_err 
tx_err
         Link detected: yes

media2 ~]# ethtool em4
Settings for em4:
         Supported ports: [ TP ]
         Supported link modes:   10baseT/Half 10baseT/Full
                                 100baseT/Half 100baseT/Full
                                 1000baseT/Half 1000baseT/Full
         Supported pause frame use: No
         Supports auto-negotiation: Yes
         Advertised link modes:  10baseT/Half 10baseT/Full
                                 100baseT/Half 100baseT/Full
                                 1000baseT/Half 1000baseT/Full
         Advertised pause frame use: Symmetric
         Advertised auto-negotiation: Yes
         Link partner advertised link modes:  10baseT/Half 10baseT/Full
                                              100baseT/Half 100baseT/Full
                                              1000baseT/Half 1000baseT/Full
         Link partner advertised pause frame use: Symmetric
         Link partner advertised auto-negotiation: Yes
         Speed: 1000Mb/s
         Duplex: Full
         Port: Twisted Pair
         PHYAD: 2
         Transceiver: internal
         Auto-negotiation: on
         MDI-X: on
         Supports Wake-on: g
         Wake-on: d
         Current message level: 0x000000ff (255)
                                drv probe link timer ifdown ifup rx_err 
tx_err
         Link detected: yes
> ethtool -S em4

media1 ~]# ethtool -S em4
NIC statistics:
      rx_octets: 246361
      rx_fragments: 0
      rx_ucast_packets: 912
      rx_mcast_packets: 8
      rx_bcast_packets: 0
      rx_fcs_errors: 0
      rx_align_errors: 0
      rx_xon_pause_rcvd: 0
      rx_xoff_pause_rcvd: 0
      rx_mac_ctrl_rcvd: 0
      rx_xoff_entered: 0
      rx_frame_too_long_errors: 0
      rx_jabbers: 0
      rx_undersize_packets: 0
      rx_in_length_errors: 0
      rx_out_length_errors: 0
      rx_64_or_less_octet_packets: 0
      rx_65_to_127_octet_packets: 0
      rx_128_to_255_octet_packets: 0
      rx_256_to_511_octet_packets: 0
      rx_512_to_1023_octet_packets: 0
      rx_1024_to_1522_octet_packets: 0
      rx_1523_to_2047_octet_packets: 0
      rx_2048_to_4095_octet_packets: 0
      rx_4096_to_8191_octet_packets: 0
      rx_8192_to_9022_octet_packets: 0
      tx_octets: 760
      tx_collisions: 0
      tx_xon_sent: 0
      tx_xoff_sent: 0
      tx_flow_control: 0
      tx_mac_errors: 0
      tx_single_collisions: 0
      tx_mult_collisions: 0
      tx_deferred: 0
      tx_excessive_collisions: 0
      tx_late_collisions: 0
      tx_collide_2times: 0
      tx_collide_3times: 0
      tx_collide_4times: 0
      tx_collide_5times: 0
      tx_collide_6times: 0
      tx_collide_7times: 0
      tx_collide_8times: 0
      tx_collide_9times: 0
      tx_collide_10times: 0
      tx_collide_11times: 0
      tx_collide_12times: 0
      tx_collide_13times: 0
      tx_collide_14times: 0
      tx_collide_15times: 0
      tx_ucast_packets: 0
      tx_mcast_packets: 8
      tx_bcast_packets: 0
      tx_carrier_sense_errors: 0
      tx_discards: 0
      tx_errors: 0
      dma_writeq_full: 0
      dma_write_prioq_full: 0
      rxbds_empty: 0
      rx_discards: 0
      rx_errors: 0
      rx_threshold_hit: 0
      dma_readq_full: 0
      dma_read_prioq_full: 0
      tx_comp_queue_full: 0
      ring_set_send_prod_index: 0
      ring_status_update: 0
      nic_irqs: 0
      nic_avoided_irqs: 0
      nic_tx_threshold_hit: 0
      mbuf_lwm_thresh_hit: 0

media2 ~]# ethtool -S em4
NIC statistics:
      rx_octets: 760
      rx_fragments: 0
      rx_ucast_packets: 0
      rx_mcast_packets: 8
      rx_bcast_packets: 0
      rx_fcs_errors: 0
      rx_align_errors: 0
      rx_xon_pause_rcvd: 0
      rx_xoff_pause_rcvd: 0
      rx_mac_ctrl_rcvd: 0
      rx_xoff_entered: 0
      rx_frame_too_long_errors: 0
      rx_jabbers: 0
      rx_undersize_packets: 0
      rx_in_length_errors: 0
      rx_out_length_errors: 0
      rx_64_or_less_octet_packets: 0
      rx_65_to_127_octet_packets: 0
      rx_128_to_255_octet_packets: 0
      rx_256_to_511_octet_packets: 0
      rx_512_to_1023_octet_packets: 0
      rx_1024_to_1522_octet_packets: 0
      rx_1523_to_2047_octet_packets: 0
      rx_2048_to_4095_octet_packets: 0
      rx_4096_to_8191_octet_packets: 0
      rx_8192_to_9022_octet_packets: 0
      tx_octets: 247887
      tx_collisions: 0
      tx_xon_sent: 0
      tx_xoff_sent: 0
      tx_flow_control: 0
      tx_mac_errors: 0
      tx_single_collisions: 0
      tx_mult_collisions: 0
      tx_deferred: 0
      tx_excessive_collisions: 0
      tx_late_collisions: 0
      tx_collide_2times: 0
      tx_collide_3times: 0
      tx_collide_4times: 0
      tx_collide_5times: 0
      tx_collide_6times: 0
      tx_collide_7times: 0
      tx_collide_8times: 0
      tx_collide_9times: 0
      tx_collide_10times: 0
      tx_collide_11times: 0
      tx_collide_12times: 0
      tx_collide_13times: 0
      tx_collide_14times: 0
      tx_collide_15times: 0
      tx_ucast_packets: 927
      tx_mcast_packets: 8
      tx_bcast_packets: 0
      tx_carrier_sense_errors: 0
      tx_discards: 0
      tx_errors: 0
      dma_writeq_full: 0
      dma_write_prioq_full: 0
      rxbds_empty: 0
      rx_discards: 0
      rx_errors: 0
      rx_threshold_hit: 0
      dma_readq_full: 0
      dma_read_prioq_full: 0
      tx_comp_queue_full: 0
      ring_set_send_prod_index: 0
      ring_status_update: 0
      nic_irqs: 0
      nic_avoided_irqs: 0
      nic_tx_threshold_hit: 0
      mbuf_lwm_thresh_hit: 0
> Of course in time the problem happens.
> > Kind regards
> > Ingo
> >
> > This is the used network config of team1:
> > media1:
> > cat /etc/sysconfig/network-scripts/ifcfg-team1_slave_0
> > # Generated by parse-kickstart
> > NAME=team1 slave 0
> > TEAM_MASTER=team1
> > DEVICETYPE=TeamPort
> > DEVICE=em3
> > ONBOOT=yes
> > UUID=b7026c5b-e9cd-457c-93d2-a2799361ed90
> >
> > cat /etc/sysconfig/network-scripts/ifcfg-team1_slave_1
> > # Generated by parse-kickstart
> > NAME=team1 slave 1
> > TEAM_MASTER=team1
> > DEVICETYPE=TeamPort
> > DEVICE=em4
> > ONBOOT=yes
> > UUID=fbe5bc56-95f5-4f9e-a5ea-ab6b6f5e5b50
> >
> > cat /etc/sysconfig/network-scripts/ifcfg-team1
> > # Generated by parse-kickstart
> > UUID=4e0090cd-32d3-4c5b-be23-9f53083da4dd
> > NAME="Team connection team1"
> > TEAM_CONFIG="{"runner": {"name": "roundrobin"}}"
> > GATEWAY=
> > IPV6_AUTOCONF=yes
> > BOOTPROTO=none
> > DEVICE=team1
> > MTU=9000
> > TYPE=Team
> > ONBOOT=yes
> > IPV6INIT=yes
> > DEVICETYPE=Team
> > IPADDR0=192.168.101.31
> > PREFIX0=24
> > DEFROUTE=yes
> > IPV4_FAILURE_FATAL=no
> > IPV6_DEFROUTE=yes
> > IPV6_PEERDNS=yes
> > IPV6_PEERROUTES=yes
> > IPV6_FAILURE_FATAL=no
> >
> > media2:
> > cat /etc/sysconfig/network-scripts/ifcfg-team1_slave_0
> > # Generated by parse-kickstart
> > NAME=team1 slave 0
> > TEAM_MASTER=team1
> > DEVICETYPE=TeamPort
> > DEVICE=em3
> > ONBOOT=yes
> > UUID=bfd8e77e-7eb8-47ea-ab7e-95286e073202
> >
> > cat /etc/sysconfig/network-scripts/ifcfg-team1_slave_1
> > # Generated by parse-kickstart
> > NAME=team1 slave 1
> > TEAM_MASTER=team1
> > DEVICETYPE=TeamPort
> > DEVICE=em4
> > ONBOOT=yes
> > UUID=93c5dfd8-3b81-4d1e-8ba9-c46f8daceb1a
> >
> > cat /etc/sysconfig/network-scripts/ifcfg-team1
> > # Generated by parse-kickstart
> > UUID=79c843db-996c-41e4-9ee5-2a8f3da244ed
> > NAME="Team connection team1"
> > TEAM_CONFIG="{"runner": {"name": "roundrobin"}}"
> > GATEWAY=
> > IPV6_AUTOCONF=yes
> > BOOTPROTO=none
> > DEVICE=team1
> > MTU=9000
> > TYPE=Team
> > ONBOOT=yes
> > IPV6INIT=yes
> > DEVICETYPE=Team
> > IPADDR0=192.168.101.32
> > PREFIX0=24
> > DEFROUTE=yes
> > IPV4_FAILURE_FATAL=no
> > IPV6_DEFROUTE=yes
> > IPV6_PEERDNS=yes
> > IPV6_PEERROUTES=yes
> > IPV6_FAILURE_FATAL=no
> > Here are some system logs from both machines:
> > media1:
> >
> > Nov  6 02:31:27 media1 corosync[3657]: [TOTEM ] Incrementing problem counter
> > for seqid 412041 iface 192.168.100.31 to [1 of 10]
> > Nov  6 02:31:29 media1 corosync[3657]: [TOTEM ] ring 0 active with no faults
> > Nov  6 02:31:29 media1 corosync[3657]: [TOTEM ] Incrementing problem counter
> > for seqid 412047 iface 192.168.100.31 to [1 of 10]
> > Nov  6 02:31:31 media1 corosync[3657]: [TOTEM ] ring 0 active with no faults
> > Nov  6 02:32:51 media1 corosync[3657]: [TOTEM ] Incrementing problem counter
> > for seqid 412315 iface 192.168.101.31 to [1 of 10]
> > Nov  6 02:32:53 media1 corosync[3657]: [TOTEM ] ring 1 active with no faults
> > Nov  6 02:32:53 media1 corosync[3657]: [TOTEM ] Incrementing problem counter
> > for seqid 412319 iface 192.168.101.31 to [1 of 10]
> > Nov  6 02:32:54 media1 corosync[3657]: [TOTEM ] Incrementing problem counter
> > for seqid 412321 iface 192.168.101.31 to [2 of 10]
> > Nov  6 02:32:55 media1 corosync[3657]: [TOTEM ] Decrementing problem counter
> > for iface 192.168.101.31 to [1 of 10]
> > Nov  6 02:32:56 media1 corosync[3657]: [TOTEM ] Incrementing problem counter
> > for seqid 412325 iface 192.168.101.31 to [2 of 10]
> > Nov  6 02:32:57 media1 corosync[3657]: [TOTEM ] Decrementing problem counter
> > for iface 192.168.101.31 to [1 of 10]
> > Nov  6 02:32:58 media1 corosync[3657]: [TOTEM ] Incrementing problem counter
> > for seqid 412327 iface 192.168.101.31 to [2 of 10]
> > Nov  6 02:32:59 media1 kernel: drbd r0: PingAck did not arrive in time.
> > Nov  6 02:32:59 media1 kernel: drbd r0: peer( Secondary -> Unknown ) conn(
> > Connected -> NetworkFailure ) pdsk( UpToDate -> DUnknown )
> > Nov  6 02:32:59 media1 kernel: block drbd0: new current UUID
> > 0C0308FA4F94F811:59E43E5FD7B26A4F:1963884285B385E6:1962884285B385E6
> > Nov  6 02:32:59 media1 kernel: drbd r0: asender terminated
> > Nov  6 02:32:59 media1 kernel: drbd r0: Terminating drbd_a_r0
> > Nov  6 02:32:59 media1 kernel: drbd r0: Connection closed
> > Nov  6 02:32:59 media1 kernel: drbd r0: conn( NetworkFailure -> Unconnected )
> > Nov  6 02:32:59 media1 kernel: drbd r0: receiver terminated
> > Nov  6 02:32:59 media1 kernel: drbd r0: Restarting receiver thread
> > Nov  6 02:32:59 media1 kernel: drbd r0: receiver (re)started
> > Nov  6 02:32:59 media1 kernel: drbd r0: conn( Unconnected -> WFConnection )
> > Nov  6 02:32:59 media1 corosync[3657]: [TOTEM ] Decrementing problem counter
> > for iface 192.168.101.31 to [1 of 10]
> > Nov  6 02:32:59 media1 corosync[3657]: [TOTEM ] Incrementing problem counter
> > for seqid 412329 iface 192.168.101.31 to [2 of 10]
> > Nov  6 02:33:01 media1 corosync[3657]: [TOTEM ] Incrementing problem counter
> > for seqid 412333 iface 192.168.101.31 to [3 of 10]
> > Nov  6 02:33:01 media1 corosync[3657]: [TOTEM ] Decrementing problem counter
> > for iface 192.168.101.31 to [2 of 10]
> > Nov  6 02:33:01 media1 crmd[4960]: notice: do_state_transition: State
> > transition S_IDLE -> S_POLICY_ENGINE [ input=I_PE_CALC cause=C_FSA_INTERNAL
> > origin=abort_transition_graph ]
> >
> > Nov  6 02:33:01 media1 pengine[4959]: notice: unpack_config: On loss of CCM
> > Quorum: Ignore
> >
> > Nov  6 02:33:01 media1 pengine[4959]: warning: unpack_rsc_op: Processing
> > failed op monitor for res_nfsserver_mediafiles on media1: unknown error (1)
> > Nov  6 02:33:01 media1 pengine[4959]: warning: unpack_rsc_op: Processing
> > failed op monitor for res_nfsserver_mediafiles on media2: unknown error (1)
> > Nov  6 02:33:01 media1 pengine[4959]: notice: process_pe_message: Calculated
> > Transition 144: /var/lib/pacemaker/pengine/pe-input-227.bz2
> >
> > Nov  6 02:33:01 media1 crmd[4960]: notice: run_graph: Transition 144
> > (Complete=0, Pending=0, Fired=0, Skipped=0, Incomplete=0,
> > Source=/var/lib/pacemaker/pengine/pe-input-227.bz2): Complete
> >
> > Nov  6 02:33:01 media1 crmd[4960]: notice: do_state_transition: State
> > transition S_TRANSITION_ENGINE -> S_IDLE [ input=I_TE_SUCCESS
> > cause=C_FSA_INTERNAL origin=notify_crmd ]
> > Nov  6 02:33:01 media1 kernel: drbd r0: Handshake successful: Agreed network
> > protocol version 101
> >
> > Nov  6 02:33:01 media1 kernel: drbd r0: Agreed to support TRIM on protocol
> > level
> >
> > Nov  6 02:33:01 media1 kernel: drbd r0: Peer authenticated using 20 bytes
> > HMAC
> >
> > Nov  6 02:33:01 media1 kernel: drbd r0: conn( WFConnection -> WFReportParams
> > )
> >
> > Nov  6 02:33:01 media1 kernel: drbd r0: Starting asender thread (from
> > drbd_r_r0 [5999])
> >
> > Nov  6 02:33:02 media1 corosync[3657]: [TOTEM ] Incrementing problem counter
> > for seqid 412337 iface 192.168.101.31 to [3 of 10]
> >
> > Nov  6 02:33:02 media1 kernel: block drbd0: drbd_sync_handshake:
> >
> > Nov  6 02:33:02 media1 kernel: block drbd0: self
> > 0C0308FA4F94F811:59E43E5FD7B26A4F:1963884285B385E6:1962884285B385E6
> > bits:192101 flags:0
> > Nov  6 02:33:02 media1 kernel: block drbd0: peer
> > 59E43E5FD7B26A4E:0000000000000000:1963884285B385E6:1962884285B385E6 bits:0
> > flags:0
> > Nov  6 02:33:02 media1 kernel: block drbd0: uuid_compare()=1 by rule 70
> >
> > Nov  6 02:33:02 media1 kernel: block drbd0: peer( Unknown -> Secondary )
> > conn( WFReportParams -> WFBitMapS ) pdsk( DUnknown -> Consistent )
> >
> > Nov  6 02:33:02 media1 kernel: block drbd0: send bitmap stats
> > [Bytes(packets)]: plain 0(0), RLE 185(1), total 185; compression: 100.0%
> >
> > Nov  6 02:33:02 media1 corosync[3657]: [TOTEM ] Incrementing problem counter
> > for seqid 412339 iface 192.168.101.31 to [4 of 10]
> >
> > Nov  6 02:33:03 media1 kernel: block drbd0: receive bitmap stats
> > [Bytes(packets)]: plain 0(0), RLE 185(1), total 185; compression: 100.0%
> >
> > Nov  6 02:33:03 media1 kernel: block drbd0: helper command: /sbin/drbdadm
> > before-resync-source minor-0
> >
> > Nov  6 02:33:03 media1 kernel: block drbd0: helper command: /sbin/drbdadm
> > before-resync-source minor-0 exit code 0 (0x0)
> >
> > Nov  6 02:33:03 media1 kernel: block drbd0: conn( WFBitMapS -> SyncSource )
> > pdsk( Consistent -> Inconsistent )
> >
> > Nov  6 02:33:03 media1 kernel: block drbd0: Began resync as SyncSource (will
> > sync 919244 KB [229811 bits set]).
> >
> > Nov  6 02:33:03 media1 kernel: block drbd0: updated sync UUID
> > 0C0308FA4F94F811:59E53E5FD7B26A4F:59E43E5FD7B26A4F:1963884285B385E6
> >
> > Nov  6 02:33:03 media1 corosync[3657]: [TOTEM ] Decrementing problem counter
> > for iface 192.168.101.31 to [3 of 10]
> >
> > Nov  6 02:33:03 media1 corosync[3657]: [TOTEM ] Incrementing problem counter
> > for seqid 412343 iface 192.168.101.31 to [4 of 10]
> >
> > Nov  6 02:33:04 media1 corosync[3657]: [TOTEM ] Incrementing problem counter
> > for seqid 412345 iface 192.168.101.31 to [5 of 10]
> >
> > Nov  6 02:33:05 media1 corosync[3657]: [TOTEM ] Incrementing problem counter
> > for seqid 412347 iface 192.168.101.31 to [6 of 10]
> >
> > Nov  6 02:33:05 media1 corosync[3657]: [TOTEM ] Decrementing problem counter
> > for iface 192.168.101.31 to [5 of 10]
> >
> > Nov  6 02:33:05 media1 corosync[3657]: [TOTEM ] Incrementing problem counter
> > for seqid 412349 iface 192.168.101.31 to [6 of 10]
> >
> > Nov  6 02:33:06 media1 corosync[3657]: [TOTEM ] Incrementing problem counter
> > for seqid 412351 iface 192.168.101.31 to [7 of 10]
> >
> > Nov  6 02:33:07 media1 corosync[3657]: [TOTEM ] Incrementing problem counter
> > for seqid 412355 iface 192.168.101.31 to [8 of 10]
> >
> > Nov  6 02:33:07 media1 corosync[3657]: [TOTEM ] Decrementing problem counter
> > for iface 192.168.101.31 to [7 of 10]
> >
> > Nov  6 02:33:07 media1 corosync[3657]: [TOTEM ] Incrementing problem counter
> > for seqid 412357 iface 192.168.101.31 to [8 of 10]
> >
> > Nov  6 02:33:08 media1 corosync[3657]: [TOTEM ] Incrementing problem counter
> > for seqid 412361 iface 192.168.101.31 to [9 of 10]
> >
> > Nov  6 02:33:09 media1 corosync[3657]: [TOTEM ] Incrementing problem counter
> > for seqid 412363 iface 192.168.101.31 to [10 of 10]
> >
> > Nov  6 02:33:09 media1 corosync[3657]: [TOTEM ] Marking seqid 412363 ringid 1
> > interface 192.168.101.31 FAULTY
> >
> > Nov  6 02:33:11 media1 crmd[4960]: notice: do_state_transition: State
> > transition S_IDLE -> S_POLICY_ENGINE [ input=I_PE_CALC cause=C_FSA_INTERNAL
> > origin=abort_transition_graph ]
> >
> > Nov  6 02:33:11 media1 pengine[4959]: notice: unpack_config: On loss of CCM
> > Quorum: Ignore
> >
> > Nov  6 02:33:11 media1 pengine[4959]: warning: unpack_rsc_op: Processing
> > failed op monitor for res_nfsserver_mediafiles on media1: unknown error (1)
> > Nov  6 02:33:11 media1 pengine[4959]: warning: unpack_rsc_op: Processing
> > failed op monitor for res_nfsserver_mediafiles on media2: unknown error (1)
> > Nov  6 02:33:11 media1 pengine[4959]: notice: process_pe_message: Calculated
> > Transition 145: /var/lib/pacemaker/pengine/pe-input-228.bz2
> >
> > Nov  6 02:33:11 media1 crmd[4960]: notice: run_graph: Transition 145
> > (Complete=0, Pending=0, Fired=0, Skipped=0, Incomplete=0,
> > Source=/var/lib/pacemaker/pengine/pe-input-228.bz2): Complete
> > Nov  6 02:33:11 media1 crmd[4960]: notice: do_state_transition: State
> > transition S_TRANSITION_ENGINE -> S_IDLE [ input=I_TE_SUCCESS
> > cause=C_FSA_INTERNAL origin=notify_crmd ]
> > Nov  6 02:33:17 media1 corosync[3657]: [TOTEM ] Automatically recovered ring
> > 1
> >
> > Nov  6 02:33:18 media1 corosync[3657]: [TOTEM ] Incrementing problem counter
> > for seqid 412387 iface 192.168.101.31 to [1 of 10]
> >
> > Nov  6 02:33:20 media1 corosync[3657]: [TOTEM ] Incrementing problem counter
> > for seqid 412389 iface 192.168.101.31 to [2 of 10]
> >
> > Nov  6 02:33:20 media1 corosync[3657]: [TOTEM ] Incrementing problem counter
> > for seqid 412391 iface 192.168.101.31 to [3 of 10]
> >
> > Nov  6 02:33:21 media1 corosync[3657]: [TOTEM ] Incrementing problem counter
> > for seqid 412393 iface 192.168.101.31 to [4 of 10]
> >
> > Nov  6 02:33:22 media1 corosync[3657]: [TOTEM ] Incrementing problem counter
> > for seqid 412397 iface 192.168.101.31 to [5 of 10]
> >
> > Nov  6 02:33:22 media1 corosync[3657]: [TOTEM ] Incrementing problem counter
> > for seqid 412399 iface 192.168.101.31 to [6 of 10]
> >
> > Nov  6 02:33:24 media1 corosync[3657]: [TOTEM ] Incrementing problem counter
> > for seqid 412403 iface 192.168.101.31 to [7 of 10]
> >
> > Nov  6 02:33:31 media1 corosync[3657]: [TOTEM ] Incrementing problem counter
> > for seqid 412421 iface 192.168.101.31 to [8 of 10]
> >
> > Nov  6 02:33:32 media1 corosync[3657]: [TOTEM ] Incrementing problem counter
> > for seqid 412423 iface 192.168.101.31 to [9 of 10]
> >
> > Nov  6 02:33:34 media1 corosync[3657]: [TOTEM ] Incrementing problem counter
> > for seqid 412429 iface 192.168.101.31 to [10 of 10]
> >
> > Nov  6 02:33:34 media1 corosync[3657]: [TOTEM ] Marking seqid 412429 ringid 1
> > interface 192.168.101.31 FAULTY
> >
> > Nov  6 02:33:36 media1 corosync[3657]: [TOTEM ] Automatically recovered ring
> > 1
> >
> > Nov  6 02:33:39 media1 corosync[3657]: [TOTEM ] Incrementing problem counter
> > for seqid 412437 iface 192.168.101.31 to [1 of 10]
> >
> > ============================================================
> > media2:
> > ============================================================
> > Nov  6 02:32:59 media2 kernel: drbd r0: sock was shut down by peer
> >
> > Nov  6 02:32:59 media2 kernel: drbd r0: peer( Primary -> Unknown ) conn(
> > Connected -> BrokenPipe ) pdsk( UpToDate -> DUnknown )
> >
> > Nov  6 02:32:59 media2 kernel: drbd r0: short read (expected size 16)
> >
> > Nov  6 02:32:59 media2 kernel: drbd r0: asender terminated
> >
> > Nov  6 02:32:59 media2 kernel: drbd r0: Terminating drbd_a_r0
> >
> > Nov  6 02:32:59 media2 kernel: drbd r0: Connection closed
> >
> > Nov  6 02:32:59 media2 kernel: drbd r0: conn( BrokenPipe -> Unconnected )
> >
> > Nov  6 02:32:59 media2 kernel: drbd r0: receiver terminated
> >
> > Nov  6 02:32:59 media2 kernel: drbd r0: Restarting receiver thread
> >
> > Nov  6 02:32:59 media2 kernel: drbd r0: receiver (re)started
> >
> > Nov  6 02:32:59 media2 kernel: drbd r0: conn( Unconnected -> WFConnection )
> >
> > Nov  6 02:33:01 media2 attrd[4940]: notice: attrd_trigger_update: Sending
> > flush op to all hosts for: master-res_drbd_mediafiles (1000)
> >
> > Nov  6 02:33:01 media2 attrd[4940]: notice: attrd_perform_update: Sent update
> > 16: master-res_drbd_mediafiles=1000
> >
> > Nov  6 02:33:01 media2 kernel: drbd r0: Handshake successful: Agreed network
> > protocol version 101
> >
> > Nov  6 02:33:01 media2 kernel: drbd r0: Agreed to support TRIM on protocol
> > level
> >
> > Nov  6 02:33:01 media2 kernel: drbd r0: Peer authenticated using 20 bytes
> > HMAC
> >
> > Nov  6 02:33:01 media2 kernel: drbd r0: conn( WFConnection -> WFReportParams
> > )
> >
> > Nov  6 02:33:01 media2 kernel: drbd r0: Starting asender thread (from
> > drbd_r_r0 [6367])
> >
> > Nov  6 02:33:01 media2 kernel: block drbd0: drbd_sync_handshake:
> >
> > Nov  6 02:33:01 media2 kernel: block drbd0: self
> > 59E43E5FD7B26A4E:0000000000000000:1963884285B385E6:1962884285B385E6 bits:0
> > flags:0
> > Nov  6 02:33:01 media2 kernel: block drbd0: peer
> > 0C0308FA4F94F811:59E43E5FD7B26A4F:1963884285B385E6:1962884285B385E6
> > bits:192101 flags:0
> > Nov  6 02:33:01 media2 kernel: block drbd0: uuid_compare()=-1 by rule 50
> >
> > Nov  6 02:33:01 media2 kernel: block drbd0: peer( Unknown -> Primary ) conn(
> > WFReportParams -> WFBitMapT ) disk( UpToDate -> Outdated ) pdsk( DUnknown ->
> > UpToDate )
> > Nov  6 02:33:02 media2 kernel: block drbd0: receive bitmap stats
> > [Bytes(packets)]: plain 0(0), RLE 185(1), total 185; compression: 100.0%
> >
> > Nov  6 02:33:03 media2 kernel: block drbd0: send bitmap stats
> > [Bytes(packets)]: plain 0(0), RLE 185(1), total 185; compression: 100.0%
> >
> > Nov  6 02:33:03 media2 kernel: block drbd0: conn( WFBitMapT -> WFSyncUUID )
> >
> > Nov  6 02:33:03 media2 kernel: block drbd0: updated sync uuid
> > 59E53E5FD7B26A4E:0000000000000000:1963884285B385E6:1962884285B385E6
> >
> > Nov  6 02:33:03 media2 kernel: block drbd0: helper command: /sbin/drbdadm
> > before-resync-target minor-0
> >
> > Nov  6 02:33:03 media2 kernel: block drbd0: helper command: /sbin/drbdadm
> > before-resync-target minor-0 exit code 0 (0x0)
> >
> > Nov  6 02:33:03 media2 kernel: block drbd0: conn( WFSyncUUID -> SyncTarget )
> > disk( Outdated -> Inconsistent )
> >
> > Nov  6 02:33:03 media2 kernel: block drbd0: Began resync as SyncTarget (will
> > sync 919244 KB [229811 bits set]).
> >
> > Nov  6 02:33:10 media2 corosync[3690]: [TOTEM ] Incrementing problem counter
> > for seqid 412364 iface 192.168.101.32 to [1 of 10]
> >
> > Nov  6 02:33:10 media2 corosync[3690]: [TOTEM ] Automatically recovered ring
> > 1
> >
> > Nov  6 02:33:10 media2 corosync[3690]: [TOTEM ] Incrementing problem counter
> > for seqid 412366 iface 192.168.101.32 to [1 of 10]
> >
> > Nov  6 02:33:11 media2 corosync[3690]: [TOTEM ] Automatically recovered ring
> > 1
> >
> > Nov  6 02:33:11 media2 attrd[4940]: notice: attrd_trigger_update: Sending
> > flush op to all hosts for: master-res_drbd_mediafiles (10)
> >
> > Nov  6 02:33:11 media2 attrd[4940]: notice: attrd_perform_update: Sent update
> > 18: master-res_drbd_mediafiles=10
> >
> > Nov  6 02:33:11 media2 corosync[3690]: [TOTEM ] Incrementing problem counter
> > for seqid 412368 iface 192.168.101.32 to [1 of 10]
> >
> > Nov  6 02:33:12 media2 corosync[3690]: [TOTEM ] ring 1 active with no faults
> >
> > Nov  6 02:33:12 media2 corosync[3690]: [TOTEM ] Incrementing problem counter
> > for seqid 412370 iface 192.168.101.32 to [1 of 10]
> >
> > Nov  6 02:33:12 media2 corosync[3690]: [TOTEM ] Automatically recovered ring
> > 1
> >
> > Nov  6 02:33:12 media2 corosync[3690]: [TOTEM ] Incrementing problem counter
> > for seqid 412372 iface 192.168.101.32 to [1 of 10]
> >
> > Nov  6 02:33:13 media2 corosync[3690]: [TOTEM ] Incrementing problem counter
> > for seqid 412374 iface 192.168.101.32 to [2 of 10]
> >
> > Nov  6 02:33:14 media2 corosync[3690]: [TOTEM ] Decrementing problem counter
> > for iface 192.168.101.32 to [1 of 10]
> >
> > Nov  6 02:33:14 media2 corosync[3690]: [TOTEM ] Incrementing problem counter
> > for seqid 412376 iface 192.168.101.32 to [2 of 10]
> >
> > Nov  6 02:33:15 media2 corosync[3690]: [TOTEM ] Incrementing problem counter
> > for seqid 412378 iface 192.168.101.32 to [3 of 10]
> >
> > Nov  6 02:33:15 media2 corosync[3690]: [TOTEM ] Automatically recovered ring
> > 1
> >
> > Nov  6 02:33:15 media2 corosync[3690]: [TOTEM ] Incrementing problem counter
> > for seqid 412380 iface 192.168.101.32 to [1 of 10]
> >
> > Nov  6 02:33:16 media2 corosync[3690]: [TOTEM ] ring 1 active with no faults
> >
> > Nov  6 02:33:16 media2 corosync[3690]: [TOTEM ] Incrementing problem counter
> > for seqid 412382 iface 192.168.101.32 to [1 of 10]
> >
> > Nov  6 02:33:16 media2 corosync[3690]: [TOTEM ] Automatically recovered ring
> > 1
> >
> > Nov  6 02:33:17 media2 corosync[3690]: [TOTEM ] Incrementing problem counter
> > for seqid 412384 iface 192.168.101.32 to [1 of 10]
> >
> > Nov  6 02:33:17 media2 corosync[3690]: [TOTEM ] Automatically recovered ring
> > 1
> >
> > Nov  6 02:33:17 media2 corosync[3690]: [TOTEM ] Incrementing problem counter
> > for seqid 412386 iface 192.168.101.32 to [1 of 10]
> >
> > Nov  6 02:33:18 media2 corosync[3690]: [TOTEM ] ring 1 active with no faults
> >
> > Nov  6 02:33:19 media2 corosync[3690]: [TOTEM ] Incrementing problem counter
> > for seqid 412388 iface 192.168.100.32 to [1 of 10]
> >
> > Nov  6 02:33:20 media2 corosync[3690]: [TOTEM ] ring 0 active with no faults
> >
> > Nov  6 02:33:35 media2 corosync[3690]: [TOTEM ] Automatically recovered ring
> > 1
> >
> > Nov  6 02:33:36 media2 corosync[3690]: [TOTEM ] Incrementing problem counter
> > for seqid 412430 iface 192.168.101.32 to [1 of 10]
> >
> > Nov  6 02:33:36 media2 corosync[3690]: [TOTEM ] Automatically recovered ring
> > 1
> >
> > Nov  6 02:33:37 media2 corosync[3690]: [TOTEM ] Incrementing problem counter
> > for seqid 412432 iface 192.168.101.32 to [1 of 10]
> >
> > Nov  6 02:33:38 media2 corosync[3690]: [TOTEM ] ring 1 active with no faults
> >
> > libteam mailing list
> > [e-mail]
> > https://lists.fedorahosted.org/mailman/listinfo/libteam

[Jiri Pirko]

Thu, Nov 06, 2014 at 05:18:18PM CET, [e-mail] wrote:
> Am 06.11.2014 um 17:03 schrieb Jiri Pirko:
> > Thu, Nov 06, 2014 at 04:26:36PM CET, [e-mail] wrote:
> > > Hello,
> > >
> > > I am currently setting up a high available NFS server cluster with 2 nodes
> > > using pacemaker + corosync + drbd + flashcache.
> > > The two nodes (media1 and media2) have 4 1G copper nics each.
> > > I formed 2 teams:
> > > team0: LACP with em1 and em2 connected to a switch for NFS services
> > > team1: roundrobin with em3 and em4 and MTU 9000 directly connected with
> > > copper patch cables for drbd sync
> > > During my testing everything was working as expected. Then I started to
> > > tranfer about 20TB from an old server to the new cluster. The copy process
> > > was fine for about 32 hours. Then corosync and drbd started to report errors
> > > on the roundrobin team1 interface (see logs at the end of this email).
> > > The drbd sync and corosync started to flap between states for several hours
> > > on team1.
> > > I checked the error counters with ifconfig and found a huge amount off TX
> > > dropped packets on team1 of the drbd slave node media2.
> > > There were NO other error counters raising but team1 on media2.
> > > To make this a bit more clear:
> > > team1 on media2 (the drbd slave) showed raising TX dropped packets
> > > em3 and em4 on media2 (the physical interfaces of team1) did not show any
> > >
> > > raising error counters.
> > > team1 on media1 (the drbd master) did not show any raising error counters.
> > > em3 and em4 on media1 (the physical interfaces of team1) did not show any
> > >
> > > raising error counters.
> > > I then thought maybe one of the two patch cables between the two machines was
> > > not 100% ok. So I removed em4 from team1 on both machines:
> > > "teamdctl team1 port remove em4"
> > > The error counters stopped raising.
> > > I thought: "Yeah! That cable must be broken!"
> > > But just to make sure I really found the problem I re-enabled em4 on both
> > > nodes:
> > > "teamdctl team1 port add em4"
> > > Ok, now both team1 interfaces had 2 links up again and the error counter
> > > restarted to raise again.
> > > I then stopped em3 on both nodes to force all traffic through em4:
> > > "teamdctl team1 port remove em3"
> > > And guess what? The error counters on team1 stopped raising!
> > > So in short:
> > > If I remove either one of the two physical interfaces from team1 everything
> > > is working without any errors. As soon as I enable both physical interfaces
> > > the error counters start to raise.
> > > After these tests I rebooted the slave node (media2) and added both ports to
> > > team1 again (Have you tried turning it off and on again...).
> > > After doing this, I now see TX dropped packets on the not yet rebooted media1
> > > team1 interface.
> > > Why do I only see these errors on the team1 interface and not on em3 and em4?
> > >
> > > Currently the copy process of 20TB is still running. I think that if I reboot
> > > media1 everything will work as expected again for some time because it worked
> > > during my initial tests.
> > > But I do think that I hit a bug in the teaming driver and the raising error
> > > counters will come back.
> > >
> > > Could anybody help?

Hmm, I went over the results, do not see anything suspicious.
Are you able to run systemtap scripts on the machine?
> > Interesting. Could you do "teamdctl team1 s d" when the issue happens
> > and post me the result?
>
> I just re-enabled port em4 on both machines and the issue happens again. Here
> are your requested results:
> media1 ~]# teamdctl team1 s d
> {
>    "ports": {
>        "em3": {
>            "ifinfo": {
>                "dev_addr": "b8:2a:72:d5:6c:e6",
>                "dev_addr_len": 6,
>                "ifindex": 4,
>                "ifname": "em3"
>            },
>            "link": {
>                "duplex": "full",
>                "speed": 1000,
>                "up": true
>            },
>            "link_watches": {
>                "list": {
>                    "link_watch_0": {
>                        "delay_down": 0,
>                        "delay_up": 0,
>                        "name": "ethtool",
>                        "up": true
>                    }
>                },
>                "up": true
>            }
>        },
>        "em4": {
>            "ifinfo": {
>                "dev_addr": "b8:2a:72:d5:6c:e6",
>                "dev_addr_len": 6,
>                "ifindex": 5,
>                "ifname": "em4"
>            },
>            "link": {
>                "duplex": "full",
>                "speed": 1000,
>                "up": true
>            },
>            "link_watches": {
>                "list": {
>                    "link_watch_0": {
>                        "delay_down": 0,
>                        "delay_up": 0,
>                        "name": "ethtool",
>                        "up": true
>                    }
>                },
>                "up": true
>            }
>        }
>    },
>    "setup": {
>        "daemonized": false,
>        "dbus_enabled": true,
>        "debug_level": 0,
>        "kernel_team_mode_name": "roundrobin",
>        "pid": 3107,
>        "pid_file": "/var/run/teamd/team1.pid",
>        "runner_name": "roundrobin",
>        "zmq_enabled": false
>    },
>    "team_device": {
>        "ifinfo": {
>            "dev_addr": "b8:2a:72:d5:6c:e6",
>            "dev_addr_len": 6,
>            "ifindex": 7,
>            "ifname": "team1"
>        }
>    }
> }
>
> media2 ~]# teamdctl team1 s d
> {
>    "ports": {
>        "em3": {
>            "ifinfo": {
>                "dev_addr": "b8:2a:72:d5:97:18",
>                "dev_addr_len": 6,
>                "ifindex": 4,
>                "ifname": "em3"
>            },
>            "link": {
>                "duplex": "full",
>                "speed": 1000,
>                "up": true
>            },
>            "link_watches": {
>                "list": {
>                    "link_watch_0": {
>                        "delay_down": 0,
>                        "delay_up": 0,
>                        "name": "ethtool",
>                        "up": true
>                    }
>                },
>                "up": true
>            }
>        },
>        "em4": {
>            "ifinfo": {
>                "dev_addr": "b8:2a:72:d5:97:18",
>                "dev_addr_len": 6,
>                "ifindex": 5,
>                "ifname": "em4"
>            },
>            "link": {
>                "duplex": "full",
>                "speed": 1000,
>                "up": true
>            },
>            "link_watches": {
>                "list": {
>                    "link_watch_0": {
>                        "delay_down": 0,
>                        "delay_up": 0,
>                        "name": "ethtool",
>                        "up": true
>                    }
>                },
>                "up": true
>            }
>        }
>    },
>    "setup": {
>        "daemonized": false,
>        "dbus_enabled": true,
>        "debug_level": 0,
>        "kernel_team_mode_name": "roundrobin",
>        "pid": 3084,
>        "pid_file": "/var/run/teamd/team1.pid",
>        "runner_name": "roundrobin",
>        "zmq_enabled": false
>    },
>    "team_device": {
>        "ifinfo": {
>            "dev_addr": "b8:2a:72:d5:97:18",
>            "dev_addr_len": 6,
>            "ifindex": 7,
>            "ifname": "team1"
>        }
>    }
> }
> > Also you can try random runner instead of roundrobin. The resulting
> > behaviour is almost the same. Maybe the issue won't happen there.
>
> I do believe that if I "restart" the roundrobin runner on media1, the problem
> is gone anyway. So for the moment I'll keep roundrobin just to keep the
> failure environment in place in case you need some more data.
> > Also, can you please post me output of:
> > ethtool -i em3
>
> media1 ~]# ethtool -i em3
> driver: tg3
> version: 3.136
> firmware-version: FFV7.8.16 bc 5720-v1.32
> bus-info: 0000:02:00.0
> supports-statistics: yes
> supports-test: yes
> supports-eeprom-access: yes
> supports-register-dump: yes
> supports-priv-flags: no
>
> media2 ~]# ethtool -i em3
> driver: tg3
> version: 3.136
> firmware-version: FFV7.8.16 bc 5720-v1.32
> bus-info: 0000:02:00.0
> supports-statistics: yes
> supports-test: yes
> supports-eeprom-access: yes
> supports-register-dump: yes
> supports-priv-flags: no
> > ethtool em3
>
> media1 ~]# ethtool em3
> Settings for em3:
>        Supported ports: [ TP ]
>        Supported link modes:   10baseT/Half 10baseT/Full
>                                100baseT/Half 100baseT/Full
>                                1000baseT/Half 1000baseT/Full
>        Supported pause frame use: No
>        Supports auto-negotiation: Yes
>        Advertised link modes:  10baseT/Half 10baseT/Full
>                                100baseT/Half 100baseT/Full
>                                1000baseT/Half 1000baseT/Full
>        Advertised pause frame use: Symmetric
>        Advertised auto-negotiation: Yes
>        Link partner advertised link modes:  10baseT/Half 10baseT/Full
>                                             100baseT/Half 100baseT/Full
>                                             1000baseT/Half 1000baseT/Full
>        Link partner advertised pause frame use: Symmetric
>        Link partner advertised auto-negotiation: Yes
>        Speed: 1000Mb/s
>        Duplex: Full
>        Port: Twisted Pair
>        PHYAD: 1
>        Transceiver: internal
>        Auto-negotiation: on
>        MDI-X: off
>        Supports Wake-on: g
>        Wake-on: d
>        Current message level: 0x000000ff (255)
>                               drv probe link timer ifdown ifup rx_err tx_err
>        Link detected: yes
>
> media2 ~]# ethtool em3
> Settings for em3:
>        Supported ports: [ TP ]
>        Supported link modes:   10baseT/Half 10baseT/Full
>                                100baseT/Half 100baseT/Full
>                                1000baseT/Half 1000baseT/Full
>        Supported pause frame use: No
>        Supports auto-negotiation: Yes
>        Advertised link modes:  10baseT/Half 10baseT/Full
>                                100baseT/Half 100baseT/Full
>                                1000baseT/Half 1000baseT/Full
>        Advertised pause frame use: Symmetric
>        Advertised auto-negotiation: Yes
>        Link partner advertised link modes:  10baseT/Half 10baseT/Full
>                                             100baseT/Half 100baseT/Full
>                                             1000baseT/Half 1000baseT/Full
>        Link partner advertised pause frame use: Symmetric
>        Link partner advertised auto-negotiation: Yes
>        Speed: 1000Mb/s
>        Duplex: Full
>        Port: Twisted Pair
>        PHYAD: 1
>        Transceiver: internal
>        Auto-negotiation: on
>        MDI-X: on
>        Supports Wake-on: g
>        Wake-on: d
>        Current message level: 0x000000ff (255)
>                               drv probe link timer ifdown ifup rx_err tx_err
>        Link detected: yes
> > ethtool -S em3
>
> media1 ~]# ethtool -S em3
> NIC statistics:
>     rx_octets: 26853394715
>     rx_fragments: 0
>     rx_ucast_packets: 227879267
>     rx_mcast_packets: 84
>     rx_bcast_packets: 6
>     rx_fcs_errors: 0
>     rx_align_errors: 0
>     rx_xon_pause_rcvd: 0
>     rx_xoff_pause_rcvd: 0
>     rx_mac_ctrl_rcvd: 0
>     rx_xoff_entered: 0
>     rx_frame_too_long_errors: 0
>     rx_jabbers: 0
>     rx_undersize_packets: 0
>     rx_in_length_errors: 0
>     rx_out_length_errors: 0
>     rx_64_or_less_octet_packets: 0
>     rx_65_to_127_octet_packets: 0
>     rx_128_to_255_octet_packets: 0
>     rx_256_to_511_octet_packets: 0
>     rx_512_to_1023_octet_packets: 0
>     rx_1024_to_1522_octet_packets: 0
>     rx_1523_to_2047_octet_packets: 0
>     rx_2048_to_4095_octet_packets: 0
>     rx_4096_to_8191_octet_packets: 0
>     rx_8192_to_9022_octet_packets: 0
>     tx_octets: 1179249531811
>     tx_collisions: 0
>     tx_xon_sent: 0
>     tx_xoff_sent: 0
>     tx_flow_control: 0
>     tx_mac_errors: 0
>     tx_single_collisions: 0
>     tx_mult_collisions: 0
>     tx_deferred: 0
>     tx_excessive_collisions: 0
>     tx_late_collisions: 0
>     tx_collide_2times: 0
>     tx_collide_3times: 0
>     tx_collide_4times: 0
>     tx_collide_5times: 0
>     tx_collide_6times: 0
>     tx_collide_7times: 0
>     tx_collide_8times: 0
>     tx_collide_9times: 0
>     tx_collide_10times: 0
>     tx_collide_11times: 0
>     tx_collide_12times: 0
>     tx_collide_13times: 0
>     tx_collide_14times: 0
>     tx_collide_15times: 0
>     tx_ucast_packets: 254815741
>     tx_mcast_packets: 8
>     tx_bcast_packets: 612
>     tx_carrier_sense_errors: 0
>     tx_discards: 0
>     tx_errors: 0
>     dma_writeq_full: 0
>     dma_write_prioq_full: 0
>     rxbds_empty: 0
>     rx_discards: 0
>     rx_errors: 0
>     rx_threshold_hit: 0
>     dma_readq_full: 0
>     dma_read_prioq_full: 0
>     tx_comp_queue_full: 0
>     ring_set_send_prod_index: 0
>     ring_status_update: 0
>     nic_irqs: 0
>     nic_avoided_irqs: 0
>     nic_tx_threshold_hit: 0
>     mbuf_lwm_thresh_hit: 0
>
> media2 ~]# ethtool -S em3
> NIC statistics:
>     rx_octets: 782620145025
>     rx_fragments: 0
>     rx_ucast_packets: 173893766
>     rx_mcast_packets: 0
>     rx_bcast_packets: 0
>     rx_fcs_errors: 0
>     rx_align_errors: 0
>     rx_xon_pause_rcvd: 0
>     rx_xoff_pause_rcvd: 0
>     rx_mac_ctrl_rcvd: 0
>     rx_xoff_entered: 0
>     rx_frame_too_long_errors: 0
>     rx_jabbers: 0
>     rx_undersize_packets: 0
>     rx_in_length_errors: 0
>     rx_out_length_errors: 0
>     rx_64_or_less_octet_packets: 0
>     rx_65_to_127_octet_packets: 0
>     rx_128_to_255_octet_packets: 0
>     rx_256_to_511_octet_packets: 0
>     rx_512_to_1023_octet_packets: 0
>     rx_1024_to_1522_octet_packets: 0
>     rx_1523_to_2047_octet_packets: 0
>     rx_2048_to_4095_octet_packets: 0
>     rx_4096_to_8191_octet_packets: 0
>     rx_8192_to_9022_octet_packets: 0
>     tx_octets: 17210360858
>     tx_collisions: 0
>     tx_xon_sent: 0
>     tx_xoff_sent: 0
>     tx_flow_control: 0
>     tx_mac_errors: 0
>     tx_single_collisions: 0
>     tx_mult_collisions: 0
>     tx_deferred: 0
>     tx_excessive_collisions: 0
>     tx_late_collisions: 0
>     tx_collide_2times: 0
>     tx_collide_3times: 0
>     tx_collide_4times: 0
>     tx_collide_5times: 0
>     tx_collide_6times: 0
>     tx_collide_7times: 0
>     tx_collide_8times: 0
>     tx_collide_9times: 0
>     tx_collide_10times: 0
>     tx_collide_11times: 0
>     tx_collide_12times: 0
>     tx_collide_13times: 0
>     tx_collide_14times: 0
>     tx_collide_15times: 0
>     tx_ucast_packets: 147864937
>     tx_mcast_packets: 23
>     tx_bcast_packets: 2
>     tx_carrier_sense_errors: 0
>     tx_discards: 0
>     tx_errors: 0
>     dma_writeq_full: 0
>     dma_write_prioq_full: 0
>     rxbds_empty: 0
>     rx_discards: 0
>     rx_errors: 0
>     rx_threshold_hit: 0
>     dma_readq_full: 0
>     dma_read_prioq_full: 0
>     tx_comp_queue_full: 0
>     ring_set_send_prod_index: 0
>     ring_status_update: 0
>     nic_irqs: 0
>     nic_avoided_irqs: 0
>     nic_tx_threshold_hit: 0
>     mbuf_lwm_thresh_hit: 0
> > ethtool -i em4
>
> media1 ~]# ethtool -i em4
> driver: tg3
> version: 3.136
> firmware-version: FFV7.8.16 bc 5720-v1.32
> bus-info: 0000:02:00.1
> supports-statistics: yes
> supports-test: yes
> supports-eeprom-access: yes
> supports-register-dump: yes
> supports-priv-flags: no
>
> media2 ~]# ethtool -i em4
> driver: tg3
> version: 3.136
> firmware-version: FFV7.8.16 bc 5720-v1.32
> bus-info: 0000:02:00.1
> supports-statistics: yes
> supports-test: yes
> supports-eeprom-access: yes
> supports-register-dump: yes
> supports-priv-flags: no
> > ethtool em4
>
> media1 ~]# ethtool em4
> Settings for em4:
>        Supported ports: [ TP ]
>        Supported link modes:   10baseT/Half 10baseT/Full
>                                100baseT/Half 100baseT/Full
>                                1000baseT/Half 1000baseT/Full
>        Supported pause frame use: No
>        Supports auto-negotiation: Yes
>        Advertised link modes:  10baseT/Half 10baseT/Full
>                                100baseT/Half 100baseT/Full
>                                1000baseT/Half 1000baseT/Full
>        Advertised pause frame use: Symmetric
>        Advertised auto-negotiation: Yes
>        Link partner advertised link modes:  10baseT/Half 10baseT/Full
>                                             100baseT/Half 100baseT/Full
>                                             1000baseT/Half 1000baseT/Full
>        Link partner advertised pause frame use: Symmetric
>        Link partner advertised auto-negotiation: Yes
>        Speed: 1000Mb/s
>        Duplex: Full
>        Port: Twisted Pair
>        PHYAD: 2
>        Transceiver: internal
>        Auto-negotiation: on
>        MDI-X: off
>        Supports Wake-on: g
>        Wake-on: d
>        Current message level: 0x000000ff (255)
>                               drv probe link timer ifdown ifup rx_err tx_err
>        Link detected: yes
>
> media2 ~]# ethtool em4
> Settings for em4:
>        Supported ports: [ TP ]
>        Supported link modes:   10baseT/Half 10baseT/Full
>                                100baseT/Half 100baseT/Full
>                                1000baseT/Half 1000baseT/Full
>        Supported pause frame use: No
>        Supports auto-negotiation: Yes
>        Advertised link modes:  10baseT/Half 10baseT/Full
>                                100baseT/Half 100baseT/Full
>                                1000baseT/Half 1000baseT/Full
>        Advertised pause frame use: Symmetric
>        Advertised auto-negotiation: Yes
>        Link partner advertised link modes:  10baseT/Half 10baseT/Full
>                                             100baseT/Half 100baseT/Full
>                                             1000baseT/Half 1000baseT/Full
>        Link partner advertised pause frame use: Symmetric
>        Link partner advertised auto-negotiation: Yes
>        Speed: 1000Mb/s
>        Duplex: Full
>        Port: Twisted Pair
>        PHYAD: 2
>        Transceiver: internal
>        Auto-negotiation: on
>        MDI-X: on
>        Supports Wake-on: g
>        Wake-on: d
>        Current message level: 0x000000ff (255)
>                               drv probe link timer ifdown ifup rx_err tx_err
>        Link detected: yes
> > ethtool -S em4
>
> media1 ~]# ethtool -S em4
> NIC statistics:
>     rx_octets: 246361
>     rx_fragments: 0
>     rx_ucast_packets: 912
>     rx_mcast_packets: 8
>     rx_bcast_packets: 0
>     rx_fcs_errors: 0
>     rx_align_errors: 0
>     rx_xon_pause_rcvd: 0
>     rx_xoff_pause_rcvd: 0
>     rx_mac_ctrl_rcvd: 0
>     rx_xoff_entered: 0
>     rx_frame_too_long_errors: 0
>     rx_jabbers: 0
>     rx_undersize_packets: 0
>     rx_in_length_errors: 0
>     rx_out_length_errors: 0
>     rx_64_or_less_octet_packets: 0
>     rx_65_to_127_octet_packets: 0
>     rx_128_to_255_octet_packets: 0
>     rx_256_to_511_octet_packets: 0
>     rx_512_to_1023_octet_packets: 0
>     rx_1024_to_1522_octet_packets: 0
>     rx_1523_to_2047_octet_packets: 0
>     rx_2048_to_4095_octet_packets: 0
>     rx_4096_to_8191_octet_packets: 0
>     rx_8192_to_9022_octet_packets: 0
>     tx_octets: 760
>     tx_collisions: 0
>     tx_xon_sent: 0
>     tx_xoff_sent: 0
>     tx_flow_control: 0
>     tx_mac_errors: 0
>     tx_single_collisions: 0
>     tx_mult_collisions: 0
>     tx_deferred: 0
>     tx_excessive_collisions: 0
>     tx_late_collisions: 0
>     tx_collide_2times: 0
>     tx_collide_3times: 0
>     tx_collide_4times: 0
>     tx_collide_5times: 0
>     tx_collide_6times: 0
>     tx_collide_7times: 0
>     tx_collide_8times: 0
>     tx_collide_9times: 0
>     tx_collide_10times: 0
>     tx_collide_11times: 0
>     tx_collide_12times: 0
>     tx_collide_13times: 0
>     tx_collide_14times: 0
>     tx_collide_15times: 0
>     tx_ucast_packets: 0
>     tx_mcast_packets: 8
>     tx_bcast_packets: 0
>     tx_carrier_sense_errors: 0
>     tx_discards: 0
>     tx_errors: 0
>     dma_writeq_full: 0
>     dma_write_prioq_full: 0
>     rxbds_empty: 0
>     rx_discards: 0
>     rx_errors: 0
>     rx_threshold_hit: 0
>     dma_readq_full: 0
>     dma_read_prioq_full: 0
>     tx_comp_queue_full: 0
>     ring_set_send_prod_index: 0
>     ring_status_update: 0
>     nic_irqs: 0
>     nic_avoided_irqs: 0
>     nic_tx_threshold_hit: 0
>     mbuf_lwm_thresh_hit: 0
>
> media2 ~]# ethtool -S em4
> NIC statistics:
>     rx_octets: 760
>     rx_fragments: 0
>     rx_ucast_packets: 0
>     rx_mcast_packets: 8
>     rx_bcast_packets: 0
>     rx_fcs_errors: 0
>     rx_align_errors: 0
>     rx_xon_pause_rcvd: 0
>     rx_xoff_pause_rcvd: 0
>     rx_mac_ctrl_rcvd: 0
>     rx_xoff_entered: 0
>     rx_frame_too_long_errors: 0
>     rx_jabbers: 0
>     rx_undersize_packets: 0
>     rx_in_length_errors: 0
>     rx_out_length_errors: 0
>     rx_64_or_less_octet_packets: 0
>     rx_65_to_127_octet_packets: 0
>     rx_128_to_255_octet_packets: 0
>     rx_256_to_511_octet_packets: 0
>     rx_512_to_1023_octet_packets: 0
>     rx_1024_to_1522_octet_packets: 0
>     rx_1523_to_2047_octet_packets: 0
>     rx_2048_to_4095_octet_packets: 0
>     rx_4096_to_8191_octet_packets: 0
>     rx_8192_to_9022_octet_packets: 0
>     tx_octets: 247887
>     tx_collisions: 0
>     tx_xon_sent: 0
>     tx_xoff_sent: 0
>     tx_flow_control: 0
>     tx_mac_errors: 0
>     tx_single_collisions: 0
>     tx_mult_collisions: 0
>     tx_deferred: 0
>     tx_excessive_collisions: 0
>     tx_late_collisions: 0
>     tx_collide_2times: 0
>     tx_collide_3times: 0
>     tx_collide_4times: 0
>     tx_collide_5times: 0
>     tx_collide_6times: 0
>     tx_collide_7times: 0
>     tx_collide_8times: 0
>     tx_collide_9times: 0
>     tx_collide_10times: 0
>     tx_collide_11times: 0
>     tx_collide_12times: 0
>     tx_collide_13times: 0
>     tx_collide_14times: 0
>     tx_collide_15times: 0
>     tx_ucast_packets: 927
>     tx_mcast_packets: 8
>     tx_bcast_packets: 0
>     tx_carrier_sense_errors: 0
>     tx_discards: 0
>     tx_errors: 0
>     dma_writeq_full: 0
>     dma_write_prioq_full: 0
>     rxbds_empty: 0
>     rx_discards: 0
>     rx_errors: 0
>     rx_threshold_hit: 0
>     dma_readq_full: 0
>     dma_read_prioq_full: 0
>     tx_comp_queue_full: 0
>     ring_set_send_prod_index: 0
>     ring_status_update: 0
>     nic_irqs: 0
>     nic_avoided_irqs: 0
>     nic_tx_threshold_hit: 0
>     mbuf_lwm_thresh_hit: 0
> > Of course in time the problem happens.
> > > Kind regards
> > > Ingo
> > >
> > > This is the used network config of team1:
> > > media1:
> > > cat /etc/sysconfig/network-scripts/ifcfg-team1_slave_0
> > > # Generated by parse-kickstart
> > > NAME=team1 slave 0
> > > TEAM_MASTER=team1
> > > DEVICETYPE=TeamPort
> > > DEVICE=em3
> > > ONBOOT=yes
> > > UUID=b7026c5b-e9cd-457c-93d2-a2799361ed90
> > >
> > > cat /etc/sysconfig/network-scripts/ifcfg-team1_slave_1
> > > # Generated by parse-kickstart
> > > NAME=team1 slave 1
> > > TEAM_MASTER=team1
> > > DEVICETYPE=TeamPort
> > > DEVICE=em4
> > > ONBOOT=yes
> > > UUID=fbe5bc56-95f5-4f9e-a5ea-ab6b6f5e5b50
> > >
> > > cat /etc/sysconfig/network-scripts/ifcfg-team1
> > > # Generated by parse-kickstart
> > > UUID=4e0090cd-32d3-4c5b-be23-9f53083da4dd
> > > NAME="Team connection team1"
> > > TEAM_CONFIG="{"runner": {"name": "roundrobin"}}"
> > > GATEWAY=
> > > IPV6_AUTOCONF=yes
> > > BOOTPROTO=none
> > > DEVICE=team1
> > > MTU=9000
> > > TYPE=Team
> > > ONBOOT=yes
> > > IPV6INIT=yes
> > > DEVICETYPE=Team
> > > IPADDR0=192.168.101.31
> > > PREFIX0=24
> > > DEFROUTE=yes
> > > IPV4_FAILURE_FATAL=no
> > > IPV6_DEFROUTE=yes
> > > IPV6_PEERDNS=yes
> > > IPV6_PEERROUTES=yes
> > > IPV6_FAILURE_FATAL=no
> > >
> > > media2:
> > > cat /etc/sysconfig/network-scripts/ifcfg-team1_slave_0
> > > # Generated by parse-kickstart
> > > NAME=team1 slave 0
> > > TEAM_MASTER=team1
> > > DEVICETYPE=TeamPort
> > > DEVICE=em3
> > > ONBOOT=yes
> > > UUID=bfd8e77e-7eb8-47ea-ab7e-95286e073202
> > >
> > > cat /etc/sysconfig/network-scripts/ifcfg-team1_slave_1
> > > # Generated by parse-kickstart
> > > NAME=team1 slave 1
> > > TEAM_MASTER=team1
> > > DEVICETYPE=TeamPort
> > > DEVICE=em4
> > > ONBOOT=yes
> > > UUID=93c5dfd8-3b81-4d1e-8ba9-c46f8daceb1a
> > >
> > > cat /etc/sysconfig/network-scripts/ifcfg-team1
> > > # Generated by parse-kickstart
> > > UUID=79c843db-996c-41e4-9ee5-2a8f3da244ed
> > > NAME="Team connection team1"
> > > TEAM_CONFIG="{"runner": {"name": "roundrobin"}}"
> > > GATEWAY=
> > > IPV6_AUTOCONF=yes
> > > BOOTPROTO=none
> > > DEVICE=team1
> > > MTU=9000
> > > TYPE=Team
> > > ONBOOT=yes
> > > IPV6INIT=yes
> > > DEVICETYPE=Team
> > > IPADDR0=192.168.101.32
> > > PREFIX0=24
> > > DEFROUTE=yes
> > > IPV4_FAILURE_FATAL=no
> > > IPV6_DEFROUTE=yes
> > > IPV6_PEERDNS=yes
> > > IPV6_PEERROUTES=yes
> > > IPV6_FAILURE_FATAL=no
> > > Here are some system logs from both machines:
> > > media1:
> > >
> > > Nov  6 02:31:27 media1 corosync[3657]: [TOTEM ] Incrementing problem counter
> > > for seqid 412041 iface 192.168.100.31 to [1 of 10]
> > > Nov  6 02:31:29 media1 corosync[3657]: [TOTEM ] ring 0 active with no faults
> > > Nov  6 02:31:29 media1 corosync[3657]: [TOTEM ] Incrementing problem counter
> > > for seqid 412047 iface 192.168.100.31 to [1 of 10]
> > > Nov  6 02:31:31 media1 corosync[3657]: [TOTEM ] ring 0 active with no faults
> > > Nov  6 02:32:51 media1 corosync[3657]: [TOTEM ] Incrementing problem counter
> > > for seqid 412315 iface 192.168.101.31 to [1 of 10]
> > > Nov  6 02:32:53 media1 corosync[3657]: [TOTEM ] ring 1 active with no faults
> > > Nov  6 02:32:53 media1 corosync[3657]: [TOTEM ] Incrementing problem counter
> > > for seqid 412319 iface 192.168.101.31 to [1 of 10]
> > > Nov  6 02:32:54 media1 corosync[3657]: [TOTEM ] Incrementing problem counter
> > > for seqid 412321 iface 192.168.101.31 to [2 of 10]
> > > Nov  6 02:32:55 media1 corosync[3657]: [TOTEM ] Decrementing problem counter
> > > for iface 192.168.101.31 to [1 of 10]
> > > Nov  6 02:32:56 media1 corosync[3657]: [TOTEM ] Incrementing problem counter
> > > for seqid 412325 iface 192.168.101.31 to [2 of 10]
> > > Nov  6 02:32:57 media1 corosync[3657]: [TOTEM ] Decrementing problem counter
> > > for iface 192.168.101.31 to [1 of 10]
> > > Nov  6 02:32:58 media1 corosync[3657]: [TOTEM ] Incrementing problem counter
> > > for seqid 412327 iface 192.168.101.31 to [2 of 10]
> > > Nov  6 02:32:59 media1 kernel: drbd r0: PingAck did not arrive in time.
> > > Nov  6 02:32:59 media1 kernel: drbd r0: peer( Secondary -> Unknown ) conn(
> > > Connected -> NetworkFailure ) pdsk( UpToDate -> DUnknown )
> > > Nov  6 02:32:59 media1 kernel: block drbd0: new current UUID
> > > 0C0308FA4F94F811:59E43E5FD7B26A4F:1963884285B385E6:1962884285B385E6
> > > Nov  6 02:32:59 media1 kernel: drbd r0: asender terminated
> > > Nov  6 02:32:59 media1 kernel: drbd r0: Terminating drbd_a_r0
> > > Nov  6 02:32:59 media1 kernel: drbd r0: Connection closed
> > > Nov  6 02:32:59 media1 kernel: drbd r0: conn( NetworkFailure -> Unconnected )
> > > Nov  6 02:32:59 media1 kernel: drbd r0: receiver terminated
> > > Nov  6 02:32:59 media1 kernel: drbd r0: Restarting receiver thread
> > > Nov  6 02:32:59 media1 kernel: drbd r0: receiver (re)started
> > > Nov  6 02:32:59 media1 kernel: drbd r0: conn( Unconnected -> WFConnection )
> > > Nov  6 02:32:59 media1 corosync[3657]: [TOTEM ] Decrementing problem counter
> > > for iface 192.168.101.31 to [1 of 10]
> > > Nov  6 02:32:59 media1 corosync[3657]: [TOTEM ] Incrementing problem counter
> > > for seqid 412329 iface 192.168.101.31 to [2 of 10]
> > > Nov  6 02:33:01 media1 corosync[3657]: [TOTEM ] Incrementing problem counter
> > > for seqid 412333 iface 192.168.101.31 to [3 of 10]
> > > Nov  6 02:33:01 media1 corosync[3657]: [TOTEM ] Decrementing problem counter
> > > for iface 192.168.101.31 to [2 of 10]
> > > Nov  6 02:33:01 media1 crmd[4960]: notice: do_state_transition: State
> > > transition S_IDLE -> S_POLICY_ENGINE [ input=I_PE_CALC cause=C_FSA_INTERNAL
> > > origin=abort_transition_graph ]
> > >
> > > Nov  6 02:33:01 media1 pengine[4959]: notice: unpack_config: On loss of CCM
> > > Quorum: Ignore
> > >
> > > Nov  6 02:33:01 media1 pengine[4959]: warning: unpack_rsc_op: Processing
> > > failed op monitor for res_nfsserver_mediafiles on media1: unknown error (1)
> > > Nov  6 02:33:01 media1 pengine[4959]: warning: unpack_rsc_op: Processing
> > > failed op monitor for res_nfsserver_mediafiles on media2: unknown error (1)
> > > Nov  6 02:33:01 media1 pengine[4959]: notice: process_pe_message: Calculated
> > > Transition 144: /var/lib/pacemaker/pengine/pe-input-227.bz2
> > >
> > > Nov  6 02:33:01 media1 crmd[4960]: notice: run_graph: Transition 144
> > > (Complete=0, Pending=0, Fired=0, Skipped=0, Incomplete=0,
> > > Source=/var/lib/pacemaker/pengine/pe-input-227.bz2): Complete
> > >
> > > Nov  6 02:33:01 media1 crmd[4960]: notice: do_state_transition: State
> > > transition S_TRANSITION_ENGINE -> S_IDLE [ input=I_TE_SUCCESS
> > > cause=C_FSA_INTERNAL origin=notify_crmd ]
> > > Nov  6 02:33:01 media1 kernel: drbd r0: Handshake successful: Agreed network
> > > protocol version 101
> > >
> > > Nov  6 02:33:01 media1 kernel: drbd r0: Agreed to support TRIM on protocol
> > > level
> > >
> > > Nov  6 02:33:01 media1 kernel: drbd r0: Peer authenticated using 20 bytes
> > > HMAC
> > >
> > > Nov  6 02:33:01 media1 kernel: drbd r0: conn( WFConnection -> WFReportParams
> > > )
> > >
> > > Nov  6 02:33:01 media1 kernel: drbd r0: Starting asender thread (from
> > > drbd_r_r0 [5999])
> > >
> > > Nov  6 02:33:02 media1 corosync[3657]: [TOTEM ] Incrementing problem counter
> > > for seqid 412337 iface 192.168.101.31 to [3 of 10]
> > >
> > > Nov  6 02:33:02 media1 kernel: block drbd0: drbd_sync_handshake:
> > >
> > > Nov  6 02:33:02 media1 kernel: block drbd0: self
> > > 0C0308FA4F94F811:59E43E5FD7B26A4F:1963884285B385E6:1962884285B385E6
> > > bits:192101 flags:0
> > > Nov  6 02:33:02 media1 kernel: block drbd0: peer
> > > 59E43E5FD7B26A4E:0000000000000000:1963884285B385E6:1962884285B385E6 bits:0
> > > flags:0
> > > Nov  6 02:33:02 media1 kernel: block drbd0: uuid_compare()=1 by rule 70
> > >
> > > Nov  6 02:33:02 media1 kernel: block drbd0: peer( Unknown -> Secondary )
> > > conn( WFReportParams -> WFBitMapS ) pdsk( DUnknown -> Consistent )
> > >
> > > Nov  6 02:33:02 media1 kernel: block drbd0: send bitmap stats
> > > [Bytes(packets)]: plain 0(0), RLE 185(1), total 185; compression: 100.0%
> > >
> > > Nov  6 02:33:02 media1 corosync[3657]: [TOTEM ] Incrementing problem counter
> > > for seqid 412339 iface 192.168.101.31 to [4 of 10]
> > >
> > > Nov  6 02:33:03 media1 kernel: block drbd0: receive bitmap stats
> > > [Bytes(packets)]: plain 0(0), RLE 185(1), total 185; compression: 100.0%
> > >
> > > Nov  6 02:33:03 media1 kernel: block drbd0: helper command: /sbin/drbdadm
> > > before-resync-source minor-0
> > >
> > > Nov  6 02:33:03 media1 kernel: block drbd0: helper command: /sbin/drbdadm
> > > before-resync-source minor-0 exit code 0 (0x0)
> > >
> > > Nov  6 02:33:03 media1 kernel: block drbd0: conn( WFBitMapS -> SyncSource )
> > > pdsk( Consistent -> Inconsistent )
> > >
> > > Nov  6 02:33:03 media1 kernel: block drbd0: Began resync as SyncSource (will
> > > sync 919244 KB [229811 bits set]).
> > >
> > > Nov  6 02:33:03 media1 kernel: block drbd0: updated sync UUID
> > > 0C0308FA4F94F811:59E53E5FD7B26A4F:59E43E5FD7B26A4F:1963884285B385E6
> > >
> > > Nov  6 02:33:03 media1 corosync[3657]: [TOTEM ] Decrementing problem counter
> > > for iface 192.168.101.31 to [3 of 10]
> > >
> > > Nov  6 02:33:03 media1 corosync[3657]: [TOTEM ] Incrementing problem counter
> > > for seqid 412343 iface 192.168.101.31 to [4 of 10]
> > >
> > > Nov  6 02:33:04 media1 corosync[3657]: [TOTEM ] Incrementing problem counter
> > > for seqid 412345 iface 192.168.101.31 to [5 of 10]
> > >
> > > Nov  6 02:33:05 media1 corosync[3657]: [TOTEM ] Incrementing problem counter
> > > for seqid 412347 iface 192.168.101.31 to [6 of 10]
> > >
> > > Nov  6 02:33:05 media1 corosync[3657]: [TOTEM ] Decrementing problem counter
> > > for iface 192.168.101.31 to [5 of 10]
> > >
> > > Nov  6 02:33:05 media1 corosync[3657]: [TOTEM ] Incrementing problem counter
> > > for seqid 412349 iface 192.168.101.31 to [6 of 10]
> > >
> > > Nov  6 02:33:06 media1 corosync[3657]: [TOTEM ] Incrementing problem counter
> > > for seqid 412351 iface 192.168.101.31 to [7 of 10]
> > >
> > > Nov  6 02:33:07 media1 corosync[3657]: [TOTEM ] Incrementing problem counter
> > > for seqid 412355 iface 192.168.101.31 to [8 of 10]
> > >
> > > Nov  6 02:33:07 media1 corosync[3657]: [TOTEM ] Decrementing problem counter
> > > for iface 192.168.101.31 to [7 of 10]
> > >
> > > Nov  6 02:33:07 media1 corosync[3657]: [TOTEM ] Incrementing problem counter
> > > for seqid 412357 iface 192.168.101.31 to [8 of 10]
> > >
> > > Nov  6 02:33:08 media1 corosync[3657]: [TOTEM ] Incrementing problem counter
> > > for seqid 412361 iface 192.168.101.31 to [9 of 10]
> > >
> > > Nov  6 02:33:09 media1 corosync[3657]: [TOTEM ] Incrementing problem counter
> > > for seqid 412363 iface 192.168.101.31 to [10 of 10]
> > >
> > > Nov  6 02:33:09 media1 corosync[3657]: [TOTEM ] Marking seqid 412363 ringid 1
> > > interface 192.168.101.31 FAULTY
> > >
> > > Nov  6 02:33:11 media1 crmd[4960]: notice: do_state_transition: State
> > > transition S_IDLE -> S_POLICY_ENGINE [ input=I_PE_CALC cause=C_FSA_INTERNAL
> > > origin=abort_transition_graph ]
> > >
> > > Nov  6 02:33:11 media1 pengine[4959]: notice: unpack_config: On loss of CCM
> > > Quorum: Ignore
> > >
> > > Nov  6 02:33:11 media1 pengine[4959]: warning: unpack_rsc_op: Processing
> > > failed op monitor for res_nfsserver_mediafiles on media1: unknown error (1)
> > > Nov  6 02:33:11 media1 pengine[4959]: warning: unpack_rsc_op: Processing
> > > failed op monitor for res_nfsserver_mediafiles on media2: unknown error (1)
> > > Nov  6 02:33:11 media1 pengine[4959]: notice: process_pe_message: Calculated
> > > Transition 145: /var/lib/pacemaker/pengine/pe-input-228.bz2
> > >
> > > Nov  6 02:33:11 media1 crmd[4960]: notice: run_graph: Transition 145
> > > (Complete=0, Pending=0, Fired=0, Skipped=0, Incomplete=0,
> > > Source=/var/lib/pacemaker/pengine/pe-input-228.bz2): Complete
> > > Nov  6 02:33:11 media1 crmd[4960]: notice: do_state_transition: State
> > > transition S_TRANSITION_ENGINE -> S_IDLE [ input=I_TE_SUCCESS
> > > cause=C_FSA_INTERNAL origin=notify_crmd ]
> > > Nov  6 02:33:17 media1 corosync[3657]: [TOTEM ] Automatically recovered ring
> > > 1
> > >
> > > Nov  6 02:33:18 media1 corosync[3657]: [TOTEM ] Incrementing problem counter
> > > for seqid 412387 iface 192.168.101.31 to [1 of 10]
> > >
> > > Nov  6 02:33:20 media1 corosync[3657]: [TOTEM ] Incrementing problem counter
> > > for seqid 412389 iface 192.168.101.31 to [2 of 10]
> > >
> > > Nov  6 02:33:20 media1 corosync[3657]: [TOTEM ] Incrementing problem counter
> > > for seqid 412391 iface 192.168.101.31 to [3 of 10]
> > >
> > > Nov  6 02:33:21 media1 corosync[3657]: [TOTEM ] Incrementing problem counter
> > > for seqid 412393 iface 192.168.101.31 to [4 of 10]
> > >
> > > Nov  6 02:33:22 media1 corosync[3657]: [TOTEM ] Incrementing problem counter
> > > for seqid 412397 iface 192.168.101.31 to [5 of 10]
> > >
> > > Nov  6 02:33:22 media1 corosync[3657]: [TOTEM ] Incrementing problem counter
> > > for seqid 412399 iface 192.168.101.31 to [6 of 10]
> > >
> > > Nov  6 02:33:24 media1 corosync[3657]: [TOTEM ] Incrementing problem counter
> > > for seqid 412403 iface 192.168.101.31 to [7 of 10]
> > >
> > > Nov  6 02:33:31 media1 corosync[3657]: [TOTEM ] Incrementing problem counter
> > > for seqid 412421 iface 192.168.101.31 to [8 of 10]
> > >
> > > Nov  6 02:33:32 media1 corosync[3657]: [TOTEM ] Incrementing problem counter
> > > for seqid 412423 iface 192.168.101.31 to [9 of 10]
> > >
> > > Nov  6 02:33:34 media1 corosync[3657]: [TOTEM ] Incrementing problem counter
> > > for seqid 412429 iface 192.168.101.31 to [10 of 10]
> > >
> > > Nov  6 02:33:34 media1 corosync[3657]: [TOTEM ] Marking seqid 412429 ringid 1
> > > interface 192.168.101.31 FAULTY
> > >
> > > Nov  6 02:33:36 media1 corosync[3657]: [TOTEM ] Automatically recovered ring
> > > 1
> > >
> > > Nov  6 02:33:39 media1 corosync[3657]: [TOTEM ] Incrementing problem counter
> > > for seqid 412437 iface 192.168.101.31 to [1 of 10]
> > >
> > > ============================================================
> > > media2:
> > > ============================================================
> > > Nov  6 02:32:59 media2 kernel: drbd r0: sock was shut down by peer
> > >
> > > Nov  6 02:32:59 media2 kernel: drbd r0: peer( Primary -> Unknown ) conn(
> > > Connected -> BrokenPipe ) pdsk( UpToDate -> DUnknown )
> > >
> > > Nov  6 02:32:59 media2 kernel: drbd r0: short read (expected size 16)
> > >
> > > Nov  6 02:32:59 media2 kernel: drbd r0: asender terminated
> > >
> > > Nov  6 02:32:59 media2 kernel: drbd r0: Terminating drbd_a_r0
> > >
> > > Nov  6 02:32:59 media2 kernel: drbd r0: Connection closed
> > >
> > > Nov  6 02:32:59 media2 kernel: drbd r0: conn( BrokenPipe -> Unconnected )
> > >
> > > Nov  6 02:32:59 media2 kernel: drbd r0: receiver terminated
> > >
> > > Nov  6 02:32:59 media2 kernel: drbd r0: Restarting receiver thread
> > >
> > > Nov  6 02:32:59 media2 kernel: drbd r0: receiver (re)started
> > >
> > > Nov  6 02:32:59 media2 kernel: drbd r0: conn( Unconnected -> WFConnection )
> > >
> > > Nov  6 02:33:01 media2 attrd[4940]: notice: attrd_trigger_update: Sending
> > > flush op to all hosts for: master-res_drbd_mediafiles (1000)
> > >
> > > Nov  6 02:33:01 media2 attrd[4940]: notice: attrd_perform_update: Sent update
> > > 16: master-res_drbd_mediafiles=1000
> > >
> > > Nov  6 02:33:01 media2 kernel: drbd r0: Handshake successful: Agreed network
> > > protocol version 101
> > >
> > > Nov  6 02:33:01 media2 kernel: drbd r0: Agreed to support TRIM on protocol
> > > level
> > >
> > > Nov  6 02:33:01 media2 kernel: drbd r0: Peer authenticated using 20 bytes
> > > HMAC
> > >
> > > Nov  6 02:33:01 media2 kernel: drbd r0: conn( WFConnection -> WFReportParams
> > > )
> > >
> > > Nov  6 02:33:01 media2 kernel: drbd r0: Starting asender thread (from
> > > drbd_r_r0 [6367])
> > >
> > > Nov  6 02:33:01 media2 kernel: block drbd0: drbd_sync_handshake:
> > >
> > > Nov  6 02:33:01 media2 kernel: block drbd0: self
> > > 59E43E5FD7B26A4E:0000000000000000:1963884285B385E6:1962884285B385E6 bits:0
> > > flags:0
> > > Nov  6 02:33:01 media2 kernel: block drbd0: peer
> > > 0C0308FA4F94F811:59E43E5FD7B26A4F:1963884285B385E6:1962884285B385E6
> > > bits:192101 flags:0
> > > Nov  6 02:33:01 media2 kernel: block drbd0: uuid_compare()=-1 by rule 50
> > >
> > > Nov  6 02:33:01 media2 kernel: block drbd0: peer( Unknown -> Primary ) conn(
> > > WFReportParams -> WFBitMapT ) disk( UpToDate -> Outdated ) pdsk( DUnknown ->
> > > UpToDate )
> > > Nov  6 02:33:02 media2 kernel: block drbd0: receive bitmap stats
> > > [Bytes(packets)]: plain 0(0), RLE 185(1), total 185; compression: 100.0%
> > >
> > > Nov  6 02:33:03 media2 kernel: block drbd0: send bitmap stats
> > > [Bytes(packets)]: plain 0(0), RLE 185(1), total 185; compression: 100.0%
> > >
> > > Nov  6 02:33:03 media2 kernel: block drbd0: conn( WFBitMapT -> WFSyncUUID )
> > >
> > > Nov  6 02:33:03 media2 kernel: block drbd0: updated sync uuid
> > > 59E53E5FD7B26A4E:0000000000000000:1963884285B385E6:1962884285B385E6
> > >
> > > Nov  6 02:33:03 media2 kernel: block drbd0: helper command: /sbin/drbdadm
> > > before-resync-target minor-0
> > >
> > > Nov  6 02:33:03 media2 kernel: block drbd0: helper command: /sbin/drbdadm
> > > before-resync-target minor-0 exit code 0 (0x0)
> > >
> > > Nov  6 02:33:03 media2 kernel: block drbd0: conn( WFSyncUUID -> SyncTarget )
> > > disk( Outdated -> Inconsistent )
> > >
> > > Nov  6 02:33:03 media2 kernel: block drbd0: Began resync as SyncTarget (will
> > > sync 919244 KB [229811 bits set]).
> > >
> > > Nov  6 02:33:10 media2 corosync[3690]: [TOTEM ] Incrementing problem counter
> > > for seqid 412364 iface 192.168.101.32 to [1 of 10]
> > >
> > > Nov  6 02:33:10 media2 corosync[3690]: [TOTEM ] Automatically recovered ring
> > > 1
> > >
> > > Nov  6 02:33:10 media2 corosync[3690]: [TOTEM ] Incrementing problem counter
> > > for seqid 412366 iface 192.168.101.32 to [1 of 10]
> > >
> > > Nov  6 02:33:11 media2 corosync[3690]: [TOTEM ] Automatically recovered ring
> > > 1
> > >
> > > Nov  6 02:33:11 media2 attrd[4940]: notice: attrd_trigger_update: Sending
> > > flush op to all hosts for: master-res_drbd_mediafiles (10)
> > >
> > > Nov  6 02:33:11 media2 attrd[4940]: notice: attrd_perform_update: Sent update
> > > 18: master-res_drbd_mediafiles=10
> > >
> > > Nov  6 02:33:11 media2 corosync[3690]: [TOTEM ] Incrementing problem counter
> > > for seqid 412368 iface 192.168.101.32 to [1 of 10]
> > >
> > > Nov  6 02:33:12 media2 corosync[3690]: [TOTEM ] ring 1 active with no faults
> > >
> > > Nov  6 02:33:12 media2 corosync[3690]: [TOTEM ] Incrementing problem counter
> > > for seqid 412370 iface 192.168.101.32 to [1 of 10]
> > >
> > > Nov  6 02:33:12 media2 corosync[3690]: [TOTEM ] Automatically recovered ring
> > > 1
> > >
> > > Nov  6 02:33:12 media2 corosync[3690]: [TOTEM ] Incrementing problem counter
> > > for seqid 412372 iface 192.168.101.32 to [1 of 10]
> > >
> > > Nov  6 02:33:13 media2 corosync[3690]: [TOTEM ] Incrementing problem counter
> > > for seqid 412374 iface 192.168.101.32 to [2 of 10]
> > >
> > > Nov  6 02:33:14 media2 corosync[3690]: [TOTEM ] Decrementing problem counter
> > > for iface 192.168.101.32 to [1 of 10]
> > >
> > > Nov  6 02:33:14 media2 corosync[3690]: [TOTEM ] Incrementing problem counter
> > > for seqid 412376 iface 192.168.101.32 to [2 of 10]
> > >
> > > Nov  6 02:33:15 media2 corosync[3690]: [TOTEM ] Incrementing problem counter
> > > for seqid 412378 iface 192.168.101.32 to [3 of 10]
> > >
> > > Nov  6 02:33:15 media2 corosync[3690]: [TOTEM ] Automatically recovered ring
> > > 1
> > >
> > > Nov  6 02:33:15 media2 corosync[3690]: [TOTEM ] Incrementing problem counter
> > > for seqid 412380 iface 192.168.101.32 to [1 of 10]
> > >
> > > Nov  6 02:33:16 media2 corosync[3690]: [TOTEM ] ring 1 active with no faults
> > >
> > > Nov  6 02:33:16 media2 corosync[3690]: [TOTEM ] Incrementing problem counter
> > > for seqid 412382 iface 192.168.101.32 to [1 of 10]
> > >
> > > Nov  6 02:33:16 media2 corosync[3690]: [TOTEM ] Automatically recovered ring
> > > 1
> > >
> > > Nov  6 02:33:17 media2 corosync[3690]: [TOTEM ] Incrementing problem counter
> > > for seqid 412384 iface 192.168.101.32 to [1 of 10]
> > >
> > > Nov  6 02:33:17 media2 corosync[3690]: [TOTEM ] Automatically recovered ring
> > > 1
> > >
> > > Nov  6 02:33:17 media2 corosync[3690]: [TOTEM ] Incrementing problem counter
> > > for seqid 412386 iface 192.168.101.32 to [1 of 10]
> > >
> > > Nov  6 02:33:18 media2 corosync[3690]: [TOTEM ] ring 1 active with no faults
> > >
> > > Nov  6 02:33:19 media2 corosync[3690]: [TOTEM ] Incrementing problem counter
> > > for seqid 412388 iface 192.168.100.32 to [1 of 10]
> > >
> > > Nov  6 02:33:20 media2 corosync[3690]: [TOTEM ] ring 0 active with no faults
> > >
> > > Nov  6 02:33:35 media2 corosync[3690]: [TOTEM ] Automatically recovered ring
> > > 1
> > >
> > > Nov  6 02:33:36 media2 corosync[3690]: [TOTEM ] Incrementing problem counter
> > > for seqid 412430 iface 192.168.101.32 to [1 of 10]
> > >
> > > Nov  6 02:33:36 media2 corosync[3690]: [TOTEM ] Automatically recovered ring
> > > 1
> > >
> > > Nov  6 02:33:37 media2 corosync[3690]: [TOTEM ] Incrementing problem counter
> > > for seqid 412432 iface 192.168.101.32 to [1 of 10]
> > >
> > > Nov  6 02:33:38 media2 corosync[3690]: [TOTEM ] ring 1 active with no faults
> > >
> > > libteam mailing list
> > > [e-mail]
> > > https://lists.fedorahosted.org/mailman/listinfo/libteam
>
> -- 
> Mit freundlichen Grüßen
>
> Ingo Brand
> Dipl.-Ing.(BA) Informationstechnik
>
> Webbilling GmbH
> Geschäftsführer: Michael Reul, Richard Wissmeyer, Guido Ullrich
> Amtsgericht Hamburg HRB 86505
> Am Marienberg 2
> 64686 Lautern
> Tel: [phone]
> Fax: [phone]
> e-mail: [e-mail]

[Ingo Brand]

Am 06.11.2014 um 17:26 schrieb Jiri Pirko:
> Thu, Nov 06, 2014 at 05:18:18PM CET, [e-mail] wrote:
> > Am 06.11.2014 um 17:03 schrieb Jiri Pirko:
> > > Thu, Nov 06, 2014 at 04:26:36PM CET, [e-mail] wrote:
> > > > Hello,
> > > >
> > > > I am currently setting up a high available NFS server cluster with 2 nodes
> > > > using pacemaker + corosync + drbd + flashcache.
> > > > The two nodes (media1 and media2) have 4 1G copper nics each.
> > > > I formed 2 teams:
> > > > team0: LACP with em1 and em2 connected to a switch for NFS services
> > > > team1: roundrobin with em3 and em4 and MTU 9000 directly connected with
> > > > copper patch cables for drbd sync
> > > > During my testing everything was working as expected. Then I started to
> > > > tranfer about 20TB from an old server to the new cluster. The copy process
> > > > was fine for about 32 hours. Then corosync and drbd started to report errors
> > > > on the roundrobin team1 interface (see logs at the end of this email).
> > > > The drbd sync and corosync started to flap between states for several hours
> > > > on team1.
> > > > I checked the error counters with ifconfig and found a huge amount off TX
> > > > dropped packets on team1 of the drbd slave node media2.
> > > > There were NO other error counters raising but team1 on media2.
> > > > To make this a bit more clear:
> > > > team1 on media2 (the drbd slave) showed raising TX dropped packets
> > > > em3 and em4 on media2 (the physical interfaces of team1) did not show any
> > > >
> > > > raising error counters.
> > > > team1 on media1 (the drbd master) did not show any raising error counters.
> > > > em3 and em4 on media1 (the physical interfaces of team1) did not show any
> > > >
> > > > raising error counters.
> > > > I then thought maybe one of the two patch cables between the two machines was
> > > > not 100% ok. So I removed em4 from team1 on both machines:
> > > > "teamdctl team1 port remove em4"
> > > > The error counters stopped raising.
> > > > I thought: "Yeah! That cable must be broken!"
> > > > But just to make sure I really found the problem I re-enabled em4 on both
> > > > nodes:
> > > > "teamdctl team1 port add em4"
> > > > Ok, now both team1 interfaces had 2 links up again and the error counter
> > > > restarted to raise again.
> > > > I then stopped em3 on both nodes to force all traffic through em4:
> > > > "teamdctl team1 port remove em3"
> > > > And guess what? The error counters on team1 stopped raising!
> > > > So in short:
> > > > If I remove either one of the two physical interfaces from team1 everything
> > > > is working without any errors. As soon as I enable both physical interfaces
> > > > the error counters start to raise.
> > > > After these tests I rebooted the slave node (media2) and added both ports to
> > > > team1 again (Have you tried turning it off and on again...).
> > > > After doing this, I now see TX dropped packets on the not yet rebooted media1
> > > > team1 interface.
> > > > Why do I only see these errors on the team1 interface and not on em3 and em4?
> > > >
> > > > Currently the copy process of 20TB is still running. I think that if I reboot
> > > > media1 everything will work as expected again for some time because it worked
> > > > during my initial tests.
> > > > But I do think that I hit a bug in the teaming driver and the raising error
> > > > counters will come back.
> > > >
> > > > Could anybody help?
>
> Hmm, I went over the results, do not see anything suspicious.
> Are you able to run systemtap scripts on the machine?

I can install systemtap and run some scripts. Please provide me some 
more information.
> > > Interesting. Could you do "teamdctl team1 s d" when the issue happens
> > > and post me the result?
> >
> > I just re-enabled port em4 on both machines and the issue happens again. Here
> > are your requested results:
> > media1 ~]# teamdctl team1 s d
> > {
> >     "ports": {
> >         "em3": {
> >             "ifinfo": {
> >                 "dev_addr": "b8:2a:72:d5:6c:e6",
> >                 "dev_addr_len": 6,
> >                 "ifindex": 4,
> >                 "ifname": "em3"
> >             },
> >             "link": {
> >                 "duplex": "full",
> >                 "speed": 1000,
> >                 "up": true
> >             },
> >             "link_watches": {
> >                 "list": {
> >                     "link_watch_0": {
> >                         "delay_down": 0,
> >                         "delay_up": 0,
> >                         "name": "ethtool",
> >                         "up": true
> >                     }
> >                 },
> >                 "up": true
> >             }
> >         },
> >         "em4": {
> >             "ifinfo": {
> >                 "dev_addr": "b8:2a:72:d5:6c:e6",
> >                 "dev_addr_len": 6,
> >                 "ifindex": 5,
> >                 "ifname": "em4"
> >             },
> >             "link": {
> >                 "duplex": "full",
> >                 "speed": 1000,
> >                 "up": true
> >             },
> >             "link_watches": {
> >                 "list": {
> >                     "link_watch_0": {
> >                         "delay_down": 0,
> >                         "delay_up": 0,
> >                         "name": "ethtool",
> >                         "up": true
> >                     }
> >                 },
> >                 "up": true
> >             }
> >         }
> >     },
> >     "setup": {
> >         "daemonized": false,
> >         "dbus_enabled": true,
> >         "debug_level": 0,
> >         "kernel_team_mode_name": "roundrobin",
> >         "pid": 3107,
> >         "pid_file": "/var/run/teamd/team1.pid",
> >         "runner_name": "roundrobin",
> >         "zmq_enabled": false
> >     },
> >     "team_device": {
> >         "ifinfo": {
> >             "dev_addr": "b8:2a:72:d5:6c:e6",
> >             "dev_addr_len": 6,
> >             "ifindex": 7,
> >             "ifname": "team1"
> >         }
> >     }
> > }
> >
> > media2 ~]# teamdctl team1 s d
> > {
> >     "ports": {
> >         "em3": {
> >             "ifinfo": {
> >                 "dev_addr": "b8:2a:72:d5:97:18",
> >                 "dev_addr_len": 6,
> >                 "ifindex": 4,
> >                 "ifname": "em3"
> >             },
> >             "link": {
> >                 "duplex": "full",
> >                 "speed": 1000,
> >                 "up": true
> >             },
> >             "link_watches": {
> >                 "list": {
> >                     "link_watch_0": {
> >                         "delay_down": 0,
> >                         "delay_up": 0,
> >                         "name": "ethtool",
> >                         "up": true
> >                     }
> >                 },
> >                 "up": true
> >             }
> >         },
> >         "em4": {
> >             "ifinfo": {
> >                 "dev_addr": "b8:2a:72:d5:97:18",
> >                 "dev_addr_len": 6,
> >                 "ifindex": 5,
> >                 "ifname": "em4"
> >             },
> >             "link": {
> >                 "duplex": "full",
> >                 "speed": 1000,
> >                 "up": true
> >             },
> >             "link_watches": {
> >                 "list": {
> >                     "link_watch_0": {
> >                         "delay_down": 0,
> >                         "delay_up": 0,
> >                         "name": "ethtool",
> >                         "up": true
> >                     }
> >                 },
> >                 "up": true
> >             }
> >         }
> >     },
> >     "setup": {
> >         "daemonized": false,
> >         "dbus_enabled": true,
> >         "debug_level": 0,
> >         "kernel_team_mode_name": "roundrobin",
> >         "pid": 3084,
> >         "pid_file": "/var/run/teamd/team1.pid",
> >         "runner_name": "roundrobin",
> >         "zmq_enabled": false
> >     },
> >     "team_device": {
> >         "ifinfo": {
> >             "dev_addr": "b8:2a:72:d5:97:18",
> >             "dev_addr_len": 6,
> >             "ifindex": 7,
> >             "ifname": "team1"
> >         }
> >     }
> > }
> > > Also you can try random runner instead of roundrobin. The resulting
> > > behaviour is almost the same. Maybe the issue won't happen there.
> >
> > I do believe that if I "restart" the roundrobin runner on media1, the problem
> > is gone anyway. So for the moment I'll keep roundrobin just to keep the
> > failure environment in place in case you need some more data.
> > > Also, can you please post me output of:
> > > ethtool -i em3
> >
> > media1 ~]# ethtool -i em3
> > driver: tg3
> > version: 3.136
> > firmware-version: FFV7.8.16 bc 5720-v1.32
> > bus-info: 0000:02:00.0
> > supports-statistics: yes
> > supports-test: yes
> > supports-eeprom-access: yes
> > supports-register-dump: yes
> > supports-priv-flags: no
> >
> > media2 ~]# ethtool -i em3
> > driver: tg3
> > version: 3.136
> > firmware-version: FFV7.8.16 bc 5720-v1.32
> > bus-info: 0000:02:00.0
> > supports-statistics: yes
> > supports-test: yes
> > supports-eeprom-access: yes
> > supports-register-dump: yes
> > supports-priv-flags: no
> > > ethtool em3
> >
> > media1 ~]# ethtool em3
> > Settings for em3:
> >         Supported ports: [ TP ]
> >         Supported link modes:   10baseT/Half 10baseT/Full
> >                                 100baseT/Half 100baseT/Full
> >                                 1000baseT/Half 1000baseT/Full
> >         Supported pause frame use: No
> >         Supports auto-negotiation: Yes
> >         Advertised link modes:  10baseT/Half 10baseT/Full
> >                                 100baseT/Half 100baseT/Full
> >                                 1000baseT/Half 1000baseT/Full
> >         Advertised pause frame use: Symmetric
> >         Advertised auto-negotiation: Yes
> >         Link partner advertised link modes:  10baseT/Half 10baseT/Full
> >                                              100baseT/Half 100baseT/Full
> >                                              1000baseT/Half 1000baseT/Full
> >         Link partner advertised pause frame use: Symmetric
> >         Link partner advertised auto-negotiation: Yes
> >         Speed: 1000Mb/s
> >         Duplex: Full
> >         Port: Twisted Pair
> >         PHYAD: 1
> >         Transceiver: internal
> >         Auto-negotiation: on
> >         MDI-X: off
> >         Supports Wake-on: g
> >         Wake-on: d
> >         Current message level: 0x000000ff (255)
> >                                drv probe link timer ifdown ifup rx_err tx_err
> >         Link detected: yes
> >
> > media2 ~]# ethtool em3
> > Settings for em3:
> >         Supported ports: [ TP ]
> >         Supported link modes:   10baseT/Half 10baseT/Full
> >                                 100baseT/Half 100baseT/Full
> >                                 1000baseT/Half 1000baseT/Full
> >         Supported pause frame use: No
> >         Supports auto-negotiation: Yes
> >         Advertised link modes:  10baseT/Half 10baseT/Full
> >                                 100baseT/Half 100baseT/Full
> >                                 1000baseT/Half 1000baseT/Full
> >         Advertised pause frame use: Symmetric
> >         Advertised auto-negotiation: Yes
> >         Link partner advertised link modes:  10baseT/Half 10baseT/Full
> >                                              100baseT/Half 100baseT/Full
> >                                              1000baseT/Half 1000baseT/Full
> >         Link partner advertised pause frame use: Symmetric
> >         Link partner advertised auto-negotiation: Yes
> >         Speed: 1000Mb/s
> >         Duplex: Full
> >         Port: Twisted Pair
> >         PHYAD: 1
> >         Transceiver: internal
> >         Auto-negotiation: on
> >         MDI-X: on
> >         Supports Wake-on: g
> >         Wake-on: d
> >         Current message level: 0x000000ff (255)
> >                                drv probe link timer ifdown ifup rx_err tx_err
> >         Link detected: yes
> > > ethtool -S em3
> >
> > media1 ~]# ethtool -S em3
> > NIC statistics:
> >      rx_octets: 26853394715
> >      rx_fragments: 0
> >      rx_ucast_packets: 227879267
> >      rx_mcast_packets: 84
> >      rx_bcast_packets: 6
> >      rx_fcs_errors: 0
> >      rx_align_errors: 0
> >      rx_xon_pause_rcvd: 0
> >      rx_xoff_pause_rcvd: 0
> >      rx_mac_ctrl_rcvd: 0
> >      rx_xoff_entered: 0
> >      rx_frame_too_long_errors: 0
> >      rx_jabbers: 0
> >      rx_undersize_packets: 0
> >      rx_in_length_errors: 0
> >      rx_out_length_errors: 0
> >      rx_64_or_less_octet_packets: 0
> >      rx_65_to_127_octet_packets: 0
> >      rx_128_to_255_octet_packets: 0
> >      rx_256_to_511_octet_packets: 0
> >      rx_512_to_1023_octet_packets: 0
> >      rx_1024_to_1522_octet_packets: 0
> >      rx_1523_to_2047_octet_packets: 0
> >      rx_2048_to_4095_octet_packets: 0
> >      rx_4096_to_8191_octet_packets: 0
> >      rx_8192_to_9022_octet_packets: 0
> >      tx_octets: 1179249531811
> >      tx_collisions: 0
> >      tx_xon_sent: 0
> >      tx_xoff_sent: 0
> >      tx_flow_control: 0
> >      tx_mac_errors: 0
> >      tx_single_collisions: 0
> >      tx_mult_collisions: 0
> >      tx_deferred: 0
> >      tx_excessive_collisions: 0
> >      tx_late_collisions: 0
> >      tx_collide_2times: 0
> >      tx_collide_3times: 0
> >      tx_collide_4times: 0
> >      tx_collide_5times: 0
> >      tx_collide_6times: 0
> >      tx_collide_7times: 0
> >      tx_collide_8times: 0
> >      tx_collide_9times: 0
> >      tx_collide_10times: 0
> >      tx_collide_11times: 0
> >      tx_collide_12times: 0
> >      tx_collide_13times: 0
> >      tx_collide_14times: 0
> >      tx_collide_15times: 0
> >      tx_ucast_packets: 254815741
> >      tx_mcast_packets: 8
> >      tx_bcast_packets: 612
> >      tx_carrier_sense_errors: 0
> >      tx_discards: 0
> >      tx_errors: 0
> >      dma_writeq_full: 0
> >      dma_write_prioq_full: 0
> >      rxbds_empty: 0
> >      rx_discards: 0
> >      rx_errors: 0
> >      rx_threshold_hit: 0
> >      dma_readq_full: 0
> >      dma_read_prioq_full: 0
> >      tx_comp_queue_full: 0
> >      ring_set_send_prod_index: 0
> >      ring_status_update: 0
> >      nic_irqs: 0
> >      nic_avoided_irqs: 0
> >      nic_tx_threshold_hit: 0
> >      mbuf_lwm_thresh_hit: 0
> >
> > media2 ~]# ethtool -S em3
> > NIC statistics:
> >      rx_octets: 782620145025
> >      rx_fragments: 0
> >      rx_ucast_packets: 173893766
> >      rx_mcast_packets: 0
> >      rx_bcast_packets: 0
> >      rx_fcs_errors: 0
> >      rx_align_errors: 0
> >      rx_xon_pause_rcvd: 0
> >      rx_xoff_pause_rcvd: 0
> >      rx_mac_ctrl_rcvd: 0
> >      rx_xoff_entered: 0
> >      rx_frame_too_long_errors: 0
> >      rx_jabbers: 0
> >      rx_undersize_packets: 0
> >      rx_in_length_errors: 0
> >      rx_out_length_errors: 0
> >      rx_64_or_less_octet_packets: 0
> >      rx_65_to_127_octet_packets: 0
> >      rx_128_to_255_octet_packets: 0
> >      rx_256_to_511_octet_packets: 0
> >      rx_512_to_1023_octet_packets: 0
> >      rx_1024_to_1522_octet_packets: 0
> >      rx_1523_to_2047_octet_packets: 0
> >      rx_2048_to_4095_octet_packets: 0
> >      rx_4096_to_8191_octet_packets: 0
> >      rx_8192_to_9022_octet_packets: 0
> >      tx_octets: 17210360858
> >      tx_collisions: 0
> >      tx_xon_sent: 0
> >      tx_xoff_sent: 0
> >      tx_flow_control: 0
> >      tx_mac_errors: 0
> >      tx_single_collisions: 0
> >      tx_mult_collisions: 0
> >      tx_deferred: 0
> >      tx_excessive_collisions: 0
> >      tx_late_collisions: 0
> >      tx_collide_2times: 0
> >      tx_collide_3times: 0
> >      tx_collide_4times: 0
> >      tx_collide_5times: 0
> >      tx_collide_6times: 0
> >      tx_collide_7times: 0
> >      tx_collide_8times: 0
> >      tx_collide_9times: 0
> >      tx_collide_10times: 0
> >      tx_collide_11times: 0
> >      tx_collide_12times: 0
> >      tx_collide_13times: 0
> >      tx_collide_14times: 0
> >      tx_collide_15times: 0
> >      tx_ucast_packets: 147864937
> >      tx_mcast_packets: 23
> >      tx_bcast_packets: 2
> >      tx_carrier_sense_errors: 0
> >      tx_discards: 0
> >      tx_errors: 0
> >      dma_writeq_full: 0
> >      dma_write_prioq_full: 0
> >      rxbds_empty: 0
> >      rx_discards: 0
> >      rx_errors: 0
> >      rx_threshold_hit: 0
> >      dma_readq_full: 0
> >      dma_read_prioq_full: 0
> >      tx_comp_queue_full: 0
> >      ring_set_send_prod_index: 0
> >      ring_status_update: 0
> >      nic_irqs: 0
> >      nic_avoided_irqs: 0
> >      nic_tx_threshold_hit: 0
> >      mbuf_lwm_thresh_hit: 0
> > > ethtool -i em4
> >
> > media1 ~]# ethtool -i em4
> > driver: tg3
> > version: 3.136
> > firmware-version: FFV7.8.16 bc 5720-v1.32
> > bus-info: 0000:02:00.1
> > supports-statistics: yes
> > supports-test: yes
> > supports-eeprom-access: yes
> > supports-register-dump: yes
> > supports-priv-flags: no
> >
> > media2 ~]# ethtool -i em4
> > driver: tg3
> > version: 3.136
> > firmware-version: FFV7.8.16 bc 5720-v1.32
> > bus-info: 0000:02:00.1
> > supports-statistics: yes
> > supports-test: yes
> > supports-eeprom-access: yes
> > supports-register-dump: yes
> > supports-priv-flags: no
> > > ethtool em4
> >
> > media1 ~]# ethtool em4
> > Settings for em4:
> >         Supported ports: [ TP ]
> >         Supported link modes:   10baseT/Half 10baseT/Full
> >                                 100baseT/Half 100baseT/Full
> >                                 1000baseT/Half 1000baseT/Full
> >         Supported pause frame use: No
> >         Supports auto-negotiation: Yes
> >         Advertised link modes:  10baseT/Half 10baseT/Full
> >                                 100baseT/Half 100baseT/Full
> >                                 1000baseT/Half 1000baseT/Full
> >         Advertised pause frame use: Symmetric
> >         Advertised auto-negotiation: Yes
> >         Link partner advertised link modes:  10baseT/Half 10baseT/Full
> >                                              100baseT/Half 100baseT/Full
> >                                              1000baseT/Half 1000baseT/Full
> >         Link partner advertised pause frame use: Symmetric
> >         Link partner advertised auto-negotiation: Yes
> >         Speed: 1000Mb/s
> >         Duplex: Full
> >         Port: Twisted Pair
> >         PHYAD: 2
> >         Transceiver: internal
> >         Auto-negotiation: on
> >         MDI-X: off
> >         Supports Wake-on: g
> >         Wake-on: d
> >         Current message level: 0x000000ff (255)
> >                                drv probe link timer ifdown ifup rx_err tx_err
> >         Link detected: yes
> >
> > media2 ~]# ethtool em4
> > Settings for em4:
> >         Supported ports: [ TP ]
> >         Supported link modes:   10baseT/Half 10baseT/Full
> >                                 100baseT/Half 100baseT/Full
> >                                 1000baseT/Half 1000baseT/Full
> >         Supported pause frame use: No
> >         Supports auto-negotiation: Yes
> >         Advertised link modes:  10baseT/Half 10baseT/Full
> >                                 100baseT/Half 100baseT/Full
> >                                 1000baseT/Half 1000baseT/Full
> >         Advertised pause frame use: Symmetric
> >         Advertised auto-negotiation: Yes
> >         Link partner advertised link modes:  10baseT/Half 10baseT/Full
> >                                              100baseT/Half 100baseT/Full
> >                                              1000baseT/Half 1000baseT/Full
> >         Link partner advertised pause frame use: Symmetric
> >         Link partner advertised auto-negotiation: Yes
> >         Speed: 1000Mb/s
> >         Duplex: Full
> >         Port: Twisted Pair
> >         PHYAD: 2
> >         Transceiver: internal
> >         Auto-negotiation: on
> >         MDI-X: on
> >         Supports Wake-on: g
> >         Wake-on: d
> >         Current message level: 0x000000ff (255)
> >                                drv probe link timer ifdown ifup rx_err tx_err
> >         Link detected: yes
> > > ethtool -S em4
> >
> > media1 ~]# ethtool -S em4
> > NIC statistics:
> >      rx_octets: 246361
> >      rx_fragments: 0
> >      rx_ucast_packets: 912
> >      rx_mcast_packets: 8
> >      rx_bcast_packets: 0
> >      rx_fcs_errors: 0
> >      rx_align_errors: 0
> >      rx_xon_pause_rcvd: 0
> >      rx_xoff_pause_rcvd: 0
> >      rx_mac_ctrl_rcvd: 0
> >      rx_xoff_entered: 0
> >      rx_frame_too_long_errors: 0
> >      rx_jabbers: 0
> >      rx_undersize_packets: 0
> >      rx_in_length_errors: 0
> >      rx_out_length_errors: 0
> >      rx_64_or_less_octet_packets: 0
> >      rx_65_to_127_octet_packets: 0
> >      rx_128_to_255_octet_packets: 0
> >      rx_256_to_511_octet_packets: 0
> >      rx_512_to_1023_octet_packets: 0
> >      rx_1024_to_1522_octet_packets: 0
> >      rx_1523_to_2047_octet_packets: 0
> >      rx_2048_to_4095_octet_packets: 0
> >      rx_4096_to_8191_octet_packets: 0
> >      rx_8192_to_9022_octet_packets: 0
> >      tx_octets: 760
> >      tx_collisions: 0
> >      tx_xon_sent: 0
> >      tx_xoff_sent: 0
> >      tx_flow_control: 0
> >      tx_mac_errors: 0
> >      tx_single_collisions: 0
> >      tx_mult_collisions: 0
> >      tx_deferred: 0
> >      tx_excessive_collisions: 0
> >      tx_late_collisions: 0
> >      tx_collide_2times: 0
> >      tx_collide_3times: 0
> >      tx_collide_4times: 0
> >      tx_collide_5times: 0
> >      tx_collide_6times: 0
> >      tx_collide_7times: 0
> >      tx_collide_8times: 0
> >      tx_collide_9times: 0
> >      tx_collide_10times: 0
> >      tx_collide_11times: 0
> >      tx_collide_12times: 0
> >      tx_collide_13times: 0
> >      tx_collide_14times: 0
> >      tx_collide_15times: 0
> >      tx_ucast_packets: 0
> >      tx_mcast_packets: 8
> >      tx_bcast_packets: 0
> >      tx_carrier_sense_errors: 0
> >      tx_discards: 0
> >      tx_errors: 0
> >      dma_writeq_full: 0
> >      dma_write_prioq_full: 0
> >      rxbds_empty: 0
> >      rx_discards: 0
> >      rx_errors: 0
> >      rx_threshold_hit: 0
> >      dma_readq_full: 0
> >      dma_read_prioq_full: 0
> >      tx_comp_queue_full: 0
> >      ring_set_send_prod_index: 0
> >      ring_status_update: 0
> >      nic_irqs: 0
> >      nic_avoided_irqs: 0
> >      nic_tx_threshold_hit: 0
> >      mbuf_lwm_thresh_hit: 0
> >
> > media2 ~]# ethtool -S em4
> > NIC statistics:
> >      rx_octets: 760
> >      rx_fragments: 0
> >      rx_ucast_packets: 0
> >      rx_mcast_packets: 8
> >      rx_bcast_packets: 0
> >      rx_fcs_errors: 0
> >      rx_align_errors: 0
> >      rx_xon_pause_rcvd: 0
> >      rx_xoff_pause_rcvd: 0
> >      rx_mac_ctrl_rcvd: 0
> >      rx_xoff_entered: 0
> >      rx_frame_too_long_errors: 0
> >      rx_jabbers: 0
> >      rx_undersize_packets: 0
> >      rx_in_length_errors: 0
> >      rx_out_length_errors: 0
> >      rx_64_or_less_octet_packets: 0
> >      rx_65_to_127_octet_packets: 0
> >      rx_128_to_255_octet_packets: 0
> >      rx_256_to_511_octet_packets: 0
> >      rx_512_to_1023_octet_packets: 0
> >      rx_1024_to_1522_octet_packets: 0
> >      rx_1523_to_2047_octet_packets: 0
> >      rx_2048_to_4095_octet_packets: 0
> >      rx_4096_to_8191_octet_packets: 0
> >      rx_8192_to_9022_octet_packets: 0
> >      tx_octets: 247887
> >      tx_collisions: 0
> >      tx_xon_sent: 0
> >      tx_xoff_sent: 0
> >      tx_flow_control: 0
> >      tx_mac_errors: 0
> >      tx_single_collisions: 0
> >      tx_mult_collisions: 0
> >      tx_deferred: 0
> >      tx_excessive_collisions: 0
> >      tx_late_collisions: 0
> >      tx_collide_2times: 0
> >      tx_collide_3times: 0
> >      tx_collide_4times: 0
> >      tx_collide_5times: 0
> >      tx_collide_6times: 0
> >      tx_collide_7times: 0
> >      tx_collide_8times: 0
> >      tx_collide_9times: 0
> >      tx_collide_10times: 0
> >      tx_collide_11times: 0
> >      tx_collide_12times: 0
> >      tx_collide_13times: 0
> >      tx_collide_14times: 0
> >      tx_collide_15times: 0
> >      tx_ucast_packets: 927
> >      tx_mcast_packets: 8
> >      tx_bcast_packets: 0
> >      tx_carrier_sense_errors: 0
> >      tx_discards: 0
> >      tx_errors: 0
> >      dma_writeq_full: 0
> >      dma_write_prioq_full: 0
> >      rxbds_empty: 0
> >      rx_discards: 0
> >      rx_errors: 0
> >      rx_threshold_hit: 0
> >      dma_readq_full: 0
> >      dma_read_prioq_full: 0
> >      tx_comp_queue_full: 0
> >      ring_set_send_prod_index: 0
> >      ring_status_update: 0
> >      nic_irqs: 0
> >      nic_avoided_irqs: 0
> >      nic_tx_threshold_hit: 0
> >      mbuf_lwm_thresh_hit: 0
> > > Of course in time the problem happens.
> > > > Kind regards
> > > > Ingo
> > > >
> > > > This is the used network config of team1:
> > > > media1:
> > > > cat /etc/sysconfig/network-scripts/ifcfg-team1_slave_0
> > > > # Generated by parse-kickstart
> > > > NAME=team1 slave 0
> > > > TEAM_MASTER=team1
> > > > DEVICETYPE=TeamPort
> > > > DEVICE=em3
> > > > ONBOOT=yes
> > > > UUID=b7026c5b-e9cd-457c-93d2-a2799361ed90
> > > >
> > > > cat /etc/sysconfig/network-scripts/ifcfg-team1_slave_1
> > > > # Generated by parse-kickstart
> > > > NAME=team1 slave 1
> > > > TEAM_MASTER=team1
> > > > DEVICETYPE=TeamPort
> > > > DEVICE=em4
> > > > ONBOOT=yes
> > > > UUID=fbe5bc56-95f5-4f9e-a5ea-ab6b6f5e5b50
> > > >
> > > > cat /etc/sysconfig/network-scripts/ifcfg-team1
> > > > # Generated by parse-kickstart
> > > > UUID=4e0090cd-32d3-4c5b-be23-9f53083da4dd
> > > > NAME="Team connection team1"
> > > > TEAM_CONFIG="{"runner": {"name": "roundrobin"}}"
> > > > GATEWAY=
> > > > IPV6_AUTOCONF=yes
> > > > BOOTPROTO=none
> > > > DEVICE=team1
> > > > MTU=9000
> > > > TYPE=Team
> > > > ONBOOT=yes
> > > > IPV6INIT=yes
> > > > DEVICETYPE=Team
> > > > IPADDR0=192.168.101.31
> > > > PREFIX0=24
> > > > DEFROUTE=yes
> > > > IPV4_FAILURE_FATAL=no
> > > > IPV6_DEFROUTE=yes
> > > > IPV6_PEERDNS=yes
> > > > IPV6_PEERROUTES=yes
> > > > IPV6_FAILURE_FATAL=no
> > > >
> > > > media2:
> > > > cat /etc/sysconfig/network-scripts/ifcfg-team1_slave_0
> > > > # Generated by parse-kickstart
> > > > NAME=team1 slave 0
> > > > TEAM_MASTER=team1
> > > > DEVICETYPE=TeamPort
> > > > DEVICE=em3
> > > > ONBOOT=yes
> > > > UUID=bfd8e77e-7eb8-47ea-ab7e-95286e073202
> > > >
> > > > cat /etc/sysconfig/network-scripts/ifcfg-team1_slave_1
> > > > # Generated by parse-kickstart
> > > > NAME=team1 slave 1
> > > > TEAM_MASTER=team1
> > > > DEVICETYPE=TeamPort
> > > > DEVICE=em4
> > > > ONBOOT=yes
> > > > UUID=93c5dfd8-3b81-4d1e-8ba9-c46f8daceb1a
> > > >
> > > > cat /etc/sysconfig/network-scripts/ifcfg-team1
> > > > # Generated by parse-kickstart
> > > > UUID=79c843db-996c-41e4-9ee5-2a8f3da244ed
> > > > NAME="Team connection team1"
> > > > TEAM_CONFIG="{"runner": {"name": "roundrobin"}}"
> > > > GATEWAY=
> > > > IPV6_AUTOCONF=yes
> > > > BOOTPROTO=none
> > > > DEVICE=team1
> > > > MTU=9000
> > > > TYPE=Team
> > > > ONBOOT=yes
> > > > IPV6INIT=yes
> > > > DEVICETYPE=Team
> > > > IPADDR0=192.168.101.32
> > > > PREFIX0=24
> > > > DEFROUTE=yes
> > > > IPV4_FAILURE_FATAL=no
> > > > IPV6_DEFROUTE=yes
> > > > IPV6_PEERDNS=yes
> > > > IPV6_PEERROUTES=yes
> > > > IPV6_FAILURE_FATAL=no

[Jiri Pirko]

Thu, Nov 06, 2014 at 05:44:37PM CET, [e-mail] wrote:
> Am 06.11.2014 um 17:26 schrieb Jiri Pirko:
> > Thu, Nov 06, 2014 at 05:18:18PM CET, [e-mail] wrote:
> > > Am 06.11.2014 um 17:03 schrieb Jiri Pirko:
> > > > Thu, Nov 06, 2014 at 04:26:36PM CET, [e-mail] wrote:
> > > > > Hello,
> > > > >
> > > > > I am currently setting up a high available NFS server cluster with 2 nodes
> > > > > using pacemaker + corosync + drbd + flashcache.
> > > > > The two nodes (media1 and media2) have 4 1G copper nics each.
> > > > > I formed 2 teams:
> > > > > team0: LACP with em1 and em2 connected to a switch for NFS services
> > > > > team1: roundrobin with em3 and em4 and MTU 9000 directly connected with
> > > > > copper patch cables for drbd sync
> > > > > During my testing everything was working as expected. Then I started to
> > > > > tranfer about 20TB from an old server to the new cluster. The copy process
> > > > > was fine for about 32 hours. Then corosync and drbd started to report errors
> > > > > on the roundrobin team1 interface (see logs at the end of this email).
> > > > > The drbd sync and corosync started to flap between states for several hours
> > > > > on team1.
> > > > > I checked the error counters with ifconfig and found a huge amount off TX
> > > > > dropped packets on team1 of the drbd slave node media2.
> > > > > There were NO other error counters raising but team1 on media2.
> > > > > To make this a bit more clear:
> > > > > team1 on media2 (the drbd slave) showed raising TX dropped packets
> > > > > em3 and em4 on media2 (the physical interfaces of team1) did not show any
> > > > >
> > > > > raising error counters.
> > > > > team1 on media1 (the drbd master) did not show any raising error counters.
> > > > > em3 and em4 on media1 (the physical interfaces of team1) did not show any
> > > > >
> > > > > raising error counters.
> > > > > I then thought maybe one of the two patch cables between the two machines was
> > > > > not 100% ok. So I removed em4 from team1 on both machines:
> > > > > "teamdctl team1 port remove em4"
> > > > > The error counters stopped raising.
> > > > > I thought: "Yeah! That cable must be broken!"
> > > > > But just to make sure I really found the problem I re-enabled em4 on both
> > > > > nodes:
> > > > > "teamdctl team1 port add em4"
> > > > > Ok, now both team1 interfaces had 2 links up again and the error counter
> > > > > restarted to raise again.
> > > > > I then stopped em3 on both nodes to force all traffic through em4:
> > > > > "teamdctl team1 port remove em3"
> > > > > And guess what? The error counters on team1 stopped raising!
> > > > > So in short:
> > > > > If I remove either one of the two physical interfaces from team1 everything
> > > > > is working without any errors. As soon as I enable both physical interfaces
> > > > > the error counters start to raise.
> > > > > After these tests I rebooted the slave node (media2) and added both ports to
> > > > > team1 again (Have you tried turning it off and on again...).
> > > > > After doing this, I now see TX dropped packets on the not yet rebooted media1
> > > > > team1 interface.
> > > > > Why do I only see these errors on the team1 interface and not on em3 and em4?
> > > > >
> > > > > Currently the copy process of 20TB is still running. I think that if I reboot
> > > > > media1 everything will work as expected again for some time because it worked
> > > > > during my initial tests.
> > > > > But I do think that I hit a bug in the teaming driver and the raising error
> > > > > counters will come back.
> > > > >
> > > > > Could anybody help?
> >
> > Hmm, I went over the results, do not see anything suspicious.
> > Are you able to run systemtap scripts on the machine?
>
> I can install systemtap and run some scripts. Please provide me some more
> information.

I'll think about it over night and provide you some test script
tomorrow.
> > > > Interesting. Could you do "teamdctl team1 s d" when the issue happens
> > > > and post me the result?
> > >
> > > I just re-enabled port em4 on both machines and the issue happens again. Here
> > > are your requested results:
> > > media1 ~]# teamdctl team1 s d
> > > {
> > >    "ports": {
> > >        "em3": {
> > >            "ifinfo": {
> > >                "dev_addr": "b8:2a:72:d5:6c:e6",
> > >                "dev_addr_len": 6,
> > >                "ifindex": 4,
> > >                "ifname": "em3"
> > >            },
> > >            "link": {
> > >                "duplex": "full",
> > >                "speed": 1000,
> > >                "up": true
> > >            },
> > >            "link_watches": {
> > >                "list": {
> > >                    "link_watch_0": {
> > >                        "delay_down": 0,
> > >                        "delay_up": 0,
> > >                        "name": "ethtool",
> > >                        "up": true
> > >                    }
> > >                },
> > >                "up": true
> > >            }
> > >        },
> > >        "em4": {
> > >            "ifinfo": {
> > >                "dev_addr": "b8:2a:72:d5:6c:e6",
> > >                "dev_addr_len": 6,
> > >                "ifindex": 5,
> > >                "ifname": "em4"
> > >            },
> > >            "link": {
> > >                "duplex": "full",
> > >                "speed": 1000,
> > >                "up": true
> > >            },
> > >            "link_watches": {
> > >                "list": {
> > >                    "link_watch_0": {
> > >                        "delay_down": 0,
> > >                        "delay_up": 0,
> > >                        "name": "ethtool",
> > >                        "up": true
> > >                    }
> > >                },
> > >                "up": true
> > >            }
> > >        }
> > >    },
> > >    "setup": {
> > >        "daemonized": false,
> > >        "dbus_enabled": true,
> > >        "debug_level": 0,
> > >        "kernel_team_mode_name": "roundrobin",
> > >        "pid": 3107,
> > >        "pid_file": "/var/run/teamd/team1.pid",
> > >        "runner_name": "roundrobin",
> > >        "zmq_enabled": false
> > >    },
> > >    "team_device": {
> > >        "ifinfo": {
> > >            "dev_addr": "b8:2a:72:d5:6c:e6",
> > >            "dev_addr_len": 6,
> > >            "ifindex": 7,
> > >            "ifname": "team1"
> > >        }
> > >    }
> > > }
> > >
> > > media2 ~]# teamdctl team1 s d
> > > {
> > >    "ports": {
> > >        "em3": {
> > >            "ifinfo": {
> > >                "dev_addr": "b8:2a:72:d5:97:18",
> > >                "dev_addr_len": 6,
> > >                "ifindex": 4,
> > >                "ifname": "em3"
> > >            },
> > >            "link": {
> > >                "duplex": "full",
> > >                "speed": 1000,
> > >                "up": true
> > >            },
> > >            "link_watches": {
> > >                "list": {
> > >                    "link_watch_0": {
> > >                        "delay_down": 0,
> > >                        "delay_up": 0,
> > >                        "name": "ethtool",
> > >                        "up": true
> > >                    }
> > >                },
> > >                "up": true
> > >            }
> > >        },
> > >        "em4": {
> > >            "ifinfo": {
> > >                "dev_addr": "b8:2a:72:d5:97:18",
> > >                "dev_addr_len": 6,
> > >                "ifindex": 5,
> > >                "ifname": "em4"
> > >            },
> > >            "link": {
> > >                "duplex": "full",
> > >                "speed": 1000,
> > >                "up": true
> > >            },
> > >            "link_watches": {
> > >                "list": {
> > >                    "link_watch_0": {
> > >                        "delay_down": 0,
> > >                        "delay_up": 0,
> > >                        "name": "ethtool",
> > >                        "up": true
> > >                    }
> > >                },
> > >                "up": true
> > >            }
> > >        }
> > >    },
> > >    "setup": {
> > >        "daemonized": false,
> > >        "dbus_enabled": true,
> > >        "debug_level": 0,
> > >        "kernel_team_mode_name": "roundrobin",
> > >        "pid": 3084,
> > >        "pid_file": "/var/run/teamd/team1.pid",
> > >        "runner_name": "roundrobin",
> > >        "zmq_enabled": false
> > >    },
> > >    "team_device": {
> > >        "ifinfo": {
> > >            "dev_addr": "b8:2a:72:d5:97:18",
> > >            "dev_addr_len": 6,
> > >            "ifindex": 7,
> > >            "ifname": "team1"
> > >        }
> > >    }
> > > }
> > > > Also you can try random runner instead of roundrobin. The resulting
> > > > behaviour is almost the same. Maybe the issue won't happen there.
> > >
> > > I do believe that if I "restart" the roundrobin runner on media1, the problem
> > > is gone anyway. So for the moment I'll keep roundrobin just to keep the
> > > failure environment in place in case you need some more data.
> > > > Also, can you please post me output of:
> > > > ethtool -i em3
> > >
> > > media1 ~]# ethtool -i em3
> > > driver: tg3
> > > version: 3.136
> > > firmware-version: FFV7.8.16 bc 5720-v1.32
> > > bus-info: 0000:02:00.0
> > > supports-statistics: yes
> > > supports-test: yes
> > > supports-eeprom-access: yes
> > > supports-register-dump: yes
> > > supports-priv-flags: no
> > >
> > > media2 ~]# ethtool -i em3
> > > driver: tg3
> > > version: 3.136
> > > firmware-version: FFV7.8.16 bc 5720-v1.32
> > > bus-info: 0000:02:00.0
> > > supports-statistics: yes
> > > supports-test: yes
> > > supports-eeprom-access: yes
> > > supports-register-dump: yes
> > > supports-priv-flags: no
> > > > ethtool em3
> > >
> > > media1 ~]# ethtool em3
> > > Settings for em3:
> > >        Supported ports: [ TP ]
> > >        Supported link modes:   10baseT/Half 10baseT/Full
> > >                                100baseT/Half 100baseT/Full
> > >                                1000baseT/Half 1000baseT/Full
> > >        Supported pause frame use: No
> > >        Supports auto-negotiation: Yes
> > >        Advertised link modes:  10baseT/Half 10baseT/Full
> > >                                100baseT/Half 100baseT/Full
> > >                                1000baseT/Half 1000baseT/Full
> > >        Advertised pause frame use: Symmetric
> > >        Advertised auto-negotiation: Yes
> > >        Link partner advertised link modes:  10baseT/Half 10baseT/Full
> > >                                             100baseT/Half 100baseT/Full
> > >                                             1000baseT/Half 1000baseT/Full
> > >        Link partner advertised pause frame use: Symmetric
> > >        Link partner advertised auto-negotiation: Yes
> > >        Speed: 1000Mb/s
> > >        Duplex: Full
> > >        Port: Twisted Pair
> > >        PHYAD: 1
> > >        Transceiver: internal
> > >        Auto-negotiation: on
> > >        MDI-X: off
> > >        Supports Wake-on: g
> > >        Wake-on: d
> > >        Current message level: 0x000000ff (255)
> > >                               drv probe link timer ifdown ifup rx_err tx_err
> > >        Link detected: yes
> > >
> > > media2 ~]# ethtool em3
> > > Settings for em3:
> > >        Supported ports: [ TP ]
> > >        Supported link modes:   10baseT/Half 10baseT/Full
> > >                                100baseT/Half 100baseT/Full
> > >                                1000baseT/Half 1000baseT/Full
> > >        Supported pause frame use: No
> > >        Supports auto-negotiation: Yes
> > >        Advertised link modes:  10baseT/Half 10baseT/Full
> > >                                100baseT/Half 100baseT/Full
> > >                                1000baseT/Half 1000baseT/Full
> > >        Advertised pause frame use: Symmetric
> > >        Advertised auto-negotiation: Yes
> > >        Link partner advertised link modes:  10baseT/Half 10baseT/Full
> > >                                             100baseT/Half 100baseT/Full
> > >                                             1000baseT/Half 1000baseT/Full
> > >        Link partner advertised pause frame use: Symmetric
> > >        Link partner advertised auto-negotiation: Yes
> > >        Speed: 1000Mb/s
> > >        Duplex: Full
> > >        Port: Twisted Pair
> > >        PHYAD: 1
> > >        Transceiver: internal
> > >        Auto-negotiation: on
> > >        MDI-X: on
> > >        Supports Wake-on: g
> > >        Wake-on: d
> > >        Current message level: 0x000000ff (255)
> > >                               drv probe link timer ifdown ifup rx_err tx_err
> > >        Link detected: yes
> > > > ethtool -S em3
> > >
> > > media1 ~]# ethtool -S em3
> > > NIC statistics:
> > >     rx_octets: 26853394715
> > >     rx_fragments: 0
> > >     rx_ucast_packets: 227879267
> > >     rx_mcast_packets: 84
> > >     rx_bcast_packets: 6
> > >     rx_fcs_errors: 0
> > >     rx_align_errors: 0
> > >     rx_xon_pause_rcvd: 0
> > >     rx_xoff_pause_rcvd: 0
> > >     rx_mac_ctrl_rcvd: 0
> > >     rx_xoff_entered: 0
> > >     rx_frame_too_long_errors: 0
> > >     rx_jabbers: 0
> > >     rx_undersize_packets: 0
> > >     rx_in_length_errors: 0
> > >     rx_out_length_errors: 0
> > >     rx_64_or_less_octet_packets: 0
> > >     rx_65_to_127_octet_packets: 0
> > >     rx_128_to_255_octet_packets: 0
> > >     rx_256_to_511_octet_packets: 0
> > >     rx_512_to_1023_octet_packets: 0
> > >     rx_1024_to_1522_octet_packets: 0
> > >     rx_1523_to_2047_octet_packets: 0
> > >     rx_2048_to_4095_octet_packets: 0
> > >     rx_4096_to_8191_octet_packets: 0
> > >     rx_8192_to_9022_octet_packets: 0
> > >     tx_octets: 1179249531811
> > >     tx_collisions: 0
> > >     tx_xon_sent: 0
> > >     tx_xoff_sent: 0
> > >     tx_flow_control: 0
> > >     tx_mac_errors: 0
> > >     tx_single_collisions: 0
> > >     tx_mult_collisions: 0
> > >     tx_deferred: 0
> > >     tx_excessive_collisions: 0
> > >     tx_late_collisions: 0
> > >     tx_collide_2times: 0
> > >     tx_collide_3times: 0
> > >     tx_collide_4times: 0
> > >     tx_collide_5times: 0
> > >     tx_collide_6times: 0
> > >     tx_collide_7times: 0
> > >     tx_collide_8times: 0
> > >     tx_collide_9times: 0
> > >     tx_collide_10times: 0
> > >     tx_collide_11times: 0
> > >     tx_collide_12times: 0
> > >     tx_collide_13times: 0
> > >     tx_collide_14times: 0
> > >     tx_collide_15times: 0
> > >     tx_ucast_packets: 254815741
> > >     tx_mcast_packets: 8
> > >     tx_bcast_packets: 612
> > >     tx_carrier_sense_errors: 0
> > >     tx_discards: 0
> > >     tx_errors: 0
> > >     dma_writeq_full: 0
> > >     dma_write_prioq_full: 0
> > >     rxbds_empty: 0
> > >     rx_discards: 0
> > >     rx_errors: 0
> > >     rx_threshold_hit: 0
> > >     dma_readq_full: 0
> > >     dma_read_prioq_full: 0
> > >     tx_comp_queue_full: 0
> > >     ring_set_send_prod_index: 0
> > >     ring_status_update: 0
> > >     nic_irqs: 0
> > >     nic_avoided_irqs: 0
> > >     nic_tx_threshold_hit: 0
> > >     mbuf_lwm_thresh_hit: 0
> > >
> > > media2 ~]# ethtool -S em3
> > > NIC statistics:
> > >     rx_octets: 782620145025
> > >     rx_fragments: 0
> > >     rx_ucast_packets: 173893766
> > >     rx_mcast_packets: 0
> > >     rx_bcast_packets: 0
> > >     rx_fcs_errors: 0
> > >     rx_align_errors: 0
> > >     rx_xon_pause_rcvd: 0
> > >     rx_xoff_pause_rcvd: 0
> > >     rx_mac_ctrl_rcvd: 0
> > >     rx_xoff_entered: 0
> > >     rx_frame_too_long_errors: 0
> > >     rx_jabbers: 0
> > >     rx_undersize_packets: 0
> > >     rx_in_length_errors: 0
> > >     rx_out_length_errors: 0
> > >     rx_64_or_less_octet_packets: 0
> > >     rx_65_to_127_octet_packets: 0
> > >     rx_128_to_255_octet_packets: 0
> > >     rx_256_to_511_octet_packets: 0
> > >     rx_512_to_1023_octet_packets: 0
> > >     rx_1024_to_1522_octet_packets: 0
> > >     rx_1523_to_2047_octet_packets: 0
> > >     rx_2048_to_4095_octet_packets: 0
> > >     rx_4096_to_8191_octet_packets: 0
> > >     rx_8192_to_9022_octet_packets: 0
> > >     tx_octets: 17210360858
> > >     tx_collisions: 0
> > >     tx_xon_sent: 0
> > >     tx_xoff_sent: 0
> > >     tx_flow_control: 0
> > >     tx_mac_errors: 0
> > >     tx_single_collisions: 0
> > >     tx_mult_collisions: 0
> > >     tx_deferred: 0
> > >     tx_excessive_collisions: 0
> > >     tx_late_collisions: 0
> > >     tx_collide_2times: 0
> > >     tx_collide_3times: 0
> > >     tx_collide_4times: 0
> > >     tx_collide_5times: 0
> > >     tx_collide_6times: 0
> > >     tx_collide_7times: 0
> > >     tx_collide_8times: 0
> > >     tx_collide_9times: 0
> > >     tx_collide_10times: 0
> > >     tx_collide_11times: 0
> > >     tx_collide_12times: 0
> > >     tx_collide_13times: 0
> > >     tx_collide_14times: 0
> > >     tx_collide_15times: 0
> > >     tx_ucast_packets: 147864937
> > >     tx_mcast_packets: 23
> > >     tx_bcast_packets: 2
> > >     tx_carrier_sense_errors: 0
> > >     tx_discards: 0
> > >     tx_errors: 0
> > >     dma_writeq_full: 0
> > >     dma_write_prioq_full: 0
> > >     rxbds_empty: 0
> > >     rx_discards: 0
> > >     rx_errors: 0
> > >     rx_threshold_hit: 0
> > >     dma_readq_full: 0
> > >     dma_read_prioq_full: 0
> > >     tx_comp_queue_full: 0
> > >     ring_set_send_prod_index: 0
> > >     ring_status_update: 0
> > >     nic_irqs: 0
> > >     nic_avoided_irqs: 0
> > >     nic_tx_threshold_hit: 0
> > >     mbuf_lwm_thresh_hit: 0
> > > > ethtool -i em4
> > >
> > > media1 ~]# ethtool -i em4
> > > driver: tg3
> > > version: 3.136
> > > firmware-version: FFV7.8.16 bc 5720-v1.32
> > > bus-info: 0000:02:00.1
> > > supports-statistics: yes
> > > supports-test: yes
> > > supports-eeprom-access: yes
> > > supports-register-dump: yes
> > > supports-priv-flags: no
> > >
> > > media2 ~]# ethtool -i em4
> > > driver: tg3
> > > version: 3.136
> > > firmware-version: FFV7.8.16 bc 5720-v1.32
> > > bus-info: 0000:02:00.1
> > > supports-statistics: yes
> > > supports-test: yes
> > > supports-eeprom-access: yes
> > > supports-register-dump: yes
> > > supports-priv-flags: no
> > > > ethtool em4
> > >
> > > media1 ~]# ethtool em4
> > > Settings for em4:
> > >        Supported ports: [ TP ]
> > >        Supported link modes:   10baseT/Half 10baseT/Full
> > >                                100baseT/Half 100baseT/Full
> > >                                1000baseT/Half 1000baseT/Full
> > >        Supported pause frame use: No
> > >        Supports auto-negotiation: Yes
> > >        Advertised link modes:  10baseT/Half 10baseT/Full
> > >                                100baseT/Half 100baseT/Full
> > >                                1000baseT/Half 1000baseT/Full
> > >        Advertised pause frame use: Symmetric
> > >        Advertised auto-negotiation: Yes
> > >        Link partner advertised link modes:  10baseT/Half 10baseT/Full
> > >                                             100baseT/Half 100baseT/Full
> > >                                             1000baseT/Half 1000baseT/Full
> > >        Link partner advertised pause frame use: Symmetric
> > >        Link partner advertised auto-negotiation: Yes
> > >        Speed: 1000Mb/s
> > >        Duplex: Full
> > >        Port: Twisted Pair
> > >        PHYAD: 2
> > >        Transceiver: internal
> > >        Auto-negotiation: on
> > >        MDI-X: off
> > >        Supports Wake-on: g
> > >        Wake-on: d
> > >        Current message level: 0x000000ff (255)
> > >                               drv probe link timer ifdown ifup rx_err tx_err
> > >        Link detected: yes
> > >
> > > media2 ~]# ethtool em4
> > > Settings for em4:
> > >        Supported ports: [ TP ]
> > >        Supported link modes:   10baseT/Half 10baseT/Full
> > >                                100baseT/Half 100baseT/Full
> > >                                1000baseT/Half 1000baseT/Full
> > >        Supported pause frame use: No
> > >        Supports auto-negotiation: Yes
> > >        Advertised link modes:  10baseT/Half 10baseT/Full
> > >                                100baseT/Half 100baseT/Full
> > >                                1000baseT/Half 1000baseT/Full
> > >        Advertised pause frame use: Symmetric
> > >        Advertised auto-negotiation: Yes
> > >        Link partner advertised link modes:  10baseT/Half 10baseT/Full
> > >                                             100baseT/Half 100baseT/Full
> > >                                             1000baseT/Half 1000baseT/Full
> > >        Link partner advertised pause frame use: Symmetric
> > >        Link partner advertised auto-negotiation: Yes
> > >        Speed: 1000Mb/s
> > >        Duplex: Full
> > >        Port: Twisted Pair
> > >        PHYAD: 2
> > >        Transceiver: internal
> > >        Auto-negotiation: on
> > >        MDI-X: on
> > >        Supports Wake-on: g
> > >        Wake-on: d
> > >        Current message level: 0x000000ff (255)
> > >                               drv probe link timer ifdown ifup rx_err tx_err
> > >        Link detected: yes
> > > > ethtool -S em4
> > >
> > > media1 ~]# ethtool -S em4
> > > NIC statistics:
> > >     rx_octets: 246361
> > >     rx_fragments: 0
> > >     rx_ucast_packets: 912
> > >     rx_mcast_packets: 8
> > >     rx_bcast_packets: 0
> > >     rx_fcs_errors: 0
> > >     rx_align_errors: 0
> > >     rx_xon_pause_rcvd: 0
> > >     rx_xoff_pause_rcvd: 0
> > >     rx_mac_ctrl_rcvd: 0
> > >     rx_xoff_entered: 0
> > >     rx_frame_too_long_errors: 0
> > >     rx_jabbers: 0
> > >     rx_undersize_packets: 0
> > >     rx_in_length_errors: 0
> > >     rx_out_length_errors: 0
> > >     rx_64_or_less_octet_packets: 0
> > >     rx_65_to_127_octet_packets: 0
> > >     rx_128_to_255_octet_packets: 0
> > >     rx_256_to_511_octet_packets: 0
> > >     rx_512_to_1023_octet_packets: 0
> > >     rx_1024_to_1522_octet_packets: 0
> > >     rx_1523_to_2047_octet_packets: 0
> > >     rx_2048_to_4095_octet_packets: 0
> > >     rx_4096_to_8191_octet_packets: 0
> > >     rx_8192_to_9022_octet_packets: 0
> > >     tx_octets: 760
> > >     tx_collisions: 0
> > >     tx_xon_sent: 0
> > >     tx_xoff_sent: 0
> > >     tx_flow_control: 0
> > >     tx_mac_errors: 0
> > >     tx_single_collisions: 0
> > >     tx_mult_collisions: 0
> > >     tx_deferred: 0
> > >     tx_excessive_collisions: 0
> > >     tx_late_collisions: 0
> > >     tx_collide_2times: 0
> > >     tx_collide_3times: 0
> > >     tx_collide_4times: 0
> > >     tx_collide_5times: 0
> > >     tx_collide_6times: 0
> > >     tx_collide_7times: 0
> > >     tx_collide_8times: 0
> > >     tx_collide_9times: 0
> > >     tx_collide_10times: 0
> > >     tx_collide_11times: 0
> > >     tx_collide_12times: 0
> > >     tx_collide_13times: 0
> > >     tx_collide_14times: 0
> > >     tx_collide_15times: 0
> > >     tx_ucast_packets: 0
> > >     tx_mcast_packets: 8
> > >     tx_bcast_packets: 0
> > >     tx_carrier_sense_errors: 0
> > >     tx_discards: 0
> > >     tx_errors: 0
> > >     dma_writeq_full: 0
> > >     dma_write_prioq_full: 0
> > >     rxbds_empty: 0
> > >     rx_discards: 0
> > >     rx_errors: 0
> > >     rx_threshold_hit: 0
> > >     dma_readq_full: 0
> > >     dma_read_prioq_full: 0
> > >     tx_comp_queue_full: 0
> > >     ring_set_send_prod_index: 0
> > >     ring_status_update: 0
> > >     nic_irqs: 0
> > >     nic_avoided_irqs: 0
> > >     nic_tx_threshold_hit: 0
> > >     mbuf_lwm_thresh_hit: 0
> > >
> > > media2 ~]# ethtool -S em4
> > > NIC statistics:
> > >     rx_octets: 760
> > >     rx_fragments: 0
> > >     rx_ucast_packets: 0
> > >     rx_mcast_packets: 8
> > >     rx_bcast_packets: 0
> > >     rx_fcs_errors: 0
> > >     rx_align_errors: 0
> > >     rx_xon_pause_rcvd: 0
> > >     rx_xoff_pause_rcvd: 0
> > >     rx_mac_ctrl_rcvd: 0
> > >     rx_xoff_entered: 0
> > >     rx_frame_too_long_errors: 0
> > >     rx_jabbers: 0
> > >     rx_undersize_packets: 0
> > >     rx_in_length_errors: 0
> > >     rx_out_length_errors: 0
> > >     rx_64_or_less_octet_packets: 0
> > >     rx_65_to_127_octet_packets: 0
> > >     rx_128_to_255_octet_packets: 0
> > >     rx_256_to_511_octet_packets: 0
> > >     rx_512_to_1023_octet_packets: 0
> > >     rx_1024_to_1522_octet_packets: 0
> > >     rx_1523_to_2047_octet_packets: 0
> > >     rx_2048_to_4095_octet_packets: 0
> > >     rx_4096_to_8191_octet_packets: 0
> > >     rx_8192_to_9022_octet_packets: 0
> > >     tx_octets: 247887
> > >     tx_collisions: 0
> > >     tx_xon_sent: 0
> > >     tx_xoff_sent: 0
> > >     tx_flow_control: 0
> > >     tx_mac_errors: 0
> > >     tx_single_collisions: 0
> > >     tx_mult_collisions: 0
> > >     tx_deferred: 0
> > >     tx_excessive_collisions: 0
> > >     tx_late_collisions: 0
> > >     tx_collide_2times: 0
> > >     tx_collide_3times: 0
> > >     tx_collide_4times: 0
> > >     tx_collide_5times: 0
> > >     tx_collide_6times: 0
> > >     tx_collide_7times: 0
> > >     tx_collide_8times: 0
> > >     tx_collide_9times: 0
> > >     tx_collide_10times: 0
> > >     tx_collide_11times: 0
> > >     tx_collide_12times: 0
> > >     tx_collide_13times: 0
> > >     tx_collide_14times: 0
> > >     tx_collide_15times: 0
> > >     tx_ucast_packets: 927
> > >     tx_mcast_packets: 8
> > >     tx_bcast_packets: 0
> > >     tx_carrier_sense_errors: 0
> > >     tx_discards: 0
> > >     tx_errors: 0
> > >     dma_writeq_full: 0
> > >     dma_write_prioq_full: 0
> > >     rxbds_empty: 0
> > >     rx_discards: 0
> > >     rx_errors: 0
> > >     rx_threshold_hit: 0
> > >     dma_readq_full: 0
> > >     dma_read_prioq_full: 0
> > >     tx_comp_queue_full: 0
> > >     ring_set_send_prod_index: 0
> > >     ring_status_update: 0
> > >     nic_irqs: 0
> > >     nic_avoided_irqs: 0
> > >     nic_tx_threshold_hit: 0
> > >     mbuf_lwm_thresh_hit: 0
> > > > Of course in time the problem happens.
> > > > > Kind regards
> > > > > Ingo
> > > > >
> > > > > This is the used network config of team1:
> > > > > media1:
> > > > > cat /etc/sysconfig/network-scripts/ifcfg-team1_slave_0
> > > > > # Generated by parse-kickstart
> > > > > NAME=team1 slave 0
> > > > > TEAM_MASTER=team1
> > > > > DEVICETYPE=TeamPort
> > > > > DEVICE=em3
> > > > > ONBOOT=yes
> > > > > UUID=b7026c5b-e9cd-457c-93d2-a2799361ed90
> > > > >
> > > > > cat /etc/sysconfig/network-scripts/ifcfg-team1_slave_1
> > > > > # Generated by parse-kickstart
> > > > > NAME=team1 slave 1
> > > > > TEAM_MASTER=team1
> > > > > DEVICETYPE=TeamPort
> > > > > DEVICE=em4
> > > > > ONBOOT=yes
> > > > > UUID=fbe5bc56-95f5-4f9e-a5ea-ab6b6f5e5b50
> > > > >
> > > > > cat /etc/sysconfig/network-scripts/ifcfg-team1
> > > > > # Generated by parse-kickstart
> > > > > UUID=4e0090cd-32d3-4c5b-be23-9f53083da4dd
> > > > > NAME="Team connection team1"
> > > > > TEAM_CONFIG="{"runner": {"name": "roundrobin"}}"
> > > > > GATEWAY=
> > > > > IPV6_AUTOCONF=yes
> > > > > BOOTPROTO=none
> > > > > DEVICE=team1
> > > > > MTU=9000
> > > > > TYPE=Team
> > > > > ONBOOT=yes
> > > > > IPV6INIT=yes
> > > > > DEVICETYPE=Team
> > > > > IPADDR0=192.168.101.31
> > > > > PREFIX0=24
> > > > > DEFROUTE=yes
> > > > > IPV4_FAILURE_FATAL=no
> > > > > IPV6_DEFROUTE=yes
> > > > > IPV6_PEERDNS=yes
> > > > > IPV6_PEERROUTES=yes
> > > > > IPV6_FAILURE_FATAL=no
> > > > >
> > > > > media2:
> > > > > cat /etc/sysconfig/network-scripts/ifcfg-team1_slave_0
> > > > > # Generated by parse-kickstart
> > > > > NAME=team1 slave 0
> > > > > TEAM_MASTER=team1
> > > > > DEVICETYPE=TeamPort
> > > > > DEVICE=em3
> > > > > ONBOOT=yes
> > > > > UUID=bfd8e77e-7eb8-47ea-ab7e-95286e073202
> > > > >
> > > > > cat /etc/sysconfig/network-scripts/ifcfg-team1_slave_1
> > > > > # Generated by parse-kickstart
> > > > > NAME=team1 slave 1
> > > > > TEAM_MASTER=team1
> > > > > DEVICETYPE=TeamPort
> > > > > DEVICE=em4
> > > > > ONBOOT=yes
> > > > > UUID=93c5dfd8-3b81-4d1e-8ba9-c46f8daceb1a
> > > > >
> > > > > cat /etc/sysconfig/network-scripts/ifcfg-team1
> > > > > # Generated by parse-kickstart
> > > > > UUID=79c843db-996c-41e4-9ee5-2a8f3da244ed
> > > > > NAME="Team connection team1"
> > > > > TEAM_CONFIG="{"runner": {"name": "roundrobin"}}"
> > > > > GATEWAY=
> > > > > IPV6_AUTOCONF=yes
> > > > > BOOTPROTO=none
> > > > > DEVICE=team1
> > > > > MTU=9000
> > > > > TYPE=Team
> > > > > ONBOOT=yes
> > > > > IPV6INIT=yes
> > > > > DEVICETYPE=Team
> > > > > IPADDR0=192.168.101.32
> > > > > PREFIX0=24
> > > > > DEFROUTE=yes
> > > > > IPV4_FAILURE_FATAL=no
> > > > > IPV6_DEFROUTE=yes
> > > > > IPV6_PEERDNS=yes
> > > > > IPV6_PEERROUTES=yes
> > > > > IPV6_FAILURE_FATAL=no

[Liang Zhao]

> Thu, Nov 06, 2014 at 05:44:37PM CET, ingo.brand(a)webbilling.de wrote:
>
> I'll think about it over night and provide you some test script
> tomorrow.

Hi Jiri,

Just wonder if there is any updates on this issue? Currently I also have a machine having exactly same issue - both underline eth have NO Tx drop, but the teamed interface is having around 25% Tx drop.

It's a HP server running RHEL 7.2. I'm happy to provide more info if you are still interested in this issue.

Thanks,
 Liang

[Jiri Pirko]

Wed, Sep 07, 2016 at 01:52:22PM CEST, [e-mail] wrote:
> > Thu, Nov 06, 2014 at 05:44:37PM CET, ingo.brand(a)webbilling.de wrote:
> >
> > I'll think about it over night and provide you some test script
> > tomorrow.
>
> Hi Jiri,
>
> Just wonder if there is any updates on this issue? Currently I also have a machine having exactly same issue - both underline eth have NO Tx drop, but the teamed interface is having around 25% Tx drop.
>
> It's a HP server running RHEL 7.2. I'm happy to provide more info if you are still interested in this issue.

You should contact RH support.

But yeah, please provide more info.
> Thanks,
> Liang
> _______________________________________________
> libteam mailing list
> [e-mail]
> https://lists.fedorahosted.org/admin/lists/[e-mail]