Stange problems with CentOS 7 and roundrobin runner
by Ingo Brand
Hello,
I am currently setting up a high available NFS server cluster with 2
nodes using pacemaker + corosync + drbd + flashcache.
The two nodes (media1 and media2) have 4 1G copper nics each.
I formed 2 teams:
team0: LACP with em1 and em2 connected to a switch for NFS services
team1: roundrobin with em3 and em4 and MTU 9000 directly connected with
copper patch cables for drbd sync
During my testing everything was working as expected. Then I started to
tranfer about 20TB from an old server to the new cluster. The copy
process was fine for about 32 hours. Then corosync and drbd started to
report errors on the roundrobin team1 interface (see logs at the end of
this email).
The drbd sync and corosync started to flap between states for several
hours on team1.
I checked the error counters with ifconfig and found a huge amount off
TX dropped packets on team1 of the drbd slave node media2.
There were NO other error counters raising but team1 on media2.
To make this a bit more clear:
- team1 on media2 (the drbd slave) showed raising TX dropped packets
- em3 and em4 on media2 (the physical interfaces of team1) did not show
any raising error counters.
- team1 on media1 (the drbd master) did not show any raising error counters.
- em3 and em4 on media1 (the physical interfaces of team1) did not show
any raising error counters.
I then thought maybe one of the two patch cables between the two
machines was not 100% ok. So I removed em4 from team1 on both machines:
"teamdctl team1 port remove em4"
The error counters stopped raising.
I thought: "Yeah! That cable must be broken!"
But just to make sure I really found the problem I re-enabled em4 on
both nodes:
"teamdctl team1 port add em4"
Ok, now both team1 interfaces had 2 links up again and the error counter
restarted to raise again.
I then stopped em3 on both nodes to force all traffic through em4:
"teamdctl team1 port remove em3"
And guess what? The error counters on team1 stopped raising!
So in short:
If I remove either one of the two physical interfaces from team1
everything is working without any errors. As soon as I enable both
physical interfaces the error counters start to raise.
After these tests I rebooted the slave node (media2) and added both
ports to team1 again (Have you tried turning it off and on again...).
After doing this, I now see TX dropped packets on the not yet rebooted
media1 team1 interface.
Why do I only see these errors on the team1 interface and not on em3 and
em4?
Currently the copy process of 20TB is still running. I think that if I
reboot media1 everything will work as expected again for some time
because it worked during my initial tests.
But I do think that I hit a bug in the teaming driver and the raising
error counters will come back.
Could anybody help?
Kind regards
Ingo
This is the used network config of team1:
media1:
cat /etc/sysconfig/network-scripts/ifcfg-team1_slave_0
# Generated by parse-kickstart
NAME=team1 slave 0
TEAM_MASTER=team1
DEVICETYPE=TeamPort
DEVICE=em3
ONBOOT=yes
UUID=b7026c5b-e9cd-457c-93d2-a2799361ed90
cat /etc/sysconfig/network-scripts/ifcfg-team1_slave_1
# Generated by parse-kickstart
NAME=team1 slave 1
TEAM_MASTER=team1
DEVICETYPE=TeamPort
DEVICE=em4
ONBOOT=yes
UUID=fbe5bc56-95f5-4f9e-a5ea-ab6b6f5e5b50
cat /etc/sysconfig/network-scripts/ifcfg-team1
# Generated by parse-kickstart
UUID=4e0090cd-32d3-4c5b-be23-9f53083da4dd
NAME="Team connection team1"
TEAM_CONFIG="{\"runner\": {\"name\": \"roundrobin\"}}"
GATEWAY=
IPV6_AUTOCONF=yes
BOOTPROTO=none
DEVICE=team1
MTU=9000
TYPE=Team
ONBOOT=yes
IPV6INIT=yes
DEVICETYPE=Team
IPADDR0=192.168.101.31
PREFIX0=24
DEFROUTE=yes
IPV4_FAILURE_FATAL=no
IPV6_DEFROUTE=yes
IPV6_PEERDNS=yes
IPV6_PEERROUTES=yes
IPV6_FAILURE_FATAL=no
media2:
cat /etc/sysconfig/network-scripts/ifcfg-team1_slave_0
# Generated by parse-kickstart
NAME=team1 slave 0
TEAM_MASTER=team1
DEVICETYPE=TeamPort
DEVICE=em3
ONBOOT=yes
UUID=bfd8e77e-7eb8-47ea-ab7e-95286e073202
cat /etc/sysconfig/network-scripts/ifcfg-team1_slave_1
# Generated by parse-kickstart
NAME=team1 slave 1
TEAM_MASTER=team1
DEVICETYPE=TeamPort
DEVICE=em4
ONBOOT=yes
UUID=93c5dfd8-3b81-4d1e-8ba9-c46f8daceb1a
cat /etc/sysconfig/network-scripts/ifcfg-team1
# Generated by parse-kickstart
UUID=79c843db-996c-41e4-9ee5-2a8f3da244ed
NAME="Team connection team1"
TEAM_CONFIG="{\"runner\": {\"name\": \"roundrobin\"}}"
GATEWAY=
IPV6_AUTOCONF=yes
BOOTPROTO=none
DEVICE=team1
MTU=9000
TYPE=Team
ONBOOT=yes
IPV6INIT=yes
DEVICETYPE=Team
IPADDR0=192.168.101.32
PREFIX0=24
DEFROUTE=yes
IPV4_FAILURE_FATAL=no
IPV6_DEFROUTE=yes
IPV6_PEERDNS=yes
IPV6_PEERROUTES=yes
IPV6_FAILURE_FATAL=no
Here are some system logs from both machines:
============================================================
media1:
============================================================
Nov 6 02:31:27 media1 corosync[3657]: [TOTEM ] Incrementing problem
counter for seqid 412041 iface 192.168.100.31 to [1 of 10]
Nov 6 02:31:29 media1 corosync[3657]: [TOTEM ] ring 0 active with no faults
Nov 6 02:31:29 media1 corosync[3657]: [TOTEM ] Incrementing problem
counter for seqid 412047 iface 192.168.100.31 to [1 of 10]
Nov 6 02:31:31 media1 corosync[3657]: [TOTEM ] ring 0 active with no faults
Nov 6 02:32:51 media1 corosync[3657]: [TOTEM ] Incrementing problem
counter for seqid 412315 iface 192.168.101.31 to [1 of 10]
Nov 6 02:32:53 media1 corosync[3657]: [TOTEM ] ring 1 active with no faults
Nov 6 02:32:53 media1 corosync[3657]: [TOTEM ] Incrementing problem
counter for seqid 412319 iface 192.168.101.31 to [1 of 10]
Nov 6 02:32:54 media1 corosync[3657]: [TOTEM ] Incrementing problem
counter for seqid 412321 iface 192.168.101.31 to [2 of 10]
Nov 6 02:32:55 media1 corosync[3657]: [TOTEM ] Decrementing problem
counter for iface 192.168.101.31 to [1 of 10]
Nov 6 02:32:56 media1 corosync[3657]: [TOTEM ] Incrementing problem
counter for seqid 412325 iface 192.168.101.31 to [2 of 10]
Nov 6 02:32:57 media1 corosync[3657]: [TOTEM ] Decrementing problem
counter for iface 192.168.101.31 to [1 of 10]
Nov 6 02:32:58 media1 corosync[3657]: [TOTEM ] Incrementing problem
counter for seqid 412327 iface 192.168.101.31 to [2 of 10]
Nov 6 02:32:59 media1 kernel: drbd r0: PingAck did not arrive in time.
Nov 6 02:32:59 media1 kernel: drbd r0: peer( Secondary -> Unknown )
conn( Connected -> NetworkFailure ) pdsk( UpToDate -> DUnknown )
Nov 6 02:32:59 media1 kernel: block drbd0: new current UUID
0C0308FA4F94F811:59E43E5FD7B26A4F:1963884285B385E6:1962884285B385E6
Nov 6 02:32:59 media1 kernel: drbd r0: asender terminated
Nov 6 02:32:59 media1 kernel: drbd r0: Terminating drbd_a_r0
Nov 6 02:32:59 media1 kernel: drbd r0: Connection closed
Nov 6 02:32:59 media1 kernel: drbd r0: conn( NetworkFailure ->
Unconnected )
Nov 6 02:32:59 media1 kernel: drbd r0: receiver terminated
Nov 6 02:32:59 media1 kernel: drbd r0: Restarting receiver thread
Nov 6 02:32:59 media1 kernel: drbd r0: receiver (re)started
Nov 6 02:32:59 media1 kernel: drbd r0: conn( Unconnected -> WFConnection )
Nov 6 02:32:59 media1 corosync[3657]: [TOTEM ] Decrementing problem
counter for iface 192.168.101.31 to [1 of 10]
Nov 6 02:32:59 media1 corosync[3657]: [TOTEM ] Incrementing problem
counter for seqid 412329 iface 192.168.101.31 to [2 of 10]
Nov 6 02:33:01 media1 corosync[3657]: [TOTEM ] Incrementing problem
counter for seqid 412333 iface 192.168.101.31 to [3 of 10]
Nov 6 02:33:01 media1 corosync[3657]: [TOTEM ] Decrementing problem
counter for iface 192.168.101.31 to [2 of 10]
Nov 6 02:33:01 media1 crmd[4960]: notice: do_state_transition: State
transition S_IDLE -> S_POLICY_ENGINE [ input=I_PE_CALC
cause=C_FSA_INTERNAL origin=abort_transition_graph ]
Nov 6 02:33:01 media1 pengine[4959]: notice: unpack_config: On loss of
CCM Quorum: Ignore
Nov 6 02:33:01 media1 pengine[4959]: warning: unpack_rsc_op: Processing
failed op monitor for res_nfsserver_mediafiles on media1: unknown error
(1)
Nov 6 02:33:01 media1 pengine[4959]: warning: unpack_rsc_op: Processing
failed op monitor for res_nfsserver_mediafiles on media2: unknown error
(1)
Nov 6 02:33:01 media1 pengine[4959]: notice: process_pe_message:
Calculated Transition 144: /var/lib/pacemaker/pengine/pe-input-227.bz2
Nov 6 02:33:01 media1 crmd[4960]: notice: run_graph: Transition 144
(Complete=0, Pending=0, Fired=0, Skipped=0, Incomplete=0,
Source=/var/lib/pacemaker/pengine/pe-input-227.bz2): Complete
Nov 6 02:33:01 media1 crmd[4960]: notice: do_state_transition: State
transition S_TRANSITION_ENGINE -> S_IDLE [ input=I_TE_SUCCESS
cause=C_FSA_INTERNAL origin=notify_crmd ]
Nov 6 02:33:01 media1 kernel: drbd r0: Handshake successful: Agreed
network protocol version 101
Nov 6 02:33:01 media1 kernel: drbd r0: Agreed to support TRIM on
protocol level
Nov 6 02:33:01 media1 kernel: drbd r0: Peer authenticated using 20
bytes HMAC
Nov 6 02:33:01 media1 kernel: drbd r0: conn( WFConnection ->
WFReportParams )
Nov 6 02:33:01 media1 kernel: drbd r0: Starting asender thread (from
drbd_r_r0 [5999])
Nov 6 02:33:02 media1 corosync[3657]: [TOTEM ] Incrementing problem
counter for seqid 412337 iface 192.168.101.31 to [3 of 10]
Nov 6 02:33:02 media1 kernel: block drbd0: drbd_sync_handshake:
Nov 6 02:33:02 media1 kernel: block drbd0: self
0C0308FA4F94F811:59E43E5FD7B26A4F:1963884285B385E6:1962884285B385E6
bits:192101 flags:0
Nov 6 02:33:02 media1 kernel: block drbd0: peer
59E43E5FD7B26A4E:0000000000000000:1963884285B385E6:1962884285B385E6
bits:0 flags:0
Nov 6 02:33:02 media1 kernel: block drbd0: uuid_compare()=1 by rule 70
Nov 6 02:33:02 media1 kernel: block drbd0: peer( Unknown -> Secondary )
conn( WFReportParams -> WFBitMapS ) pdsk( DUnknown -> Consistent )
Nov 6 02:33:02 media1 kernel: block drbd0: send bitmap stats
[Bytes(packets)]: plain 0(0), RLE 185(1), total 185; compression: 100.0%
Nov 6 02:33:02 media1 corosync[3657]: [TOTEM ] Incrementing problem
counter for seqid 412339 iface 192.168.101.31 to [4 of 10]
Nov 6 02:33:03 media1 kernel: block drbd0: receive bitmap stats
[Bytes(packets)]: plain 0(0), RLE 185(1), total 185; compression: 100.0%
Nov 6 02:33:03 media1 kernel: block drbd0: helper command:
/sbin/drbdadm before-resync-source minor-0
Nov 6 02:33:03 media1 kernel: block drbd0: helper command:
/sbin/drbdadm before-resync-source minor-0 exit code 0 (0x0)
Nov 6 02:33:03 media1 kernel: block drbd0: conn( WFBitMapS ->
SyncSource ) pdsk( Consistent -> Inconsistent )
Nov 6 02:33:03 media1 kernel: block drbd0: Began resync as SyncSource
(will sync 919244 KB [229811 bits set]).
Nov 6 02:33:03 media1 kernel: block drbd0: updated sync UUID
0C0308FA4F94F811:59E53E5FD7B26A4F:59E43E5FD7B26A4F:1963884285B385E6
Nov 6 02:33:03 media1 corosync[3657]: [TOTEM ] Decrementing problem
counter for iface 192.168.101.31 to [3 of 10]
Nov 6 02:33:03 media1 corosync[3657]: [TOTEM ] Incrementing problem
counter for seqid 412343 iface 192.168.101.31 to [4 of 10]
Nov 6 02:33:04 media1 corosync[3657]: [TOTEM ] Incrementing problem
counter for seqid 412345 iface 192.168.101.31 to [5 of 10]
Nov 6 02:33:05 media1 corosync[3657]: [TOTEM ] Incrementing problem
counter for seqid 412347 iface 192.168.101.31 to [6 of 10]
Nov 6 02:33:05 media1 corosync[3657]: [TOTEM ] Decrementing problem
counter for iface 192.168.101.31 to [5 of 10]
Nov 6 02:33:05 media1 corosync[3657]: [TOTEM ] Incrementing problem
counter for seqid 412349 iface 192.168.101.31 to [6 of 10]
Nov 6 02:33:06 media1 corosync[3657]: [TOTEM ] Incrementing problem
counter for seqid 412351 iface 192.168.101.31 to [7 of 10]
Nov 6 02:33:07 media1 corosync[3657]: [TOTEM ] Incrementing problem
counter for seqid 412355 iface 192.168.101.31 to [8 of 10]
Nov 6 02:33:07 media1 corosync[3657]: [TOTEM ] Decrementing problem
counter for iface 192.168.101.31 to [7 of 10]
Nov 6 02:33:07 media1 corosync[3657]: [TOTEM ] Incrementing problem
counter for seqid 412357 iface 192.168.101.31 to [8 of 10]
Nov 6 02:33:08 media1 corosync[3657]: [TOTEM ] Incrementing problem
counter for seqid 412361 iface 192.168.101.31 to [9 of 10]
Nov 6 02:33:09 media1 corosync[3657]: [TOTEM ] Incrementing problem
counter for seqid 412363 iface 192.168.101.31 to [10 of 10]
Nov 6 02:33:09 media1 corosync[3657]: [TOTEM ] Marking seqid 412363
ringid 1 interface 192.168.101.31 FAULTY
Nov 6 02:33:11 media1 crmd[4960]: notice: do_state_transition: State
transition S_IDLE -> S_POLICY_ENGINE [ input=I_PE_CALC
cause=C_FSA_INTERNAL origin=abort_transition_graph ]
Nov 6 02:33:11 media1 pengine[4959]: notice: unpack_config: On loss of
CCM Quorum: Ignore
Nov 6 02:33:11 media1 pengine[4959]: warning: unpack_rsc_op: Processing
failed op monitor for res_nfsserver_mediafiles on media1: unknown error
(1)
Nov 6 02:33:11 media1 pengine[4959]: warning: unpack_rsc_op: Processing
failed op monitor for res_nfsserver_mediafiles on media2: unknown error
(1)
Nov 6 02:33:11 media1 pengine[4959]: notice: process_pe_message:
Calculated Transition 145: /var/lib/pacemaker/pengine/pe-input-228.bz2
Nov 6 02:33:11 media1 crmd[4960]: notice: run_graph: Transition 145
(Complete=0, Pending=0, Fired=0, Skipped=0, Incomplete=0,
Source=/var/lib/pacemaker/pengine/pe-input-228.bz2): Complete
Nov 6 02:33:11 media1 crmd[4960]: notice: do_state_transition: State
transition S_TRANSITION_ENGINE -> S_IDLE [ input=I_TE_SUCCESS
cause=C_FSA_INTERNAL origin=notify_crmd ]
Nov 6 02:33:17 media1 corosync[3657]: [TOTEM ] Automatically recovered
ring 1
Nov 6 02:33:18 media1 corosync[3657]: [TOTEM ] Incrementing problem
counter for seqid 412387 iface 192.168.101.31 to [1 of 10]
Nov 6 02:33:20 media1 corosync[3657]: [TOTEM ] Incrementing problem
counter for seqid 412389 iface 192.168.101.31 to [2 of 10]
Nov 6 02:33:20 media1 corosync[3657]: [TOTEM ] Incrementing problem
counter for seqid 412391 iface 192.168.101.31 to [3 of 10]
Nov 6 02:33:21 media1 corosync[3657]: [TOTEM ] Incrementing problem
counter for seqid 412393 iface 192.168.101.31 to [4 of 10]
Nov 6 02:33:22 media1 corosync[3657]: [TOTEM ] Incrementing problem
counter for seqid 412397 iface 192.168.101.31 to [5 of 10]
Nov 6 02:33:22 media1 corosync[3657]: [TOTEM ] Incrementing problem
counter for seqid 412399 iface 192.168.101.31 to [6 of 10]
Nov 6 02:33:24 media1 corosync[3657]: [TOTEM ] Incrementing problem
counter for seqid 412403 iface 192.168.101.31 to [7 of 10]
Nov 6 02:33:31 media1 corosync[3657]: [TOTEM ] Incrementing problem
counter for seqid 412421 iface 192.168.101.31 to [8 of 10]
Nov 6 02:33:32 media1 corosync[3657]: [TOTEM ] Incrementing problem
counter for seqid 412423 iface 192.168.101.31 to [9 of 10]
Nov 6 02:33:34 media1 corosync[3657]: [TOTEM ] Incrementing problem
counter for seqid 412429 iface 192.168.101.31 to [10 of 10]
Nov 6 02:33:34 media1 corosync[3657]: [TOTEM ] Marking seqid 412429
ringid 1 interface 192.168.101.31 FAULTY
Nov 6 02:33:36 media1 corosync[3657]: [TOTEM ] Automatically recovered
ring 1
Nov 6 02:33:39 media1 corosync[3657]: [TOTEM ] Incrementing problem
counter for seqid 412437 iface 192.168.101.31 to [1 of 10]
============================================================
media2:
============================================================
Nov 6 02:32:59 media2 kernel: drbd r0: sock was shut down by peer
Nov 6 02:32:59 media2 kernel: drbd r0: peer( Primary -> Unknown ) conn(
Connected -> BrokenPipe ) pdsk( UpToDate -> DUnknown )
Nov 6 02:32:59 media2 kernel: drbd r0: short read (expected size 16)
Nov 6 02:32:59 media2 kernel: drbd r0: asender terminated
Nov 6 02:32:59 media2 kernel: drbd r0: Terminating drbd_a_r0
Nov 6 02:32:59 media2 kernel: drbd r0: Connection closed
Nov 6 02:32:59 media2 kernel: drbd r0: conn( BrokenPipe -> Unconnected
)
Nov 6 02:32:59 media2 kernel: drbd r0: receiver terminated
Nov 6 02:32:59 media2 kernel: drbd r0: Restarting receiver thread
Nov 6 02:32:59 media2 kernel: drbd r0: receiver (re)started
Nov 6 02:32:59 media2 kernel: drbd r0: conn( Unconnected ->
WFConnection )
Nov 6 02:33:01 media2 attrd[4940]: notice: attrd_trigger_update:
Sending flush op to all hosts for: master-res_drbd_mediafiles (1000)
Nov 6 02:33:01 media2 attrd[4940]: notice: attrd_perform_update: Sent
update 16: master-res_drbd_mediafiles=1000
Nov 6 02:33:01 media2 kernel: drbd r0: Handshake successful: Agreed
network protocol version 101
Nov 6 02:33:01 media2 kernel: drbd r0: Agreed to support TRIM on
protocol level
Nov 6 02:33:01 media2 kernel: drbd r0: Peer authenticated using 20
bytes HMAC
Nov 6 02:33:01 media2 kernel: drbd r0: conn( WFConnection ->
WFReportParams )
Nov 6 02:33:01 media2 kernel: drbd r0: Starting asender thread (from
drbd_r_r0 [6367])
Nov 6 02:33:01 media2 kernel: block drbd0: drbd_sync_handshake:
Nov 6 02:33:01 media2 kernel: block drbd0: self
59E43E5FD7B26A4E:0000000000000000:1963884285B385E6:1962884285B385E6
bits:0 flags:0
Nov 6 02:33:01 media2 kernel: block drbd0: peer
0C0308FA4F94F811:59E43E5FD7B26A4F:1963884285B385E6:1962884285B385E6
bits:192101 flags:0
Nov 6 02:33:01 media2 kernel: block drbd0: uuid_compare()=-1 by rule 50
Nov 6 02:33:01 media2 kernel: block drbd0: peer( Unknown -> Primary )
conn( WFReportParams -> WFBitMapT ) disk( UpToDate -> Outdated ) pdsk(
DUnknown -> UpToDate )
Nov 6 02:33:02 media2 kernel: block drbd0: receive bitmap stats
[Bytes(packets)]: plain 0(0), RLE 185(1), total 185; compression: 100.0%
Nov 6 02:33:03 media2 kernel: block drbd0: send bitmap stats
[Bytes(packets)]: plain 0(0), RLE 185(1), total 185; compression: 100.0%
Nov 6 02:33:03 media2 kernel: block drbd0: conn( WFBitMapT ->
WFSyncUUID )
Nov 6 02:33:03 media2 kernel: block drbd0: updated sync uuid
59E53E5FD7B26A4E:0000000000000000:1963884285B385E6:1962884285B385E6
Nov 6 02:33:03 media2 kernel: block drbd0: helper command:
/sbin/drbdadm before-resync-target minor-0
Nov 6 02:33:03 media2 kernel: block drbd0: helper command:
/sbin/drbdadm before-resync-target minor-0 exit code 0 (0x0)
Nov 6 02:33:03 media2 kernel: block drbd0: conn( WFSyncUUID ->
SyncTarget ) disk( Outdated -> Inconsistent )
Nov 6 02:33:03 media2 kernel: block drbd0: Began resync as SyncTarget
(will sync 919244 KB [229811 bits set]).
Nov 6 02:33:10 media2 corosync[3690]: [TOTEM ] Incrementing problem
counter for seqid 412364 iface 192.168.101.32 to [1 of 10]
Nov 6 02:33:10 media2 corosync[3690]: [TOTEM ] Automatically recovered
ring 1
Nov 6 02:33:10 media2 corosync[3690]: [TOTEM ] Incrementing problem
counter for seqid 412366 iface 192.168.101.32 to [1 of 10]
Nov 6 02:33:11 media2 corosync[3690]: [TOTEM ] Automatically recovered
ring 1
Nov 6 02:33:11 media2 attrd[4940]: notice: attrd_trigger_update:
Sending flush op to all hosts for: master-res_drbd_mediafiles (10)
Nov 6 02:33:11 media2 attrd[4940]: notice: attrd_perform_update: Sent
update 18: master-res_drbd_mediafiles=10
Nov 6 02:33:11 media2 corosync[3690]: [TOTEM ] Incrementing problem
counter for seqid 412368 iface 192.168.101.32 to [1 of 10]
Nov 6 02:33:12 media2 corosync[3690]: [TOTEM ] ring 1 active with no
faults
Nov 6 02:33:12 media2 corosync[3690]: [TOTEM ] Incrementing problem
counter for seqid 412370 iface 192.168.101.32 to [1 of 10]
Nov 6 02:33:12 media2 corosync[3690]: [TOTEM ] Automatically recovered
ring 1
Nov 6 02:33:12 media2 corosync[3690]: [TOTEM ] Incrementing problem
counter for seqid 412372 iface 192.168.101.32 to [1 of 10]
Nov 6 02:33:13 media2 corosync[3690]: [TOTEM ] Incrementing problem
counter for seqid 412374 iface 192.168.101.32 to [2 of 10]
Nov 6 02:33:14 media2 corosync[3690]: [TOTEM ] Decrementing problem
counter for iface 192.168.101.32 to [1 of 10]
Nov 6 02:33:14 media2 corosync[3690]: [TOTEM ] Incrementing problem
counter for seqid 412376 iface 192.168.101.32 to [2 of 10]
Nov 6 02:33:15 media2 corosync[3690]: [TOTEM ] Incrementing problem
counter for seqid 412378 iface 192.168.101.32 to [3 of 10]
Nov 6 02:33:15 media2 corosync[3690]: [TOTEM ] Automatically recovered
ring 1
Nov 6 02:33:15 media2 corosync[3690]: [TOTEM ] Incrementing problem
counter for seqid 412380 iface 192.168.101.32 to [1 of 10]
Nov 6 02:33:16 media2 corosync[3690]: [TOTEM ] ring 1 active with no
faults
Nov 6 02:33:16 media2 corosync[3690]: [TOTEM ] Incrementing problem
counter for seqid 412382 iface 192.168.101.32 to [1 of 10]
Nov 6 02:33:16 media2 corosync[3690]: [TOTEM ] Automatically recovered
ring 1
Nov 6 02:33:17 media2 corosync[3690]: [TOTEM ] Incrementing problem
counter for seqid 412384 iface 192.168.101.32 to [1 of 10]
Nov 6 02:33:17 media2 corosync[3690]: [TOTEM ] Automatically recovered
ring 1
Nov 6 02:33:17 media2 corosync[3690]: [TOTEM ] Incrementing problem
counter for seqid 412386 iface 192.168.101.32 to [1 of 10]
Nov 6 02:33:18 media2 corosync[3690]: [TOTEM ] ring 1 active with no
faults
Nov 6 02:33:19 media2 corosync[3690]: [TOTEM ] Incrementing problem
counter for seqid 412388 iface 192.168.100.32 to [1 of 10]
Nov 6 02:33:20 media2 corosync[3690]: [TOTEM ] ring 0 active with no
faults
Nov 6 02:33:35 media2 corosync[3690]: [TOTEM ] Automatically recovered
ring 1
Nov 6 02:33:36 media2 corosync[3690]: [TOTEM ] Incrementing problem
counter for seqid 412430 iface 192.168.101.32 to [1 of 10]
Nov 6 02:33:36 media2 corosync[3690]: [TOTEM ] Automatically recovered
ring 1
Nov 6 02:33:37 media2 corosync[3690]: [TOTEM ] Incrementing problem
counter for seqid 412432 iface 192.168.101.32 to [1 of 10]
Nov 6 02:33:38 media2 corosync[3690]: [TOTEM ] ring 1 active with no
faults
7 years
- 3
- 7
[PATCH] Fix teamd memory corruption issues seen by missing port unlink in ifinfo_destroy()
by Samudrala, Sridhar
This fixes teamd coredumps similar to these
systemd-coredump[11692]: Process 11345 (teamd) of user 0 dumped core.
Stack trace of thread 11345:
0 0x00007f7e6e1078d7 raise (libc.so.6)
1 0x00007f7e6e10953a abort (libc.so.6)
2 0x00007f7e6e14ada3 __libc_message (libc.so.6)
3 0x00007f7e6e1569f5 __libc_free (libc.so.6)
4 0x00007f7e6ed2779d link_free_data (libnl-route-3.so.200)
5 0x00007f7e6eafe6f0 nl_object_free (libnl-3.so.200)
6 0x00007f7e6f5a0df2 obj_input_newlink (libteam.so.5)
7 0x00007f7e6eafb0fe parse_cb (libnl-3.so.200)
8 0x00007f7e6ed2933c link_msg_parser (libnl-route-3.so.200)
9 0x00007f7e6eaf84c9 nl_cache_parse (libnl-3.so.200)
10 0x00007f7e6eafc99b nl_msg_parse (libnl-3.so.200)
11 0x00007f7e6f5a0fea ifinfo_event_handler (libteam.so.5)
12 0x00007f7e6eafdfcf nl_recvmsgs_report (libnl-3.so.200)
13 0x00007f7e6eafe339 nl_recvmsgs (libnl-3.so.200)
14 0x00007f7e6f59d8d0 cli_sock_event_handler (libteam.so.5)
15 0x00007f7e6f59e476 team_handle_events (libteam.so.5)
16 0x00000000004055d5 teamd_run_loop_run (teamd)
17 0x0000000000406a90 main (teamd)
18 0x00007f7e6e0f2fe0 __libc_start_main (libc.so.6)
19 0x000000000040755a _start (teamd)
Signed-off-by: Sridhar Samudrala <sridhar.samudrala(a)intel.com>
---
libteam/ifinfo.c | 2 ++
libteam/ports.c | 8 +++++++-
libteam/team_private.h | 1 +
3 files changed, 10 insertions(+), 1 deletion(-)
diff --git a/libteam/ifinfo.c b/libteam/ifinfo.c
index afe32da..fe370e3 100644
--- a/libteam/ifinfo.c
+++ b/libteam/ifinfo.c
@@ -213,6 +213,8 @@ static struct team_ifinfo *ifinfo_find_create(struct team_handle *th,
static void ifinfo_destroy(struct team_ifinfo *ifinfo)
{
+ if (ifinfo->linked && ifinfo->port)
+ port_unlink(ifinfo->port);
list_del(&ifinfo->list);
free(ifinfo);
}
diff --git a/libteam/ports.c b/libteam/ports.c
index 52d0c9a..9ebf30f 100644
--- a/libteam/ports.c
+++ b/libteam/ports.c
@@ -83,7 +83,8 @@ static struct team_port *port_create(struct team_handle *th,
static void port_destroy(struct team_handle *th,
struct team_port *port)
{
- ifinfo_unlink(port->ifinfo);
+ if (port->ifinfo)
+ ifinfo_unlink(port->ifinfo);
list_del(&port->list);
free(port);
}
@@ -226,6 +227,11 @@ void port_list_free(struct team_handle *th)
flush_port_list(th);
}
+void port_unlink(struct team_port *port)
+{
+ port->ifinfo = NULL;
+}
+
/* \endcond */
/**
diff --git a/libteam/team_private.h b/libteam/team_private.h
index 801f107..a07632f 100644
--- a/libteam/team_private.h
+++ b/libteam/team_private.h
@@ -105,6 +105,7 @@ int get_port_list_handler(struct nl_msg *msg, void *arg);
int port_list_alloc(struct team_handle *th);
int port_list_init(struct team_handle *th);
void port_list_free(struct team_handle *th);
+void port_unlink(struct team_port *port);
int ifinfo_event_handler(struct nl_msg *msg, void *arg);
int ifinfo_list_alloc(struct team_handle *th);
int ifinfo_list_init(struct team_handle *th);
--
2.1.0
8 years, 2 months
- 2
- 2
RE: Problems with teaming ...
by Goran Slavić
Hello Jiri,
Are there any news regarding the teaming problem we at SOX had ?
Did you manage to find time to investigate it on RH7 or ask somebody
in RH for opinion ?
I am under some pressure to solve this problem quickly (even if it
means going back to classical bonding) :-(
Can you give me the advice what to do:
- Change the port aggregation from teaming to bonding.
- Try to upgrade/reinstall kernel team module (how ?!? - I know
Linux - but I don't know it that well to just-change-kernel-modules on the
fly :-) )
- Give you some other useful data that can help you find some other
solution.
I would highly appreciate any (and I really mean ANY) help, advice
or suggestion you can give us regarding this problem.
Best regards
G.Slavic
-----Original Message-----
From: Jiri Pirko [mailto:jiri@resnulli.us]
Sent: Monday, 08 June 2015 12:22
To: Goran Slavić
Subject: Re: Problems with teaming ...
Mon, Jun 08, 2015 at 12:10:03PM CEST, gslavic(a)sox.rs wrote:
>Not today (I am over my head is some other work) but I will contact you
>tomorrow/the day after tomorrow.
>
>What is your concern ? / What does this program do ? Problem is that any
>installation or testing on this server must be approved from "higher up"
>because this is very important server to us :-)
>
>If you think that the problem is with the switch that this server is
>connected to or some other upstream equipment - I can tell you that this is
>probably not the issue. If you see the graphs from my original post the
>switch has been balancing traffic to the server almost perfectly (5-10%
>plus/minus) and that same switch has 2 other LACP aggregated port groups
>where classical bonding works in communication with other servers.
I'm 99,9% sure that the problem is inside kernel team module. For me, on
upstream kernel the issue does not happen. Looks like this is limited to
rhel7/centos7.
I would test it on rhel7, unfortunatelly I'm no longer Red Hat employee
so I don't have access to rhel7 machine.
Will try ask someone from RH.
8 years, 3 months
- 2
- 1
Re[2]: Problems with teaming ...
by Goran Slavić
Hi Jiri,
Unfortunately, this is the output of the "teamnl team0 options" command.
[root@rtr ~]# teamnl team0 options
lb_port_stats (port:enp5s0f1) \00\00\00\00\00\00\00\00
queue_id (port:enp5s0f1) 0
priority (port:enp5s0f1) 100
user_linkup_enabled (port:enp5s0f1) false
user_linkup (port:enp5s0f1) true
enabled (port:enp5s0f1) true
lb_port_stats (port:enp5s0f0) \00\00\00\00\00\00\00\00
queue_id (port:enp5s0f0) 0
priority (port:enp5s0f0) 100
user_linkup_enabled (port:enp5s0f0) false
user_linkup (port:enp5s0f0) true
enabled (port:enp5s0f0) true
lb_port_stats (port:enp4s0f1) \00\00\00\00\00\00\00\00
queue_id (port:enp4s0f1) 0
priority (port:enp4s0f1) 100
user_linkup_enabled (port:enp4s0f1) false
user_linkup (port:enp4s0f1) true
enabled (port:enp4s0f1) true
lb_port_stats (port:enp4s0f0) \00\00\00\00\00\00\00\00
queue_id (port:enp4s0f0) 0
priority (port:enp4s0f0) 100
user_linkup_enabled (port:enp4s0f0) false
user_linkup (port:enp4s0f0) true
enabled (port:enp4s0f0) true
lb_stats_refresh_interval 0
lb_hash_stats (arridx:255) \00\00\00\00\00\00\00\00
lb_hash_stats (arridx:254) \00\00\00\00\00\00\00\00
lb_hash_stats (arridx:253) \00\00\00\00\00\00\00\00
lb_hash_stats (arridx:252) \00\00\00\00\00\00\00\00
lb_hash_stats (arridx:251) \00\00\00\00\00\00\00\00
lb_hash_stats (arridx:250) \00\00\00\00\00\00\00\00
lb_hash_stats (arridx:249) \00\00\00\00\00\00\00\00
lb_hash_stats (arridx:248) \00\00\00\00\00\00\00\00
lb_hash_stats (arridx:247) \00\00\00\00\00\00\00\00
lb_hash_stats (arridx:246) \00\00\00\00\00\00\00\00
lb_hash_stats (arridx:245) \00\00\00\00\00\00\00\00
lb_hash_stats (arridx:244) \00\00\00\00\00\00\00\00
lb_hash_stats (arridx:243) \00\00\00\00\00\00\00\00
lb_hash_stats (arridx:242) \00\00\00\00\00\00\00\00
lb_hash_stats (arridx:241) \00\00\00\00\00\00\00\00
lb_hash_stats (arridx:240) \00\00\00\00\00\00\00\00
lb_hash_stats (arridx:239) \00\00\00\00\00\00\00\00
lb_hash_stats (arridx:238) \00\00\00\00\00\00\00\00
lb_hash_stats (arridx:237) \00\00\00\00\00\00\00\00
lb_hash_stats (arridx:236) \00\00\00\00\00\00\00\00
lb_hash_stats (arridx:235) \00\00\00\00\00\00\00\00
lb_hash_stats (arridx:234) \00\00\00\00\00\00\00\00
lb_hash_stats (arridx:233) \00\00\00\00\00\00\00\00
lb_hash_stats (arridx:232) \00\00\00\00\00\00\00\00
lb_hash_stats (arridx:231) \00\00\00\00\00\00\00\00
lb_hash_stats (arridx:230) \00\00\00\00\00\00\00\00
lb_hash_stats (arridx:229) \00\00\00\00\00\00\00\00
lb_hash_stats (arridx:228) \00\00\00\00\00\00\00\00
lb_hash_stats (arridx:227) \00\00\00\00\00\00\00\00
lb_hash_stats (arridx:226) \00\00\00\00\00\00\00\00
lb_hash_stats (arridx:225) \00\00\00\00\00\00\00\00
lb_hash_stats (arridx:224) \00\00\00\00\00\00\00\00
lb_hash_stats (arridx:223) \00\00\00\00\00\00\00\00
lb_hash_stats (arridx:222) \00\00\00\00\00\00\00\00
lb_hash_stats (arridx:221) \00\00\00\00\00\00\00\00
lb_hash_stats (arridx:220) \00\00\00\00\00\00\00\00
lb_hash_stats (arridx:219) \00\00\00\00\00\00\00\00
lb_hash_stats (arridx:218) \00\00\00\00\00\00\00\00
lb_hash_stats (arridx:217) \00\00\00\00\00\00\00\00
lb_hash_stats (arridx:216) \00\00\00\00\00\00\00\00
lb_hash_stats (arridx:215) \00\00\00\00\00\00\00\00
lb_hash_stats (arridx:214) \00\00\00\00\00\00\00\00
lb_hash_stats (arridx:213) \00\00\00\00\00\00\00\00
lb_hash_stats (arridx:212) \00\00\00\00\00\00\00\00
lb_hash_stats (arridx:211) \00\00\00\00\00\00\00\00
lb_hash_stats (arridx:210) \00\00\00\00\00\00\00\00
lb_hash_stats (arridx:209) \00\00\00\00\00\00\00\00
lb_hash_stats (arridx:208) \00\00\00\00\00\00\00\00
lb_hash_stats (arridx:207) \00\00\00\00\00\00\00\00
lb_hash_stats (arridx:206) \00\00\00\00\00\00\00\00
lb_hash_stats (arridx:205) \00\00\00\00\00\00\00\00
lb_hash_stats (arridx:204) \00\00\00\00\00\00\00\00
lb_hash_stats (arridx:203) \00\00\00\00\00\00\00\00
lb_hash_stats (arridx:202) \00\00\00\00\00\00\00\00
lb_hash_stats (arridx:201) \00\00\00\00\00\00\00\00
lb_hash_stats (arridx:200) \00\00\00\00\00\00\00\00
lb_hash_stats (arridx:199) \00\00\00\00\00\00\00\00
lb_hash_stats (arridx:198) \00\00\00\00\00\00\00\00
lb_hash_stats (arridx:197) \00\00\00\00\00\00\00\00
lb_hash_stats (arridx:196) \00\00\00\00\00\00\00\00
lb_hash_stats (arridx:195) \00\00\00\00\00\00\00\00
lb_hash_stats (arridx:194) \00\00\00\00\00\00\00\00
lb_hash_stats (arridx:193) \00\00\00\00\00\00\00\00
lb_hash_stats (arridx:192) \00\00\00\00\00\00\00\00
lb_hash_stats (arridx:191) \00\00\00\00\00\00\00\00
lb_hash_stats (arridx:190) \00\00\00\00\00\00\00\00
lb_hash_stats (arridx:189) \00\00\00\00\00\00\00\00
lb_hash_stats (arridx:188) \00\00\00\00\00\00\00\00
lb_hash_stats (arridx:187) \00\00\00\00\00\00\00\00
lb_hash_stats (arridx:186) \00\00\00\00\00\00\00\00
lb_hash_stats (arridx:185) \00\00\00\00\00\00\00\00
lb_hash_stats (arridx:184) \00\00\00\00\00\00\00\00
lb_hash_stats (arridx:183) \00\00\00\00\00\00\00\00
lb_hash_stats (arridx:182) \00\00\00\00\00\00\00\00
lb_hash_stats (arridx:181) \00\00\00\00\00\00\00\00
lb_hash_stats (arridx:180) \00\00\00\00\00\00\00\00
lb_hash_stats (arridx:179) \00\00\00\00\00\00\00\00
lb_hash_stats (arridx:178) \00\00\00\00\00\00\00\00
lb_hash_stats (arridx:177) \00\00\00\00\00\00\00\00
lb_hash_stats (arridx:176) \00\00\00\00\00\00\00\00
lb_hash_stats (arridx:175) \00\00\00\00\00\00\00\00
lb_hash_stats (arridx:174) \00\00\00\00\00\00\00\00
lb_hash_stats (arridx:173) \00\00\00\00\00\00\00\00
lb_hash_stats (arridx:172) \00\00\00\00\00\00\00\00
lb_hash_stats (arridx:171) \00\00\00\00\00\00\00\00
lb_hash_stats (arridx:170) \00\00\00\00\00\00\00\00
lb_hash_stats (arridx:169) \00\00\00\00\00\00\00\00
lb_hash_stats (arridx:168) \00\00\00\00\00\00\00\00
lb_hash_stats (arridx:167) \00\00\00\00\00\00\00\00
lb_hash_stats (arridx:166) \00\00\00\00\00\00\00\00
lb_hash_stats (arridx:165) \00\00\00\00\00\00\00\00
lb_hash_stats (arridx:164) \00\00\00\00\00\00\00\00
lb_hash_stats (arridx:163) \00\00\00\00\00\00\00\00
lb_hash_stats (arridx:162) \00\00\00\00\00\00\00\00
lb_hash_stats (arridx:161) \00\00\00\00\00\00\00\00
lb_hash_stats (arridx:160) \00\00\00\00\00\00\00\00
lb_hash_stats (arridx:159) \00\00\00\00\00\00\00\00
lb_hash_stats (arridx:158) \00\00\00\00\00\00\00\00
lb_hash_stats (arridx:157) \00\00\00\00\00\00\00\00
lb_hash_stats (arridx:156) \00\00\00\00\00\00\00\00
lb_hash_stats (arridx:155) \00\00\00\00\00\00\00\00
lb_hash_stats (arridx:154) \00\00\00\00\00\00\00\00
lb_hash_stats (arridx:153) \00\00\00\00\00\00\00\00
lb_hash_stats (arridx:152) \00\00\00\00\00\00\00\00
lb_hash_stats (arridx:151) \00\00\00\00\00\00\00\00
lb_hash_stats (arridx:150) \00\00\00\00\00\00\00\00
lb_hash_stats (arridx:149) \00\00\00\00\00\00\00\00
lb_hash_stats (arridx:148) \00\00\00\00\00\00\00\00
lb_hash_stats (arridx:147) \00\00\00\00\00\00\00\00
lb_hash_stats (arridx:146) \00\00\00\00\00\00\00\00
lb_hash_stats (arridx:145) \00\00\00\00\00\00\00\00
lb_hash_stats (arridx:144) \00\00\00\00\00\00\00\00
lb_hash_stats (arridx:143) \00\00\00\00\00\00\00\00
lb_hash_stats (arridx:142) \00\00\00\00\00\00\00\00
lb_hash_stats (arridx:141) \00\00\00\00\00\00\00\00
lb_hash_stats (arridx:140) \00\00\00\00\00\00\00\00
lb_hash_stats (arridx:139) \00\00\00\00\00\00\00\00
lb_hash_stats (arridx:138) \00\00\00\00\00\00\00\00
lb_hash_stats (arridx:137) \00\00\00\00\00\00\00\00
lb_hash_stats (arridx:136) \00\00\00\00\00\00\00\00
lb_hash_stats (arridx:135) \00\00\00\00\00\00\00\00
lb_hash_stats (arridx:134) \00\00\00\00\00\00\00\00
lb_hash_stats (arridx:133) \00\00\00\00\00\00\00\00
lb_hash_stats (arridx:132) \00\00\00\00\00\00\00\00
lb_hash_stats (arridx:131) \00\00\00\00\00\00\00\00
lb_hash_stats (arridx:130) \00\00\00\00\00\00\00\00
lb_hash_stats (arridx:129) \00\00\00\00\00\00\00\00
lb_hash_stats (arridx:128) \00\00\00\00\00\00\00\00
lb_hash_stats (arridx:127) \00\00\00\00\00\00\00\00
lb_hash_stats (arridx:126) \00\00\00\00\00\00\00\00
lb_hash_stats (arridx:125) \00\00\00\00\00\00\00\00
lb_hash_stats (arridx:124) \00\00\00\00\00\00\00\00
lb_hash_stats (arridx:123) \00\00\00\00\00\00\00\00
lb_hash_stats (arridx:122) \00\00\00\00\00\00\00\00
lb_hash_stats (arridx:121) \00\00\00\00\00\00\00\00
lb_hash_stats (arridx:120) \00\00\00\00\00\00\00\00
lb_hash_stats (arridx:119) \00\00\00\00\00\00\00\00
lb_hash_stats (arridx:118) \00\00\00\00\00\00\00\00
lb_hash_stats (arridx:117) \00\00\00\00\00\00\00\00
lb_hash_stats (arridx:116) \00\00\00\00\00\00\00\00
lb_hash_stats (arridx:115) \00\00\00\00\00\00\00\00
lb_hash_stats (arridx:114) \00\00\00\00\00\00\00\00
lb_hash_stats (arridx:113) \00\00\00\00\00\00\00\00
lb_hash_stats (arridx:112) \00\00\00\00\00\00\00\00
lb_hash_stats (arridx:111) \00\00\00\00\00\00\00\00
lb_hash_stats (arridx:110) \00\00\00\00\00\00\00\00
lb_hash_stats (arridx:109) \00\00\00\00\00\00\00\00
lb_hash_stats (arridx:108) \00\00\00\00\00\00\00\00
lb_hash_stats (arridx:107) \00\00\00\00\00\00\00\00
lb_hash_stats (arridx:106) \00\00\00\00\00\00\00\00
lb_hash_stats (arridx:105) \00\00\00\00\00\00\00\00
lb_hash_stats (arridx:104) \00\00\00\00\00\00\00\00
lb_hash_stats (arridx:103) \00\00\00\00\00\00\00\00
lb_hash_stats (arridx:102) \00\00\00\00\00\00\00\00
lb_hash_stats (arridx:101) \00\00\00\00\00\00\00\00
lb_hash_stats (arridx:100) \00\00\00\00\00\00\00\00
lb_hash_stats (arridx:99) \00\00\00\00\00\00\00\00
lb_hash_stats (arridx:98) \00\00\00\00\00\00\00\00
lb_hash_stats (arridx:97) \00\00\00\00\00\00\00\00
lb_hash_stats (arridx:96) \00\00\00\00\00\00\00\00
lb_hash_stats (arridx:95) \00\00\00\00\00\00\00\00
lb_hash_stats (arridx:94) \00\00\00\00\00\00\00\00
lb_hash_stats (arridx:93) \00\00\00\00\00\00\00\00
lb_hash_stats (arridx:92) \00\00\00\00\00\00\00\00
lb_hash_stats (arridx:91) \00\00\00\00\00\00\00\00
lb_hash_stats (arridx:90) \00\00\00\00\00\00\00\00
lb_hash_stats (arridx:89) \00\00\00\00\00\00\00\00
lb_hash_stats (arridx:88) \00\00\00\00\00\00\00\00
lb_hash_stats (arridx:87) \00\00\00\00\00\00\00\00
lb_hash_stats (arridx:86) \00\00\00\00\00\00\00\00
lb_hash_stats (arridx:85) \00\00\00\00\00\00\00\00
lb_hash_stats (arridx:84) \00\00\00\00\00\00\00\00
lb_hash_stats (arridx:83) \00\00\00\00\00\00\00\00
lb_hash_stats (arridx:82) \00\00\00\00\00\00\00\00
lb_hash_stats (arridx:81) \00\00\00\00\00\00\00\00
lb_hash_stats (arridx:80) \00\00\00\00\00\00\00\00
lb_hash_stats (arridx:79) \00\00\00\00\00\00\00\00
lb_hash_stats (arridx:78) \00\00\00\00\00\00\00\00
lb_hash_stats (arridx:77) \00\00\00\00\00\00\00\00
lb_hash_stats (arridx:76) \00\00\00\00\00\00\00\00
lb_hash_stats (arridx:75) \00\00\00\00\00\00\00\00
lb_hash_stats (arridx:74) \00\00\00\00\00\00\00\00
lb_hash_stats (arridx:73) \00\00\00\00\00\00\00\00
lb_hash_stats (arridx:72) \00\00\00\00\00\00\00\00
lb_hash_stats (arridx:71) \00\00\00\00\00\00\00\00
lb_hash_stats (arridx:70) \00\00\00\00\00\00\00\00
lb_hash_stats (arridx:69) \00\00\00\00\00\00\00\00
lb_hash_stats (arridx:68) \00\00\00\00\00\00\00\00
lb_hash_stats (arridx:67) \00\00\00\00\00\00\00\00
lb_hash_stats (arridx:66) \00\00\00\00\00\00\00\00
lb_hash_stats (arridx:65) \00\00\00\00\00\00\00\00
lb_hash_stats (arridx:64) \00\00\00\00\00\00\00\00
lb_hash_stats (arridx:63) \00\00\00\00\00\00\00\00
lb_hash_stats (arridx:62) \00\00\00\00\00\00\00\00
lb_hash_stats (arridx:61) \00\00\00\00\00\00\00\00
lb_hash_stats (arridx:60) \00\00\00\00\00\00\00\00
lb_hash_stats (arridx:59) \00\00\00\00\00\00\00\00
lb_hash_stats (arridx:58) \00\00\00\00\00\00\00\00
lb_hash_stats (arridx:57) \00\00\00\00\00\00\00\00
lb_hash_stats (arridx:56) \00\00\00\00\00\00\00\00
lb_hash_stats (arridx:55) \00\00\00\00\00\00\00\00
lb_hash_stats (arridx:54) \00\00\00\00\00\00\00\00
lb_hash_stats (arridx:53) \00\00\00\00\00\00\00\00
lb_hash_stats (arridx:52) \00\00\00\00\00\00\00\00
lb_hash_stats (arridx:51) \00\00\00\00\00\00\00\00
lb_hash_stats (arridx:50) \00\00\00\00\00\00\00\00
lb_hash_stats (arridx:49) \00\00\00\00\00\00\00\00
lb_hash_stats (arridx:48) \00\00\00\00\00\00\00\00
lb_hash_stats (arridx:47) \00\00\00\00\00\00\00\00
lb_hash_stats (arridx:46) \00\00\00\00\00\00\00\00
lb_hash_stats (arridx:45) \00\00\00\00\00\00\00\00
lb_hash_stats (arridx:44) \00\00\00\00\00\00\00\00
lb_hash_stats (arridx:43) \00\00\00\00\00\00\00\00
lb_hash_stats (arridx:42) \00\00\00\00\00\00\00\00
lb_hash_stats (arridx:41) \00\00\00\00\00\00\00\00
lb_hash_stats (arridx:40) \00\00\00\00\00\00\00\00
lb_hash_stats (arridx:39) \00\00\00\00\00\00\00\00
lb_hash_stats (arridx:38) \00\00\00\00\00\00\00\00
lb_hash_stats (arridx:37) \00\00\00\00\00\00\00\00
lb_hash_stats (arridx:36) \00\00\00\00\00\00\00\00
lb_hash_stats (arridx:35) \00\00\00\00\00\00\00\00
lb_hash_stats (arridx:34) \00\00\00\00\00\00\00\00
lb_hash_stats (arridx:33) \00\00\00\00\00\00\00\00
lb_hash_stats (arridx:32) \00\00\00\00\00\00\00\00
lb_hash_stats (arridx:31) \00\00\00\00\00\00\00\00
lb_hash_stats (arridx:30) \00\00\00\00\00\00\00\00
lb_hash_stats (arridx:29) \00\00\00\00\00\00\00\00
lb_hash_stats (arridx:28) \00\00\00\00\00\00\00\00
lb_hash_stats (arridx:27) \00\00\00\00\00\00\00\00
lb_hash_stats (arridx:26) \00\00\00\00\00\00\00\00
lb_hash_stats (arridx:25) \00\00\00\00\00\00\00\00
lb_hash_stats (arridx:24) \00\00\00\00\00\00\00\00
lb_hash_stats (arridx:23) \00\00\00\00\00\00\00\00
lb_hash_stats (arridx:22) \00\00\00\00\00\00\00\00
lb_hash_stats (arridx:21) \00\00\00\00\00\00\00\00
lb_hash_stats (arridx:20) \00\00\00\00\00\00\00\00
lb_hash_stats (arridx:19) \00\00\00\00\00\00\00\00
lb_hash_stats (arridx:18) \00\00\00\00\00\00\00\00
lb_hash_stats (arridx:17) \00\00\00\00\00\00\00\00
lb_hash_stats (arridx:16) \00\00\00\00\00\00\00\00
lb_hash_stats (arridx:15) \00\00\00\00\00\00\00\00
lb_hash_stats (arridx:14) \00\00\00\00\00\00\00\00
lb_hash_stats (arridx:13) \00\00\00\00\00\00\00\00
lb_hash_stats (arridx:12) \00\00\00\00\00\00\00\00
lb_hash_stats (arridx:11) \00\00\00\00\00\00\00\00
lb_hash_stats (arridx:10) \00\00\00\00\00\00\00\00
lb_hash_stats (arridx:9) \00\00\00\00\00\00\00\00
lb_hash_stats (arridx:8) \00\00\00\00\00\00\00\00
lb_hash_stats (arridx:7) \00\00\00\00\00\00\00\00
lb_hash_stats (arridx:6) \00\00\00\00\00\00\00\00
lb_hash_stats (arridx:5) \00\00\00\00\00\00\00\00
lb_hash_stats (arridx:4) \00\00\00\00\00\00\00\00
lb_hash_stats (arridx:3) \00\00\00\00\00\00\00\00
lb_hash_stats (arridx:2) \00\00\00\00\00\00\00\00
lb_hash_stats (arridx:1) \00\00\00\00\00\00\00\00
lb_hash_stats (arridx:0) \00\00\00\00\00\00\00\00
lb_tx_hash_to_port_mapping (arridx:255) 0
lb_tx_hash_to_port_mapping (arridx:254) 0
lb_tx_hash_to_port_mapping (arridx:253) 0
lb_tx_hash_to_port_mapping (arridx:252) 0
lb_tx_hash_to_port_mapping (arridx:251) 0
lb_tx_hash_to_port_mapping (arridx:250) 0
lb_tx_hash_to_port_mapping (arridx:249) 0
lb_tx_hash_to_port_mapping (arridx:248) 0
lb_tx_hash_to_port_mapping (arridx:247) 0
lb_tx_hash_to_port_mapping (arridx:246) 0
lb_tx_hash_to_port_mapping (arridx:245) 0
lb_tx_hash_to_port_mapping (arridx:244) 0
lb_tx_hash_to_port_mapping (arridx:243) 0
lb_tx_hash_to_port_mapping (arridx:242) 0
lb_tx_hash_to_port_mapping (arridx:241) 0
lb_tx_hash_to_port_mapping (arridx:240) 0
lb_tx_hash_to_port_mapping (arridx:239) 0
lb_tx_hash_to_port_mapping (arridx:238) 0
lb_tx_hash_to_port_mapping (arridx:237) 0
lb_tx_hash_to_port_mapping (arridx:236) 0
lb_tx_hash_to_port_mapping (arridx:235) 0
lb_tx_hash_to_port_mapping (arridx:234) 0
lb_tx_hash_to_port_mapping (arridx:233) 0
lb_tx_hash_to_port_mapping (arridx:232) 0
lb_tx_hash_to_port_mapping (arridx:231) 0
lb_tx_hash_to_port_mapping (arridx:230) 0
lb_tx_hash_to_port_mapping (arridx:229) 0
lb_tx_hash_to_port_mapping (arridx:228) 0
lb_tx_hash_to_port_mapping (arridx:227) 0
lb_tx_hash_to_port_mapping (arridx:226) 0
lb_tx_hash_to_port_mapping (arridx:225) 0
lb_tx_hash_to_port_mapping (arridx:224) 0
lb_tx_hash_to_port_mapping (arridx:223) 0
lb_tx_hash_to_port_mapping (arridx:222) 0
lb_tx_hash_to_port_mapping (arridx:221) 0
lb_tx_hash_to_port_mapping (arridx:220) 0
lb_tx_hash_to_port_mapping (arridx:219) 0
lb_tx_hash_to_port_mapping (arridx:218) 0
lb_tx_hash_to_port_mapping (arridx:217) 0
lb_tx_hash_to_port_mapping (arridx:216) 0
lb_tx_hash_to_port_mapping (arridx:215) 0
lb_tx_hash_to_port_mapping (arridx:214) 0
lb_tx_hash_to_port_mapping (arridx:213) 0
lb_tx_hash_to_port_mapping (arridx:212) 0
lb_tx_hash_to_port_mapping (arridx:211) 0
lb_tx_hash_to_port_mapping (arridx:210) 0
lb_tx_hash_to_port_mapping (arridx:209) 0
lb_tx_hash_to_port_mapping (arridx:208) 0
lb_tx_hash_to_port_mapping (arridx:207) 0
lb_tx_hash_to_port_mapping (arridx:206) 0
lb_tx_hash_to_port_mapping (arridx:205) 0
lb_tx_hash_to_port_mapping (arridx:204) 0
lb_tx_hash_to_port_mapping (arridx:203) 0
lb_tx_hash_to_port_mapping (arridx:202) 0
lb_tx_hash_to_port_mapping (arridx:201) 0
lb_tx_hash_to_port_mapping (arridx:200) 0
lb_tx_hash_to_port_mapping (arridx:199) 0
lb_tx_hash_to_port_mapping (arridx:198) 0
lb_tx_hash_to_port_mapping (arridx:197) 0
lb_tx_hash_to_port_mapping (arridx:196) 0
lb_tx_hash_to_port_mapping (arridx:195) 0
lb_tx_hash_to_port_mapping (arridx:194) 0
lb_tx_hash_to_port_mapping (arridx:193) 0
lb_tx_hash_to_port_mapping (arridx:192) 0
lb_tx_hash_to_port_mapping (arridx:191) 0
lb_tx_hash_to_port_mapping (arridx:190) 0
lb_tx_hash_to_port_mapping (arridx:189) 0
lb_tx_hash_to_port_mapping (arridx:188) 0
lb_tx_hash_to_port_mapping (arridx:187) 0
lb_tx_hash_to_port_mapping (arridx:186) 0
lb_tx_hash_to_port_mapping (arridx:185) 0
lb_tx_hash_to_port_mapping (arridx:184) 0
lb_tx_hash_to_port_mapping (arridx:183) 0
lb_tx_hash_to_port_mapping (arridx:182) 0
lb_tx_hash_to_port_mapping (arridx:181) 0
lb_tx_hash_to_port_mapping (arridx:180) 0
lb_tx_hash_to_port_mapping (arridx:179) 0
lb_tx_hash_to_port_mapping (arridx:178) 0
lb_tx_hash_to_port_mapping (arridx:177) 0
lb_tx_hash_to_port_mapping (arridx:176) 0
lb_tx_hash_to_port_mapping (arridx:175) 0
lb_tx_hash_to_port_mapping (arridx:174) 0
lb_tx_hash_to_port_mapping (arridx:173) 0
lb_tx_hash_to_port_mapping (arridx:172) 0
lb_tx_hash_to_port_mapping (arridx:171) 0
lb_tx_hash_to_port_mapping (arridx:170) 0
lb_tx_hash_to_port_mapping (arridx:169) 0
lb_tx_hash_to_port_mapping (arridx:168) 0
lb_tx_hash_to_port_mapping (arridx:167) 0
lb_tx_hash_to_port_mapping (arridx:166) 0
lb_tx_hash_to_port_mapping (arridx:165) 0
lb_tx_hash_to_port_mapping (arridx:164) 0
lb_tx_hash_to_port_mapping (arridx:163) 0
lb_tx_hash_to_port_mapping (arridx:162) 0
lb_tx_hash_to_port_mapping (arridx:161) 0
lb_tx_hash_to_port_mapping (arridx:160) 0
lb_tx_hash_to_port_mapping (arridx:159) 0
lb_tx_hash_to_port_mapping (arridx:158) 0
lb_tx_hash_to_port_mapping (arridx:157) 0
lb_tx_hash_to_port_mapping (arridx:156) 0
lb_tx_hash_to_port_mapping (arridx:155) 0
lb_tx_hash_to_port_mapping (arridx:154) 0
lb_tx_hash_to_port_mapping (arridx:153) 0
lb_tx_hash_to_port_mapping (arridx:152) 0
lb_tx_hash_to_port_mapping (arridx:151) 0
lb_tx_hash_to_port_mapping (arridx:150) 0
lb_tx_hash_to_port_mapping (arridx:149) 0
lb_tx_hash_to_port_mapping (arridx:148) 0
lb_tx_hash_to_port_mapping (arridx:147) 0
lb_tx_hash_to_port_mapping (arridx:146) 0
lb_tx_hash_to_port_mapping (arridx:145) 0
lb_tx_hash_to_port_mapping (arridx:144) 0
lb_tx_hash_to_port_mapping (arridx:143) 0
lb_tx_hash_to_port_mapping (arridx:142) 0
lb_tx_hash_to_port_mapping (arridx:141) 0
lb_tx_hash_to_port_mapping (arridx:140) 0
lb_tx_hash_to_port_mapping (arridx:139) 0
lb_tx_hash_to_port_mapping (arridx:138) 0
lb_tx_hash_to_port_mapping (arridx:137) 0
lb_tx_hash_to_port_mapping (arridx:136) 0
lb_tx_hash_to_port_mapping (arridx:135) 0
lb_tx_hash_to_port_mapping (arridx:134) 0
lb_tx_hash_to_port_mapping (arridx:133) 0
lb_tx_hash_to_port_mapping (arridx:132) 0
lb_tx_hash_to_port_mapping (arridx:131) 0
lb_tx_hash_to_port_mapping (arridx:130) 0
lb_tx_hash_to_port_mapping (arridx:129) 0
lb_tx_hash_to_port_mapping (arridx:128) 0
lb_tx_hash_to_port_mapping (arridx:127) 0
lb_tx_hash_to_port_mapping (arridx:126) 0
lb_tx_hash_to_port_mapping (arridx:125) 0
lb_tx_hash_to_port_mapping (arridx:124) 0
lb_tx_hash_to_port_mapping (arridx:123) 0
lb_tx_hash_to_port_mapping (arridx:122) 0
lb_tx_hash_to_port_mapping (arridx:121) 0
lb_tx_hash_to_port_mapping (arridx:120) 0
lb_tx_hash_to_port_mapping (arridx:119) 0
lb_tx_hash_to_port_mapping (arridx:118) 0
lb_tx_hash_to_port_mapping (arridx:117) 0
lb_tx_hash_to_port_mapping (arridx:116) 0
lb_tx_hash_to_port_mapping (arridx:115) 0
lb_tx_hash_to_port_mapping (arridx:114) 0
lb_tx_hash_to_port_mapping (arridx:113) 0
lb_tx_hash_to_port_mapping (arridx:112) 0
lb_tx_hash_to_port_mapping (arridx:111) 0
lb_tx_hash_to_port_mapping (arridx:110) 0
lb_tx_hash_to_port_mapping (arridx:109) 0
lb_tx_hash_to_port_mapping (arridx:108) 0
lb_tx_hash_to_port_mapping (arridx:107) 0
lb_tx_hash_to_port_mapping (arridx:106) 0
lb_tx_hash_to_port_mapping (arridx:105) 0
lb_tx_hash_to_port_mapping (arridx:104) 0
lb_tx_hash_to_port_mapping (arridx:103) 0
lb_tx_hash_to_port_mapping (arridx:102) 0
lb_tx_hash_to_port_mapping (arridx:101) 0
lb_tx_hash_to_port_mapping (arridx:100) 0
lb_tx_hash_to_port_mapping (arridx:99) 0
lb_tx_hash_to_port_mapping (arridx:98) 0
lb_tx_hash_to_port_mapping (arridx:97) 0
lb_tx_hash_to_port_mapping (arridx:96) 0
lb_tx_hash_to_port_mapping (arridx:95) 0
lb_tx_hash_to_port_mapping (arridx:94) 0
lb_tx_hash_to_port_mapping (arridx:93) 0
lb_tx_hash_to_port_mapping (arridx:92) 0
lb_tx_hash_to_port_mapping (arridx:91) 0
lb_tx_hash_to_port_mapping (arridx:90) 0
lb_tx_hash_to_port_mapping (arridx:89) 0
lb_tx_hash_to_port_mapping (arridx:88) 0
lb_tx_hash_to_port_mapping (arridx:87) 0
lb_tx_hash_to_port_mapping (arridx:86) 0
lb_tx_hash_to_port_mapping (arridx:85) 0
lb_tx_hash_to_port_mapping (arridx:84) 0
lb_tx_hash_to_port_mapping (arridx:83) 0
lb_tx_hash_to_port_mapping (arridx:82) 0
lb_tx_hash_to_port_mapping (arridx:81) 0
lb_tx_hash_to_port_mapping (arridx:80) 0
lb_tx_hash_to_port_mapping (arridx:79) 0
lb_tx_hash_to_port_mapping (arridx:78) 0
lb_tx_hash_to_port_mapping (arridx:77) 0
lb_tx_hash_to_port_mapping (arridx:76) 0
lb_tx_hash_to_port_mapping (arridx:75) 0
lb_tx_hash_to_port_mapping (arridx:74) 0
lb_tx_hash_to_port_mapping (arridx:73) 0
lb_tx_hash_to_port_mapping (arridx:72) 0
lb_tx_hash_to_port_mapping (arridx:71) 0
lb_tx_hash_to_port_mapping (arridx:70) 0
lb_tx_hash_to_port_mapping (arridx:69) 0
lb_tx_hash_to_port_mapping (arridx:68) 0
lb_tx_hash_to_port_mapping (arridx:67) 0
lb_tx_hash_to_port_mapping (arridx:66) 0
lb_tx_hash_to_port_mapping (arridx:65) 0
lb_tx_hash_to_port_mapping (arridx:64) 0
lb_tx_hash_to_port_mapping (arridx:63) 0
lb_tx_hash_to_port_mapping (arridx:62) 0
lb_tx_hash_to_port_mapping (arridx:61) 0
lb_tx_hash_to_port_mapping (arridx:60) 0
lb_tx_hash_to_port_mapping (arridx:59) 0
lb_tx_hash_to_port_mapping (arridx:58) 0
lb_tx_hash_to_port_mapping (arridx:57) 0
lb_tx_hash_to_port_mapping (arridx:56) 0
lb_tx_hash_to_port_mapping (arridx:55) 0
lb_tx_hash_to_port_mapping (arridx:54) 0
lb_tx_hash_to_port_mapping (arridx:53) 0
lb_tx_hash_to_port_mapping (arridx:52) 0
lb_tx_hash_to_port_mapping (arridx:51) 0
lb_tx_hash_to_port_mapping (arridx:50) 0
lb_tx_hash_to_port_mapping (arridx:49) 0
lb_tx_hash_to_port_mapping (arridx:48) 0
lb_tx_hash_to_port_mapping (arridx:47) 0
lb_tx_hash_to_port_mapping (arridx:46) 0
lb_tx_hash_to_port_mapping (arridx:45) 0
lb_tx_hash_to_port_mapping (arridx:44) 0
lb_tx_hash_to_port_mapping (arridx:43) 0
lb_tx_hash_to_port_mapping (arridx:42) 0
lb_tx_hash_to_port_mapping (arridx:41) 0
lb_tx_hash_to_port_mapping (arridx:40) 0
lb_tx_hash_to_port_mapping (arridx:39) 0
lb_tx_hash_to_port_mapping (arridx:38) 0
lb_tx_hash_to_port_mapping (arridx:37) 0
lb_tx_hash_to_port_mapping (arridx:36) 0
lb_tx_hash_to_port_mapping (arridx:35) 0
lb_tx_hash_to_port_mapping (arridx:34) 0
lb_tx_hash_to_port_mapping (arridx:33) 0
lb_tx_hash_to_port_mapping (arridx:32) 0
lb_tx_hash_to_port_mapping (arridx:31) 0
lb_tx_hash_to_port_mapping (arridx:30) 0
lb_tx_hash_to_port_mapping (arridx:29) 0
lb_tx_hash_to_port_mapping (arridx:28) 0
lb_tx_hash_to_port_mapping (arridx:27) 0
lb_tx_hash_to_port_mapping (arridx:26) 0
lb_tx_hash_to_port_mapping (arridx:25) 0
lb_tx_hash_to_port_mapping (arridx:24) 0
lb_tx_hash_to_port_mapping (arridx:23) 0
lb_tx_hash_to_port_mapping (arridx:22) 0
lb_tx_hash_to_port_mapping (arridx:21) 0
lb_tx_hash_to_port_mapping (arridx:20) 0
lb_tx_hash_to_port_mapping (arridx:19) 0
lb_tx_hash_to_port_mapping (arridx:18) 0
lb_tx_hash_to_port_mapping (arridx:17) 0
lb_tx_hash_to_port_mapping (arridx:16) 0
lb_tx_hash_to_port_mapping (arridx:15) 0
lb_tx_hash_to_port_mapping (arridx:14) 0
lb_tx_hash_to_port_mapping (arridx:13) 0
lb_tx_hash_to_port_mapping (arridx:12) 0
lb_tx_hash_to_port_mapping (arridx:11) 0
lb_tx_hash_to_port_mapping (arridx:10) 0
lb_tx_hash_to_port_mapping (arridx:9) 0
lb_tx_hash_to_port_mapping (arridx:8) 0
lb_tx_hash_to_port_mapping (arridx:7) 0
lb_tx_hash_to_port_mapping (arridx:6) 0
lb_tx_hash_to_port_mapping (arridx:5) 0
lb_tx_hash_to_port_mapping (arridx:4) 0
lb_tx_hash_to_port_mapping (arridx:3) 0
lb_tx_hash_to_port_mapping (arridx:2) 0
lb_tx_hash_to_port_mapping (arridx:1) 0
lb_tx_hash_to_port_mapping (arridx:0) 0
lb_tx_method hash
bpf_hash_func \20\00\00\00\02\00\00\00\30\00\00\00\28\f0\ff\ff\07\00\00\00\00\00\00\00\28\00\00\00\00\00\00\00\30\00\0 <trunc>
mcast_rejoin_interval 0
mcast_rejoin_count 0
notify_peers_interval 0
notify_peers_count 0
mode loadbalance
It is very hard for me to believe that every I hash has "00\00\00\00\00\00\00\00" in it ! What is balancing the traffic ? :-)
First let me just answer you on couple of ideas you mentioned.
1. We are the Internet Exchange so the probability that one TCP stream between two of our customers is the cause of that much disbalance is very small.. You can see from the graphics I posted to forum we are talking about 1-2 Gb/sec od traffic that is not balanced between ports while the whole trafic is between 3-5 Gb/sec! Couple of the test show graphs that are such that you can see practically all of the traffic is on single port !!!
2. Also (from the output of the teamnl team0 options) something is VERY wrong because I am not seeing any different hashes been calculated !!! There are two teamd proceses in the "ps -AL" [root@rtr ~]# ps -AL | grep team
3433 3433 ? 00:12:32 teamd
6187 6187 ? 00:00:00 teamd
3. If there is some more data you need please let me know.
Best regards
G.Slavic
-----Original Message-----
> From: "Jiri Pirko" <jiri(a)resnulli.us>
> To: "Goran Slavić" <gslavic(a)sox.rs>
> Cc: libteam(a)lists.fedorahosted.org
> Date: 06/04/15 08:13
> Subject: Re: Problems with teaming ...
>
> Wed, Jun 03, 2015 at 10:13:16PM CEST, gslavic(a)sox.rs wrote:
> >Dear Mr Jiri Pirko,
> >
> >
> >
> > My name is Goran Slavić, and I work for Serbian Open
> >eXchange. We have setup the teaming on 4x10Gb/sec ports of our routing
> >server. Unfortunetly load balancing between teamed ports doesn't work very
> >well (we have extremely UN-balanced traffic on teamed ports). Traffic
> >analysis suggests that the balancing is only done with MAC addresses and the
> >rest of configuration directives ("ipv4, ipv6, l4" ...) is ignored/not taken
> >in consideration when load balancing traffic between ports !
> >
> > You can find a detailed explanation (what we configured on
> >ports/teaming software), our configuration details and graph that are the
> >result of our testing on this link (forum where we first shown the problem
> >and asked people to help us):
> >
> > https://www.centos.org/forums/viewtopic.php?f=50&t=52121
> >
> > We would highly appreciate any help, advice or suggestion
> >you can give us regarding this problem.
>
>
> Hello Goran.
>
> It's always good idea to cc libteam mailing list for reports such as
> yours.
>
> TX port selection is done based on hash (0-255) computed by BPF function.
> Then, the hash is either used directly to determine tx port
> (hash%number_of_ports), or there is a hash-port mapping where every hash
> is mapped to a tx port. The second case is yours.
>
> You can see the mapping either in teamd debug output, or if you list
> team options directly by "teamnl team0 options". With mapping there are
> also statistics for each hash there.
>
> So in your case, there might be 3 issues:
> 1) your traffic is not differenciated enough, so most of the packets
> falls into sigle hash. That would happen for example for one tcp stream
> between 2 ip addresses.
> 2) hash function fails to produce good hash according to the packets
> coming through. You could try to dump the BPF function and look at the
> inctructions by hand. Or another thing would be to assebmle your own
> hash function and push it into kernel manually (using teamnl team0
> setoption bpf_hash_func
> BFPCODE)
> 3) mapping is wrong. Balancer might do the mapping wrongly. To figure this
> out, the debug messages or options dump will help you to find out if
> this is the case.
>
> Please gather more info so we can proceed to debug this.
>
> Thanks.
>
> Jiri
>
>
>
8 years, 3 months
- 2
- 5
Re: Problems with teaming ...
by Jiri Pirko
Wed, Jun 03, 2015 at 10:13:16PM CEST, gslavic(a)sox.rs wrote:
>Dear Mr Jiri Pirko,
>
>
>
> My name is Goran Slavić, and I work for Serbian Open
>eXchange. We have setup the teaming on 4x10Gb/sec ports of our routing
>server. Unfortunetly load balancing between teamed ports doesn't work very
>well (we have extremely UN-balanced traffic on teamed ports). Traffic
>analysis suggests that the balancing is only done with MAC addresses and the
>rest of configuration directives ("ipv4, ipv6, l4" ...) is ignored/not taken
>in consideration when load balancing traffic between ports !
>
> You can find a detailed explanation (what we configured on
>ports/teaming software), our configuration details and graph that are the
>result of our testing on this link (forum where we first shown the problem
>and asked people to help us):
>
> https://www.centos.org/forums/viewtopic.php?f=50&t=52121
>
> We would highly appreciate any help, advice or suggestion
>you can give us regarding this problem.
Hello Goran.
It's always good idea to cc libteam mailing list for reports such as
yours.
TX port selection is done based on hash (0-255) computed by BPF function.
Then, the hash is either used directly to determine tx port
(hash%number_of_ports), or there is a hash-port mapping where every hash
is mapped to a tx port. The second case is yours.
You can see the mapping either in teamd debug output, or if you list
team options directly by "teamnl team0 options". With mapping there are
also statistics for each hash there.
So in your case, there might be 3 issues:
1) your traffic is not differenciated enough, so most of the packets
falls into sigle hash. That would happen for example for one tcp stream
between 2 ip addresses.
2) hash function fails to produce good hash according to the packets
coming through. You could try to dump the BPF function and look at the
inctructions by hand. Or another thing would be to assebmle your own
hash function and push it into kernel manually (using teamnl team0
setoption bpf_hash_func
BFPCODE)
3) mapping is wrong. Balancer might do the mapping wrongly. To figure this
out, the debug messages or options dump will help you to find out if
this is the case.
Please gather more info so we can proceed to debug this.
Thanks.
Jiri
8 years, 4 months
- 1
- 0
- ← Newer
- 1
- Older →