Re: Stange problems with CentOS 7 and roundrobin runner

[Jiri Pirko]

Thu, Nov 06, 2014 at 05:44:37PM CET, [e-mail] wrote:
> Am 06.11.2014 um 17:26 schrieb Jiri Pirko:
> > Thu, Nov 06, 2014 at 05:18:18PM CET, [e-mail] wrote:
> > > Am 06.11.2014 um 17:03 schrieb Jiri Pirko:
> > > > Thu, Nov 06, 2014 at 04:26:36PM CET, [e-mail] wrote:
> > > > > Hello,
> > > > >
> > > > > I am currently setting up a high available NFS server cluster with 2 nodes
> > > > > using pacemaker + corosync + drbd + flashcache.
> > > > > The two nodes (media1 and media2) have 4 1G copper nics each.
> > > > > I formed 2 teams:
> > > > > team0: LACP with em1 and em2 connected to a switch for NFS services
> > > > > team1: roundrobin with em3 and em4 and MTU 9000 directly connected with
> > > > > copper patch cables for drbd sync
> > > > > During my testing everything was working as expected. Then I started to
> > > > > tranfer about 20TB from an old server to the new cluster. The copy process
> > > > > was fine for about 32 hours. Then corosync and drbd started to report errors
> > > > > on the roundrobin team1 interface (see logs at the end of this email).
> > > > > The drbd sync and corosync started to flap between states for several hours
> > > > > on team1.
> > > > > I checked the error counters with ifconfig and found a huge amount off TX
> > > > > dropped packets on team1 of the drbd slave node media2.
> > > > > There were NO other error counters raising but team1 on media2.
> > > > > To make this a bit more clear:
> > > > > team1 on media2 (the drbd slave) showed raising TX dropped packets
> > > > > em3 and em4 on media2 (the physical interfaces of team1) did not show any
> > > > >
> > > > > raising error counters.
> > > > > team1 on media1 (the drbd master) did not show any raising error counters.
> > > > > em3 and em4 on media1 (the physical interfaces of team1) did not show any
> > > > >
> > > > > raising error counters.
> > > > > I then thought maybe one of the two patch cables between the two machines was
> > > > > not 100% ok. So I removed em4 from team1 on both machines:
> > > > > "teamdctl team1 port remove em4"
> > > > > The error counters stopped raising.
> > > > > I thought: "Yeah! That cable must be broken!"
> > > > > But just to make sure I really found the problem I re-enabled em4 on both
> > > > > nodes:
> > > > > "teamdctl team1 port add em4"
> > > > > Ok, now both team1 interfaces had 2 links up again and the error counter
> > > > > restarted to raise again.
> > > > > I then stopped em3 on both nodes to force all traffic through em4:
> > > > > "teamdctl team1 port remove em3"
> > > > > And guess what? The error counters on team1 stopped raising!
> > > > > So in short:
> > > > > If I remove either one of the two physical interfaces from team1 everything
> > > > > is working without any errors. As soon as I enable both physical interfaces
> > > > > the error counters start to raise.
> > > > > After these tests I rebooted the slave node (media2) and added both ports to
> > > > > team1 again (Have you tried turning it off and on again...).
> > > > > After doing this, I now see TX dropped packets on the not yet rebooted media1
> > > > > team1 interface.
> > > > > Why do I only see these errors on the team1 interface and not on em3 and em4?
> > > > >
> > > > > Currently the copy process of 20TB is still running. I think that if I reboot
> > > > > media1 everything will work as expected again for some time because it worked
> > > > > during my initial tests.
> > > > > But I do think that I hit a bug in the teaming driver and the raising error
> > > > > counters will come back.
> > > > >
> > > > > Could anybody help?
> >
> > Hmm, I went over the results, do not see anything suspicious.
> > Are you able to run systemtap scripts on the machine?
>
> I can install systemtap and run some scripts. Please provide me some more
> information.

I'll think about it over night and provide you some test script
tomorrow.
> > > > Interesting. Could you do "teamdctl team1 s d" when the issue happens
> > > > and post me the result?
> > >
> > > I just re-enabled port em4 on both machines and the issue happens again. Here
> > > are your requested results:
> > > media1 ~]# teamdctl team1 s d
> > > {
> > >    "ports": {
> > >        "em3": {
> > >            "ifinfo": {
> > >                "dev_addr": "b8:2a:72:d5:6c:e6",
> > >                "dev_addr_len": 6,
> > >                "ifindex": 4,
> > >                "ifname": "em3"
> > >            },
> > >            "link": {
> > >                "duplex": "full",
> > >                "speed": 1000,
> > >                "up": true
> > >            },
> > >            "link_watches": {
> > >                "list": {
> > >                    "link_watch_0": {
> > >                        "delay_down": 0,
> > >                        "delay_up": 0,
> > >                        "name": "ethtool",
> > >                        "up": true
> > >                    }
> > >                },
> > >                "up": true
> > >            }
> > >        },
> > >        "em4": {
> > >            "ifinfo": {
> > >                "dev_addr": "b8:2a:72:d5:6c:e6",
> > >                "dev_addr_len": 6,
> > >                "ifindex": 5,
> > >                "ifname": "em4"
> > >            },
> > >            "link": {
> > >                "duplex": "full",
> > >                "speed": 1000,
> > >                "up": true
> > >            },
> > >            "link_watches": {
> > >                "list": {
> > >                    "link_watch_0": {
> > >                        "delay_down": 0,
> > >                        "delay_up": 0,
> > >                        "name": "ethtool",
> > >                        "up": true
> > >                    }
> > >                },
> > >                "up": true
> > >            }
> > >        }
> > >    },
> > >    "setup": {
> > >        "daemonized": false,
> > >        "dbus_enabled": true,
> > >        "debug_level": 0,
> > >        "kernel_team_mode_name": "roundrobin",
> > >        "pid": 3107,
> > >        "pid_file": "/var/run/teamd/team1.pid",
> > >        "runner_name": "roundrobin",
> > >        "zmq_enabled": false
> > >    },
> > >    "team_device": {
> > >        "ifinfo": {
> > >            "dev_addr": "b8:2a:72:d5:6c:e6",
> > >            "dev_addr_len": 6,
> > >            "ifindex": 7,
> > >            "ifname": "team1"
> > >        }
> > >    }
> > > }
> > >
> > > media2 ~]# teamdctl team1 s d
> > > {
> > >    "ports": {
> > >        "em3": {
> > >            "ifinfo": {
> > >                "dev_addr": "b8:2a:72:d5:97:18",
> > >                "dev_addr_len": 6,
> > >                "ifindex": 4,
> > >                "ifname": "em3"
> > >            },
> > >            "link": {
> > >                "duplex": "full",
> > >                "speed": 1000,
> > >                "up": true
> > >            },
> > >            "link_watches": {
> > >                "list": {
> > >                    "link_watch_0": {
> > >                        "delay_down": 0,
> > >                        "delay_up": 0,
> > >                        "name": "ethtool",
> > >                        "up": true
> > >                    }
> > >                },
> > >                "up": true
> > >            }
> > >        },
> > >        "em4": {
> > >            "ifinfo": {
> > >                "dev_addr": "b8:2a:72:d5:97:18",
> > >                "dev_addr_len": 6,
> > >                "ifindex": 5,
> > >                "ifname": "em4"
> > >            },
> > >            "link": {
> > >                "duplex": "full",
> > >                "speed": 1000,
> > >                "up": true
> > >            },
> > >            "link_watches": {
> > >                "list": {
> > >                    "link_watch_0": {
> > >                        "delay_down": 0,
> > >                        "delay_up": 0,
> > >                        "name": "ethtool",
> > >                        "up": true
> > >                    }
> > >                },
> > >                "up": true
> > >            }
> > >        }
> > >    },
> > >    "setup": {
> > >        "daemonized": false,
> > >        "dbus_enabled": true,
> > >        "debug_level": 0,
> > >        "kernel_team_mode_name": "roundrobin",
> > >        "pid": 3084,
> > >        "pid_file": "/var/run/teamd/team1.pid",
> > >        "runner_name": "roundrobin",
> > >        "zmq_enabled": false
> > >    },
> > >    "team_device": {
> > >        "ifinfo": {
> > >            "dev_addr": "b8:2a:72:d5:97:18",
> > >            "dev_addr_len": 6,
> > >            "ifindex": 7,
> > >            "ifname": "team1"
> > >        }
> > >    }
> > > }
> > > > Also you can try random runner instead of roundrobin. The resulting
> > > > behaviour is almost the same. Maybe the issue won't happen there.
> > >
> > > I do believe that if I "restart" the roundrobin runner on media1, the problem
> > > is gone anyway. So for the moment I'll keep roundrobin just to keep the
> > > failure environment in place in case you need some more data.
> > > > Also, can you please post me output of:
> > > > ethtool -i em3
> > >
> > > media1 ~]# ethtool -i em3
> > > driver: tg3
> > > version: 3.136
> > > firmware-version: FFV7.8.16 bc 5720-v1.32
> > > bus-info: 0000:02:00.0
> > > supports-statistics: yes
> > > supports-test: yes
> > > supports-eeprom-access: yes
> > > supports-register-dump: yes
> > > supports-priv-flags: no
> > >
> > > media2 ~]# ethtool -i em3
> > > driver: tg3
> > > version: 3.136
> > > firmware-version: FFV7.8.16 bc 5720-v1.32
> > > bus-info: 0000:02:00.0
> > > supports-statistics: yes
> > > supports-test: yes
> > > supports-eeprom-access: yes
> > > supports-register-dump: yes
> > > supports-priv-flags: no
> > > > ethtool em3
> > >
> > > media1 ~]# ethtool em3
> > > Settings for em3:
> > >        Supported ports: [ TP ]
> > >        Supported link modes:   10baseT/Half 10baseT/Full
> > >                                100baseT/Half 100baseT/Full
> > >                                1000baseT/Half 1000baseT/Full
> > >        Supported pause frame use: No
> > >        Supports auto-negotiation: Yes
> > >        Advertised link modes:  10baseT/Half 10baseT/Full
> > >                                100baseT/Half 100baseT/Full
> > >                                1000baseT/Half 1000baseT/Full
> > >        Advertised pause frame use: Symmetric
> > >        Advertised auto-negotiation: Yes
> > >        Link partner advertised link modes:  10baseT/Half 10baseT/Full
> > >                                             100baseT/Half 100baseT/Full
> > >                                             1000baseT/Half 1000baseT/Full
> > >        Link partner advertised pause frame use: Symmetric
> > >        Link partner advertised auto-negotiation: Yes
> > >        Speed: 1000Mb/s
> > >        Duplex: Full
> > >        Port: Twisted Pair
> > >        PHYAD: 1
> > >        Transceiver: internal
> > >        Auto-negotiation: on
> > >        MDI-X: off
> > >        Supports Wake-on: g
> > >        Wake-on: d
> > >        Current message level: 0x000000ff (255)
> > >                               drv probe link timer ifdown ifup rx_err tx_err
> > >        Link detected: yes
> > >
> > > media2 ~]# ethtool em3
> > > Settings for em3:
> > >        Supported ports: [ TP ]
> > >        Supported link modes:   10baseT/Half 10baseT/Full
> > >                                100baseT/Half 100baseT/Full
> > >                                1000baseT/Half 1000baseT/Full
> > >        Supported pause frame use: No
> > >        Supports auto-negotiation: Yes
> > >        Advertised link modes:  10baseT/Half 10baseT/Full
> > >                                100baseT/Half 100baseT/Full
> > >                                1000baseT/Half 1000baseT/Full
> > >        Advertised pause frame use: Symmetric
> > >        Advertised auto-negotiation: Yes
> > >        Link partner advertised link modes:  10baseT/Half 10baseT/Full
> > >                                             100baseT/Half 100baseT/Full
> > >                                             1000baseT/Half 1000baseT/Full
> > >        Link partner advertised pause frame use: Symmetric
> > >        Link partner advertised auto-negotiation: Yes
> > >        Speed: 1000Mb/s
> > >        Duplex: Full
> > >        Port: Twisted Pair
> > >        PHYAD: 1
> > >        Transceiver: internal
> > >        Auto-negotiation: on
> > >        MDI-X: on
> > >        Supports Wake-on: g
> > >        Wake-on: d
> > >        Current message level: 0x000000ff (255)
> > >                               drv probe link timer ifdown ifup rx_err tx_err
> > >        Link detected: yes
> > > > ethtool -S em3
> > >
> > > media1 ~]# ethtool -S em3
> > > NIC statistics:
> > >     rx_octets: 26853394715
> > >     rx_fragments: 0
> > >     rx_ucast_packets: 227879267
> > >     rx_mcast_packets: 84
> > >     rx_bcast_packets: 6
> > >     rx_fcs_errors: 0
> > >     rx_align_errors: 0
> > >     rx_xon_pause_rcvd: 0
> > >     rx_xoff_pause_rcvd: 0
> > >     rx_mac_ctrl_rcvd: 0
> > >     rx_xoff_entered: 0
> > >     rx_frame_too_long_errors: 0
> > >     rx_jabbers: 0
> > >     rx_undersize_packets: 0
> > >     rx_in_length_errors: 0
> > >     rx_out_length_errors: 0
> > >     rx_64_or_less_octet_packets: 0
> > >     rx_65_to_127_octet_packets: 0
> > >     rx_128_to_255_octet_packets: 0
> > >     rx_256_to_511_octet_packets: 0
> > >     rx_512_to_1023_octet_packets: 0
> > >     rx_1024_to_1522_octet_packets: 0
> > >     rx_1523_to_2047_octet_packets: 0
> > >     rx_2048_to_4095_octet_packets: 0
> > >     rx_4096_to_8191_octet_packets: 0
> > >     rx_8192_to_9022_octet_packets: 0
> > >     tx_octets: 1179249531811
> > >     tx_collisions: 0
> > >     tx_xon_sent: 0
> > >     tx_xoff_sent: 0
> > >     tx_flow_control: 0
> > >     tx_mac_errors: 0
> > >     tx_single_collisions: 0
> > >     tx_mult_collisions: 0
> > >     tx_deferred: 0
> > >     tx_excessive_collisions: 0
> > >     tx_late_collisions: 0
> > >     tx_collide_2times: 0
> > >     tx_collide_3times: 0
> > >     tx_collide_4times: 0
> > >     tx_collide_5times: 0
> > >     tx_collide_6times: 0
> > >     tx_collide_7times: 0
> > >     tx_collide_8times: 0
> > >     tx_collide_9times: 0
> > >     tx_collide_10times: 0
> > >     tx_collide_11times: 0
> > >     tx_collide_12times: 0
> > >     tx_collide_13times: 0
> > >     tx_collide_14times: 0
> > >     tx_collide_15times: 0
> > >     tx_ucast_packets: 254815741
> > >     tx_mcast_packets: 8
> > >     tx_bcast_packets: 612
> > >     tx_carrier_sense_errors: 0
> > >     tx_discards: 0
> > >     tx_errors: 0
> > >     dma_writeq_full: 0
> > >     dma_write_prioq_full: 0
> > >     rxbds_empty: 0
> > >     rx_discards: 0
> > >     rx_errors: 0
> > >     rx_threshold_hit: 0
> > >     dma_readq_full: 0
> > >     dma_read_prioq_full: 0
> > >     tx_comp_queue_full: 0
> > >     ring_set_send_prod_index: 0
> > >     ring_status_update: 0
> > >     nic_irqs: 0
> > >     nic_avoided_irqs: 0
> > >     nic_tx_threshold_hit: 0
> > >     mbuf_lwm_thresh_hit: 0
> > >
> > > media2 ~]# ethtool -S em3
> > > NIC statistics:
> > >     rx_octets: 782620145025
> > >     rx_fragments: 0
> > >     rx_ucast_packets: 173893766
> > >     rx_mcast_packets: 0
> > >     rx_bcast_packets: 0
> > >     rx_fcs_errors: 0
> > >     rx_align_errors: 0
> > >     rx_xon_pause_rcvd: 0
> > >     rx_xoff_pause_rcvd: 0
> > >     rx_mac_ctrl_rcvd: 0
> > >     rx_xoff_entered: 0
> > >     rx_frame_too_long_errors: 0
> > >     rx_jabbers: 0
> > >     rx_undersize_packets: 0
> > >     rx_in_length_errors: 0
> > >     rx_out_length_errors: 0
> > >     rx_64_or_less_octet_packets: 0
> > >     rx_65_to_127_octet_packets: 0
> > >     rx_128_to_255_octet_packets: 0
> > >     rx_256_to_511_octet_packets: 0
> > >     rx_512_to_1023_octet_packets: 0
> > >     rx_1024_to_1522_octet_packets: 0
> > >     rx_1523_to_2047_octet_packets: 0
> > >     rx_2048_to_4095_octet_packets: 0
> > >     rx_4096_to_8191_octet_packets: 0
> > >     rx_8192_to_9022_octet_packets: 0
> > >     tx_octets: 17210360858
> > >     tx_collisions: 0
> > >     tx_xon_sent: 0
> > >     tx_xoff_sent: 0
> > >     tx_flow_control: 0
> > >     tx_mac_errors: 0
> > >     tx_single_collisions: 0
> > >     tx_mult_collisions: 0
> > >     tx_deferred: 0
> > >     tx_excessive_collisions: 0
> > >     tx_late_collisions: 0
> > >     tx_collide_2times: 0
> > >     tx_collide_3times: 0
> > >     tx_collide_4times: 0
> > >     tx_collide_5times: 0
> > >     tx_collide_6times: 0
> > >     tx_collide_7times: 0
> > >     tx_collide_8times: 0
> > >     tx_collide_9times: 0
> > >     tx_collide_10times: 0
> > >     tx_collide_11times: 0
> > >     tx_collide_12times: 0
> > >     tx_collide_13times: 0
> > >     tx_collide_14times: 0
> > >     tx_collide_15times: 0
> > >     tx_ucast_packets: 147864937
> > >     tx_mcast_packets: 23
> > >     tx_bcast_packets: 2
> > >     tx_carrier_sense_errors: 0
> > >     tx_discards: 0
> > >     tx_errors: 0
> > >     dma_writeq_full: 0
> > >     dma_write_prioq_full: 0
> > >     rxbds_empty: 0
> > >     rx_discards: 0
> > >     rx_errors: 0
> > >     rx_threshold_hit: 0
> > >     dma_readq_full: 0
> > >     dma_read_prioq_full: 0
> > >     tx_comp_queue_full: 0
> > >     ring_set_send_prod_index: 0
> > >     ring_status_update: 0
> > >     nic_irqs: 0
> > >     nic_avoided_irqs: 0
> > >     nic_tx_threshold_hit: 0
> > >     mbuf_lwm_thresh_hit: 0
> > > > ethtool -i em4
> > >
> > > media1 ~]# ethtool -i em4
> > > driver: tg3
> > > version: 3.136
> > > firmware-version: FFV7.8.16 bc 5720-v1.32
> > > bus-info: 0000:02:00.1
> > > supports-statistics: yes
> > > supports-test: yes
> > > supports-eeprom-access: yes
> > > supports-register-dump: yes
> > > supports-priv-flags: no
> > >
> > > media2 ~]# ethtool -i em4
> > > driver: tg3
> > > version: 3.136
> > > firmware-version: FFV7.8.16 bc 5720-v1.32
> > > bus-info: 0000:02:00.1
> > > supports-statistics: yes
> > > supports-test: yes
> > > supports-eeprom-access: yes
> > > supports-register-dump: yes
> > > supports-priv-flags: no
> > > > ethtool em4
> > >
> > > media1 ~]# ethtool em4
> > > Settings for em4:
> > >        Supported ports: [ TP ]
> > >        Supported link modes:   10baseT/Half 10baseT/Full
> > >                                100baseT/Half 100baseT/Full
> > >                                1000baseT/Half 1000baseT/Full
> > >        Supported pause frame use: No
> > >        Supports auto-negotiation: Yes
> > >        Advertised link modes:  10baseT/Half 10baseT/Full
> > >                                100baseT/Half 100baseT/Full
> > >                                1000baseT/Half 1000baseT/Full
> > >        Advertised pause frame use: Symmetric
> > >        Advertised auto-negotiation: Yes
> > >        Link partner advertised link modes:  10baseT/Half 10baseT/Full
> > >                                             100baseT/Half 100baseT/Full
> > >                                             1000baseT/Half 1000baseT/Full
> > >        Link partner advertised pause frame use: Symmetric
> > >        Link partner advertised auto-negotiation: Yes
> > >        Speed: 1000Mb/s
> > >        Duplex: Full
> > >        Port: Twisted Pair
> > >        PHYAD: 2
> > >        Transceiver: internal
> > >        Auto-negotiation: on
> > >        MDI-X: off
> > >        Supports Wake-on: g
> > >        Wake-on: d
> > >        Current message level: 0x000000ff (255)
> > >                               drv probe link timer ifdown ifup rx_err tx_err
> > >        Link detected: yes
> > >
> > > media2 ~]# ethtool em4
> > > Settings for em4:
> > >        Supported ports: [ TP ]
> > >        Supported link modes:   10baseT/Half 10baseT/Full
> > >                                100baseT/Half 100baseT/Full
> > >                                1000baseT/Half 1000baseT/Full
> > >        Supported pause frame use: No
> > >        Supports auto-negotiation: Yes
> > >        Advertised link modes:  10baseT/Half 10baseT/Full
> > >                                100baseT/Half 100baseT/Full
> > >                                1000baseT/Half 1000baseT/Full
> > >        Advertised pause frame use: Symmetric
> > >        Advertised auto-negotiation: Yes
> > >        Link partner advertised link modes:  10baseT/Half 10baseT/Full
> > >                                             100baseT/Half 100baseT/Full
> > >                                             1000baseT/Half 1000baseT/Full
> > >        Link partner advertised pause frame use: Symmetric
> > >        Link partner advertised auto-negotiation: Yes
> > >        Speed: 1000Mb/s
> > >        Duplex: Full
> > >        Port: Twisted Pair
> > >        PHYAD: 2
> > >        Transceiver: internal
> > >        Auto-negotiation: on
> > >        MDI-X: on
> > >        Supports Wake-on: g
> > >        Wake-on: d
> > >        Current message level: 0x000000ff (255)
> > >                               drv probe link timer ifdown ifup rx_err tx_err
> > >        Link detected: yes
> > > > ethtool -S em4
> > >
> > > media1 ~]# ethtool -S em4
> > > NIC statistics:
> > >     rx_octets: 246361
> > >     rx_fragments: 0
> > >     rx_ucast_packets: 912
> > >     rx_mcast_packets: 8
> > >     rx_bcast_packets: 0
> > >     rx_fcs_errors: 0
> > >     rx_align_errors: 0
> > >     rx_xon_pause_rcvd: 0
> > >     rx_xoff_pause_rcvd: 0
> > >     rx_mac_ctrl_rcvd: 0
> > >     rx_xoff_entered: 0
> > >     rx_frame_too_long_errors: 0
> > >     rx_jabbers: 0
> > >     rx_undersize_packets: 0
> > >     rx_in_length_errors: 0
> > >     rx_out_length_errors: 0
> > >     rx_64_or_less_octet_packets: 0
> > >     rx_65_to_127_octet_packets: 0
> > >     rx_128_to_255_octet_packets: 0
> > >     rx_256_to_511_octet_packets: 0
> > >     rx_512_to_1023_octet_packets: 0
> > >     rx_1024_to_1522_octet_packets: 0
> > >     rx_1523_to_2047_octet_packets: 0
> > >     rx_2048_to_4095_octet_packets: 0
> > >     rx_4096_to_8191_octet_packets: 0
> > >     rx_8192_to_9022_octet_packets: 0
> > >     tx_octets: 760
> > >     tx_collisions: 0
> > >     tx_xon_sent: 0
> > >     tx_xoff_sent: 0
> > >     tx_flow_control: 0
> > >     tx_mac_errors: 0
> > >     tx_single_collisions: 0
> > >     tx_mult_collisions: 0
> > >     tx_deferred: 0
> > >     tx_excessive_collisions: 0
> > >     tx_late_collisions: 0
> > >     tx_collide_2times: 0
> > >     tx_collide_3times: 0
> > >     tx_collide_4times: 0
> > >     tx_collide_5times: 0
> > >     tx_collide_6times: 0
> > >     tx_collide_7times: 0
> > >     tx_collide_8times: 0
> > >     tx_collide_9times: 0
> > >     tx_collide_10times: 0
> > >     tx_collide_11times: 0
> > >     tx_collide_12times: 0
> > >     tx_collide_13times: 0
> > >     tx_collide_14times: 0
> > >     tx_collide_15times: 0
> > >     tx_ucast_packets: 0
> > >     tx_mcast_packets: 8
> > >     tx_bcast_packets: 0
> > >     tx_carrier_sense_errors: 0
> > >     tx_discards: 0
> > >     tx_errors: 0
> > >     dma_writeq_full: 0
> > >     dma_write_prioq_full: 0
> > >     rxbds_empty: 0
> > >     rx_discards: 0
> > >     rx_errors: 0
> > >     rx_threshold_hit: 0
> > >     dma_readq_full: 0
> > >     dma_read_prioq_full: 0
> > >     tx_comp_queue_full: 0
> > >     ring_set_send_prod_index: 0
> > >     ring_status_update: 0
> > >     nic_irqs: 0
> > >     nic_avoided_irqs: 0
> > >     nic_tx_threshold_hit: 0
> > >     mbuf_lwm_thresh_hit: 0
> > >
> > > media2 ~]# ethtool -S em4
> > > NIC statistics:
> > >     rx_octets: 760
> > >     rx_fragments: 0
> > >     rx_ucast_packets: 0
> > >     rx_mcast_packets: 8
> > >     rx_bcast_packets: 0
> > >     rx_fcs_errors: 0
> > >     rx_align_errors: 0
> > >     rx_xon_pause_rcvd: 0
> > >     rx_xoff_pause_rcvd: 0
> > >     rx_mac_ctrl_rcvd: 0
> > >     rx_xoff_entered: 0
> > >     rx_frame_too_long_errors: 0
> > >     rx_jabbers: 0
> > >     rx_undersize_packets: 0
> > >     rx_in_length_errors: 0
> > >     rx_out_length_errors: 0
> > >     rx_64_or_less_octet_packets: 0
> > >     rx_65_to_127_octet_packets: 0
> > >     rx_128_to_255_octet_packets: 0
> > >     rx_256_to_511_octet_packets: 0
> > >     rx_512_to_1023_octet_packets: 0
> > >     rx_1024_to_1522_octet_packets: 0
> > >     rx_1523_to_2047_octet_packets: 0
> > >     rx_2048_to_4095_octet_packets: 0
> > >     rx_4096_to_8191_octet_packets: 0
> > >     rx_8192_to_9022_octet_packets: 0
> > >     tx_octets: 247887
> > >     tx_collisions: 0
> > >     tx_xon_sent: 0
> > >     tx_xoff_sent: 0
> > >     tx_flow_control: 0
> > >     tx_mac_errors: 0
> > >     tx_single_collisions: 0
> > >     tx_mult_collisions: 0
> > >     tx_deferred: 0
> > >     tx_excessive_collisions: 0
> > >     tx_late_collisions: 0
> > >     tx_collide_2times: 0
> > >     tx_collide_3times: 0
> > >     tx_collide_4times: 0
> > >     tx_collide_5times: 0
> > >     tx_collide_6times: 0
> > >     tx_collide_7times: 0
> > >     tx_collide_8times: 0
> > >     tx_collide_9times: 0
> > >     tx_collide_10times: 0
> > >     tx_collide_11times: 0
> > >     tx_collide_12times: 0
> > >     tx_collide_13times: 0
> > >     tx_collide_14times: 0
> > >     tx_collide_15times: 0
> > >     tx_ucast_packets: 927
> > >     tx_mcast_packets: 8
> > >     tx_bcast_packets: 0
> > >     tx_carrier_sense_errors: 0
> > >     tx_discards: 0
> > >     tx_errors: 0
> > >     dma_writeq_full: 0
> > >     dma_write_prioq_full: 0
> > >     rxbds_empty: 0
> > >     rx_discards: 0
> > >     rx_errors: 0
> > >     rx_threshold_hit: 0
> > >     dma_readq_full: 0
> > >     dma_read_prioq_full: 0
> > >     tx_comp_queue_full: 0
> > >     ring_set_send_prod_index: 0
> > >     ring_status_update: 0
> > >     nic_irqs: 0
> > >     nic_avoided_irqs: 0
> > >     nic_tx_threshold_hit: 0
> > >     mbuf_lwm_thresh_hit: 0
> > > > Of course in time the problem happens.
> > > > > Kind regards
> > > > > Ingo
> > > > >
> > > > > This is the used network config of team1:
> > > > > media1:
> > > > > cat /etc/sysconfig/network-scripts/ifcfg-team1_slave_0
> > > > > # Generated by parse-kickstart
> > > > > NAME=team1 slave 0
> > > > > TEAM_MASTER=team1
> > > > > DEVICETYPE=TeamPort
> > > > > DEVICE=em3
> > > > > ONBOOT=yes
> > > > > UUID=b7026c5b-e9cd-457c-93d2-a2799361ed90
> > > > >
> > > > > cat /etc/sysconfig/network-scripts/ifcfg-team1_slave_1
> > > > > # Generated by parse-kickstart
> > > > > NAME=team1 slave 1
> > > > > TEAM_MASTER=team1
> > > > > DEVICETYPE=TeamPort
> > > > > DEVICE=em4
> > > > > ONBOOT=yes
> > > > > UUID=fbe5bc56-95f5-4f9e-a5ea-ab6b6f5e5b50
> > > > >
> > > > > cat /etc/sysconfig/network-scripts/ifcfg-team1
> > > > > # Generated by parse-kickstart
> > > > > UUID=4e0090cd-32d3-4c5b-be23-9f53083da4dd
> > > > > NAME="Team connection team1"
> > > > > TEAM_CONFIG="{"runner": {"name": "roundrobin"}}"
> > > > > GATEWAY=
> > > > > IPV6_AUTOCONF=yes
> > > > > BOOTPROTO=none
> > > > > DEVICE=team1
> > > > > MTU=9000
> > > > > TYPE=Team
> > > > > ONBOOT=yes
> > > > > IPV6INIT=yes
> > > > > DEVICETYPE=Team
> > > > > IPADDR0=192.168.101.31
> > > > > PREFIX0=24
> > > > > DEFROUTE=yes
> > > > > IPV4_FAILURE_FATAL=no
> > > > > IPV6_DEFROUTE=yes
> > > > > IPV6_PEERDNS=yes
> > > > > IPV6_PEERROUTES=yes
> > > > > IPV6_FAILURE_FATAL=no
> > > > >
> > > > > media2:
> > > > > cat /etc/sysconfig/network-scripts/ifcfg-team1_slave_0
> > > > > # Generated by parse-kickstart
> > > > > NAME=team1 slave 0
> > > > > TEAM_MASTER=team1
> > > > > DEVICETYPE=TeamPort
> > > > > DEVICE=em3
> > > > > ONBOOT=yes
> > > > > UUID=bfd8e77e-7eb8-47ea-ab7e-95286e073202
> > > > >
> > > > > cat /etc/sysconfig/network-scripts/ifcfg-team1_slave_1
> > > > > # Generated by parse-kickstart
> > > > > NAME=team1 slave 1
> > > > > TEAM_MASTER=team1
> > > > > DEVICETYPE=TeamPort
> > > > > DEVICE=em4
> > > > > ONBOOT=yes
> > > > > UUID=93c5dfd8-3b81-4d1e-8ba9-c46f8daceb1a
> > > > >
> > > > > cat /etc/sysconfig/network-scripts/ifcfg-team1
> > > > > # Generated by parse-kickstart
> > > > > UUID=79c843db-996c-41e4-9ee5-2a8f3da244ed
> > > > > NAME="Team connection team1"
> > > > > TEAM_CONFIG="{"runner": {"name": "roundrobin"}}"
> > > > > GATEWAY=
> > > > > IPV6_AUTOCONF=yes
> > > > > BOOTPROTO=none
> > > > > DEVICE=team1
> > > > > MTU=9000
> > > > > TYPE=Team
> > > > > ONBOOT=yes
> > > > > IPV6INIT=yes
> > > > > DEVICETYPE=Team
> > > > > IPADDR0=192.168.101.32
> > > > > PREFIX0=24
> > > > > DEFROUTE=yes
> > > > > IPV4_FAILURE_FATAL=no
> > > > > IPV6_DEFROUTE=yes
> > > > > IPV6_PEERDNS=yes
> > > > > IPV6_PEERROUTES=yes
> > > > > IPV6_FAILURE_FATAL=no