Team setup with two network devices

[Wolf Humble]

Hi,

I have a server with two network devices: eth1 and eth2.

I have been given 4 IP addresses via a router that I don't have any control
over.
Two of them go only to eth1, and the other two only to eth2.
Each IP address has its own gateway.

The devices, ip addresses and gateways are as following:

    eth1 – ip1: aaa.189.aaa.254 - GW: aaa.189.aaa.253
    eth1 – ip2: bbb.143.bbb.214 - GW: bbb.143.bbb.213

    eth2 – ip3: ccc.170.ccc.202  - GW: ccc.170.ccc.201
    eth2 – ip4: ddd.180.ddd.101 - GW: ddd.180.ddd.100


Two or more of the IPs will be used for virtual machines.

I have set my network up in the following way:
===========================
[#] cat ifcfg-eno1
NAME=eno1
DEVICE=eno1
DEVICETYPE=TeamPort
ONBOOT=yes
TEAM_MASTER=team1
TEAM_PORT_CONFIG="{'prio': 9}"

[#] cat ifcfg-eno2
NAME=eno2
DEVICE=eno2
DEVICETYPE=TeamPort
ONBOOT=yes
TEAM_MASTER=team1
TEAM_PORT_CONFIG="{'prio': 10}"

[#] cat  ifcfg-team1
BRIDGE=br0
DEVICE=team1
DEVICETYPE=Team
ONBOOT=yes
BOOTPROTO=none
TEAM_CONFIG="{'runner': {'name': 'lacp', 'active':
true, 'fast_rate': true,
'tx_hash': ['eth', 'ipv4', 'ipv6']},'link_watch':
{'name':
'ethtool'},"ports":{"eno1":{},"eno2":{}}}"
PROXY_METHOD=none
BROWSER_ONLY=no
DEFROUTE=yes
IPV4_FAILURE_FATAL=no
IPV6INIT=no
NAME="Team team1"
UUID=702de3eb-2e80-897c-fd52-cd0494dd8123

[#] cat ifcfg-br0
DEVICE=br0
STP=yes
BRIDGING_OPTS=priority=32768
TYPE=Bridge
BOOTPROTO=none
#
IPADDR=aaa.189.aaa.254
PREFIX=30
GATEWAY=aaa.189.aaa.253
DNS1=aaa.bbb.ccc.aaa
DNS2=aaa.bbb.ccc.bbb
#
IPADDR1=bbb.143.bbb.214
PREFIX1=30
GATEWAY1=bbb.143.bbb.213
#
IPADDR2=ccc.170.ccc.202
PREFIX2=30
GATEWAY2=ccc.170.ccc.201
#
IPADDR3=ddd.180.ddd.101
PREFIX3=30
GATEWAY3=ddd.180.ddd.100
#
DEFROUTE=yes
IPV4_FAILURE_FATAL=no
IPV6INIT=yes
IPV6_AUTOCONF=yes
IPV6_DEFROUTE=yes
IPV6_FAILURE_FATAL=no
NAME=br0
NM_CONTROLLED=YES
ONBOOT=yes
PROXY_METHOD=none
BROWSER_ONLY=no
UUID=d2d68553-f97e-7549-7a26-b34a26f29318"
===========================

[Question 1]
It took my some time to figure out that LACP, requires a specific setup on
the router (that I don't have control over), but the setup above was
answering for all my IP's.

I see via:
[#] teamnl team1 options
...
mode roundrobin
...

=> So is roundrobin the default mode, and takes over even when LACP is set
up in TEAM_CONFIG?

[Question 2]
=> Knowing that eth1 only answers requests to the two firsts IP addresses,
and eth2 the two last IP addresses, is mode roundrobin the only sensible
mode/runner in my case?
(Activebackup seems to only answer requests to the IPs connected to either
eth1 OR eth2, not both. Loadbalance seems have the same challenge . . . )

[Question 3]
=> If roundrobin is the most sensible mode, would it be enough to set up
'ifcfg-team1', like this?
[#] cat ifcfg-team1
...
TEAM_CONFIG='{"runner": {"name": "roundrobin"}}'
...

=> OR Would I need to set up
"ports":{"eno1":{},"eno2":{}} inside the
TEAM_CONFIG?

[Question 4]
=> Does it make sense to use a team in my case, or could I just drop the
team and connected the bridge(s) to eno1 and eno2 instead?
E.g: What do I gain by setting up a team in my case?

Thanks!

Best regards
Wolf

[Jamie Bainbridge]

On 25 January 2018 at 09:58, Wolf Humble <wolfhumble(a)gmail.com&gt; wrote:
>  Hi,
>
>  I have a server with two network devices: eth1 and eth2.
>
>  I have been given 4 IP addresses via a router that I don't have any control
>  over.
>  Two of them go only to eth1, and the other two only to eth2.
>  Each IP address has its own gateway.
>
>  The devices, ip addresses and gateways are as following:
>
>      eth1 – ip1: aaa.189.aaa.254 - GW: aaa.189.aaa.253
>      eth1 – ip2: bbb.143.bbb.214 - GW: bbb.143.bbb.213
>
>      eth2 – ip3: ccc.170.ccc.202  - GW: ccc.170.ccc.201
>      eth2 – ip4: ddd.180.ddd.101 - GW: ddd.180.ddd.100
>
>
>  Two or more of the IPs will be used for virtual machines.
>
>  I have set my network up in the following way:
>  ===========================
>  [#] cat ifcfg-eno1
>  NAME=eno1
>  DEVICE=eno1
>  DEVICETYPE=TeamPort
>  ONBOOT=yes
>  TEAM_MASTER=team1
>  TEAM_PORT_CONFIG="{'prio': 9}"
>
>  [#] cat ifcfg-eno2
>  NAME=eno2
>  DEVICE=eno2
>  DEVICETYPE=TeamPort
>  ONBOOT=yes
>  TEAM_MASTER=team1
>  TEAM_PORT_CONFIG="{'prio': 10}"
>
>  [#] cat  ifcfg-team1
>  BRIDGE=br0
>  DEVICE=team1
>  DEVICETYPE=Team
>  ONBOOT=yes
>  BOOTPROTO=none
>  TEAM_CONFIG="{'runner': {'name': 'lacp', 'active':
> true, 'fast_rate': true,
>  'tx_hash': ['eth', 'ipv4', 'ipv6']},'link_watch':
> {'name':
>  'ethtool'},"ports":{"eno1":{},"eno2":{}}}"
>  PROXY_METHOD=none
>  BROWSER_ONLY=no
>  DEFROUTE=yes
>  IPV4_FAILURE_FATAL=no
>  IPV6INIT=no
>  NAME="Team team1"
>  UUID=702de3eb-2e80-897c-fd52-cd0494dd8123
>
>  [#] cat ifcfg-br0
>  DEVICE=br0
>  STP=yes
>  BRIDGING_OPTS=priority=32768
>  TYPE=Bridge
>  BOOTPROTO=none
>  #
>  IPADDR=aaa.189.aaa.254
>  PREFIX=30
>  GATEWAY=aaa.189.aaa.253
>  DNS1=aaa.bbb.ccc.aaa
>  DNS2=aaa.bbb.ccc.bbb
>  #
>  IPADDR1=bbb.143.bbb.214
>  PREFIX1=30
>  GATEWAY1=bbb.143.bbb.213
>  #
>  IPADDR2=ccc.170.ccc.202
>  PREFIX2=30
>  GATEWAY2=ccc.170.ccc.201
>  #
>  IPADDR3=ddd.180.ddd.101
>  PREFIX3=30
>  GATEWAY3=ddd.180.ddd.100
>  #
>  DEFROUTE=yes
>  IPV4_FAILURE_FATAL=no
>  IPV6INIT=yes
>  IPV6_AUTOCONF=yes
>  IPV6_DEFROUTE=yes
>  IPV6_FAILURE_FATAL=no
>  NAME=br0
>  NM_CONTROLLED=YES
>  ONBOOT=yes
>  PROXY_METHOD=none
>  BROWSER_ONLY=no
>  UUID=d2d68553-f97e-7549-7a26-b34a26f29318"
>  ===========================
>
>  [Question 1]
>  It took my some time to figure out that LACP, requires a specific setup on
>  the router (that I don't have control over), but the setup above was
>  answering for all my IP's.
>
>  I see via:
>  [#] teamnl team1 options
>  ...
>  mode roundrobin
>  ...
>
>  => So is roundrobin the default mode, and takes over even when LACP is set
>  up in TEAM_CONFIG?
>
>  [Question 2]
>  => Knowing that eth1 only answers requests to the two firsts IP addresses,
>  and eth2 the two last IP addresses, is mode roundrobin the only sensible
>  mode/runner in my case?
>  (Activebackup seems to only answer requests to the IPs connected to either
>  eth1 OR eth2, not both. Loadbalance seems have the same challenge . . . )
>
>  [Question 3]
>  => If roundrobin is the most sensible mode, would it be enough to set up
>  'ifcfg-team1', like this?
>  [#] cat ifcfg-team1
>  ...
>  TEAM_CONFIG='{"runner": {"name": "roundrobin"}}'
>  ...
>
>  => OR Would I need to set up
> "ports":{"eno1":{},"eno2":{}} inside the
>  TEAM_CONFIG?
>
>  [Question 4]
>  => Does it make sense to use a team in my case, or could I just drop the
>  team and connected the bridge(s) to eno1 and eno2 instead?
>  E.g: What do I gain by setting up a team in my case? 
As far as I understand, your interfaces are both in different
broadcast domains. You cannot join these two together with any link
aggregation like teaming or bonding.

If you need bridging (to join VMs or containers to each network) then
set them up separately. eth1 in br1, this bridge has access to
networks aaa and bbb. eth2 in br2, this bridge has access to networks
ccc and ddd.

Jamie

[Wolf Humble]

Hi,

On 25 January 2018 at 09:58, Wolf Humble <wolfhumble(a)gmail.com&gt; wrote:
--snippet--
> > The devices, ip addresses and gateways are as following:
> >
> >     eth1 – ip1: aaa.189.aaa.254 - GW: aaa.189.aaa.253
> >     eth1 – ip2: bbb.143.bbb.214 - GW: bbb.143.bbb.213
> >
> >    eth2 – ip3: ccc.170.ccc.202  - GW: ccc.170.ccc.201
> >    eth2 – ip4: ddd.180.ddd.101 - GW: ddd.180.ddd.100 
--/snippet--

On Thu, Jan 25, 2018 at 8:03 AM, Jamie Bainbridge
<jamie.bainbridge(a)gmail.com&gt; wrote:
--snippet--
>  As far as I understand, your interfaces are both in different
>  broadcast domains. You cannot join these two together with any link
>  aggregation like teaming or bonding.
>
>  If you need bridging (to join VMs or containers to each network) then
>  set them up separately. eth1 in br1, this bridge has access to
>  networks aaa and bbb. eth2 in br2, this bridge has access to networks
>  ccc and ddd. 
--/snippet--

Thanks!

On more question about teaming  -- that wouldn't work in my case since
I want ALL IPs to answer at the same time -- but just curious:
It seems like teaming with activebackup would work even for interfaces
that are in different broadcast domains, as long as you don't mind
that only IPs connected to eth1 will answer when eth1 and eth2 are up,
and only IPs connected to eth2 willl answer when eth1 is down. (From
what I remember when testing the activebackup mode earlier in my
setup).
Do I understand that correctly?

Thanks again,

Wolf

[Jamie Bainbridge]

On 26 January 2018 at 01:13, Wolf Humble <wolfhumble(a)gmail.com&gt; wrote:
>  On more question about teaming  -- that wouldn't work in my case
> since
>  I want ALL IPs to answer at the same time -- but just curious:
>  It seems like teaming with activebackup would work even for interfaces
>  that are in different broadcast domains, as long as you don't mind
>  that only IPs connected to eth1 will answer when eth1 and eth2 are up,
>  and only IPs connected to eth2 willl answer when eth1 is down. (From
>  what I remember when testing the activebackup mode earlier in my
>  setup).
>  Do I understand that correctly? 
That's almost correct.

It will depend on which is the *active slave* at the time, which is a
separate thing to linkstate. You can have both slave links up and
switch which is the active one. You can change whether a preferred
slave always takes over when it's up. See the prio/sticky options on
the active-backup runner. Of course a slave must actually be up to be
selected as the active slave!

Once the active slave is decided, the system will only receive traffic
from one set of IPs (either aaa+bbb or ccc+ddd) depending on which
slave is up, hence the system will only reply to those IPs speaking in
the active slave. You're right that the incoming traffic will just
happen to be from different subnets.

However, there's nothing to stop the system trying to use its own IP
in the wrong network, say sending ARP requests for other hosts in
network aaa out the team. Whether anything responds to those requests
will depend on which slave is up.

It would also be tricky to manage which default gateway is in use
depending on which slave is up, so it would be a pretty terrible
solution for failover.

With your interface config, you may be more interested in policy
routing and multiple upstream gateways:
http://lartc.org/howto/lartc.rpdb.multiple-links.html

Jamie