Re: [Fedora-xen] VLANs on DomU domains

[Bertho Stultiens]

>  I am setting up a box with FC6 + xen with FC6 for the DomU guests.
>  I have a server with 4 gigabit ports and I want to use 802.1Q vlans on
>  one of the gig ports.
>  I have the VLANs configured in Dom0 without any addresses attached to
>  them.
>  How do I make the VLANs available to the guests? I want to make 4
>  VLANs available to each guest. I have thought of a few alternatives
>  but I cannot get any to work. 
The easy way; I have installed FC6 with 802.1Q and 4 guests. It works
very nicely. For each guest do:
1 - You need to enable vlans /etc/sysconfig/network.
	NETWORKING=yes
	VLAN=yes
	NETWORKING_IPV6=no
	HOSTNAME=xenbox2.example.com
	GATEWAY=192.168.42.1

2 - create a subinterface by file editting
/etc/sysconfig/network-scripts/ifcfg-eth0.42:
	DEVICE=eth0.42
	BOOTPROTO=static
	ONBOOT=yes
	IPADDR=192.168.42.102
	NETMASK=255.255.255.0
	TYPE=Ethernet

3 - depending on whether you want untagged packets on the same
interface, you need to edit the /etc/.../ifcfg-eth0 file too. (note: the
parent interface must be up for the tagged interface to work.) The lines
commented out will assign an IP address to the untagged interface:
	DEVICE=eth0
	BOOTPROTO=none
	ONBOOT=yes
	TYPE=Ethernet
	# BOOTPROTO=static
	# IPADDR=172.16.42.102
	# NETMASK=255.255.255.0

4 - make sure that your /etc/hosts setup is correct. If you have DNS
available 24/7, then only define localhost[.localdomain} as 127.0.0.1
and not the hostname itself.

When you restart the network in the guest, then your guest will default
to use the tagged interface vlan 42 (above example). If you want the
untagged interface as default, then you can change the gateway and
assign an IP address to it.

On the host, you need to do a similar thing. Create a subinterface
eth0.42 and assign an IP address to it. Once that is done, you can talk
to your guests via dot1Q from your host.

Beware of iptables! If you have a firewall setup, then you need to make
sure that it is done right. Doing it right, though, is a though job with
dot1Q in place. I have the firewall completely disabled on the host and
guests. My firewall is placed physically separated on another box.


>  1. Bridge the tagged 802.1Q stream from the Gig port through to the
>  guests and configure eth0.x. on the guests 
This would be above scenario.


>  2. Create a bridge on Dom0 for each VLAN so that they appear as
>  eth0...eth3 on the guests. 
The xenbrX interfaces on the host already transport all packets because
they are L2 bridges (see brctl). The bridge does not care whether the
packets are .1Q tagged or not. The only thing you need is to instruct
the kernel to look for/act on specifics by creating (sub-)interfaces.

Now, if you want the guests not to know that they are connected to a
VLAN, then you need a brigde-interface to do (un-)tagging for you. That
means that you need to create a dot1Q subinterface on the host's
physical peth[0-3] and add the subinterface to a new bridge.

Normal scenario:
peth0  -> xenbr0 -> vif0.0(host eth0)
                    vif1.0(guest1 eth0)

Dot1Q scenario:
peth0.42 -> xenbr42 -> vif1.0(guest1 eth0)

i.e.:
# brctl addbr xenbr42
# ip link set xenbr42 up
# vconfig add peth0 42
# ip link set peth0.42 up
# brctl addif xenbr42 peth0.42

In the VM config you have now something like:
vif = [ 'mac=00:16:3e:11:22:33, bridge=xenbr42', ]

The VM's eth0 interface is created as part of bridge that will fowrard
to a subinterface. The kernel will now do the tagging/untagging (note:
slow) and the VM cannot see that it is attached to a VLAN.

Theoretically it should work, but I was not able to make this scenario
work as expected though. I suspect that the hosts peth0 -> eth0 bridge
is interfering (which has a tagged interface too in my setup). Or maybe
my fingers were too fast at typing and I forgot a simple thing...



>  The xen documentation is minimal for configuring VLANs. Is there a
>  (FC6) supported way of configuring them? 
Vlans is a no-issue for xen. It is layered in the network and xen only
passes the packets.


-- 
Greetings Bertho


Bertho Stultiens
Senior Systems Manager
Mobilethink A/S

VLANs on DomU domains

[Darryl Bond]

Thanks Bertho,
I managed to get it to work something along the lines of your first 
method before I read your email. I was also unsuccessful using your 
second method.

I wanted a Dom0 host to be accessable only on an administration network 
with the guests accessing their relevant VLANs

I used one Gig (eth0) real interface as an 802.1Q trunk with the Dom0 
having no interfaces mapped to it.
The Dom0 was connected (without VLANS) to the administration network via 
eth1. I had to modify /etc/xen/xend-config.sxp to bridge eth0 otherwise 
it would bridge the active device (eth1)

eth0 --> xenbr0 -> Dom1 --> VLAN1
                                         |-> VLAN2
                                         |->...
eth1-->Dom0

(network-script 'network-bridge netdev=eth0')

The guests were then set up as normal hosts connected to an 802.1Q trunk 
network as you detailed.


I found that if I used VLANs on the Dom0 on eth0 then the trunk would 
not pass through to the guests,  Hence why Dom0 is not using VLANs on eth0.
I did not do much research to discover why except that the instant I 
removed the VLANs from Dom0 eth0 the VLAN networks started working on 
the guest.

My initial email came about because I was expecting to have to plumb the 
individual VLANs through to the guests as you also failed to do. This is 
much like VMWare ESX works.

Regards
Darryl


Bertho Stultiens wrote:
> > I am setting up a box with FC6 + xen with FC6 for the DomU
> guests.
> > I have a server with 4 gigabit ports and I want to use 802.1Q vlans on
> > one of the gig ports.
> > I have the VLANs configured in Dom0 without any addresses attached to
> > them.
> > How do I make the VLANs available to the guests? I want to make 4
> > VLANs available to each guest. I have thought of a few alternatives
> > but I cannot get any to work.
> >     
>
>  The easy way; I have installed FC6 with 802.1Q and 4 guests. It works
>  very nicely. For each guest do:
>  1 - You need to enable vlans /etc/sysconfig/network.
>  	NETWORKING=yes
>  	VLAN=yes
>  	NETWORKING_IPV6=no
>  	HOSTNAME=xenbox2.example.com
>  	GATEWAY=192.168.42.1
>
>  2 - create a subinterface by file editting
>  /etc/sysconfig/network-scripts/ifcfg-eth0.42:
>  	DEVICE=eth0.42
>  	BOOTPROTO=static
>  	ONBOOT=yes
>  	IPADDR=192.168.42.102
>  	NETMASK=255.255.255.0
>  	TYPE=Ethernet
>
>  3 - depending on whether you want untagged packets on the same
>  interface, you need to edit the /etc/.../ifcfg-eth0 file too. (note: the
>  parent interface must be up for the tagged interface to work.) The lines
>  commented out will assign an IP address to the untagged interface:
>  	DEVICE=eth0
>  	BOOTPROTO=none
>  	ONBOOT=yes
>  	TYPE=Ethernet
>  	# BOOTPROTO=static
>  	# IPADDR=172.16.42.102
>  	# NETMASK=255.255.255.0
>
>  4 - make sure that your /etc/hosts setup is correct. If you have DNS
>  available 24/7, then only define localhost[.localdomain} as 127.0.0.1
>  and not the hostname itself.
>
>  When you restart the network in the guest, then your guest will default
>  to use the tagged interface vlan 42 (above example). If you want the
>  untagged interface as default, then you can change the gateway and
>  assign an IP address to it.
>
>  On the host, you need to do a similar thing. Create a subinterface
>  eth0.42 and assign an IP address to it. Once that is done, you can talk
>  to your guests via dot1Q from your host.
>
>  Beware of iptables! If you have a firewall setup, then you need to make
>  sure that it is done right. Doing it right, though, is a though job with
>  dot1Q in place. I have the firewall completely disabled on the host and
>  guests. My firewall is placed physically separated on another box.
>
>
>    
> > 1. Bridge the tagged 802.1Q stream from the Gig port through to the
> > guests and configure eth0.x. on the guests
> >     
>
>  This would be above scenario.
>
>
>    
> > 2. Create a bridge on Dom0 for each VLAN so that they appear as
> > eth0...eth3 on the guests.
> >     
>
>  The xenbrX interfaces on the host already transport all packets because
>  they are L2 bridges (see brctl). The bridge does not care whether the
>  packets are .1Q tagged or not. The only thing you need is to instruct
>  the kernel to look for/act on specifics by creating (sub-)interfaces.
>
>  Now, if you want the guests not to know that they are connected to a
>  VLAN, then you need a brigde-interface to do (un-)tagging for you. That
>  means that you need to create a dot1Q subinterface on the host's
>  physical peth[0-3] and add the subinterface to a new bridge.
>
>  Normal scenario:
>  peth0  -> xenbr0 -> vif0.0(host eth0)
>                      vif1.0(guest1 eth0)
>
>  Dot1Q scenario:
>  peth0.42 -> xenbr42 -> vif1.0(guest1 eth0)
>
>  i.e.:
>  # brctl addbr xenbr42
>  # ip link set xenbr42 up
>  # vconfig add peth0 42
>  # ip link set peth0.42 up
>  # brctl addif xenbr42 peth0.42
>
>  In the VM config you have now something like:
>  vif = [ 'mac=00:16:3e:11:22:33, bridge=xenbr42', ]
>
>  The VM's eth0 interface is created as part of bridge that will fowrard
>  to a subinterface. The kernel will now do the tagging/untagging (note:
>  slow) and the VM cannot see that it is attached to a VLAN.
>
>  Theoretically it should work, but I was not able to make this scenario
>  work as expected though. I suspect that the hosts peth0 -> eth0 bridge
>  is interfering (which has a tagged interface too in my setup). Or maybe
>  my fingers were too fast at typing and I forgot a simple thing...
>
>
>
>    
> > The xen documentation is minimal for configuring VLANs. Is there a
> > (FC6) supported way of configuring them?
> >     
>
>  Vlans is a no-issue for xen. It is layered in the network and xen only
>  passes the packets.
>
>
>    
__________________________________________________________________________________
                                DISCLAIMER

The contents of this electronic message and any attachments are intended only
for the addressee and may contain legally privileged, personal, sensitive or
confidential information. If you are not the intended addressee, and have
received this email, any transmission, distribution, downloading, printing or
photocopying of the contents of this message or attachments is strictly
prohibited. Any legal privilege or confidentiality attached to this message and
attachments is not waived, lost or destroyed by reason of delivery to any
person other than intended addressee. If you have received this message and
are not the intended addressee you should notify the sender by return email and
destroy all copies of the message and any attachments.  Unless expressly
attributed, the views expressed in this email do not necessarily represent the
views of the company.

VLANs on DomU domains

[Bertho Stultiens]

Darryl Bond wrote:
>  I managed to get it to work something along the lines of your first
>  method before I read your email. I was also unsuccessful using your
>  second method. 
It still eludes me why the second method does not work, but the first
seems to be ok too. Users cannot change set setup, so there is no real
risk involved.


[snip]
>  I found that if I used VLANs on the Dom0 on eth0 then the trunk
> would
>  not pass through to the guests,  Hence why Dom0 is not using VLANs on eth0.
>  I did not do much research to discover why except that the instant I
>  removed the VLANs from Dom0 eth0 the VLAN networks started working on
>  the guest. 
You have to take care here because in Dom0 you have ethX and pethX.
Vlans on ethX(dom0) will work whereas those on peth0 apparently do not
as expected. My current setup is like this:

peth0 -> xenbr0 -> vif0.0 -> eth0(Dom0)+IP-> eth0.42+IP
                -> vif1.0 -> eth0(Guest1) -> eth0.42+IP
                -> vif2.0 -> eth0(Guest2) -> eth0.42+IP
                -> ...

Dom0 is part of vlan 42 and my management (no routing between them
enabled though). The uplink peth0 is a trunk with management untagged
and vlan 42 tagged.

My guess why the other setup has problems lies in several parts.
Firstly, the management MAC address (default fe:ff:ff:ff:ff:ff) of the
second bridge must be different. I forgot to change that and have had
problems with that before with STP enabled (bridge can otherwise not be
uniquely identified). And secondly, juggling with peth0.X can only be
done after xen has remapped the interfaces. The remapping occurs after
the original network is brought up. I need to go through the xen scripts
to get that right, I guess.

I'll have to do some more experimenting to get this pinned down because
it actually is interesting to have bridges that can swap/map/juggle vlan
tags.

-- 
Greetings Bertho


Bertho Stultiens
Senior Systems Manager
Mobilethink A/S

VLANs on DomU domains

[Pasi Kärkkäinen]

On Thu, Nov 23, 2006 at 09:45:20AM +0100, Bertho Stultiens wrote:
>  Darryl Bond wrote:
>  > I managed to get it to work something along the lines of your first
>  > method before I read your email. I was also unsuccessful using your
>  > second method.
>  
>  It still eludes me why the second method does not work, but the first
>  seems to be ok too. Users cannot change set setup, so there is no real
>  risk involved.
>  
>  
>  [snip]
>  > I found that if I used VLANs on the Dom0 on eth0 then the trunk would
>  > not pass through to the guests,  Hence why Dom0 is not using VLANs on eth0.
>  > I did not do much research to discover why except that the instant I
>  > removed the VLANs from Dom0 eth0 the VLAN networks started working on
>  > the guest.
>  
>  You have to take care here because in Dom0 you have ethX and pethX.
>  Vlans on ethX(dom0) will work whereas those on peth0 apparently do not
>  as expected. My current setup is like this:
>  
>  peth0 -> xenbr0 -> vif0.0 -> eth0(Dom0)+IP-> eth0.42+IP
>                  -> vif1.0 -> eth0(Guest1) -> eth0.42+IP
>                  -> vif2.0 -> eth0(Guest2) -> eth0.42+IP
>                  -> ...
>  
>  Dom0 is part of vlan 42 and my management (no routing between them
>  enabled though). The uplink peth0 is a trunk with management untagged
>  and vlan 42 tagged.
>  
>  My guess why the other setup has problems lies in several parts.
>  Firstly, the management MAC address (default fe:ff:ff:ff:ff:ff) of the
>  second bridge must be different. I forgot to change that and have had
>  problems with that before with STP enabled (bridge can otherwise not be
>  uniquely identified). And secondly, juggling with peth0.X can only be
>  done after xen has remapped the interfaces. The remapping occurs after
>  the original network is brought up. I need to go through the xen scripts
>  to get that right, I guess.
>  
>  I'll have to do some more experimenting to get this pinned down because
>  it actually is interesting to have bridges that can swap/map/juggle vlan
>  tags.
>   
Hi!

Did you figure out the correct way to do this? (set up bridge per vlan interface in
dom0, and export/bridge only untagged frames to domU's).

-- Pasi

VLANs on DomU domains

[Bertho Stultiens]

Pasi Kärkkäinen wrote:
>  Did you figure out the correct way to do this? (set up bridge per
> vlan interface in
>  dom0, and export/bridge only untagged frames to domU's). 
Actually, I never did go further investigating.


-- 
Greetings Bertho


Bertho Stultiens
Senior Systems Manager
Mobilethink A/S

VLANs on DomU domains

[Pasi Kärkkäinen]

On Mon, Feb 26, 2007 at 09:11:09AM +0100, Bertho Stultiens wrote:
>  Pasi Kärkkäinen wrote:
>  > Did you figure out the correct way to do this? (set up bridge per vlan interface in
>  > dom0, and export/bridge only untagged frames to domU's).
>  
>  Actually, I never did go further investigating.
>  
>   
OK, thanks for the reply.

I'll see what I can figure out..

-- Pasi