I am setting up a box with FC6 + xen with FC6 for the DomU guests. I have a server with 4 gigabit ports and I want to use 802.1Q vlans on one of the gig ports. I have the VLANs configured in Dom0 without any addresses attached to them. How do I make the VLANs available to the guests? I want to make 4 VLANs available to each guest. I have thought of a few alternatives but I cannot get any to work.
The easy way; I have installed FC6 with 802.1Q and 4 guests. It works very nicely. For each guest do: 1 - You need to enable vlans /etc/sysconfig/network. NETWORKING=yes VLAN=yes NETWORKING_IPV6=no HOSTNAME=xenbox2.example.com GATEWAY=192.168.42.1
2 - create a subinterface by file editting /etc/sysconfig/network-scripts/ifcfg-eth0.42: DEVICE=eth0.42 BOOTPROTO=static ONBOOT=yes IPADDR=192.168.42.102 NETMASK=255.255.255.0 TYPE=Ethernet
3 - depending on whether you want untagged packets on the same interface, you need to edit the /etc/.../ifcfg-eth0 file too. (note: the parent interface must be up for the tagged interface to work.) The lines commented out will assign an IP address to the untagged interface: DEVICE=eth0 BOOTPROTO=none ONBOOT=yes TYPE=Ethernet # BOOTPROTO=static # IPADDR=172.16.42.102 # NETMASK=255.255.255.0
4 - make sure that your /etc/hosts setup is correct. If you have DNS available 24/7, then only define localhost[.localdomain} as 127.0.0.1 and not the hostname itself.
When you restart the network in the guest, then your guest will default to use the tagged interface vlan 42 (above example). If you want the untagged interface as default, then you can change the gateway and assign an IP address to it.
On the host, you need to do a similar thing. Create a subinterface eth0.42 and assign an IP address to it. Once that is done, you can talk to your guests via dot1Q from your host.
Beware of iptables! If you have a firewall setup, then you need to make sure that it is done right. Doing it right, though, is a though job with dot1Q in place. I have the firewall completely disabled on the host and guests. My firewall is placed physically separated on another box.
- Bridge the tagged 802.1Q stream from the Gig port through to the
guests and configure eth0.x. on the guests
This would be above scenario.
- Create a bridge on Dom0 for each VLAN so that they appear as
eth0...eth3 on the guests.
The xenbrX interfaces on the host already transport all packets because they are L2 bridges (see brctl). The bridge does not care whether the packets are .1Q tagged or not. The only thing you need is to instruct the kernel to look for/act on specifics by creating (sub-)interfaces.
Now, if you want the guests not to know that they are connected to a VLAN, then you need a brigde-interface to do (un-)tagging for you. That means that you need to create a dot1Q subinterface on the host's physical peth[0-3] and add the subinterface to a new bridge.
Normal scenario: peth0 -> xenbr0 -> vif0.0(host eth0) vif1.0(guest1 eth0)
Dot1Q scenario: peth0.42 -> xenbr42 -> vif1.0(guest1 eth0)
i.e.: # brctl addbr xenbr42 # ip link set xenbr42 up # vconfig add peth0 42 # ip link set peth0.42 up # brctl addif xenbr42 peth0.42
In the VM config you have now something like: vif = [ 'mac=00:16:3e:11:22:33, bridge=xenbr42', ]
The VM's eth0 interface is created as part of bridge that will fowrard to a subinterface. The kernel will now do the tagging/untagging (note: slow) and the VM cannot see that it is attached to a VLAN.
Theoretically it should work, but I was not able to make this scenario work as expected though. I suspect that the hosts peth0 -> eth0 bridge is interfering (which has a tagged interface too in my setup). Or maybe my fingers were too fast at typing and I forgot a simple thing...
The xen documentation is minimal for configuring VLANs. Is there a (FC6) supported way of configuring them?
Vlans is a no-issue for xen. It is layered in the network and xen only passes the packets.
Thanks Bertho, I managed to get it to work something along the lines of your first method before I read your email. I was also unsuccessful using your second method.
I wanted a Dom0 host to be accessable only on an administration network with the guests accessing their relevant VLANs
I used one Gig (eth0) real interface as an 802.1Q trunk with the Dom0 having no interfaces mapped to it. The Dom0 was connected (without VLANS) to the administration network via eth1. I had to modify /etc/xen/xend-config.sxp to bridge eth0 otherwise it would bridge the active device (eth1)
eth0 --> xenbr0 -> Dom1 --> VLAN1 |-> VLAN2 |->... eth1-->Dom0
(network-script 'network-bridge netdev=eth0')
The guests were then set up as normal hosts connected to an 802.1Q trunk network as you detailed.
I found that if I used VLANs on the Dom0 on eth0 then the trunk would not pass through to the guests, Hence why Dom0 is not using VLANs on eth0. I did not do much research to discover why except that the instant I removed the VLANs from Dom0 eth0 the VLAN networks started working on the guest.
My initial email came about because I was expecting to have to plumb the individual VLANs through to the guests as you also failed to do. This is much like VMWare ESX works.
Regards Darryl
Bertho Stultiens wrote:
I am setting up a box with FC6 + xen with FC6 for the DomU guests. I have a server with 4 gigabit ports and I want to use 802.1Q vlans on one of the gig ports. I have the VLANs configured in Dom0 without any addresses attached to them. How do I make the VLANs available to the guests? I want to make 4 VLANs available to each guest. I have thought of a few alternatives but I cannot get any to work.
The easy way; I have installed FC6 with 802.1Q and 4 guests. It works very nicely. For each guest do: 1 - You need to enable vlans /etc/sysconfig/network. NETWORKING=yes VLAN=yes NETWORKING_IPV6=no HOSTNAME=xenbox2.example.com GATEWAY=192.168.42.1
2 - create a subinterface by file editting /etc/sysconfig/network-scripts/ifcfg-eth0.42: DEVICE=eth0.42 BOOTPROTO=static ONBOOT=yes IPADDR=192.168.42.102 NETMASK=255.255.255.0 TYPE=Ethernet
3 - depending on whether you want untagged packets on the same interface, you need to edit the /etc/.../ifcfg-eth0 file too. (note: the parent interface must be up for the tagged interface to work.) The lines commented out will assign an IP address to the untagged interface: DEVICE=eth0 BOOTPROTO=none ONBOOT=yes TYPE=Ethernet # BOOTPROTO=static # IPADDR=172.16.42.102 # NETMASK=255.255.255.0
4 - make sure that your /etc/hosts setup is correct. If you have DNS available 24/7, then only define localhost[.localdomain} as 127.0.0.1 and not the hostname itself.
When you restart the network in the guest, then your guest will default to use the tagged interface vlan 42 (above example). If you want the untagged interface as default, then you can change the gateway and assign an IP address to it.
On the host, you need to do a similar thing. Create a subinterface eth0.42 and assign an IP address to it. Once that is done, you can talk to your guests via dot1Q from your host.
Beware of iptables! If you have a firewall setup, then you need to make sure that it is done right. Doing it right, though, is a though job with dot1Q in place. I have the firewall completely disabled on the host and guests. My firewall is placed physically separated on another box.
- Bridge the tagged 802.1Q stream from the Gig port through to the
guests and configure eth0.x. on the guests
This would be above scenario.
- Create a bridge on Dom0 for each VLAN so that they appear as
eth0...eth3 on the guests.
The xenbrX interfaces on the host already transport all packets because they are L2 bridges (see brctl). The bridge does not care whether the packets are .1Q tagged or not. The only thing you need is to instruct the kernel to look for/act on specifics by creating (sub-)interfaces.
Now, if you want the guests not to know that they are connected to a VLAN, then you need a brigde-interface to do (un-)tagging for you. That means that you need to create a dot1Q subinterface on the host's physical peth[0-3] and add the subinterface to a new bridge.
Normal scenario: peth0 -> xenbr0 -> vif0.0(host eth0) vif1.0(guest1 eth0)
Dot1Q scenario: peth0.42 -> xenbr42 -> vif1.0(guest1 eth0)
i.e.: # brctl addbr xenbr42 # ip link set xenbr42 up # vconfig add peth0 42 # ip link set peth0.42 up # brctl addif xenbr42 peth0.42
In the VM config you have now something like: vif = [ 'mac=00:16:3e:11:22:33, bridge=xenbr42', ]
The VM's eth0 interface is created as part of bridge that will fowrard to a subinterface. The kernel will now do the tagging/untagging (note: slow) and the VM cannot see that it is attached to a VLAN.
Theoretically it should work, but I was not able to make this scenario work as expected though. I suspect that the hosts peth0 -> eth0 bridge is interfering (which has a tagged interface too in my setup). Or maybe my fingers were too fast at typing and I forgot a simple thing...
The xen documentation is minimal for configuring VLANs. Is there a (FC6) supported way of configuring them?
Vlans is a no-issue for xen. It is layered in the network and xen only passes the packets.
__________________________________________________________________________________ DISCLAIMER
The contents of this electronic message and any attachments are intended only for the addressee and may contain legally privileged, personal, sensitive or confidential information. If you are not the intended addressee, and have received this email, any transmission, distribution, downloading, printing or photocopying of the contents of this message or attachments is strictly prohibited. Any legal privilege or confidentiality attached to this message and attachments is not waived, lost or destroyed by reason of delivery to any person other than intended addressee. If you have received this message and are not the intended addressee you should notify the sender by return email and destroy all copies of the message and any attachments. Unless expressly attributed, the views expressed in this email do not necessarily represent the views of the company.
Darryl Bond wrote:
I managed to get it to work something along the lines of your first method before I read your email. I was also unsuccessful using your second method.
It still eludes me why the second method does not work, but the first seems to be ok too. Users cannot change set setup, so there is no real risk involved.
[snip]
I found that if I used VLANs on the Dom0 on eth0 then the trunk would not pass through to the guests, Hence why Dom0 is not using VLANs on eth0. I did not do much research to discover why except that the instant I removed the VLANs from Dom0 eth0 the VLAN networks started working on the guest.
You have to take care here because in Dom0 you have ethX and pethX. Vlans on ethX(dom0) will work whereas those on peth0 apparently do not as expected. My current setup is like this:
peth0 -> xenbr0 -> vif0.0 -> eth0(Dom0)+IP-> eth0.42+IP -> vif1.0 -> eth0(Guest1) -> eth0.42+IP -> vif2.0 -> eth0(Guest2) -> eth0.42+IP -> ...
Dom0 is part of vlan 42 and my management (no routing between them enabled though). The uplink peth0 is a trunk with management untagged and vlan 42 tagged.
My guess why the other setup has problems lies in several parts. Firstly, the management MAC address (default fe:ff:ff:ff:ff:ff) of the second bridge must be different. I forgot to change that and have had problems with that before with STP enabled (bridge can otherwise not be uniquely identified). And secondly, juggling with peth0.X can only be done after xen has remapped the interfaces. The remapping occurs after the original network is brought up. I need to go through the xen scripts to get that right, I guess.
I'll have to do some more experimenting to get this pinned down because it actually is interesting to have bridges that can swap/map/juggle vlan tags.
On Thu, Nov 23, 2006 at 09:45:20AM +0100, Bertho Stultiens wrote:
Darryl Bond wrote:
I managed to get it to work something along the lines of your first method before I read your email. I was also unsuccessful using your second method.
It still eludes me why the second method does not work, but the first seems to be ok too. Users cannot change set setup, so there is no real risk involved.
[snip]
I found that if I used VLANs on the Dom0 on eth0 then the trunk would not pass through to the guests, Hence why Dom0 is not using VLANs on eth0. I did not do much research to discover why except that the instant I removed the VLANs from Dom0 eth0 the VLAN networks started working on the guest.
You have to take care here because in Dom0 you have ethX and pethX. Vlans on ethX(dom0) will work whereas those on peth0 apparently do not as expected. My current setup is like this:
peth0 -> xenbr0 -> vif0.0 -> eth0(Dom0)+IP-> eth0.42+IP -> vif1.0 -> eth0(Guest1) -> eth0.42+IP -> vif2.0 -> eth0(Guest2) -> eth0.42+IP -> ...
Dom0 is part of vlan 42 and my management (no routing between them enabled though). The uplink peth0 is a trunk with management untagged and vlan 42 tagged.
My guess why the other setup has problems lies in several parts. Firstly, the management MAC address (default fe:ff:ff:ff:ff:ff) of the second bridge must be different. I forgot to change that and have had problems with that before with STP enabled (bridge can otherwise not be uniquely identified). And secondly, juggling with peth0.X can only be done after xen has remapped the interfaces. The remapping occurs after the original network is brought up. I need to go through the xen scripts to get that right, I guess.
I'll have to do some more experimenting to get this pinned down because it actually is interesting to have bridges that can swap/map/juggle vlan tags.
Hi!
Did you figure out the correct way to do this? (set up bridge per vlan interface in dom0, and export/bridge only untagged frames to domU's).
-- Pasi
Pasi Kärkkäinen wrote:
Did you figure out the correct way to do this? (set up bridge per vlan interface in dom0, and export/bridge only untagged frames to domU's).
Actually, I never did go further investigating.
On Mon, Feb 26, 2007 at 09:11:09AM +0100, Bertho Stultiens wrote:
Pasi Kärkkäinen wrote:
Did you figure out the correct way to do this? (set up bridge per vlan interface in dom0, and export/bridge only untagged frames to domU's).
Actually, I never did go further investigating.
OK, thanks for the reply.
I'll see what I can figure out..
-- Pasi
-
Bertho Stultiens -
Darryl Bond -
Pasi Kärkkäinen