In the last few days I've noticed network connectivity issues from multiple virtual machines (fedora, centos, winxp) running on a fedora 12 host. What seemed odd was that I could ping by host name, showing that both the basic network functionality as well as DNS was working. What was failing was browser access to any site outside my own subnet.
I'm reasonably sure the issue is the MTU setting for my host bridge (br0) interface. It currently shows:
# ifconfig br0 [...] UP BROADCAST RUNNING PROMISC MULTICAST MTU:576 Metric:1 [...]
I would have expected MTU:1500. In fact most of the examples I've found show other people with br0 having MTU=1500.
My short term workaround has been to manually set MTU to 576 in each of my VMs. This works, but I'm wondering:
1) Have others seen this? 2) Is there any way to manually increase MTU for the bridge interface?
WRT #2, I tried:
ifconfig br0 mtu 1500
and get this error:
SIOCSIFMTU: Invalid argument
I also tried adding MTU=1500 to ifcfg-br0. No joy.
Any ideas?
Thanks,
Joe
On 03/11/2010 02:14 PM, Joe Conway wrote:
In the last few days I've noticed network connectivity issues from multiple virtual machines (fedora, centos, winxp) running on a fedora 12 host. What seemed odd was that I could ping by host name, showing that both the basic network functionality as well as DNS was working. What was failing was browser access to any site outside my own subnet.
I'm reasonably sure the issue is the MTU setting for my host bridge (br0) interface. It currently shows:
# ifconfig br0 [...] UP BROADCAST RUNNING PROMISC MULTICAST MTU:576 Metric:1 [...]
I would have expected MTU:1500. In fact most of the examples I've found show other people with br0 having MTU=1500.
My short term workaround has been to manually set MTU to 576 in each of my VMs. This works, but I'm wondering:
- Have others seen this?
- Is there any way to manually increase MTU for the bridge interface?
WRT #2, I tried:
ifconfig br0 mtu 1500and get this error:
SIOCSIFMTU: Invalid argumentI also tried adding MTU=1500 to ifcfg-br0. No joy.
Any ideas?
"brctl show" will show how all the bridges are built. Check all interfaces under "ifconfig -a" and see if any of the participants in the bridge have a small MTU. IIRC, the smallest MTU will be propagated to the bridge so it doesn't overrun the least-capable interface. ---------------------------------------------------------------------- - Rick Stevens, Systems Engineer, C2 Hosting ricks@nerd.com - - AIM/Skype: therps2 ICQ: 22643734 Yahoo: origrps2 - - - - When you don't know what to do, walk fast and look worried. - ----------------------------------------------------------------------
On 03/11/2010 02:31 PM, Rick Stevens wrote:
On 03/11/2010 02:14 PM, Joe Conway wrote:
In the last few days I've noticed network connectivity issues from multiple virtual machines (fedora, centos, winxp) running on a fedora 12 host. What seemed odd was that I could ping by host name, showing that both the basic network functionality as well as DNS was working. What was failing was browser access to any site outside my own subnet.
I'm reasonably sure the issue is the MTU setting for my host bridge (br0) interface. It currently shows:
# ifconfig br0 [...] UP BROADCAST RUNNING PROMISC MULTICAST MTU:576 Metric:1 [...]
I would have expected MTU:1500. In fact most of the examples I've found show other people with br0 having MTU=1500.
My short term workaround has been to manually set MTU to 576 in each of my VMs. This works, but I'm wondering:
- Have others seen this?
- Is there any way to manually increase MTU for the bridge interface?
WRT #2, I tried:
ifconfig br0 mtu 1500and get this error:
SIOCSIFMTU: Invalid argumentI also tried adding MTU=1500 to ifcfg-br0. No joy.
Any ideas?
"brctl show" will show how all the bridges are built.
# brctl show bridge name bridge id STP enabled interfaces br0 8000.18a9051f09f0 no eth0 vnet0 virbr0 8000.000000000000 yes
This is pretty much as I expected based on research...
Check all interfaces under "ifconfig -a" and see if any of the participants in the bridge have a small MTU. IIRC, the smallest MTU will be propagated to the bridge so it doesn't overrun the least-capable interface.
# ifconfig -a br0 UP BROADCAST RUNNING PROMISC MULTICAST MTU:576 Metric:1 eth0 UP BROADCAST RUNNING PROMISC MULTICAST MTU:1500 Metric:1 lo UP LOOPBACK RUNNING MTU:16436 Metric:1 sit0 NOARP MTU:1480 Metric:1 virbr0 UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1 vnet0 UP BROADCAST RUNNING MULTICAST MTU:576 Metric:1 wlan0 BROADCAST MULTICAST MTU:1500 Metric:1
I had failed to notice vnet0 before -- that seems to be the culprit! Now the next question is why did it change (or get added?), and how do I fix it? Thanks for getting me to the next step!
Joe
On 03/11/2010 02:52 PM, Joe Conway wrote:
On 03/11/2010 02:31 PM, Rick Stevens wrote:
Check all interfaces under "ifconfig -a" and see if any of the participants in the bridge have a small MTU. IIRC, the smallest MTU will be propagated to the bridge so it doesn't overrun the least-capable interface.
# ifconfig -a br0 UP BROADCAST RUNNING PROMISC MULTICAST MTU:576 Metric:1 eth0 UP BROADCAST RUNNING PROMISC MULTICAST MTU:1500 Metric:1 lo UP LOOPBACK RUNNING MTU:16436 Metric:1 sit0 NOARP MTU:1480 Metric:1 virbr0 UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1 vnet0 UP BROADCAST RUNNING MULTICAST MTU:576 Metric:1 wlan0 BROADCAST MULTICAST MTU:1500 Metric:1
I had failed to notice vnet0 before -- that seems to be the culprit! Now the next question is why did it change (or get added?), and how do I fix it? Thanks for getting me to the next step!
Interestingly after a reboot:
-------------------- # brctl show bridge name bridge id STP enabled interfaces br0 8000.18a9051f09f0 no eth0
# ifconfig br0 UP BROADCAST RUNNING MULTICAST MTU:576 Metric:1 eth0 UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1 lo UP LOOPBACK RUNNING MTU:16436 Metric:1 --------------------
So what is causing br0 MTU == 576 here?
Now, if I start up a VM:
-------------------- # brctl show bridge name bridge id STP enabled interfaces br0 8000.18a9051f09f0 no eth0 vnet0
# ifconfig br0 UP BROADCAST RUNNING MULTICAST MTU:576 Metric:1 eth0 UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1 lo UP LOOPBACK RUNNING MTU:16436 Metric:1 vnet0 UP BROADCAST RUNNING MULTICAST MTU:576 Metric:1
# ifconfig vnet0 mtu 1500
# ifconfig br0 UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1 eth0 UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1 lo UP LOOPBACK RUNNING MTU:16436 Metric:1 vnet0 UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1 --------------------
So I can at least work around the issue this way after starting the VM, but still don't understand the root cause.
Joe
On Thu, 11 Mar 2010 16:20:04 -0800 Joe Conway wrote:
So I can at least work around the issue this way after starting the VM, but still don't understand the root cause.
The vnet0 may be coming from the "default" network that libvirt provides. If you are using bridging for everything, you can eradicate the default network like so:
virsh net-destroy default virsh net-undefine default
Getting rid of the default also gets rid of the dnsmasq process libvirt starts and other cruft you don't need for pure bridging (like insane junk added to iptables).
On 03/11/2010 04:37 PM, Tom Horsley wrote:
On Thu, 11 Mar 2010 16:20:04 -0800 Joe Conway wrote:
So I can at least work around the issue this way after starting the VM, but still don't understand the root cause.
The vnet0 may be coming from the "default" network that libvirt provides. If you are using bridging for everything, you can eradicate the default network like so:
That's the weird thing -- br0 has mtu == 576 before vnet0 even exists, right after booting up, and even though the only existing interface attached, eth0, has mtu == 1500.
Also worth noting is this all worked perfectly up until 2 days ago, and the only system changes have been due to "yum update" (I looked at /var/log/yum.log but nothing jumped out as an obvious cause).
virsh net-destroy default virsh net-undefine default
Getting rid of the default also gets rid of the dnsmasq process libvirt starts and other cruft you don't need for pure bridging (like insane junk added to iptables).
Unfortunately:
virsh # net-destroy default error: failed to get network 'default' error: Network not found: no network with matching name 'default'
virsh # net-list Name State Autostart -----------------------------------------
(There are none listed)
Joe
On 03/11/2010 04:20 PM, Joe Conway wrote:
Interestingly after a reboot:
# brctl show bridge name bridge id STP enabled interfaces br0 8000.18a9051f09f0 no eth0
# ifconfig br0 UP BROADCAST RUNNING MULTICAST MTU:576 Metric:1 eth0 UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1 lo UP LOOPBACK RUNNING MTU:16436 Metric:1
So what is causing br0 MTU == 576 here?
Following up, I tested on a Fedora 12 VM that was pretty much a clean install from 2 months or so ago. First I changed the network settings to match my host machine, and rebooted the VM. This is what I saw:
---------------- # ifconfig br0 UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1 eth0 UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1 lo UP LOOPBACK RUNNING MTU:16436 Metric:1 ----------------
Looks good so far.
Next I did a "yum update" and rebooted. Keep in mind that this is in a VM, so I am not running libvirt, qemu, kvm, etc. at all. Now I see:
---------------- # ifconfig br0 UP BROADCAST RUNNING MULTICAST MTU:576 Metric:1 eth0 UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1 lo UP LOOPBACK RUNNING MTU:16436 Metric:1 ----------------
Here you can see br0 MTU = 576 even though eth0 MTU is 1500. So the issue seems to be generic with some recent package update.
Unfortunately since the VM had been idle for a long while, there were hundreds of packages updated.
Anyone have any idea what package would determine the default MTU for a bridged network device?
Thanks,
Joe
On Fri, 12 Mar 2010 13:27:31 -0800 Joe Conway wrote:
Anyone have any idea what package would determine the default MTU for a bridged network device?
I can't help with that, but my fully updated f12 system shows 1500 MTU for my br0 with a windows XP VM running, so whatever is going on doesn't seem to affect me, so there is something specific on your machine maybe?
On 03/12/2010 01:35 PM, Tom Horsley wrote:
On Fri, 12 Mar 2010 13:27:31 -0800 Joe Conway wrote:
Anyone have any idea what package would determine the default MTU for a bridged network device?
I can't help with that, but my fully updated f12 system shows 1500 MTU for my br0 with a windows XP VM running, so whatever is going on doesn't seem to affect me, so there is something specific on your machine maybe?
I guess there must be ;-)
But it now shows up on both my host machine and independently on a fedora 12 virtual machine. Both are x86_64. Are you running x86_64?
Thanks,
Joe
On 03/12/2010 01:59 PM, Tom Horsley wrote:
On Fri, 12 Mar 2010 13:53:18 -0800 Joe Conway wrote:
But it now shows up on both my host machine and independently on a fedora 12 virtual machine. Both are x86_64. Are you running x86_64?
Yep. 64 bit fedora 12. Just started another ubuntu virtual machine, br0 is still at 1500.
A bit more sleuthing and I found that the culprit is dhclient. I am using a dynamically assigned address (pinned to a static IP at my dhcp server), and I bet you are not. A downgrade makes the problem go away:
yum downgrade dhclient dhclient-4.1.0p1-12.fc12.x86_64 [...] Running Transaction Installing : 12:dhclient-4.1.0p1-12.fc12.x86_64 Cleanup : 12:dhclient-4.1.1-9.fc12.x86_64 [...] # /etc/init.d/network restart [...] # ifconfig br0 [...] UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
So I guess it is a change in behavior and probably a bug in dhclient. I'll file a bug report...
Joe
On Wed, 17 Mar 2010 17:28:42 -0700 Joe Conway wrote:
A bit more sleuthing and I found that the culprit is dhclient. I am using a dynamically assigned address (pinned to a static IP at my dhcp server), and I bet you are not. A downgrade makes the problem go away:
My host machine is using a static IP, but it is the dhcp server for my other machines (including my virtual machines), so I guess dhclient isn't involved on my host, just in the guests.
On 03/17/2010 05:44 PM, Tom Horsley wrote:
On Wed, 17 Mar 2010 17:28:42 -0700 Joe Conway wrote:
A bit more sleuthing and I found that the culprit is dhclient. I am using a dynamically assigned address (pinned to a static IP at my dhcp server), and I bet you are not. A downgrade makes the problem go away:
My host machine is using a static IP, but it is the dhcp server for my other machines (including my virtual machines), so I guess dhclient isn't involved on my host, just in the guests.
Yes, I'm using dhclient for my host as well. I guess I could just switch to a static IP for my host as a workaround.
Interestingly I cannot even find dhclient among the listed components here:
https://bugzilla.redhat.com/enter_bug.cgi?format=guided&product=Fedora
Joe
On Wed, 17 Mar 2010 17:49:18 -0700 Joe Conway wrote:
Interestingly I cannot even find dhclient among the listed components here:
https://bugzilla.redhat.com/enter_bug.cgi?format=guided&product=Fedora
Little known trivia: The components are based on the source rpm names, so the component search goes:
which dhclient (see /sbin/dhclient) rpm -q -i -f /sbin/dhclient (see Source RPM: dhcp-4.1.1-9.fc12.src.rpm)
so the component is really dhcp.
On 03/17/2010 06:02 PM, Tom Horsley wrote:
On Wed, 17 Mar 2010 17:49:18 -0700 Joe Conway wrote:
Interestingly I cannot even find dhclient among the listed components here:
https://bugzilla.redhat.com/enter_bug.cgi?format=guided&product=Fedora
Little known trivia: The components are based on the source rpm names, so the component search goes:
Good to know -- thanks!
Joe
On 03/17/2010 05:49 PM, Joe Conway wrote:
On 03/17/2010 05:44 PM, Tom Horsley wrote:
On Wed, 17 Mar 2010 17:28:42 -0700 Joe Conway wrote:
A bit more sleuthing and I found that the culprit is dhclient. I am using a dynamically assigned address (pinned to a static IP at my dhcp server), and I bet you are not. A downgrade makes the problem go away:
My host machine is using a static IP, but it is the dhcp server for my other machines (including my virtual machines), so I guess dhclient isn't involved on my host, just in the guests.
Yes, I'm using dhclient for my host as well. I guess I could just switch to a static IP for my host as a workaround.
For the sake of posterity, an explanation of the root cause and a solution can be found here:
https://bugzilla.redhat.com/show_bug.cgi?id=574629
Joe
On 03/18/2010 11:53 AM, Joe Conway wrote:
On 03/17/2010 05:49 PM, Joe Conway wrote:
On 03/17/2010 05:44 PM, Tom Horsley wrote:
On Wed, 17 Mar 2010 17:28:42 -0700 Joe Conway wrote:
A bit more sleuthing and I found that the culprit is dhclient. I am using a dynamically assigned address (pinned to a static IP at my dhcp server), and I bet you are not. A downgrade makes the problem go away:
My host machine is using a static IP, but it is the dhcp server for my other machines (including my virtual machines), so I guess dhclient isn't involved on my host, just in the guests.
Yes, I'm using dhclient for my host as well. I guess I could just switch to a static IP for my host as a workaround.
For the sake of posterity, an explanation of the root cause and a solution can be found here:
Thank you _ever_ so much for posting that. I've been trying to figure out why on earth my Comcast broadband connection kept getting stuck with 576 for the MTU. The Comcast DHCP server must have been recalling that from an old lease, because my FC-12 laptop would get an MTU of 1500 but my FC-12 desktop kept getting 576 (connecting each directly to the modem).
Happy at MTU=1500 now.
-
Joe Conway -
Rick Stevens -
Robert Nichols -
Tom Horsley -
Tom Horsley