The getting started guide is missing the assignment to the em1 interface
in the initial ip address range setup. This results in flakey network
behavior when VMs are terminated.
Analysis of problem:
With default guide settings, assume two VMs are run via euca-run-instances.
The first VM will get a vnet with a mac address. It will then be added
directly into the bridge. The bridge will then take the MAC of the
vnet0 interface. Then the second VM will start and get a vnet1 with an
assigned MAC address.
Killing the first VM that was started (which has a MAC address equal to
the br0) will cause deletion of vnet0 and cause the bridge driver to
flush the arp table and choose a new bridge interface. This results in
disconnection of the remaining VM for 30-60 seconds. A simple solution
is to specify a physical bridge interface during bridge creation ie:
sudo nova-manage network create markmc 10.0.0.0/24 --bridge=br0
This is shown for the compute nodes, but not the controller nodes.
With this setting, br0 will always be assigned the MAC address of the
bridge interface. Then terminating VMs will not result in the arp table
being flushed, and loss of network connectivity.
This is shown in the "controller" section of the getting started guide
when using compute nodes, but not in the initial setup.
Note if vnet0 VM is killed, then the br0 will adopt the mac address of
vnet1. Then if vnet1 is killed the network will stall for 30-60 seconds
and lose the arp table (while killing the VM associated with vnet0 will
have no impact on the network).
This does raise the question what to do when em1 is taken out of
service. This will cause the bridge driver to flush the arp table and
pick a new device (vnet0) resulting in again flakey behavior with the
bridges when stopping virtual machines.
Just learning the ins and outs of bridges today I don't have a good
recommendation for a solution to this "em1 taken out of service" problem.
One idea I had was changing nova to add a "dummy" vnet0 before adding
em1 or any of the VM vnets (during br0 creation in nova-network). I'm
not sure if this would cause traffic to bridge out of the physical
interface properly, but it would fix the network connectivity issues and
arp table killing that occurs while a new bridge MAC is chosen.
Now off to PTO.