Adding a compute node and quantum configurations

Hello,

The quantum network side is not covered in 
https://fedoraproject.org/wiki/Getting_started_with_OpenStack_EPEL#Adding_a_... 
so I'm trying to clarify this.

On a second node I've followed the instructions in the wiki minus the 
network obviously since I'm running quantum+openvswitch (hybrid). I'm 
applying some of the instructions found here:
https://fedoraproject.org/wiki/QA:Testcase_Quantum_V2#Setup but when 
I'm doing "quantum-server-setup --plugin openvswitch" it's asking me to 
install mysql, the scripts should be modified to also ask if there's a 
DB running elsewhere (i.e. on the controller).

Anyway, I let it do its thing, install mysql and setup nova.conf, then 
modified nova.conf with the quantum section from the controller's 
nova.conf. Now my question is - in this case quantum_admin_auth_url and 
quantum_url on the node should they point to localhost or to the 
controller's address?

I'm thinking quantum_admin_auth_url should point to the keystone 
service on the controller's IP and quantum_url should be 
http://localhost:9696/?

Additionally:

quantum.conf - qpid_hostname should point to the controller's IP
quantum/api-paste.ini - [filter:authtoken] auth_host should be the 
controller
quantum/l3_agent.ini - auth_url should point to the controller
plugins/openvswitch/ovs_quantum_plugin.ini - sql_connection should be 
using the controller's mysql server

Let me know if any of that is wrong or I missed some other settings 
that should be notified.

Regards,
Lucian

Hi,
Please see my inline.
Thanks
Gary

On 11/20/2012 01:14 PM, Nux! wrote:
> Hello,
>
> The quantum network side is not covered in 
> https://fedoraproject.org/wiki/Getting_started_with_OpenStack_EPEL#Adding_a_... 
> so I'm trying to clarify this.
>
> On a second node I've followed the instructions in the wiki minus the 
> network obviously since I'm running quantum+openvswitch (hybrid). I'm 
> applying some of the instructions found here:
> https://fedoraproject.org/wiki/QA:Testcase_Quantum_V2#Setup but when 
> I'm doing "quantum-server-setup --plugin openvswitch" it's asking me 
> to install mysql, the scripts should be modified to also ask if 
> there's a DB running elsewhere (i.e. on the controller).

You do not need to install and run the Quantum service again. This only 
needs to be run once on a "controller node"

On the compute node you need only to run the Quantum agent. Please note 
that there is a utility script that you can use: quantum-node-setup

Thanks
Gary
> Anyway, I let it do its thing, install mysql and setup nova.conf, then 
> modified nova.conf with the quantum section from the controller's 
> nova.conf. Now my question is - in this case quantum_admin_auth_url 
> and quantum_url on the node should they point to localhost or to the 
> controller's address?
>
> I'm thinking quantum_admin_auth_url should point to the keystone 
> service on the controller's IP and quantum_url should be 
> http://localhost:9696/?
>
> Additionally:
>
> quantum.conf - qpid_hostname should point to the controller's IP

This is done by the abovementioned script
> quantum/api-paste.ini - [filter:authtoken] auth_host should be the 
> controller

No need
> quantum/l3_agent.ini - auth_url should point to the controller

No need - you only need one l3 agent
> plugins/openvswitch/ovs_quantum_plugin.ini - sql_connection should be 
> using the controller's mysql server

This is tread by the script. Please note that there is no access from 
the agent to the quantum database. This is done via the message broker.
> Let me know if any of that is wrong or I missed some other settings 
> that should be notified.
>
> Regards,
> Lucian

On 20.11.2012 11:26, Gary Kotton wrote:
> Hi,
> Please see my inline.
> Thanks
> Gary

Wow, that's great!

So to conclude, when adding a compute node we need to run the following 
services:
openstack-nova-compute
openstack-nova-api
quantum-openvswitch-agent

Looking at the script I see it's not dealing with chkconfig so we must 
do it manually.
> On 11/20/2012 01:14 PM, Nux! wrote:
> > Hello,
> >
> > The quantum network side is not covered in 
> > https://fedoraproject.org/wiki/Getting_started_with_OpenStack_EPEL#Adding_a_... 
> > so I'm trying to clarify this.
> >
> > On a second node I've followed the instructions in the wiki minus 
> > the network obviously since I'm running quantum+openvswitch (hybrid). 
> > I'm applying some of the instructions found here:
> > https://fedoraproject.org/wiki/QA:Testcase_Quantum_V2#Setup but when 
> > I'm doing "quantum-server-setup --plugin openvswitch" it's asking me 
> > to install mysql, the scripts should be modified to also ask if 
> > there's a DB running elsewhere (i.e. on the controller).
>
> You do not need to install and run the Quantum service again. This
> only needs to be run once on a "controller node"
>
> On the compute node you need only to run the Quantum agent. Please
> note that there is a utility script that you can use:
> quantum-node-setup
>
> Thanks
> Gary
> > Anyway, I let it do its thing, install mysql and setup nova.conf, 
> > then modified nova.conf with the quantum section from the controller's 
> > nova.conf. Now my question is - in this case quantum_admin_auth_url 
> > and quantum_url on the node should they point to localhost or to the 
> > controller's address?
> >
> > I'm thinking quantum_admin_auth_url should point to the keystone 
> > service on the controller's IP and quantum_url should be 
> > http://localhost:9696/?
> >
> > Additionally:
> >
> > quantum.conf - qpid_hostname should point to the controller's IP
>
> This is done by the abovementioned script
> > quantum/api-paste.ini - [filter:authtoken] auth_host should be the 
> > controller
>
> No need
> > quantum/l3_agent.ini - auth_url should point to the controller
>
> No need - you only need one l3 agent
> > plugins/openvswitch/ovs_quantum_plugin.ini - sql_connection should 
> > be using the controller's mysql server
>
> This is tread by the script. Please note that there is no access from
> the agent to the quantum database. This is done via the message
> broker.
> > Let me know if any of that is wrong or I missed some other settings 
> > that should be notified.
> >
> > Regards,
> > Lucian
>
> cloud mailing list
> [e-mail]
> https://admin.fedoraproject.org/mailman/listinfo/cloud

On 11/20/2012 01:47 PM, Nux! wrote:
> On 20.11.2012 11:26, Gary Kotton wrote:
> > Hi,
> > Please see my inline.
> > Thanks
> > Gary
>
> Wow, that's great!
>
> So to conclude, when adding a compute node we need to run the 
> following services:
> openstack-nova-compute
> openstack-nova-api

I am not 100% sure about the nova api. Maybe others from Nova can chime 
in here.
> quantum-openvswitch-agent

You will also need to ensure that you OVS is up and running on the host
> Looking at the script I see it's not dealing with chkconfig so we must 
> do it manually.

Yes (this is similar with the other services)
> > On 11/20/2012 01:14 PM, Nux! wrote:
> > > Hello,
> > >
> > > The quantum network side is not covered in 
> > > https://fedoraproject.org/wiki/Getting_started_with_OpenStack_EPEL#Adding_a_... 
> > > so I'm trying to clarify this.
> > >
> > > On a second node I've followed the instructions in the wiki minus 
> > > the network obviously since I'm running quantum+openvswitch 
> > > (hybrid). I'm applying some of the instructions found here:
> > > https://fedoraproject.org/wiki/QA:Testcase_Quantum_V2#Setup but when 
> > > I'm doing "quantum-server-setup --plugin openvswitch" it's asking me 
> > > to install mysql, the scripts should be modified to also ask if 
> > > there's a DB running elsewhere (i.e. on the controller).
> >
> > You do not need to install and run the Quantum service again. This
> > only needs to be run once on a "controller node"
> >
> > On the compute node you need only to run the Quantum agent. Please
> > note that there is a utility script that you can use:
> > quantum-node-setup
> >
> > Thanks
> > Gary
> > > Anyway, I let it do its thing, install mysql and setup nova.conf, 
> > > then modified nova.conf with the quantum section from the 
> > > controller's nova.conf. Now my question is - in this case 
> > > quantum_admin_auth_url and quantum_url on the node should they point 
> > > to localhost or to the controller's address?
> > >
> > > I'm thinking quantum_admin_auth_url should point to the keystone 
> > > service on the controller's IP and quantum_url should be 
> > > http://localhost:9696/?
> > >
> > > Additionally:
> > >
> > > quantum.conf - qpid_hostname should point to the controller's IP
> >
> > This is done by the abovementioned script
> > > quantum/api-paste.ini - [filter:authtoken] auth_host should be the 
> > > controller
> >
> > No need
> > > quantum/l3_agent.ini - auth_url should point to the controller
> >
> > No need - you only need one l3 agent
> > > plugins/openvswitch/ovs_quantum_plugin.ini - sql_connection should 
> > > be using the controller's mysql server
> >
> > This is tread by the script. Please note that there is no access from
> > the agent to the quantum database. This is done via the message
> > broker.
> > > Let me know if any of that is wrong or I missed some other settings 
> > > that should be notified.
> > >
> > > Regards,
> > > Lucian
> >
> > cloud mailing list
> > [e-mail]
> > https://admin.fedoraproject.org/mailman/listinfo/cloud

Right, hit some problems.
After installing openvswitch, quantum, creating the int and ex bridges 
and following the instructions for a new compute node I can see it can 
successfully contacts the controller (nova-manage service list <- lists 
the new compute service), and it even attempts to build a VM but I'm 
getting the following:

/var/log/nova/compute.log

2012-11-20 16:49:22 INFO nova.compute.resource_tracker [-] 
Compute_service record updated for node2242
2012-11-20 16:49:22 ERROR nova.network.quantumv2 [-] _get_auth_token() 
failed
2012-11-20 16:49:22 TRACE nova.network.quantumv2 Traceback (most recent 
call last):
2012-11-20 16:49:22 TRACE nova.network.quantumv2   File 
"/usr/lib/python2.6/site-packages/nova/network/quantumv2/__init__.py", 
line 38, in _get_auth_token
2012-11-20 16:49:22 TRACE nova.network.quantumv2     
httpclient.authenticate()
2012-11-20 16:49:22 TRACE nova.network.quantumv2   File 
"/usr/lib/python2.6/site-packages/quantumclient/client.py", line 199, in 
authenticate
2012-11-20 16:49:22 TRACE nova.network.quantumv2     raise 
exceptions.Unauthorized(message=body)
2012-11-20 16:49:22 TRACE nova.network.quantumv2 Unauthorized: [Errno 
111] ECONNREFUSED
2012-11-20 16:49:22 TRACE nova.network.quantumv2
2012-11-20 16:47:16 TRACE nova.network.quantumv2
2012-11-20 16:47:33 INFO nova.virt.libvirt.driver 
[req-ffb7c7cf-8c40-4622-828e-7e6916f608cc 
3c5b8ffd534347b2b30474b34f57dd3f 6cb0773ff31d480ab04ad9264bdb5f44] 
[instance: b0f4956b-6416-4270-8026-9c28ec927f42] Injecting key into 
image 727b6e19-988e-43b3-ba7c-014beafb6d29

Who refuses the connection to quantum? :-/

Then, also from compute log:

2012-11-20 16:47:58 ERROR nova.compute.manager 
[req-ffb7c7cf-8c40-4622-828e-7e6916f608cc 
3c5b8ffd534347b2b30474b34f57dd3f 6cb0773ff31d480ab04ad9264bdb5f44] 
[instance: b0f4956b-6416-4270-8026-9c28ec927f42] Instance failed to 
spawn
2012-11-20 16:47:58 TRACE nova.compute.manager [instance: 
b0f4956b-6416-4270-8026-9c28ec927f42] Traceback (most recent call last):
2012-11-20 16:47:58 TRACE nova.compute.manager [instance: 
b0f4956b-6416-4270-8026-9c28ec927f42]   File 
"/usr/lib/python2.6/site-packages/nova/compute/manager.py", line 743, in 
_spawn
2012-11-20 16:47:58 TRACE nova.compute.manager [instance: 
b0f4956b-6416-4270-8026-9c28ec927f42]     block_device_info)
2012-11-20 16:47:58 TRACE nova.compute.manager [instance: 
b0f4956b-6416-4270-8026-9c28ec927f42]   File 
"/usr/lib/python2.6/site-packages/nova/exception.py", line 117, in 
wrapped
2012-11-20 16:47:58 TRACE nova.compute.manager [instance: 
b0f4956b-6416-4270-8026-9c28ec927f42]     temp_level, payload)
2012-11-20 16:47:58 TRACE nova.compute.manager [instance: 
b0f4956b-6416-4270-8026-9c28ec927f42]   File 
"/usr/lib64/python2.6/contextlib.py", line 23, in __exit__
2012-11-20 16:47:58 TRACE nova.compute.manager [instance: 
b0f4956b-6416-4270-8026-9c28ec927f42]     self.gen.next()
2012-11-20 16:47:58 TRACE nova.compute.manager [instance: 
b0f4956b-6416-4270-8026-9c28ec927f42]   File 
"/usr/lib/python2.6/site-packages/nova/exception.py", line 92, in 
wrapped
2012-11-20 16:47:58 TRACE nova.compute.manager [instance: 
b0f4956b-6416-4270-8026-9c28ec927f42]     return f(*args, **kw)
2012-11-20 16:47:58 TRACE nova.compute.manager [instance: 
b0f4956b-6416-4270-8026-9c28ec927f42]   File 
"/usr/lib/python2.6/site-packages/nova/virt/libvirt/driver.py", line 
1062, in spawn
2012-11-20 16:47:58 TRACE nova.compute.manager [instance: 
b0f4956b-6416-4270-8026-9c28ec927f42]     block_device_info)
2012-11-20 16:47:58 TRACE nova.compute.manager [instance: 
b0f4956b-6416-4270-8026-9c28ec927f42]   File 
"/usr/lib/python2.6/site-packages/nova/virt/libvirt/driver.py", line 
1888, in _create_domain_and_network
2012-11-20 16:47:58 TRACE nova.compute.manager [instance: 
b0f4956b-6416-4270-8026-9c28ec927f42]     domain = 
self._create_domain(xml)
2012-11-20 16:47:58 TRACE nova.compute.manager [instance: 
b0f4956b-6416-4270-8026-9c28ec927f42]   File 
"/usr/lib/python2.6/site-packages/nova/virt/libvirt/driver.py", line 
1867, in _create_domain
2012-11-20 16:47:58 TRACE nova.compute.manager [instance: 
b0f4956b-6416-4270-8026-9c28ec927f42]     
domain.createWithFlags(launch_flags)
2012-11-20 16:47:58 TRACE nova.compute.manager [instance: 
b0f4956b-6416-4270-8026-9c28ec927f42]   File 
"/usr/lib/python2.6/site-packages/eventlet/tpool.py", line 187, in doit
2012-11-20 16:47:58 TRACE nova.compute.manager [instance: 
b0f4956b-6416-4270-8026-9c28ec927f42]     result = 
proxy_call(self._autowrap, f, *args, **kwargs)
2012-11-20 16:47:58 TRACE nova.compute.manager [instance: 
b0f4956b-6416-4270-8026-9c28ec927f42]   File 
"/usr/lib/python2.6/site-packages/eventlet/tpool.py", line 147, in 
proxy_call
2012-11-20 16:47:58 TRACE nova.compute.manager [instance: 
b0f4956b-6416-4270-8026-9c28ec927f42]     rv = execute(f,*args,**kwargs)
2012-11-20 16:47:58 TRACE nova.compute.manager [instance: 
b0f4956b-6416-4270-8026-9c28ec927f42]   File 
"/usr/lib/python2.6/site-packages/eventlet/tpool.py", line 76, in 
tworker
2012-11-20 16:47:58 TRACE nova.compute.manager [instance: 
b0f4956b-6416-4270-8026-9c28ec927f42]     rv = meth(*args,**kwargs)
2012-11-20 16:47:58 TRACE nova.compute.manager [instance: 
b0f4956b-6416-4270-8026-9c28ec927f42]   File 
"/usr/lib64/python2.6/site-packages/libvirt.py", line 650, in 
createWithFlags
2012-11-20 16:47:58 TRACE nova.compute.manager [instance: 
b0f4956b-6416-4270-8026-9c28ec927f42]     if ret == -1: raise 
libvirtError ('virDomainCreateWithFlags() failed', dom=self)
2012-11-20 16:47:58 TRACE nova.compute.manager [instance: 
b0f4956b-6416-4270-8026-9c28ec927f42] libvirtError: Cannot get interface 
MTU on '': No such device
2012-11-20 16:47:58 TRACE nova.compute.manager [instance: 
b0f4956b-6416-4270-8026-9c28ec927f42]
2012-11-20 16:47:59 INFO nova.compute.resource_tracker 
[req-ffb7c7cf-8c40-4622-828e-7e6916f608cc 
3c5b8ffd534347b2b30474b34f57dd3f 6cb0773ff31d480ab04ad9264bdb5f44] 
Aborting claim: [Claim b0f4956b-6416-4270-8026-9c28ec927f42: 512 MB 
memory, 0 GB disk, 1 VCPUS]
2012-11-20 16:47:59 ERROR nova.compute.manager 
[req-ffb7c7cf-8c40-4622-828e-7e6916f608cc 
3c5b8ffd534347b2b30474b34f57dd3f 6cb0773ff31d480ab04ad9264bdb5f44] 
[instance: b0f4956b-6416-4270-8026-9c28ec927f42] Build error: 
['Traceback (most recent call last):\n', '  File 
"/usr/lib/python2.6/site-packages/nova/compute/manager.py", line 496, in 
_run_instance\n    injected_files, admin_password)\n', '  File 
"/usr/lib/python2.6/site-packages/nova/compute/manager.py", line 743, in 
_spawn\n    block_device_info)\n', '  File 
"/usr/lib/python2.6/site-packages/nova/exception.py", line 117, in 
wrapped\n    temp_level, payload)\n', '  File 
"/usr/lib64/python2.6/contextlib.py", line 23, in __exit__\n    
self.gen.next()\n', '  File 
"/usr/lib/python2.6/site-packages/nova/exception.py", line 92, in 
wrapped\n    return f(*args, **kw)\n', '  File 
"/usr/lib/python2.6/site-packages/nova/virt/libvirt/driver.py", line 
1062, in spawn\n    block_device_info)\n', '  File 
"/usr/lib/python2.6/site-packages/nova/virt/libvirt/driver.py", line 
1888, in _create_domain_and_network\n    domain = 
self._create_domain(xml)\n', '  File 
"/usr/lib/python2.6/site-packages/nova/virt/libvirt/driver.py", line 
1867, in _create_domain\n    domain.createWithFlags(launch_flags)\n', '  
File "/usr/lib/python2.6/site-packages/eventlet/tpool.py", line 187, in 
doit\n    result = proxy_call(self._autowrap, f, *args, **kwargs)\n', '  
File "/usr/lib/python2.6/site-packages/eventlet/tpool.py", line 147, in 
proxy_call\n    rv = execute(f,*args,**kwargs)\n', '  File 
"/usr/lib/python2.6/site-packages/eventlet/tpool.py", line 76, in 
tworker\n    rv = meth(*args,**kwargs)\n', '  File 
"/usr/lib64/python2.6/site-packages/libvirt.py", line 650, in 
createWithFlags\n    if ret == -1: raise libvirtError 
('virDomainCreateWithFlags() failed', dom=self)\n', "libvirtError: 
Cannot get interface MTU on '': No such device\n"]
2012-11-20 16:48:18 AUDIT nova.compute.resource_tracker [-] Free ram 
(MB): 2796
2012-11-20 16:48:18 AUDIT nova.compute.resource_tracker [-] Free disk 
(GB): 225
2012-11-20 16:48:18 AUDIT nova.compute.resource_tracker [-] Free VCPUS: 
7
2012-11-20 16:48:18 INFO nova.compute.resource_tracker [-] 
Compute_service record updated for node2242.cumulus.coreix.net
2012-11-20 16:48:18 ERROR nova.network.quantumv2 [-] _get_auth_token() 
failed
2012-11-20 16:48:18 TRACE nova.network.quantumv2 Traceback (most recent 
call last):
2012-11-20 16:48:18 TRACE nova.network.quantumv2   File 
"/usr/lib/python2.6/site-packages/nova/network/quantumv2/__init__.py", 
line 38, in _get_auth_token
2012-11-20 16:48:18 TRACE nova.network.quantumv2     
httpclient.authenticate()
2012-11-20 16:48:18 TRACE nova.network.quantumv2   File 
"/usr/lib/python2.6/site-packages/quantumclient/client.py", line 199, in 
authenticate
2012-11-20 16:48:18 TRACE nova.network.quantumv2     raise 
exceptions.Unauthorized(message=body)
2012-11-20 16:48:18 TRACE nova.network.quantumv2 Unauthorized: [Errno 
111] ECONNREFUSED
2012-11-20 16:48:18 TRACE nova.network.quantumv2

/var/log/libvirt/libvirtd.log says:

2012-11-20 16:41:56.762+0000: 1497: info : libvirt version: 0.9.10, 
package: 21.el6_3.5 (CentOS BuildSystem http://bugs.centos.org, 
2012-10-11-13:57:12, c6b9.bsys.dev.centos.org)
2012-11-20 16:41:56.762+0000: 1497: error : virNetDevBridgeCreate:224 : 
Unable to create bridge virbr0: Package not installed
2012-11-20 16:45:26.423+0000: 2488: info : libvirt version: 0.9.10, 
package: 21.el6_3.5 (CentOS BuildSystem http://bugs.centos.org, 
2012-10-11-13:57:12, c6b9.bsys.dev.centos.org)
2012-11-20 16:45:26.423+0000: 2488: error : virNetDevBridgeSet:136 : 
Unable to set bridge virbr0 forward_delay: Operation not supported
2012-11-20 16:47:58.628+0000: 2482: error : virNetDevGetMTU:347 : 
Cannot get interface MTU on '': No such device
2012-11-20 16:47:58.653+0000: 2482: error : virNetDevGetIndex:657 : 
Unable to get index for interface vnet0: No such device

Selinux is in permissive mode.

What's wrong?

On 11/20/2012 06:56 PM, Nux! wrote:
> Right, hit some problems.
> After installing openvswitch, quantum, creating the int and ex bridges 
> and following the instructions for a new compute node I can see it can 
> successfully contacts the controller (nova-manage service list <- 
> lists the new compute service), and it even attempts to build a VM but 
> I'm getting the following:
>
> /var/log/nova/compute.log
>
> 2012-11-20 16:49:22 INFO nova.compute.resource_tracker [-] 
> Compute_service record updated for node2242
> 2012-11-20 16:49:22 ERROR nova.network.quantumv2 [-] _get_auth_token() 
> failed
> 2012-11-20 16:49:22 TRACE nova.network.quantumv2 Traceback (most 
> recent call last):
> 2012-11-20 16:49:22 TRACE nova.network.quantumv2   File 
> "/usr/lib/python2.6/site-packages/nova/network/quantumv2/__init__.py", 
> line 38, in _get_auth_token
> 2012-11-20 16:49:22 TRACE nova.network.quantumv2     
> httpclient.authenticate()
> 2012-11-20 16:49:22 TRACE nova.network.quantumv2   File 
> "/usr/lib/python2.6/site-packages/quantumclient/client.py", line 199, 
> in authenticate
> 2012-11-20 16:49:22 TRACE nova.network.quantumv2     raise 
> exceptions.Unauthorized(message=body)
> 2012-11-20 16:49:22 TRACE nova.network.quantumv2 Unauthorized: [Errno 
> 111] ECONNREFUSED
> 2012-11-20 16:49:22 TRACE nova.network.quantumv2
> 2012-11-20 16:47:16 TRACE nova.network.quantumv2
> 2012-11-20 16:47:33 INFO nova.virt.libvirt.driver 
> [req-ffb7c7cf-8c40-4622-828e-7e6916f608cc 
> 3c5b8ffd534347b2b30474b34f57dd3f 6cb0773ff31d480ab04ad9264bdb5f44] 
> [instance: b0f4956b-6416-4270-8026-9c28ec927f42] Injecting key into 
> image 727b6e19-988e-43b3-ba7c-014beafb6d29
>
> Who refuses the connection to quantum? :-/

In order for Nova to deploy a VM it needs to interface with Quantum. Can 
you please check that the /etc/nova.conf is updated correctly. This 
should have the keystone credentails for Quantum (you can compare with 
the nova.conf that you have on the controller node)
> Then, also from compute log:
>
> 2012-11-20 16:47:58 ERROR nova.compute.manager 
> [req-ffb7c7cf-8c40-4622-828e-7e6916f608cc 
> 3c5b8ffd534347b2b30474b34f57dd3f 6cb0773ff31d480ab04ad9264bdb5f44] 
> [instance: b0f4956b-6416-4270-8026-9c28ec927f42] Instance failed to spawn
> 2012-11-20 16:47:58 TRACE nova.compute.manager [instance: 
> b0f4956b-6416-4270-8026-9c28ec927f42] Traceback (most recent call last):
> 2012-11-20 16:47:58 TRACE nova.compute.manager [instance: 
> b0f4956b-6416-4270-8026-9c28ec927f42]   File 
> "/usr/lib/python2.6/site-packages/nova/compute/manager.py", line 743, 
> in _spawn
> 2012-11-20 16:47:58 TRACE nova.compute.manager [instance: 
> b0f4956b-6416-4270-8026-9c28ec927f42]     block_device_info)
> 2012-11-20 16:47:58 TRACE nova.compute.manager [instance: 
> b0f4956b-6416-4270-8026-9c28ec927f42]   File 
> "/usr/lib/python2.6/site-packages/nova/exception.py", line 117, in 
> wrapped
> 2012-11-20 16:47:58 TRACE nova.compute.manager [instance: 
> b0f4956b-6416-4270-8026-9c28ec927f42]     temp_level, payload)
> 2012-11-20 16:47:58 TRACE nova.compute.manager [instance: 
> b0f4956b-6416-4270-8026-9c28ec927f42]   File 
> "/usr/lib64/python2.6/contextlib.py", line 23, in __exit__
> 2012-11-20 16:47:58 TRACE nova.compute.manager [instance: 
> b0f4956b-6416-4270-8026-9c28ec927f42]     self.gen.next()
> 2012-11-20 16:47:58 TRACE nova.compute.manager [instance: 
> b0f4956b-6416-4270-8026-9c28ec927f42]   File 
> "/usr/lib/python2.6/site-packages/nova/exception.py", line 92, in wrapped
> 2012-11-20 16:47:58 TRACE nova.compute.manager [instance: 
> b0f4956b-6416-4270-8026-9c28ec927f42]     return f(*args, **kw)
> 2012-11-20 16:47:58 TRACE nova.compute.manager [instance: 
> b0f4956b-6416-4270-8026-9c28ec927f42]   File 
> "/usr/lib/python2.6/site-packages/nova/virt/libvirt/driver.py", line 
> 1062, in spawn
> 2012-11-20 16:47:58 TRACE nova.compute.manager [instance: 
> b0f4956b-6416-4270-8026-9c28ec927f42]     block_device_info)
> 2012-11-20 16:47:58 TRACE nova.compute.manager [instance: 
> b0f4956b-6416-4270-8026-9c28ec927f42]   File 
> "/usr/lib/python2.6/site-packages/nova/virt/libvirt/driver.py", line 
> 1888, in _create_domain_and_network
> 2012-11-20 16:47:58 TRACE nova.compute.manager [instance: 
> b0f4956b-6416-4270-8026-9c28ec927f42]     domain = 
> self._create_domain(xml)
> 2012-11-20 16:47:58 TRACE nova.compute.manager [instance: 
> b0f4956b-6416-4270-8026-9c28ec927f42]   File 
> "/usr/lib/python2.6/site-packages/nova/virt/libvirt/driver.py", line 
> 1867, in _create_domain
> 2012-11-20 16:47:58 TRACE nova.compute.manager [instance: 
> b0f4956b-6416-4270-8026-9c28ec927f42]     
> domain.createWithFlags(launch_flags)
> 2012-11-20 16:47:58 TRACE nova.compute.manager [instance: 
> b0f4956b-6416-4270-8026-9c28ec927f42]   File 
> "/usr/lib/python2.6/site-packages/eventlet/tpool.py", line 187, in doit
> 2012-11-20 16:47:58 TRACE nova.compute.manager [instance: 
> b0f4956b-6416-4270-8026-9c28ec927f42]     result = 
> proxy_call(self._autowrap, f, *args, **kwargs)
> 2012-11-20 16:47:58 TRACE nova.compute.manager [instance: 
> b0f4956b-6416-4270-8026-9c28ec927f42]   File 
> "/usr/lib/python2.6/site-packages/eventlet/tpool.py", line 147, in 
> proxy_call
> 2012-11-20 16:47:58 TRACE nova.compute.manager [instance: 
> b0f4956b-6416-4270-8026-9c28ec927f42]     rv = execute(f,*args,**kwargs)
> 2012-11-20 16:47:58 TRACE nova.compute.manager [instance: 
> b0f4956b-6416-4270-8026-9c28ec927f42]   File 
> "/usr/lib/python2.6/site-packages/eventlet/tpool.py", line 76, in tworker
> 2012-11-20 16:47:58 TRACE nova.compute.manager [instance: 
> b0f4956b-6416-4270-8026-9c28ec927f42]     rv = meth(*args,**kwargs)
> 2012-11-20 16:47:58 TRACE nova.compute.manager [instance: 
> b0f4956b-6416-4270-8026-9c28ec927f42]   File 
> "/usr/lib64/python2.6/site-packages/libvirt.py", line 650, in 
> createWithFlags
> 2012-11-20 16:47:58 TRACE nova.compute.manager [instance: 
> b0f4956b-6416-4270-8026-9c28ec927f42]     if ret == -1: raise 
> libvirtError ('virDomainCreateWithFlags() failed', dom=self)
> 2012-11-20 16:47:58 TRACE nova.compute.manager [instance: 
> b0f4956b-6416-4270-8026-9c28ec927f42] libvirtError: Cannot get 
> interface MTU on '': No such device
> 2012-11-20 16:47:58 TRACE nova.compute.manager [instance: 
> b0f4956b-6416-4270-8026-9c28ec927f42]
> 2012-11-20 16:47:59 INFO nova.compute.resource_tracker 
> [req-ffb7c7cf-8c40-4622-828e-7e6916f608cc 
> 3c5b8ffd534347b2b30474b34f57dd3f 6cb0773ff31d480ab04ad9264bdb5f44] 
> Aborting claim: [Claim b0f4956b-6416-4270-8026-9c28ec927f42: 512 MB 
> memory, 0 GB disk, 1 VCPUS]
> 2012-11-20 16:47:59 ERROR nova.compute.manager 
> [req-ffb7c7cf-8c40-4622-828e-7e6916f608cc 
> 3c5b8ffd534347b2b30474b34f57dd3f 6cb0773ff31d480ab04ad9264bdb5f44] 
> [instance: b0f4956b-6416-4270-8026-9c28ec927f42] Build error: 
> ['Traceback (most recent call last):\n', '  File 
> "/usr/lib/python2.6/site-packages/nova/compute/manager.py", line 496, 
> in _run_instance\n    injected_files, admin_password)\n', '  File 
> "/usr/lib/python2.6/site-packages/nova/compute/manager.py", line 743, 
> in _spawn\n    block_device_info)\n', '  File 
> "/usr/lib/python2.6/site-packages/nova/exception.py", line 117, in 
> wrapped\n    temp_level, payload)\n', '  File 
> "/usr/lib64/python2.6/contextlib.py", line 23, in __exit__\n    
> self.gen.next()\n', '  File 
> "/usr/lib/python2.6/site-packages/nova/exception.py", line 92, in 
> wrapped\n    return f(*args, **kw)\n', '  File 
> "/usr/lib/python2.6/site-packages/nova/virt/libvirt/driver.py", line 
> 1062, in spawn\n    block_device_info)\n', '  File 
> "/usr/lib/python2.6/site-packages/nova/virt/libvirt/driver.py", line 
> 1888, in _create_domain_and_network\n    domain = 
> self._create_domain(xml)\n', '  File 
> "/usr/lib/python2.6/site-packages/nova/virt/libvirt/driver.py", line 
> 1867, in _create_domain\n    domain.createWithFlags(launch_flags)\n', 
> '  File "/usr/lib/python2.6/site-packages/eventlet/tpool.py", line 
> 187, in doit\n    result = proxy_call(self._autowrap, f, *args, 
> **kwargs)\n', '  File 
> "/usr/lib/python2.6/site-packages/eventlet/tpool.py", line 147, in 
> proxy_call\n    rv = execute(f,*args,**kwargs)\n', '  File 
> "/usr/lib/python2.6/site-packages/eventlet/tpool.py", line 76, in 
> tworker\n    rv = meth(*args,**kwargs)\n', '  File 
> "/usr/lib64/python2.6/site-packages/libvirt.py", line 650, in 
> createWithFlags\n    if ret == -1: raise libvirtError 
> ('virDomainCreateWithFlags() failed', dom=self)\n', "libvirtError: 
> Cannot get interface MTU on '': No such device\n"]
> 2012-11-20 16:48:18 AUDIT nova.compute.resource_tracker [-] Free ram 
> (MB): 2796
> 2012-11-20 16:48:18 AUDIT nova.compute.resource_tracker [-] Free disk 
> (GB): 225
> 2012-11-20 16:48:18 AUDIT nova.compute.resource_tracker [-] Free VCPUS: 7
> 2012-11-20 16:48:18 INFO nova.compute.resource_tracker [-] 
> Compute_service record updated for node2242.cumulus.coreix.net
> 2012-11-20 16:48:18 ERROR nova.network.quantumv2 [-] _get_auth_token() 
> failed
> 2012-11-20 16:48:18 TRACE nova.network.quantumv2 Traceback (most 
> recent call last):
> 2012-11-20 16:48:18 TRACE nova.network.quantumv2   File 
> "/usr/lib/python2.6/site-packages/nova/network/quantumv2/__init__.py", 
> line 38, in _get_auth_token
> 2012-11-20 16:48:18 TRACE nova.network.quantumv2     
> httpclient.authenticate()
> 2012-11-20 16:48:18 TRACE nova.network.quantumv2   File 
> "/usr/lib/python2.6/site-packages/quantumclient/client.py", line 199, 
> in authenticate
> 2012-11-20 16:48:18 TRACE nova.network.quantumv2     raise 
> exceptions.Unauthorized(message=body)
> 2012-11-20 16:48:18 TRACE nova.network.quantumv2 Unauthorized: [Errno 
> 111] ECONNREFUSED
> 2012-11-20 16:48:18 TRACE nova.network.quantumv2
>
> /var/log/libvirt/libvirtd.log says:
>
> 2012-11-20 16:41:56.762+0000: 1497: info : libvirt version: 0.9.10, 
> package: 21.el6_3.5 (CentOS BuildSystem http://bugs.centos.org, 
> 2012-10-11-13:57:12, c6b9.bsys.dev.centos.org)
> 2012-11-20 16:41:56.762+0000: 1497: error : virNetDevBridgeCreate:224 
> : Unable to create bridge virbr0: Package not installed
> 2012-11-20 16:45:26.423+0000: 2488: info : libvirt version: 0.9.10, 
> package: 21.el6_3.5 (CentOS BuildSystem http://bugs.centos.org, 
> 2012-10-11-13:57:12, c6b9.bsys.dev.centos.org)
> 2012-11-20 16:45:26.423+0000: 2488: error : virNetDevBridgeSet:136 : 
> Unable to set bridge virbr0 forward_delay: Operation not supported
> 2012-11-20 16:47:58.628+0000: 2482: error : virNetDevGetMTU:347 : 
> Cannot get interface MTU on '': No such device
> 2012-11-20 16:47:58.653+0000: 2482: error : virNetDevGetIndex:657 : 
> Unable to get index for interface vnet0: No such device
>
> Selinux is in permissive mode.
>
> What's wrong?

Did you not have this working a few days ago?

On 20.11.2012 17:26, Gary Kotton wrote:
> On 11/20/2012 06:56 PM, Nux! wrote:
> > Right, hit some problems.
> > After installing openvswitch, quantum, creating the int and ex 
> > bridges and following the instructions for a new compute node I can 
> > see it can successfully contacts the controller (nova-manage service 
> > list <- lists the new compute service), and it even attempts to build 
> > a VM but I'm getting the following:
> >
> > /var/log/nova/compute.log
> >
> > 2012-11-20 16:49:22 INFO nova.compute.resource_tracker [-] 
> > Compute_service record updated for node2242
> > 2012-11-20 16:49:22 ERROR nova.network.quantumv2 [-] 
> > _get_auth_token() failed
> > 2012-11-20 16:49:22 TRACE nova.network.quantumv2 Traceback (most 
> > recent call last):
> > 2012-11-20 16:49:22 TRACE nova.network.quantumv2   File 
> > "/usr/lib/python2.6/site-packages/nova/network/quantumv2/__init__.py", 
> > line 38, in _get_auth_token
> > 2012-11-20 16:49:22 TRACE nova.network.quantumv2     
> > httpclient.authenticate()
> > 2012-11-20 16:49:22 TRACE nova.network.quantumv2   File 
> > "/usr/lib/python2.6/site-packages/quantumclient/client.py", line 199, 
> > in authenticate
> > 2012-11-20 16:49:22 TRACE nova.network.quantumv2     raise 
> > exceptions.Unauthorized(message=body)
> > 2012-11-20 16:49:22 TRACE nova.network.quantumv2 Unauthorized: 
> > [Errno 111] ECONNREFUSED
> > 2012-11-20 16:49:22 TRACE nova.network.quantumv2
> > 2012-11-20 16:47:16 TRACE nova.network.quantumv2
> > 2012-11-20 16:47:33 INFO nova.virt.libvirt.driver 
> > [req-ffb7c7cf-8c40-4622-828e-7e6916f608cc 
> > 3c5b8ffd534347b2b30474b34f57dd3f 6cb0773ff31d480ab04ad9264bdb5f44] 
> > [instance: b0f4956b-6416-4270-8026-9c28ec927f42] Injecting key into 
> > image 727b6e19-988e-43b3-ba7c-014beafb6d29
> >
> > Who refuses the connection to quantum? :-/
>
> In order for Nova to deploy a VM it needs to interface with Quantum.
> Can you please check that the /etc/nova.conf is updated correctly.
> This should have the keystone credentails for Quantum (you can 
> compare
> with the nova.conf that you have on the controller node)

Yes, my network/quantum settings in nova.conf are the same as on the 
controller...
> > 2012-11-20 16:41:56.762+0000: 1497: info : libvirt version: 0.9.10, 
> > package: 21.el6_3.5 (CentOS BuildSystem http://bugs.centos.org, 
> > 2012-10-11-13:57:12, c6b9.bsys.dev.centos.org)
> > 2012-11-20 16:41:56.762+0000: 1497: error : 
> > virNetDevBridgeCreate:224 : Unable to create bridge virbr0: Package 
> > not installed
> > 2012-11-20 16:45:26.423+0000: 2488: info : libvirt version: 0.9.10, 
> > package: 21.el6_3.5 (CentOS BuildSystem http://bugs.centos.org, 
> > 2012-10-11-13:57:12, c6b9.bsys.dev.centos.org)
> > 2012-11-20 16:45:26.423+0000: 2488: error : virNetDevBridgeSet:136 : 
> > Unable to set bridge virbr0 forward_delay: Operation not supported
> > 2012-11-20 16:47:58.628+0000: 2482: error : virNetDevGetMTU:347 : 
> > Cannot get interface MTU on '': No such device
> > 2012-11-20 16:47:58.653+0000: 2482: error : virNetDevGetIndex:657 : 
> > Unable to get index for interface vnet0: No such device
> >
> > Selinux is in permissive mode.
> >
> > What's wrong?
>
> Did you not have this working a few days ago?

I did, on the controller, it works fine, but not on this compute node. 
I'll try to dig some more, probably I misconfigured something.

On 20.11.2012 17:41, Nux! wrote:
> > > What's wrong?
> >
> > Did you not have this working a few days ago?
>
> I did, on the controller, it works fine, but not on this compute
> node. I'll try to dig some more, probably I misconfigured something.

Any reason why the keystonerc from the controller (which works fine) 
shouldn't work also on the additional node? I did change the OS_AUTH_URL 
to:
xport OS_AUTH_URL=http://controller:5000/v2.0/

Also, another thing that seems odd to me, there is no /var/glance on 
the additional node. Where are the images stored?

On 11/20/2012 08:26 PM, Nux! wrote:
> On 20.11.2012 17:41, Nux! wrote:
> > > > What's wrong?
> > >
> > > Did you not have this working a few days ago?
> >
> > I did, on the controller, it works fine, but not on this compute
> > node. I'll try to dig some more, probably I misconfigured something.
>
> Any reason why the keystonerc from the controller (which works fine) 
> shouldn't work also on the additional node? I did change the 
> OS_AUTH_URL to:
> xport OS_AUTH_URL=http://controller:5000/v2.0/
>
> Also, another thing that seems odd to me, there is no /var/glance on 
> the additional node. Where are the images stored?

I need to update the wiki for a multi host setup. Hopefully I'll get 
this done in the coming days.

In the above case the you need to add the IP for the controller to the 
hosts file.

Thanks
Gary

On 20.11.2012 18:30, Gary Kotton wrote:
> On 11/20/2012 08:26 PM, Nux! wrote:
> > On 20.11.2012 17:41, Nux! wrote:
> > > > > What's wrong?
> > > >
> > > > Did you not have this working a few days ago?
> > >
> > > I did, on the controller, it works fine, but not on this compute
> > > node. I'll try to dig some more, probably I misconfigured 
> > > something.
> >
> > Any reason why the keystonerc from the controller (which works fine) 
> > shouldn't work also on the additional node? I did change the 
> > OS_AUTH_URL to:
> > xport OS_AUTH_URL=http://controller:5000/v2.0/
> >
> > Also, another thing that seems odd to me, there is no /var/glance on 
> > the additional node. Where are the images stored?
>
> I need to update the wiki for a multi host setup. Hopefully I'll get
> this done in the coming days.

That'd be great. This is all quite confusing.
> In the above case the you need to add the IP for the controller to
> the hosts file.

Yeah, name resolution works correctly. There's also not a firewall 
issue (checked manually with telnet).

I'll keep trying in the following days.