Fatal error when migrate the VM with disk on iscsi data center

Hi,

I am testing the VM migration in oVirt engine 3.1 bteta release and 
successfully  did the migration test on NFS date center.  However, the 
test of VM migration on iSCSI data center failed.
I dumped the error and warning segments from the vdsm logs on both the 
source and destination host.  Please ignore the time stamp out of sync 
between the two hosts.
_
_*The vdsm log segment  from the source host:*

66-4da5-8310-6dcc970e5367`::migration downtime thread exiting
Thread-115166::ERROR::2012-07-25 
23:18:55,953::vm::176::vm.Vm::(_recover) 
vmId=`11e06222-2d66-4da5-8310-6dcc970e5367`::operation failed: Failed to 
connect to remote libvirt URI qemu+tcp://9.181.129.110/system
Dummy-115148::DEBUG::2012-07-25 
23:18:55,981::__init__::1249::Storage.Misc.excCmd::(_log) 'dd 
if=/rhev/data-center/0b9a4ea4-d487-11e1-b614-5254001498c4/mastersd/dom_md/inbox 
iflag=direct,fullblock count=1 bs=1024000' (cwd None)
Thread-115166::ERROR::2012-07-25 23:18:56,028::vm::240::vm.Vm::(run) 
vmId=`11e06222-2d66-4da5-8310-6dcc970e5367`::Failed to migrate
Traceback (most recent call last):
   File "/usr/share/vdsm/vm.py", line 223, in run
     self._startUnderlyingMigration()
   File "/usr/share/vdsm/libvirtvm.py", line 451, in 
_startUnderlyingMigration
     None, maxBandwidth)
   File "/usr/share/vdsm/libvirtvm.py", line 491, in f
     ret = attr(*args, **kwargs)
   File "/usr/lib/python2.7/site-packages/vdsm/libvirtconnection.py", 
line 82, in wrapper
     ret = f(*args, **kwargs)
   File "/usr/lib64/python2.7/site-packages/libvirt.py", line 1034, in 
migrateToURI2
     if ret == -1: raise libvirtError ('virDomainMigrateToURI2() 
failed', dom=self)
libvirtError: operation failed: Failed to connect to remote libvirt URI 
qemu+tcp://9.181.129.110/system


*The vdsm log segment from the destination host:*

Thread-1577::INFO::2012-07-25 
23:19:50,407::API::601::vds::(_getNetworkIp) network None: using 0
Thread-1577::INFO::2012-07-25 23:19:50,407::API::228::vds::(create) 
vmContainerLock acquired by vm 11e06222-2d66-4da5-8310-6dcc970e5367
Thread-1578::DEBUG::2012-07-25 
23:19:50,411::vm::564::vm.Vm::(_startUnderlyingVm) 
vmId=`11e06222-2d66-4da5-8310-6dcc970e5367`::Start
Thread-1577::DEBUG::2012-07-25 23:19:50,411::API::244::vds::(create) 
Total desktops after creation of 11e06222-2d66-4da5-8310-6dcc970e5367 is 1
Thread-1578::DEBUG::2012-07-25 
23:19:50,411::vm::568::vm.Vm::(_startUnderlyingVm) 
vmId=`11e06222-2d66-4da5-8310-6dcc970e5367`::_ongoingCreations acquired
Thread-1577::DEBUG::2012-07-25 
23:19:50,412::libvirtvm::2438::vm.Vm::(waitForMigrationDestinationPrepare) 
vmId=`11e06222-2d66-4da5-8310-6dcc970e5367`::migration destination: 
waiting 36s for path preparation
Thread-1578::INFO::2012-07-25 
23:19:50,412::libvirtvm::1285::vm.Vm::(_run) 
vmId=`11e06222-2d66-4da5-8310-6dcc970e5367`::VM wrapper has started
Thread-1578::WARNING::2012-07-25 
23:19:50,413::vm::398::vm.Vm::(getConfDevices) 
vmId=`11e06222-2d66-4da5-8310-6dcc970e5367`::Unknown type found, device: 
'{'device': 'unix', 'alias': 'channel0',
'type': 'channel', 'address': 
{'bus': '0', 'controller': '0', 'type':
'virtio-serial', 'port': '1'}}' 
found

Thread-1578::ERROR::2012-07-25 
23:24:50,484::vm::604::vm.Vm::(_startUnderlyingVm) 
vmId=`11e06222-2d66-4da5-8310-6dcc970e5367`::The vm start process failed
Traceback (most recent call last):
   File "/usr/share/vdsm/vm.py", line 584, in _startUnderlyingVm
     self._waitForIncomingMigrationFinish()
   File "/usr/share/vdsm/libvirtvm.py", line 1572, in 
_waitForIncomingMigrationFinish
     self._connection.lookupByUUIDString(self.id),
   File "/usr/lib/python2.7/site-packages/vdsm/libvirtconnection.py", 
line 82, in wrapper
     ret = f(*args, **kwargs)
   File "/usr/lib64/python2.7/site-packages/libvirt.py", line 2608, in 
lookupByUUIDString
     if ret is None:raise libvirtError('virDomainLookupByUUIDString() 
failed', conn=self)
libvirtError: Domain not found: no domain with matching uuid 
'11e06222-2d66-4da5-8310-6dcc970e5367' - Timed out (did not receive 
success event)
Thread-1578::DEBUG::2012-07-25 
23:24:50,485::vm::920::vm.Vm::(setDownStatus) 
vmId=`11e06222-2d66-4da5-8310-6dcc970e5367`::Changed state to Down: 
Domain not found: no domain with matching uuid '11e0622


Any idea?

-- 
Shu Ming <shuming(a)linux.vnet.ibm.com&gt;
IBM China Systems and Technology Laboratory

[Federico Simoncelli]

----- Original Message -----
>  From: "Shu Ming" <shuming(a)linux.vnet.ibm.com&gt;
>  To: "VDSM Project Development" <vdsm-devel(a)lists.fedorahosted.org&gt;
>  Sent: Wednesday, July 25, 2012 5:57:05 PM
>  Subject: [vdsm] Fatal error when migrate the VM with disk on iscsi data	center
>  
>  
>  Hi,
>  
>  I am testing the VM migration in oVirt engine 3.1 bteta release and
>  successfully did the migration test on NFS date center. However, the
>  test of VM migration on iSCSI data center failed.
>  I dumped the error and warning segments from the vdsm logs on both
>  the source and destination host. Please ignore the time stamp out of
>  sync between the two hosts.
>  
>  The vdsm log segment from the source host:
>  
>  66-4da5-8310-6dcc970e5367`::migration downtime thread exiting
>  Thread-115166::ERROR::2012-07-25
>  23:18:55,953::vm::176::vm.Vm::(_recover)
>  vmId=`11e06222-2d66-4da5-8310-6dcc970e5367`::operation failed:
>  Failed to connect to remote libvirt URI
>  qemu+tcp://9.181.129.110/system
>  Dummy-115148::DEBUG::2012-07-25
>  23:18:55,981::__init__::1249::Storage.Misc.excCmd::(_log) 'dd
>  if=/rhev/data-center/0b9a4ea4-d487-11e1-b614-5254001498c4/mastersd/dom_md/inbox
>  iflag=direct,fullblock count=1 bs=1024000' (cwd None)
>  Thread-115166::ERROR::2012-07-25 23:18:56,028::vm::240::vm.Vm::(run)
>  vmId=`11e06222-2d66-4da5-8310-6dcc970e5367`::Failed to migrate
>  Traceback (most recent call last):
>  File "/usr/share/vdsm/vm.py", line 223, in run
>  self._startUnderlyingMigration()
>  File "/usr/share/vdsm/libvirtvm.py", line 451, in
>  _startUnderlyingMigration
>  None, maxBandwidth)
>  File "/usr/share/vdsm/libvirtvm.py", line 491, in f
>  ret = attr(*args, **kwargs)
>  File "/usr/lib/python2.7/site-packages/vdsm/libvirtconnection.py",
>  line 82, in wrapper
>  ret = f(*args, **kwargs)
>  File "/usr/lib64/python2.7/site-packages/libvirt.py", line 1034, in
>  migrateToURI2
>  if ret == -1: raise libvirtError ('virDomainMigrateToURI2() failed',
>  dom=self)
>  libvirtError: operation failed: Failed to connect to remote libvirt
>  URI qemu+tcp://9.181.129.110/system 
The destination host doesn't look right (9.181.129.110).
Can you post the logs related to the vmMigrate parameters?
Eventually you could open a bug so that you can attach the full logs.

-- 
Federico