Hi, while rebooting hosted-engine VM I've the following error:
vdsm-4.12.0-10.git295a069.fc19.x86_64 (nightly) libvirt-1.0.5.4-1.fc19.x86_64 (F19) mom-0.3.2-3.fc19.noarch (manually rebuilt from master due to missing package in nightly)
Thread-621::DEBUG::2013-08-01 10:42:53,011::BindingXMLRPC::986::vds::(wrapper) return vmGetStats with {'status': {'message': 'Done', 'code': 0}, 'statsList': [{'status': 'Up', 'username': 'Unknown', 'memUsage': '0', 'acpiEnable': 'true', 'guestFQDN': '', 'pid': '26846', 'displayIp': '0', 'displayPort': '-1', 'session': 'Unknown', 'displaySecurePort': u'5900', 'cdrom': '/home/Fedora-19-x86_64-DVD.iso', 'hash': '-6106157858121768929', 'balloonInfo': {}, 'pauseCode': 'NOERR', 'clientIp': '127.0.0.1', 'kvmEnable': 'true', 'network': {u'vnet0': {'macAddr': '00:16:3e:4f:10:9a', 'rxDropped': '0', 'rxErrors': '0', 'txDropped': '0', 'txRate': '0.0', 'rxRate': '0.0', 'txErrors': '0', 'state': 'unknown', 'speed': '1000', 'name': u'vnet0'}}, 'vmId': 'fd470849-17c9-436e-818d-16232f5b032b', 'monitorResponse': '0', 'cpuUser': '0.69', 'disks': {u'hdc': {'readLatency': '0', 'apparentsize': '0', 'writeLatency': '0', 'flushLatency': '0', 'readRate': '0.00', 'truesize': '0', 'writeRate': '0.00'}, u'hda': {'readLatency': '0', 'apparentsize': '26843545600', 'writeLatency': '0', 'imageID': '9ac2ea13-1de5-4a60-83c5-8700a23203b7', 'flushLatency': '0', 'readRate': '0.00', 'truesize': '1594064896', 'writeRate': '0.00'}}, 'boot': 'd', 'statsAge': '0.74', 'elapsedTime': '2834', 'vmType': 'kvm', 'cpuSys': '0.13', 'timeOffset': -500L, 'appsList': [], 'guestIPs': '', 'displayType': 'qxl'}]} Thread-622::DEBUG::2013-08-01 10:42:58,024::BindingXMLRPC::979::vds::(wrapper) client [127.0.0.1]::call vmGetStats with ('fd470849-17c9-436e-818d-16232f5b032b',) {} Thread-622::DEBUG::2013-08-01 10:42:58,024::BindingXMLRPC::986::vds::(wrapper) return vmGetStats with {'status': {'message': 'Done', 'code': 0}, 'statsList': [{'status': 'Up', 'username': 'Unknown', 'memUsage': '0', 'acpiEnable': 'true', 'guestFQDN': '', 'pid': '26846', 'displayIp': '0', 'displayPort': '-1', 'session': 'Unknown', 'displaySecurePort': u'5900', 'cdrom': '/home/Fedora-19-x86_64-DVD.iso', 'hash': '-6106157858121768929', 'balloonInfo': {}, 'pauseCode': 'NOERR', 'clientIp': '127.0.0.1', 'kvmEnable': 'true', 'network': {u'vnet0': {'macAddr': '00:16:3e:4f:10:9a', 'rxDropped': '0', 'rxErrors': '0', 'txDropped': '0', 'txRate': '0.0', 'rxRate': '0.0', 'txErrors': '0', 'state': 'unknown', 'speed': '1000', 'name': u'vnet0'}}, 'vmId': 'fd470849-17c9-436e-818d-16232f5b032b', 'monitorResponse': '0', 'cpuUser': '0.69', 'disks': {u'hdc': {'readLatency': '0', 'apparentsize': '0', 'writeLatency': '0', 'flushLatency': '0', 'readRate': '0.00', 'truesize': '0', 'writeRate': '0.00'}, u'hda': {'readLatency': '0', 'apparentsize': '26843545600', 'writeLatency': '0', 'imageID': '9ac2ea13-1de5-4a60-83c5-8700a23203b7', 'flushLatency': '0', 'readRate': '0.00', 'truesize': '1594064896', 'writeRate': '0.00'}}, 'boot': 'd', 'statsAge': '0.76', 'elapsedTime': '2839', 'vmType': 'kvm', 'cpuSys': '0.13', 'timeOffset': -500L, 'appsList': [], 'guestIPs': '', 'displayType': 'qxl'}]} libvirtEventLoop::DEBUG::2013-08-01 10:42:59,268::vm::4714::vm.Vm::(_onLibvirtLifecycleEvent) vmId=`fd470849-17c9-436e-818d-16232f5b032b`::event Shutdown detail 0 opaque None VM Channels Listener::ERROR::2013-08-01 10:42:59,335::vmChannels::53::vds::(_handle_event) Received 00000019 on fileno 32 VM Channels Listener::DEBUG::2013-08-01 10:42:59,336::vmChannels::128::vds::(_handle_unconnected) Trying to connect fileno 32. VM Channels Listener::DEBUG::2013-08-01 10:42:59,336::guestIF::147::vm.Vm::(_connect) vmId=`fd470849-17c9-436e-818d-16232f5b032b`::Attempting connection to /var/lib/libvirt/qemu/channels/fd470849-17c9-436e-818d-16232f5b032b.com.redhat.rhevm.vdsm VM Channels Listener::DEBUG::2013-08-01 10:42:59,336::guestIF::158::vm.Vm::(_connect) vmId=`fd470849-17c9-436e-818d-16232f5b032b`::Failed to connect to /var/lib/libvirt/qemu/channels/fd470849-17c9-436e-818d-16232f5b032b.com.redhat.rhevm.vdsm with 111 libvirtEventLoop::DEBUG::2013-08-01 10:42:59,672::vm::4714::vm.Vm::(_onLibvirtLifecycleEvent) vmId=`fd470849-17c9-436e-818d-16232f5b032b`::event Stopped detail 0 opaque None libvirtEventLoop::INFO::2013-08-01 10:42:59,672::vm::2092::vm.Vm::(_onQemuDeath) vmId=`fd470849-17c9-436e-818d-16232f5b032b`::underlying process disconnected libvirtEventLoop::INFO::2013-08-01 10:42:59,672::vm::4214::vm.Vm::(releaseVm) vmId=`fd470849-17c9-436e-818d-16232f5b032b`::Release VM resources libvirtEventLoop::DEBUG::2013-08-01 10:42:59,674::libvirtconnection::101::libvirtconnection::(wrapper) Unknown libvirterror: ecode: 42 edom: 10 level: 2 message: Dominio non trovato: no domain with matching uuid 'fd470849-17c9-436e-818d-16232f5b032b' (oVirtHostedEngine) libvirtEventLoop::DEBUG::2013-08-01 10:42:59,675::sampling::292::vm.Vm::(stop) vmId=`fd470849-17c9-436e-818d-16232f5b032b`::Stop statistics collection libvirtEventLoop::DEBUG::2013-08-01 10:42:59,675::vmChannels::205::vds::(unregister) Delete fileno 32 from listener. Thread-44::DEBUG::2013-08-01 10:42:59,675::sampling::323::vm.Vm::(run) vmId=`fd470849-17c9-436e-818d-16232f5b032b`::Stats thread finished libvirtEventLoop::DEBUG::2013-08-01 10:42:59,675::libvirtconnection::101::libvirtconnection::(wrapper) Unknown libvirterror: ecode: 42 edom: 10 level: 2 message: Dominio non trovato: no domain with matching uuid 'fd470849-17c9-436e-818d-16232f5b032b' (oVirtHostedEngine) libvirtEventLoop::DEBUG::2013-08-01 10:42:59,676::task::579::TaskManager.Task::(_updateState) Task=`9be47a03-b9d9-4686-b43b-9764070a0e4f`::moving from state init -> state preparing libvirtEventLoop::INFO::2013-08-01 10:42:59,676::logUtils::44::dispatcher::(wrapper) Run and protect: teardownImage(sdUUID='ab35a1ff-700b-4354-8539-bc5f0daa6348', spUUID='0c66b59f-bfa5-475d-82c8-ac8878db2565', imgUUID='9ac2ea13-1de5-4a60-83c5-8700a23203b7', volUUID=None) libvirtEventLoop::DEBUG::2013-08-01 10:42:59,676::resourceManager::197::ResourceManager.Request::(__init__) ResName=`Storage.ab35a1ff-700b-4354-8539-bc5f0daa6348`ReqID=`7b99cc2f-21fe-4cd3-9ed3-4be8ffb3b37a`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '3299' at 'teardownImage' libvirtEventLoop::DEBUG::2013-08-01 10:42:59,676::resourceManager::541::ResourceManager::(registerResource) Trying to register resource 'Storage.ab35a1ff-700b-4354-8539-bc5f0daa6348' for lock type 'shared' libvirtEventLoop::DEBUG::2013-08-01 10:42:59,676::resourceManager::600::ResourceManager::(registerResource) Resource 'Storage.ab35a1ff-700b-4354-8539-bc5f0daa6348' is free. Now locking as 'shared' (1 active user) libvirtEventLoop::DEBUG::2013-08-01 10:42:59,676::resourceManager::237::ResourceManager.Request::(grant) ResName=`Storage.ab35a1ff-700b-4354-8539-bc5f0daa6348`ReqID=`7b99cc2f-21fe-4cd3-9ed3-4be8ffb3b37a`::Granted request libvirtEventLoop::DEBUG::2013-08-01 10:42:59,677::task::811::TaskManager.Task::(resourceAcquired) Task=`9be47a03-b9d9-4686-b43b-9764070a0e4f`::_resourcesAcquired: Storage.ab35a1ff-700b-4354-8539-bc5f0daa6348 (shared) libvirtEventLoop::DEBUG::2013-08-01 10:42:59,677::task::974::TaskManager.Task::(_decref) Task=`9be47a03-b9d9-4686-b43b-9764070a0e4f`::ref 1 aborting False libvirtEventLoop::INFO::2013-08-01 10:42:59,677::logUtils::47::dispatcher::(wrapper) Run and protect: teardownImage, Return response: None libvirtEventLoop::DEBUG::2013-08-01 10:42:59,677::task::1168::TaskManager.Task::(prepare) Task=`9be47a03-b9d9-4686-b43b-9764070a0e4f`::finished: None libvirtEventLoop::DEBUG::2013-08-01 10:42:59,677::task::579::TaskManager.Task::(_updateState) Task=`9be47a03-b9d9-4686-b43b-9764070a0e4f`::moving from state preparing -> state finished libvirtEventLoop::DEBUG::2013-08-01 10:42:59,677::resourceManager::939::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.ab35a1ff-700b-4354-8539-bc5f0daa6348': < ResourceRef 'Storage.ab35a1ff-700b-4354-8539-bc5f0daa6348', isValid: 'True' obj: 'None'>} libvirtEventLoop::DEBUG::2013-08-01 10:42:59,677::resourceManager::976::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} libvirtEventLoop::DEBUG::2013-08-01 10:42:59,677::resourceManager::615::ResourceManager::(releaseResource) Trying to release resource 'Storage.ab35a1ff-700b-4354-8539-bc5f0daa6348' libvirtEventLoop::DEBUG::2013-08-01 10:42:59,677::resourceManager::634::ResourceManager::(releaseResource) Released resource 'Storage.ab35a1ff-700b-4354-8539-bc5f0daa6348' (0 active users) libvirtEventLoop::DEBUG::2013-08-01 10:42:59,677::resourceManager::640::ResourceManager::(releaseResource) Resource 'Storage.ab35a1ff-700b-4354-8539-bc5f0daa6348' is free, finding out if anyone is waiting for it. libvirtEventLoop::DEBUG::2013-08-01 10:42:59,677::resourceManager::648::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.ab35a1ff-700b-4354-8539-bc5f0daa6348', Clearing records. libvirtEventLoop::DEBUG::2013-08-01 10:42:59,678::task::974::TaskManager.Task::(_decref) Task=`9be47a03-b9d9-4686-b43b-9764070a0e4f`::ref 0 aborting False libvirtEventLoop::WARNING::2013-08-01 10:42:59,678::clientIF::384::vds::(teardownVolumePath) Drive is not a vdsm image: VOLWM_CHUNK_MB:1024 VOLWM_CHUNK_REPLICATE_MULT:2 VOLWM_FREE_PCT:50 _blockDev:False _checkIoTuneCategories:<bound method Drive._checkIoTuneCategories of <vm.Drive object at 0x2f14210>> _customize:<bound method Drive._customize of <vm.Drive object at 0x2f14210>> _deviceXML:<disk device="cdrom" snapshot="no" type="file"><source file="/home/Fedora-19-x86_64-DVD.iso" startupPolicy="optional"/><target bus="ide" dev="hdc"/><serial></serial></disk> _makeName:<bound method Drive._makeName of <vm.Drive object at 0x2f14210>> _validateIoTuneParams:<bound method Drive._validateIoTuneParams of <vm.Drive object at 0x2f14210>> address:{u'bus': u'1', u'controller': u'0', u'type': u'drive', u'target': u'0', u'unit': u'0'} alias:ide0-1-0 apparentsize:0 blockDev:False cache:none conf:{'status': 'Up', 'bridge': 'ovirtmgmt', 'vmId': 'fd470849-17c9-436e-818d-16232f5b032b', 'pid': '26846', 'drives': [{'index': '0', 'domainID': 'ab35a1ff-700b-4354-8539-bc5f0daa6348', 'reqsize': '0', 'name': u'hda', 'format': 'raw', 'volumeInfo': {'path': '/rhev/data-center/0c66b59f-bfa5-475d-82c8-ac8878db2565/ab35a1ff-700b-4354-8539-bc5f0daa6348/images/9ac2ea13-1de5-4a60-83c5-8700a23203b7/1b63a000-a23e-46ad-9283-e7227cca4879', 'volType': 'path'}, 'address': {u'bus': u'0', u'controller': u'0', u'type': u'drive', u'target': u'0', u'unit': u'0'}, 'volumeID': '1b63a000-a23e-46ad-9283-e7227cca4879', 'apparentsize': '26843545600', 'imageID': '9ac2ea13-1de5-4a60-83c5-8700a23203b7', 'alias': u'ide0-0-0', 'readonly': 'False', 'iface': 'ide', 'truesize': '0', 'poolID': '0c66b59f-bfa5-475d-82c8-ac8878db2565', 'device': 'disk', 'shared': False, 'path': '/rhev/data-center/0c66b59f-bfa5-475d-82c8-ac8878db2565/ab35a1ff-700b-4354-8539-bc5f0daa6348/images/9ac2ea13-1de5-4a60-83c5-8700a23203b7/1b63a000-a23e-46ad-9283-e7227cca4879', 'propagateErrors': 'off', 'type': 'disk', 'volumeChain': [{'domainID': 'ab35a1ff-700b-4354-8539-bc5f0daa6348', 'vmVolInfo': {'path': '/rhev/data-center/0c66b59f-bfa5-475d-82c8-ac8878db2565/ab35a1ff-700b-4354-8539-bc5f0daa6348/images/9ac2ea13-1de5-4a60-83c5-8700a23203b7/1b63a000-a23e-46ad-9283-e7227cca4879', 'volType': 'path'}, 'leaseOffset': 0, 'volumeID': '1b63a000-a23e-46ad-9283-e7227cca4879', 'leasePath': '/rhev/data-center/0c66b59f-bfa5-475d-82c8-ac8878db2565/ab35a1ff-700b-4354-8539-bc5f0daa6348/images/9ac2ea13-1de5-4a60-83c5-8700a23203b7/1b63a000-a23e-46ad-9283-e7227cca4879.lease', 'imageID': '9ac2ea13-1de5-4a60-83c5-8700a23203b7', 'shared': False, 'path': '/rhev/data-center/0c66b59f-bfa5-475d-82c8-ac8878db2565/ab35a1ff-700b-4354-8539-bc5f0daa6348/images/9ac2ea13-1de5-4a60-83c5-8700a23203b7/1b63a000-a23e-46ad-9283-e7227cca4879'}]}, {'index': 2, 'iface': 'ide', 'name': u'hdc', 'format': 'raw', 'address': {u'bus': u'1', u'controller': u'0', u'type': u'drive', u'target': u'0', u'unit': u'0'}, 'alias': u'ide0-1-0', 'readonly': 'True', 'propagateErrors': 'off', 'shared': False, 'device': 'cdrom', 'path': '/home/Fedora-19-x86_64-DVD.iso', 'truesize': 0, 'type': 'disk'}], 'cdrom': '/home/Fedora-19-x86_64-DVD.iso', 'displaySecurePort': u'5900', 'displayPort': '-1', 'pauseCode': 'NOERR', 'clientIp': '127.0.0.1', 'nicModel': 'virtio', 'macAddr': '00:16:3e:4f:10:9a', 'vmName': 'oVirtHostedEngine', 'boot': 'd', 'devices': [{'device': 'memballoon', 'specParams': {'model': 'none'}, 'type': 'balloon'}, {'device': 'virtio-serial', 'alias': u'virtio-serial0', 'type': 'controller', 'address': {u'slot': u'0x04', u'bus': u'0x00', u'domain': u'0x0000', u'type': u'pci', u'function': u'0x0'}}, {'device': 'qxl', 'specParams': {'vram': '65536'}, 'alias': u'video0', 'type': 'video', 'address': {u'slot': u'0x02', u'bus': u'0x00', u'domain': u'0x0000', u'type': u'pci', u'function': u'0x0'}}, {'nicModel': 'virtio', 'macAddr': '00:16:3e:4f:10:9a', 'linkActive': True, 'network': 'ovirtmgmt', 'alias': u'net0', 'address': {u'slot': u'0x03', u'bus': u'0x00', u'domain': u'0x0000', u'type': u'pci', u'function': u'0x0'}, 'device': 'bridge', 'type': 'interface', 'name': u'vnet0'}, {'index': '0', 'domainID': 'ab35a1ff-700b-4354-8539-bc5f0daa6348', 'reqsize': '0', 'name': u'hda', 'format': 'raw', 'volumeInfo': {'path': '/rhev/data-center/0c66b59f-bfa5-475d-82c8-ac8878db2565/ab35a1ff-700b-4354-8539-bc5f0daa6348/images/9ac2ea13-1de5-4a60-83c5-8700a23203b7/1b63a000-a23e-46ad-9283-e7227cca4879', 'volType': 'path'}, 'address': {u'bus': u'0', u'controller': u'0', u'type': u'drive', u'target': u'0', u'unit': u'0'}, 'volumeID': '1b63a000-a23e-46ad-9283-e7227cca4879', 'apparentsize': '26843545600', 'imageID': '9ac2ea13-1de5-4a60-83c5-8700a23203b7', 'alias': u'ide0-0-0', 'readonly': 'False', 'iface': 'ide', 'truesize': '0', 'poolID': '0c66b59f-bfa5-475d-82c8-ac8878db2565', 'device': 'disk', 'shared': False, 'path': '/rhev/data-center/0c66b59f-bfa5-475d-82c8-ac8878db2565/ab35a1ff-700b-4354-8539-bc5f0daa6348/images/9ac2ea13-1de5-4a60-83c5-8700a23203b7/1b63a000-a23e-46ad-9283-e7227cca4879', 'propagateErrors': 'off', 'type': 'disk', 'volumeChain': [{'domainID': 'ab35a1ff-700b-4354-8539-bc5f0daa6348', 'vmVolInfo': {'path': '/rhev/data-center/0c66b59f-bfa5-475d-82c8-ac8878db2565/ab35a1ff-700b-4354-8539-bc5f0daa6348/images/9ac2ea13-1de5-4a60-83c5-8700a23203b7/1b63a000-a23e-46ad-9283-e7227cca4879', 'volType': 'path'}, 'leaseOffset': 0, 'volumeID': '1b63a000-a23e-46ad-9283-e7227cca4879', 'leasePath': '/rhev/data-center/0c66b59f-bfa5-475d-82c8-ac8878db2565/ab35a1ff-700b-4354-8539-bc5f0daa6348/images/9ac2ea13-1de5-4a60-83c5-8700a23203b7/1b63a000-a23e-46ad-9283-e7227cca4879.lease', 'imageID': '9ac2ea13-1de5-4a60-83c5-8700a23203b7', 'shared': False, 'path': '/rhev/data-center/0c66b59f-bfa5-475d-82c8-ac8878db2565/ab35a1ff-700b-4354-8539-bc5f0daa6348/images/9ac2ea13-1de5-4a60-83c5-8700a23203b7/1b63a000-a23e-46ad-9283-e7227cca4879'}]}, {'index': 2, 'iface': 'ide', 'name': u'hdc', 'format': 'raw', 'address': {u'bus': u'1', u'controller': u'0', u'type': u'drive', u'target': u'0', u'unit': u'0'}, 'alias': u'ide0-1-0', 'readonly': 'True', 'propagateErrors': 'off', 'shared': False, 'device': 'cdrom', 'path': '/home/Fedora-19-x86_64-DVD.iso', 'truesize': 0, 'type': 'disk'}, {'device': u'usb', 'alias': u'usb0', 'type': 'controller', 'address': {u'slot': u'0x01', u'bus': u'0x00', u'domain': u'0x0000', u'type': u'pci', u'function': u'0x2'}}, {'device': u'ide', 'alias': u'ide0', 'type': 'controller', 'address': {u'slot': u'0x01', u'bus': u'0x00', u'domain': u'0x0000', u'type': u'pci', u'function': u'0x1'}}, {'device': u'unix', 'alias': u'channel0', 'type': u'channel', 'address': {u'bus': u'0', u'controller': u'0', u'type': u'virtio-serial', u'port': u'1'}}, {'device': u'unix', 'alias': u'channel1', 'type': u'channel', 'address': {u'bus': u'0', u'controller': u'0', u'type': u'virtio-serial', u'port': u'2'}}, {'device': u'spicevmc', 'alias': u'channel2', 'type': u'channel', 'address': {u'bus': u'0', u'controller': u'0', u'type': u'virtio-serial', u'port': u'3'}}], 'smp': '2', 'vmType': 'kvm', 'timeOffset': -500L, 'memSize': '4096', 'spiceSecureChannels': 'smain,sdisplay,sinputs,scursor,splayback,srecord,ssmartcard,susbredir', 'displayIp': '0', 'display': 'qxl'} createXmlElem:<bound method Drive.createXmlElem of <vm.Drive object at 0x2f14210>> device:cdrom drv:raw format:raw getNextVolumeSize:<bound method Drive.getNextVolumeSize of <vm.Drive object at 0x2f14210>> getXML:<bound method Drive.getXML of <vm.Drive object at 0x2f14210>> iface:ide index:2 isDiskReplicationInProgress:<bound method Drive.isDiskReplicationInProgress of <vm.Drive object at 0x2f14210>> isVdsmImage:<bound method Drive.isVdsmImage of <vm.Drive object at 0x2f14210>> log:<logUtils.SimpleLogAdapter object at 0x2f23650> name:hdc networkDev:False path:/home/Fedora-19-x86_64-DVD.iso propagateErrors:off readonly:True reqsize:0 serial: shared:False truesize:0 type:cdrom volExtensionChunk:1024 watermarkLimit:536870912 Traceback (most recent call last): File "/usr/share/vdsm/clientIF.py", line 378, in teardownVolumePath res = self.irs.teardownImage(drive['domainID'], File "/usr/share/vdsm/vm.py", line 1343, in __getitem__ raise KeyError(key) KeyError: 'domainID' libvirtEventLoop::DEBUG::2013-08-01 10:42:59,679::task::579::TaskManager.Task::(_updateState) Task=`23ebd2c4-84a1-4abe-a656-fa27d36d8425`::moving from state init -> state preparing libvirtEventLoop::INFO::2013-08-01 10:42:59,679::logUtils::44::dispatcher::(wrapper) Run and protect: inappropriateDevices(thiefId='fd470849-17c9-436e-818d-16232f5b032b') libvirtEventLoop::INFO::2013-08-01 10:42:59,680::logUtils::47::dispatcher::(wrapper) Run and protect: inappropriateDevices, Return response: None libvirtEventLoop::DEBUG::2013-08-01 10:42:59,680::task::1168::TaskManager.Task::(prepare) Task=`23ebd2c4-84a1-4abe-a656-fa27d36d8425`::finished: None libvirtEventLoop::DEBUG::2013-08-01 10:42:59,680::task::579::TaskManager.Task::(_updateState) Task=`23ebd2c4-84a1-4abe-a656-fa27d36d8425`::moving from state preparing -> state finished libvirtEventLoop::DEBUG::2013-08-01 10:42:59,680::resourceManager::939::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} libvirtEventLoop::DEBUG::2013-08-01 10:42:59,681::resourceManager::976::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} libvirtEventLoop::DEBUG::2013-08-01 10:42:59,681::task::974::TaskManager.Task::(_decref) Task=`23ebd2c4-84a1-4abe-a656-fa27d36d8425`::ref 0 aborting False libvirtEventLoop::ERROR::2013-08-01 10:42:59,681::clientIF::612::vds::(dispatchLibvirtEvents) Error running VM callback Traceback (most recent call last): File "/usr/share/vdsm/clientIF.py", line 584, in dispatchLibvirtEvents v._onLibvirtLifecycleEvent(event, detail, None) File "/usr/share/vdsm/vm.py", line 4728, in _onLibvirtLifecycleEvent self._onQemuDeath() File "/usr/share/vdsm/vm.py", line 2095, in _onQemuDeath response = self.releaseVm() File "/usr/share/vdsm/vm.py", line 4246, in releaseVm self.cif.removeVmFromMonitoredDomains(self.id) File "/usr/share/vdsm/clientIF.py", line 125, in removeVmFromMonitoredDomains for dom in self.domainVmIds: RuntimeError: dictionary changed size during iteration Thread-24::DEBUG::2013-08-01 10:43:00,274::fileSD::238::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd iflag=direct if=/rhev/data-center/mnt/192.168.1.104:_home_images/ab35a1ff-700b-4354-8539-bc5f0daa6348/dom_md/metadata bs=4096 count=1' (cwd None)
The VM is in the following state: # vdsClient -s localhost getVmStats fd470849-17c9-436e-818d-16232f5b032b
fd470849-17c9-436e-818d-16232f5b032b Status = Powering down username = Unknown memUsage = 0 acpiEnable = true guestFQDN = displayPort = -1 session = Unknown displaySecurePort = 5900 timeOffset = -500 balloonInfo = {} pauseCode = NOERR network = {'vnet0': {'macAddr': '00:16:3e:4f:10:9a', 'rxDropped': '0', 'txDropped': '0', 'rxErrors': '0', 'txRate': '0.0', 'rxRate': '0.0', 'txErrors': '0', 'state': 'unknown', 'speed': '1000', 'name': 'vnet0'}} displayType = qxl cpuUser = 0.69 boot = d elapsedTime = 3400 vmType = kvm cpuSys = 0.13 appsList = [] hash = -6106157858121768929 pid = 26846 displayIp = 0 cdrom = /home/Fedora-19-x86_64-DVD.iso guestIPs = kvmEnable = true disks = {'hdc': {'readLatency': '0', 'apparentsize': '0', 'writeLatency': '0', 'flushLatency': '0', 'readRate': '0.00', 'truesize': '0', 'writeRate': '0.00'}, 'hda': {'readLatency': '0', 'apparentsize': '26843545600', 'writeLatency': '0', 'imageID': '9ac2ea13-1de5-4a60-83c5-8700a23203b7', 'flushLatency': '0', 'readRate': '0.00', 'truesize': '1594064896', 'writeRate': '0.00'}} monitorResponse = -1 statsAge = 559.96 clientIp = 127.0.0.1