vdsm-patches December 2012

vdsm-patches@lists.fedorahosted.org

51 participants
408 discussions

Change in vdsm[master]: logUtils: a timed log throttle filter for stats thread
by zhshzhou＠linux.vnet.ibm.com 30 Oct '13

30 Oct '13

Zhou Zheng Sheng has uploaded a new change for review. Change subject: logUtils: a timed log throttle filter for stats thread ...................................................................... logUtils: a timed log throttle filter for stats thread The exception log of stats thread some times contains a lot of the same messages. The normal stats log is very verbose as well. This patch adds a timed throttle filter to the stats log. If the a message in the same file same line (and same exception) is raised for above threshold times in a certain interval, the noise will be cuts down. Bug-Url: https://bugzilla.redhat.com/show_bug.cgi?id=853725 Change-Id: I60fe36ce6b324941d2cbb60ac9b40ed85dcdb858 Signed-off-by: Zhou Zheng Sheng <zhshzhou(a)linux.vnet.ibm.com> --- M vdsm.spec.in M vdsm/logUtils.py 2 files changed, 40 insertions(+), 1 deletion(-) git pull ssh://gerrit.ovirt.org:29418/vdsm refs/changes/89/9689/1 diff --git a/vdsm.spec.in b/vdsm.spec.in index 45cd87a..c8b1281 100644 --- a/vdsm.spec.in +++ b/vdsm.spec.in @@ -137,6 +137,7 @@ Requires: %{name}-python = %{version}-%{release} Requires: pyparted Requires: mom >= 0.3.0 +Requires: python-repoze-lru Requires(pre,preun): policycoreutils-python Requires(post): /usr/sbin/saslpasswd2 diff --git a/vdsm/logUtils.py b/vdsm/logUtils.py index 28c9e0a..ef082f8 100644 --- a/vdsm/logUtils.py +++ b/vdsm/logUtils.py @@ -20,10 +20,13 @@ import logging import sys +import time from functools import wraps from logging.handlers import WatchedFileHandler from logging.handlers import RotatingFileHandler from inspect import ismethod +from collections import deque +from repoze.lru import LRUCache def funcName(func): @@ -112,6 +115,41 @@ return 1 +class TimedThrottleFilter(logging.Filter): + """ + Throttle duplicated log messages if the same message is emitted too fast. + """ + + def __init__(self, threshold=3, interval=10, cacheSize=1024): + """ + Throttle the same message if it is emitted threshold times in + interval seconds. + """ + self._threshold = threshold + self._interval = interval + self._cache = LRUCache(cacheSize) + + def _makeKey(self, record): + key = record.name + record.pathname + str(record.lineno) + if record.exc_info: + key += str(record.exc_info[0]) # class of the exception raised + return key + + def filter(self, record): + key = self._makeKey(record) + stamps = self._cache.get(key, deque(maxlen=self._threshold)) + if not stamps: + self._cache.put(key, stamps) + + ctime = int(time.time()) + if (len(stamps) < self._threshold or + ctime - stamps[0] > self._interval): + stamps.append(ctime) + return 1 + + return 0 + + class QueueHandler(logging.Handler): """ This handler sends events to a queue. Typically, it would be used together @@ -178,4 +216,4 @@ WatchedFileHandler, TracebackRepeatFilter()) FilteredRotatingFileHandler = getFilteredClass( - RotatingFileHandler, TracebackRepeatFilter()) + RotatingFileHandler, TracebackRepeatFilter(), TimedThrottleFilter(3, 10)) -- To view, visit http://gerrit.ovirt.org/9689 To unsubscribe, visit http://gerrit.ovirt.org/settings Gerrit-MessageType: newchange Gerrit-Change-Id: I60fe36ce6b324941d2cbb60ac9b40ed85dcdb858 Gerrit-PatchSet: 1 Gerrit-Project: vdsm Gerrit-Branch: master Gerrit-Owner: Zhou Zheng Sheng <zhshzhou(a)linux.vnet.ibm.com>

2 11

Change in vdsm[master]: AdvancedStatsThread: Throttle duplicated exception log
by zhshzhou＠linux.vnet.ibm.com 30 Oct '13

30 Oct '13

Zhou Zheng Sheng has uploaded a new change for review. Change subject: AdvancedStatsThread: Throttle duplicated exception log ...................................................................... AdvancedStatsThread: Throttle duplicated exception log When the same exception is raised in the stats function many times, the log message is the same as well, so extra message does not help us on the problem. If the problem is not solved, the same exception will be raised again and again, then the log will contain a lot of garbages. This patch proposes a way to throttle duplicated exception log in stats functions. For the same exception in the same stats function, if it is raised too many times, just suppress the log. Bug-Url: https://bugzilla.redhat.com/show_bug.cgi?id=853725 Change-Id: I2ee04d8d82e2a14b0a003627981c28e5e64a46ab Signed-off-by: Zhou Zheng Sheng <zhshzhou(a)linux.vnet.ibm.com> --- M vdsm/utils.py 1 file changed, 45 insertions(+), 2 deletions(-) git pull ssh://gerrit.ovirt.org:29418/vdsm refs/changes/84/8884/1 diff --git a/vdsm/utils.py b/vdsm/utils.py index 002094a..7fab19a 100644 --- a/vdsm/utils.py +++ b/vdsm/utils.py @@ -291,6 +291,45 @@ return bgn_sample, end_sample, (end_time - bgn_time) + +class _ExceptionLogThrottle(object): + ''' + Record previous exception for each statsFunction, throttle the log + if the same exception is raised too many times. + ''' + + def __init__(self, log, dupThreshold=3): + self._log = log + self._dupThreshold = dupThreshold + self._exceptions = {} + + def _exceptionEqual(self, exc1, exc2): + return exc1.__class__ == exc2.__class__ and exc1.args == exc2.args + + def needSuppress(self, statsFunction, exceptObj): + if (statsFunction not in self._exceptions or + not self._exceptionEqual( + self._exceptions[statsFunction]['exception'], exceptObj)): + + self._exceptions[statsFunction] = {'exception': exceptObj, + 'dupCount': 0} + return False + + dupCount = self._exceptions[statsFunction]['dupCount'] + + if dupCount <= self._dupThreshold: + dupCount += 1 + if dupCount == self._dupThreshold: + self._log.debug( + "Log will be throttled for the same exception: %s" % + exceptObj.message) + self._exceptions[statsFunction] = {'exception': exceptObj, + 'dupCount': dupCount} + return False + + return True + + class AdvancedStatsThread(threading.Thread): """ A thread that runs the registered AdvancedStatsFunction objects @@ -311,6 +350,8 @@ self._statsTime = None self._statsFunctions = [] + + self._logThrottle = _ExceptionLogThrottle(log) def addStatsFunction(self, *args): """ @@ -395,8 +436,10 @@ statsFunction() except Exception, e: if not self.handleStatsException(e): - self._log.error("Stats function failed: %s", - statsFunction, exc_info=True) + if not self._logThrottle.needSuppress( + statsFunction, e): + self._log.error("Stats function failed: %s", + statsFunction, exc_info=True) self._stopEvent.wait(waitInterval) intervalAccum = (intervalAccum + waitInterval) % maxInterval -- To view, visit http://gerrit.ovirt.org/8884 To unsubscribe, visit http://gerrit.ovirt.org/settings Gerrit-MessageType: newchange Gerrit-Change-Id: I2ee04d8d82e2a14b0a003627981c28e5e64a46ab Gerrit-PatchSet: 1 Gerrit-Project: vdsm Gerrit-Branch: master Gerrit-Owner: Zhou Zheng Sheng <zhshzhou(a)linux.vnet.ibm.com>

6 34

Change in vdsm[master]: [WIP] Add virtual HTTP(s) server multiplexing port 54321
by zhshzhou＠linux.vnet.ibm.com 30 Oct '13

30 Oct '13

Zhou Zheng Sheng has uploaded a new change for review. Change subject: [WIP] Add virtual HTTP(s) server multiplexing port 54321 ...................................................................... [WIP] Add virtual HTTP(s) server multiplexing port 54321 Add a request router to route HTTP request to the virtual server, and XMLRPC request to the XMLRPC binding. This allow VDSM provide XMLRPC and HTTP service using the same socket binded at 54321. The request router can be extended to match path patterns and reques method, etc... This virtual HTTP server can be extened to support guest console stream and crash core dump. How to test: 1) This patch can pass the functional tests added recently when ssl = false or ssl = true 2) To test the virtual HTTP server, use the command as follow. When ssl = true wget --no-check-certificate \ --certificate=/etc/pki/vdsm/certs/vdsmcert.pem \ --private-key=/etc/pki/vdsm/keys/vdsmkey.pem \ --ca-certificate=/etc/pki/vdsm/certs/cacert.pem \ https://localhost:54321/etc/vdsm/vdsm.conf When ssl = false wget http://localhost:54321/etc/vdsm/vdsm.conf Change-Id: Ie99bf9353603ae96b0570367e22a01817e5c4e97 Signed-off-by: Zhou Zheng Sheng <zhshzhou(a)linux.vnet.ibm.com> --- M vdsm/BindingXMLRPC.py M vdsm/SecureXMLRPCServer.py 2 files changed, 71 insertions(+), 2 deletions(-) git pull ssh://gerrit.ovirt.org:29418/vdsm refs/changes/05/9605/1 diff --git a/vdsm/BindingXMLRPC.py b/vdsm/BindingXMLRPC.py index d460910..a74d197 100644 --- a/vdsm/BindingXMLRPC.py +++ b/vdsm/BindingXMLRPC.py @@ -21,6 +21,7 @@ import time from errno import EINTR import SimpleXMLRPCServer +import SimpleHTTPServer from vdsm import SecureXMLRPCServer import logging import libvirt @@ -138,15 +139,25 @@ threadLocal.flowID = self.headers.get(HTTP_HEADER_FLOWID) return r + urlRouter = SecureXMLRPCServer.UrlRouter() + urlRouter.registerHandler('/', LoggingHandler) + urlRouter.registerHandler('/RPC2', LoggingHandler) + urlRouter.registerHandler( + '/etc/vdsm/vdsm.conf', SimpleHTTPServer.SimpleHTTPRequestHandler) + # TODO: subclass SimpleHTTPServer.SimpleHTTPRequestHandler to provide + # a HTTP server forwarding the guest serial console stream and gurest + # crash core dump download + if self.enableSSL: KEYFILE, CERTFILE, CACERT = self._getKeyCertFilenames() s = SecureXMLRPCServer.SecureThreadedXMLRPCServer(server_address, keyfile=KEYFILE, certfile=CERTFILE, ca_certs=CACERT, timeout=self.serverRespTimeout, - requestHandler=LoggingHandler) + requestHandler=urlRouter.getHandlerClass()) else: s = utils.SimpleThreadedXMLRPCServer(server_address, - requestHandler=LoggingHandler, logRequests=True) + requestHandler=urlRouter.getHandlerClass(), + logRequests=True) utils.closeOnExec(s.socket.fileno()) s.lastClientTime = 0 diff --git a/vdsm/SecureXMLRPCServer.py b/vdsm/SecureXMLRPCServer.py index 2de1cf7..87ba0f4 100644 --- a/vdsm/SecureXMLRPCServer.py +++ b/vdsm/SecureXMLRPCServer.py @@ -34,6 +34,7 @@ import httplib import socket import SocketServer +import BaseHTTPServer import logging from M2Crypto import SSL, X509 @@ -114,6 +115,63 @@ return client, address +# TODO: extend the router to support path patterns and routing different +# request method (GET, POST) to different handlerClass +class UrlRouter(object): + def __init__(self): + self._handlerClasses = [] + + def registerHandler(self, path, handlerClass): + """Register a handler class with the related path + + In old style class, mro is depth first. Sometimes base handler class + overrides setup() or parse_request() methon as + BindingXMLRPC._createXMLRPCServer.LoggingHandler does. + + For the overriden method to be resolved first, the class must occur + before the ancestor class in the router handlerClass.__bases__ list. + + It is guaranteed that the router handlerClasse.__bases__ obey the + order of the handler class registered. So if you override a handler + method, register it before other handler classes to make the mothed + resoled first. + + In new style class, there is no such problem.""" + + if path in map(lambda (p, _): p, self._handlerClasses): + raise ValueError("Path %s is already registered" % path) + + self._handlerClasses.append((path, handlerClass)) + + def getHandlerClass(self): + handlerClassDict = dict(self._handlerClasses) + + class ReqHandler: + """Route the request to the appropriate handler + + Handles all HTTP GET and POST requests and attempts to route them + by URL. + """ + + def do_POST(self): + handlerClassDict[self.path].do_POST(self) + + def do_GET(self): + handlerClassDict[self.path].do_GET(self) + + bases = [] + # Remove duplicated handlerClass while keep the elements in the copy + # list remained in original order. + for _, handlerClass in self._handlerClasses: + if handlerClass not in bases: + bases.append(handlerClass) + bases.append(BaseHTTPServer.BaseHTTPRequestHandler) + + ReqHandler.__bases__ = tuple(bases) + + return ReqHandler + + class SecureXMLRPCServer(SimpleXMLRPCServer.SimpleXMLRPCServer): def __init__(self, addr, requestHandler=SimpleXMLRPCServer.SimpleXMLRPCRequestHandler, -- To view, visit http://gerrit.ovirt.org/9605 To unsubscribe, visit http://gerrit.ovirt.org/settings Gerrit-MessageType: newchange Gerrit-Change-Id: Ie99bf9353603ae96b0570367e22a01817e5c4e97 Gerrit-PatchSet: 1 Gerrit-Project: vdsm Gerrit-Branch: master Gerrit-Owner: Zhou Zheng Sheng <zhshzhou(a)linux.vnet.ibm.com>

3 12

Change in vdsm[master]: [WIP] remote serial console via HTTP virtual server
by zhshzhou＠linux.vnet.ibm.com 30 Oct '13

30 Oct '13

Zhou Zheng Sheng has uploaded a new change for review. Change subject: [WIP] remote serial console via HTTP virtual server ...................................................................... [WIP] remote serial console via HTTP virtual server How to test: We test the remote serial console using netcat, however netcat do not support SSL, so we perform this test without SSL. Firstly, in /etc/vdsm/vdsm.conf, set "ssl = False", then in the last part of /etc/libvirt/libvirtd.conf set "listen_tls = 0" "listen_tcp = 1" "auth_tcp = "none"" and in the last part of /etc/libvirt/qemu.conf, set "spice_tls=0". Restart libvirtd and vdsmd, create a virtual machine with VDSM. Fedora is the preferable guest OS because it scans serial console and automatically starts a tty programme on it. You can just use a Fedora XFCE live CD to boot your VM if you do not have an existing Fedora VM. Run net cat as follow. nc 127.0.0.1 54321 Paste the following request in net cat. POST /VM/paste-your-VM-uuid-here/dev/console HTTP/1.0 Hit enter twice, you will see the response as follow. HTTP/1.0 200 OK Server: BaseHTTP/0.3 Python/2.7.3 Date: Wed, 26 Dec 2012 09:38:27 GMT Content-type: application/octet-stream After your VM is up, hit enter again, you will get the guest console as follow. It works in full duplex mode. Fedora release 17 (Beefy Miracle) Kernel 3.3.4-5.fc17.x86_64 on an x86_64 (hvc0) localhost login: If you can edit the kernel arguments while booting guest with grub, append "console=ttyS0" and you will see the boot up messages in net cat. Change-Id: I9b2fd0979dac68a7efed4d5bbcb1866cd48a0ccb Signed-off-by: Zhou Zheng Sheng <zhshzhou(a)linux.vnet.ibm.com> --- M vdsm.spec.in M vdsm/BindingXMLRPC.py M vdsm/Makefile.am A vdsm/consoleHandler.py M vdsm/libvirtvm.py M vdsm/supervdsmServer.py M vdsm/vm.py 7 files changed, 140 insertions(+), 6 deletions(-) git pull ssh://gerrit.ovirt.org:29418/vdsm refs/changes/81/10381/1 diff --git a/vdsm.spec.in b/vdsm.spec.in index 465d548..834cfa9 100644 --- a/vdsm.spec.in +++ b/vdsm.spec.in @@ -610,6 +610,7 @@ %{_datadir}/%{vdsm_name}/blkid.py* %{_datadir}/%{vdsm_name}/caps.py* %{_datadir}/%{vdsm_name}/clientIF.py* +%{_datadir}/%{vdsm_name}/consoleHandler.py* %{_datadir}/%{vdsm_name}/API.py* %{_datadir}/%{vdsm_name}/hooking.py* %{_datadir}/%{vdsm_name}/hooks.py* diff --git a/vdsm/BindingXMLRPC.py b/vdsm/BindingXMLRPC.py index 23ef422..742f709 100644 --- a/vdsm/BindingXMLRPC.py +++ b/vdsm/BindingXMLRPC.py @@ -33,6 +33,7 @@ from vdsm.define import doneCode, errCode import API from vdsm.exception import VdsmException +import consoleHandler try: from gluster.api import getGlusterMethods _glusterEnabled = True @@ -142,12 +143,8 @@ urlRouter = SecureXMLRPCServer.UrlRouter() urlRouter.registerHandler('/$', LoggingHandler) urlRouter.registerHandler('/RPC2$', LoggingHandler) - # all files under /etc/vdsm/ urlRouter.registerHandler( - '/etc/vdsm/', SimpleHTTPServer.SimpleHTTPRequestHandler) - # TODO: subclass SimpleHTTPServer.SimpleHTTPRequestHandler to provide - # a HTTP server forwarding the guest serial console stream and guest - # crash core dump download + consoleHandler.CONSOLE_REGEX, consoleHandler.consoleHandler) if self.enableSSL: KEYFILE, CERTFILE, CACERT = self._getKeyCertFilenames() diff --git a/vdsm/Makefile.am b/vdsm/Makefile.am index a687c56..38d85ad 100644 --- a/vdsm/Makefile.am +++ b/vdsm/Makefile.am @@ -30,6 +30,7 @@ caps.py \ clientIF.py \ configNetwork.py \ + consoleHandler.py \ debugPluginClient.py \ guestIF.py \ hooking.py \ diff --git a/vdsm/consoleHandler.py b/vdsm/consoleHandler.py new file mode 100644 index 0000000..1260941 --- /dev/null +++ b/vdsm/consoleHandler.py @@ -0,0 +1,118 @@ +# +# Copyright IBM Corp. 2012 +# +# This program is free software; you can redistribute it and/or modify +# it under the terms of the GNU General Public License as published by +# the Free Software Foundation; either version 2 of the License, or +# (at your option) any later version. +# +# This program is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +# GNU General Public License for more details. +# +# You should have received a copy of the GNU General Public License +# along with this program; if not, write to the Free Software +# Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA +# +# Refer to the README and COPYING files for full details of the license +# +import os +import errno +import asyncore +import BaseHTTPServer +import urlparse + +import supervdsm +import storage.misc +import API + +CONSOLE_REGEX = (r'/VM/' + storage.misc.UUID_REGEX.pattern[1:-1] + + r'/dev/console$') + + +class ForwardSession(asyncore.dispatcher_with_send): + def __init__(self, sock, sockmap): + asyncore.dispatcher_with_send.__init__(self, sock, sockmap) + self._peer = None + + def linkPeer(self, peer): + self._peer = peer + + def unlinkPeer(self): + self._peer = None + + def sessionExit(self): + try: + self.close() + except OSError as e: + if e.errno != errno.EBADF: + raise + myPeer = self._peer + self._peer = None + if myPeer: + myPeer.unlinkPeer() + myPeer.sessionExit() + + def handle_read(self): + if not self._peer: + self.sessionExit() + return + data = self.recv(4096) + if data: + self._peer.send(data) + + def handle_close(self): + self.sessionExit() + + def readable(self): + if not (self._peer and self.connected): + return False + return len(self._peer.out_buffer) < 512 + + +class PTYSession(ForwardSession, asyncore.file_dispatcher): + def __init__(self, pty, sockmap): + ForwardSession.__init__(self, None, sockmap) + asyncore.file_dispatcher.__init__(self, pty, sockmap) + + +class consoleHandler(BaseHTTPServer.BaseHTTPRequestHandler): + wbufsize = 0 + rbufsize = 0 + disable_nagle_algorithm = True + + def do_POST(self): + self.close_connection = 1 + + pty = None + try: + vmid = urlparse.urlparse(self.path).path.split('/')[2] + vm = API.VM(vmid) + stats = vm.getStats() + if stats['status']['code'] != 0: + raise RuntimeError(stats['status']['message']) + ptyDev = stats['statsList'][0]['consolePath'] + supervdsm.getProxy().takeOwnership(ptyDev) + pty = os.open(ptyDev, os.O_RDWR | os.O_NONBLOCK | os.O_NOCTTY) + except Exception as e: + self.send_response(404, "Error open PTY for %s: %r\n" % + (self.path, e)) + self.end_headers() + self.wfile.flush() + if pty: + os.close(pty) + return + + self.send_response(200) + self.send_header("Content-type", "application/octet-stream") + self.end_headers() + self.wfile.flush() + + sockMap = {} + sockSession = ForwardSession(self.request, sockMap) + ptySession = PTYSession(pty, sockMap) + os.close(pty) + sockSession.linkPeer(ptySession) + ptySession.linkPeer(sockSession) + asyncore.loop(use_poll=True, map=sockMap) diff --git a/vdsm/libvirtvm.py b/vdsm/libvirtvm.py index d5b5e4f..a6e9361 100644 --- a/vdsm/libvirtvm.py +++ b/vdsm/libvirtvm.py @@ -1378,6 +1378,7 @@ self._getUnderlyingControllerDeviceInfo() self._getUnderlyingBalloonDeviceInfo() self._getUnderlyingWatchdogDeviceInfo() + self._getUnderlyingConsoleDeviceInfo() # Obtain info of all unknown devices. Must be last! self._getUnderlyingUnknownDeviceInfo() @@ -2881,6 +2882,16 @@ nicDev['network'] = network self.conf['devices'].append(nicDev) + def _getUnderlyingConsoleDeviceInfo(self): + ''' + Obtain console device info from libvirt + ''' + consoleElem = _domParseStr(self._lastXMLDesc).childNodes[0]. \ + getElementsByTagName('devices')[0]. \ + getElementsByTagName('console')[0] + self.conf["consolePath"] = str( + consoleElem.getElementsByTagName('source')[0].getAttribute('path')) + def _setWriteWatermarks(self): """ Define when to receive an event about high write to guest image diff --git a/vdsm/supervdsmServer.py b/vdsm/supervdsmServer.py index 1601ba5..43cc775 100755 --- a/vdsm/supervdsmServer.py +++ b/vdsm/supervdsmServer.py @@ -47,7 +47,8 @@ from storage.fileUtils import chown, resolveGid, resolveUid from storage.fileUtils import validateAccess as _validateAccess from vdsm.constants import METADATA_GROUP, EXT_UDEVADM, \ - DISKIMAGE_USER, DISKIMAGE_GROUP, P_LIBVIRT_VMCHANNELS + DISKIMAGE_USER, DISKIMAGE_GROUP, P_LIBVIRT_VMCHANNELS, \ + VDSM_USER, VDSM_GROUP from storage.devicemapper import _removeMapping, _getPathsStatus import configNetwork from vdsm.config import config @@ -292,6 +293,10 @@ def removeFs(self, path): return mkimage.removeFs(path) + @logDecorator + def takeOwnership(self, path): + chown(path, VDSM_USER, VDSM_GROUP) + def __pokeParent(parentPid, address): try: diff --git a/vdsm/vm.py b/vdsm/vm.py index e73ac6d..5a73558 100644 --- a/vdsm/vm.py +++ b/vdsm/vm.py @@ -1090,6 +1090,7 @@ 'displaySecurePort': self.conf['displaySecurePort'], 'displayType': self.conf['display'], 'displayIp': self.conf['displayIp'], + 'consolePath': self.conf['consolePath'], 'pid': self.conf['pid'], 'vmType': self.conf['vmType'], 'kvmEnable': self._kvmEnable, -- To view, visit http://gerrit.ovirt.org/10381 To unsubscribe, visit http://gerrit.ovirt.org/settings Gerrit-MessageType: newchange Gerrit-Change-Id: I9b2fd0979dac68a7efed4d5bbcb1866cd48a0ccb Gerrit-PatchSet: 1 Gerrit-Project: vdsm Gerrit-Branch: master Gerrit-Owner: Zhou Zheng Sheng <zhshzhou(a)linux.vnet.ibm.com>

4 8

Change in vdsm[master]: Fix vm block stats lost after recovery
by wudxw＠linux.vnet.ibm.com 30 Oct '13

30 Oct '13

Mark Wu has uploaded a new change for review. Change subject: Fix vm block stats lost after recovery ...................................................................... Fix vm block stats lost after recovery We don't need call preparePaths() in the code path of vm recovery, because the 'path' has been prepared on vm creatation and saved in recovery file. But the flag _volumesPrepared still needs to be set to make the block stats can be collected. Change-Id: I59a90cf909a07967f642d348e47b4928f7516c61 Signed-off-by: Mark Wu <wudxw(a)linux.vnet.ibm.com> --- M vdsm/libvirtvm.py 1 file changed, 1 insertion(+), 0 deletions(-) git pull ssh://gerrit.ovirt.org:29418/vdsm refs/changes/56/6056/1 -- To view, visit http://gerrit.ovirt.org/6056 To unsubscribe, visit http://gerrit.ovirt.org/settings Gerrit-MessageType: newchange Gerrit-Change-Id: I59a90cf909a07967f642d348e47b4928f7516c61 Gerrit-PatchSet: 1 Gerrit-Project: vdsm Gerrit-Branch: master Gerrit-Owner: Mark Wu <wudxw(a)linux.vnet.ibm.com>

3 6

Change in vdsm[master]: Source image exception when querying source image
by shuming＠linux.vnet.ibm.com 30 Oct '13

30 Oct '13

Shu Ming has uploaded a new change for review. Change subject: Source image exception when querying source image ...................................................................... Source image exception when querying source image se.StorageException exception should also be converted to se.SourceImageActionError exception Change-Id: I6abb905f1042477aa6757d6e10d6989923e955b4 Signed-off-by: Shu Ming <shuming(a)linux.vnet.ibm.com> --- M vdsm/storage/image.py 1 file changed, 0 insertions(+), 3 deletions(-) git pull ssh://gerrit.ovirt.org:29418/vdsm refs/changes/79/8479/1 diff --git a/vdsm/storage/image.py b/vdsm/storage/image.py index 9a22863..f9cb3a0 100644 --- a/vdsm/storage/image.py +++ b/vdsm/storage/image.py @@ -457,9 +457,6 @@ try: # Find all volumes of source image srcChain = self.getChain(srcSdUUID, imgUUID) - except se.StorageException: - self.log.error("Unexpected error", exc_info=True) - raise except Exception, e: self.log.error("Unexpected error", exc_info=True) raise se.SourceImageActionError(imgUUID, srcSdUUID, str(e)) -- To view, visit http://gerrit.ovirt.org/8479 To unsubscribe, visit http://gerrit.ovirt.org/settings Gerrit-MessageType: newchange Gerrit-Change-Id: I6abb905f1042477aa6757d6e10d6989923e955b4 Gerrit-PatchSet: 1 Gerrit-Project: vdsm Gerrit-Branch: master Gerrit-Owner: Shu Ming <shuming(a)linux.vnet.ibm.com>

3 3

Change in vdsm[master]: '0' is a valid master version
by shuming＠linux.vnet.ibm.com 30 Oct '13

30 Oct '13

Shu Ming has uploaded a new change for review. Change subject: '0' is a valid master version ...................................................................... '0' is a valid master version When connecting the storage pool first time with master domain version '0', exception happens not finding the temporary pool file for pool parameters. Change-Id: I736d917bfa7d3a5ac59cddcaf1a18846f53d85b0 Signed-off-by: Shu Ming <shuming(a)linux.vnet.ibm.com> --- M vdsm/storage/hsm.py 1 file changed, 1 insertion(+), 1 deletion(-) git pull ssh://gerrit.ovirt.org:29418/vdsm refs/changes/48/7948/1 diff --git a/vdsm/storage/hsm.py b/vdsm/storage/hsm.py index 1e9311b..f7f8b49 100644 --- a/vdsm/storage/hsm.py +++ b/vdsm/storage/hsm.py @@ -849,7 +849,7 @@ return pool = sp.StoragePool(spUUID, self.taskMng) - if not hostID or not scsiKey or not msdUUID or not masterVersion: + if not hostID or not scsiKey or not msdUUID or (masterVersion == None): hostID, scsiKey, msdUUID, masterVersion = pool.getPoolParams() res = pool.connect(hostID, scsiKey, msdUUID, masterVersion) if res: -- To view, visit http://gerrit.ovirt.org/7948 To unsubscribe, visit http://gerrit.ovirt.org/settings Gerrit-MessageType: newchange Gerrit-Change-Id: I736d917bfa7d3a5ac59cddcaf1a18846f53d85b0 Gerrit-PatchSet: 1 Gerrit-Project: vdsm Gerrit-Branch: master Gerrit-Owner: Shu Ming <shuming(a)linux.vnet.ibm.com>

4 5

Change in vdsm[master]: persist state in _changeBlockDev
by lvroyce＠linux.vnet.ibm.com 28 Oct '13

28 Oct '13

Royce Lv has uploaded a new change for review. Change subject: persist state in _changeBlockDev ...................................................................... persist state in _changeBlockDev persist state to make vm.conf survive recovery Change-Id: I220b72ebb360356092a2dccc25dfb9f63f638607 Signed-off-by: Royce Lv <lvroyce(a)linux.vnet.ibm.com> --- M vdsm/libvirtvm.py 1 file changed, 2 insertions(+), 0 deletions(-) git pull ssh://gerrit.ovirt.org:29418/vdsm refs/changes/17/8617/1 diff --git a/vdsm/libvirtvm.py b/vdsm/libvirtvm.py index bd43b56..abd8440 100644 --- a/vdsm/libvirtvm.py +++ b/vdsm/libvirtvm.py @@ -2028,6 +2028,8 @@ ['message']}} self.cif.teardownVolumePath(self.conf.get(vmDev)) self.conf[vmDev] = path + # persist state to keep vm conf consistency in recovery + self.saveState() return {'status': doneCode, 'vmList': self.status()} def setTicket(self, otp, seconds, connAct, params): -- To view, visit http://gerrit.ovirt.org/8617 To unsubscribe, visit http://gerrit.ovirt.org/settings Gerrit-MessageType: newchange Gerrit-Change-Id: I220b72ebb360356092a2dccc25dfb9f63f638607 Gerrit-PatchSet: 1 Gerrit-Project: vdsm Gerrit-Branch: master Gerrit-Owner: Royce Lv <lvroyce(a)linux.vnet.ibm.com>

2 2

Change in vdsm[master]: Move start_needed_srv and shutdown_conflicting_srv to vdsm-t...
by wenyi＠linux.vnet.ibm.com 27 Oct '13

27 Oct '13

Wenyi Gao has uploaded a new change for review. Change subject: Move start_needed_srv and shutdown_conflicting_srv to vdsm-tool ...................................................................... Move start_needed_srv and shutdown_conflicting_srv to vdsm-tool Change-Id: I54eca4b2cc3a17e9819a0188e5dc4c8bd28161da Signed-off-by: Wenyi Gao <wenyi(a)linux.vnet.ibm.com> --- M .gitignore M vdsm-tool/Makefile.am A vdsm-tool/base.py A vdsm-tool/needed_service.py.in M vdsm.spec.in M vdsm/vdsmd.init.in 6 files changed, 145 insertions(+), 42 deletions(-) git pull ssh://gerrit.ovirt.org:29418/vdsm refs/changes/59/8759/1 diff --git a/.gitignore b/.gitignore index 49a78ef..f9c26d4 100644 --- a/.gitignore +++ b/.gitignore @@ -41,6 +41,7 @@ vdsm/vdsmd.8 vdsm/vdsmd.init vdsm-tool/load_needed_modules.py +vdsm-tool/needed_service.py vdsm-tool/validate_ovirt_certs.py vdsm_cli/vdsClient vdsm_cli/vdscli.py diff --git a/vdsm-tool/Makefile.am b/vdsm-tool/Makefile.am index 997e339..0ed3226 100644 --- a/vdsm-tool/Makefile.am +++ b/vdsm-tool/Makefile.am @@ -27,7 +27,9 @@ dist_vdsmtool_DATA = \ __init__.py \ + base.py \ load_needed_modules.py \ + needed_service.py \ passwd.py \ validate_ovirt_certs.py \ $(NULL) diff --git a/vdsm-tool/base.py b/vdsm-tool/base.py new file mode 100644 index 0000000..9ded9a1 --- /dev/null +++ b/vdsm-tool/base.py @@ -0,0 +1,77 @@ +# Copyright IBM, Corp. 2012 +# +# This program is free software; you can redistribute it and/or modify +# it under the terms of the GNU General Public License as published by +# the Free Software Foundation; either version 2 of the License, or +# (at your option) any later version. +# +# This program is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +# GNU General Public License for more details. +# +# You should have received a copy of the GNU General Public License +# along with this program; if not, write to the Free Software +# Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA +# +# Refer to the README and COPYING files for full details of the license +# + +import subprocess + + +def exec_command(argv): + """ + This function executes a given shell command. + """ + try: + p = subprocess.Popen(argv, stdout=subprocess.PIPE, + stderr=subprocess.PIPE) + out, err = p.communicate() + rc = p.returncode + except: + raise Exception('Failed to execute "%s"', ' '.join(argv), + exc_info=True) + + return (rc, out, err) + + +def call_init_function(function, *args, **kwargs): + """ + Call a function from /etc/init.d/functions. + """ + allowed_init_function = ['success', 'failure', 'pidofproc', + 'killproc', 'daemon'] + rc = -1 + srccmd = 'SYSTEMCTL_SKIP_REDIRECT=true source /etc/init.d/functions;' + + if function not in allowed_init_function: + print('Illegal function "%s"', function) + return rc + + tmpcmd = function + ' '.join('%s=%s' % item for item in kwargs.iteritems()) + tmpcmd += ' ' + ' '.join(args) + # Only allow to execute one shell command. + if any(c in tmpcmd for c in (';', '&&', '||')): + print('Illegal function parameters "%s"', tmpcmd) + return rc + + cmd = srccmd + tmpcmd + try: + rc = subprocess.call([cmd], shell=True) + except: + raise Exception('Failed to execute "%s"', cmd, exc_info=True) + + return rc + + +def print_msg(msg, success=True): + """ + Print message followed by a color string "[ OK ]" or "[Failed]". + """ + print msg, + if success: + call_init_function('success') + else: + call_init_function('failure') + print diff --git a/vdsm-tool/needed_service.py.in b/vdsm-tool/needed_service.py.in new file mode 100644 index 0000000..5266408 --- /dev/null +++ b/vdsm-tool/needed_service.py.in @@ -0,0 +1,61 @@ +# Copyright IBM, Corp. 2012 +# +# This program is free software; you can redistribute it and/or modify +# it under the terms of the GNU General Public License as published by +# the Free Software Foundation; either version 2 of the License, or +# (at your option) any later version. +# +# This program is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +# GNU General Public License for more details. +# +# You should have received a copy of the GNU General Public License +# along with this program; if not, write to the Free Software +# Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA +# +# Refer to the README and COPYING files for full details of the license +# + +import os + +from vdsm import constants +from vdsm.tool.base import exec_command, print_msg +from vdsm.tool import expose + +EX_CHKCONFIG = '@CHKCONFIG_PATH@' +EX_SERVICE = constants.EXT_SERVICE + + +@expose('shutdown-conflicting-srv') +def shutdown_conflicting_srv(): + """ + Shutdown conflicting service + """ + srv = 'libvirt-guests' + exec_command([EX_CHKCONFIG, srv, 'off']) + rc = exec_command([EX_SERVICE, srv, 'status'])[0] + if rc == 0: + if os.path.exists('/var/lock/subsys/libvirt-guests'): + os.unlink('/var/lock/subsys/libvirt-guests') + else: + exec_command([EX_SERVICE, srv, 'stop']) + return 0 + + +@expose('start-needed-srv') +def start_needed_srv(): + """ + Start needed service + """ + services = ['iscsid', 'multipathd', 'ntpd', 'wdmd', 'sanlock'] + for srv in services: + rc = exec_command([EX_SERVICE, srv, 'status'])[0] + if rc != 0: + print 'Starting %s...' % srv + rc = exec_command([EX_SERVICE, srv, 'start'])[0] + if rc != 0: + print_msg('vdsm: Dependent %s failed to start' % srv, False) + return rc + exec_command([EX_SERVICE, 'iscsid', 'force-start']) + return 0 diff --git a/vdsm.spec.in b/vdsm.spec.in index f61a36a..dc7ee24 100644 --- a/vdsm.spec.in +++ b/vdsm.spec.in @@ -771,6 +771,8 @@ %{python_sitearch}/%{vdsm_name}/tool/passwd.py* %{python_sitearch}/%{vdsm_name}/tool/validate_ovirt_certs.py* %{python_sitearch}/%{vdsm_name}/tool/load_needed_modules.py* +%{python_sitearch}/%{vdsm_name}/tool/needed_service.py* +%{python_sitearch}/%{vdsm_name}/tool/base.py* %files tests %doc %{_datadir}/%{vdsm_name}/tests/README diff --git a/vdsm/vdsmd.init.in b/vdsm/vdsmd.init.in index 2b9ad11..35fb05e 100755 --- a/vdsm/vdsmd.init.in +++ b/vdsm/vdsmd.init.in @@ -32,8 +32,6 @@ CORE_DUMP_PATH=/var/log/core/core.%p.%t.dump DOM_METADATA_BACKUP_DIR=/var/log/vdsm/backup CORE_PATTERN=/proc/sys/kernel/core_pattern -NEEDED_SERVICES="iscsid multipathd ntpd wdmd sanlock" -CONFLICTING_SERVICES="libvirt-guests" LCONF=/etc/libvirt/libvirtd.conf QCONF=/etc/libvirt/qemu.conf @@ -130,49 +128,11 @@ fi } -shutdown_conflicting_srv() { - local srv - - for srv in $CONFLICTING_SERVICES - do - /sbin/chkconfig $srv off - if /sbin/service $srv status > /dev/null 2>&1; - then - if [ "$srv" == "libvirt-guests" ]; then - /bin/rm -f /var/lock/subsys/libvirt-guests - else - /sbin/service $srv stop - fi - fi - done - return 0 -} libvirt_should_use_upstart() { [[ -x /sbin/initctl ]] } -start_needed_srv() { - local srv - local ret_val - - for srv in $NEEDED_SERVICES - do - if ! /sbin/service $srv status > /dev/null 2>&1; - then - echo "Starting $srv..." - /sbin/service $srv start - ret_val=$? - if [ $ret_val -ne 0 ] - then - log_failure_msg "$prog: Dependent $srv failed to start" - return $ret_val - fi - fi - done - - /sbin/service iscsid force-start -} test_lo() { if ! LC_ALL=C /sbin/ifconfig lo | /bin/grep -q UP; @@ -428,7 +388,7 @@ local ret_val python @VDSMDIR@/hooks.pyc before_vdsm_start - shutdown_conflicting_srv && stop_libvirtd_sysv + /usr/bin/vdsm-tool shutdown-conflicting-srv && stop_libvirtd_sysv if ! @LIBEXECDIR@/vdsm-gencerts.sh --check; then echo -n $"Configuring a self-signed VDSM host certificate: " @@ -443,7 +403,7 @@ return $ret_val fi - start_needed_srv && start_libvirtd + /usr/bin/vdsm-tool start-needed-srv && start_libvirtd ret_val=$? if [ $ret_val -ne 0 ] then -- To view, visit http://gerrit.ovirt.org/8759 To unsubscribe, visit http://gerrit.ovirt.org/settings Gerrit-MessageType: newchange Gerrit-Change-Id: I54eca4b2cc3a17e9819a0188e5dc4c8bd28161da Gerrit-PatchSet: 1 Gerrit-Project: vdsm Gerrit-Branch: master Gerrit-Owner: Wenyi Gao <wenyi(a)linux.vnet.ibm.com>

3 7

Change in vdsm[master]: minimalistic (incomplete) test for mkimage.py with fail for ...
by josef.pacula＠gmail.com 25 Oct '13

25 Oct '13

Josef Pacula has uploaded a new change for review. Change subject: minimalistic (incomplete) test for mkimage.py with fail for non-64base data ...................................................................... minimalistic (incomplete) test for mkimage.py with fail for non-64base data Change-Id: I1a7656359beb5d116ad1919e237fe084a1208d71 Signed-off-by: Josef Pacula <josef.pacula(a)gmail.com> --- M tests/Makefile.am A tests/mkimageTests.py M vdsm/mkimage.py 3 files changed, 20 insertions(+), 1 deletion(-) git pull ssh://gerrit.ovirt.org:29418/vdsm refs/changes/75/6275/1 -- To view, visit http://gerrit.ovirt.org/6275 To unsubscribe, visit http://gerrit.ovirt.org/settings Gerrit-MessageType: newchange Gerrit-Change-Id: I1a7656359beb5d116ad1919e237fe084a1208d71 Gerrit-PatchSet: 1 Gerrit-Project: vdsm Gerrit-Branch: master Gerrit-Owner: Josef Pacula <josef.pacula(a)gmail.com>

4 10

← Newer
1
...
8
9
10
11
12
13
14
...
41
Older →

Jump to page:

2026

2025

2024

2023

2022

2021

2020

2019

2018

2017

2016

2015

2014

2013

2012

2011

vdsm-patches December 2012