Change in vdsm[master]: [WIP] Added glusterVolumeTop verb

Hello Ayal Baron, Bala.FA, Saggi Mizrahi, Federico Simoncelli, Dan Kenigsberg,

I'd like you to do a code review.  Please visit

    http://gerrit.ovirt.org/7844

to review the following change.

Change subject: [WIP] Added glusterVolumeTop verb
......................................................................

[WIP] Added glusterVolumeTop verb

Added glusterVolumeTopOpen verb
Added glusterVolumeTopRead verb
Added glusterVolumeTopWrite verb
Added glusterVolumeTopOpenDir verb
Added glusterVolumeTopReadDir verb
Added glusterVolumeTopReadPerf
verb Added glusterVolumeTopWritePerf verb

Following is the output structure of glusterVolumeTopOpen
{'statusCode' : CODE,
    'brickCount': BRICK-COUNT,
    'bricks': {BRICK-NAME: {'count':FILE-COUNT,
                'currentOpenFds': CURRENT-OPEN-FDS-COUNT,
                'maxOpen': MAX-OPEN,
                'maxOpenTime': MAX-OPEN-TIME,
                'files': [{FILE-NAME: FILE-OPEN-COUNT}, ...]
               }, ...} }

Following is the output structure of glusterVolumeTopRead
{'statusCode': CODE,
        'brickCount': BRICK-COUNT,
        'topOp': TOP-OP,
        'bricks': {BRICK-NAME: {
                    'count': FILE-COUNT,
                    'files': [{FILE-NAME: FILE-READ-COUNT}, ...]}
                  ,...}}

Following is the output structure glusterVolumeTopWrite
{'statusCode' : CODE,
        'brickCount': BRICK-COUNT,
        'topOp': TOP-OP,
        'bricks': {BRICK-NAME: {'count': FILE-COUNT,
                     'files': [{FILE-NAME: FILE-WRITE-COUNT}...]}
                   ,...}}

Following is the output structure glusterVolumeTopOpenDir
{'statusCode': CODE,
        'brickCount': BRICK-COUNT,
        'topOp': TOP-OP,
        'bricks': {BRICK-NAME: {'count':OPEN-DIR-COUNT,
                      'files': [{DIR-NAME: DIR-OPEN-COUNT}, ...]}
                  ,...}

Following is the output structure glusterVolumeTopReadDir
{'statusCode': CODE,
        'brickCount': BRICK-COUNT,
        'topOp': TOP-OP,
        'bricks': {BRICK-NAME: {'count':READ-DIR-COUNT,
                      'files': [{DIR-NAME: DIR-READ-COUNT}, ...]}
                  ,...}

Following is the output structure glusterVolumeTopReadPerf
{'statusCode': CODE,
        'brickCount': BRICK-COUNT,
        'topOp': TOP-OP,
        'bricks': {BRICK-NAME: {'fileCount':READ-COUNT,
                     'throughput': BRICK-WISE-READ-THROUGHPUT,
                     ' timeTaken': TIME-TAKEN,
                     'files': [{FILE-NAME:
                       {'throughput':FILE-READ-THROUGHPUT,
                              'time': TIME}}, ...]}
                  ,...}}

Following is the output structure glusterVolumeTopWritePerf
{'statusCode': CODE,
        'brickCount': BRICK-COUNT,
        'topOp': TOP-OP,
        'bricks': {BRICK-NAME: {'fileCount':WRITE-COUNT,
                     'throughput': BRICK-WISE-WRITE-THROUGHPUT,
                     ' timeTaken': TIME-TAKEN,
                     'files': [{FILE-NAME:
                       {'throughput':FILE-WRITE-THROUGHPUT,
                             'time': TIME}}, ...]}
                  ,...}}

Change-Id: I96486363a9acb7472014a67fcd2d5185d4f3c428
Signed-off-by: Timothy Asir <tjeyasin(a)redhat.com&gt;
---
M vdsm/gluster/api.py
M vdsm/gluster/cli.py
M vdsm/gluster/exception.py
M vdsm_cli/vdsClientGluster.py
4 files changed, 372 insertions(+), 0 deletions(-)


  git pull ssh://gerrit.ovirt.org:29418/vdsm refs/changes/44/7844/1

diff --git a/vdsm/gluster/api.py b/vdsm/gluster/api.py
index e52430b..3f493e0 100644
--- a/vdsm/gluster/api.py
+++ b/vdsm/gluster/api.py
@@ -241,6 +241,61 @@
         status = self.svdsmProxy.glusterVolumeProfileInfo(volumeName)
         return {'profileInfo': status}
 
+    @exportAsVerb
+    def volumeTopOpen(self, volumeName, brickName=None, count=None,
+                      options=None):
+        status = self.svdsmProxy.glusterVolumeTopOpen(volumeName,
+                                                          brickName, count)
+        return {'topOpen': status}
+
+    @exportAsVerb
+    def volumeTopRead(self, volumeName, brickName=None, count=None,
+                      options=None):
+        status = self.svdsmProxy.glusterVolumeTopRead(volumeName,
+                                                          brickName, count)
+        return {'topRead': status}
+
+    @exportAsVerb
+    def volumeTopWrite(self, volumeName, brickName=None, count=None,
+                      options=None):
+        status = self.svdsmProxy.glusterVolumeTopWrite(volumeName,
+                                                          brickName, count)
+        return {'topWrite': status}
+
+    @exportAsVerb
+    def volumeTopOpenDir(self, volumeName, brickName=None, count=None,
+                      options=None):
+        status = self.svdsmProxy.glusterVolumeTopOpenDir(volumeName,
+                                                          brickName, count)
+        return {'topOpenDir': status}
+
+    @exportAsVerb
+    def volumeTopWriteDir(self, volumeName, brickName=None, count=None,
+                      options=None):
+        status = self.svdsmProxy.glusterVolumeTopWriteDir(volumeName,
+                                                          brickName, count)
+        return {'topWriteDir': status}
+
+    @exportAsVerb
+    def volumeTopReadPerf(self, volumeName, blockSize=None, count=None,
+                          brickName=None, listCount=None, options=None):
+        status = self.svdsmProxy.glusterVolumeTopReadPerf(volumeName,
+                                                          blockSize,
+                                                          count,
+                                                          brickName,
+                                                          listCount)
+        return {'topReadPerf': status}
+
+    @exportAsVerb
+    def volumeTopWritePerf(self, volumeName, blockSize=None, count=None,
+                          brickName=None, listCount=None, options=None):
+        status = self.svdsmProxy.glusterVolumeTopWritePerf(volumeName,
+                                                          blockSize,
+                                                          count,
+                                                          brickName,
+                                                          listCount)
+        return {'topWritePerf': status}
+
 
 def getGlusterMethods(gluster):
     l = []
diff --git a/vdsm/gluster/cli.py b/vdsm/gluster/cli.py
index b91a04f..ba4768c 100644
--- a/vdsm/gluster/cli.py
+++ b/vdsm/gluster/cli.py
@@ -334,6 +334,66 @@
     return volumeInfoDict
 
 
+def _parseGlusterVolumeTopOpen(tree):
+    bricks = {}
+    for brick in tree.findall('volTop/brick'):
+        fileList = []
+        for file in brick.findall('file'):
+            fileList.append({file.find('filename').text:
+                             file.find('count').text})
+        bricks[brick.find('name').text] = {
+            'count': brick.find('members').text,
+            'currentOpen': brick.find('currentOpen').text,
+            'maxOpen': brick.find('maxOpen').text,
+            'maxOpenTime': brick.find('maxOpenTime').text,
+            'files': fileList}
+    status = {
+        'topOp': tree.find('volTop/topOp').text,
+        'brickCount': tree.find('volTop/brickCount').text,
+        'statusCode': tree.find('opRet').text,
+        'bricks': bricks}
+    return status
+
+
+def _parseGlusterVolumeTop(tree):
+    bricks = {}
+    for brick in tree.findall('volTop/brick'):
+        fileList = []
+        for fileTag in brick.findall('file'):
+            fileList.append({fileTag.find('filename').text:
+                             fileTag.find('count').text})
+        bricks[brick.find('name').text] = {
+            'count': brick.find('members').text,
+            'files': fileList}
+    status = {
+        'topOp': tree.find('volTop/topOp').text,
+        'brickCount': tree.find('volTop/brickCount').text,
+        'statusCode': tree.find('opRet').text,
+        'bricks': bricks}
+    return status
+
+
+def _parseGlusterVolumeTopPerf(tree):
+    bricks = {}
+    for brick in tree.findall('volTop/brick'):
+        fileList = []
+        for fileTag in brick.findall('file'):
+            fileList.append({fileTag.find('filename').text:
+                             {'count': fileTag.find('count').text,
+                             'time': fileTag.find('time').text}})
+        bricks[brick.find('name').text] = {
+            'count': brick.find('members').text,
+            'throughput': brick.find('throughput').text,
+            'timeTaken': brick.find('timeTaken').text,
+            'files': fileList}
+    status = {
+        'topOp': tree.find('volTop/topOp').text,
+        'brickCount': tree.find('volTop/brickCount').text,
+        'statusCode': tree.find("opRet").text,
+        'bricks': bricks}
+    return status
+
+
 def _parseGlusterVolumeProfileInfo(tree):
     bricks = {}
     for brick in tree.findall('volProfile/brick'):
@@ -819,3 +879,132 @@
         return _parseGlusterVolumeProfileInfo(xmltree)
     except:
         raise ge.GlusterXmlErrorException(err=out)
+
+
+@exportToSuperVdsm
+def volumeTopOpen(volumeName, brickName=None, count=None):
+    command = _getGlusterVolCmd() + ["top", volumeName, "open"]
+    if brickName:
+        command += ["brick", "%s" % brickName]
+    if count:
+        command += ["list-cnt", "%s" % count]
+    try:
+        xmltree, out = _execGlusterXml(command)
+    except ge.GlusterCmdFailedException, e:
+        raise ge.GlusterVolumeTopOpenFailedException(rc=e.rc, err=e.err)
+    try:
+        return _parseGlusterVolumeTopOpen(xmltree)
+    except:
+        raise ge.GlusterXmlErrorException(err=out)
+
+
+@exportToSuperVdsm
+def volumeTopRead(volumeName, brickName=None, count=None):
+    command = _getGlusterVolCmd() + ["top", volumeName, "read"]
+    if brickName:
+        command += ["brick", "%s" % brickName]
+    if count:
+        command += ["list-cnt", "%s" % count]
+    try:
+        xmltree, out = _execGlusterXml(command)
+    except ge.GlusterCmdFailedException, e:
+        raise ge.GlusterVolumeTopReadFailedException(rc=e.rc, err=e.err)
+    try:
+        return _parseGlusterVolumeTop(xmltree)
+    except:
+        raise ge.GlusterXmlErrorException(err=out)
+
+
+@exportToSuperVdsm
+def volumeTopWrite(volumeName, brickName=None, count=None):
+    command = _getGlusterVolCmd() + ["top", volumeName, "write"]
+    if brickName:
+        command += ["brick", "%s" % brickName]
+    if count:
+        command += ["list-cnt", "%s" % count]
+    try:
+        xmltree, out = _execGlusterXml(command)
+    except ge.GlusterCmdFailedException, e:
+        raise ge.GlusterVolumeTopWriteFailedException(rc=e.rc, err=e.err)
+    try:
+        return _parseGlusterVolumeTop(xmltree)
+    except:
+        raise ge.GlusterXmlErrorException(err=out)
+
+
+@exportToSuperVdsm
+def volumeTopOpenDir(volumeName, brickName=None, count=None):
+    command = _getGlusterVolCmd() + ["top", volumeName, "opendir"]
+    if brickName:
+        command += ["brick", "%s" % brickName]
+    if count:
+        command += ["list-cnt", "%s" % count]
+    try:
+        xmltree, out = _execGlusterXml(command)
+    except ge.GlusterCmdFailedException, e:
+        raise ge.GlusterVolumeTopOpenDirFailedException(rc=e.rc, err=e.err)
+    try:
+        return _parseGlusterVolumeTop(xmltree)
+    except:
+        raise ge.GlusterXmlErrorException(err=out)
+
+
+@exportToSuperVdsm
+def volumeTopReadDir(volumeName, bricName=None, count=None):
+    command = _getGlusterVolCmd() + ["top", volumeName, "write"]
+    if brickName:
+        command += ["brick", "%s" % brickName]
+    if count:
+        command += ["list-cnt", "%s" % count]
+    try:
+        xmltree, out = _execGlusterXml(command)
+    except ge.GlusterCmdFailedException, e:
+        raise ge.GlusterVolumeTopReadDirFailedException(rc=e.rc, err=e.err)
+    try:
+        return _parseGlusterVolumeTop(xmltree)
+    except:
+        raise ge.GlusterXmlErrorException(err=out)
+
+
+@exportToSuperVdsm
+def volumeTopReadPerf(volumeName, blockSize=None, count=None,
+                      brickName=None, listCount=None):
+    command = _getGlusterVolCmd() + ["top", volumeName, "read-perf"]
+    if blockSize:
+        command += ["bs", "%s" % blockSize]
+    if count:
+        command += ["count", "%s" % count]
+    if brickName:
+        command += ["brick", "%s" % brickName]
+    if listCount:
+        command += ["list-cnt", "%s" % listCount]
+    try:
+        xmltree, out = _execGlusterXml(command)
+    except ge.GlusterCmdFailedException, e:
+        raise ge.GlusterVolumeTopReadPerfFailedException(rc=e.rc, err=e.err)
+    try:
+        return _parseGlusterVolumeTopPerf(xmltree)
+    except:
+        raise ge.GlusterXmlErrorException(err=out)
+
+
+@exportToSuperVdsm
+def volumeTopWritePerf(volumeName, blockSize=None, count=None,
+                      brickName=None, listCount=None):
+    command = _getGlusterVolCmd() + ["top", volumeName,
"write-perf"]
+    if blockSize:
+        command += ["bs", "%s" % blockSize]
+    if count:
+        command += ["count", "%s" % count]
+    if brickName:
+        command += ["brick", "%s" % brickName]
+    if listCount:
+        command += ["list-cnt", "%s" % listCount]
+    try:
+        xmltree, out = _execGlusterXml(command)
+    except ge.GlusterCmdFailedException, e:
+        raise ge.GlusterVolumeTopWritePerfFailedException(rc=e.rc, err=e.err)
+    try:
+        return _parseGlusterVolumeTopPerf(xmltree)
+    except:
+        raise ge.GlusterXmlErrorException(err=out)
diff --git a/vdsm/gluster/exception.py b/vdsm/gluster/exception.py
index bc20dd0..b392ec8 100644
--- a/vdsm/gluster/exception.py
+++ b/vdsm/gluster/exception.py
@@ -343,6 +343,41 @@
     message = "Volume profile info failed"
 
 
+class GlusterVolumeTopOpenFailedException(GlusterVolumeException):
+    code = 4161
+    message = "Volume top open failed"
+
+
+class GlusterVolumeTopReadFailedException(GlusterVolumeException):
+    code = 4162
+    message = "Volume top read failed"
+
+
+class GlusterVolumeTopWriteFailedException(GlusterVolumeException):
+    code = 4163
+    message = "Volume top write failed"
+
+
+class GlusterVolumeTopOpenDirFailedException(GlusterVolumeException):
+    code = 4164
+    message = "Volume top open dir failed"
+
+
+class GlusterVolumeTopReadDirFailedException(GlusterVolumeException):
+    code = 4165
+    message = "Volume top read dir failed"
+
+
+class GlusterVolumeTopReadPerfFailedException(GlusterVolumeException):
+    code = 4166
+    message = "Volume top read perf failed"
+
+
+class GlusterVolumeTopWritePerfFailedException(GlusterVolumeException):
+    code = 4167
+    message = "Volume top write perf failed"
+
+
 # Host
 class GlusterHostException(GlusterException):
     code = 4400
diff --git a/vdsm_cli/vdsClientGluster.py b/vdsm_cli/vdsClientGluster.py
index 8422695..3663c63 100644
--- a/vdsm_cli/vdsClientGluster.py
+++ b/vdsm_cli/vdsClientGluster.py
@@ -221,6 +221,41 @@
         pp.pprint(status)
         return status['status']['code'],
status['status']['message']
 
+    def do_glusterVolumeTopOpen(self, args):
+        status = self.s.glusterVolumeTopOpen(args[0])
+        pp.pprint(status)
+        return status['status']['code'],
status['status']['message']
+
+    def do_glusterVolumeTopRead(self, args):
+        status = self.s.glusterVolumeTopRead(args[0])
+        pp.pprint(status)
+        return status['status']['code'],
status['status']['message']
+
+    def do_glusterVolumeTopWrite(self, args):
+        status = self.s.glusterVolumeTopWrite(args[0])
+        pp.pprint(status)
+        return status['status']['code'],
status['status']['message']
+
+    def do_glusterVolumeTopOpenDir(self, args):
+        status = self.s.glusterVolumeTopOpenDir(args[0])
+        pp.pprint(status)
+        return status['status']['code'],
status['status']['message']
+
+    def do_glusterVolumeTopReadDir(self, args):
+        status = self.s.glusterVolumeTopReadDir(args[0])
+        pp.pprint(status)
+        return status['status']['code'],
status['status']['message']
+
+    def do_glusterVolumeTopReadPerf(self, args):
+        status = self.s.glusterVolumeTop(args[0])
+        pp.pprint(status)
+        return status['status']['code'],
status['status']['message']
+
+    def do_glusterVolumeTopWritePerf(self, args):
+        status = self.s.glusterVolumeTop(args[0])
+        pp.pprint(status)
+        return status['status']['code'],
status['status']['message']
+
 
 def getGlusterCmdDict(serv):
     return {
@@ -403,4 +438,62 @@
              ('<volume_name>\n\t<volume_name> is existing volume
name',
               'get gluster volume profile info'
               )),
+        'glusterVolumeTopOpen':
+            (serv.do_glusterVolumeTopOpen,
+             ('<volume_name> [brick=<existing_brick>] '
+              '[count=<list_count>]\n\t'
+              '<volume_name> is existing volume name\n\t'
+              'get volume top open fd count and maximum fd count of '
+              'a given volume with its all brick or specified brick'
+              )),
+        'glusterVolumeTopRead':
+            (serv.do_glusterVolumeTopRead,
+             ('<volume_name> [brick=<existing_brick>] '
+              '[count=<list_count>]\n\t'
+              '<volume_name> is existing volume name\n\t'
+              'get list of highest read calls on each brick or '
+              'a specified brick of a volume'
+              )),
+        'glusterVolumeTopWrite':
+            (serv.do_glusterVolumeTopWrite,
+             ('<volume_name> [brick=<existing_brick>] '
+              '[count=<list_count>]\n\t'
+              '<volume_name> is existing volume name\n\t'
+              'get list of highest write calls on each brick or '
+              'a specified brick of a volume'
+              )),
+        'glusterVolumeTopOpenDir':
+            (serv.do_glusterVolumeTopOpenDir,
+             ('<volume_name> [brick=<existing_brick>] '
+              '[count=<list_count>]\n\t'
+              '<volume_name> is existing volume name\n\t'
+              'get list of highest open calls on directories of each brick '
+              'or a specified brick of a volume'
+              )),
+        'glusterVolumeTopReadDir':
+            (serv.do_glusterVolumeTopReadDir,
+             ('<volume_name> [brick=<existing_brick>] '
+              '[count=<list_count>]\n\t'
+              '<volume_name> is existing volume name\n\t'
+              'get list of highest read calls on directories of each brick '
+              'or a specified brick of a volume'
+              )),
+        'glusterVolumeTopReadPerf':
+            (serv.do_glusterVolumeTopReadPerf,
+             ('<volume_name> [block_size=<block_size>] '
+              '[count=<count>] [brick=<existing_brick>] '
+              '[list_count=<list_count>]\n\t'
+              '<volume_name> is existing volume name\n\t'
+              'get list of read throughput of files on bricks. '
+              'if the block size and the count is not specified, '
+              'it will give the output based on historical data'
+              )),
+        'glusterVolumeTopWritePerf':
+            (serv.do_glusterVolumeTopWritePerf,
+             ('<volume_name> [block_size=<block_size>] '
+              '[count=<count>] [brick=<existing_brick>] '
+              '[list_count=<list_count>]\n\t'
+              '<volume_name> is existing volume name\n\t'
+              'get list of write throughput of files on bricks'
+              )),
         }


--
To view, visit http://gerrit.ovirt.org/7844
To unsubscribe, visit http://gerrit.ovirt.org/settings

Gerrit-MessageType: newchange
Gerrit-Change-Id: I96486363a9acb7472014a67fcd2d5185d4f3c428
Gerrit-PatchSet: 1
Gerrit-Project: vdsm
Gerrit-Branch: master
Gerrit-Owner: Timothy Asir <tjeyasin(a)redhat.com&gt;
Gerrit-Reviewer: Ayal Baron <abaron(a)redhat.com&gt;
Gerrit-Reviewer: Bala.FA <barumuga(a)redhat.com&gt;
Gerrit-Reviewer: Dan Kenigsberg <danken(a)redhat.com&gt;
Gerrit-Reviewer: Federico Simoncelli <fsimonce(a)redhat.com&gt;
Gerrit-Reviewer: Saggi Mizrahi <smizrahi(a)redhat.com&gt;