Dima Kuznetsov has uploaded a new change for review.
Change subject: vdsm-tool: Change upgrade mechanism ......................................................................
vdsm-tool: Change upgrade mechanism
The upgrade decorator is a mechanism within vdsm-tool to allow commands to only run once. It has flags to override, but being a function decorator, it did not have any direct access to command line arguments and did access via sys.argv[2:].
There is a patch that will be sent that will add flags to vdsm-tool and will make break upgrade and its argument parsing. I'm proposing to change the upgrade from being a decorated function, to a class deriving from Upgrade with apply() method. This way args can be passed to the upgrade code (via ctor), and deriving class and extend the argument parser, this way, we'll also get a nice usage for upgrade commands: $ vdsm-tool some-upgrade --help Usage: vdsm-tool [...]
Options: [general upgrade options, e.g. --run-again] [specific upgrade options]
Change-Id: I6e1d28570dedfeff9fe60624b1db72d8cadf136a Signed-off-by: Dima Kuznetsov dkuznets@redhat.com --- M lib/vdsm/tool/unified_persistence.py M lib/vdsm/tool/upgrade.py M lib/vdsm/tool/upgrade_300_networks.py 3 files changed, 61 insertions(+), 114 deletions(-)
git pull ssh://gerrit.ovirt.org:29418/vdsm refs/changes/93/27193/1
diff --git a/lib/vdsm/tool/unified_persistence.py b/lib/vdsm/tool/unified_persistence.py index 3d4c8ab..10cdc41 100644 --- a/lib/vdsm/tool/unified_persistence.py +++ b/lib/vdsm/tool/unified_persistence.py @@ -23,7 +23,7 @@ from ..netconfpersistence import RunningConfig from ..netinfo import NetInfo, getIfaceCfg, getDefaultGateway from . import expose -from .upgrade import upgrade +from .upgrade import Upgrade
UPGRADE_NAME = 'upgrade-unified-persistence' @@ -36,7 +36,6 @@ # bootproto = 'dhcp' if there's a lease on the NIC at the moment of upgrade
-@upgrade(UPGRADE_NAME) def run(): networks, bondings = _getNetInfo() logging.debug('%s upgrade persisting networks %s and bondings %s', @@ -156,6 +155,13 @@ return config.get('vars', 'net_persistence') == 'unified'
+class UpgradeUnifiedPersistence(Upgrade): + def run(self): + if isNeeded(): + run() + return 0 + + @expose(UPGRADE_NAME) def unified_persistence(*args): """ @@ -164,6 +170,4 @@ persistence model is set as unified in /usr/lib64/python2.X/site-packages/ vdsm/config.py """ - if isNeeded(): - return run() - return 0 + return UpgradeUnifiedPersistence(*args).apply() diff --git a/lib/vdsm/tool/upgrade.py b/lib/vdsm/tool/upgrade.py index 2e19701..822a24d 100644 --- a/lib/vdsm/tool/upgrade.py +++ b/lib/vdsm/tool/upgrade.py @@ -18,58 +18,31 @@ #
import argparse -from functools import wraps import logging import logging.config -from logging.handlers import SysLogHandler import os -import sys
-from ..constants import (P_VDSM_LOG, P_VDSM_LIB, P_VDSM_CONF, VDSM_USER, - VDSM_GROUP) +from ..constants import P_VDSM_LIB from ..utils import touchFile - -sys.path.append("/usr/share/vdsm") -from storage.fileUtils import chown
class Upgrade(object): """ - Code that needs to be run exactly once can use this class as - a general, simplistic mechanism. Usage: + This is a parent class to define upgrades, it defines its own argument + parser and allows any subclass extend it, the subclass is to define run() + method that will be called if needed upon upgrade.apply().
- with Upgrade('someUpgrade') as upgrade: - if upgrade.isNeeded(): - runSomeCode() - upgrade.seal() - - Upgrade as a context manager automatically redirects logging - to the upgrade log. It also supplies the isNeeded and seal methods. - To call isNeeded and seal automatically use this module's upgrade - decorator. + upgrade.apply() parses the arguments, checks if upgrade should run, + and upon successful completion calls upgrade.seal() that marks upgrade as + done, to avoid executing it again. """ - def __init__(self, upgradeName): + def __init__(self, upgradeName, *args): self._upgradeName = upgradeName self._upgradeFilePath = os.path.join(P_VDSM_LIB, 'upgrade', self._upgradeName) - try: - # First load normal VDSM loggers, then the upgrade logger. - # This will override VDSM's root logger but will keep the other - # loggers intact. During an upgrade we add the update handler - # to all loggers. - logging.config.fileConfig(P_VDSM_CONF + 'logger.conf', - disable_existing_loggers=False) - logging.config.fileConfig(P_VDSM_CONF + 'upgrade.logger.conf', - disable_existing_loggers=False) - chown( - os.path.join(P_VDSM_LOG, 'upgrade.log'), - VDSM_USER, - VDSM_GROUP) - except Exception: - logging.getLogger('upgrade').addHandler(SysLogHandler('/dev/log')) - logging.exception("Could not init proper logging") - finally: - self.log = logging.getLogger('upgrade') + self.log = logging.getLogger('upgrade') + self._arg_parser = self._upgrade_argparser() + self._args = args
def isNeeded(self): return not os.path.exists(self._upgradeFilePath) @@ -86,72 +59,32 @@ self.log.debug("Upgrade %s successfully performed", self._upgradeName)
- def _attachUpgradeLog(self): - self._editOtherLoggers(logging.Logger.addHandler) + def _upgrade_argparser(self): + parser = argparse.ArgumentParser('vdsm-tool %s' % self._upgradeName) + parser.add_argument( + '--run-again', + dest='runAgain', + default=False, + action='store_true', + help='Run the upgrade again, even if it was ran before', + ) + return parser
- def _detachUpgradeLog(self): - self._editOtherLoggers(logging.Logger.removeHandler) + def run(self): + pass
- def _editOtherLoggers(self, edit): - """ - add/remove upgrade handler to/from all non-upgrade loggers - """ - loggers = dict(logging.Logger.manager.loggerDict.items() + - [('root', logging.getLogger())]) - for name, logger in loggers.iteritems(): - if name != 'upgrade': - for handler in logging.getLogger('upgrade').handlers: - try: - edit(logger, handler) # Call logger.edit(handler) - except TypeError: - pass - - def __enter__(self): - """ - All logging will *also* be output to the upgrade log during - the scope of this Upgrade. - """ - self._attachUpgradeLog() - return self - - def __exit__(self, type, value, traceback): - self._detachUpgradeLog() - - -def _parse_args(upgradeName): - parser = argparse.ArgumentParser('vdsm-tool %s' % upgradeName) - parser.add_argument( - '--run-again', - dest='runAgain', - default=False, - action='store_true', - help='Run the upgrade again, even if it was ran before', - ) - return parser.parse_args(sys.argv[2:]) - - -def upgrade(upgradeName): - """ - Used as a decorator for upgrades. Runs the upgrade with an Upgrade - context manager (documented above). Automatically calls isNeeded and seal - as needed. - """ - def decorator(f): - @wraps(f) - def wrapper(*args, **kwargs): - with Upgrade(upgradeName) as upgrade: - cliArguments = _parse_args(upgradeName) - if cliArguments.runAgain or upgrade.isNeeded(): - upgrade.log.debug("Running upgrade %s", upgradeName) - try: - f(*args, **kwargs) - except Exception: - upgrade.log.exception("Failed to run %s", - upgradeName) - return 1 - else: - upgrade.seal() - - return 0 - return wrapper - return decorator + def apply(self): + res = self._arg_parser.parse_known_args(self._args) + self._parsed_ns = res[0] + self._parsed_args = res[1] + if self.isNeeded() or self._parsed_ns.runAgain: + self.log.debug("Running upgrade %s", self._upgradeName) + try: + self.run() + except Exception: + self.log.exception("Failed to run %s", + self._upgradeName) + return 1 + else: + self.seal() + return 0 diff --git a/lib/vdsm/tool/upgrade_300_networks.py b/lib/vdsm/tool/upgrade_300_networks.py index 95d4e17..250b04d 100644 --- a/lib/vdsm/tool/upgrade_300_networks.py +++ b/lib/vdsm/tool/upgrade_300_networks.py @@ -24,7 +24,7 @@ from vdsm import netinfo from vdsm.constants import LEGACY_MANAGEMENT_NETWORKS from vdsm.tool import expose -from vdsm.tool.upgrade import upgrade +from vdsm.tool.upgrade import Upgrade
sys.path.append("/usr/share/vdsm") from network.configurators import ifcfg @@ -42,7 +42,6 @@ return not managementNetwork() and managementBridge()
-@upgrade(UPGRADE_NAME) def run(networks, bridges): configWriter = ifcfg.ConfigWriter()
@@ -61,6 +60,18 @@ configWriter.removeLibvirtNetwork(network, skipBackup=True)
+class Upgrade300Networks(Upgrade): + def __init__(self, networks, bridges, *args): + Upgrade.__init__(self, *args) + self._networks = networks + self._bridges = bridges + + def run(self): + if isNeeded(self._networks, self._bridges): + run(self._networks, self._bridges) + return 0 + + @expose(UPGRADE_NAME) def upgrade_networks(*args): """ @@ -73,5 +84,4 @@ networks = netinfo.networks() bridges = netinfo.bridges()
- if isNeeded(networks, bridges): - run(networks, bridges) + return Upgrade300Networks(networks, bridges, *args).apply()
oVirt Jenkins CI Server has posted comments on this change.
Change subject: vdsm-tool: Change upgrade mechanism ......................................................................
Patch Set 1:
Build Successful
http://jenkins.ovirt.org/job/vdsm_master_pep8_gerrit/8378/ : SUCCESS
http://jenkins.ovirt.org/job/vdsm_master_unit_tests_gerrit_el/7588/ : SUCCESS
http://jenkins.ovirt.org/job/vdsm_master_unit_tests_gerrit/8498/ : SUCCESS
Assaf Muller has posted comments on this change.
Change subject: vdsm-tool: Change upgrade mechanism ......................................................................
Patch Set 1:
If you git blame and look at the patches that introduced the upgrade mechanism you'd see that Saggi NACK'd inheritance, on the basis that inheritance should never be used in Python (This is an oversimplification but was the basis for his argument). I disagreed with him but implemented it without inheritance as he asked. Let's see if you can convince him now :)
Dima Kuznetsov has posted comments on this change.
Change subject: vdsm-tool: Change upgrade mechanism ......................................................................
Patch Set 2:
After a short discussion with Saggi, I've decided to get rid of the inheritance, instead, rename Upgrade to UpgradeManager, and define an interface for upgrade classes to implement.
oVirt Jenkins CI Server has posted comments on this change.
Change subject: vdsm-tool: Change upgrade mechanism ......................................................................
Patch Set 2:
Build Failed
http://jenkins.ovirt.org/job/vdsm_master_pep8_gerrit/8385/ : SUCCESS
http://jenkins.ovirt.org/job/vdsm_master_unit_tests_gerrit_el/7595/ : SUCCESS
http://jenkins.ovirt.org/job/vdsm_master_unit_tests_gerrit/8505/ : FAILURE
Dima Kuznetsov has posted comments on this change.
Change subject: vdsm-tool: Change upgrade mechanism ......................................................................
Patch Set 3:
Got rid of class, went for apply_upgrade() function as single point of interface. Added some tests for upgrade mechanism.
oVirt Jenkins CI Server has posted comments on this change.
Change subject: vdsm-tool: Change upgrade mechanism ......................................................................
Patch Set 3:
Build Successful
http://jenkins.ovirt.org/job/vdsm_master_pep8_gerrit/8414/ : SUCCESS
http://jenkins.ovirt.org/job/vdsm_master_unit_tests_gerrit_el/7624/ : SUCCESS
http://jenkins.ovirt.org/job/vdsm_master_unit_tests_gerrit/8534/ : SUCCESS
Yaniv Bronhaim has posted comments on this change.
Change subject: vdsm-tool: Change upgrade mechanism ......................................................................
Patch Set 3:
(4 comments)
http://gerrit.ovirt.org/#/c/27193/3//COMMIT_MSG Commit Message:
Line 10: to only run once. It has flags to override, but being a function Line 11: decorator, it did not have any direct access to command line arguments Line 12: and did access via sys.argv[2:]. Line 13: Line 14: There is a patch that will be sent that will add flags to vdsm-tool and now you can replace it with http://gerrit.ovirt.org/#/c/27481/ Line 15: will break upgrade and its argument parsing. I'm proposing to change the Line 16: upgrade from being a decorated function, to a function that operates on Line 17: an upgrade object (interface documented in apply_upgrade docstring) and Line 18: the command line arguments.
Line 18: the command line arguments. Line 19: Line 20: This also allows to have a single argument parser for both Line 21: general upgrade flags and specific per-upgrade flags (no current Line 22: upgrades define flags but we might want this in future). worth to mention the extendArgument func that needs to be declared Line 23: Line 24: Change-Id: I6e1d28570dedfeff9fe60624b1db72d8cadf136a
http://gerrit.ovirt.org/#/c/27193/3/tests/toolTests.py File tests/toolTests.py:
Line 179: Line 180: Line 181: class UpgradeTests(TestCase): Line 182: Line 183: class UpgraduratorTM(object): maybe upgrade.py name was the only problem... upgradurator sounds much tougher Line 184: name = 'test' Line 185: Line 186: def __init__(self, lst): Line 187: self.lst = lst
Line 195: def run(self, ns, args): Line 196: self.lst[0] += 1 Line 197: self.ns = ns Line 198: self.args = args Line 199: return 0 do you test where the output goes? ^ Line 200: Line 201: class BadUpgraduratorTM(object): Line 202: name = 'bad' Line 203:
Dima Kuznetsov has posted comments on this change.
Change subject: vdsm-tool: Change upgrade mechanism ......................................................................
Patch Set 3:
(3 comments)
http://gerrit.ovirt.org/#/c/27193/3//COMMIT_MSG Commit Message:
Line 10: to only run once. It has flags to override, but being a function Line 11: decorator, it did not have any direct access to command line arguments Line 12: and did access via sys.argv[2:]. Line 13: Line 14: There is a patch that will be sent that will add flags to vdsm-tool and
now you can replace it with http://gerrit.ovirt.org/#/c/27481/
Done Line 15: will break upgrade and its argument parsing. I'm proposing to change the Line 16: upgrade from being a decorated function, to a function that operates on Line 17: an upgrade object (interface documented in apply_upgrade docstring) and Line 18: the command line arguments.
Line 18: the command line arguments. Line 19: Line 20: This also allows to have a single argument parser for both Line 21: general upgrade flags and specific per-upgrade flags (no current Line 22: upgrades define flags but we might want this in future).
worth to mention the extendArgument func that needs to be declared
Added a mention.
It doesn't need to be declared, but can, as the message states, the full documentation of the interface is available at the docstring. Line 23: Line 24: Change-Id: I6e1d28570dedfeff9fe60624b1db72d8cadf136a
http://gerrit.ovirt.org/#/c/27193/3/tests/toolTests.py File tests/toolTests.py:
Line 195: def run(self, ns, args): Line 196: self.lst[0] += 1 Line 197: self.ns = ns Line 198: self.args = args Line 199: return 0
do you test where the output goes? ^
please explain Line 200: Line 201: class BadUpgraduratorTM(object): Line 202: name = 'bad' Line 203:
oVirt Jenkins CI Server has posted comments on this change.
Change subject: vdsm-tool: Change upgrade mechanism ......................................................................
Patch Set 4:
Build Successful
http://jenkins.ovirt.org/job/vdsm_master_pep8_gerrit/8811/ : SUCCESS
http://jenkins.ovirt.org/job/vdsm_master_unit_tests_gerrit/8947/ : SUCCESS
http://jenkins.ovirt.org/job/vdsm_master_unit_tests_gerrit_el/8021/ : SUCCESS
Antoni Segura Puimedon has posted comments on this change.
Change subject: vdsm-tool: Change upgrade mechanism ......................................................................
Patch Set 4:
(3 comments)
Looks good to me, only copyright notes.
http://gerrit.ovirt.org/#/c/27193/4/lib/vdsm/tool/unified_persistence.py File lib/vdsm/tool/unified_persistence.py:
Line 1: # Copyright 2013 Red Hat, Inc. 2013-2014 Line 2: # Line 3: # This program is free software; you can redistribute it and/or modify Line 4: # it under the terms of the GNU General Public License as published by Line 5: # the Free Software Foundation; either version 2 of the License, or
http://gerrit.ovirt.org/#/c/27193/4/lib/vdsm/tool/upgrade.py File lib/vdsm/tool/upgrade.py:
Line 1: # Copyright 2013 Red Hat, Inc. 2013-2014 Line 2: # Line 3: # This program is free software; you can redistribute it and/or modify Line 4: # it under the terms of the GNU General Public License as published by Line 5: # the Free Software Foundation; either version 2 of the License, or
http://gerrit.ovirt.org/#/c/27193/4/lib/vdsm/tool/upgrade_300_networks.py File lib/vdsm/tool/upgrade_300_networks.py:
Line 1: # Line 2: # Copyright 2011-2013 Red Hat, Inc. 2011-2014 Line 3: # Line 4: # This program is free software; you can redistribute it and/or modify Line 5: # it under the terms of the GNU General Public License as published by Line 6: # the Free Software Foundation; either version 2 of the License, or
Dima Kuznetsov has posted comments on this change.
Change subject: vdsm-tool: Change upgrade mechanism ......................................................................
Patch Set 4:
(3 comments)
http://gerrit.ovirt.org/#/c/27193/4/lib/vdsm/tool/unified_persistence.py File lib/vdsm/tool/unified_persistence.py:
Line 1: # Copyright 2013 Red Hat, Inc.
2013-2014
Done Line 2: # Line 3: # This program is free software; you can redistribute it and/or modify Line 4: # it under the terms of the GNU General Public License as published by Line 5: # the Free Software Foundation; either version 2 of the License, or
http://gerrit.ovirt.org/#/c/27193/4/lib/vdsm/tool/upgrade.py File lib/vdsm/tool/upgrade.py:
Line 1: # Copyright 2013 Red Hat, Inc.
2013-2014
Done Line 2: # Line 3: # This program is free software; you can redistribute it and/or modify Line 4: # it under the terms of the GNU General Public License as published by Line 5: # the Free Software Foundation; either version 2 of the License, or
http://gerrit.ovirt.org/#/c/27193/4/lib/vdsm/tool/upgrade_300_networks.py File lib/vdsm/tool/upgrade_300_networks.py:
Line 1: # Line 2: # Copyright 2011-2013 Red Hat, Inc.
2011-2014
Done Line 3: # Line 4: # This program is free software; you can redistribute it and/or modify Line 5: # it under the terms of the GNU General Public License as published by Line 6: # the Free Software Foundation; either version 2 of the License, or
oVirt Jenkins CI Server has posted comments on this change.
Change subject: vdsm-tool: Change upgrade mechanism ......................................................................
Patch Set 5:
Build Successful
http://jenkins.ovirt.org/job/vdsm_master_pep8_gerrit/8819/ : SUCCESS
http://jenkins.ovirt.org/job/vdsm_master_unit_tests_gerrit/8955/ : SUCCESS
http://jenkins.ovirt.org/job/vdsm_master_unit_tests_gerrit_el/8029/ : SUCCESS
Antoni Segura Puimedon has posted comments on this change.
Change subject: vdsm-tool: Change upgrade mechanism ......................................................................
Patch Set 5: Code-Review+1
oVirt Jenkins CI Server has posted comments on this change.
Change subject: vdsm-tool: Change upgrade mechanism ......................................................................
Patch Set 6:
Build Successful
http://jenkins.ovirt.org/job/vdsm_master_pep8_gerrit/9091/ : SUCCESS
http://jenkins.ovirt.org/job/vdsm_master_unit_tests_gerrit/9232/ : SUCCESS
http://jenkins.ovirt.org/job/vdsm_master_unit_tests_gerrit_el/8303/ : SUCCESS
Dima Kuznetsov has posted comments on this change.
Change subject: vdsm-tool: Change upgrade mechanism ......................................................................
Patch Set 7: Verified+1
oVirt Jenkins CI Server has posted comments on this change.
Change subject: vdsm-tool: Change upgrade mechanism ......................................................................
Patch Set 7:
Build Failed
http://jenkins.ovirt.org/job/vdsm_master_pep8_gerrit/9094/ : SUCCESS
http://jenkins.ovirt.org/job/vdsm_master_unit_tests_gerrit/9235/ : FAILURE
http://jenkins.ovirt.org/job/vdsm_master_unit_tests_gerrit_el/8306/ : SUCCESS
Dima Kuznetsov has posted comments on this change.
Change subject: vdsm-tool: Change upgrade mechanism ......................................................................
Patch Set 7: -Verified Code-Review-1
Dima Kuznetsov has posted comments on this change.
Change subject: vdsm-tool: Change upgrade mechanism ......................................................................
Patch Set 8: Verified+1
In addition to previous change, removed upgrade.log config files, as they are not needed anymore, and changed the calls to vdsm-tool that execute upgrades, to log to /var/log/vdsm/upgrade.log, as before.
This solves the issue of admin creating those logs with root:root by mistake since 'vdsm-tool upgrade-3.0.0-networks' won't write logs unless specifically told to, but would still work.
oVirt Jenkins CI Server has posted comments on this change.
Change subject: vdsm-tool: Change upgrade mechanism ......................................................................
Patch Set 8:
Build Failed
http://jenkins.ovirt.org/job/vdsm_master_pep8_gerrit/9119/ : SUCCESS
http://jenkins.ovirt.org/job/vdsm_master_unit_tests_gerrit/9262/ : FAILURE
http://jenkins.ovirt.org/job/vdsm_master_unit_tests_gerrit_el/8331/ : SUCCESS
http://jenkins.ovirt.org/job/vdsm_master_install_rpm_sanity_gerrit/623/ : FAILURE
Yaniv Bronhaim has posted comments on this change.
Change subject: vdsm-tool: Change upgrade mechanism ......................................................................
Patch Set 8: Code-Review+1
Yaniv Bronhaim has posted comments on this change.
Change subject: vdsm-tool: Change upgrade mechanism ......................................................................
Patch Set 8: -Code-Review
the patches order is wrong. http://gerrit.ovirt.org/#/c/27481/5 should be merged before this
Yaniv Bronhaim has posted comments on this change.
Change subject: vdsm-tool: Change upgrade mechanism ......................................................................
Patch Set 8: Code-Review+1
my mistake. the order correct
Assaf Muller has posted comments on this change.
Change subject: vdsm-tool: Change upgrade mechanism ......................................................................
Patch Set 8: Code-Review-1
How is logging output to other loggers/handlers (Such as the root logger, or vdsm logger) during an upgrade logged to upgrade.log?
Yaniv Bronhaim has posted comments on this change.
Change subject: vdsm-tool: Change upgrade mechanism ......................................................................
Patch Set 8:
Assaf, I'm not sure i understand your comment, but we call the upgrade verb with --verbose --append --logfile=@VDSMLOGDIR@/upgrade.log flags from the init scripts. if user runs it manually, the user should specify the log file name for output. we don't use root\vdsm logger at all, nothing load vdsm.logger.conf file as was
Assaf Muller has posted comments on this change.
Change subject: vdsm-tool: Change upgrade mechanism ......................................................................
Patch Set 8:
If you run the unified upgrade, do you see lines (in upgrade.log) like: "Adding network ovirtmgmt..."?
Assaf Muller has posted comments on this change.
Change subject: vdsm-tool: Change upgrade mechanism ......................................................................
Patch Set 8:
To make my previous question clearer: If you run the unified upgrade during VDSM start (IE: Not manually), do you see lines (in upgrade.log) like: "Adding network ovirtmgmt..."?
Dima Kuznetsov has posted comments on this change.
Change subject: vdsm-tool: Change upgrade mechanism ......................................................................
Patch Set 8:
Assaf, the ordering is truly a bit problematic, because logging facilities only added in a latter patch, http://gerrit.ovirt.org/#/c/27481 but opposite order is not perfect either because the subsequent patch breaks the present-day upgrade mechanism, that assumes it's parameters begin at sys.argv[1:].
As for the logs produced, the following is the actual log (after removing upgrade seals):
MainThread::INFO::2014-05-27 10:29:38,293::netconfpersistence::158::root::(_clearDisk) Clearing /var/run/vdsm/netconf/nets/ and /var/run/vdsm/netconf/bonds/ MainThread::INFO::2014-05-27 10:29:38,294::netconfpersistence::68::root::(setNetwork) Adding network ovirtmgmt({'nic': 'eth0vnet0', 'mtu': 1500, 'bootproto': 'dhcp', 'stp': False, 'bridged': True, 'defaultRoute': True}) MainThread::INFO::2014-05-27 10:29:38,294::netconfpersistence::158::root::(_clearDisk) Clearing /var/run/vdsm/netconf/nets/ and /var/run/vdsm/netconf/bonds/ MainThread::INFO::2014-05-27 10:29:38,294::netconfpersistence::182::root::(save) Saved new config RunningConfig({'ovirtmgmt': {'nic': 'eth0vnet0', 'mtu': 1500, 'bootproto': 'dhcp', 'stp': False, 'bridged': True, 'defaultRoute': True}}, {}) to /var/run/vdsm/netconf/nets/ and /var/run/vdsm/netconf/bonds/
Dima Kuznetsov has posted comments on this change.
Change subject: vdsm-tool: Change upgrade mechanism ......................................................................
Patch Set 8:
Upon further examination, this was missing logs, because logger was ignoring DEBUG level logs, fixed in http://gerrit.ovirt.org/#/c/27481,
Here is a comparison of old logs (lhs) and new logs (rhs): http://www.diffchecker.com/f65qrxae
Also changed the order of checks in upgrades to match the original code, upgrade function is only reached if there is any sense to run it (previously, it checked inside upgrade func).
oVirt Jenkins CI Server has posted comments on this change.
Change subject: vdsm-tool: Change upgrade mechanism ......................................................................
Patch Set 9:
Build Successful
http://jenkins.ovirt.org/job/vdsm_master_unit_tests_gerrit_el/8563/ : SUCCESS
http://jenkins.ovirt.org/job/vdsm_master_install_rpm_sanity_gerrit/649/ : SUCCESS
http://jenkins.ovirt.org/job/vdsm_master_unit_tests_gerrit/9497/ : SUCCESS
http://jenkins.ovirt.org/job/vdsm_master_pep8_gerrit/9351/ : SUCCESS
Assaf Muller has posted comments on this change.
Change subject: vdsm-tool: Change upgrade mechanism ......................................................................
Patch Set 9: Code-Review-1
(4 comments)
Sorry for the partial reviews, I hate when people do that. Just testing comments, rest looks good.
http://gerrit.ovirt.org/#/c/27193/9/tests/toolTests.py File tests/toolTests.py:
Line 370: def assertNotSealed(self, name): Line 371: self.assertFalse(self._checkSealExists(name)) Line 372: Line 373: def testRunOnce(self): Line 374: lst = [0] According to the principle of least surprise, I think you should have the number of upgrade invocations kept in the UpgraduratorTM.invocations, and not passed in as a list. Line 375: ret = upgrade.apply_upgrade(self.UpgraduratorTM(lst), 'test') Line 376: self.assertEquals(ret, 0) Line 377: self.assertEquals(lst[0], 1) Line 378: self.assertSealed('test')
Line 380: def testErrorInUpgrade(self): Line 381: bad = self.BadUpgraduratorTM() Line 382: ret = upgrade.apply_upgrade(bad, 'foobar') Line 383: self.assertEquals(ret, 1) Line 384: self.assertNotSealed('bad') 'bad' will never be sealed because the upgrade name is 'foobar'. Line 385: Line 386: def testRunMany(self): Line 387: lst = [0] Line 388: for _ in xrange(5):
Line 388: xrange xrange is not defined in Python 3. Are you guys using a library like six to ease the transition?
Line 404: upgrade.apply_upgrade(self.UpgraduratorTM(lst), 'test') Line 405: self.assertEquals(lst[0], 2) Line 406: self.assertSealed('test') Line 407: Line 408: def testNoUpgrade(self): I think you can safely remove this test... Line 409: self.assertNotSealed('test') Line 410: Line 411: def testUpgradeArgs(self): Line 412: u = self.UpgraduratorTM([0])
Dima Kuznetsov has posted comments on this change.
Change subject: vdsm-tool: Change upgrade mechanism ......................................................................
Patch Set 9:
(4 comments)
http://gerrit.ovirt.org/#/c/27193/9/tests/toolTests.py File tests/toolTests.py:
Line 370: def assertNotSealed(self, name): Line 371: self.assertFalse(self._checkSealExists(name)) Line 372: Line 373: def testRunOnce(self): Line 374: lst = [0]
According to the principle of least surprise, I think you should have the n
Ok, will change. Line 375: ret = upgrade.apply_upgrade(self.UpgraduratorTM(lst), 'test') Line 376: self.assertEquals(ret, 0) Line 377: self.assertEquals(lst[0], 1) Line 378: self.assertSealed('test')
Line 380: def testErrorInUpgrade(self): Line 381: bad = self.BadUpgraduratorTM() Line 382: ret = upgrade.apply_upgrade(bad, 'foobar') Line 383: self.assertEquals(ret, 1) Line 384: self.assertNotSealed('bad')
'bad' will never be sealed because the upgrade name is 'foobar'.
Upgrade name defined as BadUpgraduratorTM.name which is 'bad'.
The 1st argument passed is not used for name to avoid coupling with @exposed command name (in case you want to expose a single command under several names) Line 385: Line 386: def testRunMany(self): Line 387: lst = [0] Line 388: for _ in xrange(5):
Line 388: xrange
xrange is not defined in Python 3. Are you guys using a library like six to
Will change to range()
Line 404: upgrade.apply_upgrade(self.UpgraduratorTM(lst), 'test') Line 405: self.assertEquals(lst[0], 2) Line 406: self.assertSealed('test') Line 407: Line 408: def testNoUpgrade(self):
I think you can safely remove this test...
Ok Line 409: self.assertNotSealed('test') Line 410: Line 411: def testUpgradeArgs(self): Line 412: u = self.UpgraduratorTM([0])
Assaf Muller has posted comments on this change.
Change subject: vdsm-tool: Change upgrade mechanism ......................................................................
Patch Set 9:
(1 comment)
http://gerrit.ovirt.org/#/c/27193/9/tests/toolTests.py File tests/toolTests.py:
Line 380: def testErrorInUpgrade(self): Line 381: bad = self.BadUpgraduratorTM() Line 382: ret = upgrade.apply_upgrade(bad, 'foobar') Line 383: self.assertEquals(ret, 1) Line 384: self.assertNotSealed('bad')
Upgrade name defined as BadUpgraduratorTM.name which is 'bad'.
Oops, confused the argument with the upgrade name :) Line 385: Line 386: def testRunMany(self): Line 387: lst = [0] Line 388: for _ in xrange(5):
Assaf Muller has posted comments on this change.
Change subject: vdsm-tool: Change upgrade mechanism ......................................................................
Patch Set 10: Code-Review-1
No change in tests, keeping -1 for now...
Dima Kuznetsov has posted comments on this change.
Change subject: vdsm-tool: Change upgrade mechanism ......................................................................
Patch Set 10:
Oops, added to a later commit, fixing.
oVirt Jenkins CI Server has posted comments on this change.
Change subject: vdsm-tool: Change upgrade mechanism ......................................................................
Patch Set 10:
Build Successful
http://jenkins.ovirt.org/job/vdsm_master_unit_tests_gerrit_el/8583/ : SUCCESS
http://jenkins.ovirt.org/job/vdsm_master_install_rpm_sanity_gerrit/653/ : SUCCESS
http://jenkins.ovirt.org/job/vdsm_master_unit_tests_gerrit/9517/ : SUCCESS
http://jenkins.ovirt.org/job/vdsm_master_pep8_gerrit/9371/ : SUCCESS
Assaf Muller has posted comments on this change.
Change subject: vdsm-tool: Change upgrade mechanism ......................................................................
Patch Set 11: Code-Review+1
oVirt Jenkins CI Server has posted comments on this change.
Change subject: vdsm-tool: Change upgrade mechanism ......................................................................
Patch Set 11:
Build Successful
http://jenkins.ovirt.org/job/vdsm_master_unit_tests_gerrit_el/8591/ : SUCCESS
http://jenkins.ovirt.org/job/vdsm_master_install_rpm_sanity_gerrit/654/ : SUCCESS
http://jenkins.ovirt.org/job/vdsm_master_unit_tests_gerrit/9525/ : SUCCESS
http://jenkins.ovirt.org/job/vdsm_master_pep8_gerrit/9379/ : SUCCESS
mooli tayer has posted comments on this change.
Change subject: vdsm-tool: Change upgrade mechanism ......................................................................
Patch Set 11:
(1 comment)
http://gerrit.ovirt.org/#/c/27193/11/init/vdsmd_init_common.sh.in File init/vdsmd_init_common.sh.in:
Line 18: # Refer to the README and COPYING files for full details of the license Line 19: # Line 20: Line 21: VDSM_TOOL="@BINDIR@/vdsm-tool" Line 22: UPGRADE_LOGGING_PARAMS="--verbose --append --logfile=@VDSMLOGDIR@/upgrade.log" will this not break since these have not been defined yet? Line 23: prog=vdsm Line 24: Line 25: #### pre-start tasks #### Line 26: task_configure_coredump() {
Dima Kuznetsov has posted comments on this change.
Change subject: vdsm-tool: Change upgrade mechanism ......................................................................
Patch Set 11:
(1 comment)
http://gerrit.ovirt.org/#/c/27193/11/init/vdsmd_init_common.sh.in File init/vdsmd_init_common.sh.in:
Line 18: # Refer to the README and COPYING files for full details of the license Line 19: # Line 20: Line 21: VDSM_TOOL="@BINDIR@/vdsm-tool" Line 22: UPGRADE_LOGGING_PARAMS="--verbose --append --logfile=@VDSMLOGDIR@/upgrade.log"
will this not break since these have not been defined yet?
you're right, it won't fly with getopt, but order of the patches cannot be changed either (explained in comments). I'll move this to a later commit (separate or the one that adds the flags) Line 23: prog=vdsm Line 24: Line 25: #### pre-start tasks #### Line 26: task_configure_coredump() {
Yaniv Bronhaim has posted comments on this change.
Change subject: vdsm-tool: Change upgrade mechanism ......................................................................
Patch Set 11: Code-Review+1
Dima Kuznetsov has posted comments on this change.
Change subject: vdsm-tool: Change upgrade mechanism ......................................................................
Patch Set 11:
(1 comment)
http://gerrit.ovirt.org/#/c/27193/11/init/vdsmd_init_common.sh.in File init/vdsmd_init_common.sh.in:
Line 18: # Refer to the README and COPYING files for full details of the license Line 19: # Line 20: Line 21: VDSM_TOOL="@BINDIR@/vdsm-tool" Line 22: UPGRADE_LOGGING_PARAMS="--verbose --append --logfile=@VDSMLOGDIR@/upgrade.log"
you're right, it won't fly with getopt, but order of the patches cannot be
Done Line 23: prog=vdsm Line 24: Line 25: #### pre-start tasks #### Line 26: task_configure_coredump() {
oVirt Jenkins CI Server has posted comments on this change.
Change subject: vdsm-tool: Change upgrade mechanism ......................................................................
Patch Set 12:
Build Failed
http://jenkins.ovirt.org/job/vdsm_master_unit_tests_gerrit_el/9398/ : SUCCESS
http://jenkins.ovirt.org/job/vdsm_master_install_rpm_sanity_gerrit/754/ : SUCCESS
http://jenkins.ovirt.org/job/vdsm_master_pep8_gerrit/10182/ : SUCCESS
http://jenkins.ovirt.org/job/vdsm_master_unit_tests_gerrit/10338/ : SUCCESS
http://jenkins.ovirt.org/job/vdsm_master_verify-error-codes_merged/5264/ : FAILURE
http://jenkins.ovirt.org/job/vdsm_master_unit-tests_merged/3422/ : SUCCESS
Yaniv Bronhaim has posted comments on this change.
Change subject: vdsm-tool: Change upgrade mechanism ......................................................................
Patch Set 12: Code-Review+1
Dima Kuznetsov has posted comments on this change.
Change subject: vdsm-tool: Change upgrade mechanism ......................................................................
Patch Set 12: Verified+1
Verified with the following way (with subsequent patch to have logging): * manually removed old upgrade.log * manually removed seal files for unified persistance and 3.0.0 nets from /var/lib/vdsm/upgrade * removed vdsm-ovirtmgmt net with virsh * started vdsm
Both upgrades ran, as indicated by the log: http://ur1.ca/hlea0
Do not merge without http://gerrit.ovirt.org/#/c/27481, will result in upgrade log going to screen.
Dima Kuznetsov has posted comments on this change.
Change subject: vdsm-tool: Change upgrade mechanism ......................................................................
Patch Set 13:
verified after rebase as well
oVirt Jenkins CI Server has posted comments on this change.
Change subject: vdsm-tool: Change upgrade mechanism ......................................................................
Patch Set 13:
Build Failed
http://jenkins.ovirt.org/job/vdsm_master_unit_tests_gerrit_el/9611/ : SUCCESS
http://jenkins.ovirt.org/job/vdsm_master_pep8_gerrit/10396/ : SUCCESS
http://jenkins.ovirt.org/job/vdsm_master_unit_tests_gerrit/10553/ : SUCCESS
http://jenkins.ovirt.org/job/vdsm_master_verify-error-codes_merged/5477/ : SUCCESS
http://jenkins.ovirt.org/job/vdsm_master_unit-tests_merged/3635/ : SUCCESS
http://jenkins.ovirt.org/job/vdsm_master_install-rpm-sanity-fc19_created/27/ : FAILURE
http://jenkins.ovirt.org/job/vdsm_master_install-rpm-sanity-fc20_created/7/ : FAILURE
http://jenkins.ovirt.org/job/vdsm_master_install-rpm-sanity-el6_created/8/ : FAILURE
Antoni Segura Puimedon has posted comments on this change.
Change subject: vdsm-tool: Change upgrade mechanism ......................................................................
Patch Set 13:
I don't see a big need for unified_persistence.py and upgrade_300_networks.py to have an upgrade class instead of just having:
- a run that takes ns and args - an extendArgParser that extends if needed otherwise leaves the parser untouched.
I'm glad that this patch does away with the upgrade class but I would really prefer if it did away with these extra classes I mentioned above.
Dan Kenigsberg has posted comments on this change.
Change subject: vdsm-tool: Change upgrade mechanism ......................................................................
Patch Set 13: Code-Review+2
raising score
Dan Kenigsberg has submitted this change and it was merged.
Change subject: vdsm-tool: Change upgrade mechanism ......................................................................
vdsm-tool: Change upgrade mechanism
The upgrade decorator is a mechanism within vdsm-tool to allow commands to only run once. It has flags to override, but being a function decorator, it did not have any direct access to command line arguments and did access via sys.argv[2:].
The patch: http://gerrit.ovirt.org/#/c/27481/ adds flags to vdsm-tool and will break upgrade and its argument parsing. I'm proposing to change the upgrade from being a decorated function, to a function that operates on an upgrade object (interface documented in apply_upgrade docstring) and the command line arguments.
This also allows to have a single argument parser for both general upgrade flags and specific per-upgrade flags (no current upgrades define flags but we might want this in future, available via extendArgParser method, see docs).
Change-Id: I6e1d28570dedfeff9fe60624b1db72d8cadf136a Signed-off-by: Dima Kuznetsov dkuznets@redhat.com Reviewed-on: http://gerrit.ovirt.org/27193 Reviewed-by: Yaniv Bronhaim ybronhei@redhat.com Reviewed-by: Dan Kenigsberg danken@redhat.com --- M .gitignore M debian/vdsm.install M lib/vdsm/tool/unified_persistence.py M lib/vdsm/tool/upgrade.py M lib/vdsm/tool/upgrade_300_networks.py M tests/toolTests.py M vdsm.spec.in M vdsm/Makefile.am D vdsm/upgrade.logger.conf.in 9 files changed, 181 insertions(+), 165 deletions(-)
Approvals: Yaniv Bronhaim: Looks good to me, but someone else must approve Dima Kuznetsov: Verified Dan Kenigsberg: Looks good to me, approved
oVirt Jenkins CI Server has posted comments on this change.
Change subject: vdsm-tool: Change upgrade mechanism ......................................................................
Patch Set 14:
Build Successful
http://jenkins.ovirt.org/job/vdsm_master_create-rpms_merged/1544/ : SUCCESS
http://jenkins.ovirt.org/job/vdsm_master_verify-error-codes_merged/5529/ : SUCCESS
http://jenkins.ovirt.org/job/vdsm_master_unit-tests_merged/3687/ : SUCCESS
vdsm-patches@lists.fedorahosted.org