Change in vdsm[master]: iproute2: Try to remove all dynamic routes even if removal o...

Ondřej Svoboda has uploaded a new change for review.

Change subject: iproute2: Try to remove all dynamic routes even if removal of one fails
......................................................................

iproute2: Try to remove all dynamic routes even if removal of one fails

When trying to remove a non-existent route (or is its table gone?)
iproute2 fails (retcode=2) with

  RTNETLINK answers: No such process

for a reason currently unknown.

In this draft, deal with routes first, i.e. use the same try-except block
as in the caller, DynamicSourceRoute.remove(). But I think even the removal
of individual rules should be wrapped this way, in which case the try-except
block could be removed in the caller.

Change-Id: I6e63d5ceba72be91fdab5ffd9a71ceab398950b9
Signed-off-by: Ondřej Svoboda <osvoboda(a)redhat.com&gt;
---
M vdsm/network/configurators/iproute2.py
1 file changed, 5 insertions(+), 1 deletion(-)


  git pull ssh://gerrit.ovirt.org:29418/vdsm refs/changes/52/36652/1

diff --git a/vdsm/network/configurators/iproute2.py
b/vdsm/network/configurators/iproute2.py
index ed2dab5..700042c 100644
--- a/vdsm/network/configurators/iproute2.py
+++ b/vdsm/network/configurators/iproute2.py
@@ -213,7 +213,11 @@
     @staticmethod
     def removeSourceRoute(routes, rules, device):
         for route in routes:
-            routeDel(route)
+            try:
+                routeDel(route)
+            except IPRoute2Error as e:
+                logging.error('ip binary failed during source route '
+                              'removal: %s', e.message)
 
         for rule in rules:
             ruleDel(rule)


-- 
To view, visit http://gerrit.ovirt.org/36652
To unsubscribe, visit http://gerrit.ovirt.org/settings

Gerrit-MessageType: newchange
Gerrit-Change-Id: I6e63d5ceba72be91fdab5ffd9a71ceab398950b9
Gerrit-PatchSet: 1
Gerrit-Project: vdsm
Gerrit-Branch: master
Gerrit-Owner: Ondřej Svoboda <osvoboda(a)redhat.com&gt;

automation(a)ovirt.org has posted comments on this change.

Change subject: iproute2: Try to remove all dynamic routes even if removal of one fails
......................................................................


Patch Set 1:

* Update tracker::IGNORE, no Bug-Url found
* Check Bug-Url::WARN, no bug url found, make sure header matches 'Bug-Url: ' and
is a valid url.
* Check merged to previous::IGNORE, Not in stable branch (['ovirt-3.5',
'ovirt-3.4', 'ovirt-3.3'])

-- 
To view, visit http://gerrit.ovirt.org/36652
To unsubscribe, visit http://gerrit.ovirt.org/settings

Gerrit-MessageType: comment
Gerrit-Change-Id: I6e63d5ceba72be91fdab5ffd9a71ceab398950b9
Gerrit-PatchSet: 1
Gerrit-Project: vdsm
Gerrit-Branch: master
Gerrit-Owner: Ondřej Svoboda <osvoboda(a)redhat.com&gt;
Gerrit-Reviewer: automation(a)ovirt.org
Gerrit-HasComments: No

[Jenkins CI RO]

oVirt Jenkins CI Server has posted comments on this change.

Change subject: iproute2: Try to remove all dynamic routes even if removal of one fails
......................................................................


Patch Set 1:

Build Failed 

http://jenkins.ovirt.org/job/vdsm_master_unit-tests_created/14835/ : SUCCESS

http://jenkins.ovirt.org/job/vdsm_master_pep8_gerrit/14667/ : SUCCESS

http://jenkins.ovirt.org/job/vdsm_master_unit_tests_gerrit_el/13879/ : FAILURE

-- 
To view, visit http://gerrit.ovirt.org/36652
To unsubscribe, visit http://gerrit.ovirt.org/settings

Gerrit-MessageType: comment
Gerrit-Change-Id: I6e63d5ceba72be91fdab5ffd9a71ceab398950b9
Gerrit-PatchSet: 1
Gerrit-Project: vdsm
Gerrit-Branch: master
Gerrit-Owner: Ondřej Svoboda <osvoboda(a)redhat.com&gt;
Gerrit-Reviewer: automation(a)ovirt.org
Gerrit-Reviewer: oVirt Jenkins CI Server
Gerrit-HasComments: No

Ondřej Svoboda has posted comments on this change.

Change subject: iproute2: Try to remove all dynamic routes even if removal of one fails
......................................................................


Patch Set 1: Code-Review-1

(1 comment)

Missing import.

http://gerrit.ovirt.org/#/c/36652/1/vdsm/network/configurators/iproute2.py
File vdsm/network/configurators/iproute2.py:

Line 23: from vdsm import ipwrapper
Line 24: from vdsm.constants import EXT_BRCTL
Line 25: from vdsm.ipwrapper import routeAdd
Line 26: from vdsm.ipwrapper import routeDel
Line 27: from vdsm.ipwrapper import ruleAdd
from vdsm.ipwrapper import IPRoute2Error
Line 28: from vdsm.ipwrapper import ruleDel
Line 29: from vdsm.netconfpersistence import RunningConfig
Line 30: from vdsm.utils import CommandPath
Line 31: from vdsm.utils import execCmd


-- 
To view, visit http://gerrit.ovirt.org/36652
To unsubscribe, visit http://gerrit.ovirt.org/settings

Gerrit-MessageType: comment
Gerrit-Change-Id: I6e63d5ceba72be91fdab5ffd9a71ceab398950b9
Gerrit-PatchSet: 1
Gerrit-Project: vdsm
Gerrit-Branch: master
Gerrit-Owner: Ondřej Svoboda <osvoboda(a)redhat.com&gt;
Gerrit-Reviewer: Ondřej Svoboda <osvoboda(a)redhat.com&gt;
Gerrit-Reviewer: automation(a)ovirt.org
Gerrit-Reviewer: oVirt Jenkins CI Server
Gerrit-HasComments: Yes

Change in vdsm[master]: iproute2: ignore source routes already removed by the kernel

automation(a)ovirt.org has posted comments on this change.

Change subject: iproute2: ignore source routes already removed by the kernel
......................................................................


Patch Set 2:

* Update tracker::IGNORE, no Bug-Url found
* Check Bug-Url::WARN, no bug url found, make sure header matches 'Bug-Url: ' and
is a valid url.
* Check merged to previous::IGNORE, Not in stable branch (['ovirt-3.5',
'ovirt-3.4', 'ovirt-3.3'])

-- 
To view, visit https://gerrit.ovirt.org/36652
To unsubscribe, visit https://gerrit.ovirt.org/settings

Gerrit-MessageType: comment
Gerrit-Change-Id: I6e63d5ceba72be91fdab5ffd9a71ceab398950b9
Gerrit-PatchSet: 2
Gerrit-Project: vdsm
Gerrit-Branch: master
Gerrit-Owner: Ondřej Svoboda <osvoboda(a)redhat.com&gt;
Gerrit-Reviewer: Jenkins CI
Gerrit-Reviewer: Ondřej Svoboda <osvoboda(a)redhat.com&gt;
Gerrit-Reviewer: automation(a)ovirt.org
Gerrit-HasComments: No

Change in vdsm[master]: iproute2: ignore source routes already removed by the kernel

Ondřej Svoboda has posted comments on this change.

Change subject: iproute2: ignore source routes already removed by the kernel
......................................................................


Patch Set 2:

(1 comment)

A little correction.

https://gerrit.ovirt.org/#/c/36652/2//COMMIT_MSG
Commit Message:

Line 16: But it is also implied
       : that we don't have to remove the routes at all (just routing rules).
… only if we always ifdowned the network device.


-- 
To view, visit https://gerrit.ovirt.org/36652
To unsubscribe, visit https://gerrit.ovirt.org/settings

Gerrit-MessageType: comment
Gerrit-Change-Id: I6e63d5ceba72be91fdab5ffd9a71ceab398950b9
Gerrit-PatchSet: 2
Gerrit-Project: vdsm
Gerrit-Branch: master
Gerrit-Owner: Ondřej Svoboda <osvoboda(a)redhat.com&gt;
Gerrit-Reviewer: Dan Kenigsberg <danken(a)redhat.com&gt;
Gerrit-Reviewer: Ido Barkan <ibarkan(a)redhat.com&gt;
Gerrit-Reviewer: Jenkins CI
Gerrit-Reviewer: Ondřej Svoboda <osvoboda(a)redhat.com&gt;
Gerrit-Reviewer: Petr Horáček <phoracek(a)redhat.com&gt;
Gerrit-Reviewer: automation(a)ovirt.org
Gerrit-HasComments: Yes

Change in vdsm[master]: iproute2: ignore source routes already removed by the kernel

Ondřej Svoboda has posted comments on this change.

Change subject: iproute2: ignore source routes already removed by the kernel
......................................................................


Patch Set 2: Verified+1

testSetupNetworksAddDelDhcp now passes on EL7 with all configurators.

-- 
To view, visit https://gerrit.ovirt.org/36652
To unsubscribe, visit https://gerrit.ovirt.org/settings

Gerrit-MessageType: comment
Gerrit-Change-Id: I6e63d5ceba72be91fdab5ffd9a71ceab398950b9
Gerrit-PatchSet: 2
Gerrit-Project: vdsm
Gerrit-Branch: master
Gerrit-Owner: Ondřej Svoboda <osvoboda(a)redhat.com&gt;
Gerrit-Reviewer: Dan Kenigsberg <danken(a)redhat.com&gt;
Gerrit-Reviewer: Ido Barkan <ibarkan(a)redhat.com&gt;
Gerrit-Reviewer: Jenkins CI
Gerrit-Reviewer: Ondřej Svoboda <osvoboda(a)redhat.com&gt;
Gerrit-Reviewer: Petr Horáček <phoracek(a)redhat.com&gt;
Gerrit-Reviewer: automation(a)ovirt.org
Gerrit-HasComments: No

Change in vdsm[master]: iproute2: source routes are removed by the kernel on ifdown

automation(a)ovirt.org has posted comments on this change.

Change subject: iproute2: source routes are removed by the kernel on ifdown
......................................................................


Patch Set 3:

* Update tracker::IGNORE, no Bug-Url found
* Check Bug-Url::WARN, no bug url found, make sure header matches 'Bug-Url: ' and
is a valid url.
* Check merged to previous::IGNORE, Not in stable branch (['ovirt-3.5',
'ovirt-3.4', 'ovirt-3.3'])

-- 
To view, visit https://gerrit.ovirt.org/36652
To unsubscribe, visit https://gerrit.ovirt.org/settings

Gerrit-MessageType: comment
Gerrit-Change-Id: I6e63d5ceba72be91fdab5ffd9a71ceab398950b9
Gerrit-PatchSet: 3
Gerrit-Project: vdsm
Gerrit-Branch: master
Gerrit-Owner: Ondřej Svoboda <osvoboda(a)redhat.com&gt;
Gerrit-Reviewer: Dan Kenigsberg <danken(a)redhat.com&gt;
Gerrit-Reviewer: Ido Barkan <ibarkan(a)redhat.com&gt;
Gerrit-Reviewer: Jenkins CI
Gerrit-Reviewer: Ondřej Svoboda <osvoboda(a)redhat.com&gt;
Gerrit-Reviewer: Petr Horáček <phoracek(a)redhat.com&gt;
Gerrit-Reviewer: automation(a)ovirt.org
Gerrit-HasComments: No

Change in vdsm[master]: iproute2: source routes are removed by the kernel on ifdown

Petr Horáček has posted comments on this change.

Change subject: iproute2: source routes are removed by the kernel on ifdown
......................................................................


Patch Set 3: Code-Review-1

I think it would deserve a test, don't know if something like it already exists.

-- 
To view, visit https://gerrit.ovirt.org/36652
To unsubscribe, visit https://gerrit.ovirt.org/settings

Gerrit-MessageType: comment
Gerrit-Change-Id: I6e63d5ceba72be91fdab5ffd9a71ceab398950b9
Gerrit-PatchSet: 3
Gerrit-Project: vdsm
Gerrit-Branch: master
Gerrit-Owner: Ondřej Svoboda <osvoboda(a)redhat.com&gt;
Gerrit-Reviewer: Dan Kenigsberg <danken(a)redhat.com&gt;
Gerrit-Reviewer: Ido Barkan <ibarkan(a)redhat.com&gt;
Gerrit-Reviewer: Jenkins CI
Gerrit-Reviewer: Ondřej Svoboda <osvoboda(a)redhat.com&gt;
Gerrit-Reviewer: Petr Horáček <phoracek(a)redhat.com&gt;
Gerrit-Reviewer: automation(a)ovirt.org
Gerrit-HasComments: No

Change in vdsm[master]: iproute2: source routes are removed by the kernel on ifdown

automation(a)ovirt.org has posted comments on this change.

Change subject: iproute2: source routes are removed by the kernel on ifdown
......................................................................


Patch Set 4:

* Update tracker::IGNORE, no Bug-Url found
* Check Bug-Url::WARN, no bug url found, make sure header matches 'Bug-Url: ' and
is a valid url.
* Check merged to previous::IGNORE, Not in stable branch (['ovirt-3.5',
'ovirt-3.4', 'ovirt-3.3'])

-- 
To view, visit https://gerrit.ovirt.org/36652
To unsubscribe, visit https://gerrit.ovirt.org/settings

Gerrit-MessageType: comment
Gerrit-Change-Id: I6e63d5ceba72be91fdab5ffd9a71ceab398950b9
Gerrit-PatchSet: 4
Gerrit-Project: vdsm
Gerrit-Branch: master
Gerrit-Owner: Ondřej Svoboda <osvoboda(a)redhat.com&gt;
Gerrit-Reviewer: Dan Kenigsberg <danken(a)redhat.com&gt;
Gerrit-Reviewer: Ido Barkan <ibarkan(a)redhat.com&gt;
Gerrit-Reviewer: Jenkins CI
Gerrit-Reviewer: Ondřej Svoboda <osvoboda(a)redhat.com&gt;
Gerrit-Reviewer: Petr Horáček <phoracek(a)redhat.com&gt;
Gerrit-Reviewer: automation(a)ovirt.org
Gerrit-HasComments: No

Change in vdsm[master]: iproute2: source routes are removed by the kernel on ifdown

Ondřej Svoboda has posted comments on this change.

Change subject: iproute2: source routes are removed by the kernel on ifdown
......................................................................


Patch Set 4: Verified+1

There is a test, actually.

testSetupNetworksAddDelDhcp checks that both routes and routing rules are gone when the
test-network is removed.

You can the discussed testing code in the recent readability improvement:
https://gerrit.ovirt.org/#/c/41236/

As for verification, I ran all (including slow) functional tests and there were only 4
recurring failures:

======================================================================
FAIL: testSetupNetworksAddBondWithManyVlans(kwargs=False)
(functional.networkTests.NetworkTest)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/usr/share/vdsm/tests/testlib.py", line 64, in wrapper
    return f(self, *args)
  File "/usr/share/vdsm/tests/functional/networkTests.py", line 194, in wrapper
    func(*args, **kwargs)
  File "/usr/share/vdsm/tests/functional/networkTests.py", line 423, in
testSetupNetworksAddBondWithManyVlans
    self.retryAssert(assertDevStatsReported, timeout=20)
  File "/usr/share/vdsm/tests/testlib.py", line 162, in retryAssert
    return retry(expectedException=AssertionError, *args, **kwargs)
  File "/usr/lib/python2.7/site-packages/vdsm/utils.py", line 1008, in retry
    return func()
  File "/usr/share/vdsm/tests/functional/networkTests.py", line 402, in
assertDevStatsReported
    BONDING_NAME + '.' + str(tag), hostStats['network'])
AssertionError: 'bond11.0' not found in {'bond0': {'sampleTime':
1432889660.452088, 'rxDropped': '0', 'tx': '0',
'rxErrors': '0', 'rx': '0', 'txDropped':
'0', 'txRate': '0.0', 'rxRate': '0.0',
'txErrors': '0', 'state': 'down', 'speed':
'1000', 'name': 'bond0'}, 'lo': {'sampleTime':
1432889660.452088, 'rxDropped': '0', 'tx': '4385570',
'rxErrors': '0', 'rx': '4385570', 'txDropped':
'0', 'txRate': '0.0', 'rxRate': '0.0',
'txErrors': '0', 'state': 'up', 'speed':
'1000', 'name': 'lo'}, 'dummy_3OnBX':
{'sampleTime': 1432889660.452088, 'rxDropped': '0', 'tx':
'0', 'rxErrors': '0', 'rx': '0',
'txDropped': '0', 'txRate': '0.0', 'rxRate':
'0.0', 'txErrors': '0', 'state': 'down',
'speed': '1000', 'name': 'dummy_3OnBX'},
'dummy_ZWNeo': {'sampleTime': 1432889660.452088, 'rxDropped':
'0', 'tx': '0', 'rxErrors': '0', 'rx':
'0', 'txDropped': '0', 'txRate': '0.0',
'rxRate': '0.0', 'txErrors': '0', 'state':
'down', 'speed': '1000', 'name': 'dummy_ZWNeo'},
';vdsmdummy;': {'sampleTime': 143288!
 9660.452088, 'rxDropped': '0', 'tx': '0',
'rxErrors': '0', 'rx': '0', 'txDropped':
'0', 'txRate': '0.0', 'rxRate': '0.0',
'txErrors': '0', 'state': 'down', 'speed':
'1000', 'name': ';vdsmdummy;'}, 'dummy_Sa1vh':
{'sampleTime': 1432889660.452088, 'rxDropped': '0', 'tx':
'0', 'rxErrors': '0', 'rx': '0',
'txDropped': '0', 'txRate': '0.0', 'rxRate':
'0.0', 'txErrors': '0', 'state': 'down',
'speed': '1000', 'name': 'dummy_Sa1vh'}, 'bond11':
{'sampleTime': 1432889660.452088, 'rxDropped': '0', 'tx':
'55978', 'rxErrors': '0', 'rx': '0',
'txDropped': '0', 'txRate': '0.0', 'rxRate':
'0.0', 'txErrors': '0', 'state': 'up',
'speed': '1000', 'name': 'bond11'}, 'dummy_5Eslg':
{'sampleTime': 1432889660.452088, 'rxDropped': '0', 'tx':
'48890', 'rxErrors': '0', 'rx': '0',
'txDropped': '0', 'txRate': '0.0', 'rxRate':
'0.0', 'txErrors': '0', 'state': 'up',
'speed': '1000', 'name': 'dummy_5Eslg'}, 'dummy0':
{'sampleTime': 1432889660.452088, 'rxDropped': '0', 'tx':
'0', 'rxErrors': '0', 'rx': '0!
 ', 'txDropped': '0', 'txRate': '0.0',
'rxRate': '0.0', 'txErrors': '0', 'state':
'down', 'speed': '1000', 'name': 'dummy0'},
'dummy_YGKbY': {'sampleTime': 1432889660.452088, 'rxDropped':
'0', 'tx': '7088', 'rxErrors': '0', 'rx':
'0', 'txDropped': '0', 'txRate': '0.0',
'rxRate': '0.0', 'txErrors': '0', 'state':
'up', 'speed': '1000', 'name': 'dummy_YGKbY'},
'eth0': {'sampleTime': 1432889660.452088, 'rxDropped':
'0', 'tx': '1383818', 'rxErrors': '0',
'rx': '29607244', 'txDropped': '0', 'txRate':
'0.0', 'rxRate': '0.0', 'txErrors': '0',
'state': 'up', 'speed': '1000', 'name':
'eth0'}}
======================================================================
FAIL: testSetupNetworksAddBondWithManyVlans(kwargs=True)
(functional.networkTests.NetworkTest)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/usr/share/vdsm/tests/testlib.py", line 64, in wrapper
    return f(self, *args)
  File "/usr/share/vdsm/tests/functional/networkTests.py", line 194, in wrapper
    func(*args, **kwargs)
  File "/usr/share/vdsm/tests/functional/networkTests.py", line 423, in
testSetupNetworksAddBondWithManyVlans
    self.retryAssert(assertDevStatsReported, timeout=20)
  File "/usr/share/vdsm/tests/testlib.py", line 162, in retryAssert
    return retry(expectedException=AssertionError, *args, **kwargs)
  File "/usr/lib/python2.7/site-packages/vdsm/utils.py", line 1008, in retry
    return func()
  File "/usr/share/vdsm/tests/functional/networkTests.py", line 402, in
assertDevStatsReported
    BONDING_NAME + '.' + str(tag), hostStats['network'])
AssertionError: 'bond11.0' not found in {'bond0': {'sampleTime':
1432889690.469124, 'rxDropped': '0', 'tx': '0',
'rxErrors': '0', 'rx': '0', 'txDropped':
'0', 'txRate': '0.0', 'rxRate': '0.0',
'txErrors': '0', 'state': 'down', 'speed':
'1000', 'name': 'bond0'}, 'lo': {'sampleTime':
1432889690.469124, 'rxDropped': '0', 'tx': '4556517',
'rxErrors': '0', 'rx': '4556517', 'txDropped':
'0', 'txRate': '0.0', 'rxRate': '0.0',
'txErrors': '0', 'state': 'up', 'speed':
'1000', 'name': 'lo'}, 'dummy_3OnBX':
{'sampleTime': 1432889690.469124, 'rxDropped': '0', 'tx':
'0', 'rxErrors': '0', 'rx': '0',
'txDropped': '0', 'txRate': '0.0', 'rxRate':
'0.0', 'txErrors': '0', 'state': 'down',
'speed': '1000', 'name': 'dummy_3OnBX'},
'dummy_ZWNeo': {'sampleTime': 1432889690.469124, 'rxDropped':
'0', 'tx': '0', 'rxErrors': '0', 'rx':
'0', 'txDropped': '0', 'txRate': '0.0',
'rxRate': '0.0', 'txErrors': '0', 'state':
'down', 'speed': '1000', 'name': 'dummy_ZWNeo'},
';vdsmdummy;': {'sampleTime': 143288!
 9690.469124, 'rxDropped': '0', 'tx': '0',
'rxErrors': '0', 'rx': '0', 'txDropped':
'0', 'txRate': '0.0', 'rxRate': '0.0',
'txErrors': '0', 'state': 'down', 'speed':
'1000', 'name': ';vdsmdummy;'}, 'dummy_Sa1vh':
{'sampleTime': 1432889690.469124, 'rxDropped': '0', 'tx':
'0', 'rxErrors': '0', 'rx': '0',
'txDropped': '0', 'txRate': '0.0', 'rxRate':
'0.0', 'txErrors': '0', 'state': 'down',
'speed': '1000', 'name': 'dummy_Sa1vh'},
'dummy_5Eslg': {'sampleTime': 1432889690.469124, 'rxDropped':
'0', 'tx': '54050', 'rxErrors': '0', 'rx':
'0', 'txDropped': '0', 'txRate': '0.0',
'rxRate': '0.0', 'txErrors': '0', 'state':
'up', 'speed': '1000', 'name': 'dummy_5Eslg'},
'dummy0': {'sampleTime': 1432889690.469124, 'rxDropped':
'0', 'tx': '0', 'rxErrors': '0', 'rx':
'0', 'txDropped': '0', 'txRate': '0.0',
'rxRate': '0.0', 'txErrors': '0', 'state':
'down', 'speed': '1000', 'name': 'dummy0'},
'dummy_YGKbY': {'sampleTime': 1432889690.469124, 'rxDropped':
'0', 'tx': '7538', 'rxErrors': '0', 'r!
 x': '0', 'txDropped': '0', 'txRate': '0.0',
'rxRate': '0.0', 'txErrors': '0', 'state':
'up', 'speed': '1000', 'name': 'dummy_YGKbY'},
'eth0': {'sampleTime': 1432889690.469124, 'rxDropped':
'0', 'tx': '1386092', 'rxErrors': '0',
'rx': '29611812', 'txDropped': '0', 'txRate':
'0.0', 'rxRate': '0.0', 'txErrors': '0',
'state': 'up', 'speed': '1000', 'name':
'eth0'}}

======================================================================
FAIL: testSetupNetworksEmergencyDevicesCleanupVlanOverwrite(kwargs=False)
(functional.networkTests.NetworkTest)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/usr/share/vdsm/tests/testlib.py", line 64, in wrapper
    return f(self, *args)
  File "/usr/share/vdsm/tests/testValidation.py", line 170, in wrapper
    return f(*args, **kwargs)
  File "/usr/share/vdsm/tests/functional/networkTests.py", line 194, in wrapper
    func(*args, **kwargs)
  File "/usr/share/vdsm/tests/functional/networkTests.py", line 2077, in
testSetupNetworksEmergencyDevicesCleanupVlanOverwrite
    self.assertEquals(status, SUCCESS, msg)
AssertionError: nic dummy_5Eslg already enslaved to bond11

======================================================================
FAIL: test_getVdsStats (functional.networkTests.NetworkTest)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/usr/share/vdsm/tests/functional/networkTests.py", line 194, in wrapper
    func(*args, **kwargs)
  File "/usr/share/vdsm/tests/testValidation.py", line 160, in wrapper
    return f(*args, **kwargs)
  File "/usr/share/vdsm/tests/testValidation.py", line 105, in wrapper
    return f(*args, **kwargs)
  File "/usr/share/vdsm/tests/functional/networkTests.py", line 381, in
test_getVdsStats
    assertTestDevStatsReported, timeout=20)
  File "/usr/share/vdsm/tests/testlib.py", line 162, in retryAssert
    return retry(expectedException=AssertionError, *args, **kwargs)
  File "/usr/lib/python2.7/site-packages/vdsm/utils.py", line 1008, in retry
    return func()
  File "/usr/share/vdsm/tests/functional/networkTests.py", line 348, in
assertTestDevStatsReported
    left, hostStats['network'], 'could not find veth %s' % left)
AssertionError: could not find veth veth_eA9OjRtPR7
-------------------- >> begin captured logging << --------------------
root: DEBUG: /usr/bin/nmcli conn load /etc/sysconfig/network-scripts/ifcfg-veth_eA9OjRtPR7
(cwd None)
root: DEBUG: SUCCESS: <err> = ''; <rc> = 0
root: DEBUG: /usr/bin/nmcli conn load /etc/sysconfig/network-scripts/ifcfg-veth_wS5S9Gh49V
(cwd None)
root: DEBUG: SUCCESS: <err> = ''; <rc> = 0
root: DEBUG: /sbin/ip link add name veth_eA9OjRtPR7 type veth peer name veth_wS5S9Gh49V
(cwd None)
root: DEBUG: SUCCESS: <err> = ''; <rc> = 0
root: DEBUG: /sbin/ip link set dev veth_eA9OjRtPR7 up (cwd None)
root: DEBUG: SUCCESS: <err> = ''; <rc> = 0
root: DEBUG: /sbin/ip link set dev veth_wS5S9Gh49V up (cwd None)
root: DEBUG: SUCCESS: <err> = ''; <rc> = 0
root: DEBUG: /sbin/ip link del dev veth_eA9OjRtPR7 (cwd None)
root: DEBUG: SUCCESS: <err> = ''; <rc> = 0
root: DEBUG: /usr/bin/nmcli conn load /etc/sysconfig/network-scripts/ifcfg-veth_eA9OjRtPR7
(cwd None)
root: DEBUG: SUCCESS: <err> = ''; <rc> = 0
root: DEBUG: /usr/bin/nmcli conn load /etc/sysconfig/network-scripts/ifcfg-veth_wS5S9Gh49V
(cwd None)
root: DEBUG: SUCCESS: <err> = ''; <rc> = 0
--------------------- >> end captured logging << ---------------------

----------------------------------------------------------------------
Ran 116 tests in 1229.748s

FAILED (failures=4)

-- 
To view, visit https://gerrit.ovirt.org/36652
To unsubscribe, visit https://gerrit.ovirt.org/settings

Gerrit-MessageType: comment
Gerrit-Change-Id: I6e63d5ceba72be91fdab5ffd9a71ceab398950b9
Gerrit-PatchSet: 4
Gerrit-Project: vdsm
Gerrit-Branch: master
Gerrit-Owner: Ondřej Svoboda <osvoboda(a)redhat.com&gt;
Gerrit-Reviewer: Dan Kenigsberg <danken(a)redhat.com&gt;
Gerrit-Reviewer: Ido Barkan <ibarkan(a)redhat.com&gt;
Gerrit-Reviewer: Jenkins CI
Gerrit-Reviewer: Ondřej Svoboda <osvoboda(a)redhat.com&gt;
Gerrit-Reviewer: Petr Horáček <phoracek(a)redhat.com&gt;
Gerrit-Reviewer: automation(a)ovirt.org
Gerrit-HasComments: No

Change in vdsm[master]: iproute2: source routes are removed by the kernel on ifdown

Petr Horáček has posted comments on this change.

Change subject: iproute2: source routes are removed by the kernel on ifdown
......................................................................


Patch Set 4: Code-Review+1

-- 
To view, visit https://gerrit.ovirt.org/36652
To unsubscribe, visit https://gerrit.ovirt.org/settings

Gerrit-MessageType: comment
Gerrit-Change-Id: I6e63d5ceba72be91fdab5ffd9a71ceab398950b9
Gerrit-PatchSet: 4
Gerrit-Project: vdsm
Gerrit-Branch: master
Gerrit-Owner: Ondřej Svoboda <osvoboda(a)redhat.com&gt;
Gerrit-Reviewer: Dan Kenigsberg <danken(a)redhat.com&gt;
Gerrit-Reviewer: Ido Barkan <ibarkan(a)redhat.com&gt;
Gerrit-Reviewer: Jenkins CI
Gerrit-Reviewer: Ondřej Svoboda <osvoboda(a)redhat.com&gt;
Gerrit-Reviewer: Petr Horáček <phoracek(a)redhat.com&gt;
Gerrit-Reviewer: automation(a)ovirt.org
Gerrit-HasComments: No

Change in vdsm[master]: iproute2: source routes are removed by the kernel on ifdown

automation(a)ovirt.org has posted comments on this change.

Change subject: iproute2: source routes are removed by the kernel on ifdown
......................................................................


Patch Set 5:

* Update tracker::IGNORE, no Bug-Url found
* Check Bug-Url::WARN, no bug url found, make sure header matches 'Bug-Url: ' and
is a valid url.
* Check merged to previous::IGNORE, Not in stable branch (['ovirt-3.5',
'ovirt-3.4', 'ovirt-3.3'])

-- 
To view, visit https://gerrit.ovirt.org/36652
To unsubscribe, visit https://gerrit.ovirt.org/settings

Gerrit-MessageType: comment
Gerrit-Change-Id: I6e63d5ceba72be91fdab5ffd9a71ceab398950b9
Gerrit-PatchSet: 5
Gerrit-Project: vdsm
Gerrit-Branch: master
Gerrit-Owner: Ondřej Svoboda <osvoboda(a)redhat.com&gt;
Gerrit-Reviewer: Dan Kenigsberg <danken(a)redhat.com&gt;
Gerrit-Reviewer: Ido Barkan <ibarkan(a)redhat.com&gt;
Gerrit-Reviewer: Jenkins CI
Gerrit-Reviewer: Ondřej Svoboda <osvoboda(a)redhat.com&gt;
Gerrit-Reviewer: Petr Horáček <phoracek(a)redhat.com&gt;
Gerrit-Reviewer: automation(a)ovirt.org
Gerrit-HasComments: No

Change in vdsm[master]: sourceroute: don't fail to remove a dynamic source route

automation(a)ovirt.org has posted comments on this change.

Change subject: sourceroute: don't fail to remove a dynamic source route
......................................................................


Patch Set 6:

* Update tracker::IGNORE, no Bug-Url found
* Check Bug-Url::WARN, no bug url found, make sure header matches 'Bug-Url: ' and
is a valid url.
* Check merged to previous::IGNORE, Not in stable branch (['ovirt-3.5',
'ovirt-3.4', 'ovirt-3.3'])

-- 
To view, visit https://gerrit.ovirt.org/36652
To unsubscribe, visit https://gerrit.ovirt.org/settings

Gerrit-MessageType: comment
Gerrit-Change-Id: I6e63d5ceba72be91fdab5ffd9a71ceab398950b9
Gerrit-PatchSet: 6
Gerrit-Project: vdsm
Gerrit-Branch: master
Gerrit-Owner: Ondřej Svoboda <osvoboda(a)redhat.com&gt;
Gerrit-Reviewer: Dan Kenigsberg <danken(a)redhat.com&gt;
Gerrit-Reviewer: Ido Barkan <ibarkan(a)redhat.com&gt;
Gerrit-Reviewer: Jenkins CI
Gerrit-Reviewer: Ondřej Svoboda <osvoboda(a)redhat.com&gt;
Gerrit-Reviewer: Petr Horáček <phoracek(a)redhat.com&gt;
Gerrit-Reviewer: automation(a)ovirt.org
Gerrit-HasComments: No

Change in vdsm[master]: sourceroute: don't fail to remove a dynamic source route

Ondřej Svoboda has posted comments on this change.

Change subject: sourceroute: don't fail to remove a dynamic source route
......................................................................


Patch Set 6: Verified+1

testSetupNetworksAddDelDhcp works on EL7. (Even with NM running!)

-- 
To view, visit https://gerrit.ovirt.org/36652
To unsubscribe, visit https://gerrit.ovirt.org/settings

Gerrit-MessageType: comment
Gerrit-Change-Id: I6e63d5ceba72be91fdab5ffd9a71ceab398950b9
Gerrit-PatchSet: 6
Gerrit-Project: vdsm
Gerrit-Branch: master
Gerrit-Owner: Ondřej Svoboda <osvoboda(a)redhat.com&gt;
Gerrit-Reviewer: Dan Kenigsberg <danken(a)redhat.com&gt;
Gerrit-Reviewer: Ido Barkan <ibarkan(a)redhat.com&gt;
Gerrit-Reviewer: Jenkins CI
Gerrit-Reviewer: Ondřej Svoboda <osvoboda(a)redhat.com&gt;
Gerrit-Reviewer: Petr Horáček <phoracek(a)redhat.com&gt;
Gerrit-Reviewer: automation(a)ovirt.org
Gerrit-HasComments: No

Change in vdsm[master]: sourceroute: don't fail to remove a dynamic source route

[Dan Kenigsberg]

Dan Kenigsberg has posted comments on this change.

Change subject: sourceroute: don't fail to remove a dynamic source route
......................................................................


Patch Set 6: Code-Review+2

-- 
To view, visit https://gerrit.ovirt.org/36652
To unsubscribe, visit https://gerrit.ovirt.org/settings

Gerrit-MessageType: comment
Gerrit-Change-Id: I6e63d5ceba72be91fdab5ffd9a71ceab398950b9
Gerrit-PatchSet: 6
Gerrit-Project: vdsm
Gerrit-Branch: master
Gerrit-Owner: Ondřej Svoboda <osvoboda(a)redhat.com&gt;
Gerrit-Reviewer: Dan Kenigsberg <danken(a)redhat.com&gt;
Gerrit-Reviewer: Ido Barkan <ibarkan(a)redhat.com&gt;
Gerrit-Reviewer: Jenkins CI
Gerrit-Reviewer: Ondřej Svoboda <osvoboda(a)redhat.com&gt;
Gerrit-Reviewer: Petr Horáček <phoracek(a)redhat.com&gt;
Gerrit-Reviewer: automation(a)ovirt.org
Gerrit-HasComments: No

Change in vdsm[master]: sourceroute: don't fail to remove a dynamic source route

[Dan Kenigsberg]

Dan Kenigsberg has submitted this change and it was merged.

Change subject: sourceroute: don't fail to remove a dynamic source route
......................................................................


sourceroute: don't fail to remove a dynamic source route

There is a known race when removing source routing on ifdown where the device
going down was configured using ifcfg configurator (initscripts) and DHCP.

Routes are removed on ifdown by dhclient (as stated in dhclient-script(8),
HOOKS section) or the kernel, as ifdown-routes, a part of initscripts, claims:

  The routes are actually dropped just by setting the link down, so nothing
  needs to be done

At the same time, VDSM's dhclient-script hook (sourceRoute.sh) is called by
ifdown-eth with reason=STOP and triggers removal of source routing in VDSM's
sourceRoute thread. The thread then tries to remove routes and routing rules,
but is prone to failure if the routes are already removed as described above.

Even though it was tested that in this particular use case (removing source
routing on ifdown) it was not necessary for VDSM to try to remove routes
itself, the behaviour of other parts of the stack is not guaranteed forever.

iproute2 configurator, called by sourceRoute thread, now ignores the error
message saying the route was already gone:

  RTNETLINK answers: No such process

Previously this error would prevent the removal of routing rules, which would
persist on the system after ifdown. testSetupNetworksAddDelDhcp now passes
in all cases.

Change-Id: I6e63d5ceba72be91fdab5ffd9a71ceab398950b9
Signed-off-by: Ondřej Svoboda <osvoboda(a)redhat.com&gt;
Reviewed-on: https://gerrit.ovirt.org/36652
Continuous-Integration: Jenkins CI
Reviewed-by: Dan Kenigsberg <danken(a)redhat.com&gt;
---
M vdsm/network/configurators/iproute2.py
1 file changed, 10 insertions(+), 5 deletions(-)

Approvals:
  Ondřej Svoboda: Verified
  Jenkins CI: Passed CI tests
  Dan Kenigsberg: Looks good to me, approved



-- 
To view, visit https://gerrit.ovirt.org/36652
To unsubscribe, visit https://gerrit.ovirt.org/settings

Gerrit-MessageType: merged
Gerrit-Change-Id: I6e63d5ceba72be91fdab5ffd9a71ceab398950b9
Gerrit-PatchSet: 7
Gerrit-Project: vdsm
Gerrit-Branch: master
Gerrit-Owner: Ondřej Svoboda <osvoboda(a)redhat.com&gt;
Gerrit-Reviewer: Dan Kenigsberg <danken(a)redhat.com&gt;
Gerrit-Reviewer: Ido Barkan <ibarkan(a)redhat.com&gt;
Gerrit-Reviewer: Jenkins CI
Gerrit-Reviewer: Ondřej Svoboda <osvoboda(a)redhat.com&gt;
Gerrit-Reviewer: Petr Horáček <phoracek(a)redhat.com&gt;
Gerrit-Reviewer: automation(a)ovirt.org

Change in vdsm[master]: sourceroute: don't fail to remove a dynamic source route

automation(a)ovirt.org has posted comments on this change.

Change subject: sourceroute: don't fail to remove a dynamic source route
......................................................................


Patch Set 7:

* Update tracker::IGNORE, no Bug-Url found
* Set MODIFIED::IGNORE, no Bug-Url found.

-- 
To view, visit https://gerrit.ovirt.org/36652
To unsubscribe, visit https://gerrit.ovirt.org/settings

Gerrit-MessageType: comment
Gerrit-Change-Id: I6e63d5ceba72be91fdab5ffd9a71ceab398950b9
Gerrit-PatchSet: 7
Gerrit-Project: vdsm
Gerrit-Branch: master
Gerrit-Owner: Ondřej Svoboda <osvoboda(a)redhat.com&gt;
Gerrit-Reviewer: Dan Kenigsberg <danken(a)redhat.com&gt;
Gerrit-Reviewer: Ido Barkan <ibarkan(a)redhat.com&gt;
Gerrit-Reviewer: Jenkins CI
Gerrit-Reviewer: Ondřej Svoboda <osvoboda(a)redhat.com&gt;
Gerrit-Reviewer: Petr Horáček <phoracek(a)redhat.com&gt;
Gerrit-Reviewer: automation(a)ovirt.org
Gerrit-HasComments: No