Change in vdsm[master]: vm: migration: exponential downtime increment

Francesco Romani has uploaded a new change for review.

Change subject: vm: migration: exponential downtime increment
......................................................................

vm: migration: exponential downtime increment

Migration downtime is calculated using an interpolation
using the current downtime step (up to a configurabile
maximum) and the maximum downtime.

The downtime is incremented at each downtime step until
it reaches the maximum, or the migration is finished.

This patch changes the interpolation formula from linear
to exponential, being the new one:

d = c ** s

where d is the downtime, s is the step and c is the
exponentiation coefficient computed in such a way
of when s = S (number of steps to perform),
then we will have d = D (selected downtime).

Please note that this patch still preserve an linear
increment of the wait time.

Change-Id: I6401772f52ea28144452e67198bddff18f6703eb
Bug-Url: https://bugzilla.redhat.com/show_bug.cgi?id=1004101
Signed-off-by: Francesco Romani 
---
M vdsm/vm.py
1 file changed, 2 insertions(+), 1 deletion(-)

git pull ssh://gerrit.ovirt.org:29418/vdsm refs/changes/20/25820/1

diff --git a/vdsm/vm.py b/vdsm/vm.py
index c53f1d4..1d7b643 100644
--- a/vdsm/vm.py
+++ b/vdsm/vm.py
@@ -722,6 +722,7 @@
         delay_per_gib = config.getint('vars', 'migration_downtime_delay')
         memSize = int(vm.conf['memSize'])
         self._wait = (delay_per_gib * max(memSize, 2048) + 1023) / 1024
+        self._coeff = self._downtime ** (1 / float(self.DOWNTIME_STEPS))

self.daemon = True
         self.start()
@@ -735,7 +736,7 @@
             if self._stop.isSet():
                 break

-            downtime = self._downtime * (i + 1) / self.DOWNTIME_STEPS
+            downtime = int(self._coeff ** i)
             self._vm.log.debug('setting migration downtime to %d', downtime)
             self._vm._dom.migrateSetMaxDowntime(downtime, 0)

[Jenkins CI RO]

oVirt Jenkins CI Server has posted comments on this change.

Change subject: vm: migration: exponential downtime increment
......................................................................

Patch Set 1:

Build Successful

http://jenkins.ovirt.org/job/vdsm_unit_tests_gerrit_el/6692/ : SUCCESS

http://jenkins.ovirt.org/job/vdsm_master_pep8_gerrit/7483/ : SUCCESS

http://jenkins.ovirt.org/job/vdsm_unit_tests_gerrit/7593/ : SUCCESS

Michal Skrivanek has posted comments on this change.

Change subject: vm: migration: exponential downtime increment
......................................................................

Patch Set 1:

I wonder if we can factor in the convergence/progress of the migration as well. E.g. for a case when there's a lot to transfer but no trashing we actually would be able to finish migration with low downtime, but because of the time it takes to transfer the initial amount of data we go through several downtime_steps.

Something like having a threshold like "when between two progress updates the change is >10%, don't change the downtime"

Francesco Romani has posted comments on this change.

Change subject: vm: migration: exponential downtime increment
......................................................................

Patch Set 1:

I like the concept, I'll give it a try. At first glance it should be feasible with a few more lines.

Francesco Romani has posted comments on this change.

Change subject: vm: migration: exponential downtime increment
......................................................................

Patch Set 2:

Rebased

[Jenkins CI RO]

oVirt Jenkins CI Server has posted comments on this change.

Change subject: vm: migration: exponential downtime increment
......................................................................

Patch Set 2:

Build Successful

http://jenkins.ovirt.org/job/vdsm_master_pep8_gerrit/7931/ : SUCCESS

http://jenkins.ovirt.org/job/vdsm_unit_tests_gerrit_el/7141/ : SUCCESS

http://jenkins.ovirt.org/job/vdsm_unit_tests_gerrit/8043/ : SUCCESS

Itamar Heim has posted comments on this change.

Change subject: vm: migration: exponential downtime increment
......................................................................

Patch Set 2:

ping

Francesco Romani has posted comments on this change.

Change subject: vm: migration: exponential downtime increment
......................................................................

Patch Set 3:

rebased

[Jenkins CI RO]

oVirt Jenkins CI Server has posted comments on this change.

Change subject: vm: migration: exponential downtime increment
......................................................................

Patch Set 3:

Build Successful

http://jenkins.ovirt.org/job/vdsm_master_unit_tests_gerrit_el/8889/ : SUCCESS

http://jenkins.ovirt.org/job/vdsm_master_pep8_gerrit/9673/ : SUCCESS

http://jenkins.ovirt.org/job/vdsm_master_virt_functional_tests_gerrit/669/ : SUCCESS

http://jenkins.ovirt.org/job/vdsm_master_unit_tests_gerrit/9828/ : SUCCESS

Itamar Heim has posted comments on this change.

Change subject: vm: migration: exponential downtime increment
......................................................................

Patch Set 3:

ping

Michal Skrivanek has posted comments on this change.

Change subject: vm: migration: exponential downtime increment
......................................................................

Patch Set 3: Code-Review+1

I think this would help in all cases and so we should change the default behavior this way

Francesco Romani has posted comments on this change.

Change subject: vm: migration: exponential downtime increment
......................................................................

Patch Set 3: Verified+1

Verified doing migration among two hosts: one with vanilla VDSM, one with patched VDSM. Inspected the downtime progression though logs:

Old progression:
Thread-71::DEBUG::2014-08-29 15:59:56,982::migration::360::vm.Vm::(run) vmId=`56d1c657-dd76-4609-a207-c050699be5be`::migration downtime thread started
Thread-70::DEBUG::2014-08-29 16:00:20,905::migration::375::vm.Vm::(cancel) vmId=`56d1c657-dd76-4609-a207-c050699be5be`::canceling migration downtime thread
Thread-71::DEBUG::2014-08-29 16:00:20,906::migration::372::vm.Vm::(run) vmId=`56d1c657-dd76-4609-a207-c050699be5be`::migration downtime thread exiting
Thread-97::DEBUG::2014-08-29 16:24:07,093::migration::360::vm.Vm::(run) vmId=`56d1c657-dd76-4609-a207-c050699be5be`::migration downtime thread started
Thread-97::DEBUG::2014-08-29 16:24:37,094::migration::369::vm.Vm::(run) vmId=`56d1c657-dd76-4609-a207-c050699be5be`::setting migration downtime to 50
Thread-97::DEBUG::2014-08-29 16:25:07,097::migration::369::vm.Vm::(run) vmId=`56d1c657-dd76-4609-a207-c050699be5be`::setting migration downtime to 100
Thread-97::DEBUG::2014-08-29 16:25:37,100::migration::369::vm.Vm::(run) vmId=`56d1c657-dd76-4609-a207-c050699be5be`::setting migration downtime to 150
Thread-97::DEBUG::2014-08-29 16:26:07,103::migration::369::vm.Vm::(run) vmId=`56d1c657-dd76-4609-a207-c050699be5be`::setting migration downtime to 200
Thread-97::DEBUG::2014-08-29 16:26:37,106::migration::369::vm.Vm::(run) vmId=`56d1c657-dd76-4609-a207-c050699be5be`::setting migration downtime to 250
Thread-97::DEBUG::2014-08-29 16:27:07,109::migration::369::vm.Vm::(run) vmId=`56d1c657-dd76-4609-a207-c050699be5be`::setting migration downtime to 300
Thread-97::DEBUG::2014-08-29 16:27:37,112::migration::369::vm.Vm::(run) vmId=`56d1c657-dd76-4609-a207-c050699be5be`::setting migration downtime to 350
Thread-97::DEBUG::2014-08-29 16:28:07,116::migration::369::vm.Vm::(run) vmId=`56d1c657-dd76-4609-a207-c050699be5be`::setting migration downtime to 400
Thread-96::DEBUG::2014-08-29 16:28:27,684::migration::375::vm.Vm::(cancel) vmId=`56d1c657-dd76-4609-a207-c050699be5be`::canceling migration downtime thread
Thread-97::DEBUG::2014-08-29 16:28:27,684::migration::372::vm.Vm::(run) vmId=`56d1c657-dd76-4609-a207-c050699be5be`::migration downtime thread exiting

New progression:
Thread-36::DEBUG::2014-08-29 16:19:10,845::migration::375::vm.Vm::(run) vmId=`56d1c657-dd76-4609-a207-c050699be5be`::migration downtime thread started
Thread-36::DEBUG::2014-08-29 16:19:40,846::migration::384::vm.Vm::(run) vmId=`56d1c657-dd76-4609-a207-c050699be5be`::setting migration downtime to 1
Thread-36::DEBUG::2014-08-29 16:20:10,858::migration::384::vm.Vm::(run) vmId=`56d1c657-dd76-4609-a207-c050699be5be`::setting migration downtime to 1
Thread-36::DEBUG::2014-08-29 16:20:40,859::migration::384::vm.Vm::(run) vmId=`56d1c657-dd76-4609-a207-c050699be5be`::setting migration downtime to 3
Thread-36::DEBUG::2014-08-29 16:21:10,861::migration::384::vm.Vm::(run) vmId=`56d1c657-dd76-4609-a207-c050699be5be`::setting migration downtime to 6
Thread-36::DEBUG::2014-08-29 16:21:40,863::migration::384::vm.Vm::(run) vmId=`56d1c657-dd76-4609-a207-c050699be5be`::setting migration downtime to 12
Thread-36::DEBUG::2014-08-29 16:22:10,865::migration::384::vm.Vm::(run) vmId=`56d1c657-dd76-4609-a207-c050699be5be`::setting migration downtime to 22
Thread-36::DEBUG::2014-08-29 16:22:40,891::migration::384::vm.Vm::(run) vmId=`56d1c657-dd76-4609-a207-c050699be5be`::setting migration downtime to 41
Thread-36::DEBUG::2014-08-29 16:23:10,908::migration::384::vm.Vm::(run) vmId=`56d1c657-dd76-4609-a207-c050699be5be`::setting migration downtime to 77
Thread-35::DEBUG::2014-08-29 16:23:33,317::migration::390::vm.Vm::(cancel) vmId=`56d1c657-dd76-4609-a207-c050699be5be`::canceling migration downtime thread
Thread-36::DEBUG::2014-08-29 16:23:33,318::migration::387::vm.Vm::(run) vmId=`56d1c657-dd76-4609-a207-c050699be5be`::migration downtime thread exiting

Francesco Romani has posted comments on this change.

Change subject: vm: migration: exponential downtime increment
......................................................................

Patch Set 3:

another simpler migration:

old algorhythm
Thread-101::DEBUG::2014-08-29 16:29:11,355::migration::360::vm.Vm::(run) vmId=`56d1c657-dd76-4609-a207-c050699be5be`::migration downtime thread started
Thread-101::DEBUG::2014-08-29 16:29:41,356::migration::369::vm.Vm::(run) vmId=`56d1c657-dd76-4609-a207-c050699be5be`::setting migration downtime to 50
Thread-101::DEBUG::2014-08-29 16:30:11,358::migration::369::vm.Vm::(run) vmId=`56d1c657-dd76-4609-a207-c050699be5be`::setting migration downtime to 100
Thread-100::DEBUG::2014-08-29 16:30:29,243::migration::375::vm.Vm::(cancel) vmId=`56d1c657-dd76-4609-a207-c050699be5be`::canceling migration downtime thread
Thread-101::DEBUG::2014-08-29 16:30:29,244::migration::372::vm.Vm::(run) vmId=`56d1c657-dd76-4609-a207-c050699be5be`::migration downtime thread exiting

new one:
Thread-56::DEBUG::2014-08-29 16:30:40,284::migration::375::vm.Vm::(run) vmId=`56d1c657-dd76-4609-a207-c050699be5be`::migration downtime thread started
Thread-56::DEBUG::2014-08-29 16:31:10,287::migration::384::vm.Vm::(run) vmId=`56d1c657-dd76-4609-a207-c050699be5be`::setting migration downtime to 1
Thread-56::DEBUG::2014-08-29 16:31:40,306::migration::384::vm.Vm::(run) vmId=`56d1c657-dd76-4609-a207-c050699be5be`::setting migration downtime to 1
Thread-55::DEBUG::2014-08-29 16:31:54,824::migration::390::vm.Vm::(cancel) vmId=`56d1c657-dd76-4609-a207-c050699be5be`::canceling migration downtime thread
Thread-56::DEBUG::2014-08-29 16:31:54,827::migration::387::vm.Vm::(run) vmId=`56d1c657-dd76-4609-a207-c050699be5be`::migration downtime thread exiting

Francesco Romani has posted comments on this change.

Change subject: vm: migration: exponential downtime increment
......................................................................

Patch Set 3:

Formatting got completely lost. Let's retry.

First migration, longish.

Old sequence: 50,100,150,200,250,300,350,400
  New sequence:  1,   1, 3,  6, 12, 22, 41, 77

Second migration, short and simple:

Old sequence: 50,100
  New sequence:  1,  1

[Dan Kenigsberg]

Dan Kenigsberg has posted comments on this change.

Change subject: vm: migration: exponential downtime increment
......................................................................

Patch Set 3: Code-Review-1

(3 comments)

http://gerrit.ovirt.org/#/c/25820/3//COMMIT_MSG
Commit Message:

Line 18: 
Line 19: d = c ** s
Line 20: 
Line 21: where d is the downtime, s is the step and c is the
Line 22: exponentiation coefficient computed in such a way
sorry for my puritanism, but in mathematics, "c" is called the *base* of the exponentiation. "Coefficient" implies multiplication.
Line 23: of when s = S (number of steps to perform),
Line 24: then we will have d = D (selected downtime).
Line 25: 
Line 26: Please note that this patch still preserve an linear

Line 23: of when s = S (number of steps to perform),
Line 24: then we will have d = D (selected downtime).
Line 25: 
Line 26: Please note that this patch still preserve an linear
Line 27: increment of the wait time.
Could you explain why a exponential growth is better than linear?
Line 28: 
Line 29: Change-Id: I6401772f52ea28144452e67198bddff18f6703eb
Line 30: Bug-Url: https://bugzilla.redhat.com/show_bug.cgi?id=1004101

Line 26: Please note that this patch still preserve an linear
Line 27: increment of the wait time.
Line 28: 
Line 29: Change-Id: I6401772f52ea28144452e67198bddff18f6703eb
Line 30: Bug-Url: https://bugzilla.redhat.com/show_bug.cgi?id=1004101
Why is that bug secret? Please make it public, or better explain the need for this patch.

Francesco Romani has posted comments on this change.

Change subject: vm: migration: exponential downtime increment
......................................................................

Patch Set 3:

(3 comments)

http://gerrit.ovirt.org/#/c/25820/3//COMMIT_MSG
Commit Message:

Line 18: 
Line 19: d = c ** s
Line 20: 
Line 21: where d is the downtime, s is the step and c is the
Line 22: exponentiation coefficient computed in such a way
> sorry for my puritanism, but in mathematics, "c" is called the *base* of th

Your puritanism is very welcome. Will fix.
Line 23: of when s = S (number of steps to perform),
Line 24: then we will have d = D (selected downtime).
Line 25: 
Line 26: Please note that this patch still preserve an linear

Line 23: of when s = S (number of steps to perform),
Line 24: then we will have d = D (selected downtime).
Line 25: 
Line 26: Please note that this patch still preserve an linear
Line 27: increment of the wait time.
> Could you explain why a exponential growth is better than linear?

Sure, will amend the commit message.
Line 28: 
Line 29: Change-Id: I6401772f52ea28144452e67198bddff18f6703eb
Line 30: Bug-Url: https://bugzilla.redhat.com/show_bug.cgi?id=1004101

Line 26: Please note that this patch still preserve an linear
Line 27: increment of the wait time.
Line 28: 
Line 29: Change-Id: I6401772f52ea28144452e67198bddff18f6703eb
Line 30: Bug-Url: https://bugzilla.redhat.com/show_bug.cgi?id=1004101
> Why is that bug secret? Please make it public, or better explain the need f

I'll start with better explanation.

Francesco Romani has posted comments on this change.

Change subject: vm: migration: exponential downtime increment
......................................................................

Patch Set 4: Verified+1

Improved commit message and rebased.
No code changes, copied score.

[Jenkins CI RO]

oVirt Jenkins CI Server has posted comments on this change.

Change subject: vm: migration: exponential downtime increment
......................................................................

Patch Set 4:

Build Failed

http://jenkins.ovirt.org/job/vdsm_master_unit_tests_gerrit_el/11224/ : SUCCESS

http://jenkins.ovirt.org/job/vdsm_master_unit-tests_created/12166/ : SUCCESS

http://jenkins.ovirt.org/job/vdsm_master_pep8_gerrit/12013/ : SUCCESS

http://jenkins.ovirt.org/job/vdsm_master_virt_functional_tests_gerrit/1592/ : There was an infra issue, please contact [e-mail]

[Dan Kenigsberg]

Dan Kenigsberg has posted comments on this change.

Change subject: vm: migration: exponential downtime increment
......................................................................

Patch Set 4: Code-Review-1

(3 comments)

http://gerrit.ovirt.org/#/c/25820/4//COMMIT_MSG
Commit Message:

Line 26: The rationale for this patch is to minimize the
Line 27: downtime of VMs pushing for low downtime values as much
Line 28: and as long as possible.
Line 29: This way, the downtime becomes more tightly related
Line 30: to the duration of the migration itself.
please explain better why waiting for 11 seconds after waiting for 10 seconds is not very smart.
Line 31: 
Line 32: Please note that this patch still preserve an linear
Line 33: increment of the wait time.
Line 34:

http://gerrit.ovirt.org/#/c/25820/4/vdsm/virt/migration.py
File vdsm/virt/migration.py:

Line 365: 
Line 366:         delay_per_gib = config.getint('vars', 'migration_downtime_delay')
Line 367:         memSize = int(vm.conf['memSize'])
Line 368:         self._wait = (delay_per_gib * max(memSize, 2048) + 1023) / 1024
Line 369:         self._coeff = self._downtime ** (1 / float(self.DOWNTIME_STEPS))
_coeff -> _base
Line 370: 
Line 371:         self.daemon = True
Line 372:         self.start()
Line 373:

Line 379: 
Line 380:             if self._stop.isSet():
Line 381:                 break
Line 382: 
Line 383:             downtime = int(self._coeff ** i)
make sure we don't begin with 0.
Line 384:             self._vm.log.debug('setting migration downtime to %d', downtime)
Line 385:             self._vm._dom.migrateSetMaxDowntime(downtime, 0)
Line 386: 
Line 387:         self._vm.log.debug('migration downtime thread exiting')

Francesco Romani has posted comments on this change.

Change subject: vm: migration: exponential downtime increment
......................................................................

Patch Set 4: -Verified

(3 comments)

http://gerrit.ovirt.org/#/c/25820/4//COMMIT_MSG
Commit Message:

Line 26: The rationale for this patch is to minimize the
Line 27: downtime of VMs pushing for low downtime values as much
Line 28: and as long as possible.
Line 29: This way, the downtime becomes more tightly related
Line 30: to the duration of the migration itself.
> please explain better why waiting for 11 seconds after waiting for 10 secon

I'll quote David words, since his explanation is alrady pretty good.
Line 31: 
Line 32: Please note that this patch still preserve an linear
Line 33: increment of the wait time.
Line 34:

http://gerrit.ovirt.org/#/c/25820/4/vdsm/virt/migration.py
File vdsm/virt/migration.py:

Line 365: 
Line 366:         delay_per_gib = config.getint('vars', 'migration_downtime_delay')
Line 367:         memSize = int(vm.conf['memSize'])
Line 368:         self._wait = (delay_per_gib * max(memSize, 2048) + 1023) / 1024
Line 369:         self._coeff = self._downtime ** (1 / float(self.DOWNTIME_STEPS))
> _coeff -> _base

Done
Line 370: 
Line 371:         self.daemon = True
Line 372:         self.start()
Line 373:

Line 379: 
Line 380:             if self._stop.isSet():
Line 381:                 break
Line 382: 
Line 383:             downtime = int(self._coeff ** i)
> make sure we don't begin with 0.

Fixed
Line 384:             self._vm.log.debug('setting migration downtime to %d', downtime)
Line 385:             self._vm._dom.migrateSetMaxDowntime(downtime, 0)
Line 386: 
Line 387:         self._vm.log.debug('migration downtime thread exiting')

[Jenkins CI RO]

oVirt Jenkins CI Server has posted comments on this change.

Change subject: vm: migration: exponential downtime increment
......................................................................

Patch Set 5:

Build Failed

http://jenkins.ovirt.org/job/vdsm_master_unit-tests_created/12360/ : SUCCESS

http://jenkins.ovirt.org/job/vdsm_master_unit_tests_gerrit_el/11416/ : SUCCESS

http://jenkins.ovirt.org/job/vdsm_master_virt_functional_tests_gerrit/1643/ : There was an infra issue, please contact [e-mail]

http://jenkins.ovirt.org/job/vdsm_master_pep8_gerrit/12206/ : SUCCESS

[Jenkins CI RO]

oVirt Jenkins CI Server has posted comments on this change.

Change subject: vm: migration: exponential downtime increment
......................................................................

Patch Set 6:

Build Failed

http://jenkins.ovirt.org/job/vdsm_master_unit-tests_created/12365/ : SUCCESS

http://jenkins.ovirt.org/job/vdsm_master_unit_tests_gerrit_el/11421/ : SUCCESS

http://jenkins.ovirt.org/job/vdsm_master_virt_functional_tests_gerrit/1645/ : There was an infra issue, please contact [e-mail]

http://jenkins.ovirt.org/job/vdsm_master_pep8_gerrit/12210/ : SUCCESS

[Dan Kenigsberg]

Dan Kenigsberg has posted comments on this change.

Change subject: vm: migration: exponential downtime increment
......................................................................

Patch Set 6: Code-Review-1

(1 comment)

http://gerrit.ovirt.org/#/c/25820/6/vdsm/virt/migration.py
File vdsm/virt/migration.py:

Line 353:             if not self._preparingMigrationEvt:
Line 354:                     raise
Line 355: 
Line 356: 
Line 357: def exponential_downtime(downtime, steps):
would you add a unit test for this function?
> > > list(exponential_downtime(500, 10))

[451, 451, 452, 453, 454, 457, 460, 465, 472, 483]

makes very little sense to me.
Line 358:     offset = downtime - (downtime / steps)
Line 359:     base = (downtime - offset) ** (1 / float(steps))
Line 360: 
Line 361:     for i in range(steps):

Francesco Romani has posted comments on this change.

Change subject: vm: migration: exponential downtime increment
......................................................................

Patch Set 6:

(1 comment)

http://gerrit.ovirt.org/#/c/25820/6/vdsm/virt/migration.py
File vdsm/virt/migration.py:

Line 353:             if not self._preparingMigrationEvt:
Line 354:                     raise
Line 355: 
Line 356: 
Line 357: def exponential_downtime(downtime, steps):
> would you add a unit test for this function?

Because you're right, and I had a bug in my formula. Better to quickly hide this from my calculus professor :)
Line 358:     offset = downtime - (downtime / steps)
Line 359:     base = (downtime - offset) ** (1 / float(steps))
Line 360: 
Line 361:     for i in range(steps):

[Jenkins CI RO]

oVirt Jenkins CI Server has posted comments on this change.

Change subject: vm: migration: exponential downtime increment
......................................................................

Patch Set 7:

Build Failed

http://jenkins.ovirt.org/job/vdsm_master_unit-tests_created/12372/ : SUCCESS

http://jenkins.ovirt.org/job/vdsm_master_unit_tests_gerrit_el/11428/ : FAILURE

http://jenkins.ovirt.org/job/vdsm_master_virt_functional_tests_gerrit/1647/ : There was an infra issue, please contact [e-mail]

http://jenkins.ovirt.org/job/vdsm_master_pep8_gerrit/12217/ : SUCCESS

Francesco Romani has posted comments on this change.

Change subject: vm: migration: exponential downtime increment
......................................................................

Patch Set 7: Verified+1

Re-verified running a migration in the following scenario:

- patched VDSM
- set vars.migration_max_time_per_gib_mem = 0
- runned 'stress -v -m X' inside the VM to have memory stress
- performed migration

expected output:

GENji> 11:38:59 root [~]$ grep -i Downtime /var/log/vdsm//vdsm.log
Thread-39::DEBUG::2014-09-10 11:34:15,209::migration::383::vm.Vm::(run) vmId=`56d1c657-dd76-4609-a207-c050699be5be`::migration downtime thread started
Thread-39::DEBUG::2014-09-10 11:34:45,210::migration::392::vm.Vm::(run) vmId=`56d1c657-dd76-4609-a207-c050699be5be`::setting migration downtime to 51
Thread-39::DEBUG::2014-09-10 11:35:15,213::migration::392::vm.Vm::(run) vmId=`56d1c657-dd76-4609-a207-c050699be5be`::setting migration downtime to 53
Thread-39::DEBUG::2014-09-10 11:35:45,216::migration::392::vm.Vm::(run) vmId=`56d1c657-dd76-4609-a207-c050699be5be`::setting migration downtime to 56
Thread-39::DEBUG::2014-09-10 11:36:15,219::migration::392::vm.Vm::(run) vmId=`56d1c657-dd76-4609-a207-c050699be5be`::setting migration downtime to 61
Thread-39::DEBUG::2014-09-10 11:36:45,221::migration::392::vm.Vm::(run) vmId=`56d1c657-dd76-4609-a207-c050699be5be`::setting migration downtime to 71
Thread-39::DEBUG::2014-09-10 11:37:15,224::migration::392::vm.Vm::(run) vmId=`56d1c657-dd76-4609-a207-c050699be5be`::setting migration downtime to 89
Thread-39::DEBUG::2014-09-10 11:37:45,227::migration::392::vm.Vm::(run) vmId=`56d1c657-dd76-4609-a207-c050699be5be`::setting migration downtime to 121
Thread-39::DEBUG::2014-09-10 11:38:15,230::migration::392::vm.Vm::(run) vmId=`56d1c657-dd76-4609-a207-c050699be5be`::setting migration downtime to 182
Thread-39::DEBUG::2014-09-10 11:38:45,232::migration::392::vm.Vm::(run) vmId=`56d1c657-dd76-4609-a207-c050699be5be`::setting migration downtime to 294
Thread-39::DEBUG::2014-09-10 11:39:15,235::migration::392::vm.Vm::(run) vmId=`56d1c657-dd76-4609-a207-c050699be5be`::setting migration downtime to 499
Thread-39::DEBUG::2014-09-10 11:39:15,238::migration::395::vm.Vm::(run) vmId=`56d1c657-dd76-4609-a207-c050699be5be`::migration downtime thread exiting

- migration eventually was detected as stuck and aborted, but that's OK here.

[Martin Polednik]

Martin Polednik has posted comments on this change.

Change subject: vm: migration: exponential downtime increment
......................................................................

Patch Set 7: Code-Review+1

(1 comment)

Seems correct to me, +1 for patches (to think: run tests against multiple sets of (downtime, steps) in order to be really sure the properties hold still or support a proof ;))

http://gerrit.ovirt.org/#/c/25820/7/vdsm/virt/migration.py
File vdsm/virt/migration.py:

Line 358:     offset = downtime / steps
Line 359:     base = (downtime - offset) ** (1 / float(steps))
Line 360: 
Line 361:     for i in range(1, steps+1):
Line 362:         yield offset + int(base ** i)
Minor: I'd go with yield int(offset + base ** i) in order to guarantee the return value being int even if downtime ∈ ℝ(really minor imho, I'd guess that might destroy something else)
Line 363: 
Line 364: 
Line 365: class DowntimeThread(threading.Thread):
Line 366:     DOWNTIME_STEPS = config.getint('vars', 'migration_downtime_steps')

[Martin Polednik]

Martin Polednik has posted comments on this change.

Change subject: vm: migration: exponential downtime increment
......................................................................

Patch Set 7:

(1 comment)

http://gerrit.ovirt.org/#/c/25820/7/tests/vmMigrationTests.py
File tests/vmMigrationTests.py:

Line 33:     return izip(a, b)
Line 34: 
Line 35: 
Line 36: def _linear_downtime(downtime, steps):
Line 37:     for i in range(steps):
maybe range(1, steps + 1) and later downtime * i / steps for consistence with exponential downtime (also allows easier plotting, for example "plot (d / s) + ((d - d / s) ^ (1 / s)) ^ k and d * k / s for (d = 500, s = 10), (k from 1 to 10)" in wolfram alpha)
Line 38:         yield downtime * (i + 1) / steps
Line 39: 
Line 40: 
Line 41: class TestVmMigrationDowntimeSequence(TestCaseBase):

[Dan Kenigsberg]

Dan Kenigsberg has posted comments on this change.

Change subject: vm: migration: exponential downtime increment
......................................................................

Patch Set 7: Code-Review-1

(6 comments)

http://gerrit.ovirt.org/#/c/25820/7/tests/vmMigrationTests.py
File tests/vmMigrationTests.py:

Line 38:         yield downtime * (i + 1) / steps
Line 39: 
Line 40: 
Line 41: class TestVmMigrationDowntimeSequence(TestCaseBase):
Line 42:     # defaults as per config
read them from config, instead of having this comment and duplicating the value.
Line 43:     _DOWNTIME = 500
Line 44: 
Line 45:     _STEPS = 10
Line 46:

Line 61:         self.assertTrue(len(self._default) >= 2)
Line 62: 
Line 63:     def test_downtime_increasing(self):
Line 64:         for a, b in pairwise(self._default):
Line 65:             self.assertTrue(a <= b)
assertLessEqual() etc gives more info upon failure - please use more specific asserts when possible.
Line 66: 
Line 67:     def test_exponential_dowtime_never_zero(self):
Line 68:         for dt in self._default:
Line 69:             self.assertTrue(dt > 0)

Line 78:             self.assertTrue(a <= b)
Line 79: 
Line 80:     def test_exponential_same_end_value(self):
Line 81:         self.assertAlmostEqual(self._default[-1], self._linear[-1],
Line 82:                                delta=self._DELTA)
You'd also want to assert that the final step is the maximum allow downtime.

http://gerrit.ovirt.org/#/c/25820/7/vdsm/virt/migration.py
File vdsm/virt/migration.py:

Line 354:                     raise
Line 355: 
Line 356: 
Line 357: def exponential_downtime(downtime, steps):
Line 358:     offset = downtime / steps
Let's do a float division on Python2, too.
Line 359:     base = (downtime - offset) ** (1 / float(steps))
Line 360: 
Line 361:     for i in range(1, steps+1):
Line 362:         yield offset + int(base ** i)

Line 357: def exponential_downtime(downtime, steps):
Line 358:     offset = downtime / steps
Line 359:     base = (downtime - offset) ** (1 / float(steps))
Line 360: 
Line 361:     for i in range(1, steps+1):
use spaces around the + sign.
Line 362:         yield offset + int(base ** i)
Line 363: 
Line 364: 
Line 365: class DowntimeThread(threading.Thread):

Line 358:     offset = downtime / steps
Line 359:     base = (downtime - offset) ** (1 / float(steps))
Line 360: 
Line 361:     for i in range(1, steps+1):
Line 362:         yield offset + int(base ** i)
> Minor: I'd go with yield int(offset + base ** i) in order to guarantee the

+1, particularly since on Python 3, (downtime / steps) may be in ℝ∖ℕ, anyway.
Line 363: 
Line 364: 
Line 365: class DowntimeThread(threading.Thread):
Line 366:     DOWNTIME_STEPS = config.getint('vars', 'migration_downtime_steps')

Francesco Romani has posted comments on this change.

Change subject: vm: migration: exponential downtime increment
......................................................................

Patch Set 7: -Verified

(7 comments)

-Verified because needs more work.

http://gerrit.ovirt.org/#/c/25820/7/tests/vmMigrationTests.py
File tests/vmMigrationTests.py:

Line 33:     return izip(a, b)
Line 34: 
Line 35: 
Line 36: def _linear_downtime(downtime, steps):
Line 37:     for i in range(steps):
> maybe range(1, steps + 1) and later downtime * i / steps for consistence wi

Good point, will do.
Line 38:         yield downtime * (i + 1) / steps
Line 39: 
Line 40: 
Line 41: class TestVmMigrationDowntimeSequence(TestCaseBase):

Line 38:         yield downtime * (i + 1) / steps
Line 39: 
Line 40: 
Line 41: class TestVmMigrationDowntimeSequence(TestCaseBase):
Line 42:     # defaults as per config
> read them from config, instead of having this comment and duplicating the v

Sure, will fix.
Line 43:     _DOWNTIME = 500
Line 44: 
Line 45:     _STEPS = 10
Line 46:

Line 61:         self.assertTrue(len(self._default) >= 2)
Line 62: 
Line 63:     def test_downtime_increasing(self):
Line 64:         for a, b in pairwise(self._default):
Line 65:             self.assertTrue(a <= b)
> assertLessEqual() etc gives more info upon failure - please use more specif

I'd love to, but

https://docs.python.org/2/library/unittest.html#unittest.TestCase.assertLess...
-> new in version 2.7

And AFAIK it wasn't backported (git grep confirms)
Line 66: 
Line 67:     def test_exponential_dowtime_never_zero(self):
Line 68:         for dt in self._default:
Line 69:             self.assertTrue(dt > 0)

Line 78:             self.assertTrue(a <= b)
Line 79: 
Line 80:     def test_exponential_same_end_value(self):
Line 81:         self.assertAlmostEqual(self._default[-1], self._linear[-1],
Line 82:                                delta=self._DELTA)
> You'd also want to assert that the final step is the maximum allow downtime

right, I'll add.

http://gerrit.ovirt.org/#/c/25820/7/vdsm/virt/migration.py
File vdsm/virt/migration.py:

Line 354:                     raise
Line 355: 
Line 356: 
Line 357: def exponential_downtime(downtime, steps):
Line 358:     offset = downtime / steps
> Let's do a float division on Python2, too.

will fix
Line 359:     base = (downtime - offset) ** (1 / float(steps))
Line 360: 
Line 361:     for i in range(1, steps+1):
Line 362:         yield offset + int(base ** i)

Line 357: def exponential_downtime(downtime, steps):
Line 358:     offset = downtime / steps
Line 359:     base = (downtime - offset) ** (1 / float(steps))
Line 360: 
Line 361:     for i in range(1, steps+1):
> use spaces around the + sign.

oops, will fix.
Line 362:         yield offset + int(base ** i)
Line 363: 
Line 364: 
Line 365: class DowntimeThread(threading.Thread):

Line 358:     offset = downtime / steps
Line 359:     base = (downtime - offset) ** (1 / float(steps))
Line 360: 
Line 361:     for i in range(1, steps+1):
Line 362:         yield offset + int(base ** i)
> +1, particularly since on Python 3, (downtime / steps) may be in ℝ∖ℕ, anywa

Good point, that's exactly the comment I was looking for :)
Thanks!
Line 363: 
Line 364: 
Line 365: class DowntimeThread(threading.Thread):
Line 366:     DOWNTIME_STEPS = config.getint('vars', 'migration_downtime_steps')

Francesco Romani has posted comments on this change.

Change subject: vm: migration: exponential downtime increment
......................................................................

Patch Set 7:

(1 comment)

http://gerrit.ovirt.org/#/c/25820/7/tests/vmMigrationTests.py
File tests/vmMigrationTests.py:

Line 33:     return izip(a, b)
Line 34: 
Line 35: 
Line 36: def _linear_downtime(downtime, steps):
Line 37:     for i in range(steps):
> Good point, will do.

Sorry, I overlooked. This code needs to as the old one as reference, so I'm keeping it for this reason.
Line 38:         yield downtime * (i + 1) / steps
Line 39: 
Line 40: 
Line 41: class TestVmMigrationDowntimeSequence(TestCaseBase):

[Martin Polednik]

Martin Polednik has posted comments on this change.

Change subject: vm: migration: exponential downtime increment
......................................................................

Patch Set 7:

(1 comment)

http://gerrit.ovirt.org/#/c/25820/7/tests/vmMigrationTests.py
File tests/vmMigrationTests.py:

Line 33:     return izip(a, b)
Line 34: 
Line 35: 
Line 36: def _linear_downtime(downtime, steps):
Line 37:     for i in range(steps):
> Sorry, I overlooked. This code needs to as the old one as reference, so I'm

what about leaving a comment that these are equivalent? It's not hard to understand *now*, but later it might bring questions
Line 38:         yield downtime * (i + 1) / steps
Line 39: 
Line 40: 
Line 41: class TestVmMigrationDowntimeSequence(TestCaseBase):

Francesco Romani has posted comments on this change.

Change subject: vm: migration: exponential downtime increment
......................................................................

Patch Set 7:

(1 comment)

http://gerrit.ovirt.org/#/c/25820/7/tests/vmMigrationTests.py
File tests/vmMigrationTests.py:

Line 33:     return izip(a, b)
Line 34: 
Line 35: 
Line 36: def _linear_downtime(downtime, steps):
Line 37:     for i in range(steps):
> what about leaving a comment that these are equivalent? It's not hard to un

Done!
Line 38:         yield downtime * (i + 1) / steps
Line 39: 
Line 40: 
Line 41: class TestVmMigrationDowntimeSequence(TestCaseBase):

Francesco Romani has posted comments on this change.

Change subject: vm: migration: exponential downtime increment
......................................................................

Patch Set 8:

improved the tests, most notably adding permutations
smaller tuning to the exponential function after the testing

[Jenkins CI RO]

oVirt Jenkins CI Server has posted comments on this change.

Change subject: vm: migration: exponential downtime increment
......................................................................

Patch Set 8:

Build Failed

http://jenkins.ovirt.org/job/vdsm_master_unit_tests_gerrit_el/11535/ : FAILURE

http://jenkins.ovirt.org/job/vdsm_master_unit-tests_created/12479/ : SUCCESS

http://jenkins.ovirt.org/job/vdsm_master_virt_functional_tests_gerrit/1671/ : There was an infra issue, please contact [e-mail]

http://jenkins.ovirt.org/job/vdsm_master_pep8_gerrit/12324/ : SUCCESS

[Martin Polednik]

Martin Polednik has posted comments on this change.

Change subject: vm: migration: exponential downtime increment
......................................................................

Patch Set 8: Code-Review-1

(5 comments)

-1 for visibility, other than the minor things it seems to be OK (+ need answer for the steps >= 2 question)

http://gerrit.ovirt.org/#/c/25820/8/tests/vmMigrationTests.py
File tests/vmMigrationTests.py:

Line 46: class TestVmMigrationDowntimeSequence(TestCaseBase):
Line 47: 
Line 48:     @permutations(_PARAMS)
Line 49:     def test_downtime_is_sequence(self, dtime, steps):
Line 50:         self.assertTrue(len(self._default(dtime, steps)) >= 2)
can be rewritten as self.assertGreaterEqual(len(self._default(dtime, steps)), 2)

another thing, is steps really >= 2 or do we allow single step?
Line 51: 
Line 52:     @permutations(_PARAMS)
Line 53:     def test_downtime_increasing(self, dtime, steps):
Line 54:         for a, b in pairwise(self._default(dtime, steps)):

Line 51: 
Line 52:     @permutations(_PARAMS)
Line 53:     def test_downtime_increasing(self, dtime, steps):
Line 54:         for a, b in pairwise(self._default(dtime, steps)):
Line 55:             self.assertTrue(a <= b)
can be rewritten as self.assertGreater(b, a)
Line 56: 
Line 57:     @permutations(_PARAMS)
Line 58:     def test_exponential_dowtime_never_zero(self, dtime, steps):
Line 59:         for dt in self._default(dtime, steps):

Line 56: 
Line 57:     @permutations(_PARAMS)
Line 58:     def test_exponential_dowtime_never_zero(self, dtime, steps):
Line 59:         for dt in self._default(dtime, steps):
Line 60:             self.assertTrue(dt > 0)
can be rewritten as self.assertGreater(dt, 0)
Line 61: 
Line 62:     @permutations(_PARAMS)
Line 63:     def test_exponential_downtime_is_lower(self, dtime, steps):
Line 64:         # it's OK if exponential starts a little higher than linear...

Line 69: 
Line 70:         # ...but what matters is that after that, it stays lower.
Line 71:         for i, (a, b) in enumerate(zip(exp[1:], lin[1:])):
Line 72:             msg = 'step=%i/%i exp=%f lin=%f' % (i+1, steps, a, b)
Line 73:             self.assertTrue(a <= b, msg)
again, self assertGreater(b, a, msg)
Line 74: 
Line 75:     @permutations(_PARAMS)
Line 76:     def test_exponential_same_end_value(self, dtime, steps):
Line 77:         exp = self._default(dtime, steps)

Line 107: def pairwise(iterable):
Line 108:     "s -> (s0,s1), (s1,s2), (s2, s3), ..."
Line 109:     a, b = tee(iterable)
Line 110:     next(b, None)
Line 111:     return izip(a, b)
thought about this a bit, wouldn't importing islice and running izip(a, islice(b, 1, None)) be a bit nicer?
Line 112: 
Line 113: 
Line 114: def _linear_downtime(downtime, steps):
Line 115:     "this is the old formula as reference"

[Dan Kenigsberg]

Dan Kenigsberg has posted comments on this change.

Change subject: vm: migration: exponential downtime increment
......................................................................

Patch Set 8: Code-Review-1

(3 comments)

minor nit

http://gerrit.ovirt.org/#/c/25820/8/tests/vmMigrationTests.py
File tests/vmMigrationTests.py:

Line 46: class TestVmMigrationDowntimeSequence(TestCaseBase):
Line 47: 
Line 48:     @permutations(_PARAMS)
Line 49:     def test_downtime_is_sequence(self, dtime, steps):
Line 50:         self.assertTrue(len(self._default(dtime, steps)) >= 2)
> can be rewritten as self.assertGreaterEqual(len(self._default(dtime, steps)

As Francesco noted to me, these were introduced only on Python 2.7 (we still need to support 2.6).
Line 51: 
Line 52:     @permutations(_PARAMS)
Line 53:     def test_downtime_increasing(self, dtime, steps):
Line 54:         for a, b in pairwise(self._default(dtime, steps)):

Line 107: def pairwise(iterable):
Line 108:     "s -> (s0,s1), (s1,s2), (s2, s3), ..."
Line 109:     a, b = tee(iterable)
Line 110:     next(b, None)
Line 111:     return izip(a, b)
> thought about this a bit, wouldn't importing islice and running izip(a, isl

I don't really understand your suggestion; I like copying the recipe unless there's a good reason not to.
Line 112: 
Line 113: 
Line 114: def _linear_downtime(downtime, steps):
Line 115:     "this is the old formula as reference"

http://gerrit.ovirt.org/#/c/25820/8/vdsm/virt/migration.py
File vdsm/virt/migration.py:

Line 355: 
Line 356: 
Line 357: def exponential_downtime(downtime, steps):
Line 358:     offset = downtime / float(steps)
Line 359:     base = (downtime - offset) ** (1 / float(steps-1))
space around operator
Line 360: 
Line 361:     for i in range(steps):
Line 362:         yield int(offset + base ** i)
Line 363:

[Jenkins CI RO]

oVirt Jenkins CI Server has posted comments on this change.

Change subject: vm: migration: exponential downtime increment
......................................................................

Patch Set 9:

Build Failed

http://jenkins.ovirt.org/job/vdsm_master_unit_tests_gerrit_el/11759/ : FAILURE

http://jenkins.ovirt.org/job/vdsm_master_unit-tests_created/12703/ : SUCCESS

http://jenkins.ovirt.org/job/vdsm_master_virt_functional_tests_gerrit/1707/ : There was an infra issue, please contact [e-mail]

http://jenkins.ovirt.org/job/vdsm_master_pep8_gerrit/12548/ : SUCCESS

[Dan Kenigsberg]

Dan Kenigsberg has posted comments on this change.

Change subject: vm: migration: exponential downtime increment
......................................................................

Patch Set 9: Code-Review+2

[Martin Polednik]

Martin Polednik has posted comments on this change.

Change subject: vm: migration: exponential downtime increment
......................................................................

Patch Set 9: Code-Review+1

Francesco Romani has posted comments on this change.

Change subject: vm: migration: exponential downtime increment
......................................................................

Patch Set 9: Verified+1

re-verified while running a few migration, both with idle and busy VM:

Thread-444::DEBUG::2014-11-04 11:49:30,905::migration::393::vm.Vm::(run) vmId=`ace61b23-f455-4fc7-8d59-d9c469657b2a`::migration downtime thread started
Thread-444::DEBUG::2014-11-04 11:50:00,916::migration::405::vm.Vm::(run) vmId=`ace61b23-f455-4fc7-8d59-d9c469657b2a`::setting migration downtime to 51
Thread-444::DEBUG::2014-11-04 11:50:30,928::migration::405::vm.Vm::(run) vmId=`ace61b23-f455-4fc7-8d59-d9c469657b2a`::setting migration downtime to 51
Thread-444::DEBUG::2014-11-04 11:51:00,933::migration::405::vm.Vm::(run) vmId=`ace61b23-f455-4fc7-8d59-d9c469657b2a`::setting migration downtime to 53
Thread-444::DEBUG::2014-11-04 11:51:30,937::migration::405::vm.Vm::(run) vmId=`ace61b23-f455-4fc7-8d59-d9c469657b2a`::setting migration downtime to 57
Thread-444::DEBUG::2014-11-04 11:52:00,942::migration::405::vm.Vm::(run) vmId=`ace61b23-f455-4fc7-8d59-d9c469657b2a`::setting migration downtime to 65
Thread-444::DEBUG::2014-11-04 11:52:30,949::migration::405::vm.Vm::(run) vmId=`ace61b23-f455-4fc7-8d59-d9c469657b2a`::setting migration downtime to 79
Thread-444::DEBUG::2014-11-04 11:53:00,953::migration::405::vm.Vm::(run) vmId=`ace61b23-f455-4fc7-8d59-d9c469657b2a`::setting migration downtime to 108
Thread-444::DEBUG::2014-11-04 11:53:30,957::migration::405::vm.Vm::(run) vmId=`ace61b23-f455-4fc7-8d59-d9c469657b2a`::setting migration downtime to 165
Thread-443::DEBUG::2014-11-04 11:53:51,288::migration::411::vm.Vm::(stop) vmId=`ace61b23-f455-4fc7-8d59-d9c469657b2a`::stopping migration downtime thread
Thread-444::DEBUG::2014-11-04 11:53:51,289::migration::408::vm.Vm::(run) vmId=`ace61b23-f455-4fc7-8d59-d9c469657b2a`::migration downtime thread exiting
Thread-733::DEBUG::2014-11-04 11:56:48,123::migration::393::vm.Vm::(run) vmId=`ace61b23-f455-4fc7-8d59-d9c469657b2a`::migration downtime thread started
Thread-733::DEBUG::2014-11-04 11:57:18,127::migration::405::vm.Vm::(run) vmId=`ace61b23-f455-4fc7-8d59-d9c469657b2a`::setting migration downtime to 51
Thread-733::DEBUG::2014-11-04 11:57:48,131::migration::405::vm.Vm::(run) vmId=`ace61b23-f455-4fc7-8d59-d9c469657b2a`::setting migration downtime to 51
Thread-733::DEBUG::2014-11-04 11:58:18,136::migration::405::vm.Vm::(run) vmId=`ace61b23-f455-4fc7-8d59-d9c469657b2a`::setting migration downtime to 53
Thread-733::DEBUG::2014-11-04 11:58:48,140::migration::405::vm.Vm::(run) vmId=`ace61b23-f455-4fc7-8d59-d9c469657b2a`::setting migration downtime to 57
Thread-732::DEBUG::2014-11-04 11:59:00,561::migration::411::vm.Vm::(stop) vmId=`ace61b23-f455-4fc7-8d59-d9c469657b2a`::stopping migration downtime thread
Thread-733::DEBUG::2014-11-04 11:59:00,564::migration::408::vm.Vm::(run) vmId=`ace61b23-f455-4fc7-8d59-d9c469657b2a`::migration downtime thread exiting
Thread-864::DEBUG::2014-11-04 11:59:45,684::migration::393::vm.Vm::(run) vmId=`ace61b23-f455-4fc7-8d59-d9c469657b2a`::migration downtime thread started
Thread-864::DEBUG::2014-11-04 12:00:15,687::migration::405::vm.Vm::(run) vmId=`ace61b23-f455-4fc7-8d59-d9c469657b2a`::setting migration downtime to 51
Thread-864::DEBUG::2014-11-04 12:00:45,691::migration::405::vm.Vm::(run) vmId=`ace61b23-f455-4fc7-8d59-d9c469657b2a`::setting migration downtime to 51
Thread-864::DEBUG::2014-11-04 12:01:15,696::migration::405::vm.Vm::(run) vmId=`ace61b23-f455-4fc7-8d59-d9c469657b2a`::setting migration downtime to 53
Thread-864::DEBUG::2014-11-04 12:01:45,700::migration::405::vm.Vm::(run) vmId=`ace61b23-f455-4fc7-8d59-d9c469657b2a`::setting migration downtime to 57
Thread-863::DEBUG::2014-11-04 12:01:56,836::migration::411::vm.Vm::(stop) vmId=`ace61b23-f455-4fc7-8d59-d9c469657b2a`::stopping migration downtime thread
Thread-864::DEBUG::2014-11-04 12:01:56,839::migration::408::vm.Vm::(run) vmId=`ace61b23-f455-4fc7-8d59-d9c469657b2a`::migration downtime thread exiting
Thread-1065::DEBUG::2014-11-04 12:04:37,643::migration::393::vm.Vm::(run) vmId=`ace61b23-f455-4fc7-8d59-d9c469657b2a`::migration downtime thread started
Thread-1064::DEBUG::2014-11-04 12:04:56,465::migration::411::vm.Vm::(stop) vmId=`ace61b23-f455-4fc7-8d59-d9c469657b2a`::stopping migration downtime thread
Thread-1065::DEBUG::2014-11-04 12:04:56,468::migration::408::vm.Vm::(run) vmId=`ace61b23-f455-4fc7-8d59-d9c469657b2a`::migration downtime thread exiting
Thread-1108::DEBUG::2014-11-04 12:05:36,462::migration::393::vm.Vm::(run) vmId=`ace61b23-f455-4fc7-8d59-d9c469657b2a`::migration downtime thread started
Thread-1105::DEBUG::2014-11-04 12:05:57,641::migration::411::vm.Vm::(stop) vmId=`ace61b23-f455-4fc7-8d59-d9c469657b2a`::stopping migration downtime thread
Thread-1108::DEBUG::2014-11-04 12:05:57,644::migration::408::vm.Vm::(run) vmId=`ace61b23-f455-4fc7-8d59-d9c469657b2a`::migration downtime thread exiting
Thread-1217::DEBUG::2014-11-04 12:08:19,482::migration::393::vm.Vm::(run) vmId=`ace61b23-f455-4fc7-8d59-d9c469657b2a`::migration downtime thread started
Thread-1216::DEBUG::2014-11-04 12:08:39,996::migration::411::vm.Vm::(stop) vmId=`ace61b23-f455-4fc7-8d59-d9c469657b2a`::stopping migration downtime thread
Thread-1217::DEBUG::2014-11-04 12:08:40,000::migration::408::vm.Vm::(run) vmId=`ace61b23-f455-4fc7-8d59-d9c469657b2a`::migration downtime thread exiting

[Dan Kenigsberg]

Dan Kenigsberg has submitted this change and it was merged.

Change subject: vm: migration: exponential downtime increment
......................................................................

vm: migration: exponential downtime increment

Migration downtime is calculated using an interpolation
using the current downtime step (up to a configurabile
maximum) and the maximum downtime.

The downtime is incremented at each downtime step until
it reaches the maximum, or the migration is finished.

This patch changes the interpolation formula from linear
to exponential, being the new one:

d = c ** s + k

where d is the downtime, s is the step and c is the
exponentiation base computed in such a way
of when s = S (number of steps to perform),
then we will have d = D (selected downtime).
K is a constant added to make sure that downtime 'd'
is never equal to zero.

The rationale for this patch is that moving linearly from
the minimum to maximum downtime limit isn't very useful:
if if we took 1s downtime for a migration that could complete
with 0.1s downtime, that's very bad, but if we take 11s to
complete a migration that could finish in 10s, it's not a
big deal.

So, it makes more sense to step towards the final hard limit
exponentially rather than linearly.
In addition, because most of the attempts will be much shorted
than the final hard limit, it won't blow out the total time to
complete migration nearly as badly.

Please note that this patch still preserve an linear
increment of the wait time.

Along the way, we take the chance to bootstrap unit
tests for vdsm/virt/migration.py.

Change-Id: I6401772f52ea28144452e67198bddff18f6703eb
Bug-Url: https://bugzilla.redhat.com/show_bug.cgi?id=1004101
Signed-off-by: Francesco Romani 
Reviewed-on: http://gerrit.ovirt.org/25820
Reviewed-by: Dan Kenigsberg 
Reviewed-by: Martin Polednik 
---
M tests/Makefile.am
A tests/vmMigrationTests.py
M vdsm/virt/migration.py
3 files changed, 132 insertions(+), 3 deletions(-)

Approvals:
  Dan Kenigsberg: Looks good to me, approved
  Francesco Romani: Verified
  Martin Polednik: Looks good to me, but someone else must approve

[Jenkins CI RO]

oVirt Jenkins CI Server has posted comments on this change.

Change subject: vm: migration: exponential downtime increment
......................................................................

Patch Set 10:

Build Failed

http://jenkins.ovirt.org/job/vdsm_master_create-rpms-fc21-x86_64_merged/155/ : FAILURE

http://jenkins.ovirt.org/job/vdsm_master_create-rpms-el6-x86_64_merged/178/ : SUCCESS

http://jenkins.ovirt.org/job/vdsm_master_create-rpms_merged_test_debug/371/ : SUCCESS

http://jenkins.ovirt.org/job/vdsm_master_verify-error-codes_merged/6003/ : SUCCESS

http://jenkins.ovirt.org/job/vdsm_master_create-rpms-el7-x86_64_merged/176/ : SUCCESS

http://jenkins.ovirt.org/job/vdsm_master_create-rpms-fc20-x86_64_merged/171/ : FAILURE

http://jenkins.ovirt.org/job/vdsm_master_unit-tests_merged/4164/ : FAILURE