[vdsm] cleaning statistics retrieval

Hello VDSM developers,

I'd like to discuss and explain the plans for cleaning up Vm.getStats()
in vdsm/virt/vm.py, and how it affects a bug we have:
https://bugzilla.redhat.com/show_bug.cgi?id=1073478

Let's start from the bug.

To make a long story short, there is a (small) race in VDSM, probably introduced by
commit
8fedf8bde3c28edb07add23c3e9b72681cb48e49
The race can actually be triggered only if the VM is shutting down, so it is not that
bad.

Fixing the reported issue in the specific case can be done with a trivial if, and that it
what I did
in my initial http://gerrit.ovirt.org/#/c/25803/1/vdsm/vm.py,cm

However, this is just a bandaid and a proper solution is needed. This is the reason why
the following versions of http://gerrit.ovirt.org/#/c/25803 changed direction toward a
more comprehensive
approach.

And this is the core of the issue.
My initial idea is to either return success and a complete, well formed statistics set, or
return an error.
However current engine seems to not cope properly with this, and we cannot break backward
compatibility.

Looks like the only way to go is to always return success and to add a field to describe
the content of the
statistics (partial, complete...). THis is admittedly a far cry from the ideal solution,
but it is dictated
by the need to preserve the compatibility with current/old engines.

Moreover, I would like to take the chance and cleanup/refactor the statistics collection.
I already started
adding the test infrastructure: http://gerrit.ovirt.org/#/c/26536/

To summarize, what I suggest to do is:

* fix https://bugzilla.redhat.com/show_bug.cgi?id=1073478 using a simple ugly fix like the
original 
  http://gerrit.ovirt.org/#/c/25803/1/vdsm/vm.py,cm (which I'll resubmit as new
change)
* refactor and clean getStats() and friends
* on the cleaner base, properly fix the statistics collection by let getStats() always
succeed and return
  possibly partial stats, with a new field describing the content

please note that I'm not really happy about this solution, but, given the constraint,
I don't see better
alternatives.

Feedback welcome!

Bests,

-- 
Francesco Romani
RedHat Engineering Virtualization R & D
Phone: [phone]
IRC: fromani