Change in vdsm[ovirt-4.0.5]: api: Speed up schema loading

[Piotr Kliczewski]

Hello Nir Soffer,

I'd like you to do a code review.  Please visit

https://gerrit.ovirt.org/65452

to review the following change.

Change subject: api: Speed up schema loading
......................................................................

api: Speed up schema loading

Use yaml.CLoader for 13x speedup in vdsm schema loading.

Here is a comparison with other formats:

$ python bench.py

yaml: 1.053720 seconds per load
     cyaml: 0.086472 seconds per load
      json: 0.049521 seconds per load
    pickle: 0.001516 seconds per load

$ python3 bench.py

yaml: 1.340607 seconds per load
     cyaml: 0.100010 seconds per load
      json: 0.026627 seconds per load
    pickle: 0.002153 seconds per load

$ grep 'model name' /proc/cpuinfo | head -n1
model name	: Intel(R) Core(TM) i7-4770 CPU @ 3.40GHz

Clearly the best format is pickle, 700X times faster on python 2 and
638X times faster on python 3. However loading the schema in 100
milliseconds seems to be good enough for now.

The benchmark script:
https://bugzilla.redhat.com/attachment.cgi?id=1208141

Change-Id: I230fb88c279973f542f6e1cac17f43ff6f0d2f62
Bug-Url: https://bugzilla.redhat.com/1381899
Signed-off-by: Nir Soffer 
Signed-off-by: Piotr Kliczewski 
Reviewed-on: https://gerrit.ovirt.org/65230
Reviewed-by: Simone Tiraboschi 
Continuous-Integration: Jenkins CI
Reviewed-by: Piotr Kliczewski 
Tested-by: Simone Tiraboschi 
---
M lib/api/vdsmapi.py
1 file changed, 1 insertion(+), 1 deletion(-)

git pull ssh://gerrit.ovirt.org:29418/vdsm refs/changes/52/65452/1

diff --git a/lib/api/vdsmapi.py b/lib/api/vdsmapi.py
index b340f73..635aedf 100755
--- a/lib/api/vdsmapi.py
+++ b/lib/api/vdsmapi.py
@@ -92,7 +92,7 @@
         try:
             for path in paths:
                 with open(path) as f:
-                    loaded_schema = yaml.load(f)
+                    loaded_schema = yaml.load(f, Loader=yaml.CLoader)

types = loaded_schema.pop('types')
                 self._types.update(types)

gerrit-hooks has posted comments on this change.

Change subject: api: Speed up schema loading
......................................................................

Patch Set 1:

* Update Tracker::#1381899::OK, status: POST
* Check Bug-Url::OK
* Check Public Bug::#1381899::OK, public bug
* Check Product::#1381899::OK, product: vdsm
* Check TM::#1381899::OK, target milestone: ovirt-4.0.5
* Check merged to previous::IGNORE,         Not in stable branch (['ovirt-3.6', 'ovirt-4.0'])

[Nir Soffer]

Nir Soffer has posted comments on this change.

Change subject: api: Speed up schema loading
......................................................................

Patch Set 1: Code-Review+1

Francesco Romani has posted comments on this change.

Change subject: api: Speed up schema loading
......................................................................

Patch Set 1: Code-Review+2

as part of verification, please make sure yaml.Cloader is available by default (e.g. we don't need extra packages)

Francesco Romani has posted comments on this change.

Change subject: api: Speed up schema loading
......................................................................

Patch Set 1:

discard my last comment. I assume this check has already been done.

Francesco Romani has posted comments on this change.

Change subject: api: Speed up schema loading
......................................................................

Patch Set 1: Continuous-Integration+1

run CI tests locally.

Francesco Romani has posted comments on this change.

Change subject: api: Speed up schema loading
......................................................................

Patch Set 1: Verified+1

trivial change, same code as master and ovirt-4.0

Francesco Romani has submitted this change and it was merged.

Change subject: api: Speed up schema loading
......................................................................

api: Speed up schema loading

Use yaml.CLoader for 13x speedup in vdsm schema loading.

Here is a comparison with other formats:

$ python bench.py

yaml: 1.053720 seconds per load
     cyaml: 0.086472 seconds per load
      json: 0.049521 seconds per load
    pickle: 0.001516 seconds per load

$ python3 bench.py

yaml: 1.340607 seconds per load
     cyaml: 0.100010 seconds per load
      json: 0.026627 seconds per load
    pickle: 0.002153 seconds per load

$ grep 'model name' /proc/cpuinfo | head -n1
model name	: Intel(R) Core(TM) i7-4770 CPU @ 3.40GHz

Clearly the best format is pickle, 700X times faster on python 2 and
638X times faster on python 3. However loading the schema in 100
milliseconds seems to be good enough for now.

The benchmark script:
https://bugzilla.redhat.com/attachment.cgi?id=1208141

Change-Id: I230fb88c279973f542f6e1cac17f43ff6f0d2f62
Bug-Url: https://bugzilla.redhat.com/1381899
Signed-off-by: Nir Soffer 
Signed-off-by: Piotr Kliczewski 
Reviewed-on: https://gerrit.ovirt.org/65230
Reviewed-by: Simone Tiraboschi 
Continuous-Integration: Jenkins CI
Reviewed-by: Piotr Kliczewski 
Tested-by: Simone Tiraboschi 
Reviewed-on: https://gerrit.ovirt.org/65452
Reviewed-by: Francesco Romani 
Continuous-Integration: Francesco Romani 
Tested-by: Francesco Romani 
---
M lib/api/vdsmapi.py
1 file changed, 1 insertion(+), 1 deletion(-)

Approvals:
  Nir Soffer: Looks good to me, but someone else must approve
  Francesco Romani: Verified; Looks good to me, approved; Passed CI tests

gerrit-hooks has posted comments on this change.

Change subject: api: Speed up schema loading
......................................................................

Patch Set 2:

* Update Tracker::#1381899::OK, status: MERGED
* Set MODIFIED::bug 1381899::::#1381899::::OK