Change in vdsm[master]: [WIP] Implement a process to do dangerous IO in C

Saggi Mizrahi has uploaded a new change for review.

Change subject: [WIP] Implement a process to do dangerous IO in C
......................................................................

[WIP] Implement a process to do dangerous IO in C

This replaces the process pool with a process that can serve multiple
requests written in C.

This implementation is much more scalable and lightweight. Should solve
bugs related to running out of helpers, logging getting suck, python
forking deadlocking, running out of memory and other things as well.

The communication between VDSM and the IOProcess is done with json
objects.

The IOProcess starts with 3 thread:
1. requestReader - reads requests from the pipe, builds a DOM
representation of it and queues it up for handling
2. responseWriter - gets response DOMs from the queue converts them to a
JSON string and send it over the pipe
3. requestHandler - pops requests from the queue and provisions threads
for handling them. Currently we I just allocate a new thread per
request. If there is ever a need to have a thread pool this is where
the load balancing is going to sit.

Each request gets the are as a JsonNode and returns a response that is a
JsonNode as well. Most exported functions are pretty trivial and are a
good example on how to write new ones.

Unlink the ProcessPoolHelper, high level commands sit of the OopWrapper
and are run from the client side instead of being implemented in C on
the IOProcess side.

Change-Id: Ie4664d5330debbe38ba33b74ebb586ac42913b4a
Signed-off-by: Saggi Mizrahi <smizrahi(a)redhat.com&gt;
---
M configure.ac
M tests/Makefile.am
A tests/ioprocessTests.py
A tests/outOfProcessTests.py
D tests/processPoolTests.py
M vdsm.spec.in
M vdsm/constants.py.in
M vdsm/storage/Makefile.am
M vdsm/storage/fileSD.py
M vdsm/storage/fileUtils.py
M vdsm/storage/fileVolume.py
A vdsm/storage/ioprocess.py
A vdsm/storage/ioprocess/.gitignore
A vdsm/storage/ioprocess/Makefile.am
A vdsm/storage/ioprocess/exported-functions.c
A vdsm/storage/ioprocess/exported-functions.h
A vdsm/storage/ioprocess/ioprocess.c
A vdsm/storage/ioprocess/json-dom-generator.c
A vdsm/storage/ioprocess/json-dom-generator.h
A vdsm/storage/ioprocess/json-dom-parser.c
A vdsm/storage/ioprocess/json-dom-parser.h
A vdsm/storage/ioprocess/json-dom.c
A vdsm/storage/ioprocess/json-dom.h
M vdsm/storage/misc.py
M vdsm/storage/nfsSD.py
M vdsm/storage/outOfProcess.py
D vdsm/storage/processPool.py
M vdsm/storage/sd.py
M vdsm/storage/sp.py
M vdsm/storage/task.py
30 files changed, 3,018 insertions(+), 666 deletions(-)


  git pull ssh://gerrit.ovirt.org:29418/vdsm refs/changes/46/3946/1
--
To view, visit http://gerrit.ovirt.org/3946
To unsubscribe, visit http://gerrit.ovirt.org/settings

Gerrit-MessageType: newchange
Gerrit-Change-Id: Ie4664d5330debbe38ba33b74ebb586ac42913b4a
Gerrit-PatchSet: 1
Gerrit-Project: vdsm
Gerrit-Branch: master
Gerrit-Owner: Saggi Mizrahi <smizrahi(a)redhat.com&gt;

Peter V. Saveliev has posted comments on this change.

Change subject: [WIP] Implement a process to do dangerous IO in C
......................................................................


Patch Set 3: Fails

while configure: «Package requirements (glib-2.0) were not met»

# rpm -qa | grep glib2
glib2-2.22.5-7.el6.x86_64

--
To view, visit http://gerrit.ovirt.org/3946
To unsubscribe, visit http://gerrit.ovirt.org/settings

Gerrit-MessageType: comment
Gerrit-Change-Id: Ie4664d5330debbe38ba33b74ebb586ac42913b4a
Gerrit-PatchSet: 3
Gerrit-Project: vdsm
Gerrit-Branch: master
Gerrit-Owner: Saggi Mizrahi <smizrahi(a)redhat.com&gt;
Gerrit-Reviewer: Peter V. Saveliev <peet(a)redhat.com&gt;
Gerrit-Reviewer: Saggi Mizrahi <smizrahi(a)redhat.com&gt;

Saggi Mizrahi has posted comments on this change.

Change subject: [WIP] Implement a process to do dangerous IO in C
......................................................................


Patch Set 3:

install glib2-devel for bulding look at all the new buildRequires entries

--
To view, visit http://gerrit.ovirt.org/3946
To unsubscribe, visit http://gerrit.ovirt.org/settings

Gerrit-MessageType: comment
Gerrit-Change-Id: Ie4664d5330debbe38ba33b74ebb586ac42913b4a
Gerrit-PatchSet: 3
Gerrit-Project: vdsm
Gerrit-Branch: master
Gerrit-Owner: Saggi Mizrahi <smizrahi(a)redhat.com&gt;
Gerrit-Reviewer: Peter V. Saveliev <peet(a)redhat.com&gt;
Gerrit-Reviewer: Saggi Mizrahi <smizrahi(a)redhat.com&gt;

Peter V. Saveliev has posted comments on this change.

Change subject: [WIP] Implement a process to do dangerous IO in C
......................................................................


Patch Set 6: Fails

testSymlinkFail unittest fails (and thanks for the hint about -devel packages)

--
To view, visit http://gerrit.ovirt.org/3946
To unsubscribe, visit http://gerrit.ovirt.org/settings

Gerrit-MessageType: comment
Gerrit-Change-Id: Ie4664d5330debbe38ba33b74ebb586ac42913b4a
Gerrit-PatchSet: 6
Gerrit-Project: vdsm
Gerrit-Branch: master
Gerrit-Owner: Saggi Mizrahi <smizrahi(a)redhat.com&gt;
Gerrit-Reviewer: Peter V. Saveliev <peet(a)redhat.com&gt;
Gerrit-Reviewer: Saggi Mizrahi <smizrahi(a)redhat.com&gt;

Saggi Mizrahi has posted comments on this change.

Change subject: [WIP] Implement a process to do dangerous IO in C
......................................................................


Patch Set 6:

It obviously works for me, can you past the test output?
Just the failed test is fine

--
To view, visit http://gerrit.ovirt.org/3946
To unsubscribe, visit http://gerrit.ovirt.org/settings

Gerrit-MessageType: comment
Gerrit-Change-Id: Ie4664d5330debbe38ba33b74ebb586ac42913b4a
Gerrit-PatchSet: 6
Gerrit-Project: vdsm
Gerrit-Branch: master
Gerrit-Owner: Saggi Mizrahi <smizrahi(a)redhat.com&gt;
Gerrit-Reviewer: Peter V. Saveliev <peet(a)redhat.com&gt;
Gerrit-Reviewer: Saggi Mizrahi <smizrahi(a)redhat.com&gt;

Change in vdsm[master]: Implement a process to do dangerous IO in C

[Jenkins CI RO]

oVirt Jenkins CI Server has posted comments on this change.

Change subject: Implement a process to do dangerous IO in C
......................................................................


Patch Set 9: No score

Build Started http://jenkins.ovirt.info/job/vdsm_unit_tests_by_patch/80/

--
To view, visit http://gerrit.ovirt.org/3946
To unsubscribe, visit http://gerrit.ovirt.org/settings

Gerrit-MessageType: comment
Gerrit-Change-Id: Ie4664d5330debbe38ba33b74ebb586ac42913b4a
Gerrit-PatchSet: 9
Gerrit-Project: vdsm
Gerrit-Branch: master
Gerrit-Owner: Saggi Mizrahi <smizrahi(a)redhat.com&gt;
Gerrit-Reviewer: Peter V. Saveliev <peet(a)redhat.com&gt;
Gerrit-Reviewer: Saggi Mizrahi <smizrahi(a)redhat.com&gt;
Gerrit-Reviewer: Shu Ming <shuming(a)linux.vnet.ibm.com&gt;
Gerrit-Reviewer: oVirt Jenkins CI Server

Change in vdsm[master]: Implement a process to do dangerous IO in C

[Jenkins CI RO]

oVirt Jenkins CI Server has posted comments on this change.

Change subject: Implement a process to do dangerous IO in C
......................................................................


Patch Set 9:

Build Successful 

http://jenkins.ovirt.info/job/vdsm_unit_tests_by_patch/80/ : SUCCESS

--
To view, visit http://gerrit.ovirt.org/3946
To unsubscribe, visit http://gerrit.ovirt.org/settings

Gerrit-MessageType: comment
Gerrit-Change-Id: Ie4664d5330debbe38ba33b74ebb586ac42913b4a
Gerrit-PatchSet: 9
Gerrit-Project: vdsm
Gerrit-Branch: master
Gerrit-Owner: Saggi Mizrahi <smizrahi(a)redhat.com&gt;
Gerrit-Reviewer: Peter V. Saveliev <peet(a)redhat.com&gt;
Gerrit-Reviewer: Saggi Mizrahi <smizrahi(a)redhat.com&gt;
Gerrit-Reviewer: Shu Ming <shuming(a)linux.vnet.ibm.com&gt;
Gerrit-Reviewer: oVirt Jenkins CI Server

Change in vdsm[master]: Implement a process to do dangerous IO in C

Peter V. Saveliev has posted comments on this change.

Change subject: Implement a process to do dangerous IO in C
......................................................................


Patch Set 9:

Saggi, can you pls rebase the patch?

--
To view, visit http://gerrit.ovirt.org/3946
To unsubscribe, visit http://gerrit.ovirt.org/settings

Gerrit-MessageType: comment
Gerrit-Change-Id: Ie4664d5330debbe38ba33b74ebb586ac42913b4a
Gerrit-PatchSet: 9
Gerrit-Project: vdsm
Gerrit-Branch: master
Gerrit-Owner: Saggi Mizrahi <smizrahi(a)redhat.com&gt;
Gerrit-Reviewer: Peter V. Saveliev <peet(a)redhat.com&gt;
Gerrit-Reviewer: Saggi Mizrahi <smizrahi(a)redhat.com&gt;
Gerrit-Reviewer: Shu Ming <shuming(a)linux.vnet.ibm.com&gt;
Gerrit-Reviewer: oVirt Jenkins CI Server

Change in vdsm[master]: Implement a process to do dangerous IO in C

Peter V. Saveliev has posted comments on this change.

Change subject: Implement a process to do dangerous IO in C
......................................................................


Patch Set 9: Fails; I would prefer that you didn't submit this

please rebase

--
To view, visit http://gerrit.ovirt.org/3946
To unsubscribe, visit http://gerrit.ovirt.org/settings

Gerrit-MessageType: comment
Gerrit-Change-Id: Ie4664d5330debbe38ba33b74ebb586ac42913b4a
Gerrit-PatchSet: 9
Gerrit-Project: vdsm
Gerrit-Branch: master
Gerrit-Owner: Saggi Mizrahi <smizrahi(a)redhat.com&gt;
Gerrit-Reviewer: Peter V. Saveliev <peet(a)redhat.com&gt;
Gerrit-Reviewer: Saggi Mizrahi <smizrahi(a)redhat.com&gt;
Gerrit-Reviewer: Shu Ming <shuming(a)linux.vnet.ibm.com&gt;
Gerrit-Reviewer: oVirt Jenkins CI Server

Change in vdsm[master]: Implement a process to do dangerous IO in C

Itamar Heim has posted comments on this change.

Change subject: Implement a process to do dangerous IO in C
......................................................................


Patch Set 9:

ping

-- 
To view, visit http://gerrit.ovirt.org/3946
To unsubscribe, visit http://gerrit.ovirt.org/settings

Gerrit-MessageType: comment
Gerrit-Change-Id: Ie4664d5330debbe38ba33b74ebb586ac42913b4a
Gerrit-PatchSet: 9
Gerrit-Project: vdsm
Gerrit-Branch: master
Gerrit-Owner: Saggi Mizrahi <smizrahi(a)redhat.com&gt;
Gerrit-Reviewer: Itamar Heim <iheim(a)redhat.com&gt;
Gerrit-Reviewer: Saggi Mizrahi <smizrahi(a)redhat.com&gt;
Gerrit-Reviewer: Shu Ming <shuming(a)linux.vnet.ibm.com&gt;
Gerrit-Reviewer: Yaniv Bronhaim <ybronhei(a)redhat.com&gt;
Gerrit-Reviewer: oVirt Jenkins CI Server
Gerrit-HasComments: No

Change in vdsm[master]: Implement a process to do dangerous IO in C

Itamar Heim has posted comments on this change.

Change subject: Implement a process to do dangerous IO in C
......................................................................


Patch Set 9:

still relevant?

-- 
To view, visit http://gerrit.ovirt.org/3946
To unsubscribe, visit http://gerrit.ovirt.org/settings

Gerrit-MessageType: comment
Gerrit-Change-Id: Ie4664d5330debbe38ba33b74ebb586ac42913b4a
Gerrit-PatchSet: 9
Gerrit-Project: vdsm
Gerrit-Branch: master
Gerrit-Owner: Saggi Mizrahi <smizrahi(a)redhat.com&gt;
Gerrit-Reviewer: Itamar Heim <iheim(a)redhat.com&gt;
Gerrit-Reviewer: Saggi Mizrahi <smizrahi(a)redhat.com&gt;
Gerrit-Reviewer: Shu Ming <shuming(a)linux.vnet.ibm.com&gt;
Gerrit-Reviewer: Yaniv Bronhaim <ybronhei(a)redhat.com&gt;
Gerrit-Reviewer: oVirt Jenkins CI Server
Gerrit-HasComments: No

Change in vdsm[master]: Implement a process to do dangerous IO in C

Itamar Heim has abandoned this change.

Change subject: Implement a process to do dangerous IO in C
......................................................................


Abandoned

no activity, restore if relevant

-- 
To view, visit http://gerrit.ovirt.org/3946
To unsubscribe, visit http://gerrit.ovirt.org/settings

Gerrit-MessageType: abandon
Gerrit-Change-Id: Ie4664d5330debbe38ba33b74ebb586ac42913b4a
Gerrit-PatchSet: 9
Gerrit-Project: vdsm
Gerrit-Branch: master
Gerrit-Owner: Saggi Mizrahi <smizrahi(a)redhat.com&gt;
Gerrit-Reviewer: Itamar Heim <iheim(a)redhat.com&gt;
Gerrit-Reviewer: Saggi Mizrahi <smizrahi(a)redhat.com&gt;
Gerrit-Reviewer: Shu Ming <shuming(a)linux.vnet.ibm.com&gt;
Gerrit-Reviewer: Yaniv Bronhaim <ybronhei(a)redhat.com&gt;
Gerrit-Reviewer: oVirt Jenkins CI Server