Saggi Mizrahi has uploaded a new change for review.
Change subject: [WIP] Implement a process to do dangerous IO in C ......................................................................
[WIP] Implement a process to do dangerous IO in C
This replaces the process pool with a process that can serve multiple requests written in C.
This implementation is much more scalable and lightweight. Should solve bugs related to running out of helpers, logging getting suck, python forking deadlocking, running out of memory and other things as well.
The communication between VDSM and the IOProcess is done with json objects.
The IOProcess starts with 3 thread: 1. requestReader - reads requests from the pipe, builds a DOM representation of it and queues it up for handling 2. responseWriter - gets response DOMs from the queue converts them to a JSON string and send it over the pipe 3. requestHandler - pops requests from the queue and provisions threads for handling them. Currently we I just allocate a new thread per request. If there is ever a need to have a thread pool this is where the load balancing is going to sit.
Each request gets the are as a JsonNode and returns a response that is a JsonNode as well. Most exported functions are pretty trivial and are a good example on how to write new ones.
Unlink the ProcessPoolHelper, high level commands sit of the OopWrapper and are run from the client side instead of being implemented in C on the IOProcess side.
Change-Id: Ie4664d5330debbe38ba33b74ebb586ac42913b4a Signed-off-by: Saggi Mizrahi smizrahi@redhat.com --- M configure.ac M tests/Makefile.am A tests/ioprocessTests.py A tests/outOfProcessTests.py D tests/processPoolTests.py M vdsm.spec.in M vdsm/constants.py.in M vdsm/storage/Makefile.am M vdsm/storage/fileSD.py M vdsm/storage/fileUtils.py M vdsm/storage/fileVolume.py A vdsm/storage/ioprocess.py A vdsm/storage/ioprocess/.gitignore A vdsm/storage/ioprocess/Makefile.am A vdsm/storage/ioprocess/exported-functions.c A vdsm/storage/ioprocess/exported-functions.h A vdsm/storage/ioprocess/ioprocess.c A vdsm/storage/ioprocess/json-dom-generator.c A vdsm/storage/ioprocess/json-dom-generator.h A vdsm/storage/ioprocess/json-dom-parser.c A vdsm/storage/ioprocess/json-dom-parser.h A vdsm/storage/ioprocess/json-dom.c A vdsm/storage/ioprocess/json-dom.h M vdsm/storage/misc.py M vdsm/storage/nfsSD.py M vdsm/storage/outOfProcess.py D vdsm/storage/processPool.py M vdsm/storage/sd.py M vdsm/storage/sp.py M vdsm/storage/task.py 30 files changed, 3,018 insertions(+), 666 deletions(-)
git pull ssh://gerrit.ovirt.org:29418/vdsm refs/changes/46/3946/1 -- To view, visit http://gerrit.ovirt.org/3946 To unsubscribe, visit http://gerrit.ovirt.org/settings
Gerrit-MessageType: newchange Gerrit-Change-Id: Ie4664d5330debbe38ba33b74ebb586ac42913b4a Gerrit-PatchSet: 1 Gerrit-Project: vdsm Gerrit-Branch: master Gerrit-Owner: Saggi Mizrahi smizrahi@redhat.com
Peter V. Saveliev has posted comments on this change.
Change subject: [WIP] Implement a process to do dangerous IO in C ......................................................................
Patch Set 3: Fails
while configure: «Package requirements (glib-2.0) were not met»
# rpm -qa | grep glib2 glib2-2.22.5-7.el6.x86_64
-- To view, visit http://gerrit.ovirt.org/3946 To unsubscribe, visit http://gerrit.ovirt.org/settings
Gerrit-MessageType: comment Gerrit-Change-Id: Ie4664d5330debbe38ba33b74ebb586ac42913b4a Gerrit-PatchSet: 3 Gerrit-Project: vdsm Gerrit-Branch: master Gerrit-Owner: Saggi Mizrahi smizrahi@redhat.com Gerrit-Reviewer: Peter V. Saveliev peet@redhat.com Gerrit-Reviewer: Saggi Mizrahi smizrahi@redhat.com
Saggi Mizrahi has posted comments on this change.
Change subject: [WIP] Implement a process to do dangerous IO in C ......................................................................
Patch Set 3:
install glib2-devel for bulding look at all the new buildRequires entries
-- To view, visit http://gerrit.ovirt.org/3946 To unsubscribe, visit http://gerrit.ovirt.org/settings
Gerrit-MessageType: comment Gerrit-Change-Id: Ie4664d5330debbe38ba33b74ebb586ac42913b4a Gerrit-PatchSet: 3 Gerrit-Project: vdsm Gerrit-Branch: master Gerrit-Owner: Saggi Mizrahi smizrahi@redhat.com Gerrit-Reviewer: Peter V. Saveliev peet@redhat.com Gerrit-Reviewer: Saggi Mizrahi smizrahi@redhat.com
Peter V. Saveliev has posted comments on this change.
Change subject: [WIP] Implement a process to do dangerous IO in C ......................................................................
Patch Set 6: Fails
testSymlinkFail unittest fails (and thanks for the hint about -devel packages)
-- To view, visit http://gerrit.ovirt.org/3946 To unsubscribe, visit http://gerrit.ovirt.org/settings
Gerrit-MessageType: comment Gerrit-Change-Id: Ie4664d5330debbe38ba33b74ebb586ac42913b4a Gerrit-PatchSet: 6 Gerrit-Project: vdsm Gerrit-Branch: master Gerrit-Owner: Saggi Mizrahi smizrahi@redhat.com Gerrit-Reviewer: Peter V. Saveliev peet@redhat.com Gerrit-Reviewer: Saggi Mizrahi smizrahi@redhat.com
Saggi Mizrahi has posted comments on this change.
Change subject: [WIP] Implement a process to do dangerous IO in C ......................................................................
Patch Set 6:
It obviously works for me, can you past the test output? Just the failed test is fine
-- To view, visit http://gerrit.ovirt.org/3946 To unsubscribe, visit http://gerrit.ovirt.org/settings
Gerrit-MessageType: comment Gerrit-Change-Id: Ie4664d5330debbe38ba33b74ebb586ac42913b4a Gerrit-PatchSet: 6 Gerrit-Project: vdsm Gerrit-Branch: master Gerrit-Owner: Saggi Mizrahi smizrahi@redhat.com Gerrit-Reviewer: Peter V. Saveliev peet@redhat.com Gerrit-Reviewer: Saggi Mizrahi smizrahi@redhat.com
oVirt Jenkins CI Server has posted comments on this change.
Change subject: Implement a process to do dangerous IO in C ......................................................................
Patch Set 9: No score
Build Started http://jenkins.ovirt.info/job/vdsm_unit_tests_by_patch/80/
-- To view, visit http://gerrit.ovirt.org/3946 To unsubscribe, visit http://gerrit.ovirt.org/settings
Gerrit-MessageType: comment Gerrit-Change-Id: Ie4664d5330debbe38ba33b74ebb586ac42913b4a Gerrit-PatchSet: 9 Gerrit-Project: vdsm Gerrit-Branch: master Gerrit-Owner: Saggi Mizrahi smizrahi@redhat.com Gerrit-Reviewer: Peter V. Saveliev peet@redhat.com Gerrit-Reviewer: Saggi Mizrahi smizrahi@redhat.com Gerrit-Reviewer: Shu Ming shuming@linux.vnet.ibm.com Gerrit-Reviewer: oVirt Jenkins CI Server
oVirt Jenkins CI Server has posted comments on this change.
Change subject: Implement a process to do dangerous IO in C ......................................................................
Patch Set 9:
Build Successful
http://jenkins.ovirt.info/job/vdsm_unit_tests_by_patch/80/ : SUCCESS
-- To view, visit http://gerrit.ovirt.org/3946 To unsubscribe, visit http://gerrit.ovirt.org/settings
Gerrit-MessageType: comment Gerrit-Change-Id: Ie4664d5330debbe38ba33b74ebb586ac42913b4a Gerrit-PatchSet: 9 Gerrit-Project: vdsm Gerrit-Branch: master Gerrit-Owner: Saggi Mizrahi smizrahi@redhat.com Gerrit-Reviewer: Peter V. Saveliev peet@redhat.com Gerrit-Reviewer: Saggi Mizrahi smizrahi@redhat.com Gerrit-Reviewer: Shu Ming shuming@linux.vnet.ibm.com Gerrit-Reviewer: oVirt Jenkins CI Server
Peter V. Saveliev has posted comments on this change.
Change subject: Implement a process to do dangerous IO in C ......................................................................
Patch Set 9:
Saggi, can you pls rebase the patch?
-- To view, visit http://gerrit.ovirt.org/3946 To unsubscribe, visit http://gerrit.ovirt.org/settings
Gerrit-MessageType: comment Gerrit-Change-Id: Ie4664d5330debbe38ba33b74ebb586ac42913b4a Gerrit-PatchSet: 9 Gerrit-Project: vdsm Gerrit-Branch: master Gerrit-Owner: Saggi Mizrahi smizrahi@redhat.com Gerrit-Reviewer: Peter V. Saveliev peet@redhat.com Gerrit-Reviewer: Saggi Mizrahi smizrahi@redhat.com Gerrit-Reviewer: Shu Ming shuming@linux.vnet.ibm.com Gerrit-Reviewer: oVirt Jenkins CI Server
Peter V. Saveliev has posted comments on this change.
Change subject: Implement a process to do dangerous IO in C ......................................................................
Patch Set 9: Fails; I would prefer that you didn't submit this
please rebase
-- To view, visit http://gerrit.ovirt.org/3946 To unsubscribe, visit http://gerrit.ovirt.org/settings
Gerrit-MessageType: comment Gerrit-Change-Id: Ie4664d5330debbe38ba33b74ebb586ac42913b4a Gerrit-PatchSet: 9 Gerrit-Project: vdsm Gerrit-Branch: master Gerrit-Owner: Saggi Mizrahi smizrahi@redhat.com Gerrit-Reviewer: Peter V. Saveliev peet@redhat.com Gerrit-Reviewer: Saggi Mizrahi smizrahi@redhat.com Gerrit-Reviewer: Shu Ming shuming@linux.vnet.ibm.com Gerrit-Reviewer: oVirt Jenkins CI Server
Itamar Heim has posted comments on this change.
Change subject: Implement a process to do dangerous IO in C ......................................................................
Patch Set 9:
ping
Itamar Heim has posted comments on this change.
Change subject: Implement a process to do dangerous IO in C ......................................................................
Patch Set 9:
still relevant?
Itamar Heim has abandoned this change.
Change subject: Implement a process to do dangerous IO in C ......................................................................
Abandoned
no activity, restore if relevant
vdsm-patches@lists.fedorahosted.org