Unwanted process creation when importing storage.sd

I have a python program that is making use of the vdscli and storage python
modules.  I noticed that my program was creating a bunch of sub-processes that I
did not request.  After some investigation I determined the chain of imports
that is causing these threads to be created:

Traceback (most recent call last):
  File "./vdsmi", line 7, in <module>
    from backend.Backend import Backend
  File "/home/aglitke/src/vdsm-interface/backend/Backend.py", line 5, in
<module>
    from storage.sd import LOCALFS_DOMAIN, DATA_DOMAIN
  File "/usr/share/vdsm/storage/sd.py", line 30, in <module>
    import resourceFactories
  File "/usr/share/vdsm/storage/resourceFactories.py", line 29, in
<module>
    import image
  File "/usr/share/vdsm/storage/image.py", line 27, in <module>
    import volume
  File "/usr/share/vdsm/storage/volume.py", line 33, in <module>
    import task
  File "/usr/share/vdsm/storage/task.py", line 58, in <module>
    import outOfProcess as oop
  File "/usr/share/vdsm/storage/outOfProcess.py", line 38, in <module>
    _globalPool = ProcessPool(MAX_HELPERS, GRACE_PERIOD, DEFAULT_TIMEOUT)

outOfProcess.py is initializing a global ProcessPool at module import which is
causing the unwanted threads.  In order to prevent this, I believe we need to
either clean up this mess of cross-dependencies or require that users of
outOfProcess call a global processPoolInit() function if the ProcessPool is
needed.  I pulled in an awful lot of code in an effort to load two integer
constants :)

-- 
Adam Litke <agl(a)us.ibm.com&gt;
IBM Linux Technology Center

[Dan Kenigsberg]

On Sat, Sep 10, 2011 at 11:03:56AM -0500, Adam Litke wrote:
>  I have a python program that is making use of the vdscli and storage
> python
>  modules.  I noticed that my program was creating a bunch of sub-processes that I
>  did not request.  After some investigation I determined the chain of imports
>  that is causing these threads to be created:
>  
>  Traceback (most recent call last):
>    File "./vdsmi", line 7, in <module>
>      from backend.Backend import Backend
>    File "/home/aglitke/src/vdsm-interface/backend/Backend.py", line 5, in
> <module>
>      from storage.sd import LOCALFS_DOMAIN, DATA_DOMAIN
>    File "/usr/share/vdsm/storage/sd.py", line 30, in <module>
>      import resourceFactories
>    File "/usr/share/vdsm/storage/resourceFactories.py", line 29, in
> <module>
>      import image
>    File "/usr/share/vdsm/storage/image.py", line 27, in <module>
>      import volume
>    File "/usr/share/vdsm/storage/volume.py", line 33, in <module>
>      import task
>    File "/usr/share/vdsm/storage/task.py", line 58, in <module>
>      import outOfProcess as oop
>    File "/usr/share/vdsm/storage/outOfProcess.py", line 38, in <module>
>      _globalPool = ProcessPool(MAX_HELPERS, GRACE_PERIOD, DEFAULT_TIMEOUT)
>  
>  outOfProcess.py is initializing a global ProcessPool at module import which is
>  causing the unwanted threads.  In order to prevent this, I believe we need to
>  either clean up this mess of cross-dependencies or require that users of
>  outOfProcess call a global processPoolInit() function if the ProcessPool is
>  needed.  I pulled in an awful lot of code in an effort to load two integer
>  constants :) 
Yes, this has become more painful with

commit 1c487d9007a7235404bcb467daf71d335bbbc26a
Author: Saggi Mizrahi <smizrahi(a)redhat.com&gt;
Date:   Mon Jul 25 16:18:36 2011 +0300

    BZ#725967 - one process pool to rule them all

    Instead of having a process pool per domain use one pool but limit the amount of
    workers a domain can hold at any given time.

that creates gazillion processes on startup. Unfortunately, this is not the
worst effect of this patch; in many senses master is currently broken.
Hopefully https://fedorahosted.org/pipermail/vdsm-patches/2011-September/001768.html
would fix it soon.

Dan.

On Sun, Sep 11, 2011 at 10:22:04AM +0300, Dan Kenigsberg wrote:
>  On Sat, Sep 10, 2011 at 11:03:56AM -0500, Adam Litke wrote:
>  > outOfProcess.py is initializing a global ProcessPool at module import which is
>  > causing the unwanted threads.  In order to prevent this, I believe we need to
>  > either clean up this mess of cross-dependencies or require that users of
>  > outOfProcess call a global processPoolInit() function if the ProcessPool is
>  > needed.  I pulled in an awful lot of code in an effort to load two integer
>  > constants :)
>  
>  Yes, this has become more painful with
>  
>  commit 1c487d9007a7235404bcb467daf71d335bbbc26a
>  Author: Saggi Mizrahi <smizrahi(a)redhat.com&gt;
>  Date:   Mon Jul 25 16:18:36 2011 +0300
>  
>      BZ#725967 - one process pool to rule them all
>  
>      Instead of having a process pool per domain use one pool but limit the amount of
>      workers a domain can hold at any given time.
>  
>  that creates gazillion processes on startup. Unfortunately, this is not the
>  worst effect of this patch; in many senses master is currently broken.
>  Hopefully https://fedorahosted.org/pipermail/vdsm-patches/2011-September/001768.html
>  would fix it soon. 
Excellent, thanks Dan.

-- 
Adam Litke <agl(a)us.ibm.com&gt;
IBM Linux Technology Center