VDSM Meeting for fun an profit 25/7/2011
by smizrahi＠redhat.com
====ACTION LIST====
Review patches
WIKI - Go over the list an choose a bullet to write about
Register to mailing lists
Review patches
====ACTION LIST====
move bug to lvm archive
Stuff we talked about:
-Fun facts about about umbilical cords
-PVs were sometimes created with small 1MB metadata. This is fixed,
correcting VMs in the wild is still being discussed.
-Fix for libvirt restart during migration
-Spm params where not being taken form metadata, defaults were being
used instead. This is now fixed!
-Begining of a move to autotools instead of dankentools
12 years, 9 months
Re: dirty shared memory on python threading
by smizrahi＠redhat.com
OK, I tested it more thoroughly and I was right all along (as usual)
memory is shared and RSS includes the shared pages.
I attached a python script I wrote to compare actual pages between
processes.
fork() acts as expected even in python. The problem is that after the
fork VDSM keeps changing the memory and the children keep a copy of the
memory.
This means that even though they'll never use it a copy will exists.
Because of how we spawn processes. child processes can have different
versions of VDSM thus actually ending up taking all the memory not
because they rewrote it but because VDSM did.
Comparing OOPs that spawned at almost the same time they share a lot of
memory
194 (776 KB) only in A, 2146 (8584 KB) shared, 194 (776 KB) only in B
OOP processes that spawned a few hours apart have a lot less in common
2336 (9344 KB) only in A, 4 (16 KB) shared, 2521 (10084 KB) only in B
So spawning all the processes at once instead of gradually would solve
this memory problem.
I added the script if you want to test it yourselves on your hosts and
test progs
On 07/21/2011 09:52 AM, Dan Kenigsberg wrote:
> On Wed, Jul 20, 2011 at 11:41:33AM -0400, David Malcolm wrote:
>> On Wed, 2011-07-20 at 13:39 +0300, Moran Goldboim wrote:
>>> Hi Dave,
>>>
>>> We are suffering high memory usage on vdsm as a result of using
>>> threading in python, child process is taking most of the father memory.
>>> for example:
>>>
>>> import multiprocessing
>>> import time
>>> l = [1000] * 2**20
>>> def run():
>>> time.sleep(10000)
>>> multiprocessing.Process(target=run).start()
>> Multiprocessing is one of my least favorite modules. It doesn't use
>> threads so much as it forks child processes, without calling exec:
>> each child process within multiprocessing has an entire copy of the
>> parent, which gradually leads to copy-on-write differences, even for
>> supposedly read-only operations, given that python uses reference
>> counting (since all of the refcounts are constantly going through
>> inc/dec operations, so pages rapidly get written to).
> Ouch. Though I'm not sure this explains the issue with the example
> script: the child process does nothing, so it states should not change
> with time. It appeared to me that the pages became dirty
> instantaneously. Am I missing smething?
>
>>
>>> grep Shared_Dirty /proc/22986/smaps | awk '{s=s+$2} END {print s}'
>>> 11272
>>>
>>> top:
>>> PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
>>> 22985 mgoldboi 20 0 184m 14m 2908 S 0.0 0.4 0:00.04 python
>>> 22986 mgoldboi 20 0 184m 12m 616 S 0.0 0.3 0:00.00 python
>>>
>>> is it a known issue? can we do something about it?
>> How much do you need the child to inherit state from the parent? How
>> long-lived is the child? In theory, I guess you could start "del"-ing
>> things within the child, and run a garbage collection first. But if you
>> merely need a subprocess, could you use the "subprocess" module instead?
>> (which does an "exec" after the fork, so you're starting with a clean
>> slate).
> We used to have our own inter-process messaging with Python 2.4, and
> where very happy to ditch it in favor of multiprocessing... I am not
> sure we can go back to that. Saggi, can you find a way around this?
>
>> FWIW I wrote a tool to analyse python memory usage: gdb-heap; see
>> http://fedoraproject.org/wiki/Features/MemoryDebuggingTools
>> However, it is unpolished and (ironically) very memory-hungry itself.
>>
>> Hope this is helpful
> It is, in an unhappy way.
>
> Thanks,
> Dan.
12 years, 9 months
Vdsm development becoming open
by Dan Kenigsberg
Hi,
Vdsm, which is the node management daemon for Red Hat Enterprise
Virtualization, has been released under GPLv2+ since ever.
Its development, however, was confined within Red Hat. Now this is
changing: we have a public git repository [1], and several mailing lists:
- vdsm-devel [2], for technical discussions about the project and its
code base.
- vdsm-users [3], for questions by users, who do not want to be
swamped by technicalities.
- vdsm-patches [4], for submitting and reviewing changes to vdsm.
Most of us currently use an in-doors Gerrit [5] server to review our
patches. We strive to make this server publicly-accessible, but until
then, if you would like to improve vdsm, please git-send-email your
patches to vdsm-patches.
Best regards,
Dan.
[1] http://git.fedorahosted.org/git/?p=vdsm.git
[2] https://fedorahosted.org/mailman/listinfo/vdsm-devel
[3] https://fedorahosted.org/mailman/listinfo/vdsm-users
[4] https://fedorahosted.org/mailman/listinfo/vdsm-patches
[5] http://code.google.com/p/gerrit/
12 years, 9 months