Outrageous memory usage with cElementTree

[Shahms King]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Michael Schwendt wrote:
> On Fri, 07 Apr 2006 13:38:23 -0700, Shahms King wrote:
> 
>> I was trying to figure out why my small python application was using
>> outrageous amounts of memory on FC5 and managed to track it down to a
>> single anonymous rw mapping that's 32380K.  It only gets mapped after
>> importing cElementTree but I can't figure out why.  It doesn't happen
>> under FC4 or FC3.  Is this memory actually used or is it another
>> artifact of Linux's often questionable RSS statistics?  32MB is a lot of
>> memory to be using for no apparent reason, especially as it is
>> per-process unshared memory.  It's particularly vexing using mod_python
>> and kid (which uses cElementTree, if available) as it means that every
>> single httpd process uses ~50MB base memory, which is ridiculous.
> 
> With FC5 and Python repoquery I'm facing increased heap space
> usage by a factor of 7-10 compared with FC4 or older.
> 
> Heap size of the repoquery process grows quickly to and beyond
> 
>   Size:            586796 kB
>   Rss:             540100 kB
> 
> compared with FC4:
> 
>   Size:             77632 kB
>   Rss:              77628 kB
> 
> I've stopped looking into it, but Yum is not the culprit and SQLite
> is not either.
> 

Thanks for the response, at least I know I'm not going crazy ;-)

I'm looking into it more deeply.  Bug #188327 (currently on the
python-elementtree component) give a little more detail.  I doubt that
elementtree is to blame, but it's the only package I've found so far
that triggers the problem. Specifically, elementtree.ElementTree causes
the jump in memory usage.  I've copied ElementTree.py into its own file
and am trying to figure out what, exactly, causes it as the module
doesn't appear to do anything special...

- --
Shahms E. King <shahms at shahms.com>
Multnomah ESD

Public Key:
http://shahms.mesd.k12.or.us/~sking/shahms.asc
Fingerprint:
1612 054B CE92 8770 F1EA  AB1B FEAB 3636 45B2 D75B
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.2.2 (GNU/Linux)
Comment: Using GnuPG with Fedora - http://enigmail.mozdev.org

iD8DBQFENuzI/qs2NkWy11sRAiG4AJ0eMXz6m0LNN8kdQV52unorW854TwCgvDFd
prETxyVeU8KRT/onvWKqUlk=
=62n9
-----END PGP SIGNATURE-----