using iterators to reduce memory use

[Mike McLean]

These changes cause the hub to use generators for some potentially large
queries internally. The point is to reduce memory usage.

In the case that a large query is returned via rpc, the marshaller has
been extended to handle generators. The marshaller will still create a
giant pile of xml, but we at least still save the memory for the
original data.

I also have some work on an iterating marshaller that will not keep the
entire xml return in memory, but that's a bit more complicated, so I'm
going to leave that for another day.

There are other cases (e.g. repo_init) where we have very large queries
internally. In these cases, this patch set will result in significant
memory savings.

-------------- next part --------------
A non-text attachment was scrubbed...
Name: 0001-Use-iterators-for-potentially-large-queries-to-reduc.patch
Type: text/x-patch
Size: 11170 bytes
Desc: not available
URL: <http://lists.fedoraproject.org/pipermail/buildsys/attachments/20140729/c15bd15b/attachment-0011.bin>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: 0002-use-iteration-in-readTaggedRPMS.patch
Type: text/x-patch
Size: 3303 bytes
Desc: not available
URL: <http://lists.fedoraproject.org/pipermail/buildsys/attachments/20140729/c15bd15b/attachment-0012.bin>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: 0003-rework-repo_init-for-iterator-efficiency.patch
Type: text/x-patch
Size: 7140 bytes
Desc: not available
URL: <http://lists.fedoraproject.org/pipermail/buildsys/attachments/20140729/c15bd15b/attachment-0013.bin>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: 0004-QueryProcessor-iteration-refinements.patch
Type: text/x-patch
Size: 2674 bytes
Desc: not available
URL: <http://lists.fedoraproject.org/pipermail/buildsys/attachments/20140729/c15bd15b/attachment-0014.bin>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: 0005-limit-request-size.patch
Type: text/x-patch
Size: 1262 bytes
Desc: not available
URL: <http://lists.fedoraproject.org/pipermail/buildsys/attachments/20140729/c15bd15b/attachment-0015.bin>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: 0006-use-pid-in-cursor-name.patch
Type: text/x-patch
Size: 941 bytes
Desc: not available
URL: <http://lists.fedoraproject.org/pipermail/buildsys/attachments/20140729/c15bd15b/attachment-0016.bin>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: 0007-fix-variable-name.patch
Type: text/x-patch
Size: 1021 bytes
Desc: not available
URL: <http://lists.fedoraproject.org/pipermail/buildsys/attachments/20140729/c15bd15b/attachment-0017.bin>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: 0008-fix-to-address-for-gc-notices-ticket-268.patch
Type: text/x-patch
Size: 830 bytes
Desc: not available
URL: <http://lists.fedoraproject.org/pipermail/buildsys/attachments/20140729/c15bd15b/attachment-0018.bin>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: 0009-return-a-generator-from-maven_tag_archives.patch
Type: text/x-patch
Size: 6040 bytes
Desc: not available
URL: <http://lists.fedoraproject.org/pipermail/buildsys/attachments/20140729/c15bd15b/attachment-0019.bin>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: 0010-handle-maven_tag_archives-generator-in-updateMavenBu.patch
Type: text/x-patch
Size: 4164 bytes
Desc: not available
URL: <http://lists.fedoraproject.org/pipermail/buildsys/attachments/20140729/c15bd15b/attachment-0020.bin>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: 0011-chunksize-option-for-iterate.patch
Type: text/x-patch
Size: 2737 bytes
Desc: not available
URL: <http://lists.fedoraproject.org/pipermail/buildsys/attachments/20140729/c15bd15b/attachment-0021.bin>