Re: using iterators to reduce memory use

[Mike McLean]

On 07/29/2014 05:43 PM, Mike McLean wrote:
>  These changes cause the hub to use generators for some potentially
> large
>  queries internally. The point is to reduce memory usage.
>  
>  In the case that a large query is returned via rpc, the marshaller has
>  been extended to handle generators. The marshaller will still create a
>  giant pile of xml, but we at least still save the memory for the
>  original data.
>  
>  I also have some work on an iterating marshaller that will not keep the
>  entire xml return in memory, but that's a bit more complicated, so I'm
>  going to leave that for another day.
>  
>  There are other cases (e.g. repo_init) where we have very large queries
>  internally. In these cases, this patch set will result in significant
>  memory savings. 
A few more related changes, additional logging and a bugfix.

Also, will probably drop the chunksize patch, doesn't seem to matter
when unless the db connection has a lot of lag.

These changes are working well for me and are likely to land in git soon.