using iterators to reduce memory use

[Mike Bonnet]

On 7/30/14, 2:32 PM, Mike McLean wrote:
> On 07/29/2014 05:43 PM, Mike McLean wrote:
>> These changes cause the hub to use generators for some potentially large
>> queries internally. The point is to reduce memory usage.
>>
>> In the case that a large query is returned via rpc, the marshaller has
>> been extended to handle generators. The marshaller will still create a
>> giant pile of xml, but we at least still save the memory for the
>> original data.
>>
>> I also have some work on an iterating marshaller that will not keep the
>> entire xml return in memory, but that's a bit more complicated, so I'm
>> going to leave that for another day.
>>
>> There are other cases (e.g. repo_init) where we have very large queries
>> internally. In these cases, this patch set will result in significant
>> memory savings.
>
> A few more related changes, additional logging and a bugfix.
>
> Also, will probably drop the chunksize patch, doesn't seem to matter
> when unless the db connection has a lot of lag.
>
> These changes are working well for me and are likely to land in git soon.

Here's a small patch to fix chain-maven, which can override a G:A:V in 
the tag with a build from the chain.  Tested locally and it fixes the 
regression in chain-maven builds for me.

-------------- next part --------------
From 7f391b08bf04e3872c57ede21d651cfb7c4f957b Mon Sep 17 00:00:00 2001
From: Mike Bonnet <mikeb at redhat.com>
Date: Thu, 31 Jul 2014 19:39:32 -0400
Subject: [PATCH] maven-chain builds can override a G:A:V from the tag with a
 previous build in the chain, update the index accordingly

---
 hub/kojihub.py | 5 +++--
 1 file changed, 3 insertions(+), 2 deletions(-)

diff --git a/hub/kojihub.py b/hub/kojihub.py
index 62220cf..ed17f23 100644
--- a/hub/kojihub.py
+++ b/hub/kojihub.py
@@ -10428,8 +10428,9 @@ class HostExports(object):
                                 archive['artifact_id'], {}).setdefault(
                                     archive['version'], archive['build_id'])
                     if idx_build != archive['build_id']:
-                        logger.error("Found multiple builds for %(group_id)s:%(artifact_id)s:%(version)s. Current build: %(build_id)i", archive)
-                        logger.error("Indexed build id was %i", idx_build)
+                        logger.error("Overriding build for %(group_id)s:%(artifact_id)s:%(version)s.", archive)
+                        logger.error("Current build is %s, new build is %s.", idx_build, archive['build_id'])
+                        maven_build_index[archive['group_id']][archive['artifact_id']][archive['version']] = archive['build_id']
 
         ignore.extend(task_deps.values())
 
-- 
1.8.2.1