Enable post-koji-build for rawhide (dist-f14)

[James Laska]

Greetings,

Currently, post-koji-build monitors for new builds in 'dist-f14-update-candidate'.
 Until Fedora 14 is branched (in a few months), all rawhide/f14 builds are tagged with
'dist-f14'.  The following patches change post-koji-build to use the contents of
repoinfo.conf instead of hard coding the tags and architectures to search.

By enabling tests where there may not be a parent repo, some tests may fail to find a
previous package (aka rpmguard).  I've also added a check in koji_utils.py
list_previous_releases() to handle the condition when no parent repo is found.  Will and I
discussed some techniques for addressing this in subsequent patches.

While in the post-koji-build watcher, I also knocked out a FIXME to add a --prevtime
command-line option.  This proved helpful for testing.

Comments appreciated.

Thanks,
James

[PATCH 1/2] Check if parent_for_tag has key before accessing. Add FIXME for future improvement

[James Laska]

---
 lib/python/koji_utils.py |   20 +++++++++++++-------
 1 files changed, 13 insertions(+), 7 deletions(-)

diff --git a/lib/python/koji_utils.py b/lib/python/koji_utils.py
index c9fad5d..3a1039a 100644
--- a/lib/python/koji_utils.py
+++ b/lib/python/koji_utils.py
@@ -114,13 +114,19 @@ class SimpleKojiClientSession(koji.ClientSession):
         max_evr = (epoch, version, release)
         '''
         prev_rlss = {}
-        for p in parents_for_tag[tag]:
-            ptag = repoinfo.get(p,'tag')
-            if ('-testing' in ptag or '-candidate' in ptag) and \
-            not unstable_tags:
-                # skip unstable tags
-                continue
-            prev_rlss[ptag] = self.latest_by_tag(ptag, name, max_evr=max_evr)
+        if parents_for_tag.has_key(tag):
+            for p in parents_for_tag[tag]:
+                ptag = repoinfo.get(p,'tag')
+                if ('-testing' in ptag or '-candidate' in ptag) and \
+                not unstable_tags:
+                    # skip unstable tags
+                    continue
+                prev_rlss[ptag] = self.latest_by_tag(ptag, name, max_evr=max_evr)
+        else:
+            '''FIXME - rawhide doesn't have a parent repo.  In order to
find
+            the previously released package, may need to query the rawhide
+            repo, or use koji tag history'''
+
         return prev_rlss
 
     def list_previous_release(self, name, tag, max_evr=None,
-- 
1.7.0.1

[PATCH 2/2] Use repoinfo to get list of tags and arches to watch for. Add prevtime command-line argument.

[James Laska]

---
 hooks/post-koji-build/watch-koji-builds.py |   51 +++++++++++++++------------
 1 files changed, 28 insertions(+), 23 deletions(-)

diff --git a/hooks/post-koji-build/watch-koji-builds.py
b/hooks/post-koji-build/watch-koji-builds.py
index ae12471..62aa0e6 100755
--- a/hooks/post-koji-build/watch-koji-builds.py
+++ b/hooks/post-koji-build/watch-koji-builds.py
@@ -18,6 +18,7 @@ import xmlrpclib
 import subprocess
 import optparse
 from autoqa import koji_utils
+from autoqa.repoinfo import repoinfo
 
 # Directory where we cache info between runs
 cachedir = '/var/cache/autoqa/watch-koji-builds'
@@ -31,21 +32,14 @@ except OSError, e:
 #     alternately: read from e.g. /etc/autoqa/autoqa.conf
 kojiserver = 'http://koji.fedoraproject.org/kojihub'
 
-# Set of tags to look for
-taglist = set(('dist-f10-updates-candidate',
-               'dist-f11-updates-candidate',
-               'dist-f12-updates-candidate',
-               'dist-f13-updates-candidate',
-               'dist-f14-updates-candidate',
-))
-archlist = ('i686', 'x86_64', 'noarch')
-
-
 def get_prevtime():
-    try: 
+    '''Returns the prevtime to use when looking for new builds.  Value will
be
+       either 1) the mtime of the cachefile, or 2) if no cachefile is found, current
+       time - 3 hours.'''
+    try:
         return os.stat(cachefile)[8] # mtime
     except:
-        return 0
+        return time.time() - 10800
 
 def load_untagged():
     '''Load the list of untagged builds seen by previous runs'''
@@ -64,6 +58,7 @@ def save_list(pkglist):
     out.close()
 
 def new_builds_since(session, taglist, prevtime):
+
     untagged_builds = []
     tagged_builds = {}
     for tag in taglist:
@@ -79,8 +74,9 @@ def new_builds_since(session, taglist, prevtime):
         elif not session.build_failed(nvr):
             # No tag yet but the build is still OK - check again later 
             untagged_builds.append(nvr)
-    #print "untagged: %s" % " ".join(untagged_builds)
-    #print "tagged: %s" % " ".join(tagged_builds)
+    if opts.verbose:
+        print "untagged: %s" % " ".join(untagged_builds)
+        print "tagged: %s" % " ".join(tagged_builds)
     return (tagged_builds, untagged_builds)
 
 if __name__ == '__main__':
@@ -90,17 +86,22 @@ if __name__ == '__main__':
 tags for new builds and kick off tests when new builds/packages are found.')
     parser.add_option('--dryrun', '--dry-run',
action='store_true',
         help='Do not actually execute commands, just show what would be done')
+    parser.add_option('-p', '--prevtime', action='store',
+        type='float', default=get_prevtime(),
+        help='How far back to look for new builds (seconds since last epoch)')
     parser.add_option('--verbose', action='store_true',
         help='Print extra information')
     (opts, args) = parser.parse_args()
 
-    # load prevtime from some saved timestamp
-    prevtime = get_prevtime()
-    if not prevtime:
-        print "No previous run - checking builds in the past 3 hours"
-        prevtime = time.time() - 10800
+    # Using repoinfo, establish the set of tags to look for
+    taglist = set()
+    for repo in repoinfo.repos():
+        taglist.add(repoinfo.get(repo, 'tag'))
+
     if opts.verbose:
-        print "Looking up builds since %s" % time.ctime(prevtime)
+        print "Looking up builds since %s (%s)" % (opts.prevtime,
time.ctime(opts.prevtime))
+        print "Looking for builds with tags %s" % " ".join(taglist)
+
     # Set up the koji connection
     kojiopts = {} # Possible items: user, password, debug_xmlrpc, debug..
     session = koji_utils.SimpleKojiClientSession(kojiserver, kojiopts)
@@ -109,7 +110,7 @@ tags for new builds and kick off tests when new builds/packages are
found.')
     try:
         session.ensure_connection()
         (tagged_builds, untagged_builds) = new_builds_since(session,
-                                            taglist, prevtime)
+                                            taglist, opts.prevtime)
         if not opts.dryrun:
             save_list(untagged_builds)
         for tag, builds in tagged_builds.items():
@@ -119,12 +120,16 @@ tags for new builds and kick off tests when new builds/packages are
found.')
                 harnesscall = ['autoqa', 'post-koji-build',
                                '--name', b['name'],
                                '--kojitag', tag]
+
                 # Invoke autoqa with an --arch arg for each arch we care about
-                testarches = set(arches).intersection(archlist)
+                repoarches = set(repoinfo.getrepo_by_tag(tag).get("arches"))
+                testarches = set(repoarches).intersection(arches)
+
                 # NOTE HACK: autoqa doesn't handle 'noarch', use
'x86_64'
-                if 'noarch' in testarches:
+                if 'noarch' in arches:
                     testarches.discard('noarch')
                     testarches.add('x86_64')
+
                 for arch in testarches:
                     harnesscall += ['--arch', arch]
                 if b['epoch']:
-- 
1.7.0.1

[PATCH 1/2] Check if parent_for_tag has key before accessing. Add FIXME for future improvement

[Will Woods]

On Thu, 2010-05-27 at 11:56 -0400, James Laska wrote:
>  +        if parents_for_tag.has_key(tag): 
Unfortunately parents_for_tag is a hardcoded dict in koji_utils.py
(boo!), so this will fail for F13, because it doesn't have
dist-f13-updates-candidate in it yet.

I'm working on another patch that fixes that; I'll combine it with this
patch and send something out in a bit.

-w

[PATCH 2/2] Use repoinfo to get list of tags and arches to watch for. Add prevtime command-line argument.

[Will Woods]

On Thu, 2010-05-27 at 11:56 -0400, James Laska wrote:
>  ---
>   hooks/post-koji-build/watch-koji-builds.py |   51 +++++++++++++++------------
>   1 files changed, 28 insertions(+), 23 deletions(-) 
This patch all looks fine to me. Thanks for fixing up that hardcoded
taglist nonsense.

-w

[PATCH 2/2] Use repoinfo to get list of tags and arches to watch for. Add prevtime command-line argument.

[James Laska]

On Thu, 2010-05-27 at 17:44 -0400, Will Woods wrote:
>  On Thu, 2010-05-27 at 11:56 -0400, James Laska wrote:
>  > ---
>  >  hooks/post-koji-build/watch-koji-builds.py |   51 +++++++++++++++------------
>  >  1 files changed, 28 insertions(+), 23 deletions(-)
>  
>  This patch all looks fine to me. Thanks for fixing up that hardcoded
>  taglist nonsense. 
Thanks for the feedback, pushed to master

http://git.fedorahosted.org/git/?p=autoqa.git;a=commit;h=580734d4f2aa768a...

Thanks,
James