autoqa-devel August 2010

autoqa-devel@lists.fedorahosted.org

7 participants
87 discussions

Re: [PATCH] add upgradepath test
by Kamil Paral 24 Aug '10

24 Aug '10

----- "James Laska" <jlaska(a)redhat.com> wrote: > > See attached patch to convey the stand-alone idea noted above. Ok, I see your point now. Yes, we can do it like that. We will create "stand-alone" script check_upgradepath.py, that will depend on autoqa (therefore be able to use autoqa library and repoinfo.conf), but can be run alone like: $ ./check_upgradepath.py --envrs blobby-0.9b-1.fc13 --kojitag dist-f13-updates I'm not sure how large the benefit is over calling it through the hook, but surely the CLI interface is a little simpler than: $ autoqa post-bodhi-update --title blobby-0.9b-1.fc13 --target-tag dist-f13-updates blobby-0.9b-1.fc13 --local -t upgradepath Both should achieve the same functionality.

4 5

[AutoQA] #217: autoqa job failures
by fedora-badges 24 Aug '10

24 Aug '10

#217: autoqa job failures --------------------+------------------------------------------------------- Reporter: jlaska | Owner: Type: defect | Status: new Priority: major | Milestone: Hot issues Component: tests | Version: 1.0 Keywords: | --------------------+------------------------------------------------------- After Kamil mentioned this earlier in the week, I noticed not many post- koji-build jobs were processing. Inspecting the logs, it appears they are failing .... For example, the following traceback comes from http://test1250.test.redhat.com/results/44857-autotest/brutus.test.redhat.c… {{{ 08/19 13:01:22 INFO | autotest:0913| FAIL rpmlint rpmlint timestamp=1282249190 localtime=Aug 19 16:19:50 Unhandled str: Unhandled TypeError: 'NoneType' object is unsubscriptable 08/19 13:01:22 INFO | autotest:0913| Traceback (most recent call last): 08/19 13:01:22 INFO | autotest:0913| File "/usr/share/autotest/common_lib/test.py", line 570, in _call_test_function 08/19 13:01:22 INFO | autotest:0913| return func(*args, **dargs) 08/19 13:01:22 INFO | autotest:0913| File "/usr/share/autotest/common_lib/test.py", line 279, in execute 08/19 13:01:22 INFO | autotest:0913| postprocess_profiled_run, args, dargs) 08/19 13:01:22 INFO | autotest:0913| File "/usr/share/autotest/common_lib/test.py", line 201, in _call_run_once 08/19 13:01:22 INFO | autotest:0913| self.run_once(*args, **dargs) 08/19 13:01:22 INFO | autotest:0913| File "/usr/share/autotest/site_tests/rpmlint/rpmlint.py", line 70, in run_once 08/19 13:01:22 INFO | autotest:0913| collection_name = repoinfo.getrepo_by_tag(kojitag)['collection_name'] 08/19 13:01:22 INFO | autotest:0913| TypeError: 'NoneType' object is unsubscriptable 08/19 13:01:22 INFO | autotest:0913| 08/19 13:01:22 INFO | autotest:0913| None 08/19 13:01:22 INFO | autotest:0913| END FAIL rpmlint rpmlint timestamp=1282249190 localtime=Aug 19 16:19:50 }}} This AutoQA instance is running the older 0.3 branch. I gather the tests have changed significantly since. -- Ticket URL: <https://fedorahosted.org/autoqa/ticket/217> AutoQA <http://autoqa.fedorahosted.org> Automated QA project

1 5

Re: [PATCH] check for previous releases even in the requested repo, not just in parents
by Kamil Paral 24 Aug '10

24 Aug '10

----- "James Laska" <jlaska(a)redhat.com> wrote: > On Fri, 2010-08-20 at 17:42 +0200, Kamil Páral wrote: > > --- > > lib/python/koji_utils.py | 51 > ++++++++++++++++++++------------------------- > > 1 files changed, 23 insertions(+), 28 deletions(-) > > > > diff --git a/lib/python/koji_utils.py b/lib/python/koji_utils.py > > index f8ac077..87b32e9 100644 > > --- a/lib/python/koji_utils.py > > +++ b/lib/python/koji_utils.py > > <snip> > > Thanks Kamil, > > As we discussed in the Fedora QA meeting today. I have applied this > patch to the running AutoQA instance and rescheduled several > previously > failed rpmguard jobs. I consider the results a PASS, but welcome any > additional tests you'd like me to run. > > First, a package that does *not* have previous builds ... > gtk-nodoka-engine-0.7.5-1.fc15 > * Without the patch, the job (#45485) failed (see > http://fpaste.org/cOJh/) > * With the patch, the job (#45506) runs to completion, but is > not > able to find a previous build to compare against in dist-f15. > This is accurate, looking in 'koji' I cannot find another > dist-f15 build to compare against > > (http://koji.fedoraproject.org/koji/packageinfo?packageID=4871) > > N: There is no previous build of > gtk-nodoka-engine-0.7.5-1.fc15 in parents of dist-f15 > tag. > > Next, a package that does have previous builds ... > schroot-1.4.2-5.fc14 > * Without patch - rpmguard result - > > https://fedorahosted.org/pipermail/autoqa-results/2010-July/028449.html > * With patch - rpmguard result - > > https://fedorahosted.org/pipermail/autoqa-results/2010-August/030026.html > > The results match, and the confirm what I'd expect to see based on > koji > builds > (http://koji.fedoraproject.org/koji/packageinfo?packageID=8929) Yes, I also believe that the behaviour is correct now. Thanks for testing it, James, and thanks for pushing it, Will. Closing ticket #217.

1 0

[PATCH] check for previous releases even in the requested repo, not just in parents
by Kamil Paral 24 Aug '10

24 Aug '10

--- lib/python/koji_utils.py | 51 ++++++++++++++++++++------------------------- 1 files changed, 23 insertions(+), 28 deletions(-) diff --git a/lib/python/koji_utils.py b/lib/python/koji_utils.py index f8ac077..87b32e9 100644 --- a/lib/python/koji_utils.py +++ b/lib/python/koji_utils.py @@ -23,6 +23,7 @@ import time import koji from repoinfo import repoinfo import rpmUtils.miscutils +import sys # XXX fetch from /etc/koji.conf, section 'koji' kojiurl = 'http://koji.fedoraproject.org/kojihub' @@ -101,40 +102,34 @@ class SimpleKojiClientSession(koji.ClientSession): '''Get most recent releases of package in specified tag (inheritence enabled). Returns dictionary in the format {tag: NVR object}. Optional max_evr says that all returned release versions must be lesser then - it. If you enable unstable_tags, then also 'testing' and 'candidate' - tags are considered. + it. If you enable unstable_tags, then also '-testing' tags are considered. + max_evr = (epoch, version, release) ''' prev_rlss = {} # minor hack - assume -candidate parents = stable+updates+testing # TODO: provide this info in repoinfo instead + # since there's no repo for candidate builds, just look at -testing and below if tag.endswith('-updates-candidate'): - testtag = tag.replace('-candidate','-testing') - testrepo = repoinfo.getrepo_by_tag(testtag) - # -testing parents = stable+updates. add -testing and we're good. - parents = testrepo['parents'] + [testrepo['name']] - else: - parents = repoinfo.getrepo_by_tag(tag)['parents'] - if parents: - for p in parents: - ptag = repoinfo.get(p,'tag') - if ('-testing' in ptag or '-candidate' in ptag) and \ - not unstable_tags: - # skip unstable tags - continue - prev_rlss[ptag] = self.latest_by_tag(ptag, name, max_evr=max_evr) - else: - # NOTE: for tags with no parents (e.g. rawhide) we obviously can't - # check the parent repos' tags to get the previously released - # package NVRs. Instead we use koji's tagHistory command to find - # the previous version of this package that had this tag. - hist = self.tagHistory(tag=tag, package=name) - # remove untagged builds - hist = filter(lambda h: h['revoke_event'] is None, hist) - # ensure sorting by time - hist.sort(key=lambda h: h['create_ts']) - # hist[-2] = not the currently-tagged build, the one before that - prev_rlss[tag] = self.getBuild(hist[-2]['build_id']) + tag = tag.replace('-candidate','-testing') + + testrepo = repoinfo.getrepo_by_tag(tag) + if testrepo is None: + print >> sys.stderr, "Error: There's no repo definition for tag '%s'" % tag + return {} + parents = testrepo['parents'] + repos = parents + [testrepo['name']] + + if not unstable_tags: + for repo in repos[:]: + rtag = repoinfo.get(repo, 'tag') + if ('-testing' in rtag or '-candidate' in rtag): + # remove repo with unstable tag + repos.remove(repo) + + for repo in repos: + rtag = repoinfo.get(repo, 'tag') + prev_rlss[rtag] = self.latest_by_tag(rtag, name, max_evr=max_evr) return prev_rlss def list_previous_release(self, name, tag, max_evr=None, -- 1.7.2.1

2 1

[PATCH] don't crash when repoinfo.getrepo_by_tag returns None
by Kamil Paral 20 Aug '10

20 Aug '10

--- lib/python/util.py | 3 +++ tests/initscripts/initscripts.py | 6 +++--- tests/rpmguard/rpmguard.py | 5 +++-- tests/rpmlint/rpmlint.py | 4 ++-- 4 files changed, 11 insertions(+), 7 deletions(-) diff --git a/lib/python/util.py b/lib/python/util.py index ed90ba4..b6d0013 100644 --- a/lib/python/util.py +++ b/lib/python/util.py @@ -96,6 +96,9 @@ def checkhash(filename, algo, hexdigest): def check_opt_in(pkgname, dist_release): """take a package name and distribution release and check for an autoqa-optin file""" + if not pkgname or not dist_release: + return False + url = 'http://fedorapeople.org/packages/' fullurl = url + '%s/%s/%s/autoqa-optin' % (pkgname[0].lower(), pkgname, dist_release) tmout = socket.getdefaulttimeout() diff --git a/tests/initscripts/initscripts.py b/tests/initscripts/initscripts.py index 0806107..1885003 100644 --- a/tests/initscripts/initscripts.py +++ b/tests/initscripts/initscripts.py @@ -152,9 +152,9 @@ class initscripts(AutoQATest): self.highlights = "Following tests returned zero exit code: %s\n" % repr(tests_exit_status_zero) self.highlights += "Following tests returned NON-zero exit code: %s\n" % repr(tests_exit_status_nonzero) - #add subscribed package owners to mail-to - collection_name = repoinfo.getrepo_by_tag(kojitag)['collection_name'] - if collection_name and autoqa.util.check_opt_in(name, collection_name): + # email results to mailing list and to pkg owner if they optin + repo = repoinfo.getrepo_by_tag(kojitag) + if repo is not None and autoqa.util.check_opt_in(name, repo['collection_name']): #FIXME - hardcoded partial email address here - obviously sub-par self.mail_to.append('%s-owner(a)fedoraproject.org' % name) diff --git a/tests/rpmguard/rpmguard.py b/tests/rpmguard/rpmguard.py index a0e7e74..b317089 100644 --- a/tests/rpmguard/rpmguard.py +++ b/tests/rpmguard/rpmguard.py @@ -188,8 +188,9 @@ rpmlint | cut -d ' ' -f 1", retain_output=True) self.summary = '%u warnings for package %s' % (warning_count, envr) self.outputs = log - collection_name = repoinfo.getrepo_by_tag(kojitag)['collection_name'] - if collection_name and autoqa.util.check_opt_in(name, collection_name): + # email results to mailing list and to pkg owner if they optin + repo = repoinfo.getrepo_by_tag(kojitag) + if repo is not None and autoqa.util.check_opt_in(name, repo['collection_name']): #FIXME - hardcoded partial email address here - obviously sub-par self.mail_to.append('%s-owner(a)fedoraproject.org' % name) diff --git a/tests/rpmlint/rpmlint.py b/tests/rpmlint/rpmlint.py index adcfb13..1189298 100644 --- a/tests/rpmlint/rpmlint.py +++ b/tests/rpmlint/rpmlint.py @@ -69,8 +69,8 @@ class rpmlint(AutoQATest): self.summary = '%s for package %s' % (summary, envr) # email results to mailing list and to pkg owner if they optin - collection_name = repoinfo.getrepo_by_tag(kojitag)['collection_name'] - if collection_name and autoqa.util.check_opt_in(name, collection_name): + repo = repoinfo.getrepo_by_tag(kojitag) + if repo is not None and autoqa.util.check_opt_in(name, repo['collection_name']): #FIXME - hardcoded partial email address here - obviously sub-par self.mail_to.append('%s-owner(a)fedoraproject.org' % name) -- 1.7.2.1

2 1

Re: [PATCH] add upgradepath test
by Kamil Paral 20 Aug '10

20 Aug '10

----- "Will Woods" <wwoods(a)redhat.com> wrote: > On Thu, 2010-08-19 at 14:58 -0400, James Laska wrote: > > I was interested in testing out just the upgrade-path portion of the > > test (not the autotest/autoqa integration). How would you feel > about > > moving the test to a stand-alone script, and then changing > > upgradepath.py to call the stand-alone script and process output? No, don't say this, Vojta will try to kill me! That is exactly how he started to implement that and I talked him out of it :) > > > > I think this would make it a lot easier for folks without a full > autoqa > > and autotest-server setup to run the test at home. > > I agree with James here - it's best if we can run the tests outside of > AutoQA, both for debugging purposes and so we can extend/reuse test > code > elsewhere. > There are a few reasons why I recommended Vojta to write it as a single AutoQA script: 1. This task is very short and simple, we can easily join the the test logic and AutoQA test object stuff. 2. This task requires access to almost whole repoinfo.conf. Using it as a standalone script would require keeping a copy of repoinfo.conf next to it or passing all those information with command line options (all those repo URLs and stuff, that wouldn't be pretty). 3. This task requires a lot of functionality that we already implemented in our autoqa python library, like querying koji for latest package with some tag, comparing EVR or easily working with repoinfo. Using a standalone script would still require autoqa library installed (and therefore whole autoqa effectively) or Vojta would have to re-implement all that stuff from scratch. Of course it is possible to have this as a standalone script, but it seemed extremely cumbersome to me and I discouraged Vojta from doing that. The code would be twice as long, handling many options passing and re-implementing some autoqa code. But maybe I'm just looking at it all wrong. Tell me what you think.

2 2

[PATCH] add upgradepath test
by Vojtěch Aschenbrenner 20 Aug '10

20 Aug '10

Hello, I created new test for AutoQA named 'upgradepath'. It will check if package which you want to push into repository has correct version. Also there will be warning if you're pushing into stable repo or pushing older or the same version of package. Requests for comments :-). --- diff --git a/tests/upgradepath/control b/tests/upgradepath/control new file mode 100644 index 0000000..997c81e --- /dev/null +++ b/tests/upgradepath/control @@ -0,0 +1,14 @@ +# vim: set syntax=python +# Notice: Most recent documentation is available at doc/control.template. +TIME="SHORT" +AUTHOR = "Vojtech Aschenbrenner <vaschenb(a)redhat.com>" +DOC = """ +This test runs upgradepath to catch problems with package versions +in repositories. +""" +NAME = 'upgradepath' +TEST_TYPE = 'CLIENT' # SERVER can be used for tests that need multiple machines +TEST_CLASS = 'General' +TEST_CATEGORY = 'Functional' + +job.run_test('upgradepath', config=autoqa_conf, **autoqa_args) diff --git a/tests/upgradepath/control.autoqa b/tests/upgradepath/control.autoqa new file mode 100644 index 0000000..6b2ea92 --- /dev/null +++ b/tests/upgradepath/control.autoqa @@ -0,0 +1,9 @@ +# vim: set syntax=python +# Notice: Most recent documentation is available at doc/control.autoqa.template. + +# upgradepath can be run just once and on any architecture +archs = ['noarch'] + +# do not run automaticaly +execute = False + diff --git a/tests/upgradepath/upgradepath.py b/tests/upgradepath/upgradepath.py new file mode 100755 index 0000000..8c4547e --- /dev/null +++ b/tests/upgradepath/upgradepath.py @@ -0,0 +1,147 @@ +#!/usr/bin/python +# +# Copyright 2010, Red Hat, Inc. +# +# This program is free software; you can redistribute it and/or modify +# it under the terms of the GNU General Public License as published by +# the Free Software Foundation; either version 2 of the License, or +# (at your option) any later version. +# +# This program is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +# GNU General Public License for more details. +# +# You should have received a copy of the GNU General Public License along +# with this program; if not, write to the Free Software Foundation, Inc., +# 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA. +# +# Author: Vojtech Aschenbrenner <vaschenb(a)redhat.com> + +import os +import sys +import operator +import autoqa.koji_utils +import autoqa.util +import rpmUtils.miscutils + +from autoqa.test import AutoQATest +from autoqa.decorators import ExceptionCatcher +from autotest_lib.client.bin.test_config import config_loader +from autotest_lib.client.common_lib import error + + +class upgradepath(AutoQATest): + version = 1 # increment if setup() changes + test_result = 'PASS' + log = [] + envr_list = set() + + def initialize(self, config): + self.config = config_loader(config, self.tmpdir) + #URL of logs/results stored on autotest-server + self.autotest_url = autoqa.util.make_autotest_url(self.config) + + def setup(self): + pass + + def compare(self, matching_build, tags, op): + koji = autoqa.koji_utils.SimpleKojiClientSession() + for tag in tags: + latest_build = koji.latest_by_tag(tag, matching_build['name']) + if latest_build is None: + msg = tag + '... OK' + self.log.append(msg) + print msg + + msg = "-> Package %s in %s doesn't exist" % (matching_build['name'], tag) + self.log.append(msg) + print msg + else: + result = autoqa.koji_utils.compareEVR(matching_build, latest_build) + + if op(result, 0): + msg = tag + '... OK' + print msg + self.log.append(msg) + else: + self.test_result = 'FAIL' + msg = tag + '... FAIL' + print msg + self.log.append(msg) + self.envr_list.add(matching_build['envr']) + msg = "-> Pushing package: %s\n-> Latest package: %s" % (matching_build['envr'], latest_build['nvr']) + self.log.append(msg) + print msg + + @ExceptionCatcher("self.run_once_failed") + def run_once(self, envrs, name, kojitag, **kwargs): + reponames = [reponame for reponame in autoqa.koji_utils.repoinfo.repos() if not reponame.endswith('-testing')] + repotags = [autoqa.koji_utils.repoinfo.getrepo(reponame)['tag'] for reponame in reponames] + repotags.sort() + try: + current_tag = repotags.index(kojitag) + except ValueError: + self.test_result = 'FAIL' + msg = "ERROR: Entered bad kojitag" + print msg + self.log.append(msg) + self.envr_list.add(matching_build['envr']) + raise error.TestFail + + hi_tags = repotags[(current_tag + 1):] # tags higher than current + low_eq_tags = repotags[:(current_tag + 1)] # tags lower or equal + + for envr in envrs: + msg = 40*'=' + '\n' + envr + '\n' + 40*'=' + self.log.append(msg) + print 40*'=' + print envr + print 40*'=' + + # our testing package + (name, version, release, epoch, arch) = rpmUtils.miscutils.splitFilename(envr) + matching_build = { + 'name': name, + 'version' : version, + 'release' : release + '.' + arch, + 'epoch' : epoch, + 'envr' : envr, + } + + if kojitag.find('updates') == -1 and repotags[current_tag] != repotags[-1]: + msg = "Warning: Pushing into stable repository" + self.log.append(msg) + print msg + + koji = autoqa.koji_utils.SimpleKojiClientSession() + latest_build = koji.latest_by_tag(kojitag, matching_build['name']) + if latest_build is not None: + result = autoqa.koji_utils.compareEVR(matching_build, latest_build) + if result != 1: + msg = "Warning: Pushing older or current version of package" + self.log.append(msg) + print msg + + # compare with lower or equal tags, so version has to be greater or equal + self.compare(matching_build, low_eq_tags, operator.ge) + # compare with higher tags, so version has to be lower or equal + self.compare(matching_build, hi_tags, operator.le) + + # mailing stuff + self.result = self.test_result + + self.outputs = "" + for i in self.log: + self.outputs += i + '\n' + self.outputs += '\n \n' + + packages_fail = "" + for i in self.envr_list: + packages_fail += i + ', ' + packages_fail = packages_fail[:-2] + self.summary = packages_fail + + if self.test_result == 'FAIL': + raise error.TestFail +

4 3

[AutoQA] #199: Standardize (some) AutoQA test args
by fedora-badges 19 Aug '10

19 Aug '10

#199: Standardize (some) AutoQA test args -----------------------+---------------------------------------------------- Reporter: wwoods | Owner: Type: task | Status: new Priority: major | Milestone: Multi-hook tests Component: docs/wiki | Version: 1.0 Keywords: | -----------------------+---------------------------------------------------- The various hooks that AutoQA provides each have different test args. Some of these arguments are similar or identical across hooks - for example, 'config' applies to all hooks. Each hook also has an argument that represents the name of the thing to be tested, but this varies from hook to hook - it's a reponame for post-repo-update, treename for post-tree- compose, etc. We should document and standardize these argument names so that it's easier to write tests that will run for multiple hooks. For example, maybe all hooks should use 'name' to represent the name of the thing being tested. -- Ticket URL: <https://fedorahosted.org/autoqa/ticket/199> AutoQA <http://autoqa.fedorahosted.org> Automated QA project

1 3

depcheck - next steps!
by Will Woods 18 Aug '10

18 Aug '10

Time for a Depcheck Roundup! Whee! So now that I've messed with the unittests a bit and proven to myself that yes, depcheck does what it's supposed to (at least for *trivial* cases) it's time to look at the next steps: == write control/control.autoqa/test wrapper object == I took a stab at this in the depcheck branch but I'm pretty sure I did it wrong / left some stuff out. Kamil/Joza, if you guys can give me some help/pointers here I'd appreciate it... but don't worry, I'll send the patch to the list for a final review before I merge anything. == integrate mash == Basically, we need to be using the 'mash' tool rather than 'createrepo' to create the repos we use when testing updates, otherwise we'll miss anything involving multilib (like the infamous nss-softokn bug). I'm not at all sure how mash works - does anyone want to volunteer to figure this out? == more test cases == Once we're using mash instead of createrepo, we'll want to a) write a unittest or two to be sure mash is doing what it's supposed to, and b) write some depcheck unittests that prove depcheck handles multilib problems (and normal multilib packages) properly. == Get it running in autoqa! == Technically, we could start running depcheck *before* we finish the mash work - we just would ignore all multilib issues. I feel like we should wait until depcheck is working *correctly* before wiring it up, but maybe it would be helpful to have it partially working? What do you all think? == And beyond.. == Once we're satisfied that depcheck is working and doing the right thing.. then we actually talk to rel-eng and infrastructure about wiring up the proper bits so we can *reject* packages that fail depcheck. Exciting stuff! Expect flamewars! Just to state the obvious: we're probably not going to have depcheck working by F14Alpha. Oh well. But I am undaunted - we're getting reeally close. Anything else depcheck-related that I'm forgetting about here? I need to review the trac tickets, I think.. -w

1 0

some unittests in depcheck branch
by Will Woods 18 Aug '10

18 Aug '10

Hi, all - Last week we talked about writing unittests for depcheck - Kamil and I tried to list a few required test cases, and I did a little experimenting with the unittest module in the depcheck branch. Here's an example test: def test_depcheck_good_update(self): '''Make sure depcheck accepts a good update as expected''' (n, v, r) = ('mint', '2.0', '1') # twice as minty as mint 1.0! p = rpmfluff.SimpleRpmBuild(n,v,r) p.make() (rv, problems, ok_packages) = depcheck_main(self.mainrepo.id, [p.get_built_rpm(self.arch)]) self.assertEqual(len(problems), 1) self.assertTrue(problems[0].startswith(u'Success')) okp = ok_packages[0] self.assertEqual((n,v,r),(okp.name, okp.version, okp.release)) self.assertEqual(rv, 2) p.clean() (there's a setUp method that creates a very simple repo which contains three packages - 'julep', which depends on 'mint' and 'bourbon = 1.0') Kamil, is this the sort of thing you were thinking of? I think this makes sense but I'd love some input from the rest of you. If you have a chance, check out the 'depcheck' branch and have a look at the unit tests, in tests/depcheck/depcheck.py. I'd love suggestions / more unittests / whatever. Thanks! -w

1 0

← Newer
1
2
3
4
5
6
7
8
9
Older →

2026

2025

2024

2023

2022

2021

2020

2019

2018

2017

2016

2015

2014

2013

2012

2011

2010

2009

autoqa-devel August 2010