August 2010 - autoqa-devel - Fedora mailing-lists

Re: [PATCH] add upgradepath test
by Kamil Paral 24 Aug '10

24 Aug '10

----- "James Laska" <jlaska(a)redhat.com> wrote: > > See attached patch to convey the stand-alone idea noted above. Ok, I see your point now. Yes, we can do it like that. We will create "stand-alone" script check_upgradepath.py, that will depend on autoqa (therefore be able to use autoqa library and repoinfo.conf), but can be run alone like: $ ./check_upgradepath.py --envrs blobby-0.9b-1.fc13 --kojitag dist-f13-updates I'm not sure how large the benefit is over calling it through the hook, but surely the CLI interface is a little simpler than: $ autoqa post-bodhi-update --title blobby-0.9b-1.fc13 --target-tag dist-f13-updates blobby-0.9b-1.fc13 --local -t upgradepath Both should achieve the same functionality.

4 5

[AutoQA] #217: autoqa job failures
by fedora-badges 24 Aug '10

24 Aug '10

#217: autoqa job failures --------------------+------------------------------------------------------- Reporter: jlaska | Owner: Type: defect | Status: new Priority: major | Milestone: Hot issues Component: tests | Version: 1.0 Keywords: | --------------------+------------------------------------------------------- After Kamil mentioned this earlier in the week, I noticed not many post- koji-build jobs were processing. Inspecting the logs, it appears they are failing .... For example, the following traceback comes from http://test1250.test.redhat.com/results/44857-autotest/brutus.test.redhat.c… {{{ 08/19 13:01:22 INFO | autotest:0913| FAIL rpmlint rpmlint timestamp=1282249190 localtime=Aug 19 16:19:50 Unhandled str: Unhandled TypeError: 'NoneType' object is unsubscriptable 08/19 13:01:22 INFO | autotest:0913| Traceback (most recent call last): 08/19 13:01:22 INFO | autotest:0913| File "/usr/share/autotest/common_lib/test.py", line 570, in _call_test_function 08/19 13:01:22 INFO | autotest:0913| return func(*args, **dargs) 08/19 13:01:22 INFO | autotest:0913| File "/usr/share/autotest/common_lib/test.py", line 279, in execute 08/19 13:01:22 INFO | autotest:0913| postprocess_profiled_run, args, dargs) 08/19 13:01:22 INFO | autotest:0913| File "/usr/share/autotest/common_lib/test.py", line 201, in _call_run_once 08/19 13:01:22 INFO | autotest:0913| self.run_once(*args, **dargs) 08/19 13:01:22 INFO | autotest:0913| File "/usr/share/autotest/site_tests/rpmlint/rpmlint.py", line 70, in run_once 08/19 13:01:22 INFO | autotest:0913| collection_name = repoinfo.getrepo_by_tag(kojitag)['collection_name'] 08/19 13:01:22 INFO | autotest:0913| TypeError: 'NoneType' object is unsubscriptable 08/19 13:01:22 INFO | autotest:0913| 08/19 13:01:22 INFO | autotest:0913| None 08/19 13:01:22 INFO | autotest:0913| END FAIL rpmlint rpmlint timestamp=1282249190 localtime=Aug 19 16:19:50 }}} This AutoQA instance is running the older 0.3 branch. I gather the tests have changed significantly since. -- Ticket URL: <https://fedorahosted.org/autoqa/ticket/217> AutoQA <http://autoqa.fedorahosted.org> Automated QA project

1 5

Re: [PATCH] check for previous releases even in the requested repo, not just in parents
by Kamil Paral 24 Aug '10

24 Aug '10

----- "James Laska" <jlaska(a)redhat.com> wrote: > On Fri, 2010-08-20 at 17:42 +0200, Kamil Páral wrote: > > --- > > lib/python/koji_utils.py | 51 > ++++++++++++++++++++------------------------- > > 1 files changed, 23 insertions(+), 28 deletions(-) > > > > diff --git a/lib/python/koji_utils.py b/lib/python/koji_utils.py > > index f8ac077..87b32e9 100644 > > --- a/lib/python/koji_utils.py > > +++ b/lib/python/koji_utils.py > > <snip> > > Thanks Kamil, > > As we discussed in the Fedora QA meeting today. I have applied this > patch to the running AutoQA instance and rescheduled several > previously > failed rpmguard jobs. I consider the results a PASS, but welcome any > additional tests you'd like me to run. > > First, a package that does *not* have previous builds ... > gtk-nodoka-engine-0.7.5-1.fc15 > * Without the patch, the job (#45485) failed (see > http://fpaste.org/cOJh/) > * With the patch, the job (#45506) runs to completion, but is > not > able to find a previous build to compare against in dist-f15. > This is accurate, looking in 'koji' I cannot find another > dist-f15 build to compare against > > (http://koji.fedoraproject.org/koji/packageinfo?packageID=4871) > > N: There is no previous build of > gtk-nodoka-engine-0.7.5-1.fc15 in parents of dist-f15 > tag. > > Next, a package that does have previous builds ... > schroot-1.4.2-5.fc14 > * Without patch - rpmguard result - > > https://fedorahosted.org/pipermail/autoqa-results/2010-July/028449.html > * With patch - rpmguard result - > > https://fedorahosted.org/pipermail/autoqa-results/2010-August/030026.html > > The results match, and the confirm what I'd expect to see based on > koji > builds > (http://koji.fedoraproject.org/koji/packageinfo?packageID=8929) Yes, I also believe that the behaviour is correct now. Thanks for testing it, James, and thanks for pushing it, Will. Closing ticket #217.

1 0

[PATCH] check for previous releases even in the requested repo, not just in parents
by Kamil Paral 23 Aug '10

23 Aug '10

--- lib/python/koji_utils.py | 51 ++++++++++++++++++++------------------------- 1 files changed, 23 insertions(+), 28 deletions(-) diff --git a/lib/python/koji_utils.py b/lib/python/koji_utils.py index f8ac077..87b32e9 100644 --- a/lib/python/koji_utils.py +++ b/lib/python/koji_utils.py @@ -23,6 +23,7 @@ import time import koji from repoinfo import repoinfo import rpmUtils.miscutils +import sys # XXX fetch from /etc/koji.conf, section 'koji' kojiurl = 'http://koji.fedoraproject.org/kojihub' @@ -101,40 +102,34 @@ class SimpleKojiClientSession(koji.ClientSession): '''Get most recent releases of package in specified tag (inheritence enabled). Returns dictionary in the format {tag: NVR object}. Optional max_evr says that all returned release versions must be lesser then - it. If you enable unstable_tags, then also 'testing' and 'candidate' - tags are considered. + it. If you enable unstable_tags, then also '-testing' tags are considered. + max_evr = (epoch, version, release) ''' prev_rlss = {} # minor hack - assume -candidate parents = stable+updates+testing # TODO: provide this info in repoinfo instead + # since there's no repo for candidate builds, just look at -testing and below if tag.endswith('-updates-candidate'): - testtag = tag.replace('-candidate','-testing') - testrepo = repoinfo.getrepo_by_tag(testtag) - # -testing parents = stable+updates. add -testing and we're good. - parents = testrepo['parents'] + [testrepo['name']] - else: - parents = repoinfo.getrepo_by_tag(tag)['parents'] - if parents: - for p in parents: - ptag = repoinfo.get(p,'tag') - if ('-testing' in ptag or '-candidate' in ptag) and \ - not unstable_tags: - # skip unstable tags - continue - prev_rlss[ptag] = self.latest_by_tag(ptag, name, max_evr=max_evr) - else: - # NOTE: for tags with no parents (e.g. rawhide) we obviously can't - # check the parent repos' tags to get the previously released - # package NVRs. Instead we use koji's tagHistory command to find - # the previous version of this package that had this tag. - hist = self.tagHistory(tag=tag, package=name) - # remove untagged builds - hist = filter(lambda h: h['revoke_event'] is None, hist) - # ensure sorting by time - hist.sort(key=lambda h: h['create_ts']) - # hist[-2] = not the currently-tagged build, the one before that - prev_rlss[tag] = self.getBuild(hist[-2]['build_id']) + tag = tag.replace('-candidate','-testing') + + testrepo = repoinfo.getrepo_by_tag(tag) + if testrepo is None: + print >> sys.stderr, "Error: There's no repo definition for tag '%s'" % tag + return {} + parents = testrepo['parents'] + repos = parents + [testrepo['name']] + + if not unstable_tags: + for repo in repos[:]: + rtag = repoinfo.get(repo, 'tag') + if ('-testing' in rtag or '-candidate' in rtag): + # remove repo with unstable tag + repos.remove(repo) + + for repo in repos: + rtag = repoinfo.get(repo, 'tag') + prev_rlss[rtag] = self.latest_by_tag(rtag, name, max_evr=max_evr) return prev_rlss def list_previous_release(self, name, tag, max_evr=None, -- 1.7.2.1

2 1

[PATCH] don't crash when repoinfo.getrepo_by_tag returns None
by Kamil Paral 20 Aug '10

20 Aug '10

--- lib/python/util.py | 3 +++ tests/initscripts/initscripts.py | 6 +++--- tests/rpmguard/rpmguard.py | 5 +++-- tests/rpmlint/rpmlint.py | 4 ++-- 4 files changed, 11 insertions(+), 7 deletions(-) diff --git a/lib/python/util.py b/lib/python/util.py index ed90ba4..b6d0013 100644 --- a/lib/python/util.py +++ b/lib/python/util.py @@ -96,6 +96,9 @@ def checkhash(filename, algo, hexdigest): def check_opt_in(pkgname, dist_release): """take a package name and distribution release and check for an autoqa-optin file""" + if not pkgname or not dist_release: + return False + url = 'http://fedorapeople.org/packages/' fullurl = url + '%s/%s/%s/autoqa-optin' % (pkgname[0].lower(), pkgname, dist_release) tmout = socket.getdefaulttimeout() diff --git a/tests/initscripts/initscripts.py b/tests/initscripts/initscripts.py index 0806107..1885003 100644 --- a/tests/initscripts/initscripts.py +++ b/tests/initscripts/initscripts.py @@ -152,9 +152,9 @@ class initscripts(AutoQATest): self.highlights = "Following tests returned zero exit code: %s\n" % repr(tests_exit_status_zero) self.highlights += "Following tests returned NON-zero exit code: %s\n" % repr(tests_exit_status_nonzero) - #add subscribed package owners to mail-to - collection_name = repoinfo.getrepo_by_tag(kojitag)['collection_name'] - if collection_name and autoqa.util.check_opt_in(name, collection_name): + # email results to mailing list and to pkg owner if they optin + repo = repoinfo.getrepo_by_tag(kojitag) + if repo is not None and autoqa.util.check_opt_in(name, repo['collection_name']): #FIXME - hardcoded partial email address here - obviously sub-par self.mail_to.append('%s-owner(a)fedoraproject.org' % name) diff --git a/tests/rpmguard/rpmguard.py b/tests/rpmguard/rpmguard.py index a0e7e74..b317089 100644 --- a/tests/rpmguard/rpmguard.py +++ b/tests/rpmguard/rpmguard.py @@ -188,8 +188,9 @@ rpmlint | cut -d ' ' -f 1", retain_output=True) self.summary = '%u warnings for package %s' % (warning_count, envr) self.outputs = log - collection_name = repoinfo.getrepo_by_tag(kojitag)['collection_name'] - if collection_name and autoqa.util.check_opt_in(name, collection_name): + # email results to mailing list and to pkg owner if they optin + repo = repoinfo.getrepo_by_tag(kojitag) + if repo is not None and autoqa.util.check_opt_in(name, repo['collection_name']): #FIXME - hardcoded partial email address here - obviously sub-par self.mail_to.append('%s-owner(a)fedoraproject.org' % name) diff --git a/tests/rpmlint/rpmlint.py b/tests/rpmlint/rpmlint.py index adcfb13..1189298 100644 --- a/tests/rpmlint/rpmlint.py +++ b/tests/rpmlint/rpmlint.py @@ -69,8 +69,8 @@ class rpmlint(AutoQATest): self.summary = '%s for package %s' % (summary, envr) # email results to mailing list and to pkg owner if they optin - collection_name = repoinfo.getrepo_by_tag(kojitag)['collection_name'] - if collection_name and autoqa.util.check_opt_in(name, collection_name): + repo = repoinfo.getrepo_by_tag(kojitag) + if repo is not None and autoqa.util.check_opt_in(name, repo['collection_name']): #FIXME - hardcoded partial email address here - obviously sub-par self.mail_to.append('%s-owner(a)fedoraproject.org' % name) -- 1.7.2.1

2 1

Re: [PATCH] add upgradepath test
by Kamil Paral 20 Aug '10

20 Aug '10

----- "Will Woods" <wwoods(a)redhat.com> wrote: > On Thu, 2010-08-19 at 14:58 -0400, James Laska wrote: > > I was interested in testing out just the upgrade-path portion of the > > test (not the autotest/autoqa integration). How would you feel > about > > moving the test to a stand-alone script, and then changing > > upgradepath.py to call the stand-alone script and process output? No, don't say this, Vojta will try to kill me! That is exactly how he started to implement that and I talked him out of it :) > > > > I think this would make it a lot easier for folks without a full > autoqa > > and autotest-server setup to run the test at home. > > I agree with James here - it's best if we can run the tests outside of > AutoQA, both for debugging purposes and so we can extend/reuse test > code > elsewhere. > There are a few reasons why I recommended Vojta to write it as a single AutoQA script: 1. This task is very short and simple, we can easily join the the test logic and AutoQA test object stuff. 2. This task requires access to almost whole repoinfo.conf. Using it as a standalone script would require keeping a copy of repoinfo.conf next to it or passing all those information with command line options (all those repo URLs and stuff, that wouldn't be pretty). 3. This task requires a lot of functionality that we already implemented in our autoqa python library, like querying koji for latest package with some tag, comparing EVR or easily working with repoinfo. Using a standalone script would still require autoqa library installed (and therefore whole autoqa effectively) or Vojta would have to re-implement all that stuff from scratch. Of course it is possible to have this as a standalone script, but it seemed extremely cumbersome to me and I discouraged Vojta from doing that. The code would be twice as long, handling many options passing and re-implementing some autoqa code. But maybe I'm just looking at it all wrong. Tell me what you think.

2 2

[PATCH] add upgradepath test
by Vojtěch Aschenbrenner 20 Aug '10

20 Aug '10

Hello, I created new test for AutoQA named 'upgradepath'. It will check if package which you want to push into repository has correct version. Also there will be warning if you're pushing into stable repo or pushing older or the same version of package. Requests for comments :-). --- diff --git a/tests/upgradepath/control b/tests/upgradepath/control new file mode 100644 index 0000000..997c81e --- /dev/null +++ b/tests/upgradepath/control @@ -0,0 +1,14 @@ +# vim: set syntax=python +# Notice: Most recent documentation is available at doc/control.template. +TIME="SHORT" +AUTHOR = "Vojtech Aschenbrenner <vaschenb(a)redhat.com>" +DOC = """ +This test runs upgradepath to catch problems with package versions +in repositories. +""" +NAME = 'upgradepath' +TEST_TYPE = 'CLIENT' # SERVER can be used for tests that need multiple machines +TEST_CLASS = 'General' +TEST_CATEGORY = 'Functional' + +job.run_test('upgradepath', config=autoqa_conf, **autoqa_args) diff --git a/tests/upgradepath/control.autoqa b/tests/upgradepath/control.autoqa new file mode 100644 index 0000000..6b2ea92 --- /dev/null +++ b/tests/upgradepath/control.autoqa @@ -0,0 +1,9 @@ +# vim: set syntax=python +# Notice: Most recent documentation is available at doc/control.autoqa.template. + +# upgradepath can be run just once and on any architecture +archs = ['noarch'] + +# do not run automaticaly +execute = False + diff --git a/tests/upgradepath/upgradepath.py b/tests/upgradepath/upgradepath.py new file mode 100755 index 0000000..8c4547e --- /dev/null +++ b/tests/upgradepath/upgradepath.py @@ -0,0 +1,147 @@ +#!/usr/bin/python +# +# Copyright 2010, Red Hat, Inc. +# +# This program is free software; you can redistribute it and/or modify +# it under the terms of the GNU General Public License as published by +# the Free Software Foundation; either version 2 of the License, or +# (at your option) any later version. +# +# This program is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +# GNU General Public License for more details. +# +# You should have received a copy of the GNU General Public License along +# with this program; if not, write to the Free Software Foundation, Inc., +# 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA. +# +# Author: Vojtech Aschenbrenner <vaschenb(a)redhat.com> + +import os +import sys +import operator +import autoqa.koji_utils +import autoqa.util +import rpmUtils.miscutils + +from autoqa.test import AutoQATest +from autoqa.decorators import ExceptionCatcher +from autotest_lib.client.bin.test_config import config_loader +from autotest_lib.client.common_lib import error + + +class upgradepath(AutoQATest): + version = 1 # increment if setup() changes + test_result = 'PASS' + log = [] + envr_list = set() + + def initialize(self, config): + self.config = config_loader(config, self.tmpdir) + #URL of logs/results stored on autotest-server + self.autotest_url = autoqa.util.make_autotest_url(self.config) + + def setup(self): + pass + + def compare(self, matching_build, tags, op): + koji = autoqa.koji_utils.SimpleKojiClientSession() + for tag in tags: + latest_build = koji.latest_by_tag(tag, matching_build['name']) + if latest_build is None: + msg = tag + '... OK' + self.log.append(msg) + print msg + + msg = "-> Package %s in %s doesn't exist" % (matching_build['name'], tag) + self.log.append(msg) + print msg + else: + result = autoqa.koji_utils.compareEVR(matching_build, latest_build) + + if op(result, 0): + msg = tag + '... OK' + print msg + self.log.append(msg) + else: + self.test_result = 'FAIL' + msg = tag + '... FAIL' + print msg + self.log.append(msg) + self.envr_list.add(matching_build['envr']) + msg = "-> Pushing package: %s\n-> Latest package: %s" % (matching_build['envr'], latest_build['nvr']) + self.log.append(msg) + print msg + + @ExceptionCatcher("self.run_once_failed") + def run_once(self, envrs, name, kojitag, **kwargs): + reponames = [reponame for reponame in autoqa.koji_utils.repoinfo.repos() if not reponame.endswith('-testing')] + repotags = [autoqa.koji_utils.repoinfo.getrepo(reponame)['tag'] for reponame in reponames] + repotags.sort() + try: + current_tag = repotags.index(kojitag) + except ValueError: + self.test_result = 'FAIL' + msg = "ERROR: Entered bad kojitag" + print msg + self.log.append(msg) + self.envr_list.add(matching_build['envr']) + raise error.TestFail + + hi_tags = repotags[(current_tag + 1):] # tags higher than current + low_eq_tags = repotags[:(current_tag + 1)] # tags lower or equal + + for envr in envrs: + msg = 40*'=' + '\n' + envr + '\n' + 40*'=' + self.log.append(msg) + print 40*'=' + print envr + print 40*'=' + + # our testing package + (name, version, release, epoch, arch) = rpmUtils.miscutils.splitFilename(envr) + matching_build = { + 'name': name, + 'version' : version, + 'release' : release + '.' + arch, + 'epoch' : epoch, + 'envr' : envr, + } + + if kojitag.find('updates') == -1 and repotags[current_tag] != repotags[-1]: + msg = "Warning: Pushing into stable repository" + self.log.append(msg) + print msg + + koji = autoqa.koji_utils.SimpleKojiClientSession() + latest_build = koji.latest_by_tag(kojitag, matching_build['name']) + if latest_build is not None: + result = autoqa.koji_utils.compareEVR(matching_build, latest_build) + if result != 1: + msg = "Warning: Pushing older or current version of package" + self.log.append(msg) + print msg + + # compare with lower or equal tags, so version has to be greater or equal + self.compare(matching_build, low_eq_tags, operator.ge) + # compare with higher tags, so version has to be lower or equal + self.compare(matching_build, hi_tags, operator.le) + + # mailing stuff + self.result = self.test_result + + self.outputs = "" + for i in self.log: + self.outputs += i + '\n' + self.outputs += '\n \n' + + packages_fail = "" + for i in self.envr_list: + packages_fail += i + ', ' + packages_fail = packages_fail[:-2] + self.summary = packages_fail + + if self.test_result == 'FAIL': + raise error.TestFail +

4 3

[AutoQA] #199: Standardize (some) AutoQA test args
by fedora-badges 19 Aug '10

19 Aug '10

#199: Standardize (some) AutoQA test args -----------------------+---------------------------------------------------- Reporter: wwoods | Owner: Type: task | Status: new Priority: major | Milestone: Multi-hook tests Component: docs/wiki | Version: 1.0 Keywords: | -----------------------+---------------------------------------------------- The various hooks that AutoQA provides each have different test args. Some of these arguments are similar or identical across hooks - for example, 'config' applies to all hooks. Each hook also has an argument that represents the name of the thing to be tested, but this varies from hook to hook - it's a reponame for post-repo-update, treename for post-tree- compose, etc. We should document and standardize these argument names so that it's easier to write tests that will run for multiple hooks. For example, maybe all hooks should use 'name' to represent the name of the thing being tested. -- Ticket URL: <https://fedorahosted.org/autoqa/ticket/199> AutoQA <http://autoqa.fedorahosted.org> Automated QA project

1 3

depcheck - next steps!
by Will Woods 17 Aug '10

17 Aug '10

Time for a Depcheck Roundup! Whee! So now that I've messed with the unittests a bit and proven to myself that yes, depcheck does what it's supposed to (at least for *trivial* cases) it's time to look at the next steps: == write control/control.autoqa/test wrapper object == I took a stab at this in the depcheck branch but I'm pretty sure I did it wrong / left some stuff out. Kamil/Joza, if you guys can give me some help/pointers here I'd appreciate it... but don't worry, I'll send the patch to the list for a final review before I merge anything. == integrate mash == Basically, we need to be using the 'mash' tool rather than 'createrepo' to create the repos we use when testing updates, otherwise we'll miss anything involving multilib (like the infamous nss-softokn bug). I'm not at all sure how mash works - does anyone want to volunteer to figure this out? == more test cases == Once we're using mash instead of createrepo, we'll want to a) write a unittest or two to be sure mash is doing what it's supposed to, and b) write some depcheck unittests that prove depcheck handles multilib problems (and normal multilib packages) properly. == Get it running in autoqa! == Technically, we could start running depcheck *before* we finish the mash work - we just would ignore all multilib issues. I feel like we should wait until depcheck is working *correctly* before wiring it up, but maybe it would be helpful to have it partially working? What do you all think? == And beyond.. == Once we're satisfied that depcheck is working and doing the right thing.. then we actually talk to rel-eng and infrastructure about wiring up the proper bits so we can *reject* packages that fail depcheck. Exciting stuff! Expect flamewars! Just to state the obvious: we're probably not going to have depcheck working by F14Alpha. Oh well. But I am undaunted - we're getting reeally close. Anything else depcheck-related that I'm forgetting about here? I need to review the trac tickets, I think.. -w

1 0

some unittests in depcheck branch
by Will Woods 17 Aug '10

17 Aug '10

Hi, all - Last week we talked about writing unittests for depcheck - Kamil and I tried to list a few required test cases, and I did a little experimenting with the unittest module in the depcheck branch. Here's an example test: def test_depcheck_good_update(self): '''Make sure depcheck accepts a good update as expected''' (n, v, r) = ('mint', '2.0', '1') # twice as minty as mint 1.0! p = rpmfluff.SimpleRpmBuild(n,v,r) p.make() (rv, problems, ok_packages) = depcheck_main(self.mainrepo.id, [p.get_built_rpm(self.arch)]) self.assertEqual(len(problems), 1) self.assertTrue(problems[0].startswith(u'Success')) okp = ok_packages[0] self.assertEqual((n,v,r),(okp.name, okp.version, okp.release)) self.assertEqual(rv, 2) p.clean() (there's a setUp method that creates a very simple repo which contains three packages - 'julep', which depends on 'mint' and 'bourbon = 1.0') Kamil, is this the sort of thing you were thinking of? I think this makes sense but I'd love some input from the rest of you. If you have a chance, check out the 'depcheck' branch and have a look at the unit tests, in tests/depcheck/depcheck.py. I'd love suggestions / more unittests / whatever. Thanks! -w

1 0

Re: [PATCH 5/5] allow hooks to throw parser error on incorrectly provided arguments
by Kamil Paral 12 Aug '10

12 Aug '10

----- "Will Woods" <wwoods(a)redhat.com> wrote: > On Tue, 2010-08-10 at 15:32 +0200, Kamil Páral wrote: > > The hooks can now use their process_testdata method to check the > > correctness of provided options and arguments and abort the program > if > > something is wrong. It also allows hooks to have some mandatory > options > > defined. > > This patch looks fine, but one question: > > > -def process_testdata(opts, args, arch, **extra): > > +def process_testdata(parser, opts, args, **extra): > > Where'd 'arch' go? Was it just not needed? That was part of my control.autoqa patch: http://git.fedorahosted.org/git/?p=autoqa.git;a=commitdiff;h=55ddd92dfffe45… If you see the current workflow, hook.process_testdata() is now called just once for the whole job, not several times for every arch. This is needed, because hook.process_testdata() must be executed before control.autoqa is evaluated, and control.autoqa needs also to be run just once for the whole job, not once for every arch (otherwise control.autoqa wouldn't be able to modify the archlist). The result is the hooks are now not able to modify test input values according to a specific architecture. They have to pass some general value or the autoqa harness must be run several times for different architectures if needed (in the aforementioned patch I have modified watch-composes.py in exactly that way - it runs autoqa several times, because different archs need different arguments).

2 1

[PATCH #199] Standardize AutoQA test arguments
by Kamil Paral 11 Aug '10

11 Aug '10

This is an attempt to solve ticket #199: https://fedorahosted.org/autoqa/ticket/199 I standardized the test argument names amongst the hooks as much as possible. I also wanted to rename 'kojitag' to just 'tag', but 'tag' is a reserved word in Autotest harness (similarly to 'url'), so I left it intact. As usual, I also added a few more goodies to the patchset, so we now use native python objects for passing lists of arguments, and the hooks are now able to specify mandatory arguments and fail on bad input. I tested the changes a little, no regressions found. Comments welcome. Thanks, Kamil

2 11

Re: [PATCH 1/5] Use native python list when passing list of ENVRs for bodhi tests
by Kamil Paral 11 Aug '10

11 Aug '10

----- "Will Woods" <wwoods(a)redhat.com> wrote: > On Tue, 2010-08-10 at 15:32 +0200, Kamil Páral wrote: > > --- > > hooks/post-bodhi-update/README | 2 +- > > hooks/post-bodhi-update/hook.py | 3 +-- > > 2 files changed, 2 insertions(+), 3 deletions(-) > > One thing to note - you've changed the definition of testdata. > Previously, we (informally) defined it to be a dictionary of key=val > pairs, where both key and val were objects of type 'str'. > > Passing a value of type 'list' will still work, because autoqa is > using > %r to format testdata when it writes it into the control file, and the > result of repr(listval) can be evaluated to reproduce the original > list. > > So this is fine - in fact we can pass list, str, dict, set, or any > other > type where the result of repr(val) can be evaluated back into the > original object. > > But NOTE: while I'm fairly sure that all builtin python types should > work, this does NOT mean we can throw ANY python object into testdata > and have it come out the other side intact. For example, if we tried > to > pass (let's say) a bugzilla.Bug() into testdata, we'd end up with a > str > value like: > '<Bug #1337 on https://bugzilla.redhat.com/xmlrpc.cgi at 0x999f06c>' > > But that's being a little pedantic - since we're populating testdata > from the results of OptionParser, we're unlikely to be passing > anything > other than str and lists of strs. So this patch is fine, so long as we > remember not to get too fancy with our values in the future. I have quite forgotten we write out this data into the control file. But still, passing native python list seemed to me a better choice than using comma/space separated strings (different approach for different hooks) and easier for test writers. Sure, we can't pass on arbitrary object type.

1 0

[PATCH] treat ghost files appropriately, i.e. match rpm's behaviour
by François Cami 10 Aug '10

10 Aug '10

From: Francois Cami <fcami(a)fedoraproject.org> --- tests/conflicts/potential_conflict.py | 2 +- 1 files changed, 1 insertions(+), 1 deletions(-) diff --git a/tests/conflicts/potential_conflict.py b/tests/conflicts/potential_conflict.py index 62f62f3..c18dba0 100755 --- a/tests/conflicts/potential_conflict.py +++ b/tests/conflicts/potential_conflict.py @@ -197,7 +197,7 @@ for repo in my.repos.listEnabled(): print repo.urls[0] for pkg in my.pkgSack.returnNewestByNameArch(): - for fn in pkg.filelist + pkg.ghostlist: + for fn in pkg.filelist: if not fulldict.has_key(fn): fulldict[fn] = [] fulldict[fn].append(pkg) -- 1.7.2

2 1

[AutoQA] #216: pst: signature check?
by fedora-badges 10 Aug '10

10 Aug '10

#216: pst: signature check? --------------------+------------------------------------------------------- Reporter: wwoods | Owner: Type: task | Status: new Priority: major | Milestone: Package Update Acceptance Test Plan - package sanity tests Component: tests | Version: 1.0 Keywords: | --------------------+------------------------------------------------------- It's not explicitly stated in the Package Sanity Test Plan but I think verifying the package signature might be part of Package Sanity? We need to verify package signatures for packages that are supposed to be pushed as updates - there have been at least two recent incidents of unsigned packages landing in the live repos. -- Ticket URL: <https://fedorahosted.org/autoqa/ticket/216> AutoQA <http://autoqa.fedorahosted.org> Automated QA project

1 0

Correct handling of %ghost
by James Laska 10 Aug '10

10 Aug '10

Greetings, Testing against F-14-Alpha highlighted several potential file conflicts between MTA's. For details see https://bugzilla.redhat.com/show_bug.cgi?id=621864. Long story short, turns out potential_conflicts.py is incorrectly including %ghost files when comparing files between packages. François Cami and Seth Vidal made the following patch and tested to confirm that the %ghost files are no longer included when comparing file conflicts. Thanks, James

1 1

New test for autoqa - upgradepath
by Vojtěch Aschenbrenner 10 Aug '10

10 Aug '10

Hi, I created new test for autoqa. Upgradepath will check if version of pushing package has good version. See ticket #123 at https://fedorahosted.org/autoqa/ticket/123 . This is first version, which will NOT consider update-testing repository (just dist-fXX and dist-fXX-update). You can get it from upgradepath branch in git. Try for example: $ autoqa post-bodhi-update --updateid FEDORA-2010-11132 \ --title mc-4.7.1-2.fc11 --targettag dist-f11 1:mc-4.7.1-2.fc11 \ --arch x86_64 -t upgradepath --local This will check if 1:mc-4.7.1-2.fc11 could be pushed to dist-f11. So every lower repos have to have lower or equal version and higher repos higher or equal version. Next version will consider update-testing repos. Request for comments :-). Vojta

3 4

[PATCH] schedule post-bodhi-update watcher with cron
by Kamil Paral 09 Aug '10

09 Aug '10

I just wonder, shouldn't we start scheduling post-bodhi-update watcher with cron? Patch attached.

2 7

AutoQATest work (jskladan branch) update
by Will Woods 09 Aug '10

09 Aug '10

So! I've been testing Josef's work on the AutoQATest class and I think I've got the rough edges filed down so it's basically working as expected! You can check out my updates to his work on the jskladan branch - I'm going to keep testing things on our local setup here until I'm more sure it's ready, and then I should hopefully be merging that branch on Monday or Tuesday. -w

2 2

Re: [AutoQA] #42: Automate test case - Basic Functionality / X basic input handling
by fedora-badges 06 Aug '10

06 Aug '10

#42: Automate test case - Basic Functionality / X basic input handling ---------------------+------------------------------------------------------ Reporter: wwoods | Owner: Type: task | Status: new Priority: major | Milestone: Future test cases Component: tests | Version: 1.0 Resolution: | Keywords: ---------------------+------------------------------------------------------ Changes (by kparal): * milestone: => Future test cases -- Ticket URL: <https://fedorahosted.org/autoqa/ticket/42#comment:2> AutoQA <http://autoqa.fedorahosted.org> Automated QA project

1 0

Re: [AutoQA] #41: Automate test case - Basic Functionality / X startup/basic display configuration
by fedora-badges 06 Aug '10

06 Aug '10

#41: Automate test case - Basic Functionality / X startup/basic display configuration ---------------------+------------------------------------------------------ Reporter: wwoods | Owner: Type: task | Status: new Priority: major | Milestone: Future test cases Component: tests | Version: 1.0 Resolution: | Keywords: ---------------------+------------------------------------------------------ Changes (by kparal): * milestone: => Future test cases -- Ticket URL: <https://fedorahosted.org/autoqa/ticket/41#comment:2> AutoQA <http://autoqa.fedorahosted.org> Automated QA project

1 0

[AutoQA] #190: Test idea - ABI compatability check
by fedora-badges 06 Aug '10

06 Aug '10

#190: Test idea - ABI compatability check -------------------------+-------------------------------------------------- Reporter: jlaska | Owner: Type: enhancement | Status: new Priority: minor | Milestone: Component: tests | Version: 1.0 Keywords: | -------------------------+-------------------------------------------------- Remi Collet posted a blog article on how to use the [http://ispras.linux- foundation.org/index.php/ABI_compliance_checker ABI compliance checker] to ensure that ABI is maintained during updated releases of a package (this could be used to compare versions changes as well). See the full post at http://blog.famillecollet.com/post/2010/06/20/ABI -:-stability-check -- Ticket URL: <https://fedorahosted.org/autoqa/ticket/190> AutoQA <http://autoqa.fedorahosted.org> Automated QA project

1 2

Re: [AutoQA] #31: Test for unowned directories
by fedora-badges 06 Aug '10

06 Aug '10

#31: Test for unowned directories ----------------------+----------------------------------------------------- Reporter: dgregor | Owner: Type: task | Status: new Priority: minor | Milestone: Future test cases Component: tests | Version: 1.0 Resolution: | Keywords: ----------------------+----------------------------------------------------- Changes (by kparal): * milestone: => Future test cases -- Ticket URL: <https://fedorahosted.org/autoqa/ticket/31#comment:2> AutoQA <http://autoqa.fedorahosted.org> Automated QA project

1 0

Re: [AutoQA] #59: Add debugrepo check
by fedora-badges 06 Aug '10

06 Aug '10

#59: Add debugrepo check --------------------+------------------------------------------------------- Reporter: scop | Owner: Type: task | Status: new Priority: minor | Milestone: Future test cases Component: tests | Version: Resolution: | Keywords: --------------------+------------------------------------------------------- Changes (by kparal): * milestone: => Future test cases -- Ticket URL: <https://fedorahosted.org/autoqa/ticket/59#comment:1> AutoQA <http://autoqa.fedorahosted.org> Automated QA project

1 0

Re: [AutoQA] #92: Adapt font tests for autoqa
by fedora-badges 06 Aug '10

06 Aug '10

#92: Adapt font tests for autoqa ---------------------+------------------------------------------------------ Reporter: wwoods | Owner: Type: task | Status: new Priority: minor | Milestone: Future test cases Component: tests | Version: 1.0 Resolution: | Keywords: ---------------------+------------------------------------------------------ Changes (by kparal): * milestone: => Future test cases -- Ticket URL: <https://fedorahosted.org/autoqa/ticket/92#comment:2> AutoQA <http://autoqa.fedorahosted.org> Automated QA project

1 0

[AutoQA] #118: New test proposal: Python debugability
by fedora-badges 06 Aug '10

06 Aug '10

#118: New test proposal: Python debugability --------------------+------------------------------------------------------- Reporter: kparal | Owner: Type: task | Status: new Priority: minor | Milestone: Component: tests | Version: 1.0 Keywords: | --------------------+------------------------------------------------------- From Adam Williamson: I've cut some of the context, but basically David wants to write a test case for checking whether Python debugging is possible as intended, and I asked exactly how he wanted it to be used. {{{ On Tue, 2010-01-26 at 15:30 -0500, David Malcolm wrote: > > > Can I request a test case to cover debuggability of the Python > runtime > > > please (both in Fedora and in RHEL). > > > > > > This is in relation to: > > > https://bugzilla.redhat.com/show_bug.cgi?id=556975 > > > https://bugzilla.redhat.com/show_bug.cgi?id=558977 > > > https://bugzilla.redhat.com/show_bug.cgi?id=557772 > > > > > > as there seem to be gcc and gdb issues, which are conspiring to > make > > > python impossible to debug. > > > > > > > > > The requirement is: within "gdb python", I must be able to select > a > > > PyEval_EvalFrameEx frame, and have the following work: > > > (gdb) print co > > > (gdb) print f > > > > > > rather that have <variable optimized out> > > > > > > so that I can do this: > > > (gdb) print (char*)(((PyStringObject*)co->co_name)->ob_sval) > > > to get the function name > > > > > > (gdb) print (char*)(((PyStringObject*)co->co_filename)->ob_sval) > > > to get the source filename > > > > > > and > > > > > > (gdb) print f->f_lineno > > > to get the approximate source line number. > > > > > > If the above isn't working, it becomes extremely hard to > meaningfully > > > debug any issues that arise inside Python. > This is probably scriptable, and a good candidate for AutoQA and foe > RHTS. > }}} -- Ticket URL: <https://fedorahosted.org/autoqa/ticket/118> AutoQA <http://autoqa.fedorahosted.org> Automated QA project

1 2

[PATCH 1/2] add repoinfo.getreleases() and get_bodhi_releases()
by Will Woods 05 Aug '10

05 Aug '10

--- lib/python/repoinfo.py | 6 ++++++ repoinfo.conf | 4 ++++ 2 files changed, 10 insertions(+), 0 deletions(-) diff --git a/lib/python/repoinfo.py b/lib/python/repoinfo.py index 420becc..3a8c1f8 100644 --- a/lib/python/repoinfo.py +++ b/lib/python/repoinfo.py @@ -39,6 +39,12 @@ class RepoinfoConfig(object): return [t.strip() for t in itemlist.split(',')] def getarches(self, name): return self._getlist(name, 'arches') + def getreleases(self): + '''Return the list of known, non-EOL releases.''' + return [r for r in repoinfo.repos() if self.config.get(r,'isactiverelease')] + def get_bodhi_releases(self): + '''Return the list of releases known to Bodhi.''' + return [r.upper() for r in self.getreleases()] def getparents(self, name): '''Given a repo name, give the names of its parent repos e.g.: getparents('f13-updates-testing') -> ['f13','f13-updates]''' diff --git a/repoinfo.conf b/repoinfo.conf index 3241044..d36dd93 100644 --- a/repoinfo.conf +++ b/repoinfo.conf @@ -8,6 +8,7 @@ baseurl = http://download.fedoraproject.org/pub/fedora/linux goldurl = %(baseurl)s/releases/%(path)s/Everything/%(arch)s/os updatesurl = %(baseurl)s/updates/%(path)s/%(arch)s rawhideurl = %(baseurl)s/%(path)s/%(arch)s/os +isactiverelease = # Rawhide [rawhide] @@ -19,6 +20,7 @@ collection_name = devel # Fedora 14 [f14] +isactiverelease = yes path = development/14 url = %(rawhideurl)s collection_name = F-14 @@ -37,6 +39,7 @@ collection_name = F-14 # Fedora 13 [f13] +isactiverelease = yes path = 13 url = %(goldurl)s collection_name = F-13 @@ -55,6 +58,7 @@ collection_name = F-13 # Fedora 12 [f12] +isactiverelease = yes path = 12 url = %(goldurl)s collection_name = F-12 -- 1.7.2

2 3

[PATCH] post-bodhi-update watcher: a few fixes
by Kamil Paral 05 Aug '10

05 Aug '10

These two patches fix some issues with post-bodhi-update watcher, as discussed on IRC. I haven't really studied working with repoinfo.conf-style config files, so please review if it is correct. Thanks. It is tested and it works.

2 5

[PUSHED] post-bodhi-update: fix hook to conform to control.autoqa patchset
by Kamil Paral 04 Aug '10

04 Aug '10

This was an obvious omission of the control.autoqa patchset, I forgot to fix post-bodhi-update hook. I pushed it straightaway. http://git.fedorahosted.org/git/?p=autoqa.git;a=commit;h=4dff3d9761eaafca3e…

1 0

[PATCH] deprecate email opt-in for koji watcher
by Kamil Paral 04 Aug '10

04 Aug '10

I'm not sure about all consequences of this patch. But it seems to me that we don't need collection_name and email opt-in anymore. What do you think, Will?

2 4

[AutoQA] #184: Add tags for destructive testing
by fedora-badges 02 Aug '10

02 Aug '10

#184: Add tags for destructive testing ----------------------------+----------------------------------------------- Reporter: kparal | Owner: Type: task | Status: new Priority: major | Milestone: Virtualization Component: infrastructure | Version: 1.0 Keywords: | ----------------------------+----------------------------------------------- We need to add tags into autotest and autoqa tests that will determine whether test is (may be) destructive. This tag must be propagated and appropriate test client must be selected. After finishing this test the test machine must be restored to a previous safe state. -- Ticket URL: <https://fedorahosted.org/autoqa/ticket/184> AutoQA <http://autoqa.fedorahosted.org> Automated QA project

1 1

[AutoQA] #181: Quick and dirty support for Virtualization
by fedora-badges 02 Aug '10

02 Aug '10

#181: Quick and dirty support for Virtualization ----------------------------+----------------------------------------------- Reporter: kparal | Owner: Type: task | Status: new Priority: major | Milestone: Virtualization Component: infrastructure | Version: 1.0 Keywords: | ----------------------------+----------------------------------------------- The task is to implement quick and dirty support for virtualization for AutoQA tests. We can already install autotest-client into virt machine, but there are some tests which must not run inside virt environment (tests using virt themselves for example). Therefore we must ensure all our tests are marked properly ("may use virt", "may not use virt"), all our tests clients are marked properly ("is virt", "is bare metal"), and this information are propagated from AutoQA to autotest-server and used properly when selecting appropriate host and scheduling jobs. This could be quite easy, there is some support for tags in autotest already. As an enhancement, we may think about preferences. Do we want to mark some test that it is preferred to run it in virt, but may also be run on bare metal? How do we achieve correct behavior for autotest-server for such task? For example: We may want to prefer executing sanity tests in virt env, but it's not really mandatory, just desired. Food for thoughts. -- Ticket URL: <https://fedorahosted.org/autoqa/ticket/181> AutoQA <http://autoqa.fedorahosted.org> Automated QA project

1 10

[AutoQA] #207: post-koji-build watcher does not watch for new updates-candidate builds
by fedora-badges 02 Aug '10

02 Aug '10

#207: post-koji-build watcher does not watch for new updates-candidate builds ----------------------+----------------------------------------------------- Reporter: kparal | Owner: Type: defect | Status: new Priority: major | Milestone: Package Update Acceptance Test Plan Component: watchers | Version: 1.0 Keywords: | ----------------------+----------------------------------------------------- I have found out that our post-koji-build watches only these tags: dist-f10 dist-f10-updates dist-f10-updates-testing dist-f11 dist-f11-updates dist-f11-updates-testing dist-f12 dist-f12-updates dist-f12-updates-testing dist-f13 dist-f13-updates dist-f13-updates- testing dist-f14 Which is incorrect, we should be watching dist-f*-updates-candidate (and maybe dist-f14, I don't know). We don't need to watch for the rest of it. The change happened in commit 580734d4f2aa768aefa1c0498904df2296747918. See the diff and inspect current contents of repoinfo.conf to grasp the change. The result is that we don't currently test almost any packages except f14 ones. Do we have it already deployed on our production machine? What we can do about it? Well, I hope someone with a deep understanding of repoinfo stuff knows what's the best approach :) It seems to me that we could define the *-updates-testing repositories in repoinfo.conf (and make sure we didn't break something else) and retrieve and select just those in watch-koji-builds. -- Ticket URL: <https://fedorahosted.org/autoqa/ticket/207> AutoQA <http://autoqa.fedorahosted.org> Automated QA project

2 5

[AutoQA] #200: Multiple control files / multiple test wrappers
by fedora-badges 02 Aug '10

02 Aug '10

#200: Multiple control files / multiple test wrappers --------------------+------------------------------------------------------- Reporter: wwoods | Owner: Type: task | Status: new Priority: major | Milestone: Multi-hook tests Component: tests | Version: 1.0 Keywords: | --------------------+------------------------------------------------------- For a test to run under multiple hooks, it may need multiple control files, since we may want to do slightly different things to prepare for each test. We could have multiple test wrappers and conditionalize the job.run_test() argument based on the calling hook, for example: if hook == 'post-koji-build': job.run_test('mytest_koji', name=name, envrs=envrs, ...) elif hook == 'post-bodhi-update': job.run_test('mytest_bodhi', name=name, envrs=envrs, ...) Alternately, we could have multiple control.HOOKNAME files - one for each hook the test can be run under. This would also allow us to eliminate the use of the 'testlist' file to determine the tests that are eligible for use in each hook - each test would be able to declare how to run itself under each hook. This ticket requires deciding on a method for handling multi-hook tests, documenting that method, and (perhaps) converting existing tests. -- Ticket URL: <https://fedorahosted.org/autoqa/ticket/200> AutoQA <http://autoqa.fedorahosted.org> Automated QA project

1 2

[AutoQA] #211: Update repoinfo.conf - Add branch f14
by fedora-badges 02 Aug '10

02 Aug '10

#211: Update repoinfo.conf - Add branch f14 ----------------------------+----------------------------------------------- Reporter: jkeating | Owner: Type: task | Status: new Priority: critical | Milestone: Component: infrastructure | Version: 1.0 Keywords: | ----------------------------+----------------------------------------------- We branched, SOp says make a ticket. -- Ticket URL: <https://fedorahosted.org/autoqa/ticket/211> AutoQA <http://autoqa.fedorahosted.org> Automated QA project

1 2

[AutoQA] #195: Write HelloWorld test
by fedora-badges 02 Aug '10

02 Aug '10

#195: Write HelloWorld test --------------------+------------------------------------------------------- Reporter: kparal | Owner: kparal Type: task | Status: new Priority: major | Milestone: Component: tests | Version: 1.0 Keywords: | --------------------+------------------------------------------------------- Write a HelloWorld test that will just serve for testing purposes. More information here: https://fedorahosted.org/pipermail/autoqa-devel/2010-June/000727.html If anyone is eager to write this, feel free to reassign. -- Ticket URL: <https://fedorahosted.org/autoqa/ticket/195> AutoQA <http://autoqa.fedorahosted.org> Automated QA project

1 5

Update repoinfo.conf for F-14 branch
by James Laska 02 Aug '10

02 Aug '10

Greetings, Per ticket#211, I have updated repoinfo.conf for the new F-14 branch. Instructions for updating repoinfo.conf are on the wiki at https://fedoraproject.org/wiki/How_to_update_AutoQA_repoinfo.conf. I have tested this change and confirmed that watch-koji-builds.py goes from monitoring these repos: Looking for builds with tags dist-f10-updates-candidate dist-f11-updates-candidate dist-f12-updates-candidate dist-f13-updates-candidate dist-f14 To monitoring for koji builds in these repos: Looking for builds with tags dist-f10-updates-candidate dist-f11-updates-candidate dist-f12-updates-candidate dist-f13-updates-candidate dist-f14-updates-candidate dist-f15 Thanks, James

2 2

2026

2025

2024

2023

2022

2021

2020

2019

2018

2017

2016

2015

2014

2013

2012

2011

2010

2009

autoqa-devel August 2010