autoqa-devel March 2011

autoqa-devel@lists.fedorahosted.org

10 participants
103 discussions

[AutoQA] #292: When publishing results to bodhi, link to output.log
by fedora-badges 30 Mar '11

30 Mar '11

#292: When publishing results to bodhi, link to output.log --------------------+------------------------------------------------------- Reporter: jlaska | Owner: Type: task | Status: new Priority: minor | Milestone: Finger Food Component: tests | Keywords: --------------------+------------------------------------------------------- Another suggestion from sgallagh... Now that we save all test output to output.log, can we link to it when posting results to bodhi, rather than posting to the top-level results directory? For example, with https://admin.fedoraproject.org/updates/sssd-1.5.1-8.fc15,libldb-1.0.0-2.fc…, we display ... {{{ autoqa - 2011-03-23 16:45:29 AutoQA: upgradepath test PASSED on noarch. The result can be found at: http://autoqa.fedoraproject.org/results/72138-autotest/qa03.c.fedoraproject…. }}} Perhaps instead we can link to ... {{{ autoqa - 2011-03-23 16:45:29 AutoQA: upgradepath test PASSED on noarch. The result can be found at: http://autoqa.fedoraproject.org/results/72138-autotest/qa03.c.fedoraproject…. }}} This way, maintainers don't have to hunt for the test results. -- Ticket URL: <https://fedorahosted.org/autoqa/ticket/292> AutoQA <http://autoqa.fedorahosted.org> Automated QA project

1 10

[AutoQA] #286: upgradepath doesn't report update results as a whole but per-package
by fedora-badges 30 Mar '11

30 Mar '11

#286: upgradepath doesn't report update results as a whole but per-package ----------------------+----------------------------------------------------- Reporter: kparal | Owner: Type: defect | Status: new Priority: critical | Milestone: Hot issues Component: tests | Keywords: ----------------------+----------------------------------------------------- See https://admin.fedoraproject.org/updates/vte3-0.27.90-2.fc15,gucharmap-2.33.… ,gedit-plugins-2.91.0-3.fc15 Upgradepath sends 3 different results to this update: {{{ 03/23 16:44:12 INFO |upgradepat:0214| Sending results to Bodhi: vte3-0.27.90-2.fc15,gucharmap-2.33.2-6.fc15,gedit-plugins-2.91.0-3.fc15 FAILED 03/23 16:45:06 INFO |upgradepat:0214| Sending results to Bodhi: vte3-0.27.90-2.fc15,gucharmap-2.33.2-6.fc15,gedit-plugins-2.91.0-3.fc15 PASSED 03/23 16:47:37 INFO |upgradepat:0214| Sending results to Bodhi: vte3-0.27.90-2.fc15,gucharmap-2.33.2-6.fc15,gedit-plugins-2.91.0-3.fc15 PASSED }}} We have to first group the packages into an update and then send one summary result. Like we do in depcheck. -- Ticket URL: <https://fedorahosted.org/autoqa/ticket/286> AutoQA <http://autoqa.fedorahosted.org> Automated QA project

1 5

[PATCH 1/3] 286 - Upgradepath reports results to updates
by Josef Skladanka 30 Mar '11

30 Mar '11

--- tests/upgradepath/upgradepath.py | 53 ++++++++++++++++++++++++++----------- 1 files changed, 37 insertions(+), 16 deletions(-) diff --git a/tests/upgradepath/upgradepath.py b/tests/upgradepath/upgradepath.py index b7527f3..c2f8c92 100755 --- a/tests/upgradepath/upgradepath.py +++ b/tests/upgradepath/upgradepath.py @@ -25,6 +25,7 @@ import rpmUtils.miscutils from autoqa.repoinfo import repoinfo from autoqa.test import AutoQATest from autoqa.decorators import ExceptionCatcher +from autoqa.bodhi_utils import bodhi_post_testresult, query_update class upgradepath(AutoQATest): version = 1 # increment if setup() changes @@ -202,24 +203,44 @@ class upgradepath(AutoQATest): self.summary = '%s for %s' % (summary, update_id) - # send results to Bodhi - print 'Sending results to Bodhi...' - exc = False + # This code simply takes all nvrs (=builds), and creates a dictionary + # of update metadata (taken from bodhi). + # The variable 'updates' then looks like + # + # updates = {updatename : bodhi_metadata} + + exc = False # variable to detect presence of builds not-found-in-bodhi + updates = {} for nvr in nvrs: - result = self.nvr_results[nvr] - try: - update = autoqa.bodhi_utils.query_update(nvr) - assert update is not None, 'No such update object in Bodhi: %s' % nvr - update_title = update['title'] - print 'Sending results to Bodhi: %s %s' % (update_title, result) - autoqa.bodhi_utils.bodhi_post_testresult(update_title, 'upgradepath', - result, self.autotest_url, 'noarch') - except AssertionError, e: - msg = 'Failed to send Bodhi results to %s:\n %s' % (nvr, e) - print msg - self.outputs.append(msg) + bodhi_info = None + bodhi_info = query_update(nvr) + if bodhi_info is None: + print >> sys.stderr, "Update containing %s not found in Bodhi" % nvr exc = True - # if assert failed (some Bodhi update doesn't exist), end the test as NEEDS_INSPECTION + continue + + if bodhi_info['title'] not in updates.keys(): + updates[bodhi_info['title']] = bodhi_info + + # Becase bodhi is working on top of updates, we need to check whether + # all nvrs (builds) from the respective update were accepted. If at least + # one of the nvrs in update was in rejected/ignored set, whole + # update is 'failed'. + + update_result = dict() # update_title -> update_result + for update in updates.values(): + update_result[update['title']] = 'PASSED' + for build in update['builds']: + if self.result_order.index(self.nvr_results[build['nvr']]) > self.result_order.index(update_result[update['title']]): + update_result[update['title']] = self.nvr_results[build['nvr']] + + # Report all results to Bodhi + for title, result in update_result.items(): + print "Posting to Bodhi: %s %s" % (title, result) + bodhi_post_testresult(title, 'upgradepath', result, + self.autotest_url, 'noarch', karma=0) + + # if some Bodhi update doesn't exist, end the test as NEEDS_INSPECTION if exc and self.result == 'PASSED': print >> sys.stderr, 'Nonexistent updates found, setting result to NEEDS_INSPECTION' self.result = 'NEEDS_INSPECTION' -- 1.7.2.3

3 12

Problem with scheduling Depcheck run in Autotest
by Josef Skladanka 30 Mar '11

30 Mar '11

Hi, for some reason, autoqa is behaving 'weird': # autoqa post-bodhi-update-batch --targettag dist-f13-updates --arch x86_64 package_x_y_z --dry-run /usr/bin/atest job create --reboot_before=never --reboot_after=never -m *x86_64 -f /tmp/autoqa-control.ppy374 post-bodhi-update-batch:upgradepath.noarch /usr/bin/atest job create --reboot_before=never --reboot_after=never -m *x86_64 -d package_x_y_z -f /tmp/autoqa-control.lewOvW post-bodhi-update-batch:depcheck.x86_64 I don't know why there is the "-d package_x_y_z" part on the second line, but because of this parameter job create fails: # /usr/bin/atest job create --reboot_before=never --reboot_after=never -m *x86_64 -d package_x_y_z -f /tmp/autoqa-control.lewOvW post-bodhi-update-batch:depcheck.x86_64 Operation create_job failed: KeyError: 'package_x_y_z' I tried to find the problem, but I'm probably too stupid today, and can not determine what's wrong. J.

1 1

[AutoQA] #273: Change watcher-name based events 'parsing' to event names
by fedora-badges 30 Mar '11

30 Mar '11

#273: Change watcher-name based events 'parsing' to event names -------------------------+-------------------------------------------------- Reporter: jskladan | Owner: Type: enhancement | Status: new Priority: minor | Milestone: Finger Food Component: core | Keywords: -------------------------+-------------------------------------------------- This task consists of two sub-tasks. First part is simply changing the 'autoqa watcher_name parameters' calls in watchers to 'autoqa event_name parameters'. This also brings a need to re-structure the importing of 'hook.py' - at the moment, all watchers have hook.py file in their directory structure. This contains additional rules for parameter parsing. We'll need to redo the directory structure based on the event names, and change the importing routine in autoqa script. -- Ticket URL: <https://fedorahosted.org/autoqa/ticket/273> AutoQA <http://autoqa.fedorahosted.org> Automated QA project

1 6

[AutoQA] #267: Remove 'frontends' (israwhidebroken) directory from the master branch in git
by fedora-badges 30 Mar '11

30 Mar '11

#267: Remove 'frontends' (israwhidebroken) directory from the master branch in git ----------------------+----------------------------------------------------- Reporter: jskladan | Owner: Type: task | Status: new Priority: trivial | Milestone: Finger Food Component: core | Keywords: ----------------------+----------------------------------------------------- Since we're not building/packaging israwhidebroken any more, I'd suggest removing the 'frontends' directory alltogether - it will stay in the git history, if we ever decide we need it, and will not mess up our grep searches any more :) -- Ticket URL: <https://fedorahosted.org/autoqa/ticket/267> AutoQA <http://autoqa.fedorahosted.org> Automated QA project

4 8

[AutoQA] #276: Support for independent test development
by fedora-badges 29 Mar '11

29 Mar '11

#276: Support for independent test development -------------------------+-------------------------------------------------- Reporter: jlaska | Owner: Type: enhancement | Status: new Priority: major | Milestone: 0.5.0 Component: core | Keywords: -------------------------+-------------------------------------------------- We need support for tests that are loosely coupled from autoqa itself. While many tests are very specific to the version of autoqa, we have more tests that need to mature independently. This ticket is to track some small-ish improvements so we can support updating tests without updating autoqa packaging. It's certainly a long-term goal to support maintainer contributed tests, but for now this will *only* track allowing our current batch of maintainer contributed tests to mature independently from autoqa development cycles. Some thoughts ... 1. Since this mostly concerns tests that are developed outside of the autoqa-devel team, it might be appropriate to support maintainers storing tests in their git-dist checkout. And we keep that content sync'd on the autoqa server by way of cron. That seems appropriately hacky until we get something better, and allows us to explore storing tests along-side the code. This is not required for this support, but would be interesting to explore. 2. Update 'autoqa' to add support for recursive test discovery. The autoqa script needs to look into sub-directories of tests stored in testdir ... not just the high-level directory 3. Update 'autoqa' and 'autoqa.conf' to support multiple values in testdir configuration value -- Ticket URL: <https://fedorahosted.org/autoqa/ticket/276> AutoQA <http://autoqa.fedorahosted.org> Automated QA project

1 8

[AutoQA] #281: Test AutoQA agains production versions of python
by fedora-badges 29 Mar '11

29 Mar '11

#281: Test AutoQA agains production versions of python --------------------+------------------------------------------------------- Reporter: tflink | Owner: Type: task | Status: new Priority: major | Milestone: 0.5.0 Component: core | Keywords: --------------------+------------------------------------------------------- With the recent deployment of autoqa-0.4.4 there were a couple of issues with syntax version mismatch since the production environment is still running 2.4 while recent versions of Fedora use 2.7. Two options for preventing this from happening again are: 1. Set up a staging environment that is identical to production 2. Start testing against older versions of python ([http://codespeak.net/tox/ tox] is a good tool for this) I think that tox is a good long term solution but the only way it would work is if all the autoqa code was loaded at unit test time (which it currently isn't). Matching the staging environment to the production environment might be a better choice for the short term. -- Ticket URL: <https://fedorahosted.org/autoqa/ticket/281> AutoQA <http://autoqa.fedorahosted.org> Automated QA project

1 1

[AutoQA] #277: Branched release uses -updates-pending differently
by fedora-badges 29 Mar '11

29 Mar '11

#277: Branched release uses -updates-pending differently --------------------+------------------------------------------------------- Reporter: kparal | Owner: Type: task | Status: new Priority: major | Milestone: 0.5.0 Component: core | Keywords: --------------------+------------------------------------------------------- I just discovered that branched release (F15 ATM) uses dist-f15-updates- pending tags, even though no stable update repo is enabled. It seems that before 'Final Change Deadline' Bodhi uses -updates-pending tag for tracking of updates going to the main dist-f15 repository. This can confuse some of our tests. We have to check. Also we might need to mark this somehow in repoinfo.conf to correctly execute (or not execute) our tests. -- Ticket URL: <https://fedorahosted.org/autoqa/ticket/277> AutoQA <http://autoqa.fedorahosted.org> Automated QA project

1 4

[AutoQA] #262: Rethink the watcher/event/test architecture
by fedora-badges 29 Mar '11

29 Mar '11

#262: Rethink the watcher/event/test architecture --------------------+------------------------------------------------------- Reporter: kparal | Owner: Type: task | Status: new Priority: major | Milestone: 0.5.0 Component: core | Keywords: --------------------+------------------------------------------------------- We've hit the limits of our architecture. The one watcher-one event style is no longer sufficient. We need to think about our needs and change that architecture, so that we are more flexible with watchers, events, target architectures, batch scheduling, etc. I have mentioned some bits at: https://fedorahosted.org/pipermail/autoqa-devel/2011-January/001522.html Let's refactor the architecture and get rid of many hacks that are currently necessary. -- Ticket URL: <https://fedorahosted.org/autoqa/ticket/262> AutoQA <http://autoqa.fedorahosted.org> Automated QA project

1 3

← Newer
1
2
3
4
5
6
7
8
9
10
11
Older →

2026

2025

2024

2023

2022

2021

2020

2019

2018

2017

2016

2015

2014

2013

2012

2011

2010

2009

autoqa-devel March 2011