autoqa-devel January 2011

autoqa-devel@lists.fedorahosted.org

7 participants
78 discussions

[AutoQA] #243: doc: How to run AutoQA tests
by fedora-badges 29 Mar '11

29 Mar '11

#243: doc: How to run AutoQA tests ---------------------------+------------------------------------------------ Reporter: kparal | Owner: Type: task | Status: new Priority: major | Milestone: Component: documentation | Keywords: ---------------------------+------------------------------------------------ We currently have "How to run AutoQA tests" section at [[BR]] https://fedoraproject.org/wiki/Writing_AutoQA_Tests#How_to_run_AutoQA_tests Because the page tends to grow, it is no longer optimal to have it together. It should probably make sense to have separate page for "Writing AutoQA Tests" and separate page for "Running AutoQA Tests". Maybe the new page could be somehow merged with [[BR]] https://fedoraproject.org/wiki/Verifying_AutoQA_tests -- Ticket URL: <https://fedorahosted.org/autoqa/ticket/243> AutoQA <http://autoqa.fedorahosted.org> Automated QA project

1 1

[AutoQA] #247: upgradepath: possible broken-updates scenario for new packages
by fedora-badges 29 Mar '11

29 Mar '11

#247: upgradepath: possible broken-updates scenario for new packages --------------------+------------------------------------------------------- Reporter: kparal | Owner: Type: defect | Status: new Priority: major | Milestone: Component: tests | Keywords: --------------------+------------------------------------------------------- Here is a scenario that will create broken upgrade path and our current tests will not catch it: 1. Package foo is available for F12 (foo-1.0-1.fc12), but not for F13. 2. foo-1.0-1.fc13 is sent to f13-updates-pending as a new package update. upgradepath passes. 3. foo-2.0-1.fc12 is sent to f12-updates-pending. Since there is still no foo in F13 (AutoQA passed but it has not been pushed by RelEng team), upgradepath passes. 4. foo-2.0-1.fc12 is pushed to f12-updates. 5. foo-1.0-1.fc13 is pushed to f13-updates. 6. Broken upgrade path. What can we do about it? We can break the scenario between steps 4 and 5. But we can't probably do that just by watching -pending tags. This requires cooperation from the updates manager (i.e. Bodhi). Bodhi must know that if some foo build enters f12-updates, and there is also another foo build in f13-updates- pending, it must notify us (over Fedora Message Bus) and ask us to retest the latter build. We can also break the scenario between steps 2 and 3. When we run upgradepath on foo in f12-updates-pending, we can take into account not just f13-updates, but also f13-updates-pending. But this is quite controversial. Some builds in -updates-pending may be denied by RelEng team, but we could have failed some other builds because of them. We will probably solve this ticket only when we have advanced infrastructure ready (ResultDB, Fedora Message Bus). Without it this seems pretty hard to be solved properly. -- Ticket URL: <https://fedorahosted.org/autoqa/ticket/247> AutoQA <http://autoqa.fedorahosted.org> Automated QA project

1 2

[AutoQA] #246: Add temporary hack for test re-scheduling
by fedora-badges 08 Mar '11

08 Mar '11

#246: Add temporary hack for test re-scheduling --------------------+------------------------------------------------------- Reporter: kparal | Owner: Type: task | Status: new Priority: major | Milestone: 0.4.4 Component: core | Keywords: --------------------+------------------------------------------------------- Until we have a proper support for test re-scheduling (ticket #245) we need some temporary hack to enable test re-scheduling in most common cases. We plan to have upgradepath and depcheck sending comments to Bodhi. There is probably no problem with depcheck, because it re-evaluates the whole contents of -pending tag on its every update. But there are some issues with upgradepath, one use case is described in ticket #245. We need to: 1. get a list of use cases for which we need to reschedule upgradepath or depcheck 2. inquire whether Bodhi changes timestamp (or whatever) of the updates in those cases (i.e. we would detect it and run tests again properly even with our current watcher implementation) 3. in the remaining cases decide what to do with it. For example we can make temporary upgradepath behave the same way as depcheck does, i.e. test all builds in -updates-pending for every update of that tag. Because we will have Bodhi comment duplication prevention, it could work quite fine. -- Ticket URL: <https://fedorahosted.org/autoqa/ticket/246> AutoQA <http://autoqa.fedorahosted.org> Automated QA project

1 2

[AutoQA] #240: Make depcheck and upgradepath send comments to Bodhi
by fedora-badges 08 Mar '11

08 Mar '11

#240: Make depcheck and upgradepath send comments to Bodhi --------------------+------------------------------------------------------- Reporter: kparal | Owner: Type: task | Status: new Priority: major | Milestone: 0.4.4 Component: tests | Keywords: --------------------+------------------------------------------------------- After we have the necessary plumbing, don't forget to make depcheck and upgradepath actually send their results to Bodhi. -- Ticket URL: <https://fedorahosted.org/autoqa/ticket/240> AutoQA <http://autoqa.fedorahosted.org> Automated QA project

3 4

[AutoQA] #260: Merge new_koji_watcher branch with master
by fedora-badges 07 Mar '11

07 Mar '11

#260: Merge new_koji_watcher branch with master ----------------------+----------------------------------------------------- Reporter: jskladan | Owner: Type: task | Status: new Priority: major | Milestone: 0.4.4 Component: core | Keywords: ----------------------+----------------------------------------------------- $SUBJ - changes from #204 & #248 are present and (mostly) reviewed in that branch. -- Ticket URL: <https://fedorahosted.org/autoqa/ticket/260> AutoQA <http://autoqa.fedorahosted.org> Automated QA project

1 3

New Koji Watcher (watching -pending tags & batch possibility)
by Josef Skladanka 23 Feb '11

23 Feb '11

Hello gang, After today's testing, I believe that the new koji watcher is ready to be reviewed and poked by the rest of you :) The code is located in the new_koji_watcher branch in git, latest version should be here: http://git.fedorahosted.org/git/?p=autoqa.git;a=commit;h=634133dc452abde0ae… Comments and critique are appreciated as always joza

3 9

depcheck test, version 2
by Will Woods 14 Feb '11

14 Feb '11

This patch adds the depcheck test. Detailed change info/development history can be found by poking around the 'depcheck' branch. This version of the patch should address all the issues found by James and Kamil[1] in their reviews. (Thanks for the feedback, guys.) The most interesting bit is that depcheck_main was split into two parts: do_depcheck and print_depcheck_results. do_depcheck returns much more info than depcheck_main did, which allows for more unittests and things. (nearly) full list of changes since the last version of this patch: * test: - separate files for CLI, library, and unittests - handle pending/accepted like normal yum repos (use YumBase.add_enable_repo) - handle new packages (i.e. not updates to existing packages) correctly - refactor depcheck_main into do_depcheck and print_depcheck_results - add unittests for new packages and ignored packages * wrapper: - install mash in setup() - add missing 'import re' - initialize results dict - fetch_nvrs(): fix missing 'koji', fix filename arg, download status output - use new query_update() and bodhi_already_commented() methods - fix NameError caused by missing 'updateid' variable * control files: - fix incorrect comment in control.autoqa [1] ..except the 'tags and timing' problem of pending/accepted changing during the test. That still need to be addressed, but first we need code to work with! tests/depcheck/control | 14 + tests/depcheck/control.autoqa | 6 + tests/depcheck/depcheck | 96 +++++++ tests/depcheck/depcheck.py | 185 +++++++++++++ tests/depcheck/depcheck_lib.py | 337 +++++++++++++++++++++++ tests/depcheck/depcheck_unittests.py | 489 ++++++++++++++++++++++++++++++++++ 6 files changed, 1127 insertions(+), 0 deletions(-) -w

4 9

[AutoQA] #264: Run eclipse test suite for new eclipse koji builds
by fedora-badges 04 Feb '11

04 Feb '11

#264: Run eclipse test suite for new eclipse koji builds --------------------+------------------------------------------------------- Reporter: jlaska | Owner: Type: task | Status: new Priority: minor | Milestone: Future test cases Component: tests | Keywords: --------------------+------------------------------------------------------- Spoke to overholt at FUDCon Tempe and he would like to have the existing eclipse test suite run each time the 'eclipse' package is built in koji. The initial estimate of work involves ... 1. Create a test wrapper responsible for downloading koji packages, preparing test configuration, run test, send results to eclipse- owner(a)fedoraproject.org 2. <what else>? Overholt can provide more details on test configuration aspects. Andrew, if you are able to create a script (bash or python), that handles preparing the tests and executing them. We can then integrate that within AutoQA. -- Ticket URL: <https://fedorahosted.org/autoqa/ticket/264> AutoQA <http://autoqa.fedorahosted.org> Automated QA project

1 1

[AutoQA] #263: Run kvm-autotest test suite for new virt* koji builds
by fedora-badges 31 Jan '11

31 Jan '11

#263: Run kvm-autotest test suite for new virt* koji builds --------------------+------------------------------------------------------- Reporter: jlaska | Owner: Type: task | Status: new Priority: minor | Milestone: Future test cases Component: tests | Keywords: --------------------+------------------------------------------------------- jforbes requested running kvm-autotest in the Fedora Run upstream kvm-autotest suite for koji builds. Preliminary estimate on the work involved includes ... 1. Create/use test wrapper - download koji builds, download/locate ISO's, parse results and email to -owner(a)fedoraproject.org 2. Figure out how and where to store release-specific test configuration (dist-git) 3. Identify additional bare-metal hardware to deploy in autoqa.fedoraproject.org (AMD) 4. Create/maintain Fedora+1 (e.g. F15), Fedora (e.g. F14) and Fedora-1 (e.g. F13) releases -- Ticket URL: <https://fedorahosted.org/autoqa/ticket/263> AutoQA <http://autoqa.fedorahosted.org> Automated QA project

1 0

Update to autoqa-0.4.4-0.1.pre
by James Laska 26 Jan '11

26 Jan '11

Hey gang, Kamil asked to start pulling together some early builds for the upcoming 0.4.4 release. I've included a patch to update the spec file and adjust the %changelog. I tried to generate a meaningful (but not overly verbose) %changelog using the command: # git log --format="%h %gd %s (%aE)" --date short v0.4.3-1..HEAD Please let me know if I missed anything. Thanks, James

2 3

2026

2025

2024

2023

2022

2021

2020

2019

2018

2017

2016

2015

2014

2013

2012

2011

2010

2009

autoqa-devel January 2011