autoqa-devel April 2012

autoqa-devel@lists.fedorahosted.org

6 participants
21 discussions

Playing around with Sikuli
by Tim Flink 05 Jun '12

05 Jun '12

I've been wanting to try out Sikuli (http://www.sikuli.org) for a while now and finally got around to making it build and work on Fedora. The basic concept is to create GUI test cases based on images instead of absolute (x,y) coordinates (like selenium), scripted key sequences (openqa) or using an accessability layer (like dogtail). I've put together a tarball with source and build scripts if anyone else wants to try it out: http://tflink.fedorapeople.org/tools/sikuli/README_build-sikuli.txt http://tflink.fedorapeople.org/tools/sikuli/fedora-sikuli-20120220.tar.gz The distribution method is a really dirty hack but I've been able to build on both F15 and F16 after installing a huge number of -devel dependencies and building all of Sikuli's direct dependencies from source. Xpresser (https://wiki.ubuntu.com/Xpresser) might be another option - it's a re-implementation of the concepts from Sikuli. It seems to be missing some of Sikuli's features but would likely be quite a bit easier to build and integrate. One of the things I have in mind is to attempt automating at least some of the installation test cases because a lot of them pass most of the time and they can be a bit mind-numbing to do over and over again. I'm of the opinion that our human resources could be put to better use than monkey-button-pushing their way through simple test cases. A non-trivial amount of work would be needed before we could fully utilize this (fixing the code so that it is package-able, better integration with VNC etc.) but I'm wondering if it could be a good candidate for automating some of the installation test cases. I've written two test cases for Sikuli thus far: a simple DVD graphical install and graphical firstboot. I had to workaround some timing issues due to occasional lag in the VM's VNC interface but they seem to be working well with F17 Alpha RC2 for now. F17 DVD Basic Graphical Install: http://tflink.fedorapeople.org/tools/sikuli/f17-basicGraphicalDVD_sikuli.ta… F17 Graphical Firstboot: http://tflink.fedorapeople.org/tools/sikuli/f17-graphicalFirstboot_sikuli.t… - NOTE: This only works some of the time due to rhbz#750527 which was closed with no bug comment by the developers. I'm going to bug them about this but will likely update the test case to work with unpredictable order of the firstboot screens. To run those test cases, extract them somewhere and point the Sikuli IDE at the extracted directories. I'm going to be running my basic graphical install testcase for different F17 releases to get an initial feel for how fragile the test cases are and how much effort would be required to maintain these kinds of test cases over a release. Anyhow, long email over. Any thoughts on whether this might be worth investigating farther? Tim

3 4

[AutoQA] #415: results not being reported to bodhi (Expectation Error)
by fedora-badges 11 May '12

11 May '12

#415: results not being reported to bodhi (Expectation Error) -----------------------+------------------------ Reporter: tflink | Owner: Type: defect | Status: new Priority: critical | Milestone: Hot issues Component: core | Keywords: Blocked By: | Blocking: -----------------------+------------------------ I was just pinged regarding the lack of a depcheck run for a bodhi update - https://admin.fedoraproject.org/updates/abattis-cantarell- fonts-0.0.8-1.fc17 After looking into the logs: - http://autoqa.fedoraproject.org/results/315536-autotest/virt01.qa/debug/cli… we're getting errors when attempting to make the bodhi comment to report the results: {{{ 03/29 14:16:37 ERROR|bodhi_util:0249| An error occured: ServerError(https://admin.fedoraproject.org/updates/list, 417, Expectation Failed) 03/29 14:16:37 ERROR|bodhi_util:0250| Could not connect to bodhi! 03/29 14:16:37 ERROR|bodhi_util:0250| Could not post a comment to bodhi }}} I'm not sure what's going on here but will investigate and update here as I learn more. -- Ticket URL: <https://fedorahosted.org/autoqa/ticket/415> AutoQA <http://autoqa.fedorahosted.org> Automated QA project

1 6

autoqa-results usage questions
by Tim Flink 02 May '12

02 May '12

I guess that the autoqa-results list is still causing some problems for fedora-infra. I'm honestly not sure what the problem is with our current usage of the list - I didn't really get a clear answer to that, just that they want to understand more about our usage in case there is a better solution than what we're currently using. The old archives are going to be deleted but I've downloaded a compressed copy of the old archives in case we ever decide to parse them. I've been asked for the following information: - What do we do with the list - What do we plan to do with the list With the recent changes that Kamil made, the only thing that I can think of is "notification of failed or questionable test runs". Is there anything that I'm missing? Tim

2 4

Question about check-upgrade
by Sergio Belkin 18 Apr '12

18 Apr '12

Hi, I've received yesterday a mail about broken upgrade path of UpTools package. UpTools has the same EVR to both F15 and F16 too. So was it a bug of check-upgrade-paths.py isn't it? How often is run that test? Today I've followed instructions from http://jlaska.wordpress.com/2010/06/01/fedora-package-maintainers-want-test… is that right? I've run: autoqa-optin UpTools F-15 Tell me if I'm wrong -- -- Sergio Belkin http://www.sergiobelkin.com Watch More TV http://sebelk.blogspot.com LPIC-2 Certified - http://www.lpi.org

2 2

[RFC] Autotest API Fixes
by Lucas Meneghel Rodrigues 18 Apr '12

18 Apr '12

IMPORTANT: The results of this thread will affect 3rd parties developing on top of autotest in the near future. Hi guys, Due to the packaging requirements that we've been working during the past several months, I decided to finally tackle the challenge of doing some major cleanup on the autotest API structure. I have a patchset here: https://github.com/autotest/autotest/pull/294 It's a massive patchset, that was for the most of it auto generated, plus a bunch of fixes that I've been working on over the past week. With this, everyone developing code on top of autotest will have to perform changes on their code, so consider this as a heads up. I've tested the patches over the few weeks, and I'm up to the point I see no more (at least obvious) problems on it. Bottom line, the patches change the autotest structure in the following way -> No more autotest/client/bin dir -> API: autotest_lib -> autotest -> API: autotest.client.common_lib -> autotest.client.shared As an example, here's how the imports on client/bin/job.py looks in master: from autotest_lib.client.bin import client_logging_config from autotest_lib.client.bin import utils, parallel, kernel, xen from autotest_lib.client.bin import profilers, boottool, harness from autotest_lib.client.bin import config, sysinfo, test, local_host from autotest_lib.client.bin import partition as partition_lib from autotest_lib.client.common_lib import base_job from autotest_lib.client.common_lib import error, barrier, log, logging_manager from autotest_lib.client.common_lib import base_packages, packages from autotest_lib.client.common_lib import global_config from autotest_lib.client.tools import html_report And how they're going to look after the patchset is applied: from autotest.client import client_logging_config from autotest.client import utils, parallel, kernel, xen from autotest.client import profilers, boottool, harness from autotest.client import config, sysinfo, test, local_host from autotest.client import partition as partition_lib from autotest.client.shared import base_job from autotest.client.shared import error, barrier, log, logging_manager from autotest.client.shared import base_packages, packages from autotest.client.shared import global_config from autotest.client.tools import html_report I know this will cause inconveniences, but I believe it'll be well worth the effort. Please feel free to review the changes and give us your opinion. Thanks!

1 1

code review request
by Hongqing Yang 17 Apr '12

17 Apr '12

the rats_install has updated to run three times to test repo fedora, fedora+updates, fedora+updates+updates-testing. Please review the codes in branch rats_install and the diffs are as below: diff --git a/lib/autoqa/virtguest.py b/lib/autoqa/virtguest.py index ba9af47..53ac16a 100644 --- a/lib/autoqa/virtguest.py +++ b/lib/autoqa/virtguest.py @@ -215,8 +215,7 @@ class VirtGuest(object): virt_args += ['--extra-args', '%s ks=%s' % (extra_args, kickstart)] elif kickstart.startswith(".") or kickstart.startswith("/"): virt_args += ['--initrd-inject=%s' % kickstart] - virt_args += ['--extra-args', '%s ks=file:/%s' % (extra_args, - os.path.basename(kickstart))] + virt_args += ['--extra-args', '%s ks=file:/ks.cfg' % extra_args] if self.diskcount > 1: for n in range(1, self.diskcount): # e.g. 5 -> [1,2,3,4] diff --git a/tests/rats_install/control b/tests/rats_install/control index 2baffca..8e227d8 100644 --- a/tests/rats_install/control +++ b/tests/rats_install/control @@ -12,7 +12,4 @@ The test cases covered by this test are: """ MAINTAINER = "Hongqing Yang <hoyang(a)redhat.com>" -# run test with different packages group -for pkg in ['fedora','updates','updates-testing']: - autoqa_args['pkg'] = pkg - job.run_test('rats_install', tag=pkg, **autoqa_args) +job.run_test('rats_install', **autoqa_args) diff --git a/tests/rats_install/fedora.cfg b/tests/rats_install/fedora.cfg deleted file mode 100644 index 238c7b3..0000000 --- a/tests/rats_install/fedora.cfg +++ /dev/null @@ -1,26 +0,0 @@ -# Basic kickstart file for RATS -install -lang en_US.UTF-8 -keyboard us -network --device link --activate -rootpw --plaintext autoqa -timezone --utc America/New_York -# partitioning - nuke and start fresh -zerombr -clearpart --all --initlabel -autopart -bootloader --location=mbr -reboot -repo --name=fedora -# packages should be a superset of critpath -%packages -# @base should be selected by default -# critpath groups list taken from https://fedoraproject.org/wiki/Critical_path_package#Implementation -@core -@critical-path-base -@critical-path-gnome -@critical-path-apps -@critical-path-kde -@critical-path-lxde -@critical-path-xfce -%end diff --git a/tests/rats_install/install.py b/tests/rats_install/install.py index 7e040c0..429e8e7 100755 --- a/tests/rats_install/install.py +++ b/tests/rats_install/install.py @@ -81,7 +81,7 @@ def _get_parser(): return parser -def parse_args(args=None, pkg=None): +def parse_args(args=None): parser = _get_parser() (opt, args) = parser.parse_args(args) @@ -109,10 +109,8 @@ def parse_args(args=None, pkg=None): anaconda.logdir = opt.logdir # if ksfile is not specifited, use the local default one - if pkg == 'updates': - opt.ksfile = os.path.join(opt.bindir,'updates.cfg') - elif pkg == 'updates-testing': - opt.ksfile = os.path.join(opt.bindir,'updates-testing.cfg') + if not opt.ksfile: + opt.ksfile = os.path.join(opt.bindir,'ks.cfg') opt.keyval_file = os.path.join(opt.logdir, 'keyval') @@ -180,10 +178,10 @@ def system_sanity(): #http://fedoraproject.org/wiki/QA:Rawhide_Acceptance_Test_Plan#Installability_2 -def main(args=None, pkg=None): +def main(args=None): # Process command-line arguments - (opt, baseurl) = parse_args(args, pkg) + (opt, baseurl) = parse_args(args) if not system_sanity(): return False diff --git a/tests/rats_install/ks.cfg b/tests/rats_install/ks.cfg new file mode 100644 index 0000000..e38e523 --- /dev/null +++ b/tests/rats_install/ks.cfg @@ -0,0 +1,26 @@ +# Basic kickstart file for RATS +install +lang en_US.UTF-8 +keyboard us +network --device link --activate +rootpw --plaintext autoqa +timezone --utc America/New_York +# partitioning - nuke and start fresh +zerombr +clearpart --all --initlabel +autopart +bootloader --location=mbr +reboot + +# packages should be a superset of critpath +%packages +# @base should be selected by default +# critpath groups list taken from https://fedoraproject.org/wiki/Critical_path_package#Implementation +@core +@critical-path-base +@critical-path-gnome +@critical-path-apps +@critical-path-kde +@critical-path-lxde +@critical-path-xfce +%end diff --git a/tests/rats_install/rats_install.py b/tests/rats_install/rats_install.py index 2a29f36..35c53c5 100644 --- a/tests/rats_install/rats_install.py +++ b/tests/rats_install/rats_install.py @@ -50,9 +50,9 @@ class rats_install(AutoQATest): self.detail.arch = arch # set self.detail.id if name: - self.detail.id = "%s-%s-%s" % (name, arch, kwargs['pkg']) + self.detail.id = "%s-%s" % (name, arch) else: - self.detail.id = kwargs['pkg']+':'+baseurl + self.detail.id = baseurl @ExceptionCatcher() @@ -71,7 +71,7 @@ class rats_install(AutoQATest): sys.stdout = util.Tee(sys.stdout, out, autoclose=False) sys.stderr = util.Tee(sys.stderr, out, autoclose=False) # run the test - success = install.main(args,kwargs['pkg']) + success = install.main(args) # collect results if success: self.detail.result = 'PASSED' diff --git a/tests/rats_install/updates-testing.cfg b/tests/rats_install/updates-testing.cfg deleted file mode 100644 index 761474c..0000000 --- a/tests/rats_install/updates-testing.cfg +++ /dev/null @@ -1,28 +0,0 @@ -# Basic kickstart file for RATS -install -lang en_US.UTF-8 -keyboard us -network --device link --activate -rootpw --plaintext autoqa -timezone --utc America/New_York -# partitioning - nuke and start fresh -zerombr -clearpart --all --initlabel -autopart -bootloader --location=mbr -reboot -repo --name=fedora -repo --name=updates -repo --name=updates-testing -# packages should be a superset of critpath -%packages -# @base should be selected by default -# critpath groups list taken from https://fedoraproject.org/wiki/Critical_path_package#Implementation -@core -@critical-path-base -@critical-path-gnome -@critical-path-apps -@critical-path-kde -@critical-path-lxde -@critical-path-xfce -%end diff --git a/tests/rats_install/updates.cfg b/tests/rats_install/updates.cfg deleted file mode 100644 index 4df9dc7..0000000 --- a/tests/rats_install/updates.cfg +++ /dev/null @@ -1,27 +0,0 @@ -# Basic kickstart file for RATS -install -lang en_US.UTF-8 -keyboard us -network --device link --activate -rootpw --plaintext autoqa -timezone --utc America/New_York -# partitioning - nuke and start fresh -zerombr -clearpart --all --initlabel -autopart -bootloader --location=mbr -reboot -repo --name=fedora -repo --name=updates -# packages should be a superset of critpath -%packages -# @base should be selected by default -# critpath groups list taken from https://fedoraproject.org/wiki/Critical_path_package#Implementation -@core -@critical-path-base -@critical-path-gnome -@critical-path-apps -@critical-path-kde -@critical-path-lxde -@critical-path-xfce -%end

1 0

Bodhi bug: don't allow the same build in two different updates
by Kamil Paral 17 Apr '12

17 Apr '12

Some updates today ended with ABORTED when running depcheck. It seems to be caused by Bodhi allowing the same build to be in two different updates. Reported here: https://fedorahosted.org/bodhi/ticket/682

1 0

autoqa-results now receive only aborted tests
by Kamil Paral 16 Apr '12

16 Apr '12

Hello, because ResultsDB instances work fine, I decided to lower the burden on fedora mailman and our mailboxes and disabled "ok" test results notifications coming from autoqa production server into autoqa-results ML. This is now set in autoqa.conf both on production and staging: result_email_blacklist = ^.*: (PASSED|FAILED|INFO);.*$ That means autoqa-results will receive only CRASHED, ABORTED and NEEDS_INSPECTION emails. ResultsDB doesn't support any notification system yet, so it's useful. All the rest can be found in ResultsDBs. Kamil

1 0

[AutoQA] #418: rpmguard: IndexError: list index out of range
by fedora-badges 05 Apr '12

05 Apr '12

#418: rpmguard: IndexError: list index out of range ---------------------+------------------------ Reporter: kparal | Owner: Type: defect | Status: new Priority: minor | Milestone: Hot issues Component: tests | Keywords: Blocked By: | Blocking: ---------------------+------------------------ A crash! http://autoqa.fedoraproject.org/results/319127-autotest/virt03.qa/rpmguard/… {{{ N: Comparing openchange-devel-0.11-4.fc17 and openchange-devel-1.0-1.fc17 (archs: x86_64) ... W: requirement-added libmapiserver.so.0()(64bit) W: requirement-added openchange-server = 1.0-1.fc17 Traceback (most recent call last): File "/usr/share/autotest/tests/rpmguard/rpmguard", line 371, in <module> main() File "/usr/share/autotest/tests/rpmguard/rpmguard", line 112, in main printPRCOchanges(output, 'CONFLICTS', 'conflict') File "/usr/share/autotest/tests/rpmguard/rpmguard", line 317, in printPRCOchanges rem_names = [line.split()[2] for line in removed] IndexError: list index out of range }}} -- Ticket URL: <https://fedorahosted.org/autoqa/ticket/418> AutoQA <http://autoqa.fedorahosted.org> Automated QA project

1 0

Fwd: Dependencies on Bodhi Updates
by Kamil Paral 02 Apr '12

02 Apr '12

This is a highly-relevant discussion from our perspective. Linking here. ----- Forwarded Message ----- From: "Stephen Gallagher" <sgallagh(a)redhat.com> To: devel(a)lists.fedoraproject.org Sent: Monday, March 26, 2012 9:53:09 PM Subject: Dependencies on Bodhi Updates As requested during the FESCo meeting, I am going to try to summarize some of the issues inherent in the way that Bodhi updates currently work. First, I'll try to explain the goals and constraints: 1) The stable 'fedora-updates' yum repository should NEVER exist in a state where any package has dependency issues. In other words, it should never be possible for an update to be pushed to stable that breaks cleanly updating any other package. 2) Updates must be possible and (ideally) timely. This is probably self-evident. 3) Packages pushed to the stable 'fedora-updates' yum repository should (ideally) not introduce regressions in packages that depend on them. 4) New features in "superpackages" such as Firefox, GNOME or FreeIPA that have many and varied dependencies may require new features in packages they depend on in order to enhance or fix the superpackage. In the trivial example, a package (let's say libtalloc) needs to make an update to fix a bug. This package requires nothing new from its dependencies and is a self-contained fix. For this example, it is simple to just build libtalloc in koji and then create a Bodhi update and pass it through "updates-testing", get karma and *poof* off to "fedora-updates". Now let's extend the example. Suppose that we have another package libtevent that has libtalloc as a dependency. Libtevent's maintainer wants to add a new feature to libtevent, but the patch from upstream depends on the bug in libtalloc having been fixed in order for the new feature to work properly. In this situation, the maintainer of libtevent would build libtevent with an explicit Requires: libtalloc >= <version> in the specfile (possibly pulling libtalloc into the BuildRoot overrides if necessary) and then test it locally to see that it works. So now we have our first updates dependency issue. If we submit libtevent as its own update, it is possible that it will achieve its karma requirement before libtalloc does. It would then be pushed to the "fedora-updates" repository and then introduce a dependency issue in the stable repository (because users trying to update libtevent would be unable to update libtalloc without enabling the updates-testing repository). The current recommended approach is to bundle the two updates into a single one carrying multiple packages. The first problem with this is that you must have commit privilege on all packages that you are bundling into an update. If you do not, then you need to track down a provenpackager to do it for you. Now let's make the problem even more fun. Consider that the update to libtevent might be coming in because it is necessary for a new feature in libldb, which is in turn providing new functionality necessary for SSSD. So now we have four packages all sitting in the same update. The problem with this is that the tendency will be to only test the most user-visible package(s) in the set. In this particular case, that might be SSSD. So people would likely test SSSD and, if nothing went wrong, consider the entire update stable. But wait! SSSD isn't the only package that depends on libldb, libtevent and libtalloc. So too does the samba package. Suppose that the bugfix in libtalloc, after resolving the original issue, results in exposing another more serious bug in samba? Now we need to pull a samba update into this same update series. A contrived example, you say? That would never happen, bugfixes aren't likely to do that. Well, for one example: https://admin.fedoraproject.org/updates/FEDORA-2011-11845 In this particular example, we knew up-front that it was going to necessitate a rebuild of several dependent packages and we coordinated a single release to address them. So in this case, the proper approach was to bundle them together in a single update. This worked because we specifically knew that the libtevent change was going to break other packages. But what about when we don't know that? Let's take another example: https://admin.fedoraproject.org/updates/FEDORA-2011-17399 In this case, there was a security bug reported against Firefox. Such things are serious, and acted on quickly. However, the bug was actually fixed in the nss package, and Firefox, Xulrunner and friends were rebuilt against that nss package. The problem was this: the fix made to the nss package introduced regressions in every other package that depended on it. However, because the default install of Firefox contained no issues, it rapidly received the necessary karma points and the whole update was pushed to stable. It then broke nearly every application in Fedora that relied on cryptography. The problem here was sociological, not technological. The only package that received testing was Firefox. It's hard to say without evidence whether the problem would have been averted by having nss go through its own update, but I strongly suspect that what we would have seen was greater testing on actual nss features for that specific update. Of course, we now have the same potential for an issue that I described above: If we had separate updates for nss and for Firefox, chances would be highly-likely that Firefox would be pushed to stable via karma points rapidly, whereas nss (which requires much more careful testing) might be left behind in updates-testing. So I really see two options for improving these situations: 1) https://fedorahosted.org/bodhi/ticket/663 I opened this ticket two months ago (to silence). The idea would be to add the ability for bodhi updates to mark other updates as a dependency, so that in the example above, Firefox could have been marked as ready for stable, but not pushed until the nss update was also marked as ready for stable. This to me seems like the best long-term solution. I'd also like to mention that Ubuntu's Launchpad system has this capability. 2) We could continue on the "single update for multiple packages" approach, but revamp the karma system so that each SRPM gets its own karma, rather than the update as a whole. Then, the whole update would not be pushed via autokarma until all of the dependent packages had sufficient karma (or the owner of the update could push them after the stable wait period, of course). Discuss. -- devel mailing list devel(a)lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/devel

1 0

2024

2023

2022

2021

2020

2019

2018

2017

2016

2015

2014

2013

2012

2011

2010

2009

autoqa-devel April 2012