autoqa-devel September 2010

autoqa-devel@lists.fedorahosted.org

8 participants
88 discussions

repoinfo.conf and older releases
by James Laska 22 Sep '10

22 Sep '10

Greetings, Currently, we don't remove old releases from repoinfo.conf. However, we do mark some of them (only the dist-fXX entries) as 'isactiverelease=no'. This affects some test results in that we don't have a way to determine what releases are no longer needed for testing. Specifically, this affects the post-repo-update watcher, and the upgradepath test. For example with upgradepath, we are testing a proposed f13-updates-testing package against everything from dist-f10 to dist-f15 (see sample result [1]). While the results from older unmaintained releases may be interesting, they shouldn't affect the test outcome. Some options to address this ... 1. Use isactiverelease more - In a recent commit (6b3ae44dee122b030bcede1a2059a7533ee8203f), we introduced the 'isactiverelease' repoinfo.conf option. This is currently only used on the unmaintained dist-fNN entires. I'd propose we use this for all unmaintained repoinfo.conf entries (includes -updates and -updates-testing) *and* updating any tests or scripts to honor this option. 2. Remove (or comment out) unmaintained releases - Certainly an option, but my preference is #1 for some reason. But if folks overwhelmingly feel that just removing older entries is ideal ... I can't think of any objections. Thoughts/concerns/comments? Note, I think there is another upgradepath problem present in this test ... I'll leave that for another mail (any takers on what it is?). Thanks, James [1] https://fedorahosted.org/pipermail/autoqa-results/2010-September/037544.html

1 0

#201 - First spin of using MASH
by Josef Skladanka 22 Sep '10

22 Sep '10

Hello gang! Script provided in the attached patch contains: - mash config file (autoqa.mash) - magical script (mash_packages.py) mash_packages.py is a script, which downloads packages from koji (according to the supplied envr), creates a repository structure using these packages, and then runs MASH on this repository. Be aware, that MASH only supports ['ppc', 'x86_64', 'sparc', 's390x'] Other architectures are skipped (as there is no need for mash there, i guess). To run the script you need to: - install the mash (yum install mash) - symlink (or copy) the autoqa.mash to /etc/mash to run the script, simply call it like this python mash_package.py /tmp/some_dir curl-7.19.6-10.fc12 The first argument is the directory in which the repo will be created, the second is envr (curl is multilib btw.) I'll be glad for any feedback Joza

2 1

Minor bug fixes and improvements to 0.4.1-1
by James Laska 22 Sep '10

22 Sep '10

Greetings, As noted earlier, I deployed autoqa-0.4.1-1 yesterday. It has been quietly running jobs for about a day now. After reviewing test results, I noticed several issues. This mail attempts to summarize the issues and propose patches to correct the problems. = conflicts test failures = 1) Failing to find packages or mirrors This appears to be a result of running the test in a different environment. When run previously, download.fedoraproject.org would result to an internal Red Hat mirror which was (for the most part) always updated and accurate. With our tests now running in Fedora infrastructure, we are at the mercy of MirrorManager to provide an appropriate mirror. It seems that we rarely get an updated mirror. I have 2 proposed solutions to this problem. 1. Force our deployed systems to use the infrastructure mirro by updating repoinfo.conf appropriately (tested and works) 2. Request MirrorManager configuration change so that any requests from infrastructure systems are always routed to the private infrastructure mirror 2) Sloooooow.... This test seems to run for a *long* time now (~15 minutes). I seem to remember this taking, maybe 30 seconds tops. Perhaps something with Fedora 13 changed how this test runs. Not urgent, but perhaps something to follow-up with skvidal for guidance/ideas on. Option#2 is my preference as this mirrors our internal setup, and keeps our test+configuration consistent inside and outside of Fedora infrastructure. I'll raise this idea with the infrastructure team shortly. UPDATE: mmcgrath updated MirrorManager so option#2 seems like the way forward. This change should propogate sometime today. = upgradepath test failures = > https://fedorahosted.org/pipermail/autoqa-results/2010-September/035783.html > Traceback (most recent call last): > File "/usr/share/autotest/common_lib/test.py", line 570, in _call_test_function > return func(*args, **dargs) > File "/usr/share/autotest/common_lib/test.py", line 279, in execute > postprocess_profiled_run, args, dargs) > File "/usr/share/autotest/common_lib/test.py", line 201, in _call_run_once > self.run_once(*args, **dargs) > File "/usr/lib/python2.6/site-packages/autoqa/decorators.py", line 71, in newf > f_result = f(*args, **kwargs) #call the decorated function > File "/usr/share/autotest/tests/upgradepath/upgradepath.py", line 92, in run_once > assert kojitag in repotags, 'Requested unsupported kojitag: %s' % kojitag > AssertionError: Requested unsupported kojitag: dist-f14-updates-testing See proposed patch. I've adjusted upgradepath.py so that it ... 1) Uses autoqa.repoinfo directly, and not through autoqa.koji_utils.repoinfo 2) Includes '-testing' repos when attempting to find applicable tags = autoqa script = 1) Do not reboot before or after running a job I'd like to include a previously posted (but not accepted) patch that prevents test clients from rebooting before or after tests. We have a *lot* of tests running now, and reboots are expensive as jobs queue up while a system is out of action. I've been manually applying this patch to our "production" instance for months now. It's safe to include, the only question I have is .. "is there a better, more appropriate way, to make this change?" 2) Restrict known_hooks to sub-directories of hookdir Since I moved /etc/cron.d/autoqa to /usr/share/autoqa/autoqa.cron, it was being listed as a valid hook. I've changed the known_hooks filter to only list sub-directories. I've also moved autoqa.cron into %{_docdir} in another patch. Moving it there is in line with several other packages that include recommended cron files. = rats_sanity = > https://fedorahosted.org/pipermail/autoqa-results/2010-September/035666.html This seems to be straightforward, corrected a missing import. I've tested this fix and it's in production now. With this fix, the test proceeds but encounters another problem. I believe this may be a valid failure discovered by the test (see https://fedorahosted.org/pipermail/autoqa-results/2010-September/035811.html) = autoqa.spec = 1) Move autoqa.cron into the %{_docdir} 2) Bump the release to autoqa-0.4.2-1 Thanks, James

3 16

rpmlint improvements
by Alexander Todorov 21 Sep '10

21 Sep '10

Hello folks, I want to discuss topics in tickets: https://fedorahosted.org/autoqa/ticket/149 https://fedorahosted.org/autoqa/ticket/150 I did a test run of rpmlint against the latest RHEL6 tree and filed some bugs. Some were legitimate but others turned out to be false negatives and developers got angry. Then it turned out that rpmlint needs more features in order to be used reliably without making developers unhappy. So I talked to twoerner who is rpmlint maintainer for RHEL6 and exchanged some ideas. What we think is a good idea and needs to be present in upstream is having a rpmlint-data package which will provide rules.$pkg-name files for packages that need special handling. Then rpmlint will take into account those files and act accordingly. The rules files will be maintained by a separate maintainer and not by the packagers. Also any change into those rules/exceptions needs to be approved with a review process to avoid pkg maintainers manipulating the rules simply to silence out rpmlint output. This will guarantee accurate results. This data package will be optional to rpmlint and not affect core functionality. It will most likely be different for Fedora and RHEL as well. This is more or less the same as proposed in ticket 149 with lintian tool for Debian. What do you think? Any drawbacks/gotchas you may think of? The rules files format need to support the following (as identified so far): * List file paths/directories with special permissions and uid/gid settings. For example CUPS has such. audit also has such (in RHEL for certification purposes) * Ignore some test cases for particular package(or file). For example: gcc.x86_64: E: devel-dependency glibc-devel gcc.x86_64: W: devel-file-in-non-devel-package This looks normal since gcc is a compiler but will be an error for another package (e.g. an application). jlaska told me that Fedora QA already uses rpmlint. Have you guys identified other areas/tests that need special handling or exclusion ? What are those? Have you thought about the file format? It needs to be easy to maintain and easy to read - probably just plain text. It also needs to be flexible so that we can add to it without breaking compatibility. We can probably have a separate file (format) for every test/package that rpmlint performs. Just to let you know that I'm happy to work on the rpmlint code and push it upstream if noone else has spare cycles. Regards, Alexander.

1 1

[AutoQA] #227: duplicate results
by fedora-badges 17 Sep '10

17 Sep '10

#227: duplicate results ----------------------------+----------------------------------------------- Reporter: notting | Owner: Type: defect | Status: new Priority: major | Milestone: Component: infrastructure | Version: 1.0 Keywords: | ----------------------------+----------------------------------------------- I built initscripts-9.21-1.fc15. I got duplicate results from AutoQA for rpmlint and rpmguard. One said (for example): Stored logs available at <http://autoqa.fedoraproject.org/results/3894-autotest/qa03.c.fedoraproject.…> The other said: Stored logs available at <http://test1250.test.redhat.com/results/50486-autotest/brutus.test.redhat.c…> The test results were slightly different as well. -- Ticket URL: <https://fedorahosted.org/autoqa/ticket/227> AutoQA <http://autoqa.fedorahosted.org> Automated QA project

1 1

[AutoQA] #225: Package custom yum-utils to fix bug#622710
by fedora-badges 17 Sep '10

17 Sep '10

#225: Package custom yum-utils to fix bug#622710 -----------------------+---------------------------------------------------- Reporter: jlaska | Owner: jlaska Type: task | Status: new Priority: minor | Milestone: Packaging, Review, & Deployment Component: packaging | Version: 1.0 Keywords: | -----------------------+---------------------------------------------------- As noted by Kamil, a new yum-utils package is needed by autoqa tests in order to address [https://bugzilla.redhat.com/show_bug.cgi?id=622710#c1 bug#622710]. I'll take this ticket and build a new yum-utils that contains the current released version (''yum-utils-1.1.28-1.fc13.noarch'') and the [http://yum.baseurl.org/gitweb?p=yum- utils.git;a=commitdiff_plain;h=92ea5b33601015d9b2fb009f5ebd2aac6b6dd2df upstream patch referenced in the bugzilla]. -- Ticket URL: <https://fedorahosted.org/autoqa/ticket/225> AutoQA <http://autoqa.fedorahosted.org> Automated QA project

1 2

Re: [PATCH 0/3] multihook patchset - completion
by Kamil Paral 16 Sep '10

16 Sep '10

----- "Will Woods" <wwoods(a)redhat.com> wrote: > On Tue, 2010-09-14 at 13:04 +0200, Kamil Páral wrote: > > This is the remaining part of the multihook patchset. Now all raised > exceptions > > should be reported as CRASHED, together with a short exception > description in > > the summary. All tests were modified not to use exceptions for > ending > > correctly. > > > > All this code is also available in multitests branch, so it's enough > to just > > merge the whole branch. > > All the patches look good to me. Merged into master. Thanks a lot! Great, thanks! > > I think James may be working on rolling this into a new (0.4.5?) > autoqa > release shortly. 0.4.1 would seem more appropriate to me :) > > -w > > _______________________________________________ > autoqa-devel mailing list > autoqa-devel(a)lists.fedorahosted.org > https://fedorahosted.org/mailman/listinfo/autoqa-devel

3 2

[AutoQA] #221: Convert package repository to repos.fedorapeople.org/autoqa
by fedora-badges 16 Sep '10

16 Sep '10

#221: Convert package repository to repos.fedorapeople.org/autoqa -----------------------+---------------------------------------------------- Reporter: jlaska | Owner: Type: task | Status: new Priority: minor | Milestone: Packaging, Review, & Deployment Component: packaging | Version: 1.0 Keywords: | -----------------------+---------------------------------------------------- It would make me feel more comfortable having the repositories less tied to my name. It seems to transient that way, and for some reason, feels one step closer to permanent if the packages are housed at http://repos.fedorapeople.org -- Ticket URL: <https://fedorahosted.org/autoqa/ticket/221> AutoQA <http://autoqa.fedorahosted.org> Automated QA project

1 1

[PATCH 0/3] multihook patchset - completion
by Kamil Paral 15 Sep '10

15 Sep '10

This is the remaining part of the multihook patchset. Now all raised exceptions should be reported as CRASHED, together with a short exception description in the summary. All tests were modified not to use exceptions for ending correctly. All this code is also available in multitests branch, so it's enough to just merge the whole branch. Josef Skladanka (1): ExceptionCatcher hands over the Exception object Kamil Páral (2): raise exception only when you want to report the result as CRASHED be more concise when run_once fails, let's have shorter email subjects lib/python/decorators.py | 4 ++-- lib/python/test.py | 22 ++++++++++++---------- tests/rats_install/rats_install.py | 2 -- tests/rats_sanity/rats_sanity.py | 4 +--- tests/repoclosure/repoclosure.py | 3 --- tests/upgradepath/upgradepath.py | 15 ++------------- 6 files changed, 17 insertions(+), 33 deletions(-) -- 1.7.2.2

2 5

Re: Creating autoqa-watchers sub-package?
by Kamil Paral 14 Sep '10

14 Sep '10

----- "James Laska" <jlaska(a)redhat.com> wrote: > Greetings, > > As autoqa is currently packaged, all tests, watchers, configs and > cronjobs are included in the main autoqa package. This works fine > except for a minor annoyance. Anytime you install 'autoqa', unless > you > are actually scheduling jobs, you needed comment or remove the > watcher > notification /etc/cron.d/autoqa. This gets annoying the more I > deploy > autoqa. > > I'd like propose moving the watcher scripts and cronjob into a > sub-package (see attached patch for detail). > > When most people install 'autoqa', they want the test library and > tests. > This won't change that behavior. The only change will be for anyone > setting up a test server. I'll need to update the existing wiki > documentation to note installing 'autoqa-watchers' when setting up a > test server [1]. > > Comments/concerns/ideas? Splitting is a good idea. Just a remark: If you put watchers (i.e. hooks) into a separate package but leave autoqa harness in the original package, then the autoqa harness can't be used (it doesn't work separately, it needs hook.py files to operate). So people won't be able to just try and run some test, even manually. I see several ways: 1. Let's create autotest and autotest-client packages. Autotest-client will contain only stuff really required for clients, i.e. autotest and autoqa library. Autotest package will contain the rest. Easy to deploy, but people that want to experiment with it must install all (and disable cron jobs). 2. Your idea, autotest and autotest-watchers packages. Easy to deploy, people still can't just run a test without the full installation. But this time they can at least have a look at the tests' source code. 3. Creating autotest and autotest-server packages. Autotest-server will contain just the crontab file. Silly? :) 4. What about commenting out the crontab file by default? If someone really wants to deploy a server, he will just uncomment the file (and maybe tweak the settings anyway). That file is left intact on upgrade, so that shouldn't be a problem, it's a one-time task. And we will document that on the wiki (the same way we have now documented that everyone not wanting to run a server should comment out that file). I would like to provide people easy way to try and play with autoqa. Although the last solution seems to be extremely simple, it seems to work well enough. Would that help you too? Or do you prefer another way? What do you think?

2 2

← Newer
1
2
3
4
5
6
7
8
9
Older →

2025

2024

2023

2022

2021

2020

2019

2018

2017

2016

2015

2014

2013

2012

2011

2010

2009

autoqa-devel September 2010