repoinfo.conf and older releases
by James Laska
Greetings,
Currently, we don't remove old releases from repoinfo.conf. However, we
do mark some of them (only the dist-fXX entries) as
'isactiverelease=no'. This affects some test results in that we don't
have a way to determine what releases are no longer needed for testing.
Specifically, this affects the post-repo-update watcher, and the
upgradepath test.
For example with upgradepath, we are testing a proposed
f13-updates-testing package against everything from dist-f10 to dist-f15
(see sample result [1]). While the results from older unmaintained
releases may be interesting, they shouldn't affect the test outcome.
Some options to address this ...
1. Use isactiverelease more - In a recent commit
(6b3ae44dee122b030bcede1a2059a7533ee8203f), we introduced the
'isactiverelease' repoinfo.conf option. This is currently only
used on the unmaintained dist-fNN entires. I'd propose we use
this for all unmaintained repoinfo.conf entries (includes
-updates and -updates-testing) *and* updating any tests or
scripts to honor this option.
2. Remove (or comment out) unmaintained releases - Certainly an
option, but my preference is #1 for some reason. But if folks
overwhelmingly feel that just removing older entries is
ideal ... I can't think of any objections.
Thoughts/concerns/comments?
Note, I think there is another upgradepath problem present in this
test ... I'll leave that for another mail (any takers on what it is?).
Thanks,
James
[1]
https://fedorahosted.org/pipermail/autoqa-results/2010-September/037544.html
13 years, 7 months
#201 - First spin of using MASH
by Josef Skladanka
Hello gang!
Script provided in the attached patch contains:
- mash config file (autoqa.mash)
- magical script (mash_packages.py)
mash_packages.py is a script, which downloads packages
from koji (according to the supplied envr), creates a
repository structure using these packages, and then
runs MASH on this repository.
Be aware, that MASH only supports ['ppc', 'x86_64', 'sparc', 's390x']
Other architectures are skipped (as there is no need for
mash there, i guess).
To run the script you need to:
- install the mash (yum install mash)
- symlink (or copy) the autoqa.mash to /etc/mash
to run the script, simply call it like this
python mash_package.py /tmp/some_dir curl-7.19.6-10.fc12
The first argument is the directory in which the repo will
be created, the second is envr (curl is multilib btw.)
I'll be glad for any feedback
Joza
13 years, 7 months
Minor bug fixes and improvements to 0.4.1-1
by James Laska
Greetings,
As noted earlier, I deployed autoqa-0.4.1-1 yesterday. It has been quietly
running jobs for about a day now. After reviewing test results, I noticed
several issues. This mail attempts to summarize the issues and propose patches
to correct the problems.
= conflicts test failures =
1) Failing to find packages or mirrors
This appears to be a result of running the test in a different environment.
When run previously, download.fedoraproject.org would result to an internal Red
Hat mirror which was (for the most part) always updated and accurate. With our
tests now running in Fedora infrastructure, we are at the mercy of
MirrorManager to provide an appropriate mirror. It seems that we rarely get an
updated mirror. I have 2 proposed solutions to this problem.
1. Force our deployed systems to use the infrastructure mirro by updating
repoinfo.conf appropriately (tested and works)
2. Request MirrorManager configuration change so that any requests from
infrastructure systems are always routed to the private infrastructure
mirror
2) Sloooooow....
This test seems to run for a *long* time now (~15 minutes). I seem to remember
this taking, maybe 30 seconds tops. Perhaps something with Fedora 13 changed
how this test runs. Not urgent, but perhaps something to follow-up with
skvidal for guidance/ideas on.
Option#2 is my preference as this mirrors our internal setup, and keeps our
test+configuration consistent inside and outside of Fedora infrastructure.
I'll raise this idea with the infrastructure team shortly.
UPDATE: mmcgrath updated MirrorManager so option#2 seems like the way forward.
This change should propogate sometime today.
= upgradepath test failures =
> https://fedorahosted.org/pipermail/autoqa-results/2010-September/035783.html
> Traceback (most recent call last):
> File "/usr/share/autotest/common_lib/test.py", line 570, in _call_test_function
> return func(*args, **dargs)
> File "/usr/share/autotest/common_lib/test.py", line 279, in execute
> postprocess_profiled_run, args, dargs)
> File "/usr/share/autotest/common_lib/test.py", line 201, in _call_run_once
> self.run_once(*args, **dargs)
> File "/usr/lib/python2.6/site-packages/autoqa/decorators.py", line 71, in newf
> f_result = f(*args, **kwargs) #call the decorated function
> File "/usr/share/autotest/tests/upgradepath/upgradepath.py", line 92, in run_once
> assert kojitag in repotags, 'Requested unsupported kojitag: %s' % kojitag
> AssertionError: Requested unsupported kojitag: dist-f14-updates-testing
See proposed patch. I've adjusted upgradepath.py so that it ...
1) Uses autoqa.repoinfo directly, and not through autoqa.koji_utils.repoinfo
2) Includes '-testing' repos when attempting to find applicable tags
= autoqa script =
1) Do not reboot before or after running a job
I'd like to include a previously posted (but not accepted) patch that prevents
test clients from rebooting before or after tests. We have a *lot* of tests
running now, and reboots are expensive as jobs queue up while a system is out
of action.
I've been manually applying this patch to our "production" instance for months
now. It's safe to include, the only question I have is .. "is there a better,
more appropriate way, to make this change?"
2) Restrict known_hooks to sub-directories of hookdir
Since I moved /etc/cron.d/autoqa to /usr/share/autoqa/autoqa.cron, it was being
listed as a valid hook. I've changed the known_hooks filter to only list
sub-directories. I've also moved autoqa.cron into %{_docdir} in another patch.
Moving it there is in line with several other packages that include recommended
cron files.
= rats_sanity =
> https://fedorahosted.org/pipermail/autoqa-results/2010-September/035666.html
This seems to be straightforward, corrected a missing import. I've tested this
fix and it's in production now.
With this fix, the test proceeds but encounters another problem. I believe
this may be a valid failure discovered by the test (see
https://fedorahosted.org/pipermail/autoqa-results/2010-September/035811.html)
= autoqa.spec =
1) Move autoqa.cron into the %{_docdir}
2) Bump the release to autoqa-0.4.2-1
Thanks,
James
13 years, 7 months
rpmlint improvements
by Alexander Todorov
Hello folks,
I want to discuss topics in tickets:
https://fedorahosted.org/autoqa/ticket/149
https://fedorahosted.org/autoqa/ticket/150
I did a test run of rpmlint against the latest RHEL6 tree and filed some bugs.
Some were legitimate but others turned out to be false negatives and developers
got angry.
Then it turned out that rpmlint needs more features in order to be used reliably
without making developers unhappy. So I talked to twoerner who is rpmlint
maintainer for RHEL6 and exchanged some ideas.
What we think is a good idea and needs to be present in upstream is having a
rpmlint-data package which will provide rules.$pkg-name files for packages that
need special handling. Then rpmlint will take into account those files and act
accordingly. The rules files will be maintained by a separate maintainer and not
by the packagers. Also any change into those rules/exceptions needs to be
approved with a review process to avoid pkg maintainers manipulating the rules
simply to silence out rpmlint output. This will guarantee accurate results.
This data package will be optional to rpmlint and not affect core functionality.
It will most likely be different for Fedora and RHEL as well. This is more or
less the same as proposed in ticket 149 with lintian tool for Debian.
What do you think? Any drawbacks/gotchas you may think of?
The rules files format need to support the following (as identified so far):
* List file paths/directories with special permissions and uid/gid settings. For
example CUPS has such. audit also has such (in RHEL for certification purposes)
* Ignore some test cases for particular package(or file). For example:
gcc.x86_64: E: devel-dependency glibc-devel
gcc.x86_64: W: devel-file-in-non-devel-package
This looks normal since gcc is a compiler but will be an error for another
package (e.g. an application).
jlaska told me that Fedora QA already uses rpmlint. Have you guys identified
other areas/tests that need special handling or exclusion ? What are those?
Have you thought about the file format? It needs to be easy to maintain and easy
to read - probably just plain text. It also needs to be flexible so that we can
add to it without breaking compatibility. We can probably have a separate file
(format) for every test/package that rpmlint performs.
Just to let you know that I'm happy to work on the rpmlint code and push it
upstream if noone else has spare cycles.
Regards,
Alexander.
13 years, 7 months
Re: [PATCH 0/3] multihook patchset - completion
by Kamil Paral
----- "Will Woods" <wwoods(a)redhat.com> wrote:
> On Tue, 2010-09-14 at 13:04 +0200, Kamil Páral wrote:
> > This is the remaining part of the multihook patchset. Now all raised
> exceptions
> > should be reported as CRASHED, together with a short exception
> description in
> > the summary. All tests were modified not to use exceptions for
> ending
> > correctly.
> >
> > All this code is also available in multitests branch, so it's enough
> to just
> > merge the whole branch.
>
> All the patches look good to me. Merged into master. Thanks a lot!
Great, thanks!
>
> I think James may be working on rolling this into a new (0.4.5?)
> autoqa
> release shortly.
0.4.1 would seem more appropriate to me :)
>
> -w
>
> _______________________________________________
> autoqa-devel mailing list
> autoqa-devel(a)lists.fedorahosted.org
> https://fedorahosted.org/mailman/listinfo/autoqa-devel
13 years, 7 months
[PATCH 0/3] multihook patchset - completion
by Kamil Paral
This is the remaining part of the multihook patchset. Now all raised exceptions
should be reported as CRASHED, together with a short exception description in
the summary. All tests were modified not to use exceptions for ending
correctly.
All this code is also available in multitests branch, so it's enough to just
merge the whole branch.
Josef Skladanka (1):
ExceptionCatcher hands over the Exception object
Kamil Páral (2):
raise exception only when you want to report the result as CRASHED
be more concise when run_once fails, let's have shorter email
subjects
lib/python/decorators.py | 4 ++--
lib/python/test.py | 22 ++++++++++++----------
tests/rats_install/rats_install.py | 2 --
tests/rats_sanity/rats_sanity.py | 4 +---
tests/repoclosure/repoclosure.py | 3 ---
tests/upgradepath/upgradepath.py | 15 ++-------------
6 files changed, 17 insertions(+), 33 deletions(-)
--
1.7.2.2
13 years, 7 months
Re: Creating autoqa-watchers sub-package?
by Kamil Paral
----- "James Laska" <jlaska(a)redhat.com> wrote:
> Greetings,
>
> As autoqa is currently packaged, all tests, watchers, configs and
> cronjobs are included in the main autoqa package. This works fine
> except for a minor annoyance. Anytime you install 'autoqa', unless
> you
> are actually scheduling jobs, you needed comment or remove the
> watcher
> notification /etc/cron.d/autoqa. This gets annoying the more I
> deploy
> autoqa.
>
> I'd like propose moving the watcher scripts and cronjob into a
> sub-package (see attached patch for detail).
>
> When most people install 'autoqa', they want the test library and
> tests.
> This won't change that behavior. The only change will be for anyone
> setting up a test server. I'll need to update the existing wiki
> documentation to note installing 'autoqa-watchers' when setting up a
> test server [1].
>
> Comments/concerns/ideas?
Splitting is a good idea. Just a remark: If you put watchers (i.e. hooks)
into a separate package but leave autoqa harness in the original package,
then the autoqa harness can't be used (it doesn't work separately, it needs
hook.py files to operate). So people won't be able to just try and run some
test, even manually.
I see several ways:
1. Let's create autotest and autotest-client packages. Autotest-client will
contain only stuff really required for clients, i.e. autotest and autoqa
library. Autotest package will contain the rest. Easy to deploy, but people
that want to experiment with it must install all (and disable cron jobs).
2. Your idea, autotest and autotest-watchers packages. Easy to deploy, people
still can't just run a test without the full installation. But this time they
can at least have a look at the tests' source code.
3. Creating autotest and autotest-server packages. Autotest-server will
contain just the crontab file. Silly? :)
4. What about commenting out the crontab file by default? If someone really
wants to deploy a server, he will just uncomment the file (and maybe tweak
the settings anyway). That file is left intact on upgrade, so that shouldn't
be a problem, it's a one-time task. And we will document that on the wiki
(the same way we have now documented that everyone not wanting to run a server
should comment out that file).
I would like to provide people easy way to try and play with autoqa. Although
the last solution seems to be extremely simple, it seems to work well enough.
Would that help you too? Or do you prefer another way? What do you think?
13 years, 7 months
