[AutoQA] #150: "artificial ignorance": suppress errors and warnings that aren't important
by fedora-badges
#150: "artificial ignorance": suppress errors and warnings that aren't important
-------------------------+--------------------------------------------------
Reporter: dmalcolm | Owner:
Type: enhancement | Status: new
Priority: major | Milestone: Package Sanity Tests
Component: tests | Version: 1.0
Keywords: |
-------------------------+--------------------------------------------------
There are some interesting ideas in here about suppressing a logfile to
show up the information that's truly important:
http://www.ranum.com/security/computer_security/papers/ai/index.html
I think that the rpmlint test needs to have a set of filters, and should
suppress output matching certain patterns. For example
https://fedorahosted.org/pipermail/autoqa-results/2010-April/013903.html
shows:
{{{
gnome-python2-gnome.i686: W: spelling-error Summary(en_US) libgnome -> lib
gnome, lib-gnome, cognomen
}}}
i.e. it's complaining about a reference to "libgnome" within gnome-
python2-gnome, which is clearly a false-positive.
For this case we might have a python filtering process that says something
like this (caveat: untested):
{{{
patterns = [
": W: spelling-error \S+ libgnome .*",
]
def should_ignore(line):
for pat in patterns
# May want to precompile the patterns:
m = re.match(pat, line)
if m:
return True
# No match:
return False
for line in rpmlint_output
if not should_ignore(line):
print(line)
}}}
and then gradually grow patterns. The idea is to make it nice and
lightweight, and Fedora QA can then easily take ownership of the
suppresssions, without having to get patches into rpmlint.
--
Ticket URL: <https://fedorahosted.org/autoqa/ticket/150>
AutoQA <http://autoqa.fedorahosted.org>
Automated QA project
13 years, 5 months
Re: Any further discussion on depcheck & the Glorious Future?
by Kamil Paral
----- "Will Woods" <wwoods(a)redhat.com> wrote:
> Hey all,
>
> I've finished the series of blog posts about depcheck and the
> Glorious
> Future of Fedora QA (as I see it):
>
> [1/3] http://qa-rockstar.livejournal.com/10187.html
> [2/3] http://qa-rockstar.livejournal.com/10368.html
> [3/3] http://qa-rockstar.livejournal.com/10507.html
> [bonus details]: http://wwoods.fedorapeople.org/depcheck.html
Thanks Will for those blogposts, really great. I have some more technical
questions:
1. Do we need hosts with different architectures for running depcheck?
Let's imagine we have these packages proposed as updates:
foo-1.2-3.fc13
|-----foo-1.2-3.i686.rpm
|-----foo-1.2-3.x86_64.rpm
|-----foo-data-1.2-3.noarch.rpm
bar-4.5-6.fc13
|-----bar-4.5-6.fc13.noarch.rpm
How do I test those packages? Should I get two machines and test i686
rpm files on i686 hardware, x86_64 rpm files on x86_64 hardware, and
noarch rpm files on any hardware? Or should I test the whole set of
RPMs twice, once on each hardware arch?
2. I have imagined a situation where two updates are accepted into
the -pending tag in quick succession. Thus we can end up with two
depcheck tests running simultaneously. And the one that was started
earlier can finish up later. It could then report incorrect result.
Do you have an idea what we can we do about that? Maybe we can
somehow write down the state of the -pending tag in the time of
starting the test, and if it is different from when the test is
finished, we can just throw away the results? Just a wild idea.
13 years, 5 months
Re: Update to autoqa-0.4.3-1
by Kamil Paral
----- "James Laska" <jlaska(a)redhat.com> wrote:
> Greetings gang,
>
> Per previous discussion on the list
> (https://fedorahosted.org/pipermail/autoqa-devel/2010-November/001286.html),
> and of course using the roadmap Kamil posted earlier
> (https://fedoraproject.org/wiki/AutoQA_Roadmap), I updated to
> autoqa-0.4.3-1.
> In reply to this mail, I sent three patches that I included when
> tagging
> autoqa-0.4.3-1. I've applied these to master already. Hopefully that
> wasn't
> in poor form, we can easily adjust if there are any concerns regarding
> the
> patches. Two of the patches should be familiar (autoqa.conf and
> repoinfo.conf). The third patch resolves ticket#239.
Thanks, James, patches look completely fine. My quick smoke test succeeded.
Let's deploy! :)
13 years, 5 months
Update to autoqa-0.4.3-1
by James Laska
Greetings gang,
Per previous discussion on the list
(https://fedorahosted.org/pipermail/autoqa-devel/2010-November/001286.html),
and of course using the roadmap Kamil posted earlier
(https://fedoraproject.org/wiki/AutoQA_Roadmap), I updated to autoqa-0.4.3-1.
In reply to this mail, I sent three patches that I included when tagging
autoqa-0.4.3-1. I've applied these to master already. Hopefully that wasn't
in poor form, we can easily adjust if there are any concerns regarding the
patches. Two of the patches should be familiar (autoqa.conf and
repoinfo.conf). The third patch resolves ticket#239.
More details on the changes since autoqa-0.4.2-1 are listed below.
cb8c8fd 2010-11-15 Bump to autoqa-0.4.3-1 (jlaska(a)redhat.com)
ff03bab 2010-11-15 Ticket#239 - install hunspell-en during rpmlint setup() (jlaska(a)redhat.com)
f3cb33f 2010-11-15 Remove ppc from arches. f14 is now officially released, update path appropriately. Remove unsupported Fedora releases (jlaska(a)redhat.com)
399d703 2010-11-15 only library modules with main loop should be executable, others should not (kparal(a)redhat.com)
4e6bdc6 2010-11-10 remove unused imports, call parent initialize() method when overriding it (kparal(a)redhat.com)
1462e47 2010-11-09 upgradepath: improve error message clarity (kparal(a)redhat.com)
6670ee2 2010-10-14 allow tests to distinguish CRASHED and ABORTED (kparal(a)redhat.com)
a5ce337 2010-10-13 Reviewed. (mkrizek(a)redhat.com)
821f162 2010-10-11 Added reviewed initscript tests. (jskladan(a)redhat.com)
e0f77a1 2010-10-06 Include traceback in email output (kparal(a)redhat.com)
82baf5f 2010-10-06 Removed individual XYZ_failed methods in test.py (jskladan(a)redhat.com)
78cacd5 2010-10-05 guard setup() with ExceptionCatcher (kparal(a)redhat.com)
8e4e894 2010-10-05 allow self.outputs and self.highlights to be lists of strings (kparal(a)redhat.com)
03ca1e8 2010-10-04 Automatically log test output (kparal(a)redhat.com)
c6e5d3a 2010-10-01 Correct missing config_loader import (jlaska(a)redhat.com)
a2bd1b1 2010-10-01 Tell post-tree-compose that composes are enabled for F14 (jlaska(a)redhat.com)
885e0e1 2010-09-30 tidy up upgradepath test code (kparal(a)redhat.com)
993e394 2010-09-29 upgradepath doesn't support -testing repos yet, disable scheduling (kparal(a)redhat.com)
Thanks,
James
13 years, 5 months
AutoQA Roadmap
by Kamil Paral
Hello all,
I have created https://fedoraproject.org/wiki/AutoQA_Roadmap .
Since our manpower decreased (best wishes to wwoods) we really need to
get more focused and selective in our future goals. I would like us to
state some near-future goals that are doable, that have visible benefits
and then concentrate the team on achieving that.
Sometimes the tasks can be divided easily among several people,
sometimes they may not. But as much as possible we should avoid working
on several projects each, as was often the case now. Let's finish up
one task and then go to another, what do you say?
In the roadmap, I'm partially duplicating our Trac Roadmap:
https://fedorahosted.org/autoqa/roadmap
Still, it seemed to me that on wiki page one could easily see the most
important features we want to have in a release. Better than in Trac,
where it is split up more in a functional units rather than time
schedule units.
I believe that the best part to concentrate upon right now are depcheck
and upgradepath tests. They bring a huge benefit to the Fedora project,
we just need to finish them and hook them up. Therefore I propose Bodhi
comments support in depcheck and upgradepath as the nearest important
feature. Rewriting bodhi watcher is related to that. I would also like
to see AutoQA fully prepared for a staging environment (it almost is
now) - we will get hardware in the meantime. That would be 0.4.4 release.
After we have this work done, I think it's about time to finally
set up ResultDB (we can have some preview in 0.4.4). Having it production
ready would be a next major step, 0.5.0 release.
Please look at the wiki page and tell us what you think.
Thanks,
Kamil
13 years, 5 months
Re: autotest_lib can't be used outside autotest process
by Kamil Paral
----- "James Laska" <jlaska(a)redhat.com> wrote:
> On Tue, 2010-11-09 at 12:59 -0500, Kamil Paral wrote:
> > Martin played a little today with AutoQA and discovered that he
> can't
> > run our watchers if he uses autotest_lib module inside them.
> >
> > The reason wasn't clear for me immediately, so I look at it a
> little
> > closer.
> >
> > The reason is that autotest libraries are not in a standard Python
> PATH.
> > The only way to gain access to autotest_lib module is to import
> > /usr/share/autotest/client/bin/common.py. And that happens only
> when
> > you run autotest tool, like this:
> >
> > # autoqa post-koji-build -k dist-f14 foo -t helloworld --local
> > -- or (subsequently) like this --
> > # ~autotest/client/bin/autotest /tmp/autoqa-control.9xlSGi
>
> Definitely something specific to the peculiar way that autotest is
> packaged. I can see why, it makes sense for their project to be
> portable to any directory. Just import the common module, and it
> will
> do the dirty work of finding the remaining libraries and setting up
> sys.path. Since most of our work is specific to a single distro, we
> obviously like things to be in their standard locations.
You're right, that works. I can do something like:
# PYTHONPATH=/usr/share/autotest/ python
>>> import client.bin.common
>>> import autotest_lib
>>>
So if we have the right PYTHONPATH (which we can solve by installing
a .pth file into /usr/lib/python/site-packages) and if we keep the right
order of imports (common.py first), we can use autotest libraries also
outside the autotest process itself.
>
> > For those scripts that we want to run outside autotest environment
> > (like watchers, autoqa script itself, stand-alone parts of our
> tests)
> > we can't use autotest_lib module, we have to rely just on our
> autoqa
> > library (and more specifically on those parts of our autoqa library
> > that don't rely on autotest_lib themselves).
> >
> > So I just want to point out that if you see this kind of problem,
> > this is the reason.
> >
> > The question remains - is this desirable or not desirable? Would we
>
> > like to see autotest modules be accessible as a standard Python
> module?
> > Do we need it?
>
> I may not fully understand the problem, but I don't think we should
> need
> to integrate watchers with autotest. Was there a specific need to
> integrate the watcher with autotest?
I don't think there was a strong need for it and I don't think it would
be wise. We just found it by coincidence.
>
> Additionally, I get nervous when we integrate out test scripts with
> autotest. Obviously, we need to integrate with the current test
> harness
> to some degree. However, I want to make sure we understand the links
> and they are documented well in the event we need to swap out for a
> different back-end harness/scheduler.
>
> This does help clarify though which files can link to autotest and
> which
> cannot. Obviously, the control file and the test object file can
> link
> against autotest. But any test script should be stand-alone. For
> example, tests/conflicts/potential_conflict.py and
> tests/rpmguard/rpmguard. They were designed to be run stand-alone on
> a
> system and shouldn't link with autotest at all. Am I correct?
Yes, I have the same opinion. If it is doable and reasonable we really
should have a standalone test script that doesn't rely on autotest.
In some cases, however, we have put the logic directly into the test
object (e.g. upgradepath test), because it seems like a best approach
in those scenarios (although it has some disadvantages).
>
> > I don't think we need it right now, just some food for thought.
>
> Definitely, thanks for pointing this out!
>
> While researching packaging autotest, the packager will have to
> answer
> this question of why aren't some files in standard system locations.
> I've been assuming that answering that question is a distinct effort
> from anything going on with our autoqa work. Did I get that wrong?
I am not sure. I think the current state (non-standard location)
is sufficient for us, at least for now.
13 years, 5 months
autotest_lib can't be used outside autotest process
by Kamil Paral
Martin played a little today with AutoQA and discovered that he can't
run our watchers if he uses autotest_lib module inside them.
The reason wasn't clear for me immediately, so I look at it a little
closer.
The reason is that autotest libraries are not in a standard Python PATH.
The only way to gain access to autotest_lib module is to import
/usr/share/autotest/client/bin/common.py. And that happens only when
you run autotest tool, like this:
# autoqa post-koji-build -k dist-f14 foo -t helloworld --local
-- or (subsequently) like this --
# ~autotest/client/bin/autotest /tmp/autoqa-control.9xlSGi
For those scripts that we want to run outside autotest environment
(like watchers, autoqa script itself, stand-alone parts of our tests)
we can't use autotest_lib module, we have to rely just on our autoqa
library (and more specifically on those parts of our autoqa library
that don't rely on autotest_lib themselves).
So I just want to point out that if you see this kind of problem,
this is the reason.
The question remains - is this desirable or not desirable? Would we
like to see autotest modules be accessible as a standard Python module?
Do we need it?
I don't think we need it right now, just some food for thought.
Cheers,
Kamil
13 years, 5 months
Re: upgradepath: FAIL; 0 OK, 1 FAILED for GMT-4.5.5-2.fc13
by Kamil Paral
----- "James Laska" <jlaska(a)redhat.com> wrote:
> On Tue, 2010-11-09 at 07:44 -0500, Kamil Paral wrote:
> > ----- "James Laska" <jlaska(a)redhat.com> wrote:
> > >
> > > Aaah, I understand the difference and why I was confused by the
> > > result.
> > > Thanks for explaining.
> >
> > What do you think about this one?
>
> Definitely helps me understand the output better. SOme comments
> below.
>
> > ===============================================
> > >From f9bcda33b036e7df25a50d2b1a4a72c6ef1a0619 Mon Sep 17 00:00:00
> 2001
> > From: =?UTF-8?q?Kamil=20P=C3=A1ral?= <kparal(a)redhat.com>
> > Date: Tue, 9 Nov 2010 13:39:41 +0100
> > Subject: [PATCH] upgradepath: improve error message clarity
> >
> > Use human sentence in the error message instead of a symbolic
> notation
> > ---
> > tests/upgradepath/upgradepath.py | 18 ++++++++++++++----
> > 1 files changed, 14 insertions(+), 4 deletions(-)
> >
> > diff --git a/tests/upgradepath/upgradepath.py
> b/tests/upgradepath/upgradepath.py
> > index edbe4ea..d42e39d 100755
> > --- a/tests/upgradepath/upgradepath.py
> > +++ b/tests/upgradepath/upgradepath.py
> > @@ -50,9 +50,19 @@ class upgradepath(AutoQATest):
> > self.outputs = []
> > self.highlights = []
> >
> > - def compare(self, matching_build, tags, op, opstr):
> > + def compare(self, matching_build, tags, op):
> > koji = autoqa.koji_utils.SimpleKojiClientSession()
> > result = 'PASSED'
> > +
> > + # convert 'op' to human language for error message clarity
> > + opstr = {operator.eq: 'equal to',
> > + operator.ge: 'greater or equal than',
>
> Maybe rephrase the last part as "... equal to'
>
> > + operator.gt: 'greater than',
> > + operator.le: 'lesser or equal than',
>
> Maybe rephrase as "less than or equal to'
>
> > + operator.lt: 'lesser than',
>
> Same, maybe rephrase the last part as "... equal to'
Thanks for your correction.
>
> > + operator.ne: 'not equal to'}
> > + assert op in opstr, 'Used unsupported operator: %s' % op
> > +
>
> Heh, that's funny. I was looking to see if the operator module
> itself
> had some text for each of those functions, so I like your opstr
> dictionary above.
Unfortunately it doesn't.
>
> > for tag in tags:
> > latest_build = koji.latest_by_tag(tag,
> matching_build['name'])
> > if latest_build is None:
> > @@ -80,7 +90,7 @@ class upgradepath(AutoQATest):
> > print msg
> > self.outputs.append(msg)
> > if not ok:
> > - msg = '\tFailed condition: Requested package %s
> Latest package' % opstr
> > + msg = '\tError: Requested package must be %s
> the latest package' % opstr[op]
>
> Future-proofing ... are there any operators we aren't adding to the
> opstr dictionary that might result in a traceback at some future
> date?
> Should we convert the access to opstr.get(op, "UNKNOWN")?
I would rather see it crashed than silently doing anything.
I don't suppose we will need more operators, and if we do, we can amend
the dictionary.
>
> Or should we stuff an assert in there somewhere (assert
> optsr.has_key(op))?
Oh yes, the assert is right after the dictionary declaration.
>
> > print msg
> > self.outputs.append(msg)
> > result = 'FAILED'
> > @@ -138,9 +148,9 @@ class upgradepath(AutoQATest):
> > self.highlights.append(msg)
> >
> > # compare with lower or equal tags, so version has to
> be greater or equal
> > - result = self.compare(matching_build, low_eq_tags,
> operator.ge, '>=')
> > + result = self.compare(matching_build, low_eq_tags,
> operator.ge)
> > # compare with higher tags, so version has to be lower
> or equal
> > - result2 = self.compare(matching_build, hi_tags,
> operator.le, '<=')
> > + result2 = self.compare(matching_build, hi_tags,
> operator.le)
> >
> > # compute pkg result
> > if self.result_order.index(result2) >
> self.result_order.index(result):
> > --
> > 1.7.3.2
> >
> > ====================================================
> >
> > # autoqa post-bodhi-update --title xxx --target-tag dist-f13-updates
> GMT-4.5.5-2.fc13 -t upgradepath --local
> > 13:37:27 INFO | ========================================
> > 13:37:27 INFO | GMT-4.5.5-2.fc13 into dist-f13-updates
> > 13:37:27 INFO | ========================================
> > 13:37:29 INFO | [ OK ] dist-f10
> > 13:37:29 INFO | Latest package: GMT-4.3.1-2.fc10
> > 13:37:29 INFO | [ OK ] dist-f10-updates
> > 13:37:29 INFO | Latest package: GMT-4.4.0-1.fc10
> > 13:37:30 INFO | [ OK ] dist-f11
> > 13:37:30 INFO | Latest package: GMT-4.4.0-2.fc11
> > 13:37:30 INFO | [ OK ] dist-f11-updates
> > 13:37:30 INFO | Latest package: GMT-4.5.0-4.fc11
> > 13:37:31 INFO | [ OK ] dist-f12
> > 13:37:31 INFO | Latest package: GMT-4.5.1-1.fc12
> > 13:37:32 INFO | [ OK ] dist-f12-updates
> > 13:37:32 INFO | Latest package: GMT-4.5.3-3.fc12
> > 13:37:32 INFO | [ OK ] dist-f13
> > 13:37:32 INFO | Latest package: GMT-4.5.2-1.fc13
> > 13:37:33 INFO | [ OK ] dist-f13-updates
> > 13:37:33 INFO | Latest package: GMT-4.5.3-3.fc13
> > 13:37:34 INFO | [FAIL] dist-f14
> > 13:37:34 INFO | Latest package: GMT-4.5.3-3.fc14
> > 13:37:34 INFO | Error: Requested package must be lesser or equal
> than the latest package
> > 13:37:37 INFO | [ OK ] dist-f14-updates
> > 13:37:37 INFO | No such package: GMT
> > 13:37:37 INFO | [ OK ] dist-f15
> > 13:37:37 INFO | Latest package: GMT-4.5.5-2.fc15
> > 13:37:37 INFO | RESULT: FAILED
> > 13:37:37 INFO | SUMMARY: 1 FAILED
> > 13:37:37 INFO | ****************************************
>
> Looks good to me ... definitely addresses my confusion.
I'll commit to master soon. Thanks for feedback.
13 years, 5 months
