TRAC tickets bcc'd to the mailing list?
by James Laska
Hey folks,
Would people be interested in having trac ticket updates automatically
bcc'd to the mailing list? I thought it might increase the visibility
the milestones people are working on. But wasn't sure if people would
consider it SPAM.
Thanks,
James
13 years, 4 months
logging (was: rpmguard-integration branch available)
by Kamil Paral
----- "James Laska" <jlaska(a)redhat.com> wrote:
>
> I played around with a quick'n'dirty patch that uses the logging
> module
> (which autotest extends for us).
>
> http://pastie.org/747317
>
> It's definitely not perfect or something I would consider committing.
> I
> just was curious how the logging module would work with our test
> wrappers. The nice thing is autotest splits the output based on
> loglevel for us ...
>
> # ls /usr/share/autotest/client/results/post-koji-build
> \:rpmguard.x86_64/rpmguard/debug/
> rpmguard.DEBUG rpmguard.ERROR rpmguard.INFO
> rpmguard.WARNING
>
> # cat /usr/share/autotest/client/results/post-koji-build
> \:rpmguard.x86_64/rpmguard/debug/rpmguard.ERROR
> rpmlint.noarch |
> 348
> kB 00:00 ... ] 0.0 B/s | 53 kB --:-- ETA
> 12/17 11:15:16 ERROR| rpmguard:0124| added REQUIRES
> libportaudio.so.2
> removed REQUIRES libpulse.so.0
> removed REQUIRES libpulse.so.0(PULSE_0)
> 12/17 11:15:24 ERROR| rpmguard:0124| added REQUIRES
> libportaudio.so.2()(64bit)
> removed REQUIRES libpulse.so.0()(64bit)
> removed REQUIRES libpulse.so.0(PULSE_0)(64bit)
> 12/17 11:15:27 ERROR| rpmguard:0124| added REQUIRES
> libportaudio.so.2()(64bit)
> removed REQUIRES libpulse.so.0()(64bit)
> removed REQUIRES libpulse.so.0(PULSE_0)(64bit)
> 12/17 11:15:34 ERROR| rpmguard:0124| added REQUIRES
> libportaudio.so.2
> removed REQUIRES libpulse.so.0
> removed REQUIRES libpulse.so.0(PULSE_0)
>
> The format of the logs is something autotest has defined, and we can
> override if desired. Also, in this example I was assuming that any
> output from rpmguard was considered a failure. So that output was
> sent
> using logging.error(...).
>
> The logging module is pretty flexible. It's easy to add additional
> logging destinations (files, stdout etc...), each with their own
> output
> formats. So we could continue creating the single rpmguard.log
> results
> file if desired. In fact, we could possible have this done for every
> test by providing an AutoQA specific output handler in addition to
> what
> autotest has defined.
>
> Thoughts/comments/concerns?
I would like to use it, even though I still see value in having self.results
separated from general logging facility (but it all depends how we want our
output to look like). I will play with it after Christmas.
13 years, 5 months
Re: rpmguard-integration branch available
by Kamil Paral
----- "James Laska" <jlaska(a)redhat.com> wrote:
> Nice work Kamil, this looks great. I don't mind the output format to
> be
> honest, but I think the discussion you and Will had during the recent
> QA
> meeting [1] about collapsing results from different architectures
> together makes sense. We probably want to err on the side of
> displaying
> less information for developer review than scaring them away with
> tons
> of repeated information. Besides, we can scare them later by just
> providing the results against their builds :)
Already as a ticket:
https://fedorahosted.org/autoqa/ticket/94
I want to have it implemented this week. Good idea, Will.
>
> I don't think this is specific to the rpmguard test, but it's a
> little
> confusing for me (not knowing the internals) to see two different
> scripts 'rpmguard' and 'rpmguard.py' in the test directory. Looking
> further, I believe rpmguard.py provides the class that autotest calls
> from the provided control file and is intended to be imported only.
> Is
> that correct?
Actually, I think it's rather specific to rpmguard test. In other tests
you usually use some binary from fedora repos located in system paths
(you just do 'yum install foo' at setup), so we can afford to have test
named same as the binary (e.g. rpmlint binary and rpmlint test).
For rpmguard it got a little complicated, because I already develop
the rpmguard in the same directory (and we don't have it packaged
or something). And because the wrapper must be called <test>.py, I
renamed rpmguard.py to rpmguard and created the wrapper with the prior
name.
There are a few solutions:
1. change test naming conventions
2. rename the rpmguard test
3. move rpmguard upstream to another folder
4. move rpmguard upstream to its own project (overkill?)
Third option is probably the easiest. What you think? Just tell me
where would you prefer to put it so we won't make mess of the project
structure.
>
> Some other thoughts ...
>
> * I like how you're using the setup() method and checking for
> specific versions of required software. I imagine this might
> become a common thing, I wonder if in the future we could
> offer
> a common require_version() method in the autoqa base package
> * In the initialize() method ... should we look into using
> mktemp
> (or similar) here instead of os.makedirs? Could multiple
> runs
> of the test pick-up stale data?
I have to admit cheating, I just copied that stuff from Will's rpmlint
test :) I haven't studied if we have some conventions where to save data
etc. So maybe Will could better answer that. Surely I don't need the
data to be persistent in this case so mktemp works well for me. Multiple
runs should work ok in current implementation, because the urlgrabber
seems to be overwriting existing files.
> * In run_once(), I like that you are displaying errors on
> stdout
> as well as in the log. I wonder if we could rely on the
> python
> logging module (or a common autoqa logging subclass to
> provide
> proper format and loglevels) to send results to multiple
> places?
> The trick I think will be writing this such that it can still
> run in stand-alone mode. Looks like Lucas and Michael are
> doing
> similar things with integrating the KVM tests into upstream
> autotest [2]. Is this something we could make use of as
> well?
>
> [1]
> https://fedoraproject.org/wiki/QA/Meetings/20091214#rpmguard_integration
> [2] http://patchwork.kernel.org/patch/40190/
Some kind of logging would be nice, I was also thinking about this
when I printed the messages both to stdout and results. Now when
I want to reformat the rpmguard test output to eliminate duplicities
I am not sure if I would use it (the results output will be probably
different from stdout), but for most tests this could simplify
things.
I hope our one and only autoqa guru wwoods can tell us more about this
topic :)
>
> _______________________________________________
> autoqa-devel mailing list
> autoqa-devel(a)lists.fedorahosted.org
> https://fedorahosted.org/mailman/listinfo/autoqa-devel
13 years, 5 months
rpmguard-integration branch available
by Kamil Paral
Hello,
I'm happy to announce that in the git there is a
kparal/rpmguard-integration branch available containing my latest
effort to bring rpmguard to autoqa. I would be glad for any feedback
provided. If you want to see it in action easily, you can run e.g.
this command:
$ autoqa post-koji-build --name ctdb --kojitag dist-f12-updates-candidate --arch x86_64 --arch i686 ctdb-1.0.108-1.fc12 -t rpmguard
(append --local if you have only autotest-client installed)
That should be working until ctdb gets the update-candidate tag off.
Example output is here:
http://pastebin.com/m73c45d08
I have some concerns about current implementation, what should be
improved and when the test could fail, but I'll cover that in some
further emails.
What about the output format, Will? Should something be changed?
Thanks,
Kamil
13 years, 5 months
Re: [PATCH] Show rpmlint in test listing
by Kamil Paral
----- "Kamil Páral" <kparal(a)redhat.com> wrote:
> I would like to demostrate autoqa possibilities on rpmlint test case,
> but
> currently due to the hacks in koji watcher the rpmlint test in not
> shown
> in test listing when running in the local mode. This patch fixes that
> (let's
> determine architectures before getting test listing from hook).
>
Since there were no objections against this patch, I will commit it. When we have some means to determine arch-independent tests (as Will mentioned in 'architecture for tests' email) these hacks should not be necessary.
13 years, 5 months
Re: architecture for tests
by Will Woods
----- "Kamil Paral" <kparal(a)redhat.com> wrote:
>
> The --arch tag specifies not which packages to test but on which
> machines to run. My fault.
Right, this is pretty confusing, but the --arch argument to autoqa tells autoqa which machines to *schedule tests on*.
> > I have looked at rpmlint test and it simply tests
> > all architectures of a NVR. But is that correct?
> > Because if one architecture is built sooner then the other,
>
> I was told that 'updates-candidate' tag is applies only
> after all architectures were built.
Correct. So the post-koji-build hook operates on a *complete build*, which may include packages of many arches. It launches autoqa and tells it to schedule jobs for each arch included in that build.
This makes sense because a normal functional test (for example: making sure 'ls' works in a new 'coreutils' build) needs to be scheduled and run on an x86_64 machine to check the x86_64 package, and again on i686, and ppc, and so on.
But *some* tests, like rpmlint, can operate on any arch, and check the whole contents of the build. So it would be a waste of time to run the test on more than one arch.
>
> When studying this issue, I don't quite understand current
> wwoods' code:
>
> def process_testlist(opts, args, testlist):
> # XXX TODO we may pull new tests from CVS at this point
> # XXX HACK: only run rpmlint on one single arch, since one test
> will cover
> # the packages for all arches
> if 'x86_64' not in opts.arch and 'rpmlint' in testlist:
> testlist.remove('rpmlint')
> return testlist
>
> Why would that mean that rpmlint is not run multiple times?
> I think when "--arch x86_64 --arch i686" is specified it will
> run on both architectures.
Hm. That seems like a reasonable deduction - if you have both, we won't throw rpmlint out of testlist, and so rpmlint will get scheduled for both arches. Probably we should get rid of that hack altogether and have the control file for rpmlint abort/skip the test if arch is not 'x86_64' - or, in the future, 'noarch' - see below.
> When only "--arch i686" is specified
> rpmlint will not run at all (which is bad).
Hm. You might be right about that.
For noarch packages, watch-koji-build.py substitutes 'x86_64' for 'noarch', so we ensure the test gets run. (Yes, this is another hack, but we didn't have a way to implement 'autoqa --arch noarch' yet. We've got a plan for that now.)
But I think you're right - for a package that *only* had i686, it might not run at all. We could extend the hack in watch-koji-build.py to also substitute 'noarch' for 'src'; this would ensure we always schedule the test for at least one arch.
So as mentioned above, in the future individual tests that are arch-independent (like rpmlint/rpmguard) could do something like exiting the control file early if arch is not 'noarch'. That would ensure that the test only runs once, and then we can remove the hack from process_testlist in post-koji-build/hook.py.
-w
13 years, 5 months
[PATCH] Show rpmlint in test listing
by Kamil Paral
I would like to demostrate autoqa possibilities on rpmlint test case, but
currently due to the hacks in koji watcher the rpmlint test in not shown
in test listing when running in the local mode. This patch fixes that (let's
determine architectures before getting test listing from hook).
I have reverted variable archlist back to opts.arch, because opts is
also transfered to other methods.
13 years, 5 months
Re: architecture for tests
by Kamil Paral
----- "Kamil Paral" <kparal(a)redhat.com> wrote:
> I have a question,
I will reply to myself, I got it all wrong.
> how does a test (rpmguard in my case)
> know, which architecture to test? This is a method header:
>
> def run_once(self, nvr, name, kojitag):
>
> I receive the package NVR, but architecture is not included
> in that. On the other hand, the architecture is communicated
> from the watcher to the harness:
>
> autoqa post-koji-build --name vim --kojitag dist-f10-updates-candidate
> --arch x86_64 vim-7.2.315-1.fc10
>
The --arch tag specifies not which packages to test but on which
machines to run. My fault.
> I have looked at rpmlint test and it simply tests
> all architectures of a NVR. But is that correct?
> Because if one architecture is built sooner then the other,
I was told that 'updates-candidate' tag is applies only
after all architectures were built. Therefore my question
is pointless.
> say x86_64 sooner then i686, then our tests may pick it up
> in that moment and test only x86_64 arch. After a while when
> we check again:
> 1) will we test the i686 arch, or will we consider this NVR
> as "done"?
> 2) if we pick up the NVR again, won't we test now both
> architectures (and therefore report x86_64 errors twice)?
>
> Thanks for explanation.
When studying this issue, I don't quite understand current
wwoods' code:
def process_testlist(opts, args, testlist):
# XXX TODO we may pull new tests from CVS at this point
# XXX HACK: only run rpmlint on one single arch, since one test will cover
# the packages for all arches
if 'x86_64' not in opts.arch and 'rpmlint' in testlist:
testlist.remove('rpmlint')
return testlist
Why would that mean that rpmlint is not run multiple times?
I think when "--arch x86_64 --arch i686" is specified it will
run on both architectures. When only "--arch i686" is specified
rpmlint will not run at all (which is bad). I would need the
same behaviour as rpmlint (I can also check packages on
arbitrary architecture), but I don't think the hack is doing
the right thing.
13 years, 6 months
