autoqa-devel December 2009

autoqa-devel@lists.fedorahosted.org

6 participants
15 discussions

TRAC tickets bcc'd to the mailing list?
by James Laska 05 Jan '10

05 Jan '10

Hey folks, Would people be interested in having trac ticket updates automatically bcc'd to the mailing list? I thought it might increase the visibility the milestones people are working on. But wasn't sure if people would consider it SPAM. Thanks, James

2 2

wiki - Verifying AutoQA tests
by Kamil Paral 04 Jan '10

04 Jan '10

Hello, I have tried to create another piece of documentation, that could help new test developers testing their new tests :) https://fedoraproject.org/wiki/Verifying_AutoQA_tests Feel free to improve. Kamil

3 4

logging (was: rpmguard-integration branch available)
by Kamil Paral 18 Dec '09

18 Dec '09

----- "James Laska" <jlaska(a)redhat.com> wrote: > > I played around with a quick'n'dirty patch that uses the logging > module > (which autotest extends for us). > > http://pastie.org/747317 > > It's definitely not perfect or something I would consider committing. > I > just was curious how the logging module would work with our test > wrappers. The nice thing is autotest splits the output based on > loglevel for us ... > > # ls /usr/share/autotest/client/results/post-koji-build > \:rpmguard.x86_64/rpmguard/debug/ > rpmguard.DEBUG rpmguard.ERROR rpmguard.INFO > rpmguard.WARNING > > # cat /usr/share/autotest/client/results/post-koji-build > \:rpmguard.x86_64/rpmguard/debug/rpmguard.ERROR > rpmlint.noarch | > 348 > kB 00:00 ... ] 0.0 B/s | 53 kB --:-- ETA > 12/17 11:15:16 ERROR| rpmguard:0124| added REQUIRES > libportaudio.so.2 > removed REQUIRES libpulse.so.0 > removed REQUIRES libpulse.so.0(PULSE_0) > 12/17 11:15:24 ERROR| rpmguard:0124| added REQUIRES > libportaudio.so.2()(64bit) > removed REQUIRES libpulse.so.0()(64bit) > removed REQUIRES libpulse.so.0(PULSE_0)(64bit) > 12/17 11:15:27 ERROR| rpmguard:0124| added REQUIRES > libportaudio.so.2()(64bit) > removed REQUIRES libpulse.so.0()(64bit) > removed REQUIRES libpulse.so.0(PULSE_0)(64bit) > 12/17 11:15:34 ERROR| rpmguard:0124| added REQUIRES > libportaudio.so.2 > removed REQUIRES libpulse.so.0 > removed REQUIRES libpulse.so.0(PULSE_0) > > The format of the logs is something autotest has defined, and we can > override if desired. Also, in this example I was assuming that any > output from rpmguard was considered a failure. So that output was > sent > using logging.error(...). > > The logging module is pretty flexible. It's easy to add additional > logging destinations (files, stdout etc...), each with their own > output > formats. So we could continue creating the single rpmguard.log > results > file if desired. In fact, we could possible have this done for every > test by providing an AutoQA specific output handler in addition to > what > autotest has defined. > > Thoughts/comments/concerns? I would like to use it, even though I still see value in having self.results separated from general logging facility (but it all depends how we want our output to look like). I will play with it after Christmas.

1 0

Re: rpmguard-integration branch available
by Kamil Paral 18 Dec '09

18 Dec '09

----- "James Laska" <jlaska(a)redhat.com> wrote: > Nice work Kamil, this looks great. I don't mind the output format to > be > honest, but I think the discussion you and Will had during the recent > QA > meeting [1] about collapsing results from different architectures > together makes sense. We probably want to err on the side of > displaying > less information for developer review than scaring them away with > tons > of repeated information. Besides, we can scare them later by just > providing the results against their builds :) Already as a ticket: https://fedorahosted.org/autoqa/ticket/94 I want to have it implemented this week. Good idea, Will. > > I don't think this is specific to the rpmguard test, but it's a > little > confusing for me (not knowing the internals) to see two different > scripts 'rpmguard' and 'rpmguard.py' in the test directory. Looking > further, I believe rpmguard.py provides the class that autotest calls > from the provided control file and is intended to be imported only. > Is > that correct? Actually, I think it's rather specific to rpmguard test. In other tests you usually use some binary from fedora repos located in system paths (you just do 'yum install foo' at setup), so we can afford to have test named same as the binary (e.g. rpmlint binary and rpmlint test). For rpmguard it got a little complicated, because I already develop the rpmguard in the same directory (and we don't have it packaged or something). And because the wrapper must be called <test>.py, I renamed rpmguard.py to rpmguard and created the wrapper with the prior name. There are a few solutions: 1. change test naming conventions 2. rename the rpmguard test 3. move rpmguard upstream to another folder 4. move rpmguard upstream to its own project (overkill?) Third option is probably the easiest. What you think? Just tell me where would you prefer to put it so we won't make mess of the project structure. > > Some other thoughts ... > > * I like how you're using the setup() method and checking for > specific versions of required software. I imagine this might > become a common thing, I wonder if in the future we could > offer > a common require_version() method in the autoqa base package > * In the initialize() method ... should we look into using > mktemp > (or similar) here instead of os.makedirs? Could multiple > runs > of the test pick-up stale data? I have to admit cheating, I just copied that stuff from Will's rpmlint test :) I haven't studied if we have some conventions where to save data etc. So maybe Will could better answer that. Surely I don't need the data to be persistent in this case so mktemp works well for me. Multiple runs should work ok in current implementation, because the urlgrabber seems to be overwriting existing files. > * In run_once(), I like that you are displaying errors on > stdout > as well as in the log. I wonder if we could rely on the > python > logging module (or a common autoqa logging subclass to > provide > proper format and loglevels) to send results to multiple > places? > The trick I think will be writing this such that it can still > run in stand-alone mode. Looks like Lucas and Michael are > doing > similar things with integrating the KVM tests into upstream > autotest [2]. Is this something we could make use of as > well? > > [1] > https://fedoraproject.org/wiki/QA/Meetings/20091214#rpmguard_integration > [2] http://patchwork.kernel.org/patch/40190/ Some kind of logging would be nice, I was also thinking about this when I printed the messages both to stdout and results. Now when I want to reformat the rpmguard test output to eliminate duplicities I am not sure if I would use it (the results output will be probably different from stdout), but for most tests this could simplify things. I hope our one and only autoqa guru wwoods can tell us more about this topic :) > > _______________________________________________ > autoqa-devel mailing list > autoqa-devel(a)lists.fedorahosted.org > https://fedorahosted.org/mailman/listinfo/autoqa-devel

2 2

rpmguard-integration branch available
by Kamil Paral 18 Dec '09

18 Dec '09

Hello, I'm happy to announce that in the git there is a kparal/rpmguard-integration branch available containing my latest effort to bring rpmguard to autoqa. I would be glad for any feedback provided. If you want to see it in action easily, you can run e.g. this command: $ autoqa post-koji-build --name ctdb --kojitag dist-f12-updates-candidate --arch x86_64 --arch i686 ctdb-1.0.108-1.fc12 -t rpmguard (append --local if you have only autotest-client installed) That should be working until ctdb gets the update-candidate tag off. Example output is here: http://pastebin.com/m73c45d08 I have some concerns about current implementation, what should be improved and when the test could fail, but I'll cover that in some further emails. What about the output format, Will? Should something be changed? Thanks, Kamil

3 11

architecture visualization
by Kamil Paral 16 Dec '09

16 Dec '09

Hi, I'm working on AutoQA documentation and I tried to make a diagram visualizing AutoQA architecture. Here it is: https://fedoraproject.org/wiki/AutoQA_architecture If you have any suggestions or corrections, please provide a comment here or adjust the image yourselves (ODG source attached on wiki too). Thanks, Kamil

4 3

Re: [PATCH] Show rpmlint in test listing
by Kamil Paral 14 Dec '09

14 Dec '09

----- "Kamil Páral" <kparal(a)redhat.com> wrote: > I would like to demostrate autoqa possibilities on rpmlint test case, > but > currently due to the hacks in koji watcher the rpmlint test in not > shown > in test listing when running in the local mode. This patch fixes that > (let's > determine architectures before getting test listing from hook). > Since there were no objections against this patch, I will commit it. When we have some means to determine arch-independent tests (as Will mentioned in 'architecture for tests' email) these hacks should not be necessary.

1 0

Re: architecture for tests
by Will Woods 10 Dec '09

10 Dec '09

----- "Kamil Paral" <kparal(a)redhat.com> wrote: > > The --arch tag specifies not which packages to test but on which > machines to run. My fault. Right, this is pretty confusing, but the --arch argument to autoqa tells autoqa which machines to *schedule tests on*. > > I have looked at rpmlint test and it simply tests > > all architectures of a NVR. But is that correct? > > Because if one architecture is built sooner then the other, > > I was told that 'updates-candidate' tag is applies only > after all architectures were built. Correct. So the post-koji-build hook operates on a *complete build*, which may include packages of many arches. It launches autoqa and tells it to schedule jobs for each arch included in that build. This makes sense because a normal functional test (for example: making sure 'ls' works in a new 'coreutils' build) needs to be scheduled and run on an x86_64 machine to check the x86_64 package, and again on i686, and ppc, and so on. But *some* tests, like rpmlint, can operate on any arch, and check the whole contents of the build. So it would be a waste of time to run the test on more than one arch. > > When studying this issue, I don't quite understand current > wwoods' code: > > def process_testlist(opts, args, testlist): > # XXX TODO we may pull new tests from CVS at this point > # XXX HACK: only run rpmlint on one single arch, since one test > will cover > # the packages for all arches > if 'x86_64' not in opts.arch and 'rpmlint' in testlist: > testlist.remove('rpmlint') > return testlist > > Why would that mean that rpmlint is not run multiple times? > I think when "--arch x86_64 --arch i686" is specified it will > run on both architectures. Hm. That seems like a reasonable deduction - if you have both, we won't throw rpmlint out of testlist, and so rpmlint will get scheduled for both arches. Probably we should get rid of that hack altogether and have the control file for rpmlint abort/skip the test if arch is not 'x86_64' - or, in the future, 'noarch' - see below. > When only "--arch i686" is specified > rpmlint will not run at all (which is bad). Hm. You might be right about that. For noarch packages, watch-koji-build.py substitutes 'x86_64' for 'noarch', so we ensure the test gets run. (Yes, this is another hack, but we didn't have a way to implement 'autoqa --arch noarch' yet. We've got a plan for that now.) But I think you're right - for a package that *only* had i686, it might not run at all. We could extend the hack in watch-koji-build.py to also substitute 'noarch' for 'src'; this would ensure we always schedule the test for at least one arch. So as mentioned above, in the future individual tests that are arch-independent (like rpmlint/rpmguard) could do something like exiting the control file early if arch is not 'noarch'. That would ensure that the test only runs once, and then we can remove the hack from process_testlist in post-koji-build/hook.py. -w

1 0

[PATCH] Show rpmlint in test listing
by Kamil Paral 09 Dec '09

09 Dec '09

I would like to demostrate autoqa possibilities on rpmlint test case, but currently due to the hacks in koji watcher the rpmlint test in not shown in test listing when running in the local mode. This patch fixes that (let's determine architectures before getting test listing from hook). I have reverted variable archlist back to opts.arch, because opts is also transfered to other methods.

2 2

Re: architecture for tests
by Kamil Paral 04 Dec '09

04 Dec '09

----- "Kamil Paral" <kparal(a)redhat.com> wrote: > I have a question, I will reply to myself, I got it all wrong. > how does a test (rpmguard in my case) > know, which architecture to test? This is a method header: > > def run_once(self, nvr, name, kojitag): > > I receive the package NVR, but architecture is not included > in that. On the other hand, the architecture is communicated > from the watcher to the harness: > > autoqa post-koji-build --name vim --kojitag dist-f10-updates-candidate > --arch x86_64 vim-7.2.315-1.fc10 > The --arch tag specifies not which packages to test but on which machines to run. My fault. > I have looked at rpmlint test and it simply tests > all architectures of a NVR. But is that correct? > Because if one architecture is built sooner then the other, I was told that 'updates-candidate' tag is applies only after all architectures were built. Therefore my question is pointless. > say x86_64 sooner then i686, then our tests may pick it up > in that moment and test only x86_64 arch. After a while when > we check again: > 1) will we test the i686 arch, or will we consider this NVR > as "done"? > 2) if we pick up the NVR again, won't we test now both > architectures (and therefore report x86_64 errors twice)? > > Thanks for explanation. When studying this issue, I don't quite understand current wwoods' code: def process_testlist(opts, args, testlist): # XXX TODO we may pull new tests from CVS at this point # XXX HACK: only run rpmlint on one single arch, since one test will cover # the packages for all arches if 'x86_64' not in opts.arch and 'rpmlint' in testlist: testlist.remove('rpmlint') return testlist Why would that mean that rpmlint is not run multiple times? I think when "--arch x86_64 --arch i686" is specified it will run on both architectures. When only "--arch i686" is specified rpmlint will not run at all (which is bad). I would need the same behaviour as rpmlint (I can also check packages on arbitrary architecture), but I don't think the hack is doing the right thing.

1 0

2024

2023

2022

2021

2020

2019

2018

2017

2016

2015

2014

2013

2012

2011

2010

2009

autoqa-devel December 2009