Re: post-bodhi-update early test results
On Tue, 2010-06-15 at 15:59 -0400, James Laska wrote:
Greetings,
Since I have an extra virt autotest-server setup, I decided to run
through a few post-bodhi-update tests using the autoqa-0.3.5-pre1
package [1].
A few observations, some are probably known/expected, but I'll list them
here for others.
1. I needed to create /usr/share/autoqa/post-bodhi-update/testlist
for /usr/bin/autoqa to recognize the watcher. I just added
rpmlint to test (see step#4 for results)
I was thinking - maybe we should have a really stupid "helloworld"-style
test that just basically logs its inputs and reports success. This would
allow us to test new hooks without needing to find an appropriate test.
2. See attached output from running watch-bodhi-requests.py for
the
first time. Nothing horrible here. Perhaps reducing the error
output produced by calling load_update_ids() when no cachefiles
are present?
Yeah, I intended that we would remove/quiet that code once we were
satisfied that the watcher wasn't doing anything stupid.
3. Attempting to schedule an autotest job using
watch-bodhi-requests.py fails due to incorrect command-line
arguments (see below). The attached patch resolves that
problem.
Ah doh! I forgot to change that. Patch looks fine; please apply.
4. Re-running the '/usr/bin/autoqa' using the hook patch
from
step#3 schedules an autotest job using the attached control
file. Obviously, the job.run_test() call for rpmlint, doesn't
use the same arguments that post-bodhi-update is creating. So
'rpmlint' might not be a good test. I'll try using the
'depcheck' test next.
So maybe let's write that "helloworld" test, and try that out instead of
trying to get depcheck working first?
All it would need to do is something vaguely like:
def run_once(self, *args, **kwargs):
msg = "args = %s\nkwargs = %s" % (args, kwargs)
self.results = msg
print msg
# maybe email the message here just to be sure email is working
# etc.
...except writing the control file might be a bit tricky since we don't
necessarily know the key names. hrm.
So Maybe autoqa ought to be writing all the hook arguments into a dict
instead of individual args, so we can pass along all the autoqa
arguments without knowing their names.
This is a valuable exercise, though - we're starting to have tests that
could apply to multiple hooks (rpmlint for example) and we're going to
need ways to handle that.
For example we might also need to standardize on argument names - so
(e.g.) --target-tag will always mean "this is the tag that this package
is trying to enter" regardless of whether we're doing post-koji-build or
post-bodhi-update or whatever.
In the meantime, though, we can just write this 'echo_inputs' test
specifically for post-bodhi-update, and worry about how to make it work
for multiple hooks later.
So. Anyone want to take a crack at writing this test?
-w