9:28 a.m.
On Fri, 2009-11-13 at 03:33 -0500, Kamil Paral wrote:
I have been thinking about ticket #52:
https://fedorahosted.org/autoqa/ticket/52
The main idea behind it is: "I have the test script written,
let me see if it actually works in AutoQA." I don't know if
my patch is the best approach available, but it served my
needs.
Imagine a situation: You have a working test. You have written
a control file and a test wrapper. Now, how do you know it
works? You know your test works, but it is hard to ensure
that your test will get correct command-line parameters,
because you have no way to try it.
Forwarding (and transformation) of parameters works in this way:
watcher -> autoqa -> -> hook -> control file -> wrapper ->
-> your test.
Oof, that's a long way where many mistakes can happen.
The only current way is to install autotest-server, which is
painful. I would like to lower this entrance barrier by avoiding
that.
I have found that many watcher scripts have hidden undocumented
option --dryrun. Then they only print command they would have
run. That's great. Still there is a long way from this command
to actual test execution command.
Also, there are some use cases in which you can't really separate
your test from the watcher. For example the repoclosure test - it
can work perfect when you test it manually, but it depends on some
repository metadata that the _watcher_ downloads. Then you simply
don't know if all the data is downloaded ok, in correct formats,
correct locations, etc. And if you don't want to hack and slash
the watcher source code, then the only method is to run the whole
thing.
Because Will is developing some more libraries for tests, the code
can soon became even more intertwined and there can be even bigger
desire for running the whole thing together.
The last use case is when contributing to autoqa directly and
wanting to see "something" running. Just to have some idea how
it works.
Which takes me to a list of possible improvements:
1. Standardize "--dry-run" option for the all watchers, document
it (let them have --help). It helps and it is easy.
2a. Provide "--dry-run" also for the autoqa itself, so it wouldn't
actually run the tests, it would just print all the commands
that would be run for the particular hook (so printing lines
like "repoclosure --foo=bar ..." etc).
This is hard, because autotest is assembling the execution
commands from our provided control files/wrappers. I haven't
found any --dry-run support in autotest. Therefore I think this
can't be done. But if it could be, it would be cool.
2b. Allow people to run single hook/test without the need of
autotest-server. That can be most easily done by directly
calling autotest-client. This is just for
development/demonstrative purposes. Just to see if everything
is working ok and if the new test can be deployed on the "real"
autoqa instance.
This can be done as config option (my patch), command-line
option, whatever.
Suggestions, comments, remarks, corrections, anything... welcome.
Great mail Kamil! You're right on by pointing out barriers to entry and
work up plans to address that. Will and I were discussing this topic
earlier this week. One thing that jumped out was that there might
different approaches on how to solve the problems. Which is fine
really, since that can often lead to new ways to solve a problem. But
more than different methods, perhaps it isn't yet clear how we're
expecting people to solve problems for themselves.
With AutoQA so early on it's development, and with FUDCon looming, the
thought occurred that it might be helpful to outline all the ways we
anticipate people will interact with AutoQA. Really, I see this as an
exercise/thought_experiment to outline use cases ... and ensure we have
plans in place to document or address them.
I've started a *rough* outline of what I was thinking at
https://fedoraproject.org/wiki/AutoQA_Use_Cases.
Can you help make sure I have your use cases listed. I'm going to
continue fleshing this out in the next few days. Please feel free to
update the wiki directly, or add thoughts to the Talk page.
Thanks,
James