This patch involves tickets:
https://fedorahosted.org/autoqa/ticket/315 - Create per-item logs for multi-item tests
https://fedorahosted.org/autoqa/ticket/316 - depcheck: extract possible cause of failure
https://fedorahosted.org/autoqa/ticket/318 - Provide access to test documentation
https://fedorahosted.org/autoqa/ticket/298 - test.py - split postprocess_iteration
reporting into standalone methods
https://fedorahosted.org/autoqa/ticket/319 - Create HTML log output if possible
and it is the result of my, jskladan's and vhumpa's work.
I'll the describe the patch in detail here, but I decided not to attach it nor to post
it into reviewboard, because it is simply too large. Also I expect that it will receive a
little more polish in the following days, so it's not absolutely finalized yet. But if
you see anything you would like to change, please, send your comments.
You can display the patch by command:
$ git fetch
$ git diff origin/master..origin/pretty -M
Most important changes:
1. TestDetail class
All the test's outcomes are now stored in a single class TestDetail. It works as a
simple container for test result, summary, output, highlights and a few additional
information. It is easier now to call methods that operate on these information, you just
pass them your TestDetail instance.
By default every test has a default "main" TestDetail instance created in
self.detail. You can access self.detail to change the values of result, summary, etc. For
a simple single-item tests (like rpmlint), this main TestDetail instance is all they need.
For complex multi-item tests (like upgradepath) you need to create a TestDetail instance
for every "report" you want to create. In the case of upgradepath and depcheck
this means creating separate TestDetail for every Bodhi update tested.
2. TestDetail.update_result() method
TestDetail class remembers list of result keywords sorted by importance. If you use
TestDetail.update_result(result) method, it will set the test result only if your provided
test result is more important (i.e. "worse") then the current test result. This
way you can easily handle all error states inside your script and only as the last line
include:
self.detail.update_result('PASSED')
3. self.log() method
We used to do a pretty complicated tasks when printing and saving some output. E.g. you
would print in out on the stderr, and then you would append it to self.outputs and maybe
even self.highlights. Now you just use:
self.log(message, stderr=True, highlight=True)
For multi-item tests (like upgradepath) you can also use self.log() to print to several
TestDetail instances at once.
4. automatic log creation, different terminology
We now have several types of logs (debug, output.log and pretty log). We want to have them
transparent and simple for test authors. Debug log is just for us, the developers, so we
don't mention it. The former output.log was renamed to full.log and we mention it as
little as possible, preferably no at all. And the pretty log is referenced simply as
"log" or "test log" and that's the only one we hope test authors
will work with.
Full log (full.log) is created automatically at the end of every test run. It contains
everything that has been logged via self.log() method (provided that @param
printout==True). It does not contain output of "print foo" messages (that's
just in the debug log).
Test log (aka pretty log) is also created automatically at the end of every test run and
it's populated from main TestDetail instance (self.detail). Except for the case when
the test author has created some test log already manually, then we don't create
another one automatically. Test log is created in HTML and this file is used for reporting
results everywhere (it is either sent or linked).
5. self.post_results() method
There is now a single method for reporting results. It takes care of everything - creating
log file, sending email, sending opt-in email and sending Bodhi comment. The first two are
mandatory and the last two are optional - you have to provide correct parameters.
By default the results are posted automatically at the end of the test run. It means for
simple tests where you don't need anything fancy (like Bodhi comments or opt-in
emails), you don't have to care about reporting results, everything is handled for
you. For more advanced uses you have to call the self.post_results() method. For
multi-items tests like upgradepath or depcheck you'll need to call
self.post_results(test_detail) for every "item" (Bodhi update in our case) you
want to report results for.
6. Test logs are in HTML
Test logs are now in HTML. It will allow us to do pretty formatting, nice highlighting,
etc. When sending as an email, only a plaintext overview header and a link to proper HTML
log is included in the email body.
Example:
http://kparal.fedorapeople.org/autoqa/upgradepath2.html
7. All tests modified
All tests were re-written to the new architecture. Most of them don't take any
advantage of the new log features yet.
rpmlint example:
http://kparal.fedorapeople.org/autoqa/rpmlint.html
Upgradepath and depcheck received more love and they should creater nice per-update logs
with just the relevant info included. Depcheck now tries to filter out irrelevant messages
and keep only the interesting ones in the log.
upgradepath example:
http://kparal.fedorapeople.org/autoqa/upgradepath.html
depcheck example:
http://kparal.fedorapeople.org/autoqa/depcheck.html
We were working on it hard even today. Some code may be still rough. All comments
welcome.
Thanks,
Kamil